You already know: Python stores every value as an object and lists hold pointers to those objects; NumPy is fast because it packs numbers side by side (Python internals, sections 2–3).
Why this matters for Atlas. The fastest parts of AI systems — inference engines, CUDA kernels, tokenizers — are written in C++ or similar languages. In C++ you decide where data lives and when it's freed. That control is where the speed comes from, and also where the bugs come from.
All C++ examples compile with
clang++ -std=c++17 -O2 file.cpp.
1. Stack and heap: two places to put data
The question. When a function creates a variable, where does that memory come from, and when does it go away?
Start from nothing. A running program has two main areas for data:
- The stack — a scratchpad for each function call. When a function starts, it pushes its local variables; when it returns, they're popped off instantly. Allocation is just moving a pointer: extremely fast. But sizes must be known at compile time and the space is small (often a few MB).
- The heap — a large pool for data whose size or lifetime isn't known in advance. You request memory (
new, or a container likestd::vectordoes it for you) and it lives until someone frees it. More flexible, slower to allocate.
Real-life analogy. The stack is your desk: quick to put things on, cleared when you finish the task. The heap is a storage unit: rent any size, keep it as long as you like, but you must remember to give it back.
Worked example.
#include <iostream>
#include <vector>
int sum_small() {
int numbers[4] = {1, 2, 3, 4}; // on the stack: size known, freed on return
int total = 0;
for (int n : numbers) total += n;
return total;
}
int main() {
std::vector<int> big(1'000'000, 1); // the vector object is on the stack,
// its 1M ints live on the heap
long total = 0;
for (int n : big) total += n;
std::cout << sum_small() << " " << total << "\n"; // 10 1000000
} // big's heap memory is freed here, automatically
Common traps
- Returning a pointer to a local (stack) variable: the memory is gone once the function returns.
- Huge arrays on the stack (
int x[10'000'000];) cause a stack overflow. Usestd::vector.
In production / interviews. Allocation in hot loops is a classic latency killer in inference servers; engineers pre-allocate buffers once and reuse them. Interview question: "Why is allocating on the stack faster than on the heap?"
The same idea, one level up: memory pools. GPU memory allocation (cudaMalloc) is far slower than a stack push, so frameworks don't call it per tensor. PyTorch's CUDA caching allocator keeps freed blocks and hands them out again. That's why nvidia-smi keeps showing memory "in use" after your tensors are gone, and why PyTorch reports both allocated and reserved memory. Inference engines go further and reserve one large pool up front, then carve it into fixed-size blocks for the KV cache (KV cache engineering). The principle is the one this section teaches: allocate rarely, reuse often.
From production
A C++ tokenizer service allocates a new std::vector<int> for every request's token IDs, and p99 latency spikes under load. Profiling shows heavy time in malloc/free.
What's the standard fix?
Show the answer
C. Reuse a per-thread buffer: reserve() it once, clear() between requests, so capacity is kept and no new allocation happens — clear() keeps the vector's capacity, so a reused buffer stops hitting the allocator. A huge stack array risks stack overflow, and list allocates per node, which is worse.
2. Pointers and references: addresses, not copies
The question. How do you let a function work on a 1 GB array without copying it?
Start from nothing. Every byte of memory has an address (a number). A pointer is a variable that stores an address. A reference is an alias — another name for an existing object, which can't be null and can't be re-pointed.
Passing a large object by value copies it. Passing by reference (const std::vector<float>&) passes only its location — 8 bytes, however big the data. Python does something similar automatically: all names are references (Python internals, section 2).
Real-life analogy. Giving someone your home address (pointer) instead of building them a copy of your house.
Worked example.
#include <iostream>
#include <vector>
float sum_copy(std::vector<float> v) { // copies every element
float s = 0; for (float x : v) s += x; return s;
}
float sum_ref(const std::vector<float>& v) { // no copy, read-only
float s = 0; for (float x : v) s += x; return s;
}
void scale(std::vector<float>& v, float k) { // no copy, can modify
for (float& x : v) x *= k;
}
int main() {
std::vector<float> w(4, 1.0f);
int a = 7;
int* p = &a; // p holds a's address
*p = 9; // write through the pointer
scale(w, 2.0f);
std::cout << a << " " << sum_ref(w) << " " << sum_copy(w) << "\n"; // 9 8 8
}
Common traps
- Dangling pointers: pointing to memory that has been freed. Reading it is undefined behaviour — it may "work" in testing and crash in production.
- Null pointers: dereferencing
nullptrcrashes. Prefer references when "no object" isn't a valid case.
In production / interviews. Accidental copies of big tensors or strings are a common silent slowdown in C++ ML code. Code reviewers look for const& on large parameters.
The program prints 9 8 8. Writing through p changed a to 9. scale doubled the four 1.0s in place (no copy), so both sums are 8: sum_ref read the original, and sum_copy summed a private copy of it.
The Python connection. When you pass a NumPy array or a PyTorch tensor from Python into a C++ extension (through pybind11 or PyTorch's C++ API), the well-written path passes a reference to the existing buffer, not a copy. That's why torch.from_numpy(arr) is instant and shares memory with arr: change one and the other changes. By contrast, torch.tensor(arr) copies. Same data, very different cost and aliasing behaviour, exactly the value-vs-reference distinction from this section.
Quick check
t = torch.from_numpy(arr) followed by arr[0] = 99. What is t[0]?
Show the answer
A. 99, because from_numpy shares arr's memory buffer instead of copying it — from_numpy wraps the existing buffer (a reference), so writes through either name are visible through the other. torch.tensor(arr) makes an independent copy.
3. RAII and smart pointers: let scope clean up for you
The question. If you must free every heap allocation yourself, how do you avoid leaks when code returns early or throws an error?
Start from nothing. C++'s answer is RAII — Resource Acquisition Is Initialization. Tie every resource (memory, a file, a GPU buffer, a lock) to an object on the stack. The object's constructor acquires the resource; its destructor releases it. Destructors run automatically when the object goes out of scope — on normal return and when an exception passes through.
Smart pointers apply RAII to heap memory:
std::unique_ptr<T>— one owner; freed when the owner goes away. Use this by default.std::shared_ptr<T>— shared ownership with a reference count, freed when the last owner goes away (the same idea as Python's refcount).
Real-life analogy. A hotel key card that deactivates itself when you check out — you can't forget to return it.
Worked example.
#include <iostream>
#include <memory>
struct GpuBuffer { // stand-in for any resource
size_t bytes;
explicit GpuBuffer(size_t b) : bytes(b) { std::cout << "alloc " << bytes << "\n"; }
~GpuBuffer() { std::cout << "free " << bytes << "\n"; }
};
void step(bool fail) {
auto buf = std::make_unique<GpuBuffer>(1024);
if (fail) return; // early return: still freed
std::cout << "using buffer\n";
} // freed here otherwise
int main() {
step(false);
step(true);
auto shared = std::make_shared<GpuBuffer>(64);
auto other = shared; // two owners
std::cout << "owners: " << shared.use_count() << "\n"; // 2
}
Common traps
- Raw
new/deletein application code. Preferstd::make_uniqueand containers. shared_ptrcycles (A owns B, B owns A) never free — the same problem Python's cycle collector solves.
In production / interviews. Inference servers wrap CUDA memory, streams and file handles in RAII types so error paths can't leak GPU memory. Interview question: "What does RAII guarantee, and what happens to a unique_ptr when an exception is thrown?"
The output is alloc 1024, using buffer, free 1024 for the normal call. The early-return call prints alloc 1024 then free 1024, with no "using buffer", and it still freed. Then alloc 64, owners: 2, and finally free 64 when both shared owners go out of scope at the end of main.
You already use RAII in Python. with open(path) as f:, with lock: and with torch.no_grad(): are Python's version: the context manager acquires on entry and releases on exit, including when an exception passes through (Decorators and context managers). C++ gets it from destructors, Python from __enter__ / __exit__. Either way, cleanup is tied to a scope, not to remembering a line of code on every path.
From production
A C++ function opens a CUDA stream, allocates a device buffer with a raw pointer, and calls cudaFree at the end. A new validation check in the middle now returns early on bad input.
What problem appears, and what's the idiomatic fix?
Show the answer
D. The early return skips cudaFree, so every bad request leaks GPU memory; wrap the buffer in an RAII type (unique_ptr with a custom deleter) so it's freed on every exit path — Manual cleanup at the end of a function is skipped by early returns and exceptions. RAII ties the free to scope exit, so every path, including future ones, releases the buffer.
4. Move semantics: hand over, don't copy
The question. A function builds a 500 MB tensor and returns it. Does returning it copy 500 MB?
Start from nothing. Many objects are small "handles" (pointer + size) to big heap data. Copying the handle and the data is expensive. Moving copies just the handle and leaves the source empty — the big data never moves in memory. C++ moves automatically when the source is a temporary (like a return value) and lets you request it with std::move when you're done with a variable.
Real-life analogy. Selling a house by handing over the keys, instead of building an identical house for the buyer.
Worked example.
#include <iostream>
#include <string>
#include <utility>
#include <vector>
std::vector<float> make_embeddings(size_t n) {
std::vector<float> v(n, 0.5f);
return v; // moved (or elided), never deep-copied
}
int main() {
auto emb = make_embeddings(1'000'000);
std::vector<std::string> queue;
std::string prompt = "Explain virtual memory";
queue.push_back(std::move(prompt)); // steal the string's buffer
std::cout << emb.size() << " queued=" << queue[0]
<< " left-behind-size=" << prompt.size() << "\n";
// prompt is valid but unspecified (here: empty) — don't rely on its value
}
Common traps
- Using a variable after
std::moveas if it still held its value. std::moveon aconstobject silently copies (you can't steal from something you promised not to change).
The program prints 1000000 queued=Explain virtual memory left-behind-size=0. The million-float vector came back without a deep copy. The prompt's character buffer was handed to the queue, leaving prompt empty (valid, but don't rely on its contents).
In production / interviews. Request queues, batchers and token buffers in inference servers move data between stages instead of copying it: a request's prompt and token IDs are moved into the scheduler's queue, then into a batch. On the Python side, the analogy is passing a tensor to the next stage instead of calling .clone(). Clone only when you genuinely need an independent copy. Interview question: "What does std::move actually do?" (Answer: nothing at run time by itself. It's a cast that allows a move constructor to steal the resources.)
Quick check
After queue.push_back(std::move(prompt));, which statement about prompt is correct?
Show the answer
B. It's in a valid but unspecified state (often empty): you may assign to it or destroy it, but don't read its old value — Moved-from standard library objects stay valid but unspecified. Reassigning or destroying them is fine. Reading them for their old value is a logic bug.
5. Cache locality: the hidden speed limit
The question. Two loops do the same additions. Why can one be several times slower?
Start from nothing. Memory isn't one speed. Roughly: registers (instant) → L1 cache (~1 ns) → L2/L3 (a few ns) → main RAM (~100 ns). The CPU fetches memory in cache lines of 64 bytes. When you read one float, its 15 neighbours come along for free. Code that walks memory in order gets those neighbours "for free"; code that jumps around pays the RAM price on almost every access.
A 2-D array in C/C++/NumPy is stored row by row (row-major). Walking along a row is sequential; walking down a column jumps a whole row each step.
Real-life analogy. Reading a book page by page versus reading the first word of every page, then the second word of every page.
Worked example (runs in Python via NumPy, same memory rules).
import time
import numpy as np
a = np.ones((4000, 4000), dtype=np.float32) # row-major (C order)
t = time.perf_counter()
rows = sum(float(a[i, :].sum()) for i in range(a.shape[0])) # contiguous slices
t_rows = time.perf_counter() - t
t = time.perf_counter()
cols = sum(float(a[:, j].sum()) for j in range(a.shape[1])) # strided slices
t_cols = time.perf_counter() - t
assert rows == cols
print(f"row-wise {t_rows:.3f}s, column-wise {t_cols:.3f}s, "
f"column/row = {t_cols / t_rows:.1f}x")
Math. Number of cache lines touched when reading N floats: sequentially ≈ N × 4 bytes / 64 bytes = N/16; with a stride of one row ≈ N (a new line for every element, unless they're already cached). Up to 16× more memory traffic for the same work.
Common traps
- "Array of structs" (
struct {x, y, z, label}per point) when you only readx— you drag the rest through the cache. "Struct of arrays" (allxtogether) is often faster. - Linked lists for hot data: every node is a jump to a random address.
In production / interviews. This is the same idea GPU engineers obsess over as memory coalescing (D13), and the reason FlashAttention restructures attention to stay in fast memory. Interview question: "Why is iterating a matrix column-by-column slower in C order, and how would you fix it?"
Quick check
A C++ inference server returns a reference to a local std::vector from a function and occasionally crashes. Why?
Show the answer
C. The local vector is destroyed when the function returns, so the reference dangles; return by value (moved or copy-elided) instead — Stack objects die at scope exit. Returning by value costs nothing extra thanks to move semantics and copy elision.
Check yourself
- Recall: Where does a
std::vector's data live — stack or heap? - Apply: A function takes
std::vector<float> dataand only reads it. What should the parameter type be? - Predict: In
step(true)from section 3, is the buffer freed?
Answers
- The vector object (pointer, size, capacity) can be on the stack; its elements are on the heap.
const std::vector<float>&— no copy, read-only.- Yes. The
unique_ptr's destructor runs on the early return and frees it.
Exercise
Write a C++ function top_score(const std::vector<float>& scores) that returns the index of the highest score without copying the vector, and a ScoredLesson type held in a std::vector that is filled using push_back(std::move(...)). Measure, using std::chrono, the difference between summing a 4000×4000 row-major array by rows and by columns.
Solution sketch
#include <chrono>
#include <iostream>
#include <string>
#include <vector>
size_t top_score(const std::vector<float>& s) {
size_t best = 0;
for (size_t i = 1; i < s.size(); ++i) if (s[i] > s[best]) best = i;
return best;
}
struct ScoredLesson { std::string title; float score; };
int main() {
std::vector<ScoredLesson> lessons;
std::string t = "Virtual memory";
lessons.push_back({std::move(t), 0.92f});
const size_t n = 4000;
std::vector<float> a(n * n, 1.0f);
auto time_it = [&](bool by_rows) {
auto start = std::chrono::steady_clock::now();
double total = 0;
for (size_t i = 0; i < n; ++i)
for (size_t j = 0; j < n; ++j)
total += by_rows ? a[i * n + j] : a[j * n + i];
auto ms = std::chrono::duration<double, std::milli>(std::chrono::steady_clock::now() - start).count();
std::cout << (by_rows ? "rows " : "cols ") << ms << " ms (" << total << ")\n";
};
time_it(true);
time_it(false);
std::cout << "best index: " << top_score({0.1f, 0.9f, 0.4f}) << "\n"; // 1
}
This unlocks:
- Operating systems — who actually hands out stack and heap memory, and what "virtual" addresses are.
- D13 GPU memory hierarchy — the same cache-locality rules, with far bigger penalties.
- D12 KV cache — preallocating and reusing big buffers.
Sources: cppreference — std::unique_ptr, std::move, value categories; C++ Core Guidelines (R.1 RAII, F.16 parameter passing); Ulrich Drepper, What Every Programmer Should Know About Memory (2007).