Daniel's got a whole thing this week about memory leaks — not the abstract "don't forget to free" lecture, but the actual taxonomy of what breaks in real codebases and how you hunt it down before the OOM killer does it for you. He wants us to start with the anatomy. What a memory leak actually is at the OS level, versus what developers loosely call one. He thinks there are several distinct failure modes that all get lumped together, and he wants them teased apart — genuine unreachable-allocation leaks in manually-managed languages, the malloc without free, the ownership confusion across library boundaries. Then the "logical" leaks in garbage-collected languages, where objects stay reachable but are never used again — the unbounded cache, the listener registered and never deregistered, the global registry that only ever grows, the closure that captures far more scope than anyone intended. Heap fragmentation that looks like a leak in RSS but isn't. And the ones that aren't heap at all — file descriptors, sockets, threads, mmap regions, child processes that never get reaped.
That last category alone is worth the price of admission. People see RSS climbing and assume heap, but I've seen a service brought down by a file descriptor leak that looked identical on a dashboard. Same upward slope.
He also wants the language-specific classics. Python reference cycles and the __del__ problem, plus C-extension leaks the GC can't see. JavaScript detached DOM nodes and closures holding onto old state. Java ClassLoader leaks and ThreadLocals in pooled threads. Go goroutines blocked forever on a channel nobody will ever write to. And Rust's Rc and Arc cycles — leaking safely, which is explicitly allowed.
It is. The Rust book says so right there in chapter fifteen — memory leaks are memory safe. Which sounds like a zen koan until you've debugged one.
Then the detection half, which is what he really cares about. The actual workflow when you suspect a leak. How do you distinguish legitimate growth over time from unbounded growth that eventually kills the box? Real tooling, named and explained with how you invoke and read them — Valgrind, Massif, AddressSanitizer's LeakSanitizer, heaptrack, jemalloc and tcmalloc heap profiling, Python's tracemalloc, objgraph, memray, guppy, Chrome DevTools heap snapshots and the three-snapshot comparison technique, JVM tooling with jmap and Eclipse MAT and the retained size versus shallow size distinction, Go's pprof heap and goroutine profiles, and continuous production profilers like Pyroscope or Parca where you watch allocation flamegraphs drift over days.
That's a lot of tools. But they each catch a different failure pattern, and no single one catches all of them. That's the thing people miss.
Then the OOM side — how Linux picks its victim, what oom_score and oom_score_adj mean, what shows up in dmesg, how to read those logs forensically. The cgroups v2 and container angle — memory.max versus memory.high, memory.pressure, why a container gets killed at its limit while the host has gigabytes free, why exit code 137 tells you almost nothing on its own. Early warning tooling — earlyoom, systemd-oomd, PSI-based alerting. How to wire Prometheus with cAdvisor or node_exporter so you get paged on the slope of the curve rather than after the kill.
The slope. That's the whole game. A single snapshot hides the slope, and the slope is the whole story.
He finishes with the discipline side — regression tests for leaks, soak testing, RAII and smart pointers, weak references, bounded caches, arena allocation. And he wants honest disagreement between us on whether "just restart it nightly" is pragmatic or an admission of defeat.
Oh, I have thoughts on that one. But let's start with the anatomy, because half the arguments about restarting come from not knowing which kind of leak you're dealing with.
Let's do it. So what actually is a memory leak at the OS level?
At the kernel's view, it's almost nothing. The kernel doesn't know what a leak is. It knows pages. A process asks for virtual memory through brk or mmap, the kernel hands over address space, and physical pages get faulted in when the process touches them. The kernel tracks RSS — resident set size — which is how many physical pages are actually backed by RAM right now. But whether the process still intends to use those pages? The kernel has no idea. That's entirely the allocator's problem, and the allocator only knows what the process told it. A leak, in the strict sense, is a block that was malloc'd and the pointer was lost — nobody will ever call free on it. The allocator keeps it marked live, and the kernel keeps the pages hot, forever.
And that's the genuine unreachable-allocation leak. The C classic.
Right. And it's not just "I forgot to call free." The nastier version is ownership confusion across library boundaries. You call into a C library that allocates and returns a pointer, and the documentation is ambiguous about whether you're supposed to free it or the library will. Or the library frees it internally on some code path but not others. Or the allocation happens in one shared object and the free is expected to happen in another, but they were compiled against different allocators. Those bugs can survive for years because they only leak on error paths nobody tests.
And then there's the second category Daniel flagged — the logical leak in a garbage-collected language.
Which is, in some ways, harder. Because every byte is reachable. The GC is doing its job perfectly. The object graph says "this is still in use," and the GC has no basis to disagree. The problem is that "in use" and "should be in use" are different things. The classic is the unbounded cache. You add items, you never evict, the cache grows until it eats the heap. A cache without eviction is a memory leak disguised as a feature.
That's a good line.
It's not mine, but it's right. The listener pattern is the same shape — you register a callback on an event bus and never deregister. The event bus holds a reference, so the object is reachable forever. Global registries, singletons with append-only collections, closures that capture a giant scope when they only needed one field — all the same failure pattern. The memory is reachable, the GC sees it as live, and your heap grows without bound. LeakSanitizer and Valgrind are completely blind to this. They report zero leaks, because technically there are zero leaks.
So you've got a process whose RSS is climbing, and the leak detector says it's clean. That's a special kind of frustration.
And that's where you reach for heap profiling, not leak detection. You need to know what's accumulating, not what's orphaned.
The third category Daniel mentioned is fragmentation.
This one is underappreciated. Glibc's ptmalloc creates per-hread arenas to reduce contention. A handful of live objects can prevent an entire arena from being returned to the OS. The process has forty-five megabytes of heap actually in use, but RSS is sitting at one point three gigabytes. It's not leaking. The allocator just can't give the memory back because the live objects are scattered across arenas and there's no contiguous free region big enough to unmap.
So it draws a leak-shaped graph but it's not a leak.
And the fix isn't "find the bug" — the fix is swap the allocator. jemalloc or tcmalloc. There's a case study from April where swapping glibc for jemalloc cut production memory by forty-seven percent on a bursty async workload. No code change. Just the allocator.
And the fourth category — the non-eap leaks.
File descriptors are the big one. Every open socket, every open file, every pipe, every eoll instance — they all consume kernel memory that doesn't show up in heap profiles at all. A process can have a perfectly flat heap and still grow its RSS through unreclaimed socket buffers. Threads are another — each thread has a stack, typically eight megabytes of virtual space, and if you're spawning threads and never joining them, that adds up. Mmap regions that never get munmapped. Child processes that never get reaped — they sit in zombie state consuming a PID and a slot in the process table, and if you leak enough of them you can't fork anymore.
The zombie process one is sneaky because it doesn't bloat RSS much. It just exhausts a finite resource silently.
And then one day your cron jobs stop running because fork returns EAGAIN and nobody knows why.
Let's go through the language-specifics. Start with Python.
Python's got two distinct problems that interact badly. The first is reference cycles. Two objects that reference each other — the GC can handle those, normally. The cycle detector finds them and collects them. Unless one of them has a __del__ finalizer. Then the GC refuses to collect the cycle because it can't determine a safe finalization order. Those objects sit in gc.garbage forever.
And the second problem?
C extensions. numpy arrays, pandas DataFrames — they allocate memory through malloc at the C level, outside Python's GC visibility. If the Python wrapper object gets collected but the underlying C allocation doesn't get freed — either because of a bug in the extension or because the extension author expected you to call some cleanup method you didn't — you get a leak that tracemalloc will never see.
tracemalloc only traces Python allocations.
Correct. Which is why Bloomberg built memray — it traces both Python and native allocations, outputs flamegraphs, and can show you that your "Python memory leak" is actually a C library allocating and never freeing. That's the first thing to check when tracemalloc says everything is fine but RSS keeps climbing.
JavaScript.
Detached DOM nodes. You remove an element from the DOM tree, but a JavaScript closure somewhere still holds a reference to it. The node can't be garbage collected. The three-snapshot technique in Chrome DevTools is the standard way to find these — take a heap snapshot, do some operations, take another snapshot, do more operations, take a third. Then in the third snapshot's Summary view, filter to objects allocated between snapshots one and two. Those objects should have been freed by snapshot three. If they're still there, they're leaked.
And closures holding onto old state is the other JS classic.
A closure captures its entire scope, not just the variables it uses. If you have a closure inside a loop that references one small thing, but the loop body also had a giant array in scope, that array is captured too. Modern engines have gotten better at optimizing this — V8 does scope analysis to drop uncaptured variables — but it's not perfect, and in complex code it's easy to accidentally retain far more than you meant to.
Java.
ClassLoader leaks are the JVM's signature move. Every time you redeploy a web application, the application server creates a new ClassLoader. If anything from the old ClassLoader is still reachable — a ThreadLocal in a pooled thread, a static collection somewhere, a logging framework that cached a reference — the old ClassLoader and every class it loaded can never be garbage collected. Metaspace grows without bound. Eclipse MAT is the tool here — you load the heap dump, run the Leak Suspects report, and it walks the dominator tree to show you which object is holding the ClassLoader alive.
And the retained size versus shallow size distinction?
Shallow size is the object itself. Retained size is everything that would be freed if this object were collected. A HashMap entry might have a shallow size of forty bytes but a retained size of two gigabytes because it's the only thing keeping a giant cache reachable. The dominator tree in MAT shows you exactly this — you walk up from the biggest retained set until you find the single object responsible.
Go.
Goroutines blocked forever on a channel nobody will ever write to. That's the one. You spin up a goroutine to handle a request, it tries to send on a channel, the receiver exited early, the send blocks forever. The goroutine never terminates. Its stack — which starts at two kilobytes but can grow to a gigabyte — stays allocated. And everything on that stack, every heap object it references, stays reachable. It's a GC root that never goes away. pprof's goroutine profile shows you exactly how many goroutines are sitting at which line of code. If you see a number that only goes up, you've got a leak.
Go one point twenty-six added that experimental goroutineleak profile, right?
Yes — it uses GC reachability to identify goroutines blocked on primitives that are themselves unreachable. So a goroutine waiting on a channel that nothing else can possibly write to. It's still experimental, you enable it with GOEXPERIMENT equals goroutineleakprofile, but it's a huge step forward from manually staring at goroutine stacks.
Rust.
Rc and Arc cycles. Rust explicitly documents that memory leaks are safe. An Rc cycle where two nodes point to each other will never be dropped — the reference count never hits zero. The ownership system prevents use-after-free and double-free, but it doesn't prevent "never-free." Weak references are the escape hatch — Rc downgrade to Weak, Arc downgrade to Weak, and the cycle can be broken. But it's opt-in. You have to know to do it.
So Rust gives you the tools to avoid leaks but doesn't make it impossible to create them.
Which is the right tradeoff for a systems language. If you need guaranteed leak freedom, you need a language with a tracing GC that can collect cycles. But Rust's position is: we'll prevent the dangerous memory bugs, and for leaks we'll give you the tools and get out of your way.
Let's move to detection. Someone suspects a leak in production. What's the first step?
Confirm the trend. Don't look at a single RSS snapshot. Watch it over hours. If RSS climbs past where the workload should have settled — if it's still going up at hour six when the cache should be warm and the connection pools should be full — you've got something. The slope matters more than the absolute value. And you need to rule out the look-alikes before you start debugging. Is this actually a leak, or is it fragmentation? Is it a cache that's legitimately filling? Is it the GC heap not returning memory to the OS because the runtime decided not to? Misdiagnose this and you'll tune an allocator for hours chasing a bug that's actually in your code.
And if it is a leak, the tool depends on the language.
For C and C++, the staging environment gets Valgrind with leak-check equals full. It classifies leaks as definitely lost or possibly lost, with allocating call stacks. The problem is the overhead — ten to fifty times slowdown. You can't run that in production. For CI, AddressSanitizer with detect_leaks equals one is about two times overhead, which is manageable. And heaptrack traces every allocation with stack traces and gives you a flamegraph you can explore interactively. That's the workflow from the case study in March — Valgrind to find the leak category, ASan to narrow it down in CI, heaptrack to visualize the allocation site.
And for production, jemalloc profiling.
jemalloc dash dash prof enables heap profiling with far lower overhead than Valgrind. You set MALLOC_CONF equals prof colon true, and you can capture heap dumps from a live process. It won't tell you about reachable unbounded growth — it's still looking at what's allocated, not what's logically leaked — but for native leaks it's the production-safe option.
Python's toolchain.
tracemalloc is in the standard library since Python three point four. You call tracemalloc dot start, let it run, take a snapshot, take another snapshot later, and call compare_to between them. It shows you the top allocations by file and line number, ranked by the difference. That's your first stop. If tracemalloc doesn't show anything but RSS is climbing, you've got a C-extension leak and you need memray. objgraph is useful for reference cycle hunting — it can show you the backreferences to an object, the chain that's keeping it alive. And gc dot DEBUG_UNCOLLECTABLE will show you the objects stuck in uncollectable cycles because of __del__.
The Chrome DevTools three-snapshot technique you mentioned.
That's the gold standard for JavaScript in the browser. For Node.js, you can take heap snapshots programmatically with the v8 module, or use the dash dash inspect flag and connect Chrome DevTools to a running Node process. Same technique applies — take snapshots, diff them, find what's accumulating.
JVM?
jmap dash dump colon live, format equals b, file equals heap dot hprof, and the process ID. That gives you a heap dump. Load it into Eclipse MAT. The Leak Suspects Report automates most of the analysis — it identifies the largest accumulated objects, walks the dominator tree, and gives you a shortlist of suspects. The dominator tree is the key concept: it shows which objects, if removed, would free the most memory. You walk up from the biggest retained sets and you find the one HashMap or ThreadLocal or ClassLoader that's anchoring everything.
And Go.
net slash http slash pprof is built in. You expose slash debug slash pprof slash heap and slash debug slash pprof slash goroutine. For the heap, you take a baseline profile, wait, take another, and run go tool pprof dash base baseline dot pb dot gz leak dot pb dot gz. That shows you the allocations that happened between the two snapshots. For goroutines, debug equals one gives you aggregated stacks — how many goroutines are at each call site. If you see a number that only goes up, you know where they're blocked. debug equals two dumps every goroutine individually, but be careful — on a process with a hundred thousand goroutines, that's a stop-the-world pause you'll feel.
And the continuous profilers — Pyroscope, Parca.
These are always-on. They sample at something like a hundred hertz with about one percent overhead, store the profiles with timestamps, and let you query them over time ranges. The killer feature is differential profiling — you deploy, wait, and compare the flamegraph from before the deploy to after. New allocations show up as a different color. Parca uses eBPF so it doesn't need instrumentation — it profiles at the host level. If you've got a leak that takes three days to become visible, you can zoom out to a seventy-two-hour window and watch the allocation flamegraph drift. That's a completely different workflow from "attach a debugger and hope."
Let's get to the OOM side, because that's usually how you find out you have a problem.
The OOM killer wakes up when the kernel can't satisfy an allocation. It runs oom_badness on every process. The score is roughly RSS plus swap entries plus page table pages, plus oom_score_adj scaled by total pages divided by a thousand. Highest score gets killed. oom_score_adj ranges from minus a thousand — completely immune — to plus a thousand — guaranteed victim. Kernel threads and PID one are never killed.
And the process whose allocation failed isn't necessarily the one that gets killed.
Almost never, in practice. The allocation fails in process A, the OOM killer looks around, and process B has the highest score. Process B dies. Process A continues. This is how a leak in your web server gets your monitoring agent killed — exactly when you most want monitoring to be running.
Reading the dmesg output.
Four sections. First, the trigger line — which process's allocation failed. Second, the memory state dump — active_anon, active_file, slab_reclaimable, free. If active_file plus inactive_file is near zero, the page cache is exhausted and the system is thrashing. If slab_unreclaimable is dominating, you might have a kernel memory leak. Third, the task table — every process with its RSS, anon-rss, file-rss, page table bytes, and oom_score_adj. Fourth, the kill line — which process was actually killed and its memory breakdown.
And the forensic reading — how do you work out which process was actually at fault?
You look at the task table and find the process with the highest RSS that has a less-negative oom_score_adj than the victim. That's your likely culprit. The victim was just the biggest thing that wasn't protected. If your database has oom_score_adj minus five hundred and your web server has zero, the web server dies even if the database is the one that's leaked to sixty gigabytes.
The container angle — cgroups v2.
Three gates. memory.low is protection — the kernel won't reclaim below this unless it's desperate. memory.high is throttling — when the cgroup hits this, allocations stall and the process gets throttled. memory.max is the hard limit — hitting this triggers the cgroup OOM killer. That's a separate killer from the global one. A container gets killed at its limit while the host has gigabytes free because the cgroup OOM killer only looks at the cgroup's own memory, not the host's. memory.events tracks high, max, oom, and oom_kill counters monotonically. memory.pressure gives you PSI — pressure stall information — with some and full averages over ten, sixty, and three hundred second windows. If full is rising, every task in the cgroup was simultaneously blocked on memory. No useful work was happening at all.
Exit code 137.
Tells you the process received SIGKILL. That's it. It doesn't distinguish between a cgroup OOM kill, kubectl delete pod, or someone running kill dash nine. You have to check dmesg or journalctl dash k for the Memory cgroup out of memory colon prefix to confirm it was actually an OOM kill. Without that check, you're flying blind.
Early warning tooling.
earlyoom is a userspace daemon that monitors oom_score and kills processes before the kernel OOM killer fires, based on configured memory and swap thresholds. systemd-oomd uses cgroups v2 and PSI — it monitors pressure and kills descendant cgroups when pressure exceeds thresholds. The advantage of systemd-oomd is that it uses the same metrics the kernel uses to decide when the system is actually suffering, rather than a simple percentage threshold.
And the Prometheus side — what do you actually alert on?
container_memory_working_set_bytes from cAdvisor. That's the metric the OOM killer counts — anonymous memory plus swap cache, minus inactive file pages. It's what matters for OOM. Alert on the slope, not a threshold. deriv of container_memory_working_set_bytes over thirty minutes, sustained above zero for hours. For Go specifically, alert on go_goroutines greater than go_goroutines offset one hour times one point two — more than twenty percent goroutine growth over an hour. That catches goroutine leaks before they become memory leaks.
So you get paged when the curve starts bending up, not when the process is already dead.
The kill is the last data point. By then you're doing forensics, not monitoring.
Okay. The discipline side. How do you write a regression test for a leak?
For native code, you run ASan or LSan in CI with halt_on_error equals one. Build with dash fsanitize equals address, run tests with ASAN_OPTIONS equals detect_leaks equals one colon halt_on_error equals one. If a test leaks, the build fails. For Rust, there's a crate called navian-memcheck published last month — it asserts that memory plateaus after warmup as a cargo test assertion. It catches reachable unbounded growth that Valgrind and LeakSanitizer miss, because the memory is still referenced. For other languages, you write a soak test — run the service under production-like load for hours, sample RSS and open file descriptors over time, fail if RSS climbs past a budget or FDs grow without bound.
The prevention patterns.
RAII and smart pointers for C++. Weak references for cycles — Python's weakref, Rust's Weak, Java's WeakReference. Bounded caches with eviction — LRU, TTL, something that guarantees the cache stops growing. Context managers and defer for resource cleanup. Arena allocation — allocate from a big block and free the whole thing at once, which eliminates individual free bugs entirely. And for Go, goleak dot VerifyTestMain in CI to catch goroutine leaks in tests.
Which brings us to the restart question. Daniel wants us to disagree on this.
I'll start. "Just restart it nightly" is an admission of defeat. It hides the signal. The next person to touch that service inherits a problem with no graph pointing at it. The container restart cycle — Kubernetes brings it back, RSS drops to baseline, the leak resumes, the cycle repeats — makes this look like an intermittent crash when it's actually a deterministic leak on a timer. And a leak is also a correctness smell, not only a resource one. Something in your program's logic is wrong. The fact that you can outrun it with a cron job doesn't make it less wrong.
I don't think that's quite right.
Okay. Tell me why.
Because "correctness" isn't the only axis. There are batch workers, scrapers, one-off data pipeline stages where the process runs for six hours, leaks two hundred megabytes, and then exits. A scheduled restart at hour four costs you nothing and prevents the OOM kill entirely. That's not hiding a problem — that's acknowledging that the cost of fixing the leak exceeds the cost of the restart. Some military missile software literally provisions double the expected maximum leak and accepts it. That's an engineering choice, not laziness.
Military missile software has a fixed mission duration. Most of the services I've seen restarted nightly are web servers that someone intends to keep running for years.
Sure, and for a long-lived service I agree with you. But the absolutist position — "all leaks must be fixed" — ignores that some leaks are in dependencies you don't control, in C extensions you can't patch, in allocator behavior you can't change without a rebuild that requires approval from a team that no longer exists. In those cases, restarting is a legitimate mitigation while you work on the real fix, which might take months.
I'll grant that for dependency leaks. If the bug is in a library you can't upgrade, restarting is a reasonable stopgap. What I object to is restarting as the permanent solution. Because what happens is the restart becomes institutionalized, the original engineer leaves, and two years later someone triples the traffic and the leak outpaces the restart window and now you've got a production incident at three AM and nobody knows where the leak is because the graphs have been flat for two years thanks to the nightly restart.
That's a process problem, not a technical one. The restart should be documented as a mitigation with a linked bug ticket. If your team doesn't do that, the problem is your team, not the restart.
Fair. But I've never seen a team that actually does that consistently.
Then we agree on the principle and disagree on whether humans can be trusted to execute it.
I think that's about right. I'm less optimistic about human processes than you are.
You're a donkey who reads kernel source for fun. You were never going to be optimistic about human processes.
That's... not wrong.
Hilbert: We ran a VAX cluster in eighty-five that had a memory leak in the terminal driver. DEC wasn't going to fix it — the patch cycle was eighteen months and we weren't a big enough customer to matter. So we put a cron job in to bounce the terminal controller process every Sunday at four AM. Ran like that for three years. Nobody forgot why the cron job was there because the guy who wrote it, Marty, put a comment in the crontab that said "DEC terminal driver leaks like a sieve, do not remove." Marty was not subtle.
Did anyone ever remove it?
Hilbert: New sysadmin did, in eighty-eight. Took about six hours for the first terminals to lock up. He put it back. Wrote his own comment under Marty's. "Marty was right." That machine ran until ninety-two with that cron job. Nobody ever fixed the driver.
So you're on Corn's side.
Hilbert: I'm on the side of knowing which problem you're solving. The problem wasn't "fix the terminal driver." The problem was "the terminals lock up." The cron job solved the actual problem. The driver bug was somebody else's problem, and DEC never did fix it. Marty checked the patch notes for three years. Nothing.
Three years of patch notes.
Hilbert: He was thorough.
I still think for code you own, you fix the leak.
Hilbert: Sure. If you own it. Marty didn't own the terminal driver. Most people restarting containers tonight don't own the thing that's leaking. They own the outage. Different problem.
And that's the distinction. Own the leak, fix the leak. Don't own the leak, own the mitigation.
I can live with that framing. I still think the mitigation should come with a very loud comment.
Hilbert: Marty would agree.
This has been My Weird Prompts. Thanks to our producer Hilbert Flumingtop.
Find us at my weird prompts dot com, or email us at show at my weird prompts dot com. We'll be back soon.
