You have a server. It's up. Everything works. And it's dying by inches. Daniel sent us a case study from his own homelab this week that might be the cleanest example of this I've ever seen. Let me read you what he wrote.
He says — I've got an MCP gateway running in a container. It spawns tool servers as child processes. One of those tool servers shells out to headless Chromium for web scraping. The gateway process never calls wait on any of its children. Ever. Three days later I SSH in and find two hundred thirty-three live Chromium processes holding about two point four gigabytes of RAM, plus two hundred fifty-four zombies. This is on a box with thirty gigs total, already into swap. Nothing crashed. Nothing alerted. Every tool worked perfectly the entire time. I only found it because I went looking for where the memory had gone.
He adds — the monitoring stack, Prometheus, Grafana, cAdvisor, node-exporter, was running and healthy the whole time. It collected all the data. Nobody had written the alert. And then he asks us to explain how process reaping actually works, why PID one doesn't save you inside containers, and what the real defenses are. So let's talk about the quietest way a Linux system can fail, and why your monitoring stack probably won't tell you.
The thing that gets me about this story — two hundred thirty-three Chromium processes. That's not a slow leak. That's a hemorrhage. And nobody noticed because the service kept responding. Health check passed. Uptime green. Meanwhile the kernel's sitting there like a librarian who's been handed two hundred fifty-four books with no return date and just... shelved them.
A librarian who never speaks.
The kernel does not complain. That's the design. And it's a good design, for what it was designed for. But let's start with what the kernel actually does when a process dies, because the answer is surprisingly minimal.
So let's get into the kernel mechanics. The details matter here.
When a child process exits, it doesn't vanish. It sits in a state the kernel calls — and this is the actual constant in the source — EXIT_ZOMBIE. The kernel frees everything the process owned. All its memory. All its file descriptors. All its network sockets. Gone. What remains is the task struct, which is the kernel's internal bookkeeping entry for that process, and inside it the exit code. That's it. A few hundred bytes. The zombie is a receipt waiting to be collected.
A receipt. So it's not undead. It's a Post-it note.
That's exactly what it is. The parent process created this child. The parent has the right to ask, how did my child die? Did it exit cleanly? Was it killed by a signal? The kernel preserves that answer until the parent calls wait, or waitpid, or waitid — any of the wait family of system calls. The waitpid man page is explicit about this. The kernel retains only the exit status and resource usage data. Everything else is gone.
So the thing people get wrong — zombies consume memory. They don't.
They don't. They consume a process table slot. And process table slots are finite. The default maximum PID on most Linux systems is thirty-two thousand seven hundred sixty-eight. That's kernel dot pid max. You can raise it, but it's not infinite. Run out of slots, and fork starts failing. No new processes. That's a very bad day.
And two hundred fifty-four zombies is nowhere near that limit, which is why nothing broke.
Right. Two hundred fifty-four is a warning, not a catastrophe. But it tells you something is wrong. Some parent process is not fulfilling its contract. And that's the word I'd use — contract. Fork creates the child. Wait collects it. Those two calls are a pair. If you never call wait, the kernel holds that receipt forever.
Okay, so that's a zombie. What's an orphan?
An orphan is a child whose parent died first. The parent exits while the child is still running. The kernel doesn't leave the child parentless — it re-parents it to PID one. On a full Linux system, PID one is init. Systemd, or whatever init system you're running. And init is designed to reap. Its main loop calls wait. It collects orphans automatically. This is the design assumption that works beautifully on bare metal.
So on bare metal, if I fire up a background process from a shell script and the script exits, systemd just... adopts it?
The shell exits, the child gets re-parented to PID one, and systemd calls wait when the child eventually exits. The receipt gets collected. No zombie. It's a elegant piece of design that's been in Unix since the 1970s. The assumption is baked in at the architecture level — PID one is special. PID one has responsibilities. And for forty years, on every server, PID one was init. That assumption held.
And inside a container?
Inside a container, PID one is whatever the ENTRYPOINT or CMD is. Your Node app. Your Python script. Your Go binary. Most applications have no idea they've inherited a reaping duty. They never call wait. They don't even know they're PID one. So orphans accumulate as zombies. The parent died, the kernel re-parented to PID one, and PID one is a Flask application that's just sitting there serving HTTP and has no concept of process reaping.
This is why Docker's init flag exists.
Yes. Docker added the dash dash init flag in version one point thirteen, January twenty seventeen. It injects a tiny program called tini as PID one inside the container. Tini is about a hundred lines of C. Its entire job is to loop on waitpid with the WNOHANG flag, reaping any children that have exited, and forwarding signals to the main process. That's it. A hundred lines. And it solves the problem completely.
A hundred lines of C that prevent a two point four gigabyte memory leak.
I mean, when you put it that way, it's almost embarrassing how little code it takes.
And the Phusion blog post from twenty fifteen — this was one of the first people to really document this, right?
January twenty fifteen. They were running Ruby on Rails apps inside Docker containers using Unicorn as the app server. Unicorn spawns worker processes. The master process never called wait on its workers. Inside a container, Unicorn was PID one. Workers would exit, get re-parented to the Unicorn master, and the master — which was never designed to be PID one — just ignored them. Zombies everywhere. That blog post was a wake-up call for a lot of people.
So the container twist is: the kernel's design assumes PID one is competent. Inside a container, that assumption breaks.
Completely. And it's not the kernel's fault. The kernel is doing exactly what it was told. The contract says the parent must call wait. The parent isn't calling wait. The kernel doesn't escalate. It just keeps the books.
Now here's the distinction Daniel's story really drives home, and I want to sit on this for a second. A leaked zombie costs a PID table slot. A leaked live process costs real memory. And the parent that doesn't reap often also doesn't kill.
This is the part that makes the MCP gateway case so much worse than a zombie problem. The gateway spawned tool servers. The tool servers spawned Chromium. The tool servers exited. Chromium didn't. It kept running. The gateway never called wait on the tool servers, so they became zombies. But Chromium — the grandchild — was orphaned, re-parented to the gateway, and the gateway had no idea it was now responsible for two hundred thirty-three live browser processes.
And each one holding memory. File handles. Possibly network connections.
Headless Chromium is not lightweight. A single instance can easily hold eighty to a hundred and twenty megabytes depending on what page it's rendering. Two hundred thirty-three of them — you do the math. Two point four gigs of RAM just sitting there, doing nothing, serving no purpose, because nobody told them to stop.
The gateway thought it was an API server. The kernel thought it was a process supervisor. Nobody told the gateway.
And that mismatch is the whole problem. Software that spawns children needs to know it spawned children. If it doesn't track them and reap them, the kernel will keep them around — either as zombies if they exited, or as live processes if they didn't. It's like hiring contractors to do a job, never checking if they finished, and then being surprised when they're still sitting in the break room three days later drinking your coffee.
And the break room only has so many chairs.
Eventually you run out of chairs. Or in this case, memory.
There's actually a mechanism for this. The prctl system call with PR_SET_CHILD_SUBREAPER.
Yes. You can mark a process as a subreaper. When a descendant of that process is orphaned, it gets re-parented to the nearest ancestor that's a subreaper, rather than all the way up to PID one. Systemd uses this. It lets you create reaping hierarchies. But again — your Node app isn't calling prctl. Most developers don't even know this exists.
I didn't know it existed until this week.
Most people don't. It's a fairly obscure kernel feature. But it's been there since Linux three point four. It's the kind of thing that only matters when you're deep in the weeds of process management.
So that's the theory. Now let me tell you about a server that lived this theory for three days without anyone noticing.
Let's walk through the timeline. Day one. The MCP gateway starts up. It's handling requests. A tool server gets spawned to do some web scraping. That tool server spawns Chromium. The scrape finishes. The tool server exits. Chromium — well, Chromium might still be running. The gateway never collects the tool server's exit status. Zombie number one.
And Chromium is now an orphan.
Re-parented to the gateway. Still running. Still holding memory. Day one ends. Maybe a dozen zombies, maybe a dozen live Chromium processes. The box has thirty gigs of RAM. Nobody notices a few hundred megabytes.
Day two.
More requests. More tool servers. More Chromium instances. The gateway is spawning and forgetting, spawning and forgetting. The process table is filling up with zombies. The live Chromium count is climbing. Swap usage starts to tick up. But the gateway is responding. Every tool works. The health check endpoint returns two hundred OK.
And the monitoring stack — Prometheus scraping, Grafana dashboards, cAdvisor tracking container metrics, node exporter pushing host stats — all of it is green.
All green. And here's the thing. The data was there. cAdvisor was tracking the container's memory usage. Node exporter was tracking system memory and swap. The number of processes — you can get that from node exporter. The data was being collected every fifteen seconds. It was sitting in Prometheus. Nobody had written the alert.
Day three.
Two hundred thirty-three live Chromium processes. Two point four gigabytes of RAM leaked. Two hundred fifty-four zombies. The box is in swap. Everything is slower — but still working. Still responding. No crash. No OOM kill. The kernel hasn't run out of memory yet because swap is absorbing the pressure. But it's dying by inches.
And Daniel only finds it because he wonders why the box feels sluggish and goes looking.
He runs ps aux and sees two hundred thirty-three lines of Chromium. That moment — I've had that moment. You SSH in, run top, and your stomach drops because you see something you didn't expect to see, and it's been there for days.
The system stayed up the whole way down. That's the part that should make people uncomfortable.
Because most of our monitoring is built around the question, is it up? Is the process running? Is the endpoint responding? Those are binary checks. Up or down. Alive or dead. This failure mode is neither. It's alive but multiplying. The process is up. It's very up. It's more up than it's ever been. And that's the problem.
Two hundred thirty-three times up.
And a health check that just pings the process — is Chromium running? — passes. It's running. All two hundred thirty-three of them are running. What you need is a health check that asks, how many Chromium processes are running? And if the answer is more than, say, five, something is wrong.
So what do you actually do about this? Let me give you a checklist.
First, the cause. If you're running containers, use dash dash init or include tini in your image. This fixes the reaping problem at the root. PID one becomes a real init process that calls wait. Orphans get reaped. Zombies get cleaned up. This is the correct fix.
And the tradeoff?
You need to change your Docker run command or your Docker Compose file or your image. It's a one-line change. Init colon true in Compose, or dash dash init on the command line. It's not hard, but you have to know to do it. And a lot of people don't.
Second defense. Periodic reaper jobs.
A cron job or a systemd timer that walks the process tree and calls waitpid on orphaned zombies. This treats the symptom, not the cause. But it works on software you don't control. If you're running a third-party container that doesn't use an init system and you can't rebuild the image, a reaper job is your safety net.
The downside being it's reactive. Zombies accumulate between runs.
Right. If your reaper runs every five minutes, you can still accumulate zombies for five minutes. For most cases that's fine. For the MCP gateway case, five minutes of Chromium spawning could be dozens of processes. But it's better than three days.
Third. Liveness probes and watchdogs. And this is where I want to dwell on the distinction you made. Is the process alive is a bad question.
It's a genuinely bad health check when the failure pattern is alive but multiplying. You need to check process count. A simple probe that runs pgrep chromium and counts the lines — if it's above a threshold, the probe fails. Kubernetes will restart the pod. Even without Kubernetes, a cron job that checks every minute and sends an alert is better than nothing.
The specific difference: ps aux pipe grep chromium pipe wc minus l versus pgrep minus x chromium. One counts instances. The other just confirms existence.
And the existence check will pass with two hundred thirty-three instances. The count check will scream.
Fourth. Resource limits as a blast-radius control.
cgroups. Container memory limits. If Daniel's MCP gateway container had a memory limit of, say, one gigabyte, the OOM killer would have stepped in long before two point four gigs leaked. The container would have been killed and restarted. That's not a fix — the leak still happened. But the blast radius is contained. The rest of the system stays healthy.
And the honest answer is usually both.
Fix the cause and sweep up after it. Because you rarely control every piece of software you run. The MCP gateway is your code. You can add waitpid. But headless Chromium — you didn't write Chromium. You don't control how it behaves when its parent dies. You need layers. Proper init in the container. Process count alerts. Memory limits. A reaper job as a backstop. Any one of these would have caught this. All of them together means you sleep better.
The monitoring stack on that box was healthy. Prometheus was scraping. Grafana was graphing. cAdvisor was tracking. Node exporter was exporting. It was all working perfectly.
That's the uncomfortable point Daniel's story forces. Collecting a metric and noticing a metric are completely different things. The data was there. Every fifteen seconds, the system was writing down exactly what was happening. Memory climbing. Process count climbing. Swap usage climbing. It was all in Prometheus. Nobody had written the alert rule.
Most homelabs only ever do the first half.
They set up the monitoring stack. They get the pretty Grafana dashboards. They feel good because they can see their CPU usage and their memory usage and their disk IO. But they never write the alerts. They never teach the system what to look for. And so the data just sits there, beautifully visualized, while the server quietly fills up with two hundred thirty-three Chromium processes.
The dashboard is a mirror. The alert is a smoke detector. They're not the same thing.
That's the line of the episode right there. A dashboard shows you what's happening. An alert tells you something is wrong. If you only have dashboards, you have to be looking at the dashboard at the right moment to catch the problem. And nobody is looking at the dashboard at three in the morning on day two of a slow memory leak.
The checklist, in order. One — check if your containers use init or tini. Two — add a process count alert, not just a process existence check. Three — set container memory limits as blast radius controls. Four — if you have a long-running process that spawns children, audit whether it ever calls wait.
Five — write the alert. Go into Prometheus right now, or whatever you're using, and write the rule. Container memory usage above eighty percent for more than ten minutes. Process count above some reasonable threshold. Swap usage climbing. Pick one. But write it. Because the data is already being collected. It's just not being noticed.
That brings us to the uncomfortable question this whole episode has been leading to.
What other silent degradations are hiding in your homelab right now?
File descriptor leaks.
Connection pool exhaustion. Every database connection your app ever opened but never closed, sitting there in CLOSE_WAIT.
Disk space slowly filling with logs that nobody rotates.
The pattern is the same every time. Slow, invisible, no crash. The system stays up. The health checks pass. Until one day they don't, and by then it's been failing for weeks.
The steelman here — someone could reasonably say, look, if the service was working fine for three days, and Daniel only noticed because he went digging, was this actually a problem? Two point four gigs on a thirty gig box with swap — the system handled it.
That's fair. The system did handle it. Until it wouldn't have. The trajectory was the problem. Three more days and it's five gigs. A week and it's ten. Eventually the OOM killer wakes up, and it doesn't always kill the thing you want it to kill. It might kill your database instead of Chromium. The fact that the bridge held doesn't mean the cracks weren't real.
The point isn't that the system crashed. The point is that it was degrading toward a crash and nobody knew.
That's the kind of thing that turns a Tuesday morning into a very bad Tuesday morning, with no warning and no obvious cause, because you weren't watching the thing that was slowly failing.
One open question before we close. We talked about process reaping, but this pattern — silent resource accumulation — shows up everywhere. File handles. Sockets. Disk space. Memory fragmentation. The monitoring gap is the same in every case. What else is quietly multiplying in your homelab right now that you'd only find if you went looking?
That's the question to sit with. And maybe the answer is, go looking. Right after this episode. SSH into your boxes and run ps aux. Check your zombie count. Check your process count. Check what's holding memory. It takes five minutes.
Thanks as always to our producer Hilbert Flumingtop.
This has been My Weird Prompts.
If this episode made you uncomfortable about your own homelab, leave us a review — and then go check your process count.
We'll be back soon.
