Daniel's been watching the Linux admin forums catch fire again. This time it's Proxmox versus SmartOS versus just plain Linux, and he noticed something. The whole debate conflates the virtualization platform with the hypervisor itself. He wants to strip away the branding and talk about the actual layer between the silicon and the guest OS. Three questions. What does that layer actually do? How is it fundamentally different from what Docker does? And what do the levels of abstraction actually look like, from raw hardware up to whatever's writing on top of the hypervisor?
This is the kind of question where the answer is a trapdoor. Literally. That's the mechanism at the heart of all of this. And most people running Proxmox or ESXi have never heard of it.
A trapdoor.
The CPU has a trapdoor built into the silicon now. When a guest operating system tries to do something privileged, the floor opens and it falls into the hypervisor's lap. That's the whole game. Everything else, the management dashboards, the backup systems, the clustering, that's all wrapping paper.
So let's unwrap it. What's the hypervisor actually doing at the most basic level?
It's a traffic cop. It takes one set of physical resources, CPU cores, RAM, storage, network interfaces, and partitions them into multiple sets of virtual resources. Each guest VM gets what looks like its own complete computer. The hypervisor's job is to maintain that illusion without letting the guests collide.
And there are two families of these things.
Right. Type one runs directly on the hardware, no operating system underneath. Type two runs as a process inside a host OS. That's the textbook distinction. But it's been blurry for years now, because KVM, the Kernel-based Virtual Machine, is technically a type two hypervisor that turns the Linux kernel itself into a type one. You load a kernel module, and suddenly Linux is a hypervisor. The host OS becomes the hypervisor.
Which is what makes the whole Proxmox versus SmartOS debate confusing before you even get started. Proxmox isn't a hypervisor. It's a management platform that wraps KVM and LXC.
And SmartOS is an operating system based on illumos that ships with KVM built in, plus its own container system called Zones. So both of them use KVM under the hood for full virtual machines. The fight isn't about hypervisors at all. It's about control planes and container runtimes and ZFS versions.
But Daniel's question goes deeper than that. He wants to know what KVM is actually doing. What happens when a guest VM executes an instruction?
Let's walk through it. Intel and AMD added hardware virtualization extensions years ago, VT-x and AMD-V. These add a new CPU execution mode called VMX non-root operation. There's root mode, where the hypervisor runs, and non-root mode, where guests run. The guest thinks it's running directly on the CPU, but it's in this constrained ring.
And the constraint is enforced by a data structure.
The VMCS. Virtual Machine Control Structure. It lives in memory and it defines everything about the guest. The register state, the instruction pointer, and critically, a bitmap that says which operations should cause an exit. When the guest hits one of those, the CPU hardware itself saves the guest state into the VMCS, loads the hypervisor state, and transfers control. That's a VM Exit. No software involved in the interception. The silicon does it.
So the hypervisor doesn't have to watch the guest. The guest runs at full speed until it does something it's not allowed to do, and then the hardware rats it out.
That's the trapdoor. The guest is running along, executing normal instructions at native speed, and then it tries to modify CR3 to switch page tables. The CPU checks the VMCS, sees that CR3 modifications are configured to exit, and bam. The floor opens. The hypervisor gets control, looks at the exit reason field in the VMCS, sees it was a CR3 access, emulates what the guest wanted to do but on the real hardware, updates the guest's virtual CR3 in the VMCS, and then executes a VM Entry to resume the guest. The guest has no idea anything happened.
How long does that round trip take?
On modern hardware, a few hundred cycles for a simple exit. It used to be thousands. The hardware has gotten very good at this. But it's still not free, and if you have a workload that causes frequent exits, you feel it.
And that's just the CPU side. What about memory?
Memory is where it gets clever. The guest thinks it has a contiguous block of physical RAM starting at address zero. The host has already allocated real physical pages scattered all over the place. So you need a second level of address translation.
Two sets of page tables.
Extended Page Tables on Intel, Nested Page Tables on AMD. The guest maintains its own page tables, mapping virtual addresses to what it thinks are physical addresses. The hypervisor maintains the EPT, mapping those guest physical addresses to actual host physical addresses. The CPU's memory management unit walks both tables in hardware during a TLB miss. The guest never sees the real addresses. It's a clean, hardware-enforced wall.
So the guest's physical address zero might be host physical address four gigabytes and change, and the guest will never know.
And it can't find out. There's no instruction that leaks the host physical address from non-root mode. The hardware guarantees it.
That's the isolation. Now, the other half of this is devices. The guest needs a disk and a network card, and those don't virtualize themselves.
This is where QEMU enters the picture. KVM is a kernel module. It exposes a device file, slash dev slash kvm, to userspace. You open that file, make ioctl calls on it, and you can create and run virtual machines. But KVM only handles CPU and memory virtualization. It does not handle devices.
So QEMU is the userspace process that does everything else.
QEMU provides the device model. It can emulate real hardware, an Intel e1000 network card, a Cirrus video card, an IDE controller. The guest OS loads its standard driver for that hardware, and when the driver writes to a memory-mapped I/O register, that write causes a VM Exit. KVM sees it's an I/O access, doesn't know what to do with it, and hands it off to QEMU. QEMU interprets the write, updates its internal device state, and returns.
And that's slow.
Painfully slow compared to native. Every register access is a full VM Exit, a context switch to QEMU in userspace, and back. For a network card pushing gigabits of traffic, that's a disaster.
So they invented paravirtualization.
virtio. Instead of pretending to be real hardware, virtio says, look, you're a guest operating system. You know you're in a virtual machine. Let's stop pretending. Here's a shared memory ring buffer. You put your network packets in this buffer, you ring a doorbell, and the hypervisor picks them up. No emulation, no register-level exits. Just a memory buffer and a notification mechanism.
So the guest driver has to be virtio-aware.
Yes. You install the virtio-net driver in the guest, and it knows to use the ring buffer instead of banging on fake hardware registers. The performance difference is enormous. You go from maybe a few hundred megabits on an emulated e1000 to near line-rate ten gigabit with virtio.
That's the trade-off. Full emulation means the guest runs an unmodified OS with stock drivers, at a performance cost. Paravirtualization means you modify the guest, but you get near-native speed.
And almost everyone chooses paravirtualization now. Every modern OS ships with virtio drivers. The only time you use full emulation is during installation, before you've loaded the virtio drivers, or for some ancient legacy guest that can't be modified.
Let me see if I have this straight. KVM handles CPU and memory in the kernel. QEMU handles devices in userspace. The guest runs in non-root mode on the CPU. Privileged instructions trap to KVM. I/O traps to KVM and gets forwarded to QEMU. Memory is translated through two layers of page tables in hardware.
That's the architecture. And it means the Linux kernel is doing the most performance-critical work, the CPU and memory virtualization, while QEMU handles the messy, slower device stuff in userspace where it's easier to develop and debug.
Now, Daniel's second question. How is this fundamentally different from Docker?
The boundary. In virtualization, the boundary is the hardware instruction set. In containerization, the boundary is the system call interface. Completely different layer.
Walk me through the same example. A process inside a Docker container wants to read a file.
The process calls the read system call. That call goes to the host kernel. There's no second kernel in a container. The host kernel looks at the process, sees which namespaces it belongs to, and uses the mount namespace to figure out which filesystem root to use. It uses the PID namespace to show the process a different view of process IDs. But it's the same kernel, the same system call handler, the same everything.
So there's no trapdoor.
No trapdoor. No VM Exit. No second set of page tables. The isolation is implemented in the kernel's data structures, not in the CPU's execution modes.
What are those data structures?
Namespaces and cgroups. Namespaces provide isolation. The PID namespace means a process inside the container sees itself as PID one, but the host kernel knows it's really PID four thousand and something. The network namespace gives the container its own network interfaces and routing table. The mount namespace gives it its own filesystem tree. The UTS namespace gives it its own hostname. The IPC namespace isolates System V IPC resources.
And cgroups limit what the container can consume.
CPU shares, memory limits, block I/O throttling, network bandwidth. The kernel enforces those limits at the resource accounting level. If a container tries to allocate more memory than its cgroup allows, the kernel's out-of-memory killer fires, but only inside that cgroup.
So the security model is completely different.
Radically different. In a VM, the security boundary is enforced by the CPU hardware. Non-root mode cannot access hypervisor memory. Period. To escape a VM, you need to find a bug in the virtual hardware, a bug in KVM, or a bug in QEMU's device emulation. These bugs exist, but they're rare and they get patched urgently.
And in a container?
The security boundary is the Linux kernel's system call interface. If there's a kernel bug that allows privilege escalation, and you can trigger it from inside a container, you get root on the host. The container's isolation is only as strong as the kernel's ability to correctly implement namespaces and capabilities. A kernel exploit is a container escape.
That's the trade-off everyone argues about. VMs are heavier but more isolated. Containers are lighter but share the attack surface of the kernel.
And there's a performance dimension. That VM Exit loop has a cost. Even with hardware virtualization and virtio and EPT, you're paying a tax on every privileged operation. A container pays almost no tax. The system call goes straight to the host kernel. There's no second-level address translation, because there's no second kernel with its own virtual memory map. The container process is just a process with some extra kernel data structures attached.
How big is the difference in practice?
For CPU-bound workloads, it's single-digit percentages now. Hardware virtualization has gotten very good. For I/O-heavy workloads, especially with virtio, it's also quite small. The bigger difference is density. You can run hundreds of containers on a machine that might only run a dozen VMs, because each VM carries the overhead of a full kernel, its own page cache, its own scheduler.
And yet sometimes you need a VM. Different kernel, different OS.
That's the killer feature. You cannot run a Windows container on a Linux host. You can't run a FreeBSD container on a Linux host. Containers share the kernel. VMs run their own kernel. If you need isolation strong enough to run untrusted workloads from different customers, you use VMs. The cloud providers run customer workloads in VMs, and then they might run containers inside those VMs.
Nested virtualization. A trapdoor inside a trapdoor.
And it works. It's not fast, but it works.
So let's bring this back to the debate Daniel mentioned. Proxmox versus SmartOS. How does knowing the mechanism clarify that fight?
It clarifies that they're not really competing at the hypervisor level. Both use KVM. The differences are in the control plane and the container technology. Proxmox gives you a web interface, a clustered filesystem, backup tools, and LXC for containers. SmartOS gives you Zones for containers, ZFS deeply integrated, DTrace for observability, and a different management philosophy.
SmartOS is the hybrid approach. Zones are OS-level virtualization, like containers, but they predate Docker by years. They're part of the illumos kernel.
And they're extremely lightweight. A Zone is basically a label on a process group. Creating a Zone takes seconds, and you can run thousands of them. But they share the illumos kernel. If you need a different kernel, you spin up a KVM guest. That's the hybrid. Zones for lightweight isolation where the kernel matches, KVM for full isolation where it doesn't.
And Proxmox does something similar with LXC and KVM.
Same pattern. LXC for Linux containers, KVM for full VMs. The difference is in the details. Zones have a longer track record in production, they're deeply integrated with ZFS for snapshotting and cloning, and SmartOS is a more opinionated system. Proxmox is easier to get started with, has a bigger community, and runs on standard Linux.
But neither of them is the hypervisor. The hypervisor is KVM. The platform is the management layer on top.
And that's the misconception Daniel was pointing at. People say Proxmox is a hypervisor. It's not. It's a control plane. It's a web application that calls KVM and LXC APIs. You could do everything Proxmox does with shell scripts and the KVM command line. It would be miserable, but you could do it.
Some people do.
Some people do, and they're very proud of it, and their systems are unmaintainable. But the point is, the platform debate is about operations, not about virtualization mechanics. The mechanics are the same.
The real choice is about what you want your operational layer to look like. Do you want a Debian-based system with a web UI and a large community? Proxmox. Do you want an illumos-based system with DTrace and Zones and a smaller but very dedicated community? SmartOS. Do you want to build your own operational layer on top of raw KVM and libvirt? Plain Linux.
Each of those choices has implications for storage, networking, monitoring, high availability. But the trapdoor underneath is identical. The VM Exit loop, the EPT, the virtio ring buffers, all of it works the same way regardless of which dashboard you clicked to create the VM.
Which brings me to something I've been wondering. If hardware virtualization is this good, and containers are this light, is there a future where the hypervisor just becomes part of the CPU? No kernel module, no userspace component, just silicon that does it all?
We're already moving in that direction. The hardware has absorbed more and more of the virtualization logic. Intel and AMD keep adding features that reduce VM Exit frequency. There's APIC virtualization so interrupt handling doesn't exit. There's posted interrupt processing. There's virtual interrupt delivery. Each generation moves more of the hypervisor's work into the CPU itself.
The hypervisor gets thinner.
Thinner and thinner. At some point, the hypervisor might be a few thousand lines of code that just configure the hardware and get out of the way. The CPU would handle scheduling, memory translation, interrupt routing, all of it in silicon.
The platform layer, the Proxmox and SmartOS and vSphere layer, that becomes the real differentiator.
It already is. The hypervisor is commoditized. KVM is free and it's excellent. The value is in the management, the orchestration, the backup and disaster recovery, the integration with storage and networking. That's where the engineering effort goes now.
It's funny. We spend all this time arguing about platforms, and the most important piece of infrastructure is this invisible layer that most people don't even know exists. The trapdoor.
The trapdoor is what makes the cloud possible. Every EC2 instance, every DigitalOcean droplet, every Google Compute Engine VM, they all run on this same mechanism. VM Exit, EPT, virtio. The cloud is just a very large number of trapdoors.
The people who built the first one didn't have any of this hardware support. They did it all in software.
IBM did it in the seventies. VM slash three seventy. The Control Program was a hypervisor that created virtual machines on System three seventy mainframes. Each VM ran its own copy of CMS, the Conversational Monitor System, and they were fully isolated. No hardware virtualization extensions. They had to trap and emulate every privileged instruction in software.
How did that perform?
Well enough to be a commercial product for decades. The mainframe people figured out virtualization before most of us were born, and they did it without any help from the CPU.
Hilbert's been making faces behind the glass. He's got something to say about this.
Hilbert: VM slash three seventy. CP dash sixty seven before that. I ran it.
You ran a mainframe hypervisor?
Hilbert: Summer of ninety-one. Intern at an insurance company in Hartford. They had a four three four one. I was supposed to be filing, but the sysadmin took a liking to me. Showed me how to IPL the thing. We had six virtual machines running on it, each one doing something different. Batch processing, timesharing, a database.
What was the database?
Hilbert: IMS. Hierarchical. Green screen terminals.
That's incredible. You were working on the direct ancestor of everything we've been talking about.
Hilbert: The commands are the same. You define a virtual machine, you give it storage, you IPL it. Different syntax, same concept. They solved this problem fifty years ago.
Then you moved on from insurance to what, exactly?
Hilbert: Systems integrator. Late nineties. Company sold server consolidation services. This was before VMware got big. We'd go into a shop that had fifteen servers each running at ten percent utilization, and we'd consolidate them.
How?
Hilbert: Dual boot. We'd put two operating systems on one machine, with a script that would swap out the fstab and reboot into the other one.
That's not server consolidation. That's a boot menu.
Hilbert: We called it temporal virtualization.
Temporal virtualization.
Hilbert: Marketing term. Sounded better than dual boot with a script. You'd run the payroll system during the day, then at night the script would swap the fstab, reboot, and the machine would come up as the batch processing server. Morning, swap back.
Did it work?
Hilbert: Most of the time. We lost a client's payroll data once. Race condition in the script. It swapped the fstab but the old one didn't get written back to disk properly. Machine came up, couldn't find its root filesystem. Payroll for two hundred people.
What happened?
Hilbert: We restored from tape. Took three days. Client was not happy.
Is that why you're a producer now?
Hilbert: I'm a producer now because the company got bought by a larger company that got bought by a larger company and I took the severance. But the payroll incident didn't help.
Temporal virtualization. I'm going to remember that.
Hilbert: The mainframe people would have laughed at us. They had real virtualization in the seventies, and we were out there rebooting servers with shell scripts and calling it innovation. The Linux admins arguing about Proxmox today, they don't know how good they have it.
They're standing on the shoulders of a four three four one.
Hilbert: A bad shell script. Don't forget the shell script.
Before we wrap up, I want to hit the misconception that started this whole conversation.
The misconception is that Proxmox is a hypervisor. It's not. It's a management platform. The hypervisor is KVM, and KVM is a kernel module that turns Linux into a type one hypervisor. That distinction matters because when you're debugging performance problems or thinking about security boundaries, you need to know which layer you're actually touching.
The broader misconception is that containers are just lightweight VMs. They're not. They virtualize a completely different boundary. VMs virtualize hardware. Containers virtualize the operating system interface. That difference determines everything about their security, performance, and use cases.
Here's the open question I keep coming back to. If hardware-assisted virtualization keeps getting better, and the hypervisor keeps getting thinner, at what point does the hypervisor stop being software at all? Do we end up with CPUs that just do virtualization natively, where creating a VM is an instruction?
I think we're headed toward a world where the hypervisor is a firmware feature. The CPU and chipset handle the isolation, the memory translation, the interrupt routing. An operating system just asks the firmware for a new virtual machine and gets back a handle. The software layer becomes purely about management and policy.
Which means the real debate isn't Proxmox versus SmartOS. It's about how we want to manage a capability that's increasingly built into the silicon. The trapdoor is becoming part of the floor.
That's the thing Daniel was really asking about. The trapdoor is the most important piece of infrastructure you never see. It's the magic that makes the cloud possible, and it's disappearing into the hardware one generation at a time.
Hilbert Flumingtop produced this episode, and if he ever offers to consolidate your servers, ask about the shell script first.
This has been My Weird Prompts. You can find every episode at my weird prompts dot com, or email the show at show at my weird prompts dot com. We'll be back soon.
