While everyone is out there talking about the latest cloud-based chatbots and who has the slickest web interface, there is a whole other world of AI happening in places you will never see on a map. I am talking about windowless, reinforced bunkers with armed guards and cooling systems that draw more water than a mid-sized city.
It is the high-stakes world of sovereign compute and classified clusters. Herman Poppleberry here, and honestly Corn, this is the side of the industry that actually makes my hair stand up. We are seeing the birth of what I call the "Intelligence Factory" in its most literal, physical sense. It’s not a metaphor anymore. It’s a factory where the raw material is electricity and the finished product is strategic insight.
Today's prompt from Daniel is about this exact shift—the move away from the cloud by the organizations that actually have the highest stakes. We are looking at governments, intelligence agencies, and militaries that are essentially saying "thanks, but no thanks" to the public cloud for their most sensitive workloads. They are looking at the convenience of a web portal and deciding that the risk profile just doesn’t compute.
And it is a fascinating reversal. Ten years ago, the mantra was "cloud first" for everything. If you weren't migrating to AWS or Azure, you were considered a dinosaur. We saw the CIA move to the cloud with the C2S contract back in 2013, and everyone thought that was the end of the on-prem data center. But for AI, especially at the scale required by the US Intelligence Community or the UK's Department for Science, Innovation and Technology—known as DSIT—the physics and the security requirements are forcing a massive boomerang effect back to on-premises hardware.
By the way, fun fact for the listeners—Google Gemini 1.5 Flash is actually helping us put this script together today. A bit of irony there, using a cutting-edge cloud model to talk about why people are running away from the cloud. It’s like using a commercial airliner to write a manual on how to build a private bunker.
Well, it is the right tool for the job! But to Daniel's point, when we talk about "on-prem" in this context, we aren't talking about a server rack in a closet. We are talking about infrastructure that rivals the biggest commercial data centers but with layers of physical and digital shielding that would make a bank vault look like a screen door. Think about the difference between a public library and a restricted archive inside the Pentagon. Both have books, but the way you access them and the walls around them are worlds apart.
So, why now? Why is the cloud suddenly not enough for these guys? Is it just old-school paranoia, or is there a technical wall they are hitting? I mean, AWS has GovCloud. Isn't that supposed to be the "secure" version?
It is a bit of both, but mostly it is about sovereignty and the "air-gap" imperative. If you are the UK government and you are running the DSIT compute programme, which has allocated hundreds of millions of pounds for domestic infrastructure, you are doing that because you cannot have your national security insights sitting on a server that someone else ultimately controls the kill switch for. Even with GovCloud, you are still operating on someone else's hardware, using someone else's proprietary hypervisors. If that provider has a global outage or a supply chain compromise, your national defense goes dark.
But wait, if they’re building these "sovereign" clusters, are they actually using different chips? Or is it the same NVIDIA H100s everyone else is fighting over, just behind a bigger fence?
It’s the same silicon, but the configuration is fundamentally different. In a public cloud, those H100s are virtualized. They’re sliced up so multiple customers can use them. In a sovereign cluster, you have "bare metal" access. Every clock cycle of that GPU is dedicated to one mission. There’s no "noisy neighbor" effect where someone else’s cat video generation slows down your satellite image analysis. You own the silicon, the firmware, and the thermal envelope.
Right, and it’s not just about who owns the box. It’s about the data itself. If you’re an intelligence agency, your training data isn't just "the internet." It’s intercepted communications, satellite imagery, and human intelligence. You can't exactly upload that to a public S3 bucket, even with the best encryption in the world. Imagine trying to explain to a security auditor that you uploaded top-secret satellite coordinates to a server that also hosts a teenage influencer’s dance videos.
You hit the nail on the head. The US Intelligence Community is reportedly operating a classified AI cluster that uses upwards of five thousand H100 equivalents. Think about the sheer density of that. An H100 draws about seven hundred watts at peak. You multiply that by five thousand, add in the networking and the storage, and you are looking at a facility that needs five to ten megawatts of power just for the compute. That doesn't even count the lights or the coffee machines for the guards.
That is enough to power a small town. And that's just the electricity to run the chips. What about the heat? You can't just open a window in a SCIF—that's a Sensitive Compartmented Information Facility for those who don't spend their lives in bunkers. If you have five thousand H100s running at full tilt, that room is going to turn into a pizza oven in about thirty seconds.
The cooling is where it gets truly wild. We are moving toward full liquid-to-chip cooling for these high-density clusters. You are talking about millions of gallons of water annually. When you build these on-prem, you have to solve for that infrastructure yourself. You can't just lean on Amazon's utility hookups. You need massive heat exchangers, industrial-grade pumping stations, and often, redundant power sources like dedicated substations or even small modular reactors in some future-looking proposals. I heard about a facility where they had to build a dedicated water treatment plant just to handle the mineral content of the local water because it was fouling the micro-channels in the cooling plates.
But how does that work in practice? If you’re a government agency, do you just buy a warehouse and start plumbing?
It’s more like a civil engineering project. You start with the power grid. You need a "dual-feed" from two different substations. Then you build the shell—usually reinforced concrete with copper mesh embedded in the walls for that Faraday cage effect. Then you bring in the specialized HVAC. It’s a multi-year construction project before a single GPU ever gets plugged in.
Is there a specific protocol for how these buildings are laid out? Like, do you put the GPUs in the middle and the guards on the outside?
It’s actually layered like an onion. You have the "Perimeter Layer" with physical barriers. Then the "Support Layer" where the cooling pumps and backup generators live. Then the "Data Hall" which is the heart of the beast. But inside that hall, you have "Security Enclaves." Even if you have clearance to be in the building, you might not have clearance to be in the room where the most sensitive model weights are stored. It’s a nested doll of security.
It’s funny you mention the scale because most companies are still struggling to even figure out their AI governance. I saw a report recently from Arcast saying only about ten percent of organizations are actually ready to deploy AI at scale. Most are still in "innovation theater" mode. But for the military and intelligence sectors, theater isn't an option. They are dealing with what's being called the "Year of Truth" for AI.
That Arcast data is a great reality check. While most CEOs are talking about AI at board meetings and maybe playing with a customized GPT, the organizations Daniel mentioned are dealing with the physical reality of TEMPEST standards. That is the requirement to prevent electromagnetic leakage. If you have a massive GPU cluster crunching classified data, those chips emit radio frequency signals. A sophisticated adversary could, in theory, "listen" to the vibrations of the hardware or the fluctuations in the power line to reconstruct what is being processed.
So you have to wrap the entire data center in a Faraday cage. I’ve heard about people using "white noise" generators for electronics—is it that extreme?
It’s more extreme. You have "red" and "black" separation. Red lines carry unencrypted classified data; black lines carry encrypted or unclassified data. They can’t even be in the same cable tray. You have physical gaps between the wires to prevent "crosstalk" where one signal might bleed into another via induction. When you’re running thousands of GPUs, the electromagnetic "noise" is deafening. Shielding that so a spy satellite or a van parked outside can’t pick it up is a massive engineering hurdle that the public cloud just isn't built for.
But wait, if you have these massive electromagnetic signatures, doesn't that make the bunker a giant target? Like, "Hey, look at this massive radio-silent spot in the middle of Virginia, there must be something important there."
That’s the "Signature Management" problem. You don't just shield it; you try to blend it in. Some facilities are built deep underground, using the earth itself as a shield. Others are disguised as mundane industrial parks or municipal water works. If you look at a satellite map, it just looks like a warehouse for a plumbing supply company. But that "plumbing company" has a security fence that's electrified and a dedicated fiber optic line that goes straight to the Pentagon.
And then there is the "air-gap" itself. In the world of high-end security, an air-gapped system is one that is physically disconnected from the internet. No cables, no Wi-Fi, no Bluetooth. If you want to move data in, you're literally carrying a secure drive through a series of checkpoints. How do you even keep an AI model updated in that environment? You can’t exactly run git pull or pip install.
It is a nightmare for DevOps, but it is the only way to ensure that a model weight—which is the "brain" of the AI—doesn't get exfiltrated. We talked about securing model weights in a previous episode, but the on-prem reality takes it to the extreme. If you are training a sovereign model for a country like Israel or the UK, that weight file is a national asset. If it leaks, your strategic advantage evaporates. You have to use "sneakernet"—physical media moved by people with high-level clearances. You have a "clean room" for data ingestion where files are scanned for steganography or malicious code before they are allowed near the main cluster.
So, let’s say a researcher finds a new paper on Arxiv and wants to try a new training technique. What’s the actual workflow? Do they print the paper out and re-type the code?
Almost! They usually have a "low-side" network where they can do research. They write the code there, then they put it on a secure USB drive—which is often tracked with a serial number and a signature. That drive goes through a "scanning station" which is basically a digital car wash. It checks for malware, unexpected hidden partitions, and even "logic bombs." If it passes, the drive is physically moved across the gap to the "high-side" network. It’s tedious, it’s slow, and it’s the only way to sleep at night if you're the CISO of an intel agency.
It’s the "Intelligence Factory" concept again. If the factory is on someone else’s land, you don’t really own the product. But let’s talk about the "Boomerang Effect." We spent the last decade telling everyone that the cloud is cheaper, faster, and better. Is the cost-benefit analysis actually shifting back to on-prem for GPUs? I mean, those H100s aren't cheap. They’re like thirty or forty thousand dollars a pop.
At a certain scale, the math flips. If you are renting ten thousand H100s from a cloud provider, you are paying a massive premium for their overhead and profit margin. If you have a steady, twenty-four-seven workload—which these agencies do—the math starts to favor owning the hardware. The "Total Cost of Ownership" or TCO includes the facility, the power, the specialized staff, and the hardware itself. Over a three-to-five-year horizon, owning the cluster can be significantly cheaper than cloud credits, especially when you factor in that you aren't paying for egress fees to move your massive datasets around.
Plus, you avoid vendor lock-in. If you build your entire workflow on a specific provider's proprietary AI stack, you're their prisoner. If they raise prices or change their API, you're stuck. For a national government, that's a strategic vulnerability.
And let’s talk about "Compute Sovereignty." If there is a geopolitical flare-up and a foreign cloud provider decides to throttle your access or "pause" your account because of a policy change in their home country, you are in big trouble. We saw this with various sanctions and trade wars. If the UK relies on a US-based cloud for its defense AI, and a future US administration decides to change the terms of the relationship, the UK’s defense capabilities are held hostage.
It’s like a digital version of the strategic petroleum reserve. You need a strategic compute reserve. You need the ability to keep the lights on even if the rest of the world cuts you off.
That is actually a perfect way to put it. And it isn't just about the "big" models. It’s about the specific, fine-tuned applications. Think about a military using AI for real-time sensor fusion on a battlefield. You’re taking radar data, infrared, signals intelligence, and satellite feeds and mashing them into a single "God’s eye view." You can’t send that data to a server in Northern Virginia and wait for a response. You need the compute to be local, secure, and under your absolute control.
How does that work with mobile units? You can't exactly drag a liquid-cooled bunker onto a battlefield.
That’s the "Tactical Edge" problem. You have the "Intelligence Factory" back at home base doing the heavy lifting—the training and the massive data crunching. Then you "distill" those models into smaller, more efficient versions that can run on ruggedized hardware in the field. But even those field units are increasingly "on-prem" in a sense. They are self-contained clusters inside an armored vehicle or a shipping container. They don't rely on a satellite link to the cloud to function. If the link is cut, the AI keeps working.
But what about the latency? If you’re air-gapped and on-prem, aren’t you sacrificing the speed of the modern web?
Inside the facility, the latency is actually lower. You’re using InfiniBand or specialized fiber optics that connect the GPUs at speeds the public internet can’t dream of. The "latency" is in the human process—getting the data in and out. But once the data is inside the bunker, the processing is lightning fast because you own the whole pipe. You aren't sharing bandwidth with a million people streaming Netflix.
I wonder about the talent gap, though. It’s one thing to hire a guy to manage a cloud account. You just need someone who knows their way around a dashboard. It’s another thing to hire a team that can manage a liquid-cooled, air-gapped, ten-megawatt data center. Those people don't exactly grow on trees. You need a plumber who understands high-performance computing and a sysadmin who knows how to handle a Glock.
That is one of the biggest bottlenecks. The skillset required to run an "Intelligence Factory" is a mix of high-end electrical engineering, thermal dynamics, cybersecurity, and AI infrastructure. And these people need top-secret clearances. The competition for that talent between the private sector and the government is essentially a war right now. If you’re a top-tier AI engineer, do you go to OpenAI for a million-dollar salary and a view of San Francisco, or do you go to a windowless basement in Maryland for a government GS-level salary?
How does the government win that fight? Is it just patriotism, or is it the "cool factor" of the data they get to work with?
It’s the data. If you’re an AI researcher, the chance to work on "the real stuff"—data that no one else in the world has access to—is a massive draw. You’re not just optimizing an ad-click algorithm; you’re literally looking for needles in a global haystack to prevent a conflict. That’s a powerful motivator. But the government is also having to get creative with "commercial-to-government" fellowships and specialized contracting to bridge that pay gap.
Do they ever use "bounties"? Like, "Find this specific pattern in the satellite data and we'll pay off your student loans"?
Not quite that informal, but they do have "innovation hubs" like AFWERX or DIU that act as bridges. They bring in commercial talent for short bursts to solve specific problems. But the core "Intelligence Factory" staff? Those are the lifers. They are the ones who know where the literal and figurative bodies are buried in the code.
So, if I’m a technical leader listening to this, and I’m sitting on some very sensitive IP—maybe I’m in biotech or high-end manufacturing—should I be looking at the basement again? Is the cloud dream over for the "sensitive" crowd?
I wouldn't say the dream is over, but the "one size fits all" approach is definitely dead. We are entering a bifurcated era. For your general HR bots, your marketing copy generators, and your public-facing apps, the cloud is still king. It’s efficient and scalable. But for the "Crown Jewels"—the core IP, the sensitive customer data, the proprietary research—you are seeing a move toward a hybrid model. Keep the "brain" of your company in a private, on-prem cluster, and use the cloud for the "muscles" that interact with the public.
It’s like the difference between renting a storage unit for your old furniture versus building a safe in your house for your gold bars. You don’t put your wedding ring in a public locker at the bus station.
And the lead times are the other thing people miss. If you decide today that you want a ten-thousand-GPU on-prem cluster, you are looking at an eighteen-month lead time, minimum. You have to secure the GPUs, which are still in high demand, but more importantly, you have to build or retro-fit the power and cooling infrastructure. You can't just plug ten thousand H100s into a standard wall outlet. I’ve seen projects delayed by a year just because the local utility company couldn't provide the necessary transformer capacity.
I can imagine the look on the face of the local power company when you tell them you need twenty megawatts by next Tuesday. "Sure, let me just flip the 'Small Town' switch to 'Off' for you."
They would laugh you out of the room. It requires years of planning with local utilities. This is why we are seeing "sovereign compute" become a major plank of national policy. Governments are realizing that compute capacity is a fundamental utility, like water or electricity. If you don't have it, you can't compete in the twenty-first century. It’s a "compute-or-die" world. Look at countries like Saudi Arabia or the UAE—they are investing billions into domestic GPU clusters because they don't want to be dependent on Western cloud providers forever.
Is there a risk of "compute silos"? Where every country has their own AI that doesn't talk to anyone else's AI?
That’s already happening. We are moving toward a fragmented "splinternet" for AI. You have the Western models, the Chinese models, and now these burgeoning sovereign models. They are trained on different data, with different cultural values and strategic objectives. If you’re a global corporation, you might have to run three different versions of your AI just to comply with local "compute sovereignty" laws.
It’s interesting to see how this connects back to what we’ve talked about regarding the "Intelligence Factory" in the cloud. In episode six-hundred-seventy-five, we looked at how the cloud was being rebuilt for AI. But now, that same blueprint is being taken "off-grid" into these classified environments. Is the hardware actually different, or is it just the building around it?
The technical architecture is actually very similar. You are using the same InfiniBand networking, the same NVLink fabric for the GPUs, and the same high-speed storage arrays like WEKA or VAST. The difference is the "wrapper." Instead of a multi-tenant cloud where you share a physical building with a thousand other companies, you are in a single-tenant, hardened facility where every single packet of data is accounted for. Even the firmware on the network switches is scrutinized for "phone-home" bugs.
What about the software side? If you are air-gapped, you can't just "pip install" a new library or pull a container from Docker Hub. How do you keep the engineers from losing their minds?
That is one of the biggest friction points. You have to have a "dirty-to-clean" pipeline. You download the software on a separate, internet-connected network, run it through a battery of automated security tests, scan it for every possible vulnerability, and then move it across the air-gap via a controlled "data diode"—a device that only allows data to flow in one direction. It slows down development significantly. You can't just try a new library on a whim. But for the US Intelligence Community, that trade-off is mandatory. They are worried about "supply chain attacks" where an AI library might have a hidden backdoor that leaks data once it’s installed inside the secure zone.
Do they have their own version of GitHub inside the fence?
They do. They have internal mirrors of almost every major repository. But those mirrors are curated. If a library has a known vulnerability, it doesn't get in. If it has suspicious dependencies, it doesn't get in. It’s like living in a gated community where every visitor has to have a background check before they can even walk on the sidewalk.
It sounds like the ultimate "trust no one" environment. Which makes sense for people whose job is literally to trust no one. But what does this mean for the big cloud providers? If the biggest spenders—the governments—start building their own stuff, does that hurt the bottom line for the Amazons and Microsofts of the world?
Not necessarily, because they are the ones often helping to build these private clusters! Microsoft has "Azure Government Secret," and Amazon has "AWS Top Secret." They are essentially building "private clouds" inside the government's own facilities. It’s a specialized business model where they provide the software stack and the hardware expertise, but the government provides the bunker and the guards. It’s a "managed service" but the service is happening behind a ten-foot thick concrete wall.
So they’re basically "ghost-writing" the data centers. They provide the script and the actors, but the government owns the theater and sells the tickets.
In a way, yes. But we are also seeing some countries move away from the US-based giants entirely. France and Germany, for example, have been very vocal about wanting "European" sovereign compute that doesn't rely on US-headquartered companies, even if the servers are physically located in Europe. They are worried about things like the US Cloud Act, which could theoretically allow the US government to subpoena data held by a US company, regardless of where the server is. If you're the French Ministry of Defense, you don't want a US judge to have the legal authority to look at your AI models.
How do they bridge that gap? If they aren't using Amazon or Microsoft, who are they using?
They are building their own consortiums. You see companies like OVHcloud in France or T-Systems in Germany trying to build "European-only" stacks. It’s a hard road because the R&D budget of a Microsoft or a Google is bigger than the entire tech budget of some medium-sized countries. But for "sovereignty," they are willing to pay the price of being slightly behind the curve if it means they have total control.
It’s a geopolitical chess match played with silicon and electricity. And it seems like we’re only in the opening moves. What’s the next level of this? Do we end up with "AI embassies" where a country's compute is considered sovereign territory?
It’s not that far-fetched. We already see data sovereignty laws that dictate where data can be stored. The logical next step is "compute sovereignty" where the actual processing has to happen within specific borders and under specific legal frameworks. For the military and intelligence sectors, this is already the reality. We might see "Compute Treaties" in the future, where countries agree on how much processing power they can have, similar to nuclear arms limitation talks. "You can have ten thousand H100s, but no more."
"The SALT Treaty for GPUs." I can see the headlines now. But let's look at the practical side for a second. If I'm a developer working in one of these bunkers, what is my day-to-day like? Is it just like working at a tech startup, but with more badges?
It’s much slower. You spend a lot of time waiting for security clearances on your code. You can't just Google a solution to a bug because your workstation isn't connected to the internet. You have to use "offline" documentation. If you want to use a new open-source model, you have to wait weeks for it to be "vetted" and brought across the gap. It’s a very different culture. It’s "move slow and secure things" rather than "move fast and break things."
That sounds incredibly frustrating for someone used to the modern dev cycle.
It is. But the stakes are different. If you "break things" in a classified environment, you don't just lose a few hours of uptime; you might compromise national security. The people who thrive there are the ones who find deep satisfaction in solving incredibly complex problems under extreme constraints. It’s like being a submariner—you’re in a high-tech, high-pressure environment, cut off from the world, doing something vital.
I think the biggest takeaway here is that "The Cloud" is not an inevitable destination. It was a convenient detour for a lot of things, but for the most powerful and sensitive AI workloads, the physical world still matters. The bunker is back, and it’s filled with liquid-cooled GPUs. We’re seeing a return to the "Big Iron" era, but instead of IBM mainframes, it’s NVIDIA clusters.
And for the engineers listening, I think the message is clear: if you can learn how to build and maintain these high-security, high-density environments, you are going to be in very high demand. The "Intelligence Factory" is the new frontier of civil engineering and computer science combined. You need to understand how to balance a load on a three-phase power circuit just as well as you understand how to tune a transformer model.
What about the "legacy" systems? Do these bunkers have ten-year-old GPUs sitting next to the H100s?
Sometimes! You'd be surprised how much "legacy AI" is still running on older V100 or A100 clusters because the cost of migrating a specialized, validated model to a new architecture is too high. In the intelligence world, "if it ain't broke, don't fix it" is a very real thing. They will run a cluster until the fans literally fall off if the model it’s running is still providing accurate insights.
Just make sure you can pass a background check and don't mind working in a room with no windows. And maybe learn to enjoy the hum of ten thousand cooling fans.
A small price to pay for being at the literal center of the AI revolution. It’s where the real magic—and the real power—is being concentrated.
Well, I think we've covered the "why" and the "how" of this boomerang back to on-prem. It’s about power, it’s about cooling, and it’s about the absolute fundamental need for security that the public cloud just can't touch for the highest-stakes players. Daniel, thanks for the prompt. It really forced us to look at the physical reality beneath the digital hype.
It’s a fascinating shift. We’re moving from the era of "AI as a Service" to "AI as Infrastructure." It’s no longer just a software layer; it’s a physical asset that nations are going to guard as closely as their gold reserves or their nuclear silos.
And on that note, we should probably wrap this up before someone asks for our clearances or starts wondering why we know so much about Faraday cages.
Good call. I’ve already said too much about the water treatment plants.
Thanks as always to our producer, Hilbert Flumingtop, for keeping the gears turning behind the scenes and ensuring our own little "Intelligence Factory" stays online.
And a big thanks to Modal for providing the GPU credits that power this show. They make the complex stuff look easy, which is exactly what you want when you're trying to navigate this landscape.
This has been My Weird Prompts. If you enjoyed the deep dive into the windowless world of sovereign compute, go ahead and follow us on Spotify so you never miss an episode. We have a lot more ground to cover as this "Year of Truth" unfolds.
We'll see you next time, hopefully from somewhere with a bit more natural light.
Stay weird. And stay secure.
