Here's a question that sounds paranoid until you've lived through the outage that makes it obvious. If your entire infrastructure is in AWS, and you've got it spread across three regions with auto-scaling and multi-AZ databases and all the rest — and then AWS IAM goes down globally — what exactly did all that redundancy buy you?
Nothing. It bought you a very expensive, very well-architected single point of failure.
That's the tension Daniel's poking at this week. He sent us a long one, and it starts with something I said earlier that caught his ear — the phrase "multi-cloud redundancy." His immediate reaction was, wait, is this actually a real thing people do? Because we've all seen the high-profile provider-wide outages. Multi-region within one cloud is redundancy within a single organization's control plane, their IAM, their DNS infrastructure. A provider-wide failure takes the whole thing down. So his question is: can you architect a genuine failover path between completely different clouds — AWS to Azure, Google to AWS, whatever the combination — in a way that actually survives a provider-level catastrophe? And he's got two specific patterns he wants us to trace. One is the Kubernetes-orchestrated cross-cloud failover at scale. The other is the much smaller pattern — a single VPS with a standby on a totally different provider, no Kubernetes, no orchestration layer, just a clean failover path. And he threw in a third wrinkle too — the cloud-to-on-premise pattern, where the primary is in the cloud and the secondary is a rack in a closet somewhere, or vice versa. So we went looking for teams who have actually built this — not just architected it on a whiteboard, but run the drill and measured the seconds.
And the answer to Daniel's first question — is this actually a thing — is yes, absolutely, and the documentation is surprisingly detailed. Multiple teams have published full case studies with measured RTO numbers, specific failure modes they hit during drills, and the exact fixes. The through-line in every single one is that the architecture diagrams are the easy part. The hard part is trusting the automation enough to let it run without a human in the loop.
That trust problem shows up at every scale, which is what makes this worth an episode. Let's start with the big end — the teams running Kubernetes across multiple clouds, because that's where the most detailed documentation lives.
The case study I keep coming back to is from the GeekyAnts team, published in June. They built an AWS EKS to Azure AKS warm standby. PostgreSQL streaming replication from the primary in AWS to a standby in Azure. Route fifty-three DNS failover routing. And they drilled it. Their first drill took thirty-five minutes and failed in three places.
Thirty-five minutes and three failure points on the first try. That's an honest number. And I want to pause on that because thirty-five minutes sounds bad if you're comparing it to the hundred-and-fourteen-second target they eventually hit. But for a first drill of a cross-cloud failover? That's actually a success. They proved the path existed. The rest was tuning.
It is. And they got it down to a measured one hundred and fourteen seconds RTO with zero-second RPO. Breakdown: twelve seconds for database promotion, thirty-three seconds for AKS pod startup, and sixty to ninety seconds for DNS propagation. The Azure standby runs at about eighteen percent of the primary AWS cost.
Eighteen percent. So you're paying less than a fifth of your primary bill for a warm standby that can take over in under two minutes. That's the kind of math that makes a CFO stop asking why you need two cloud bills. And let's be real about what that eighteen percent gets you. It's not just insurance against an AWS-wide outage. It's insurance against the thing where your AWS account gets flagged by some automated fraud detection system at two in the morning and suddenly your entire production environment is frozen while you're on hold with support. That's happened to people. It's not theoretical.
And the orchestration tooling landscape has gotten mature. Karmada is a CNCF incubation project now — it provides centralised multi-cloud Kubernetees management, speaking Kubernetees-native APIs, with advanced scheduling including cluster affinity and multi-dimension HA across regions, availability zones, and providers. There's Cilium ClusterMesh, which was used in a production case study for a regulated payments workload spanning AWS, GCP, and Azure. They reduced region-level RTO from twenty-two minutes manual to forty-seven seconds automatic.
Twenty-two minutes of a human on a call, typing commands, to forty-seven seconds of the cluster handling it itself. That's the gap between having a failover architecture and having one you actually use. And I want to unpack what those twenty-two minutes actually look like in practice. It's not twenty-two minutes of calm, deliberate execution. It's someone getting paged at 3 AM, fumbling for their laptop, logging into the VPN, pulling up the runbook, realizing the runbook is six months out of date and references a dashboard that doesn't exist anymore, then typing commands by hand while their phone is blowing up with messages from the VP of Engineering asking for a status update. That's the twenty-two minutes. Forty-seven seconds is the cluster noticing the failure, making the switch, and the on-call engineer waking up to a notification that says "failover complete" instead of "everything is on fire."
And then there's the GitOps pattern — ArgoCD plus Kustomize overlays. The survival-grade-infra repo on GitHub uses a single repository with Kustomize overlays for cloud-specific patches. Different container registries for ECR versus GCR, different Kafka brokers — all deployed to both EKS and GKE simultaneously from one CI pipeline. The repo is public, it's documented, and it estimates the full active-active setup at about eleven hundred dollars a month.
Which sounds steep until you read their note that a single hour of downtime for a business processing two million dollars a day is about eighty-three thousand dollars. The architecture pays for itself after preventing forty-nine minutes of outage. And that math shifts dramatically depending on your revenue. If you're processing two hundred thousand a day, it's eight thousand three hundred an hour. Still pays for itself in under ten hours of prevented downtime over the lifetime of the setup. The threshold where this stops making sense is a lot lower than people assume.
But here's where the GeekyAnts team's documentation gets really valuable. They published the specific failure pattern they hit during drills, and these are the things that never show up in architecture diagrams. First one: PostgreSQL promotion timing. You run the promotion command, you assume the database is ready for connections — it's not. There's a gap. They fixed it by polling until the database confirms readiness.
So the automation said "promotion complete" and started sending traffic to a database that wasn't accepting connections yet. How long was the gap?
They didn't publish the exact number, but long enough to cause cascading failures in the application layer. Pods started, tried to connect, got refused, went into CrashLoopBackoff. The automation wasn't waiting for the thing it had just done to actually take effect.
That's such a common class of bug. The tool reports success because it successfully issued the command, not because the command's effects have fully materialized. It's the difference between "I sent the letter" and "the letter arrived and someone read it."
Second failure pattern: circular DNS dependency. Their Route fifty-three health check was pointing at the same domain being failed over. So when the domain failed over, the health check followed it — and then reported the new target as unhealthy because the failover hadn't propagated yet, which triggered another failover.
That's the kind of bug that only surfaces when you actually run the drill. On a whiteboard, the health check and the failover are two separate boxes with an arrow between them. In production, they're the same DNS record. And the really insidious thing is that it probably worked fine in partial failure scenarios. It only breaks when the entire primary region goes away, which is exactly the scenario you're building the failover for.
The fix is simple once you see it — a dedicated health subdomain that never participates in failover routing. But you have to discover the problem first. Third failure: double-failover state. Running failover twice without failback in between causes PostgreSQL to error. They added guard checks at script start. Fourth: sequence conflicts on failback. After multiple drill cycles, PostgreSQL sequences fall behind the actual max ID. One-line sequence reset before write tests fixes it.
And the fifth one is the one that makes me twitch. Shell scripts that fail quietly. Missing variables, failed commands — the script just keeps going into a broken state.
The fix is set -euo pipefail, which is three flags that should be in every shell script anyone ever writes for production. set -e exits on error, set -u treats unset variables as an error, and set -o pipefail makes a pipeline fail if any command in the pipeline fails, not just the last one. But the deeper point is that all five of these are operational problems, not architectural ones. The architecture was correct from day one. The automation wasn't trustworthy until they'd found and fixed every one of these.
There's a quote from their write-up that captures this perfectly. "Automation is not the hard part. Trust is. Most teams still want a human involved before traffic moves between clouds during a real incident."
And that's the thing. You can have a forty-seven-second automatic failover with Cilium ClusterMesh, but if your on-call engineer doesn't trust it and jumps in to "help," you've just turned an automatic process into a manual one with unpredictable results. The discipline is letting it run. And that discipline is cultural, not technical. You can't install it with a Helm chart.
There's an analogy here that I think works. It's like those automated trading systems in finance. The ones with circuit breakers that halt trading when things get too volatile. The whole point is to take the human out of the loop because humans panic and make things worse. But someone still has to resist the urge to override the circuit breaker. The GeekyAnts team built a circuit breaker for their infrastructure. The question is whether the on-call engineer can resist the urge to grab the keyboard.
Let's shift to the pattern Daniel asked about that surprised me more — the small scale. A single VPS with a failover on a different cloud, no Kubernetes, no orchestration layer. I went into this assuming the answer was going to be "theoretically possible but nobody actually does it." And then I found the Midpath case study.
This one is delightful. March twenty twenty-six. Three VPS machines at about twelve dollars a month each, across three different providers — Linode, E2E Networks, and Contabo. Serving two hundred and twenty thousand API requests a day. No Kubernetes. No load balancer. No central coordinator of any kind.
How?
Cooperative clients. The clients themselves pick the nearest healthy node based on latency. Each node runs a local PostgreSQL read replica. Writes go through SSH tunnels — they use a tool called Mole — to a single primary database node. So reads survive any node failure. Writes stop if the primary database goes down, but the system degrades gracefully rather than falling over completely.
So the intelligence lives in the client, not in the infrastructure. That's a fundamentally different design philosophy than the enterprise patterns. Instead of building a smart control plane that routes traffic, you build smart clients that figure out where to go.
And it's a pattern that used to be more common. If you go back to the early 2000s, a lot of distributed systems had client-side load balancing and failover. The rise of managed load balancers and service meshes abstracted that away. Midpath is essentially saying, what if we went back to that simpler model? The tradeoff is that every client needs the logic, which works if you control the client library. It doesn't work if you're serving a public API to arbitrary consumers.
The whole thing costs about forty dollars a month. That's cheaper than a single managed database cluster on any of the major clouds.
And their blog post has a line that I think is the thesis of this whole episode. "The cost is not buying throughput. It's buying the ability to deploy without downtime, keep reads alive when the primary dies, lose a full node or an entire provider without an incident, and sleep through most failure scenarios."
Forty dollars a month to sleep through provider outages. That's the pitch.
The CubePath guide from April lays out the same pattern in a more prescriptive way. Cloudflare API-based DNS failover scripts, rclone for file replication, Terraform for multi-provider infrastructure as code. This is the blueprint for Daniel's second pattern — a single VPS with a standby on a different cloud, and you don't need Kubernetes to do it. You need DNS, you need a way to sync data, and you need to test it.
The Midpath architecture accepts a tradeoff that the enterprise patterns don't have to make. Writes stop when the primary database goes down. That's a deliberate design choice — they decided that degrading to read-only during a provider outage was acceptable for their workload. And for a lot of applications, it probably is. Think about a dashboard, a reporting tool, a documentation site, a read-heavy API. Users can still see their data. They just can't make changes for a while.
The survival-grade-infra repo takes the opposite approach — full active-active across AWS and GCP with CockroachDB cross-cloud replication, independent Cloudflare DNS failover, and automated blackout testing scripts. But that's eleven hundred dollars a month and significant operational complexity. The GeekyAnts team's DR maturity ladder puts true active-active writes across clouds at Level five, and their description is blunt: "powerful, but expensive and operationally complex. Most teams don't need this."
Level five is the summit. Most organizations, they say, sit between Level zero and Level two and think they're at Level three. The drills reveal the truth.
Let's walk through that ladder properly because it's worth having the full picture. Level zero is no DR at all. You're running on hope. Level one is backups that may or may not restore. You're doing something, but you've never tried to actually recover from them. Level two is documented procedures that have never been tested. You've got a runbook, it looks good, nobody's ever followed it under pressure. Level three is tested failover with human involvement. You've run the drill, a human executes the steps, and it works. Level four is automated failover with human approval. The system can do it itself but a person has to say "go." Level five is fully automatic with no human in the loop. The system detects the failure, executes the failover, and notifies the humans after the fact.
And the jump from two to three is the hardest one, because it's the first time you actually find out whether any of it works. It's the moment Hilbert's going to tell us about in a minute, where the thing you built on paper meets the thing that actually exists.
Let's talk about the third pattern Daniel raised — cloud to on-premise. Google Cloud's official architecture documentation describes three patterns. Cold: minimal resources in the cloud DR project, a Deployment Manager template creates the environment on failover, data restored from Cloud Storage backups. Warm standby: a minimal database instance runs continuously in Google Cloud, receiving replicated transactions from the on-premise primary. Hot HA: both sides serve production traffic simultaneously with DNS weighted routing.
The cold pattern is the cheapest and slowest. You're essentially storing your infrastructure as code and your data as backups, and you only materialize the infrastructure when you need it. The hot HA pattern is the most expensive and fastest. And the warm standby is the middle ground that most teams actually land on. But each of these has a hidden cost that the documentation doesn't emphasize. Cold failover means your team has to be fluent in the Deployment Manager or Terraform workflow under pressure. Hot HA means you're running two full environments and dealing with data consistency issues continuously, not just during outages.
There's also a public runbook from HybridOps that documents the full failback process — moving workloads from cloud DR back to on-premise RKE2 clusters after an incident resolves. That's the part nobody thinks about. You fail over to the cloud, the on-premise issue gets fixed, and now you have to go back. The failback is often harder than the failover because you're moving from a running system to one that's been idle.
Data divergence during the outage window. You've been writing to the cloud database for six hours while the on-premise one was down. Now you need to reconcile those writes back to the primary. That's not a script — that's a migration. And migrations are the kind of thing where you want a human thinking carefully about edge cases, not an automated process running at 3 AM.
The solitary.cloud guide from January makes the same point. "Data is the hardest part. For object storage, adopt S3-compatible replication or continuous sync tools. For databases, prefer asynchronous logical replication for cross-cloud replication."
There's an unresolved problem that runs through every case study we looked at. The write-path single point of failure at small scale. The Midpath architecture explicitly accepts that writes stop when the primary database goes down. The GeekyAnts architecture uses a single writable PostgreSQL primary. Nobody has solved true active-active writes across clouds at a price point that makes sense for small teams.
And maybe that's fine. The Midpath team made a conscious decision that read-only degradation during an outage was acceptable. Their users could still access data, still see their dashboards, still run reports — they just couldn't make changes. For a lot of SaaS products, that's a perfectly reasonable degraded state. And honestly, I think more teams should be asking themselves that question explicitly. What's the minimum viable version of your product that still provides value? If the answer is "read-only access to existing data," you've just dramatically simplified your DR requirements.
The solitary.cloud guide recommends a pragmatic middle path that I think is the right takeaway for most listeners. Warm standby with a VPS from a different provider as the fallback. You get most of the resilience benefit at a fraction of the complexity of a full multi-cloud Kubernetes setup. And you can always add complexity later if you actually need it. Start with the forty-dollar version. If your business grows to the point where read-only degradation is costing you real money, that's a good problem to have.
And there's a secondary benefit that the GeekyAnts team mentioned in passing that I think is underappreciated. Multi-cloud changes your negotiating position. When you can realistically move workloads, pricing conversations with providers get a lot more interesting.
That's not a technical argument, but it's a real one. And it doesn't even require you to actually move. The credible threat of movement is often enough. If your AWS account manager knows you have a working, tested failover to Azure, the conversation about reserved instance pricing goes differently than if you're locked in.
It's the infrastructure equivalent of having another job offer when you're negotiating a raise. You don't have to take it. You just have to be able to.
Hilbert: I had a setup like this once.
This is Hilbert Flumingtop, our producer, who has been quietly listening and apparently has a story.
Hilbert: Not the Kubernetes stuff. The Midpath thing, before anyone called it that. Twenty eighteen. I was running a small SaaS — customer-facing API for invoice generation. Nothing fancy. Single Linode VPS, forty dollars a month, PostgreSQL, a few hundred lines of Python. Made about three thousand a month off it. Linode had a datacenter-wide network issue. Newark, I think. Took down my API for three days.
Hilbert: Three days of revenue gone. Customers ringing. I spent the next month building a failover to a DigitalOcean droplet. Cron jobs, rsync, a shell script that updated DNS if the primary stopped responding to pings. Never tested it.
Hilbert: I was too scared to test it. The API was making money. If I ran the drill and broke something, I'd be down during business hours. So I had a failover I'd never tested. Six months later Linode went down again. The failover script had bit-rotted. The DigitalOcean droplet was running an old kernel that didn't match the production database version. The whole thing failed silently — the script ran, updated DNS, pointed traffic at a droplet that couldn't start PostgreSQL.
The kernel version mismatch. That's the small-scale version of the GeekyAnts team's PostgreSQL promotion timing bug. Different symptom, same root cause — the failover path had never been walked. The environment had drifted and nobody knew because nobody was checking.
Hilbert: I had a failover that gave me false confidence. That's worse than having no failover. With no failover, you know you're exposed and you act accordingly. With an untested one, you think you're covered and you're not. It's the difference between knowing you don't have a seatbelt and thinking you have one that's going to unlatch on impact.
The GeekyAnts team's quote — "disaster recovery is less about having the right architecture and more about having a process your team actually trusts." Your story is that exact problem at the smallest possible scale. One person, two VPSes, and a script that had never survived contact with reality.
Hilbert: The Midpath people are right about the forty dollars buying sleep. But the sleep doesn't come from the architecture. It comes from having run the drill enough times that you stop being surprised by what breaks. You've seen the failure pattern. You've fixed them. You know the thing works because you've watched it work, repeatedly, under conditions you controlled.
Hilbert: I sold the business in twenty twenty. Told the buyer about the failover. He asked if I'd ever tested it. I said no. He laughed. Then he tested it. It took him four hours to get it working. Four hours of downtime, during business hours, to fix something that was supposed to prevent downtime.
Four hours. That's longer than the original outage you were trying to protect against.
Hilbert: Yep. The cure was worse than the disease because I'd let the cure expire.
That's the operational reality underneath every case study we've discussed. The GeekyAnts team's first drill took thirty-five minutes and failed in three places. Cilium ClusterMesh got it down to forty-seven seconds but only after multiple drill cycles. The Midpath team explicitly documents that they run failover drills regularly. The common thread isn't the technology — it's the willingness to break things on purpose. On a Tuesday afternoon. When nothing is wrong. Just to prove you can.
Hilbert's story really drives home the point that runs through every single case study we looked at. The architecture is the easy part. The trust is the hard part. And trust only comes from testing. Not testing once. Testing regularly. Because as Hilbert discovered, the environment drifts. Kernels get updated on one side and not the other. Database versions diverge. API endpoints change. Your failover is a living thing that rots if you don't exercise it.
There's one open question I want to leave hanging. We found no evidence of a widely-used multi-cloud failover as a managed service. Karmada, Cilium ClusterMesh, CoreDNS-GSLB — these are tools, not turnkey products. Every documented implementation involves significant custom engineering. Will we see a SaaS product emerge that abstracts this? Or is the complexity too domain-specific — every application's data model, every team's risk tolerance, every budget constraint is different enough that a one-size-fits-all product doesn't make sense?
I suspect the answer is that the tooling will keep improving but the operational discipline can't be productized. You can't buy a SaaS that runs your drills for you and builds your team's muscle memory. That part is just work. And the teams that do the work sleep better than the teams that buy the tool and skip the drills.
The solitary.cloud guide put it well. "Resilience is not about adding more providers; it's about designing predictable, tested paths so your critical workloads can continue when a provider falters." The word "tested" is doing all the work in that sentence.
If you take one thing from this episode, it's that the gap between having a multi-cloud architecture and having one you can trust is a drill. One drill. Run it. It will fail. The GeekyAnts team's first drill took thirty-five minutes and broke in three places. Hilbert's failover script had bit-rotted and the kernel didn't match. Your first drill will fail too. That's not a reason to avoid it — that's the whole point of doing it. The failure is the deliverable. It tells you what you don't know.
And the second thing is that you don't need Kubernetees and an eleven-hundred-dollar-a-month budget to do this. Forty dollars a month, three VPSes, DNS failover, and the willingness to accept read-only degradation during an outage gets you remarkably far. The Midpath team proved it. Start there. Run the drill. Fix what breaks. Run it again. Then decide if you need more.
Thanks to Hilbert Flumingtop for producing, and for the story that I suspect a lot of listeners recognized themselves in. I certainly did.
This has been My Weird Prompts. If you want to go deeper on any of the case studies we discussed — the GeekyAnts hundred-and-fourteen-second failover, the Midpath forty-dollar multi-cloud setup, the survival-grade-infra active-active architecture — the show notes have links to everything.
Email us at show at my weird prompts dot com. We'll be back soon.
