There's a comedy sketch from Israel that's been making the rounds for years. A driver, increasingly furious, getting nonsense turn after nonsense turn from his navigation app. And then it cuts to the startup office. A guy at a desk with a microphone, a manager leaning over his shoulder, physically deciding which way to send this man.
The Waze call center.
The Waze call center. And Daniel wrote in about this, because it stuck with him. He says early Waze did give bizarre reroutes, probably spotty connectivity, and the sketch took that frustration and pushed it to its logical absurdity. Somebody is just telling you where to go. And Daniel says, obviously, that's comedy. It's implausible. But then he says something sharper. He says he's encountered things in tech where what's marketed as high-tech automation is actually people working hard at desks.
He's not wrong.
He gives one example. A friend worked at a startup advertising AI that could parse complex logistics contracts. Cutting-edge algorithm, unwieldy legal documents into machine-readable format. The reality was a satellite office full of minimum-wage workers following a style guide and typing fields into a database through a front end. And so Daniel's question is this. What are some other comical examples, over the years, of technology that was sold as highly automated and turned out to be essentially manual? The real-world equivalent of the Waze call center.
There are so many.
That's what I was afraid of.
There's a word for it. Fauxtomation. Automation that isn't automation. Human labor wearing a machine costume.
And it has a pedigree. Because before it was a business model, it was a research method. In human-computer interaction, there's a thing called a Wizard of Oz prototype. You sit a user down in front of what they believe is a working system. It responds to them intelligently. And behind a one-way mirror, a researcher is typing the responses by hand.
It was completely legitimate. You're testing whether the interaction concept works before you spend two years building the thing that might not.
The method is honest. The user is deceived for the duration of a study, and then you tell them. The business model is the same trick with the disclosure removed and the researcher replaced by someone making eleven dollars an hour.
And the deception scales. That's the part that matters. A Wizard of Oz prototype has maybe twenty participants in a lab. Amazon Go had millions of shoppers who believed the ceiling was doing the work.
So that's the arc. We've got the cases. Amazon Go, Facebook M, Expensify, Uber, Taco Bell. Then we get into why this keeps happening. The economics, the incentives, the labor.
Let's start with Amazon Go, because it's the purest version of the thing. Just Walk Out. You walk in, you take what you want, you walk out. No checkout, no scanning, no line. Amazon marketed it as computer vision and sensor fusion. Cameras tracking every item and every hand.
The ceiling knows what you took.
The ceiling did not know what you took. In 2024, it came out that the system relied on roughly a thousand workers in India manually reviewing transactions. Watching video, labeling what shoppers picked up, correcting the bill.
A thousand people. To run a convenience store.
Across the fleet, yes. And the detail that gets me is the latency. The receipt didn't appear instantly because the model was that fast. It appeared after a human had gone through the footage. Which means somewhere in that experience, a shopper is standing outside checking their phone, and the actual computation is a person in Hyderabad scrubbing through a video of them debating between two flavors of chips.
That's a very expensive way to avoid a cashier.
It's an expensive way to look like you've replaced a cashier. Amazon has since scaled the technology back substantially. The flagship stores moved toward a model that's closer to what it actually always was.
I want to linger on the economics there, because it's not just that they hired a thousand people. It's that they hired a thousand people and still charged the same prices, still claimed the same efficiency, still told investors this was a technology company.
Right. The labor was real, but it was accounted for as R and D. It was a research expense, not a cost of goods sold. So the unit economics looked like software even though the operation was a call center with extra steps.
And when you're a company valued on the premise that you've solved something nobody else has solved, the thousand people in India are not a feature of the business. They're a rounding error you hope nobody audits.
Which is exactly what happened. The audit came, the story collapsed, and the technology got quietly rebranded as something more modest.
Facebook M. 2015.
Facebook M launched inside Messenger as a virtual assistant. You could ask it to book a restaurant, find a gift, make a call, research something. It was presented as an AI that could complete tasks for you. And it was, in the sense that a task got completed.
By whom.
By a team of human contractors. The AI handled the simple stuff. The interesting requests, the ambiguous ones, the ones with any judgment in them, went to people. The marketing implied full automation. The reality was a concierge service with a chat interface.
And it died in 2018.
Shut down. And I think the reason is instructive. If you're paying humans to do the work, you don't have an AI product. You have a staffing agency with a very expensive front end. The economics only work if the AI is actually doing the work, and M's AI wasn't.
So the better the human contractors were, the worse the business looked.
That's a paradox worth sitting with. Every time a human solved a problem the AI couldn't, they were proving the AI wasn't ready. The product got better and the pitch got weaker at the same time.
Did the users ever figure it out?
Some did. There were reports of people asking M to do something and getting a response that felt too human, too specific, too contextual. And Facebook never really denied it. They just let the ambiguity sit there, because ambiguity was worth more than honesty.
Expensify.
This one's the darkest of the set. SmartScan. You photograph a receipt, the app extracts the merchant, the amount, the date, the line items. Marketed as smart OCR and machine learning.
And the smart part was.
Amazon Mechanical Turk workers. People manually transcribing receipts. And think about what's on a receipt. Medical bills. Pharmacy purchases. Therapy invoices. Personal expenses people assumed were being read by a machine.
They were being read by a machine. The machine was a person.
A low-paid crowdworker on the other side of the world, looking at your itemized medical history because your employer's expense system asked you to upload it.
That's a consent problem dressed up as a feature.
And it's the one where the deception has a real victim who never agreed to anything. The Amazon Go shopper is being watched, but they're in a store. The Expensify user is uploading something private to a company that told them a computer would read it.
What did Expensify say when this came out?
They defended it. They said the human review was part of the quality process, that it improved accuracy. Which is true, and also entirely beside the point. The user wasn't told, the user wasn't asked, and the user had no way to opt out of a stranger seeing their prescriptions.
The quality argument is the tell. It's the same argument every fauxtomation company makes. The humans are there for quality. The humans are there because the machine can't do it. Those are the same sentence, and only one of them gets said out loud.
Uber.
Uber's self-driving program. For years, the pitch was driverless ride-hailing. And for years, the cars had human safety drivers in them. That's not the scandal by itself. Every AV program has safety drivers. The problem is the marketing ran ahead of the reality by a decade.
And the intervention rates.
High enough in the early years that the cars were effectively human-driven with automated assistance. The human was doing the driving and the software was doing the suggesting. And then in 2018, in Tempe, Arizona, a pedestrian was killed by one of those vehicles. The safety driver was looking down at her phone. The system detected the pedestrian but didn't classify her correctly in time to brake.
And that's where the gap between the pitch and the product stops being funny.
It stops being funny because a person died. The marketing said the car could see. The car could not see well enough. And the safety driver, who was the actual redundancy, wasn't doing the job the marketing had implicitly assigned her.
The safety driver is the perfect fauxtomation figure, though. She's the human the system depends on, and she's the human the system is designed to make you forget. The whole point of the pitch is that she isn't necessary. And the whole reality is that she is.
And when she fails, the story becomes about her failure, not about the fact that the system was built on the assumption that she wouldn't fail. The human is the scapegoat and the safety net at the same time.
Taco Bell.
Taco Bell announced in 2024 that it was testing AI-powered voice ordering at drive-throughs. You pull up, you talk to the machine, it takes your order. And it turned out that some of those interactions had humans listening in, correcting the AI when it got confused.
So the AI takes your order, and when the AI can't take your order, a person takes your order, and the customer experiences it as the AI taking their order.
The customer hears one voice. Behind it, there's a model and a person and a decision about which one handles this particular taco.
The pattern across all five is the same. The marketing promises full automation. The operation runs on significant human labor. The labor is offshore, low-paid, and invisible. And the customer is never told.
The Wizard of Oz, every time. Somebody behind the curtain, pulling the levers.
Why do they do it. That's the real question.
Because real AI is expensive, slow, and uncertain. You don't know if it'll work. You don't know when. You don't know if the thing you build will be good enough to ship. Hiring people is cheap, fast, and predictable.
And the marketing benefit is enormous.
Enormous. Saying "AI-powered" in 2016, in 2020, even now, moves valuation. It gets you press. It gets you investors who want to fund the future. It gets you customers who want to buy the future. The word does work that the product can't.
So the labor cost gets hidden.
Hidden in offshore subsidiaries, contractor networks, crowdwork platforms. And here's the part that ties the whole thing together. Amazon launched Mechanical Turk in 2005. Named after the eighteenth-century chess automaton. The Turk was a mechanical chess player that toured Europe and beat everyone, and it turned out to have a human hidden inside the cabinet.
They named their human-labor marketplace after the most famous fake automation in history.
And they called it artificial artificial intelligence. That's the actual tagline. They were being honest about it, in a way that only works if nobody reads the name. Requesters post tasks, workers complete them for small fees. It's a way to scale human labor while calling it AI.
And it's shutting down.
Amazon announced it would shut down Mechanical Turk in 2026. End of an era. But the practice it enabled is more widespread than ever. The platform is dying. The model is thriving.
So the marketplace closes and the fauxtomation just moves to the next platform.
It doesn't need Mechanical Turk. It needs any pool of people willing to do piecework for low pay. And there's always one.
There's a structural thing here I want to get at. The AI eating itself problem.
Model collapse. If you train a model on its own outputs, it degrades. It drifts. It loses the distribution of real human data and starts producing mush that looks like language and means less and less.
So the AI needs human-generated data to stay functional.
It needs humans. Not just to fix the edge cases, but to keep the whole thing from dissolving. And the industry is hiding the humans who provide that data, because admitting you need them undercuts the story.
So the dependency is real and the invisibility is a choice.
The invisibility is a marketing choice. The dependency is a technical fact. And they're in direct conflict, which is why you get these elaborate structures to keep the workers out of frame.
What about the ethics of it. Because there's a version of this where it's just a joke. Ha ha, the AI was a guy. But the guy is making minimum wage and looking at your medical receipts.
The workers are invisible, underpaid, and often exposed to sensitive content. Expensify users didn't know strangers were reading their receipts. Amazon Go shoppers didn't know their movements were being watched by people in India. There's a consent gap and a privacy gap, and the companies almost never address either.
And the economic distortion.
This is the part I find most interesting. Fauxtomation lets a company claim the productivity gains of automation while paying human wages. Often below what those workers would earn in a transparent labor market, because the whole point is that they're invisible. And it undermines trust in the category. When "AI-powered" can mean "a guy in a call center," nobody can tell what's real anymore.
The word stops meaning anything.
The word has largely stopped meaning anything. That's the cost. Not just to the deceived customer, but to every company actually building the real thing, because they're competing against a cheaper lie.
So where does it go. Does fauxtomation disappear as AI gets better?
I don't think so. There's always an edge case. There's always a long tail. There's always a query the model can't handle and a task that needs judgment. The question isn't whether the humans disappear. It's whether companies will be honest about the humans who remain.
Which is a question about disclosure, not technology.
It's entirely a question about disclosure. The technology is fine. The humans are fine. The problem is the curtain.
I want to go back to something you said about the Turk. That they named it after a fake and called it artificial artificial intelligence. Because that's almost a confession.
It's a confession that works as branding. The people who get the joke think it's clever. The people who don't get the joke just hear "artificial intelligence" twice, which sounds twice as advanced.
That's the whole industry in one product name.
There's a version of this where you're generous and you say, look, every new technology goes through a phase where the demo is more real than the product. The Wright brothers' first flight was five seconds. You have to start somewhere.
And there's a version where you're not generous.
The not-generous version is that the deception is the product. That the pitch is calibrated to extract investment and press and customers before the technology exists, and the humans are the bridge that lets you ship something while you wait.
Which one do you believe.
I think it depends on the company, and I think the honest answer is that most of them start in the first version and drift into the second. You begin with a genuine research goal and a stopgap. The stopgap works. The customers are happy. The investors are happy. And then the stopgap becomes the business, and nobody wants to be the one to say the emperor is a guy named Dave.
There's the thing I keep circling. The workers doing this aren't doing low-skill work. That's the misconception. The AI can't handle these tasks. That's why the humans are there. The humans are doing the hard part and being paid like they're doing the easy part.
The whole reason there's a human in the loop is that the human is better than the model. And the human is paid as if the model is better than the human. It's a complete inversion of the actual value.
So the better the AI gets, the more the remaining humans are doing the hard work, and the less they're paid, because the story requires them to be invisible.
The story requires them to be a rounding error. If they're visible, they're expensive. If they're invisible, they're cheap. And the invisibility is the product.
Hilbert: The best AI we ever had was a guy named Dave.
Sorry?
Hilbert: He was the human API. When the model failed, the call went to Dave, and Dave answered it. He'd read the ticket and type a response. Took him about ninety seconds. Customers thought it was the system. It was Dave.
How long did that run?
Hilbert: Two years. Then they replaced him with a script that returned "I'm sorry, I didn't understand that." Cost saving. Dave was the best AI we ever had, and they fired him for being too expensive.
What was the company selling?
Hilbert: Sentiment analysis. Proprietary AI. My job was to read the customer emails and tag them happy, angry, or confused. Manually. The AI was a dashboard that displayed my tags. They told me never to tell the clients there were people involved.
And there were people involved.
Hilbert: There were eleven of us. We had a quota. Four hundred emails a day each. You get fast at it. You learn to tell angry from confused in the first line.
Did you think it was wrong?
Hilbert: I thought it was a job. The pay was fine. The problem wasn't the deception. The problem was that the clients never knew we existed, so we never got credit for anything. The dashboard got the credit. The dashboard was a table with colored rows.
There's a phrase your manager used to say.
Hilbert: "The best AI is a human who doesn't know they're AI." He said it in meetings. He thought it was funny. I don't think he meant it as a joke.
It's not really a joke.
Hilbert: It's a business model. Anyway, I had one and I don't have it anymore. The dashboard. They shut the whole thing down when the contract ended. I don't know what happened to Dave.
The thing about Dave is that he was the actual capability. The model was the interface.
Hilbert: The model was the interface. Dave was the product. And when they replaced Dave with a script, the product got worse and the interface stayed the same, so nobody noticed for a while. That's the part that bothers me. Nobody noticed.
Because the script said "I'm sorry, I didn't understand that," and the customers assumed that was normal.
Hilbert: They assumed that was the AI being an AI. They'd been trained to expect it. So the failure looked like a feature, and Dave looked like a cost, and the spreadsheet said cut Dave.
There's a whole theory of institutional blindness in that.
Hilbert: There's a whole theory of a guy in a car park waiting for me. I said I'd be twenty minutes and it's been forty. I'll be back.
The misconception I want to kill is that "AI-powered" means fully automated. It doesn't. It means a company has decided to describe its operation using the word AI, and that description may or may not correspond to anything.
And the second misconception, which is worse, is that fauxtomation is rare. It's not a fringe practice. It's the standard playbook. Startups do it. Amazon does it. Facebook did it. It's everywhere, and it's been everywhere since at least 2005.
The humans behind it aren't doing simple work either. They're doing the work the AI can't. That's why they're there. And they're paid as if they're doing nothing.
So the open question is how much of what we use every day is powered by people we'll never see. And whether it matters if the output is good.
It matters when the output is your medical receipts.
It matters when the output is a person's life, in Tempe.
As regulation grows, will companies have to disclose human involvement? Will consumers start demanding human-free labels? Or does fauxtomation just become the norm, an open secret everyone knows and nobody mentions?
I don't know. I don't.
That's where I'll leave it. One more thing before we go. Hilbert Flumingtop produces this show, and he does it while apparently having somewhere to be.
He's been producing it for a while.
This has been My Weird Prompts. If you're enjoying the show, rate and review us wherever you listen. It helps. We'll be back soon.
See you then.
