Here's what Daniel sent in this week. He's been thinking about Waze, which, fair enough, everyone here uses it. And a few years back there was an Israeli comedy sketch where a driver keeps getting these bizarre reroutes, and the reveal is that it's not an algorithm at all. It's a guy in a startup office with a microphone, and a manager standing over his shoulder telling him which way to send people.
The manager is the best part.
It is. So Daniel says, obviously that was comedy, completely implausible, nobody actually thinks Waze works that way. But he wants to know about the cases where it turned out to be true. Technology that was sold as cutting edge automation that was, underneath, just people at desks. He mentions a friend who worked at a startup advertising an AI that could parse complex logistics contracts into machine readable format. The reality was a satellite office full of minimum wage workers following a style guide and typing the fields into a database by hand.
That's the whole episode right there.
And then he asks three things. What are the other comical examples that have come to light over the years. Why do companies misrepresent manual work as automation. And what does it mean, ethically and economically, for the workers doing the invisible work and for the consumers paying for the promise.
So let's start with the most famous recent case.
And it's a doozy.
Amazon Just Walk Out. Launched with enormous fanfare. The store of the future, no checkout lines, you just walk out with your items and the cameras and the computer vision figure out what you took and charge you. The pitch was that this was a breakthrough in computer vision, that Amazon had solved something nobody else had solved.
And the reality?
In 2024, The Information reported that roughly a thousand workers in India were manually reviewing transactions. Watching footage, labeling what people picked up, correcting the AI's mistakes. A thousand people. For a few dozen stores.
A thousand people is not a rounding error. That's a call center.
That's a call center with a computer vision demo attached. And Amazon's response was interesting. They said the humans were validating the AI, not replacing it. That the model was doing the work and the humans were just checking.
Which is a distinction that sounds meaningful until you ask how much of the work is being checked.
Right. If a thousand people are reviewing transactions for a few dozen stores, the ratio tells you something. The humans aren't the exception to the system. They're the load bearing wall.
So the AI was the validation layer, not the other way around.
That's exactly backwards from how it was sold. And that's the pattern we're going to see over and over. The demo is real. The scale up is human.
Let's define what we're actually talking about here, because there's a legitimate version of this and a deceptive version, and they get conflated.
There's a term for the deceptive version. Fauxtomation. Technology marketed as automated that secretly relies on human labor. And the key word is secretly. It's not that humans are involved. It's that the marketing is built on the claim that they aren't.
And the legitimate version is the Wizard of Oz prototype. Where you build something that looks automated, but behind the curtain a human is doing the work, and you do that to test whether anyone wants the thing before you spend three years building the real version.
That's a standard product development technique. It's honest internally, it's temporary, and the whole point is to learn whether the demand is real before you commit engineering resources.
The difference is duration and disclosure.
Duration and disclosure. The Wizard of Oz prototype is a question you're asking. Fauxtomation is an answer you're faking. And the faking continues after the product has launched, after the funding round has closed, after the customers have signed contracts based on the claim.
So the sketch Daniel's talking about, the Waze call center, that's funny because it's absurd. But the structure of it, a person at a desk pretending to be a system, that's not absurd at all.
That's a business model.
Let's go through the cases, because Daniel asked for the comical examples and there are more than people realize.
Google Duplex is the one that sticks with me. Twenty eighteen. The demo where the AI calls a hair salon and books an appointment, and it says "mm hmm" and "got it" in this eerily natural way, and the salon owner on the other end has no idea she's talking to a machine. It was impressive. It was also, in some cases, humans.
Google claimed the demo was real.
Google claimed the demo was real, and I believe the demo was real. But when it went into wider testing, there were reports that human operators were involved in some calls. And the reason is obvious in retrospect. The demo was a controlled scenario. Real calls are messy. Accents, background noise, people changing their minds mid sentence.
The last ten percent.
The last ten percent is ninety percent of the effort. That's the whole thing. Getting a model to ninety percent accuracy on a narrow task is a solved problem. Getting it to ninety nine point nine percent, which is what you need for something that touches money or scheduling or anything a customer will notice, that's where the bodies are buried.
And that's where the people come in.
Facebook M is the other classic. Twenty fifteen to twenty eighteen. Marketed as an AI powered personal assistant inside Messenger. You'd ask it to do things, book reservations, find information, and it would do them. And it was heavily reliant on human contractors.
How heavily?
Heavily enough that the contractors were the ones doing the work, and the AI was assisting them, not the other way around. Facebook was relatively upfront about this, to be fair. They described it as AI assisted by humans. But the marketing impression was that you were talking to an AI.
And Daniel's friend's logistics startup is the purest version of this, because there's no AI at all. There's a style guide and a front end.
That's the Waze call center. The algorithm is a person with a style guide. The cutting edge AI is a satellite office full of people being paid minimum wage to type legal fields into a database.
What I want to know is how common this is. Because Daniel's asking, and I think the honest answer is more common than anyone admits.
It's very common. And the reason it's hard to measure is that the companies doing it have no incentive to disclose it, and the companies not doing it have no incentive to point fingers, because they might be doing a little of it themselves.
So why does it happen? What's the mechanism?
Three reasons, and they stack. First, AI is hard. The last ten percent of accuracy takes ninety percent of the effort, and at some point the engineering cost of closing that gap exceeds the cost of just hiring people to close it manually.
So it's a make versus buy decision.
It's a make versus buy decision where the buy option is a person in a cheaper timezone. Second reason, investors and customers pay for the story of automation. The valuation multiple on an AI company is different from the multiple on a staffing company. If you can tell the AI story and deliver the staffing reality, you capture the AI multiple.
That's the arbitrage.
That's the arbitrage. And third, human labor is cheap enough in certain geographies that faking it is economically viable, at least until you scale. A thousand workers in India reviewing transactions is a real cost, but it's a cost that's lower than the cost of building the computer vision system that would actually do the job.
Until it isn't.
Until it isn't. And that's when the story breaks.
How does it break? How does this stuff come to light?
Usually journalism. The Information on Amazon. Sometimes regulatory filings, sometimes former employees speaking out. Occasionally a customer notices something that doesn't add up.
What does that look like?
The chatbot that's suspiciously good at edge cases and suspiciously bad at simple ones. Or the AI that responds at exactly the speed of a person typing. Or the automation that takes a suspiciously long lunch break.
The lunch break is a tell.
The lunch break is a tell. And when it comes out, the companies almost never admit it. They reframe. Human in the loop. Hybrid AI. AI assisted. The language is designed to make the human involvement sound like a feature rather than a confession.
Which brings us to the economics, because that's where this gets interesting.
The margin isn't from efficiency. That's the thing. If you're paying minimum wage in India or the Philippines to do work that the customer thinks is being done by a model, your margin isn't a technology margin. It's a labor arbitrage margin dressed up as technology.
And that distorts competition.
It distorts competition badly. A genuine automation startup has to build the thing, which costs money, and then charge enough to recover that cost. A fauxtomation company just hires people and charges the AI price. On paper, the fauxtomation company looks more efficient. It looks like it has better unit economics.
So the honest company loses the deal.
The honest company loses the deal, and then either has to cut corners to compete or go out of business. Which means the market selects for fauxtomation. That's the perverse incentive.
Let's talk about the workers, because they're the ones holding this up and they're the ones who get nothing out of it.
They're invisible by design. That's the point of the whole arrangement. They can't put "I was the AI" on a resume. They can't point to the product and say I built that. They're contractors, usually, with no benefits, no job security, and no path to advancement.
And when the company either automates or gets caught?
They're the first to go. If the company gets caught, the story becomes "we've eliminated the manual review process" and the workers are laid off. If the company automates, same outcome. They were never employees in the meaningful sense. They were a stopgap that the company was always planning to eliminate.
There's something grim about that. You're hired to be the thing that gets replaced.
You're hired to be the training data. And then you're discarded when the training is done.
What about the consumers? What do we lose?
We're paying for a promise that isn't kept. When you walk out of an Amazon Go store, you think you're being tracked by cameras and algorithms. You're actually being watched by a person in India reviewing the footage. There's a privacy dimension there that nobody signed up for.
You consented to the algorithm.
You consented to the algorithm. You didn't consent to a person in another country watching you pick up a sandwich and put it back.
And the knock-on effect is trust.
Trust is the big one. If every AI powered product might secretly be a person, how do you evaluate any claim? You can't. You either become cynical and assume everything is fake, or you become credulous and assume everything is real. Neither is good.
The cynicism is the rational response, which is the problem.
The rational response is to assume the marketing is lying, which means the companies that are telling the truth get punished for it. That's the erosion.
Let me push back on this a little, because there's a counterargument that I think is worth taking seriously.
Go ahead.
Every automation wave has had human scaffolding. Telephone operators, computer operators, the people who used to do the calculations that gave computers their name. Humans were always in the loop. Isn't this just how technology develops?
That's the strongest version of the counterargument, and it's partly right. Humans have always been the scaffolding. The difference is the deception. The telephone operator wasn't marketed as an automatic switching system. She was marketed as a telephone operator.
So the issue isn't the humans. It's the claim.
It's the claim. The question isn't whether humans are involved. It's whether the marketing is honest about it. And in the fauxtomation cases, it isn't. The humans are hidden precisely because disclosing them would undermine the pitch.
So the ethical problem is the concealment, not the labor.
The concealment, and the fact that the concealment is what makes the economics work. If Amazon had said "we have a thousand people in India reviewing transactions and some cameras," the story would be different. The valuation would be different. The whole thing falls apart.
Which means the deception isn't incidental. It's load bearing.
It's the product.
There's one more thing I want to get to, which is what happens as AI gets better. Because the line is going to blur.
It's already blurring. The honest version of human in the loop is a real thing. A model does the work, a person checks it, the person catches the errors the model makes. That's a legitimate architecture. The problem is that "human in the loop" has become the euphemism that fauxtomation hides behind.
So the same phrase covers both the honest and the dishonest version.
The same phrase covers both, which means the phrase is useless as a signal. You can't tell from the outside whether the human is the exception handler or the engine.
And the incentive to fake it doesn't go away as the technology improves, because the gap between the demo and the deployment never closes entirely.
The gap never closes entirely. There's always a tail of cases the model can't handle, and the question is always whether you handle that tail with engineering or with people. And people are usually cheaper, at least until you get caught.
So the Waze sketch stays relevant.
The Waze sketch stays relevant. It was comedy, but it's also a documentary.
Hilbert: The word you keep using is deception.
That's the word.
Hilbert: It's not the word I'd use. I worked at a place that sold automated customer service to mid sized businesses. Forty of us in a windowless office on the second floor of a building that also had a dentist. We were the automation. There was a script and a style guide and a guy named Dave who was really good at sounding like a robot. Dave got employee of the month twice.
What was the product actually doing?
Hilbert: It was a chatbot. You'd type a question into a box on a company's website, and the answer would come back. The customer thought they were talking to software. They were talking to me, or to Dave, or to the woman at the desk next to me who was doing her master's degree in the evenings and typed faster than anyone I've ever met.
And the customers didn't know?
Hilbert: Some of them figured it out. That's the part nobody talks about. They'd notice that the chatbot was very good at the weird questions and very bad at the simple ones, because we had a script for the weird ones and the simple ones were supposed to be handled by the actual software, which was terrible. So you'd get someone asking about a refund policy in three languages and getting a perfect answer, and then someone asking what time you close and getting nonsense.
The inversion.
Hilbert: The inversion. And some customers would ask, straight out, am I talking to a person. And the script said to say I am an AI assistant. That line was in the style guide. I typed it more times than I can count. I always felt bad about that one.
What happened to the company?
Hilbert: They won an award. Industry award for customer service automation. The metric they submitted was containment rate. The percentage of chats that didn't escalate to a human.
But every chat was handled by a human.
Hilbert: Every chat was handled by a human. So the containment rate was one hundred percent. We won the award for it. They put it on the wall in the windowless office, right above the desk where Dave sat.
The award was in the room with the AI.
Hilbert: The award was in the room with the AI. I don't know what happened to it. I left before they did. A relative is expecting me, I have to go.
The containment rate was a hundred percent because there was nothing to contain.
There's something about that award that's better than any of the examples we've been through. It's the whole thing in one object. A trophy for automation, displayed in the room where the humans sat, measuring a metric that was meaningless because the humans were the system.
It's the purest version of the pattern. The metric was designed to measure how well the AI was doing, and it measured a hundred percent, and the reason it measured a hundred percent is that the AI wasn't doing anything.
It reminds me of something I read about the early days of IBM's Watson. After it won Jeopardy, they tried to sell it to hospitals as a cancer diagnosis tool. And there were reports that the system was recommending unsafe treatment options, and that the way it was being used in practice was that human doctors were quietly ignoring its suggestions and doing what they already knew was right. The AI was the front end. The humans were the actual diagnostic engine. And nobody wanted to say that out loud because the whole pitch was that Watson had beaten the humans.
And the hospitals had paid for Watson.
The hospitals had paid for Watson. So the doctors were doing the work, the AI was getting the credit, and the patients were being told they were getting cutting edge machine intelligence.
That's the same structure. The human is the engine, the AI is the interface, and the marketing inverts the two.
So if fauxtomation is this common, and the incentive to fake it doesn't go away, how do we ever trust a claim of automation? Is it regulation, is it journalism, or do we just get cynical?
I don't think regulation can keep up. The technology moves faster than the rulemaking, and the disclosure requirements are easy to game. Journalism has done the most work here, but it's episodic. It catches the big cases and misses the small ones.
Which leaves cynicism.
Which leaves cynicism, and cynicism has a cost. It punishes the honest companies along with the dishonest ones.
And as the technology improves, the line blurs further. The honest human in the loop and the dishonest one look identical from the outside.
They look identical, which means the burden shifts to the buyer. You have to ask, and the seller has to answer, and the answer has to be specific. Not AI powered. Not human in the loop. How many people, doing what, where.
The Waze sketch was comedy. But it's also a documentary.
It's a documentary with a laugh track.
This has been My Weird Prompts. Our producer is Hilbert Flumingtop. The show is at my weird prompts dot com, and you can find it on all podcast platforms. If you enjoyed this episode, leave a review. It helps other listeners find the show.
We'll be back soon.
See you tomorrow.
