Daniel's been building a voice pipeline on his phone, and the way he describes it, the transcription part is basically solved. The cleanup part is where it falls apart.
Right, and that's the part nobody puts in the demo. You record thirty seconds, the words come out, everyone claps. Nobody claps at the part where you have to turn that word salad into something a human would actually read.
So here's what he sent. He was thinking about one small model for one narrow job. Dates and numbers. Because ASR will hand you back "nineteen eighty-four" when you wanted digits, and it'll spell out a year like it's reading a bedtime story.
Every time. And it's not even consistent about it. You'll get "nineteen eighty-four" in one sentence and "1984" in the next, and there's no rule you can point to for why.
Then he wondered whether that same small model could also do paragraph breaks and punctuation restoration, two more things ASR is naive about. And then he starts listing. He reckons he could come up with six or seven discrete small-model tasks for this one use case. His example is scrubbing meta instructions. You say "Tomorrow I need to buy bananas at the supermarket, wait, I meant grapes," and the model should render that as "Tomorrow I need to buy grapes at the supermarket." It has to notice the correction is an instruction, not content.
That's a rewrite. That's not a cleanup.
Right. And then he lays out the fork. One multimodal model, audio in, clean text out, every rule stuffed into a long system prompt. Or ASR for the transcription, then iterative improvement in a pipeline. Three questions. Is there a commonly accepted practical limit on how many small models you can chain. What inference tooling or parameters are actually optimized for multi-model workflows where models load and unload between stages. And how do you orchestrate the whole thing, especially on-device with limited VRAM.
And he flags his own bias, which I appreciate. He says he's drawn toward consolidation and that even two chained models makes him nervous.
So let's take the pipeline side seriously, because that's where the interesting engineering is.
The real question underneath this isn't which architecture is better. It's what the pipeline costs you, and where that cost shows up.
Draw the two cleanly. Architecture A, one multimodal model. Audio goes in, cleaned text comes out, all your heuristics live in a long system prompt. Architecture B, ASR produces a raw transcript, then N successive text-transformation stages each do one narrow job.
The appeal of B is obvious to anyone who's debugged anything. Each stage is independently testable. Each stage is independently swappable. And a small model doing one bounded task can be small, which matters enormously if you want this on a phone.
The appeal of A is that you load one model, you make one inference call, there's no intermediate serialization, no error compounding between stages, and the model can use acoustic context that a text-only cleanup stage has already thrown away. That last one is the part people forget.
A text stage never heard the audio. It only sees what the ASR decided the audio was. So if the ASR misheard "grapes" as "grapes" but with the wrong intonation, the text stage has no way to know the speaker was correcting themselves, because that information lived in the prosody and it's gone.
The correction signal in Daniel's example is partly acoustic. The "wait" has a particular shape. A text-only stage sees the word "wait" and has to infer everything from that.
So the tension the whole episode pulls on is consolidation instinct versus modularity instinct. And the modularity instinct has a hardware bill attached.
Daniel's task list is a good test case because each task has a different input-output shape. That's not a detail, that's the whole problem.
Before we can ask how many models is too many, we have to ask what each of these models is actually being asked to do.
Start with the easy one. Number and date rendering. That's near-deterministic normalization. Tiny output space. You're mapping "nineteen eighty-four" to four digits, "the third of March" to a date format. A model with a hundred million parameters can do that in its sleep.
And you could almost do it with a regex, except the cases where you can't are exactly the cases that matter. "He was born in eighty-four" versus "he scored eighty-four points." One's a year, one's a number, and the disambiguation needs context.
So even the easy one has a judgment call buried in it. It's just a small one.
Punctuation restoration is a step up. That's sentence-boundary judgment. You're deciding where a thought ends. Still fairly local, but you're making a call, not a lookup.
And the call is ambiguous sometimes. Speech doesn't have commas. You're imposing structure that wasn't there.
Paragraph segmentation is where the window starts to matter. Deciding that these four sentences belong together and the fifth one starts a new topic is discourse-level judgment. You can't do it one sentence at a time. You need to hold a longer span in view.
And the span you need is unbounded in principle. A topic can run for a while. If your context window is two hundred tokens, you're going to cut paragraphs in the wrong place.
And then -instruction scrubbing, which is the hardest of the four. It requires detecting that a span is an instruction rather than content, and then applying it. That's a rewrite that deletes or replaces content. It's not a relabel.
Which is the structural point I want to sit on. These tasks are not equivalent. And the ones that need long context are the ones that fight the small-model premise. A hundred-million-parameter model can normalize a date. A hundred-million-parameter model cannot reliably decide that "wait, I meant grapes" is an edit instruction and then rewrite the sentence before it.
Because that requires modeling intent. It has to understand that the speaker changed their mind, and that the change reaches backward.
And it has to do that without over-firing. Because people say "wait" as filler all the time. "I need to buy bananas, wait, and also milk." That's not a correction. That's just someone talking.
So the hard part isn't just detecting the correction. It's distinguishing a correction from a continuation that happens to start with the same word.
And the failure isn't graceful. If it gets it wrong, you either delete something the person said or you leave the correction in as content. Both are bad.
The first one is worse. Silently deleting a user's words is the kind of thing that makes them stop trusting the tool.
So now the arithmetic. This is the real answer to "is there a limit." If each stage is ninety-seven percent accurate on its own, six stages in series gives you roughly zero point nine seven to the sixth.
Which is about eighty-three percent end-to-end.
Eighty-three percent. So six stages that are each individually excellent have compounded down to something you'd be embarrassed to ship.
And ninety-seven percent per stage is optimistic for the hard tasks. Punctuation restoration is probably there. -instruction scrubbing is not.
What's a realistic number for the hard ones?
Eighty-five, maybe ninety on a good day, and that's with a model that's bigger than you wanted. So now you're compounding ninety percent six times and you're down around fifty-three percent.
Which is a coin flip. That's not a product.
So the limit isn't a hard number of models. The limit is where your end-to-end accuracy budget runs out. And that budget depends entirely on how good each stage actually is.
There's an asymmetry that matters too. Some stages are idempotent and low-risk. Punctuation restoration rarely breaks meaning. If it inserts a comma in the wrong place, you can live with it.
And some stages are destructive. -instruction scrubbing deletes content. A destructive stage early in the chain poisons everything downstream, because every later stage is now operating on text that's already wrong.
A destructive stage late in the chain is at least inspectable. You can see what it did and you can see what it changed.
So position in the chain matters as much as the stage itself. That's not intuitive. People order stages by perceived difficulty, not by blast radius.
And the right ordering is probably the opposite. Put the destructive stages last, where their damage is contained and visible.
Although there's a counter-argument. If the destructive stage is last, everything upstream has to operate on uncorrected text, which might be harder.
True. There's no free ordering. You're trading compounding against context quality.
On the "commonly accepted limit" question, I want to be honest, because I looked and there isn't a canonical number. There's no paper that says seven. What people actually do is keep chains short. Two to four stages is the common range. Beyond that you see people either merge stages into one model or move to a single model with a structured prompt.
Which is exactly what Daniel's instinct is telling him. He said two chained models makes him nervous. That nervousness is correct, and it's correct for the reason we just worked out.
It's not superstition. It's the error budget.
Now the counter-argument for the monolith, because it's stronger than people give it credit for. A multimodal model with a long system prompt doesn't have stage boundaries. So it can trade off between tasks. It can decide that this span needs number normalization but not punctuation work. It can leave something alone.
A pipeline can't do that. Each stage runs unconditionally. Stage three doesn't know that stage two already handled this sentence correctly. It just does its thing.
Which is how you get the fourth stage second-guessing correct work.
That's a real failure mode, and it's under-discussed. A cleanup stage that assumes its input is broken will break input that wasn't broken.
There's a debugging advantage to the pipeline that people underrate, though. When the output is wrong, you can bisect the chain. You inspect the intermediate text after stage two, after stage three. You find where it went wrong.
With a monolith you get one wrong answer and no intermediate state. You're staring at input and output and guessing.
So the pipeline buys you observability. That's not nothing. That's valuable when you're iterating.
It's the difference between a stack trace and "it didn't work."
And it's worth saying that the monolith's prompt gets harder to maintain as you add rules. You're not just adding a stage, you're adding a paragraph to a system prompt that every other rule now has to coexist with. Rules start colliding. The model has to arbitrate between them, and it does that invisibly.
Which is its own kind of error compounding. It's just hidden inside one call instead of spread across six.
So the reframe for this whole segment. The question isn't how many models can I chain. The question is how many decisions can I afford to make in series before the error budget is gone.
And that number is smaller than people want it to be.
So if the limit is an error budget and not a model count, the next question is what it costs you in hardware to run even a short chain.
Because Daniel's second question is the one that actually bites. He's on-device. He has a VRAM ceiling and it's not negotiable.
Lay out the arithmetic. The ASR model is the big one. It has to be resident for the transcription pass. Then each small model needs its weights resident during its stage. If you naively keep everything loaded, your peak memory is the sum of all stages plus the ASR model.
Which on a phone is exactly the constraint he flagged. You can have a beautiful six-stage pipeline and no way to run it.
So the elegant answer is load and unload between stages. Load the ASR model, transcribe, unload it. Load stage one, run it, unload. Load stage two, and so on.
And that trades VRAM for latency. The tax is cold-start latency. Loading weights from storage takes time, and you pay it once per stage per utterance if you're doing this per-request.
In streaming dictation, that's brutal. You're paying six model loads to clean up one sentence.
There's an escape hatch, and it's the standard way people make this tractable. Batching. If you're processing a batch of transcripts rather than streaming, you run stage one across the whole batch, unload, run stage two across the whole batch, unload, and so on.
The load cost gets amortized across the batch. You pay it once for a hundred transcripts instead of once per transcript.
The catch is that batching is incompatible with low-latency streaming dictation. If the user is watching words appear as they speak, you can't wait to accumulate a batch. You're back to paying the load cost per utterance.
Which is a genuine fork. Batch mode is tractable. Streaming mode is where the pipeline gets expensive.
There's a middle path people try, which is to keep the smallest stages co-resident and only swap the big ones. You pay one load for the ASR model and then run three tiny stages that all fit in the leftover budget.
Which works until the stages don't all fit, and then you're back to scheduling.
Now the tooling question, and I want to be careful here because specific engine names and parameter names should be verified rather than taken from me. The structural point is what matters.
Name the categories.
Three categories. First, engines that support explicit model load and unload, or model eviction policies. You want residency to be something you control, not something that happens as a side effect of process lifetime.
Second?
Engines that support multiple models resident at once with a shared memory pool. That's the other strategy. Instead of swapping, you co-resident everything and let the scheduler manage it.
Third, serving frameworks that treat each stage as a separate worker and handle the scheduling for you. You define the pipeline, the framework handles the handoffs.
The load-bearing claim is that you want an engine where model residency is a first-class, controllable concept. Everything else is implementation detail that changes every six months.
The thing to look for is residency as a knob, not residency as an accident.
Exactly that. If you can't tell the engine when to evict, you can't build the pipeline you want.
Now the quantization angle, because this reframes the whole constraint.
A hundred-million-parameter model at four-bit quantization is on the order of tens of megabytes. So the aggregate weight footprint of several small stages may be smaller than the ASR model alone.
Which means the thing you were worried about isn't the thing that's actually the problem.
The ASR model is the memory hog. The cleanup chain is cheap. If you quantize aggressively, you can fit six small stages in less space than the transcription model takes by itself.
That's a useful reframe. Daniel's worried about the chain, and the chain is the small part.
The interesting optimization isn't "how do I fit six small models." It's "can I run the ASR model and the cleanup stages at different times so they never coexist in VRAM."
Which is a scheduling problem, not a model-size problem.
Scheduling problems are tractable in a way that memory problems aren't. You can sequence things. You can't make a model smaller than it is.
Now the orchestration layer, because this is the part people underestimate when they sketch the pipeline on a whiteboard.
Something has to own the sequence. Something has to handle the intermediate text handoffs. Something has to decide when to load and when to unload. Something has to deal with a stage failing mid-chain.
What does failure look like? Does the chain halt? Does it pass the text through unchanged and log it? Does it retry?
Those are design decisions, and if you don't make them explicitly, you get whatever the default is. Which is usually "crash and lose the transcript."
That's a real piece of software. It's not a config file.
It's the part that turns a demo into a product, and it's the part nobody budgets for.
The trade-off summary. Pipeline gives you modularity, testability, and bisectable debugging. At the cost of error compounding, serialization overhead, and a scheduling and orchestration burden.
Monolith gives you one load and cross-task trade-offs. At the cost of opacity and a much bigger single model.
The choice depends on which cost you can actually pay. On-device, with a hard VRAM ceiling and a latency budget, that's not an abstract question.
Hilbert: Anyway, the fourth guy, we called him the closer, and his whole job was reading it back against the tape.
Sorry, go back.
Hilbert: The shop did tape-to-text. Early two thousands. You'd get a cassette or a minidisc, and it went down a line. First person typed the raw audio. Second person fixed the numbers and the dates. Third person added punctuation and broke it into paragraphs. Fourth person read it back against the tape and caught what the first three got wrong.
The pipeline was people.
Hilbert: Four stations. And the closer was the best-paid one, because he was the one who caught things. That was the whole theory. You stack the checks and the output gets cleaner.
And did it?
Hilbert: Quality went down. Not up. We added the closer and the error rate got worse.
Why?
Hilbert: Because he started fixing things that weren't broken. He'd get a transcript that was correct and he'd find something to change, because that was his job. If he sent it through untouched, what was he for? So he'd second-guess the punctuation, he'd move a comma, he'd decide a number should be spelled out when the second station had already decided it shouldn't.
The verification stage assumed the upstream was wrong.
Hilbert: Every stage assumes the previous stage was wrong. That's what a cleanup stage is. And the closer had the most room to assume it, because his job was the last look.
The stage that was supposed to catch errors was generating them.
Hilbert: He wasn't bad at it. He was good. That was the problem. A bad closer would have just rubber-stamped it. He cared, so he found things.
That's the destructive-stage point from earlier, except the destruction is over-correction.
Hilbert: The other thing. The shop eventually collapsed the middle two stations into one person. The numbers guy and the punctuation guy became one job.
Why?
Hilbert: Because the handoff between them was where most of the errors came from. Not the work. The handoff. The punctuation guy would get the numbers guy's output and not know why a number was written the way it was, so he'd change it. If one person did both, there was no handoff to lose information at.
The serialization boundary itself was the failure point.
Hilbert: It's where the context gets dropped. Every handoff, something doesn't make it across.
The closer.
Hilbert: The closer. He had a name, but everybody called him the closer. I don't think I ever knew his actual first name. He had a mug that said it and I never read it.
The fix wasn't a better closer. It was fewer stations.
Hilbert: Fewer stations and one person holding more of the context. That's what the shop landed on. It's also, I'd guess, what your model pipeline is going to land on.
Which brings us back to the design question. Not how many models, but how many irreversible decisions.
Whether each stage can reject an earlier stage's output rather than blindly transforming it. The closer couldn't do that. He had no way to say "this is fine, I'm leaving it." His whole role pushed him toward changing something.
If the real constraint is the error budget, then the move might be to make each stage verifiable. So a later stage can look at what it was handed and decide to pass it through untouched.
Which is a different architecture. That's a pipeline with a reject option, not a pipeline that always transforms.
The second open question. Does the load and unload scheduling problem get easier or harder as on-device accelerators get better at multi-model residency?
If residency stops being expensive, the whole trade-off shifts. The pipeline gets cheaper and the monolith's advantage shrinks.
If it doesn't, then batching is the only tractable path, and streaming dictation stays hard.
Either way, the interesting question isn't how many models. It's how many irreversible decisions you're making in series.
The one thing people get wrong about this whole thing, if they take one idea away.
They think the memory constraint is the cleanup models. It isn't. Quantized, several small stages aggregate to less than the ASR model takes by itself. The ASR model is the hog.
If you're budgeting VRAM, look at the transcription model first. The chain you were worried about is the cheap part.
That's the misconception. Everyone sizes the pipeline by counting models, and the count is the wrong unit.
Thanks as always to Hilbert Flumingtop, who produces this show and who has apparently been in a transcription shop this entire time.
This has been My Weird Prompts.
If you want to send us something, email us at show at my weird prompts dot com.
We'll be back soon.
See you then.
