Okay. I need everyone to know that Daniel has spent his week doing something I consider deranged, and I say that with love.
He's been running speech-to-text models on his phone. On purpose. For fun.
For science, allegedly. Here's what he wrote in. He's had this goal for about two years now, going fully dictation-first, no typing. And Whisper's been the big name in open source ASR the whole time, but he's never found it works well on his phone specifically. He's not doing this to save money on API calls. Reception in his new area is spotty, so local ASR means he can keep dictating through a dead zone and push the audio out whenever the signal comes back. He's tried basically every local model you can run on Android. Moonshine, the rest of them. NVIDIA's Parakeet series is the one that's consistently impressed him.
Which is interesting, because Whisper is what everyone reaches for first.
Right. He landed on an app called Whisperian, and then the actual hard part started, which was trying the variants. He dictates for minutes at a time, so the real-time streaming models fall apart on him. The ones doing context boundary generation fit his style much better. And then the surprise: he got better results out of Parakeet v2 than v3. He suspects it's because v2 is English-only. He was also startled that the 1.1 billion parameter model would even run on the phone. His hunch is that 0.6B v2 at 4-bit quantization is the sweet spot for his voice and his hardware.
Three decisions, not one.
Three decisions. Version, size, quantization. Each one moves accuracy and latency. And here's where he's stuck. He knows how to benchmark ASR on his desktop. He doesn't know of any tooling for Android, and the catch is that the inference has to happen on-device or the benchmark tells you nothing. So his plan is to generate ten sentences, record them with each variant on the phone, push the transcripts to the cloud, and compute word error rate offline. He knows he won't get inference time in milliseconds. He figures it's enough to rank them. And he wants to know if we've got better ideas. He's budgeting about an hour.
An hour. That's the part I want to talk about.
Naturally. So let's get into it, because there's a lot here and the first thing worth untangling is why Whisper, the model everyone names first, is the wrong tool for this job entirely.
Whisper was never built for a phone. It was built for batch transcription. You hand it a file, it hands you text, and it does that extremely well. The architecture is an encoder-decoder transformer, and the smallest useful variants are not small. Base is seventy-four million parameters. Small is two hundred forty-four million. Medium is seven hundred sixty-nine million. Large is one and a half billion.
And those numbers mean nothing to a person until you put them next to a phone.
Right, so think about what a phone actually gives you. You've got a mobile NPU with a fixed compute budget, you've got memory bandwidth that's shared with everything else on the device, and you've got thermal headroom that disappears the moment the thing warms up. Whisper large at one and a half billion parameters, even quantized, is asking a phone to do something it was not designed to do. And the deeper problem isn't size, it's shape. Whisper wants to see a thirty-second window of audio and decode it in one go. That's fine when you're transcribing a podcast file on a laptop. It's terrible when you're holding a phone and want words to appear as you speak.
So the mismatch is architectural, not just a matter of the model being too fat.
It's both, but the architecture is the part people miss. Whisper's decoder attends over the whole encoded chunk. There's no incremental state you can carry forward cheaply. So a mobile implementation either waits for a full window before producing anything, which feels broken to a user, or it hacks around the architecture and loses accuracy. Neither is what you want when you're dictating for four minutes straight.
And Parakeet is different how?
Parakeet is NVIDIA's family, built on their NeMo toolkit. The sizes Daniel's working with are six hundred million and one point one billion. Same rough order of magnitude as Whisper medium, but designed from the start for efficient inference and deployment, which means the operators and the memory access patterns are friendlier to accelerators. It's the difference between a model that was trained and then someone tried to make it fast, and a model that was trained with deployment in mind.
That's a real distinction. A lot of models get optimized after the fact and it shows.
It shows in exactly the place Daniel cares about. And then there's the v2 versus v3 thing, which I think is the most interesting result he reported. v2 is English-only. v3 is multilingual. He got better results from v2.
Which sounds backwards. More languages, more capability, right?
That's the intuition and it's wrong. Parameter count is capacity, and capacity gets spent. If you train a model to handle forty languages, those parameters are shared across all of them. The model has to represent phoneme inventories, vocabularies, and syntax for everything it's been taught. When you ask it to transcribe English, it's still carrying all of that. A model trained only on English spends every parameter it has on the one language you actually speak. For a single-language user, that focus is worth more than the breadth.
So v3 isn't worse. It's just carrying luggage Daniel never asked it to carry.
That's a clean way to think about it. And it explains why he could run the 1.1B model and still prefer the 0.6B one. The bigger model has more capacity, but if that capacity is spread across languages he doesn't use, the effective capacity for his English is smaller than the English-only model's total.
Okay, so that's decision one. Version. Now quantization, because that's the one I understand least and I suspect it's doing more work than people give it credit for.
Quantization is reducing the precision of the model's weights. Normally those weights are stored in sixteen-bit floating point. Quantize to four-bit integers and you've cut the memory footprint by roughly four times, and you've also made the arithmetic cheaper because integer operations are faster and cooler than floating point on most mobile silicon.
Four times smaller. That's the difference between a model that loads and one that doesn't.
It's often exactly that. But the cost is real. Four-bit integers can only represent sixteen distinct values. So every weight gets rounded to the nearest one of sixteen levels. Most of the time the model barely notices. But on rare words, unusual names, strong accents, technical vocabulary, that rounding error can flip a decision. And it flips it silently. You don't get a warning, you get a wrong word.
Which is precisely the failure Daniel's going to be measuring, whether he knows it or not.
Which is why the benchmark matters. Four-bit is usually the practical floor on mobile, and it's usually good enough.
Here's the thing I want to poke at. He said streaming models never do well with his style. I want to understand why, because "streaming" sounds like it should be strictly better. Lower latency, words as you go.
Streaming models process audio in small chunks, often a few hundred milliseconds at a time, and they commit to output early. That's the whole point. Low latency means you can't wait to see how the sentence ends. So the model is essentially guessing at the end of a clause before it's heard the rest of it.
And Daniel doesn't speak in tidy hundred-millisecond units.
Nobody does. He dictates for minutes. He pauses mid-thought, he restarts sentences, he runs clauses together. A streaming model has already emitted text for the first half of a sentence by the time the second half arrives and contradicts it. Context boundary generation is the fix. Instead of chopping on a fixed clock, the system looks for natural boundaries, a pause, a breath, a syntactic break, and transcribes those segments whole. The model gets enough context to resolve ambiguity before it commits.
So it's the difference between transcribing what he said and transcribing what he was saying at the two-second mark.
That's the whole thing. And it's a use-case fit question, not a quality question. A streaming model is the right call for live captioning. It's the wrong call for someone composing prose by voice.
There's a thread on Hacker News about Apple's SpeechAnalyzer that's been chewing through exactly this, Parakeet versus Whisper on-device, and the numbers people are posting line up with what Daniel found. Whisper is not the automatic winner once you leave the desktop.
It isn't, and that's the misconception worth killing early. Whisper is excellent. It's just excellent at a different job.
So Daniel's got his three dials. Version, size, quantization. And he's got a hypothesis. Now he wants to test it, and this is where the episode turns, because the tooling situation is bleak.
It's bleak, and I want to be precise about why. On desktop, benchmarking ASR is a solved problem. You've got the audio, you've got the reference transcript, you run the model, you compute word error rate, you're done. There are libraries that do this. There's a whole ecosystem.
And on Android?
Nothing. Or close enough to nothing that it doesn't matter. There's no standard harness that says "run this model on this phone, feed it this audio, give me accuracy and latency." The apps that run these models are consumer apps. They're built to transcribe your voice memos, not to expose an evaluation API.
And the catch Daniel identified is the killer. The inference has to happen on the device. If you benchmark in the cloud, you're measuring a server's behavior, not your phone's. You lose the memory pressure, you lose the thermal throttling, you lose the exact quantization behavior on that specific NPU. The benchmark becomes fiction.
That's the crux. The thing you're trying to measure only exists on the device. Which means you can't outsource the measurement.
So we've got a real gap. And I want to be fair to his proposed approach, because I think it's better than he's giving himself credit for. Ten sentences, recorded with each variant, transcripts saved to the cloud, word error rate computed offline.
Let's evaluate it honestly. Strengths first. It requires no custom tooling. It measures exactly what he cares about, accuracy on his voice, on his hardware, with his microphone. And it's cheap. An hour, like he said.
Weaknesses.
It doesn't capture latency. He knows that. It doesn't capture memory usage or thermal behavior. And ten sentences is a small sample. If two models are within a couple of percentage points of each other on word error rate, ten sentences won't separate them. The confidence interval on a sample that small is enormous.
How enormous?
Roughly, if you're measuring a rate on ten sentences, your margin of error is plus or minus thirty percentage points. Which means the benchmark can tell you that one model is dramatically better. It cannot tell you that one model is slightly better.
So it's a screening tool, not a ranking tool.
That's exactly what it is. And that's fine, as long as he knows it. If the gap is big, the test finds it. If the gap is small, he needs more sentences or a different method.
Which brings us to the other ideas. And I want to put these on the table properly, because he asked for them. First one: does the app expose anything scriptable? An intent, an API, a share target?
Worth checking, and I'd check it before building anything. If Whisperian or any of these apps will accept audio through an Android intent and return a transcript, you can script the whole loop with the Android Debug Bridge. Push the audio, fire the intent, pull the transcript, repeat. That turns a manual hour into a batch job.
And if it doesn't?
Then you build the smallest possible thing that does. Not an app for users. A test harness. You take the model's inference library, ONNX Runtime, TensorFlow Lite, or NVIDIA's own NeMo tooling if it's portable, and you write a minimal Android app that loads the model, feeds it a fixed set of audio files, and writes out the transcript plus a timestamp before and after each inference. That timestamp is your latency measurement.
And that gets him the milliseconds he said he wasn't going to get.
It does. It's more than an hour of work, though. I'd call it an afternoon if everything goes smoothly and a weekend if it doesn't. And there's a real chance it doesn't, because mobile ML build tooling is where desktop tooling was ten years ago.
Which is itself the story. This isn't Daniel's problem. This is a structural gap. Mobile ML tooling lags desktop ML tooling by years, and it lags worst in evaluation. Everyone's building inference runtimes for phones. Almost nobody's building measurement harnesses for phones.
Because the people building the runtimes are shipping products, and the people who need the measurements are researchers, and the two groups don't overlap as much as they should.
Third option. ADB scripting if the app supports it, which we covered. Fourth: does he even need Android-specific tooling? He could export the transcripts and compute word error rate offline with standard tooling. Hugging Face's evaluate library does word error rate. That part's solved.
That part's solved, and it's the part he already planned to do. The unsolved part is getting the transcripts out of the phone reliably.
So let me try to give him something actionable. If I were him, here's what I'd do. First, spend fifteen minutes checking whether the app exposes any automation surface at all. Intent, share target, filesystem access to the model output. If it does, script it and you've got a repeatable harness.
Second, if it doesn't, don't build an app yet. Do the manual ten-sentence test first, but do it properly. Use the same ten sentences across every variant. Record them once, cleanly, and feed the identical audio to each model. Otherwise you're measuring your own inconsistency, not the models.
That's important. Same audio, every variant. If he re-records for each model, the comparison is worthless.
And third, if the manual test shows a clear winner, stop. He doesn't need a harness. He needs a decision. The harness only earns its keep if the results are close or he wants to keep testing new models as they ship.
Which he will, because new models ship constantly.
Then the fourth thing is to build the harness, but build it as a general on-device evaluation tool, not an ASR-specific one. The loop is the same for any model. Feed input, capture output, timestamp both ends, export. Once you've got that, you can evaluate any on-device model on that phone, not just speech.
And that's the second-order point worth making. The workflow is the asset. The specific benchmark result is disposable. Models change every few months. A reusable evaluation pipeline on his phone is worth more than any single comparison.
There's one more thing I'd add, and it's a measurement Daniel didn't mention. Speaker adaptation. Some of these models have a warmup or adaptation phase where accuracy improves after they've heard you for a while. If he benchmarks cold, he might be measuring the wrong thing.
Does Parakeet do that?
I'm not certain it does anything adaptive in the deployed app. I don't want to overstate it. But it's worth controlling for. Run each model twice on the same audio and see if the second pass differs. If it does, you've got a variable you need to hold constant.
Fair. So let's say he does all this. He gets his numbers. What's he actually going to learn?
If his hunch is right, he learns that 0.6B v2 at four-bit beats 1.1B v2 at higher precision for his voice. Which is a counterintuitive result and worth knowing. It would mean the extra capacity of the bigger model is being eaten by something, either quantization differences or inference path differences on his specific hardware.
And if his hunch is wrong?
Then he learns that his intuition about the sweet spot was off, which is also worth an hour. The whole point of a benchmark is to lose an argument with reality.
I want to flag something about his framing, though. He said he wouldn't get inference time in milliseconds, and he seemed to accept that as the trade. But I don't think he has to. A stopwatch on the screen, or even counting seconds, gets you within a second of the truth. If one model takes two seconds and another takes eight, you don't need millisecond precision to make the call.
Latency is usually bimodal in practice. Models are either fast enough or they're not. The exact number rarely changes the decision.
So the lightweight version captures more than he thinks. Accuracy from word error rate, latency from a crude timer, and a subjective feel for whether it's usable. That's a complete picture for his purposes.
It is. And I'd add one more piece of data that costs nothing. Note the battery drain and whether the phone gets hot. Those are the things that make a model unusable in daily life, and no benchmark captures them.
A model that's accurate and fast but turns your phone into a hand warmer is not the sweet spot.
It's not. And that's the kind of thing you only learn by living with it.
Okay, so here's where I land. Daniel's plan is good. It's better than good, it's the right shape. Manual, same audio across variants, transcripts exported, word error rate computed offline. The improvements are: use a fixed sentence set, run each model twice to control for adaptation, capture crude latency with a timer, and note thermals and battery. Then, if the results are close, build the harness. Not before.
And if he builds it, build it general. For on-device evaluation.
Which brings us to the metric itself. Because word error rate is the standard, and I want to spend a minute on whether it's actually the right thing to optimize, given that Daniel's use case is dictating prose he's going to publish.
Word error rate is substitutions plus deletions plus insertions, divided by the total number of words in the reference. It treats every error as equally bad.
And they're not equally bad.
They're not. If a model transcribes "the" as "a," that's an error. If it transcribes a technical term as something else entirely, that's also an error. Same weight in the metric. Completely different consequences for the user.
Which is exactly the kind of thing that would make a model look great on paper and be miserable in practice.
There's a real risk there. A model can have a lower word error rate and still be worse for Daniel, if its errors cluster on the words that matter to him.
So the metric he should maybe be tracking is not just word error rate but something like meaning-altering error rate. Errors that change what the sentence means.
That's a harder thing to compute automatically, but it's very easy to compute by hand on ten sentences. He's already reading the transcripts. He can just mark which errors would have required him to go back and fix them.
Which is a better use of his hour than chasing a decimal point on word error rate.
It is. And it's the kind of thing that only shows up when you actually look at the output instead of the number.
Which is a good place to pause, because I think our producer has been sitting on something this whole time.
Hilbert: The patient became a patent. That's the one that got me.
Say more.
Hilbert: I did transcription for a medical group. Early voice-to-text, the kind that shipped on a CD. They bought it to cut the transcription pool. I was in the pool. My job for about a year was correcting the machine.
Correcting it how?
Hilbert: The system heard "patient" as "patent." Every time. Every time. And in our documents that word is load-bearing. "The patient was seen" became "the patent was seen." The billing people flagged it. The compliance people flagged it. We had a meeting about it.
One substitution, and it cascaded.
Hilbert: It wasn't the spelling. Nobody cares about the spelling. It was that the sentence stopped meaning anything. And the error rate on that system looked fine. It was the errors that mattered that were the problem.
So you'd have scored it well and it was useless.
Hilbert: I kept a notebook. Every error I caught, I wrote it down. Which word, which model version, what it should have been. Filled most of a spiral notebook over that year. Still have it in a box somewhere.
That's a labelled error corpus. That's the thing Daniel's trying to build.
Hilbert: It's a notebook. Anyway, the point is you want to count the errors that make you rewrite the sentence. The other ones you fix without thinking.
Which is exactly the distinction Herman was drawing. Meaning-altering versus cosmetic.
Hilbert: I've got to move a chest freezer before it gets dark. It's in the garage and it's not going to move itself.
A chest freezer.
Hilbert: It's empty. Mostly. I'll be back for the mix.
So Daniel's got his answer, and it's a better answer than the one he came in with. Word error rate gets him a ranking. Meaning-altering errors get him a decision.
The notebook Hilbert mentioned is the proof that the second metric is the one that survives contact with real work. He was doing this by hand thirty years ago and the lesson hasn't changed.
Here's the misconception I want to put to bed. The one I think most people carry into this. The belief that Whisper is the default best open source ASR model, full stop.
It's the default for batch transcription on a desktop, and it's excellent at that. It is not the default for a phone. Its architecture and its size make it a poor fit for on-device dictation, and NVIDIA's Parakeet models routinely beat it there. Daniel's own testing says the same thing.
The second one, which is subtler. The belief that bigger is better. That the 1.1B model has to beat the 0.6B model.
Not if the bigger model is multilingual and the smaller one is English-only. For a single-language user, focused capacity wins. Daniel ran the bigger model and still preferred the smaller one, and the reason is that v3's parameters are spread across languages he never speaks.
The open question I'm left with. Does he build the harness, or does he take the manual result and run?
I'd take the manual result and run, unless the numbers come back close. An hour gets him a decision. A weekend gets him a tool. The tool only pays off if he's going to keep testing.
The bigger question. Whether word error rate is even the right target, or whether the whole field should be tracking meaning-altering errors separately. Hilbert's notebook suggests the second one is what actually matters, and nobody's built a standard for it.
Nobody has. Which means Daniel, with ten sentences and a text editor, could build a better evaluation than the one that ships with most tooling.
That's the episode. Thanks to Hilbert Flumingtop for producing, and for the freezer update.
If you got something out of this one, leave us a review wherever you listen. It helps other people find the show.
This has been My Weird Prompts.
The human-AI collaboration podcast. We'll be back soon.
