Sixty percent. That's what Daniel's fine-tune hit on code-switched Hebrew, up from somewhere under twenty across every leading model he tested.
On two hours of his own voice, yes. Two hours of short sentences he recorded on a phone, using an app Claude wrote for him.
Which is the part that should stop people. Two hours of one man talking into a phone comfortably beat out models trained on a hundred thousand hours of audio.
Trained on a hundred and twenty thousand hours, in Parakeet's case. That's the V2 checkpoint. Six hundred million parameters, a six percent word error rate on the Open ASR leaderboard, and it runs about three thousand times faster than real time. It is a excellent English model. It just has no idea what Hebrew is.
So Daniel's prompt has two parts to it. He shipped the fine-tune, the benchmark looked good, and now he's stuck on the follow-up questions. First one came from Claude originally. If thirty percent of your training sentences should be ordinary English with no Hebrew mixed in, is thirty the right number? Second one is Hannah's, and it's the better question. If the whole thing was trained on one Irishman, who else does it actually help?
And I'd argue those two questions are secretly the same question.
Go on.
Both of them are asking what a fine-tune actually captures. The control sentences are about what it learns. The speaker question is about who it learns it from.
Let's get into it then. Start with what code-switching actually sounds like, because people hear that term and picture something much tidier than reality.
That's the thing. It's not two languages in clean blocks. It's an English sentence with makolet dropped in the middle, or misrad hapnim, or bitachon. You don't switch languages at the sentence boundary. You switch for one noun and switch back.
And the listener doesn't notice. That's the whole point. To the speaker it's one stream of speech.
To the model it's a trap, because Whisper-family models do internal language identification per utterance. That's how they decide which decoding path to take, and it means the model is constantly trying to answer a question the speaker never asked. Is this an English word or did the language change?
And with one short dictation burst, there's almost no context to answer with.
There's no signal, so it guesses. Which is why you get gibberish, or worse, a confident English word that is close to the Hebrew one but wrong.
Give me an example of the confident-wrong one. That's the failure people don't expect.
Makolet becomes "mocklet." Bitachon becomes "bitchen." The model hears something Hebrew-shaped, decides it's English, and produces the nearest English word that fits the acoustics. It's not noise. It's a real word, spelled correctly, and completely wrong. And because it's a real word, your spell-checker never flags it.
That's the nasty version. A human proofreader skims past it because it looks like language.
And it's worse in a dictation context, because you're not reading for meaning. You're reading for whether the transcript matches what you said, and "mocklet" is close enough to makolet that your eye slides off it.
Parakeet is a different failure entirely, though. Whisper at least knows Hebrew exists.
Parakeet V2 is English-only, full stop. And here's the piece that matters. Hebrew isn't in Parakeet V3's list either. That's twenty-five languages, and every single one of them is European. There is no Semitic language anywhere in it.
So the model has probably never seen a Hebrew character.
Almost certainly not. Those tokens are not in the vocabulary at all. The model card says it outright. If a word is not trained in the language model and not presented in vocabulary, the word is not likely to be recognized.
So Daniel isn't teaching Parakeet a new accent. And he isn't really teaching it a new language in the way people usually mean that either.
He's teaching it a new alphabet. New tokens that did not exist for this model twelve months ago. And that reframes the transfer question completely, because now we're not asking whether an accent carries over. We're asking whether tokens carry over.
And those are very different bets. An accent is a surface variation on sounds the model already knows. A token is a unit the model has no representation for at all.
Right. When you fine-tune for an accent, you're nudging existing weights toward a different distribution of the same phonemes. When you fine-tune for new tokens, you're asking the model to allocate representational space that didn't exist. That's a bigger ask, and it's a different kind of learning.
So the model now has Hebrew tokens it never had before. The question is what else it learned along the way, and that's where the control sentences come in.
The advice Daniel got was that roughly thirty percent of the training data should be plain, everyday English with no code-switching in it at all. The reasoning being that you stop the model overfitting on the code-switching itself.
Which sounds right, before you think about it too hard. What does the number actually buy you?
It buys you a contrast. You're not just teaching the model a target, you're teaching it the boundary. Here is what it looks like when I speak Hebrew, and here is what it looks like when I don't.
And without the second half of that, the model only ever sees Hebrew appearing. So it learns that Hebrew appears, not when it appears.
Right. It starts treating the Hebrew tokens as something that just happens in your speech at a certain rate, rather than something tied to a specific word.
Which is exactly the failure Daniel described. English words rendered as Hebrew.
And that failure mode is documented, which is the part I'd want him to know. There's a paper from Ugan and Waibel at Interspeech this year. They state flatly that fine-tuning on synthetic code-switching data generally degrades strong monolingual baselines.
Meaning you can fix the mixing and break the English.
Consistently. It's not an edge case, it's the default outcome. Their fix cut code-switched word errors by about thirty-three percent while improving overall word error rate by five percent, and it held monolingual performance steady. That's the trade Daniel's control sentences are buying.
So the direction of the advice is well supported. Nobody's arguing you should train exclusively on the target.
Nobody serious, no.
Then what about the number? Is thirty right?
Here's the honest answer. Nobody has published a control ratio for code-switching fine-tuning. It isn't there.
That's a real gap, not a gap you're papering over.
It's a real gap. The closest rigorous evidence is Maison and Estève, from twenty twenty-three, and it's about accents rather than code-switching. They mixed accented and non-accented French at different ratios and got U-shaped error curves.
U-shaped meaning?
Error falls as you add the other domain, then rises again when you strip the target domain out entirely. Best results were at more balanced mixes. What that tells you is that the optimum is task-specific, and that blindly maxing out on data is not the answer.
So there's a floor and a ceiling. Too little control and the model overfits the target. Too much control and you're not teaching the target at all.
And the shape of the curve matters more than any single point on it. If you're on the flat bottom of the U, small changes to the ratio don't hurt you. If you're on one of the arms, they do.
And the specific thirty percent?
Is a heuristic. A reasonable one, but a heuristic. The literature's optima ranged between fifty and ninety percent non-target data depending on the task, which puts thirty on the aggressive end.
So Daniel shouldn't agonise over it.
He should ablate it. Run the same fine-tune at thirty, fifty, seventy percent control and see. It's a weekend of recording and a few training runs.
There's a sharper piece of evidence I want to get to, though. Forgetting happens even when the model has already seen the target.
That's the Maison and Estève finding I keep coming back to. A model fine-tuned on Common Voice scored eighteen point seven word error rate on a different accented dataset, even though that second dataset was in its pre-training data.
So fine-tuning doesn't just add. It overwrites.
The model doesn't file the new behaviour alongside the old one. It replaces.
That's the bit that should make anyone nervous about fine-tuning. You're not stacking skills. You're trading them.
And you don't get to choose the trade. The model decides what it's willing to lose.
Friends, are you tired of your speech recognition model misunderstanding you? Well, so was Gerald. Gerald runs a falafel stand in Ramat Gan, and Gerald told me, he said, Larry, my transcription software turned my entire lunch order into the Book of Leviticus. Then I tried the new Chutzpah Switch Pro, and now my model understands every word I say, in every language, including three I made up. My cousin Shlomo, sorry, my cousin Avi, same man, different day, he tried it and his model started transcribing his thoughts before he had them. Chutzpah Switch Pro. It switches. It pross. Available nowhere.
So Daniel's own benchmark has a result in it that cuts against everything people assume about fine-tuning.
It does. He fine-tuned on about ninety minutes of audio and improved the smaller Whisper models but made the larger ones worse. Tiny, base, small, all got better. And then the exception.
Code-switching improved dramatically across every model size.
Every size. The thing that the base model is worst at is the thing fine-tuning is best at fixing.
That's the insight worth sitting with. Fine-tuning helps where the model is ignorant, not where it's merely imprecise.
And it explains why Daniel's Parakeet result is so big. He wasn't polishing a good model. He was handing a model tokens it had never possessed.
It also explains why the bigger Whisper models got worse. They were already good at English. There was nothing for the fine-tune to add on that axis, so it only had room to subtract.
The larger the model, the more it has to lose and the less it has to gain from narrow data. That's the pattern.
So back to the thirty percent. Is it about right?
If a third of his users' utterances actually code-switch, then thirty percent control is defensible on distribution grounds alone. You'd be matching the training mix to the inference mix, which is the only principled way to pick it.
And if his users code-switch more than that?
Then he's over-controlling, and he's spending training sentences on English the model already handles. That's the cost. Every control sentence is a sentence not teaching Hebrew.
Cue the overfitting risk.
Which is real. Without enough control data, the model learns Hebrew tokens as a positional or acoustic pattern rather than a lexical one. It hears a certain rhythm and reaches for a Hebrew word that isn't there.
How bad does that get?
There's a paper on Whisper hallucination that found around one percent of transcriptions contained entirely invented phrases. Not errors, invented text. And there's a well-documented case of someone fine-tuning Whisper-small on Hinglish data and getting both a performance drop and more hallucination.
Invented text is a different category of failure. A wrong word is a wrong word. Invented text is the model writing its own script.
And it's the hardest failure to catch, because it reads fluently. You only notice when it says something you never said.
You can make a model worse at everything by trying to make it better at one thing.
That's the whole tension. And it's why Daniel's own note about epochs matters. He wrote up a gist on this. Single-speaker data carries higher overfitting risk, so fewer epochs. Three to five. For accented speech, five to ten. For code-switching, ten to fifteen.
Interesting that code-switching earns the most epochs.
Because it's the hardest thing to learn. It needs the most passes to stick, and it's also the thing most likely to tip into overfitting if the data is narrow.
So the epochs and the control ratio are pulling in the same direction. Both are trying to give the model enough exposure to learn the new thing without letting it forget the old one.
And both are guesses calibrated on intuition rather than measurement. Which is fine, as long as you know that's what they are.
So the answer to question one is the direction is right, the number is unproven, and his own results already show where the real gains are.
That's the summary. Though I'd put it differently. The thirty percent is probably fine, and it's also the least interesting knob he has.
That's all about what the model learned. The other half of Daniel's question is who it learned it from.
And that's where it gets more interesting, because the finding that answers it is counter-intuitive.
Hannah's question, for the record. She pointed out that Daniel is one Irishman in a sea of American accents, and asked whether any of this transfers.
Good question, and the literature has a cleaner answer than I expected. There's work from Stanford last year, Roll and colleagues. Adaptation has two phases.
Name them.
The first is speaker-specific. Four to six examples from the same speaker give you about twenty percent relative error reduction. Then there's a variety-general phase, where larger numbers of examples confer similar benefits even when they come from different speakers.
So at low data, it has to be you.
And then somewhere between ten and twelve examples, the same-speaker advantage disappears entirely. It's worth one point one percentage points at four to six shots, and nothing at ten to twelve.
Daniel has two hours.
He is nowhere near the speaker-specific phase. He blew past it in the first ten minutes of recording. The question isn't whether his voice dominates the model. It's what accent group he just adapted to.
The two hours isn't a liability in the way people assume. It's more data than the phase transition needs.
Far more. The interesting question is what that surplus data does once the speaker-specific phase is over. It's not learning his voice anymore. It's learning something broader, and the literature says that something is variety.
Which brings the Irish accent back into play.
Here's where the research flatters him, against expectation. Maison and Estève are clear that a model fine-tuned on accent A has reduced performance on accent B, even in the same language. Accent mismatch is real.
The worry is legitimate.
The fix is documented. Training on multiple accents gave up to twenty-five percent relative error reduction on accents the model had never seen, compared to single-accent training.
The answer is more accents, not fewer.
More accents. But then Roll and colleagues found something that inverts the entire business. Low-resource varieties gain most from adaptation. German-accented English, thirty-five percent better. Indian-accented English, thirty-nine percent. US English, twenty-two.
The minority accent is the better teacher.
That's exactly what it says. Daniel's Irish accent may make his fine-tune a better donor for other non-American speakers than an American one would have been.
He's spent the whole project assuming his accent was a limitation.
It might be the asset. Which is a much better answer than he was expecting, and I'd want him to hear it plainly.
What about gender, though? That's the half of Hannah's question with no paper behind it.
No study isolating male-to-female transfer in single-speaker ASR fine-tuning. There isn't one. The literature does accent and speaker variety. Nobody has run the gender version.
So you're inferring.
I'm inferring, and I'll say so. The best proxy is the same finding about same-speaker advantage fading at higher example counts. If speaker identity stops mattering past ten shots, gender is probably a smaller factor than accent.
Why would that be, though? Acoustically.
Accent differences are systematic. They move whole vowel categories. They change prosody across an entire utterance. Gender differences are mostly fundamental frequency and formant scaling, and the acoustic front-end has seen women across a hundred and twenty thousand hours of pre-training.
The model has heard women.
It has heard millions of women. It has heard far fewer Irish-accented men saying makolet.
That's the asymmetry. Accent is the axis, not gender.
Accent is the axis. And it's the one he can fix cheaply.
How cheaply?
Roll and colleagues found about twenty-five to thirty seconds of transcribed audio captures most of the available adaptation benefit. That's a handful of sentences.
He doesn't need thousands of sentences from women. He needs a couple of dozen.
A couple of dozen from a few different accents. American, British, one or two others. Same LLM-generated sentences, read aloud, a few minutes per speaker. That would test the transfer hypothesis directly.
What does he do with the result?
If Hebrew recognition holds up on an American speaker, the tokens transferred, and the accent worry was overstated. If it collapses, he's built a model for one Irishman and he knows it.
That's the experiment worth running, and it costs him an afternoon.
The other thing to say is how much of the Hebrew gain even can transfer, because the Hebrew is the novel part and nobody has measured it. Every paper we've got is about accent adaptation in monolingual settings. There is no literature on cross-speaker transfer of newly added tokens.
Daniel's benchmark might be the only data point.
As far as I can tell, it is. Which is either exciting or lonely depending on your temperament.
Regional is a word they use when they don't want to say wrong.
Morning, Hilbert.
There was an actor, Moishe Krendel, worked out of a studio on Allenby. Dubbed a children's cartoon into Galitzianer. Whole production, twelve episodes. Then the broadcaster heard it, said it was too regional, and had it re-dubbed.
Same actor?
Flatter. They kept the flat one, obviously.
And the original?
I have it. Only copy left. Biscuit tin, top of the wardrobe, next to the fuse box.
You have a tape?
I have a tape.
Do you have a tape player?
No.
Of course not.
Been meaning to get one since the tin came to me. That's eleven years. Still meaning to.
What do you think about the broadcaster's decision?
I think about it more than I think about most things. He got paid twice for the same twelve episodes. That's the only good thing that came out of it.
The flatter version won.
The flatter version always wins. Ask your Irishman.
I might.
He should keep his accent. That's all I'll say.
So the two things Daniel can actually settle with a weekend of recording. The control ratio and the accent transfer.
Both empirical, both cheap. He could ablate the control percentage at thirty, fifty and seventy and just look at the curves. And he could get three speakers with different accents to read the same fifty sentences and re-run the benchmark.
The thing that stays unresolved regardless.
The Hebrew tokens. Nobody has measured how much of that gain transfers across speakers, because the accent isn't the novel part. The alphabet is.
He's standing in the one spot the literature hasn't reached.
He could be the first data point on it. That's not nothing.
The control sentences are there to keep the model honest about the boundary. That's the one wrong idea to kill, by the way.
That thirty percent is some published constant.
It isn't. It's a heuristic, and it's probably fine, and the paper that would settle it doesn't exist.
The other one. That a single speaker's fine-tune is doomed to only work for that speaker.
It isn't. Same-speaker advantage is gone by ten shots, and he has two hours.
Which means he's not the bottleneck. The accent coverage is.
That's fixable for the price of a few afternoons and some friends with opinions about how he says makolet.
Thanks to Hilbert Flumingtop, who produces this and owns at least one tape in a biscuit tin.
For more along these lines, there's episode four, If Your Voice Ages, Does Your Fine-Tune Become Useless; episode five, Fine-Tuning ASR For Maximal Usability; and episode nine, Benchmarking Custom ASR Tools - Beyond The WER. This has been My Weird Prompts. If you got something out of this one, a review helps other people find it.
If you've got a question of your own, send us your own prompt on Telegram at t dot me slash MWP listener bot. We'll be back soon.
