Daniel's been thinking about a problem that anyone who uses voice-to-text regularly will recognize immediately. He sends in prompts for this show by dictating into his phone. He used to send the raw audio straight into the pipeline, and we'd transcribe it here. But he switched to sending text because he wanted to review the transcription first — catch the mistakes before they landed in an episode. The thing is, he'd rather go back to sending audio. He just needs confidence that the transcription is right.
And the specific failure he's worried about is the one where the model hears a technical term it doesn't know and quietly substitutes something that sounds similar but means something completely different. His example: saying "LORAs" as in low-rank adaptations, and getting back "Lauras" as in multiple women named Laura.
Right. So his proposal is essentially: run the same audio through multiple transcription passes — either different models or the same model multiple times — and then reconcile the outputs. If three out of four runs say "LORA," you go with "LORA." And he's asking whether this architecture has a name, whether the reconciliation step could be handled by an LLM acting as a judge rather than a programmatic diff, and how you'd actually build the thing. Specifically, the hardest part: taking several divergent transcriptions and producing one clean, non-repetitive reconciled output.
This lands right in a space I've been following closely. The architecture Daniel's describing does have a name — actually, it has several, depending on which part of the stack you're looking at. The broad pattern is called ensemble decoding, or sometimes multi-hypothesis reconciliation. In the speech recognition literature, it's been around for decades.
Decades?
The old-school version, yeah. Before neural models took over, the standard approach in automatic speech recognition was to generate an N-best list — the top N most probable transcriptions from a single model — and then rescore them with a separate language model. That's a form of ensemble right there. The difference with what Daniel's proposing is that he wants to run entirely separate inference passes, possibly across different model architectures, and then reconcile at the text level rather than at the probability level.
So the core idea isn't new, but the implementation he's sketching — multiple full transcription jobs plus an LLM judge — is a modern twist on it.
And the modern twist matters because the old methods assumed you had access to the model's internal probabilities. You'd look at the confidence scores for each word and pick the highest. But with cloud APIs — OpenAI Whisper, Deepgram, AssemblyAI — you don't always get token-level confidence. Sometimes you just get the final text. So you can't do probability-weighted voting. You have to work with the text outputs directly.
Which is where the LLM-as-judge idea comes in.
Right. And that pattern — using an LLM to evaluate or reconcile outputs from other models — has its own name in the current literature. People call it LLM-as-a-judge, or sometimes LLM evaluator. It's mostly been applied to things like evaluating the quality of other LLM outputs, grading summaries, that kind of thing. Using it for transcription reconciliation is less common, but the principle transfers cleanly.
Let me push on the basic version first, the programmatic one. Daniel said it sounds hard to implement, and I think he's right. If you've got three transcriptions of the same thirty-second clip, they're not going to line up word for word. One might have an extra "um," another might split a sentence differently, the third might have hallucinated a whole phrase that sounds plausible but wasn't in the audio.
This is the alignment problem, and it's the hardest part. You can't just do a line-by-line diff because the transcriptions won't have the same number of lines, or even the same number of sentences. The standard approach is to use a text alignment algorithm — usually something based on the Levenshtein distance at the word level, or more sophisticated methods like the Gale-Church algorithm originally developed for aligning parallel corpora in machine translation.
Gale-Church. That's a name I haven't heard in a while.
It's old but it works. The idea is you align the texts at the sentence or word level, and then for each aligned segment, you take a vote. If two out of three transcriptions have "LORA" at position seventeen, and the third has "Laura," you pick "LORA." Simple majority voting.
Except it's not simple when the alignments are messy. What if one transcription splits "low-rank adaptation" into two words and another hyphenates it?
That's where it gets fiddly. You need a normalization step first — lowercase everything, strip punctuation, maybe expand contractions. Then you align, then you vote, then you reconstruct the casing and punctuation from the majority output. It's doable, but Daniel's instinct that it's harder than it sounds is completely correct. I've seen production implementations of this, and they're usually several hundred lines of Python with a lot of edge-case handling.
Several hundred lines of edge cases sounds about right for anything involving human speech.
The other thing is, simple majority voting assumes the errors are independent. If you're running the same model three times with the same temperature setting, the errors might be correlated — the model might consistently mishear "LORA" as "Laura" because it doesn't have that token in its vocabulary. In that case, all three runs agree on the wrong answer, and voting doesn't help.
So the ensemble only works if the models make different kinds of mistakes.
That's the key insight, yeah. The best results come from using different model architectures — say, OpenAI Whisper large versus Deepgram Nova versus AssemblyAI's Conformer model. They've been trained on different data, they have different weaknesses. Whisper might be better with technical vocabulary, Deepgram might handle background noise better, and so on. When they disagree, the disagreement itself is a signal.
And when they all agree on the wrong thing, you're just confidently wrong.
Confidently wrong in triplicate. Which brings us to the LLM-as-judge approach, which I think is actually the smarter path here.
Walk me through it.
So instead of programmatic alignment and voting, you take all three transcriptions, you feed them into an LLM — something like Claude or GPT-4 — and you give it a prompt that says, essentially: here are three independent transcriptions of the same audio. Your job is to produce a single reconciled transcription. Use majority agreement where the transcriptions differ. Preserve the most technically precise version of any term that appears to be a specialized vocabulary item. Output only the final transcription.
That's... elegant. The LLM handles the alignment implicitly because it's reading all three and understanding them semantically, not just doing string matching.
It's doing something closer to what a human proofreader would do. A human looking at three transcriptions doesn't align words in a grid and count votes. They read all three, notice where they diverge, and use context to figure out which version makes the most sense. The LLM can do the same thing. "LORA" appears twice, "Laura" appears once, and the surrounding text is about machine learning — it's going to pick "LORA" every time.
Even if it's never seen "LORA" in its training data? I mean, low-rank adaptation is a pretty specific term.
It doesn't need to have seen the term in its training data to reason about it. It sees the pattern — this word appears multiple times in a technical context, the alternative is a common name that makes no sense in context — and it infers the correct reading. That's the power of using a language model rather than a string-matching algorithm. You're leveraging semantic understanding.
There's a cost question here, though. Daniel said transcription is cheap, and it is — Whisper API calls are fractions of a cent per minute. But running everything through an LLM judge adds a second cost layer. For a podcast prompt it's negligible, but if you're doing this at scale, it adds up.
It does. But you can mitigate that. The judge model doesn't need to be the biggest, most expensive one. For reconciliation tasks, a mid-tier model works fine — you're not asking it to generate novel content, you're asking it to compare and reconcile. Claude Haiku or GPT-4-mini would handle this perfectly well at a fraction of the cost of the full models.
And the latency is lower too, which matters if this is part of a pipeline.
Right. The other thing is, you can structure the prompt to make the judge's job easier. Instead of dumping all three transcriptions and saying "figure it out," you can pre-process them slightly — align them roughly, highlight the points of disagreement, and ask the LLM to focus specifically on those. That reduces the token count and makes the reconciliation more reliable.
So the pipeline would be: audio in, split to three transcription services, collect the outputs, optionally pre-align to flag divergences, then feed to the judge LLM, get back the final text.
That's the shape of it. And I'd add one more step: a confidence marker. The judge can output not just the reconciled text but also annotations on words or phrases where the transcriptions disagreed. So Daniel can see at a glance, "these three words had low agreement, I should double-check them."
That's smart. It closes the loop back to his original workflow, where he was manually reviewing the transcription before sending. With confidence markers, he's still reviewing, but he's only reviewing the parts that actually need attention.
It's not fully autonomous — it's augmented review. The machine handles the easy ninety-five percent, flags the hard five percent, and the human makes the final call on the flagged bits.
Let's talk about the name question. Daniel asked what this methodology is called. You mentioned ensemble decoding and LLM-as-a-judge. Is there a more specific term for the whole pattern?
In the current literature, the closest named pattern is probably "LLM-as-a-judge" or "LLM evaluator," but those are broad. For the specific transcription use case, I've seen it called "multi-STT voting" or "transcription ensemble." There'sn't a settled, widely-recognized name for the full pipeline Daniel's describing — multiple transcription engines plus LLM reconciliation. Which honestly means he gets to name it.
Dangerous territory. Giving Daniel naming rights.
I'm thinking something like "consensus transcription" or "reconciled multi-model transcription." Not catchy, but descriptive.
"Transcription by committee." Every word has to survive a vote.
The committee model is actually a useful way to think about it. Each transcription engine is a committee member with its own biases and blind spots. The LLM judge is the committee chair who listens to everyone and then writes the final report.
And like any committee, the output is only as good as the diversity of its members. If you pack the committee with three instances of the same model, you get groupthink.
Which circles back to the independence point. The ideal setup is three different engines. But even with the same engine run three times, there's some value if you vary the parameters slightly — different temperature settings, different prompt prefixes. The outputs won't be identical, and the differences can surface uncertainty.
What about the practical implementation? Daniel asked specifically about packages and approaches for the reconciliation step.
For the alignment piece, if you go programmatic, the workhorses are libraries like jiwer for word error rate calculation — it includes alignment utilities — or textalign which is a Python port of those older alignment algorithms I mentioned. For the voting, it's mostly custom code, but there are packages like ensemble-transcription on GitHub, though they tend to be research code, not production-ready.
Research code. So it works on the author's machine with the author's test files.
And the author's specific Python environment from eighteen months ago, yes. For production, you'd probably build it yourself. The core logic isn't that complex — it's the edge cases that eat time. Things like: what do you do when one transcription has a completely hallucinated sentence that doesn't appear in the others? Simple voting would include it. You need a threshold — if a segment appears in fewer than half the transcriptions, drop it.
The hallucination problem is real. I've seen Whisper invent entire phrases that sound phonetically plausible but were never spoken.
It's especially bad with silence. Some models will hallucinate text when the audio goes quiet, because they've been trained to always produce output. So you get things like "thank you for watching" appended to the end of a clip that was just someone pausing.
Or the classic, "subscribe to my channel."
That one's so common it's practically a watermark. But the LLM judge handles this beautifully because it can recognize that "subscribe to my channel" makes no sense in the middle of a technical discussion about model fine-tuning, and it'll drop it even if two out of three transcriptions include it.
So the LLM judge isn't just doing majority voting — it's applying a plausibility filter.
Right. It's majority voting plus semantic coherence. That's the real advantage over the programmatic approach.
Let me play out a scenario. Daniel sends a thirty-second audio prompt. It hits the pipeline. Three transcription services return their outputs. The judge gets all three. One says "Let's talk about how Lauras could work in tax models." The other two say "LORAs." The judge sees "tax models" — which could go either way, honestly, tax models exist — but also sees that two out of three say "LORAs," and the surrounding context of the show is AI and machine learning. It picks "LORAs." But what if the audio was ambiguous? What if Daniel actually did say "Lauras" and was making a joke about people named Laura working in tax modeling?
Then the judge gets it wrong, and the error mode is the same as the original problem — a substitution that changes the meaning. But the probability of that happening is much lower than the probability of a single transcription engine making the same error. That's the whole point of the ensemble.
You're reducing the error rate, not eliminating it.
You're never eliminating it. Speech recognition is inherently probabilistic. There's always some residual error rate. The question is whether you can push it low enough that the remaining errors are acceptable for the use case. For a podcast prompt, where Daniel can still review the flagged uncertainties, I think this gets you well into acceptable territory.
What about the latency? He's currently dictating into his phone and getting a transcription back in, what, a couple of seconds? If he adds two more transcription calls plus an LLM reconciliation step, that's going to take longer.
It depends on the architecture. If you run the three transcription calls in parallel, the latency is roughly the slowest of the three plus the judge step. Whisper API can do a thirty-second clip in maybe two or three seconds. The judge step with a fast model is another second or two. So you're looking at maybe five seconds total. That's noticeable but not unusable.
And if you run them sequentially, it's three times longer.
Don't run them sequentially. There's no reason to. The calls are independent.
So we've got a name — ensemble decoding, or let's call it reconciled multi-model transcription. We've got an architecture — parallel transcription calls, optional pre-alignment, LLM judge, confidence markers. We've got the key insight that model diversity matters more than the number of runs. What about the packages? You mentioned a few for alignment, but Daniel asked specifically about the reconciliation step.
For the LLM-as-a-judge approach, the implementation is mostly prompt engineering plus API calls. You don't need a special package for the reconciliation — you need a well-structured prompt. Something like: "You are a transcription reconciliation system. Below are three independent transcriptions of the same audio. Produce a single reconciled transcription following these rules: one, prefer the majority reading where transcriptions disagree; two, prefer technical terminology over common words when context supports it; three, flag any word with less than unanimous agreement by wrapping it in brackets; four, remove hallucinated content that doesn't fit the surrounding context."
The bracketing for low-confidence words is a nice touch.
It gives Daniel back exactly what he had in his manual workflow — the ability to spot-check the uncertain parts. Except now he's only spot-checking maybe five percent of the text instead of the whole thing.
Let's talk about where this breaks. You mentioned correlated errors. What else?
The biggest failure mode is when the audio itself is bad — heavy background noise, overlapping speakers, strong accents that all the models struggle with equally. In that case, all three transcriptions might be garbled in similar ways, and the judge has nothing to work with. Garbage in, garbage out, just with more steps.
And the judge might actually make things worse in that scenario, by trying to reconcile garbled inputs into something that looks coherent but is completely wrong.
That's a real risk. The judge's strength — semantic coherence — becomes a liability when the inputs are so noisy that any coherent output is necessarily a fabrication. It's the hallucination problem all over again, just one level up.
So you need a confidence threshold at the input level too. If all three transcriptions have low average confidence — or if they diverge so wildly that alignment is impossible — the system should flag the whole thing for manual review rather than trying to reconcile.
Yes. A pre-judge gate. If the pairwise word error rate between the transcriptions exceeds some threshold — say, thirty percent — don't bother with reconciliation. Just send the audio to a human.
Or in Daniel's case, send it back to him with a note saying "this one was messy, you might want to re-record."
Which is still better than the current workflow, because most of the time it'll work fine and he won't have to think about it.
I want to zoom out for a second. The pattern Daniel's describing — multiple models, judge, reconciliation — this generalizes way beyond transcription. You could use the same architecture for any task where you want higher reliability than a single model can provide.
It's a general pattern for AI reliability. You see it in things like constitutional AI, where one model generates and another evaluates. You see it in retrieval-augmented generation, where multiple retrieval sources feed into a single generation step. The transcription use case is just a particularly clean example because the inputs are identical — same audio file — and the outputs are directly comparable.
And the cost of running multiple models has dropped enough that this is practical for consumer applications, not just research labs.
That's the thing that's changed in the last couple of years. Five years ago, running three separate speech-to-text models on every query would have been prohibitively expensive and slow. Now it's... what, maybe a tenth of a cent in API costs? The economics have shifted to make ensemble methods viable at scale.
So the bottleneck isn't cost anymore, it's engineering complexity.
And the engineering complexity is mostly in the edge cases, as usual. The happy path — three clean transcriptions that mostly agree — is straightforward. It's the unhappy paths that take the work. What do you do when one transcription service times out? When one returns a completely different language? When the audio contains a code-switch between English and Hebrew and only one of the three models handles it well?
Daniel's in Jerusalem. Code-switching between English and Hebrew is not a hypothetical for him.
And most commercial speech-to-text models are trained primarily on monolingual English. Whisper handles code-switching reasonably well, but it's not perfect. So for Daniel's specific use case, model selection matters a lot. He'd want at least one model in the ensemble that's known to handle Hebrew-English code-switching well.
Which model would that be?
Whisper large is decent. There's also a model called Ivrit-AI that's specifically trained on Hebrew, but it's more focused on pure Hebrew than code-switching. Honestly, for the Jerusalem tech scene use case, I'd probably include Whisper, Deepgram, and maybe a fine-tuned Whisper variant that's been trained on multilingual technical content.
And the judge model needs to handle the code-switched output too.
The judge has it easier because it's working with text, not audio. Most modern LLMs handle multilingual text fine. Claude and GPT-4 both handle Hebrew-English code-switching without issues.
So the pipeline for Daniel specifically would be: audio in, parallel calls to Whisper, Deepgram, and maybe a third service, all chosen for multilingual capability. Pre-alignment to flag divergences. Judge LLM reconciles, with special attention to Hebrew-English boundary words. Confidence markers on low-agreement terms. Output to Daniel with flags for manual review.
That's the blueprint. And the nice thing is, once you've built it for transcription, the same architecture works for translation, for summarization, for anything where you want to combine multiple model outputs into a single higher-quality result.
We should probably give Daniel a concrete answer on the name, since he asked directly.
Fair. The academic literature calls this ensemble decoding or multi-hypothesis reconciliation. The industry pattern is LLM-as-a-judge. For the specific pipeline — multiple STT engines plus LLM reconciliation — there'sn't a widely-adopted name yet. I'd call it reconciled ensemble transcription. But honestly, Daniel can call it whatever he wants and it'll probably stick, at least in his own documentation.
"Daniel's committee method." Every word confirmed by a vote.
I'd read that paper.
The other thing he asked about was the specific difficulty of producing one non-repetitive output from divergent transcriptions. That's really the crux of the engineering challenge, isn't it?
It is. And the LLM judge solves it differently than the programmatic approach. The programmatic approach aligns, votes, and then has to reconstruct — and the reconstruction step is where you often get stilted or repetitive output, because you're stitching together fragments from different sources. The LLM judge doesn't stitch — it reads and re-writes. The output is a fresh generation informed by the inputs, not a patchwork of the inputs.
Which produces more natural text.
Much more natural. And it handles the repetition problem implicitly. If two transcriptions both include the same sentence, the judge writes it once. If they include slightly different versions of the same idea, the judge picks the cleaner version. You don't need explicit deduplication logic.
The tradeoff is that the judge might paraphrase rather than transcribe exactly. If Daniel's original words were precise and the judge "improves" them slightly, that could be a problem.
That's a real concern, and it's why the prompt engineering matters. You need to instruct the judge to reconcile, not to edit. "Preserve the original wording wherever the transcriptions agree" should be an explicit instruction. The judge should only be making choices where the transcriptions diverge, not polishing the prose.
"You are a reconciliation system, not an editor." That's the first line of the prompt.
And you can reinforce it with examples in the prompt — show the judge a sample where it should preserve the original wording even if it's slightly awkward, and only intervene where there's a genuine disagreement.
Daniel's going to build this, isn't he.
I'd be surprised if he doesn't have a prototype running by the end of the week. This is exactly the kind of pipeline engineering he enjoys.
And then we'll get audio prompts back in the show pipeline, which was the whole point.
Full circle. Though I have to say, the text prompts have been working fine.
They have. But there's something about hearing Daniel's actual voice in the episode that the text-to-speech reading doesn't capture. The pacing, the emphasis. It's a different quality of presence.
Agreed. And if this architecture gives him confidence that the transcription won't mangle his technical terms, I think we get the best of both worlds — the authenticity of audio with the accuracy of reviewed text.
The committee will protect him from the Lauras.
The Lauras will not prevail.
Hilbert: You're talking about running the same thing three times and then asking a fourth thing which version is right.
That's... yes. That's the architecture.
Hilbert: I did something like this in ninety-four. Not with speech. With barcodes. I was working inventory at a warehouse in Hartford — this is before scanners got good, the handheld ones would misread maybe one in forty labels if the print was smudged. So we had three people scan the same pallet, three different scanners, and I'd sit at the terminal comparing the readouts. If two out of three matched, that was the number. If all three were different, someone went and looked at the physical label.
That's literally the same voting architecture, just with barcodes instead of phonemes.
Hilbert: The hard part wasn't the voting. It was when two scanners agreed on the wrong number. Happened with certain digits — three and eight looked similar to the older scanners, especially if the label was creased. Two out of three would confidently say eight, and we'd ship a pallet to the wrong dock. Took us six months to figure out the pattern.
Correlated errors.
Hilbert: That's what you called it. Yeah. The scanners all had the same weakness, so the vote didn't help. We ended up replacing one of the scanner brands with a different model that had a different misread pattern. Then when they disagreed, at least one of them was probably right.
That's exactly the model diversity point. Did the replacement fix it?
Hilbert: Mostly. We still had a bin of misdirected pallets every month, but it went from about twenty to maybe two. Good enough for the warehouse manager. He was a practical guy. Didn't need perfection, just needed the error rate low enough that the trucking company stopped complaining.
What I'm taking from this is that the architecture works, but the model selection is the part that actually determines whether it works well.
Hilbert: The models are the whole thing. The voting is just arithmetic. Anyone can do arithmetic.
Hilbert, I'm curious — in your barcode setup, how did you handle the case where all three scanners gave different reads? You said someone checked the physical label. Was that common?
Hilbert: Maybe one in two hundred pallets. Usually meant the label was torn or had something spilled on it. The interesting cases were when all three agreed but the number didn't match the manifest. That meant the label was wrong from the start — misprint at the factory. No amount of voting fixes a bad source.
Garbage in, garbage out.
Hilbert: Garbage in, three scanners agree it's garbage, you ship the garbage to Vermont.
Vermont specifically?
Hilbert: That's where the misdirected pallets usually ended up. There was a distribution center in Burlington that had a similar address code to ours. I think about forty pallets went there over the two years I worked that job. They never sent them back. I always wondered what they did with them.
They probably built their own warehouse out of misdirected pallets.
Hilbert: Wouldn't surprise me.
The point about all three agreeing on a wrong source — that's the limitation Daniel would hit if the audio itself is the problem. If he mumbles, or if there's construction noise in the background, all three transcription engines might agree on the wrong word.
Hilbert: Then you need a fourth thing that listens to the audio directly. Not the text. The sound.
A human.
Hilbert: A human works. Or you just accept that some percentage will be wrong and move on. That warehouse manager I mentioned — he had a sign above his desk. Said "perfect is the enemy of shipped." I think about that sign a lot.
That's... actually a pretty good engineering philosophy for this kind of system.
Hilbert: He was a practical guy.
I keep thinking about the confidence markers you mentioned, Herman. The brackets around low-agreement words. That's the bridge between "accept the error rate" and "review everything manually." It lets Daniel decide case by case whether the uncertainty matters.
And the nice thing is, the judge can be tuned for how aggressive the bracketing is. If Daniel wants to review more, you lower the threshold — bracket anything with less than unanimous agreement. If he wants to review less, you raise it — only bracket words where the transcriptions are evenly split.
The knob goes from "trust but verify" to "verify everything."
And he can adjust it over time as he builds confidence in the system. Start conservative, lots of brackets, review everything. After a few dozen prompts where the bracketed words turn out to be correct, he can loosen the threshold.
That's the adoption curve for any automated system. Start with the human in the loop, gradually reduce the human's involvement as trust builds.
The difference here is the human never fully leaves the loop. Daniel's still reviewing, just more efficiently. It's augmented transcription rather than automated transcription.
Which is honestly where most AI applications should land. Full autonomy is brittle. Augmented human judgment is robust.
And it's a better fit for the use case. A podcast prompt isn't a high-volume, low-stakes application where you can tolerate occasional errors. It's low-volume, medium-stakes — Daniel cares that his words come through correctly because they're going into a published episode.
The stakes are "Herman will spend twenty minutes analyzing a mistranscribed word and we'll all be confused."
Which has happened.
I think we've covered the ground Daniel asked about. The architecture has a name — or several names, ensemble decoding being the most established. The LLM-as-a-judge pattern is the modern implementation that handles the reconciliation step better than programmatic diffing. The key engineering insight is that model diversity matters more than the number of runs. And the practical advice is: parallel calls, a well-structured judge prompt, confidence markers on the output, and a pre-judge gate for audio that's too noisy to reconcile.
And the packages. For alignment: jiwer, textalign. For the judge: any LLM API with a good prompt. The rest is custom code, because the edge cases are specific to your audio, your domain, your tolerance for error.
Daniel's going to want to build the whole thing as a single Python script with a YAML config file, isn't he.
I'd bet on it. Probably call it something like "reconciled-stt" and put it on GitHub within the week.
We'll find out when the next audio prompt lands in the pipeline.
Looking forward to it. This has been My Weird Prompts. Thanks to our producer Hilbert Flumingtop for keeping this show running — and apparently for inventing ensemble methods in a Hartford warehouse in nineteen ninety-four.
If you want to send us your own prompts, whether by text or audio — we'll handle either — email the show at show at my weird prompts dot com. Or visit my weird prompts dot com for the full archive.
We'll be back soon.
I'm going to go think about those forty pallets in Burlington.
