Daniel's been thinking about the voices you're hearing right now — literally the audio of this podcast — and he's zeroed in on something that's been quietly bugging him. He writes in that we've talked about TTS models in several recent episodes, and the tech powering this show is Chatterbox by Resemble AI. The quality has leapt forward so far that the pre-AI generation of speech synthesis is barely recognizable next to what we have now. But here's the twist he wants to dig into — and it's a sign of maturity in the field, not a shortcoming. Real human conversation is messy. People cut each other off. They overlap. They laugh, gasp, trail off mid-thought, catch themselves in tangents and circle back. Nobody takes perfectly spaced, clean turns. So the architectural challenge for anyone producing a podcast like this — script generation agent feeding into TTS — is actually to make the dialogue more imperfect. Not to polish it further, but to roughen it up in the right ways. He asks what modality makes the most sense for this, whether multimodal models that compact the whole pipeline into one step might be better suited, and what early-stage approaches are being prototyped to close the realism gap in TTS audio — especially for long-form, multi-person dialogue with distinct personalities. So today we're going to pull back the curtain on our own voices and why they don't sound quite human yet.
And I love this question because it inverts the entire history of the field. For decades the goal was intelligibility — make the machine pronounce every word clearly, no artifacts, no noise. We spent billions of dollars and probably millions of engineer-hours scrubbing imperfection out of synthetic speech. And now Daniel's asking: can you put some of it back in, but deliberately, in the places where it belongs? That's a genuinely hard problem.
It's like we spent a century breeding the seeds out of watermelons and now someone's asking if we can put them back.
That's... actually a perfect analogy. And the reason it's hard is that the seeds aren't just random black dots you sprinkle in. They have a structure, a distribution, a relationship to the fruit. Human disfluency works the same way. A filler word isn't noise — it's a signal that the speaker is holding the floor while retrieving the next word. An overlap isn't a bug — it's often a sign of engagement, of someone anticipating where the sentence is going. The model has to learn the grammar of imperfection, not just inject chaos.
So let's start by unpacking what we mean by realism — because it's not just about sounding human, it's about sounding like a human in a conversation.
Right. The realism gap is the distance between what current TTS produces and what actual human dialogue sounds like. Current systems give you clean, turn-based speech. Speaker A finishes. Brief silence. Speaker B begins. Every utterance is a self-contained, grammatically complete unit. No one ever says "uh," no one ever starts a sentence and abandons it halfway through, no one ever talks over anyone else. And your brain notices. Not consciously, necessarily — but long-form listening creates this subtle unease, this sense that something is off. You're hearing two people who never interrupt, never hesitate, never breathe wrong. Real humans don't talk like that. Real humans are a mess.
I've been told I'm very deliberate.
You're a sloth. You're an outlier and should not have been counted.
Fair. But the point stands — the perfection is the problem. It's the uncanny valley of conversation. We've made the audio so clean that it loops back around to feeling synthetic.
And the core tension here is fascinating. The entire history of speech synthesis was a war against disfluency. Early concatenative TTS — where you stitch together recorded phonemes — sounded choppy and unnatural because the transitions between units were jarring. Then parametric models smoothed things out but sounded buzzy and robotic. Then WaveNet and the neural revolution gave us that buttery, natural prosody. Every step was about removing artifacts, removing glitches, removing anything that reminded you a machine was speaking. And now we're at the point where the remaining artifacts are the absence of human imperfection. We need to teach the machine to stumble convincingly.
So to understand why our dialogue sounds too perfect, we need to look under the hood of how TTS models are built.
The first thing to understand is what these models are trained on. The overwhelming majority of TTS training data is single-speaker, clean audio recorded in a studio. One person reading sentences into a microphone in a treated room. No background noise, no other voices, no interruptions — because why would there be? The task was "produce this one person's voice reading this text." And the models got extraordinarily good at exactly that. Chatterbox, the system generating our voices, is trained on precisely this kind of data. It learns the mapping from text to speech for a specific voice in ideal conditions.
So it's never seen a conversation.
It's never seen a conversation. The training data contains no examples of two people talking. No overlaps. No backchannels — those little "mm-hmm" and "uh-huh" sounds listeners make to show they're following. No laughter that bleeds across speaker boundaries. The model has no statistical representation of what it sounds like when one person starts speaking before another person has finished. It doesn't know that in English, the average gap between turns is about two hundred milliseconds — roughly the time it takes to blink. It doesn't know that when someone is about to yield the floor, their pitch tends to drop and their final syllable lengthens. None of that structure is in the training distribution.
Two hundred milliseconds. That's... nothing.
It's nothing. And here's the thing — when two humans are really engaged in a conversation, the gaps get even shorter. Sometimes negative, which is just a fancy way of saying they overlap. A study out of Columbia a few years back found that in casual conversation among friends, overlaps occur in something like thirty to forty percent of turn transitions. Not interruptions in the rude sense — collaborative overlaps, where the second speaker is completing the first speaker's thought or adding emphasis. "I was thinking we could —" "— go to the place on Fourth Street?" That kind of thing. Current TTS dialogue has zero percent overlap, because the pipeline can't do it.
Walk me through why the pipeline can't do it. Because on the surface, it seems like you could just... overlap the audio tracks.
You could, and that's actually what some naive approaches try — just layer the waveforms in post-production. But it sounds terrible, and here's why. The pipeline is linear: text generation first, then TTS. The language model writes a script. That script is a sequence of clean, alternating turns. Corn says something. Then Herman says something. Then Corn says something. Each utterance is generated in isolation and sent to the TTS engine separately. The TTS engine renders Speaker A's line. Then it renders Speaker B's line. At no point does either system have access to the other speaker's acoustic context.
Acoustic context meaning...
Meaning what the other voice actually sounds like in that moment. When I interrupt you in real life, I hear the pitch you're at, the volume you're at, the exact phoneme you're on — and my brain times my interruption based on that acoustic signal. I'm not reading your transcript. I'm hearing your voice. The TTS pipeline has no equivalent. It's working from text, and text has no pitch, no timing, no prosody. So even if you try to overlap the audio after the fact, the two voices don't interact acoustically. They sound like two recordings playing at the same time, not two people in the same room.
Two radios on the same desk.
And there's a deeper problem too, which is prosodic entrainment. This is one of those phenomena that's obvious once you hear about it but most people never notice consciously. When two people talk, they unconsciously match each other's speech rate, pitch range, and even accent features over the course of a conversation. It's a social bonding mechanism. You slow down if the other person slows down. Your pitch converges toward theirs. Current TTS models don't do this at all because each speaker is synthesized independently with fixed voice parameters. My voice always has the same average pitch, the same speaking rate, regardless of what you just said or how you said it. A real conversation breathes — the voices adapt to each other. Our voices just... recite.
So even at the state of the art, even with Chatterbox producing impressive individual voice quality, the dialogue as a whole is fundamentally missing the interaction layer.
And it's not a flaw in Chatterbox specifically. It's a category problem. Every TTS system in production today is optimized for single-speaker clarity. The evaluation metrics bear this out — mean opinion score tests ask listeners to rate naturalness, but they're almost always rating isolated utterances, not conversations. The entire field has been optimizing for a task that's adjacent to what we need but not the same thing.
Let me push on something. Daniel mentioned that the script itself might be part of the problem — that the text generation step produces dialogue that's too clean even before TTS touches it.
He's absolutely right, and this is the misconception I want to flag early. A lot of people think the realism gap is purely a TTS problem — just make the audio sound more natural and we're done. But the text is already unnatural. Language models, even good ones, tend to produce dialogue that's too structured. Every turn is a complete paragraph. Nobody says "wait, no, start over." Nobody loses their train of thought. The model is optimizing for coherence and informativeness, and real conversation is full of moments that are neither coherent nor informative in the traditional sense. They're social. They're phatic. They're the verbal equivalent of grooming.
I do sometimes catch myself mid-tangent.
You do, and that's in the script. But the script has to explicitly write that in. The language model has to decide "Corn will now interrupt himself." It's choreographed imperfection. And choreographed imperfection is its own kind of uncanny — if every disfluency is perfectly placed, it reads as artificial in a different way. The model needs to generate disfluency emergently, not by following stage directions.
Which brings us to the frontier. If the current pipeline is the problem, what's the alternative?
So this is where it gets really interesting. The most promising direction — and it's still early, I want to be clear about that — is multimodal models that generate audio directly from text, bypassing the script-then-TTS pipeline entirely. Instead of "write text, then speak text," you have a single model that maps from the semantic representation straight to a multi-speaker audio waveform. The model is reasoning in a shared latent space that represents both the linguistic content and the acoustic properties simultaneously.
Unpack "latent space" for a second.
Think of it as the model's internal representation — the compressed, abstract encoding of everything it knows about the input. In a text-only model, that latent space only captures linguistic patterns. In an audio-only model, it only captures acoustic patterns. But in a multimodal model trained on both text and audio, the latent space can represent things like "this word is being interrupted right now" or "this sentence is trailing off in pitch because the speaker is uncertain." The model doesn't have to decide the words first and then figure out how to say them — it generates both in one pass, and the acoustic decisions can feed back into the linguistic ones.
So the model could decide mid-generation that an overlap should happen, and the overlap would be acoustically coherent because both voices are being synthesized from the same latent representation.
That's the idea. There are research prototypes moving in this direction. Audio language models — systems that treat audio tokens the way text language models treat word tokens — can generate speech directly. Google's AudioLM, which they published a couple of years ago, showed you could generate continuations of audio prompts with remarkably natural prosody, including hesitations and filled pauses that emerged from the model rather than being scripted. It wasn't doing multi-speaker dialogue, but the architecture is extensible. You can imagine training a model on conversational corpora — real recorded dialogues with speaker labels — and having it learn to generate multi-speaker audio with natural turn-taking dynamics baked in.
What are the conversational corpora that exist for this?
That's one of the bottlenecks. There's Switchboard, which is a classic — hundreds of hours of telephone conversations between strangers, meticulously transcribed with timing annotations. There's the Corpus of Spontaneous Japanese, which captures casual conversation with detailed prosodic markup. There's the Santa Barbara Corpus of Spoken American English, which is smaller but very richly annotated. The problem is scale. These corpora are tiny compared to the datasets used to train foundation models. Switchboard is about two hundred sixty hours of audio. For comparison, the datasets used to train large language models contain trillions of words. We simply don't have conversational audio at internet scale.
Because conversations are private, mostly.
Because conversations are private, and the ones that are public — podcasts, interviews, broadcast media — tend to be more structured than casual speech. A podcast host and guest are performing conversation, not having one. They're more coherent, more turn-based, less overlapping than two friends at a kitchen table. So even the publicly available conversational data is biased toward the kind of clean dialogue we're trying to escape.
So we're training on performances of conversation to generate performances of conversation, and wondering why it sounds performed.
That's the recursion, yes. And it gets worse when you add the personality problem. Daniel asked about distinct personalities in multi-person audio. Right now, we handle that with speaker embeddings — essentially a vector that captures the acoustic fingerprint of a specific voice. You train the model with these embeddings, and at inference time you say "generate audio with embedding A for speaker one and embedding B for speaker two." It works reasonably well for short utterances. But over long-form content, like a twenty-five-minute podcast, speaker identity drifts. The model gradually loses track of which voice is which, or the voices start to converge toward some average. Maintaining consistent vocal personality across thousands of tokens of generated audio is an open research problem.
And personality isn't just voice timbre. It's word choice, pacing, humor, sarcasm.
It's everything. A sarcastic remark isn't just the words — it's a specific prosodic contour, often with exaggerated pitch variation and a particular timing pattern. The model has to know not just that this line is sarcastic, but how sarcasm sounds in this specific voice. And it has to do that while also managing turn-taking, backchanneling, and all the other layers of conversational realism. The complexity compounds fast.
So let's talk about what people are actually trying. What are the early-stage approaches being prototyped?
There are a few threads worth following. One is what researchers call conversational TTS — a two-stage approach where you first generate a dialogue graph that includes not just the words but annotations for timing, overlap, and prosodic events. Then a second stage synthesizes the audio with an overlap-aware acoustic model that can render two voices simultaneously while maintaining acoustic coherence. The dialogue graph is essentially a score — like sheet music for conversation — that specifies who speaks when, where the overlaps happen, and what the prosodic contour should be.
And who writes the score?
That's the question. In current prototypes, it's mostly rule-based or uses a separate model trained on annotated conversational data. The rules encode things like "if speaker B's response is an acknowledgment like 'yeah' or 'right,' it can overlap with the end of speaker A's turn." Or "if speaker A's pitch is falling and their speech rate is slowing, that's a turn-yielding cue, and speaker B can begin." These are linguistic findings that have been known for decades — conversation analysts in the nineteen-seventies were documenting turn-taking mechanics in exquisite detail. The challenge is operationalizing that knowledge in a generative model.
So we're encoding nineteen-seventies sociology into twenty-twenties neural networks.
We are, and it's oddly beautiful. Harvey Sacks and Emanuel Schegloff would be astonished. But the rule-based approach has limits. Real conversation is too variable, too context-dependent. You can't write enough rules. So the other thread is end-to-end learning — train a model on enough conversational audio that it internalizes the patterns without explicit rules. This is where multimodal models become essential. If you can train a single model on paired text and multi-speaker audio, it can learn that certain linguistic contexts — a question, an incomplete thought, a moment of high emotion — correlate with specific acoustic patterns like overlapping speech or filled pauses.
The training data problem rears its head again.
Always. But there's a clever workaround that's gaining traction: synthetic data generation. You take existing single-speaker TTS voices and you simulate conversations by having them interact according to probabilistic turn-taking models. You generate thousands of hours of synthetic conversational audio, complete with overlaps and backchannels, and you use that to train a multi-speaker model. It's bootstrapping — the synthetic data isn't perfect, but it gives the model a prior, a rough sense of what conversational dynamics look like, which can then be fine-tuned on the smaller amount of real conversational data that exists.
That's the "fake it till you make it" school of machine learning.
Which has a surprisingly good track record. But there's a pitfall here, and it connects back to something Daniel mentioned. Deliberately injecting disfluencies can backfire if you do it mechanically. If the model learns to insert an "um" every seven seconds because that's the average rate in the training data, it sounds worse than no "ums" at all. The disfluencies have to be motivated — they have to occur at points of genuine cognitive load, like when the speaker is retrieving a difficult word or reformulating a complex thought. The model needs to learn the why, not just the what.
The uncanny valley of imperfection.
A perfectly placed "um" is more uncanny than no "um" at all, because your brain detects that it's being performed rather than experienced. The listener's subconscious is an unforgiving critic.
What about reinforcement learning as a way to tune this? Daniel mentioned RLHF in passing — could you train a reward model on human judgments of conversational naturalness?
I think that's probably where this is headed. You'd need a different kind of evaluation than the standard mean opinion score. Instead of asking listeners "does this sound natural?" for isolated utterances, you'd ask them to rate entire conversations — "does this feel like two people talking?" The reward model would learn to predict those ratings, and then you'd use it to fine-tune the generative model. The tricky part is that conversational naturalness is holistic. You can't decompose it into "the overlaps were good but the backchannels were weird." It's a gestalt judgment. Listeners know it when they hear it, but they can't always articulate why.
Which makes it hard to optimize for.
Hard, but not impossible. The same problem exists in image generation — "does this face look real?" is a holistic judgment, and we've gotten very good at optimizing for it. The difference is scale. We have millions of labeled faces. We don't have millions of labeled conversations.
Where does that leave someone like Daniel, producing a podcast with current tools? What can actually be done today?
Today, the practical answer is mostly script-level intervention with some light audio post-processing. You can instruct the language model to generate disfluencies, false starts, and interruptions explicitly in the text. You can add stage directions that the TTS system ignores but that shape the cadence of the written dialogue — things like leaving sentences unfinished, using ellipses, writing in overlaps by having one speaker's line begin mid-sentence. Then in post, you can manually adjust timing — tighten gaps, layer in occasional overlaps by editing the audio tracks. It's labor-intensive and it's never going to feel fully spontaneous, but it's what we've got.
The other thing you can do is vary the prompting strategy. If you prompt the language model with examples of real conversational transcripts — messy ones, with overlaps marked and disfluencies transcribed — it's more likely to produce dialogue in that style. It's still text, but it's text that carries the fingerprints of speech. The TTS system will render it more naturally because the underlying prosody of the written language is different. Shorter sentences. More fragments. More questions that trail off.
Before we wrap up, Hilbert's been making faces back there. I think he has something to say.
Hilbert: I used to do this by hand.
Do what by hand?
Hilbert: Make fake conversations sound real. Late nineties, I was producing radio drama for a public station in Vermont. Tiny operation. We had two actors, two mics, and a quarter-inch tape deck. The scripts were clean — playwrights write clean, it's what they do — but the recordings came out stiff. So I'd sit there with a razor blade and splicing tape, cutting out breaths, moving pauses around, sometimes adding a stutter by snipping a syllable and pasting it back in.
You physically cut the tape.
Hilbert: That was the job. I spent three hours once on a thirty-second exchange. Two characters arguing. The words were fine but the rhythm was wrong — the interruptions landed a beat too late, so it sounded like they were waiting for each other. I had to shave milliseconds off the gaps, one splice at a time, until the anger felt real.
Three hours for thirty seconds.
Hilbert: Still have the reel somewhere. I could dig it out if you want to hear it.
I think we're good.
Hilbert: Suit yourself. But here's the thing. You're trying to automate what I did with a razor blade. The problem isn't the model — it's that you're asking it to be a director, not just a voice actor. A voice actor reads lines. A director knows that the pause before "I'm fine" is the whole performance.
That's... actually the most concise framing of this I've heard.
Hilbert: We also used a two-mic setup — cardioid mics, positioned so each actor bled a little into the other's channel. That bleed is what made the overlaps sound like they were in the same room. Your models are trying to simulate that bleed in software, without the room, without the mics, without the air. Good luck.
Did you ever get it perfect? The thirty-second scene?
Hilbert: It aired. People said it sounded spontaneous. That's the thing about this work — if you do it right, nobody knows you did anything at all.
That's the paradox, isn't it? The goal is invisibility. The best conversational TTS will be the one where listeners stop noticing they're listening to TTS. Not because it's perfectly clean, but because it's perfectly messy in the ways they expect.
That's a hard act to follow. Let's step back and think about where this is all heading. The open question that keeps nagging at me is whether we'll ever fully accept AI dialogue that's imperfect in the right ways, or whether there will always be something that feels off — some ineffable quality of listening that the machine can't fake.
I think the line will blur faster than we expect. As multimodal models improve, the distinction between scripted and improvised audio will erode. Podcasts like this one might eventually be generated in a single pass — not a script fed to TTS, but a model that decides in real time when to interrupt, when to hesitate, when to laugh. The model won't be reading a script. It'll be... conversing. With itself, in two voices, but conversing nonetheless.
The goal isn't to make AI sound human.
No. It's to make it sound like it's listening.
The misconception I want to leave people with — the thing most coverage gets wrong — is that this is purely a TTS problem. Add better audio, problem solved. But the realism gap starts in the text. The script itself is too clean, too structured, too much like writing and not enough like speech. You can't fix it in post if the words were never alive to begin with.
The corollary: adding random filler words doesn't help. "Um" is not a spice you sprinkle on. It's a signal that belongs in specific cognitive contexts, and if the model doesn't understand those contexts, the disfluencies make things worse, not better. Imperfection has a grammar. We're just starting to learn it.
Thanks to our producer Hilbert Flumingtop, who apparently spent the nineties with a razor blade and a dream.
This has been My Weird Prompts. We'll be back soon.
See you tomorrow.
