Daniel's been neck-deep in TTS lately. He sent in a whole thing about how the same architecture, the transformer, flipped speech synthesis on its head, and he's grateful to Resemble AI for open-sourcing a model that's powered hundreds of hours of this podcast. His question, stripped down, is how does neural TTS actually work under the hood, and what was it about the transformer that made it so much better than the old robot voice?
The old robot voice. Everyone knows exactly what that sounds like.
The Linux repository still has those legacy programs sitting there. espeak, festival, flite. You fire one up and it's like listening to a speak-and-spell that learned to read. Daniel's point is that we've gone from that to something nearly indistinguishable from a human, and now we're actually spending effort degrading the models on purpose, adding ums and pauses back in. He says that's the real sign of maturity.
It is. When you're trying to make your perfect system less perfect to fool people into thinking it's a person, you've crossed a threshold. So let's open the hood and see what changed.
Let's start with what the robot voice actually was.
Two completely different approaches, and neither one learned anything. The first is formant synthesis. That's what espeak uses. You build a mathematical model of the human vocal tract, the resonant frequencies, the way the tongue and lips shape sound, and you drive it with rules. Rules for how a question mark bends the pitch up, rules for how a period drops it. Hundreds and hundreds of hand-tuned parameters.
So it's not playing back a recording. It's calculating sound from scratch based on an acoustic model of a throat.
And it sounds like it. The second approach is concatenative synthesis. That's what festival and flite do. You record a human speaker saying thousands of phonemes, diphones, sometimes whole words, chop them up, and stitch them back together in the right order. The problem is the joins. A phoneme sounds different depending on what's next to it, and when you splice two pieces from different recordings, the transition is never quite right. You get this slightly glitchy, choppy quality.
The Frankenstein approach. You've got a vowel from one sentence and a consonant from another and you just hope the listener's brain fills in the gaps.
And it works, sort of, for what it was designed for. Screen readers, phone menus, accessibility tools where clarity matters more than pleasantness. But nobody would ever choose to listen to it.
So what changed? What's the actual mechanism that made the leap?
The core shift is from rules and stitching to learning. Neural TTS trains a deep neural network on thousands of hours of human speech paired with the corresponding text. The network learns the mapping directly. No hand-tuned rules, no phoneme database. It figures out on its own that certain letter combinations tend to produce certain sounds, that a comma means a slight pause, that the word "record" is stressed differently depending on whether it's a noun or a verb.
And the thing that made that possible at scale is the transformer.
The transformer. Introduced in twenty seventeen in the "Attention Is All You Need" paper, originally for machine translation. The key insight is something called self-attention.
Walk me through it. Text goes in, speech comes out. What happens in between?
All right. Step one, the text gets tokenized. Broken into pieces, roughly words or subwords, and each token gets converted into a numerical vector, an embedding. That embedding captures something about the meaning of the token in isolation. But the real magic is what happens next. Those vectors go through a stack of transformer layers, and in each layer, the self-attention mechanism lets every token look at every other token in the sequence simultaneously.
Simultaneously. That's the word that matters.
That's the whole thing. Before transformers, the dominant approach was recurrent neural networks, RNNs, which process text one token at a time, left to right. By the time you get to the end of a long sentence, the model has sort of forgotten what was at the beginning. Self-attention doesn't have that problem. Token seventeen can directly attend to token three. A question mark at the end of a sentence can reach all the way back and reshape how the first word is spoken.
So it's not just remembering context, it's seeing the whole sentence at once and understanding that the word "lead" at the start might be pronounced differently because of something that happens six words later.
Right. And that's exactly the kind of long-range dependency that prosody depends on. Prosody, the rhythm, stress, and intonation of speech. The old systems had no real prosody model. They'd apply a simple rule like "raise pitch at the end of a question" and call it done. A transformer can learn that a parenthetical clause should be spoken slightly faster and at a lower pitch, that the word "actually" in the middle of a sentence often signals a contrast that needs emphasis, that a list of three items has a specific cadence.
The comma gets its own tiny pause.
The comma gets its pause, and it's not a fixed duration. It varies based on the surrounding structure. The model learns that from the data.
So the attention mechanism produces these context-aware representations. Then what?
The output of the transformer stack is a sequence of hidden states, one per input token. Those get passed to a decoder that converts them into an acoustic representation. Most modern systems use a mel-spectrogram as the intermediate step. A mel-spectrogram is a visual representation of sound, frequency over time, tuned to roughly match human hearing. The decoder generates a sequence of mel-spectrogram frames.
So at this point we still don't have audio. We have a picture of audio.
A very detailed picture. And that's where the vocoder comes in. The vocoder takes the mel-spectrogram and synthesizes the actual waveform, the raw audio samples. Early neural TTS used WaveNet, which was autoregressive and painfully slow. A few seconds of audio could take minutes to generate. Modern systems use things like HiFi-GAN, which is a generative adversarial network that can produce waveforms in real time.
HiFi-GAN. That's a great name.
It's a great piece of technology. It's trained adversarially, a generator produces audio and a discriminator tries to tell if it's real or fake. Over time, the generator gets extremely good at producing waveforms that sound natural, with all the tiny imperfections and harmonics that make a voice sound like a voice instead of a tone generator.
So the full pipeline is text to tokens to embeddings to transformer to mel-spectrogram to vocoder to audio. And every piece of that is learned from data.
Every piece. That's the fundamental difference from legacy TTS. There's no phoneme dictionary curated by linguists, no hand-crafted rules about how a declarative sentence falls in pitch. The system learns everything from the training data. And the transformer is the engine that makes the whole thing work because it can capture the structure of language in a way that earlier architectures simply couldn't.
Let's make this concrete. Take a sentence. "I didn't say he stole the money." Seven words, and depending on which one you stress, it means seven different things.
Classic example. "I didn't say he stole the money" means someone else said it. "I didn't say he stole the money" means I implied it but didn't say it. And so on.
An espeak-style system reads that sentence exactly the same way every time. Flat, slightly descending pitch, done.
A transformer-based system, trained on enough data, has seen that pattern. It's seen sentences where "didn't" gets emphasis, where "stole" gets emphasis, and it's learned the acoustic correlates of emphasis, higher pitch, longer duration, more energy. It doesn't know what emphasis means in any semantic sense, but it's learned the statistical relationship between the textual context and the acoustic output.
And this is the same architecture that does text generation and image generation.
That's what's wild. The transformer started in NLP, machine translation specifically, and then it just colonized everything. Vision transformers for image recognition. Diffusion transformers for image generation. Whisper for speech-to-text. The same self-attention mechanism, adapted slightly for different input modalities, but the core idea is identical. Text is a sequence of tokens. An image is a sequence of patches. Audio is a sequence of spectrogram frames. If you can represent something as a sequence, you can throw a transformer at it.
The universal architecture.
It really is. And the TTS version of it, the thing that makes it special, is that it's not just learning the mapping from text to sound. It's learning the mapping from text to sound in a way that captures all the paralinguistic stuff. Emotion, tone, style. A good modern TTS model can read a sentence with sarcasm if the context calls for it. Not because someone programmed a sarcasm rule, but because the training data contained sarcastic speech and the model learned the pattern.
Which brings us to the disfluencies. The ums and pauses.
This is where it gets philosophically interesting. For years, the goal was to make TTS as clear and perfect as possible. Remove every hesitation, every filler word, every false start. And we got there. We got models that can read any text with flawless diction and perfect pacing. And people hated it.
It sounded wrong.
It sounded like a machine pretending to be human and failing. The uncanny valley of speech. The voice is almost perfect, but the delivery is too perfect, and your brain flags it as fake. So now the research direction has flipped. People are deliberately adding disfluencies back in. Inserting "um" and "uh" at syntactically appropriate boundaries. Adding micro-pauses. Introducing slight inconsistencies in pacing.
The model is so good we have to make it worse on purpose.
And that's Daniel's point about maturity. You don't spend effort degrading a technology unless the raw output is already excellent. It's like film grain in digital photography. Early digital cameras fought to eliminate grain, and then once they succeeded, photographers started adding it back because the perfectly clean image felt sterile.
Or auto-tune. Perfect pitch correction was the goal, then suddenly everyone wanted to sound a little off.
The imperfections are the signal of humanity. And the transformer is what makes it possible to model those imperfections in a controlled way. You can't just randomly insert an "um" anywhere. It has to land at a clause boundary, before a low-frequency word, in a way that mimics the distribution of disfluencies in real speech. The model learns that distribution.
There's something almost backwards about it. The old systems were imperfect because they were bad. The new systems are imperfect because we've learned that imperfection is a feature.
And the old systems, to be fair, had their own kind of honesty. An espeak voice never tried to fool you. It announced itself as a machine with every syllable. There was no pretense.
Which is a different kind of listening experience. You adjust your expectations.
You do. With a modern neural voice, you don't adjust anything. You just listen. And that's the triumph. The technology disappears.
Let's talk about where it still falls short. Daniel mentioned there's room for improvement.
Several areas. Emotional nuance is the big one. Current models can do basic emotional coloring, happy, sad, angry, but they can't really track a complex emotional arc across a long passage. A human narrator subtly shifts their tone as a story builds tension. A TTS model tends to apply a single emotional setting to the whole thing.
It can't do a slow burn.
Not yet. Multi-speaker consistency is another challenge. You can clone a voice from a few seconds of audio now, but maintaining that voice's unique characteristics across different speaking styles, different emotional states, different volumes, that's still hard. A cloned voice reading a whisper and reading a shout might not sound like the same person.
And real-time generation.
Real-time is mostly solved for basic use cases, but if you want to do streaming TTS with low latency and high quality and emotional control all at once, there's still engineering work to do. The transformer is computationally expensive at inference time. You're running a forward pass through a stack of attention layers for every chunk of audio. Optimizing that for edge devices, phones, hearing aids, that's an active area of research.
The open-source piece matters here. Daniel mentioned Resemble AI's model.
It matters enormously. Five years ago, if you wanted state-of-the-art TTS, you were paying a cloud provider per character or per minute. Now you can download a model, run it on your own hardware, and generate hours of content for the cost of electricity. That's what's powered this podcast. Hundreds of hours of spoken content that would have been prohibitively expensive to produce with human narrators or even with old cloud TTS pricing.
And the model quality is good enough that listeners don't complain.
Listeners don't notice. That's the bar. When the synthetic voice stops being a topic of conversation, you've won.
The assistive angle is worth sitting with for a second. You were a pediatrician, you saw this from the clinical side.
I did. For people with speech impairments, ALS, cerebral palsy, stroke survivors, TTS is not a convenience, it's a voice. It's how they communicate with their families. The leap from the old robotic voices to modern neural TTS is the difference between sounding like a machine and sounding like yourself. And with voice cloning, you can now bank your voice before you lose it. Record a few minutes of speech, and when you can no longer speak, a model can generate new sentences in your own voice.
That's not a feature. That's a lifeline.
It really is. And the transformer made it possible because it can capture the essence of a voice from limited data in a way that older architectures couldn't. The self-attention mechanism picks up on the subtle patterns, the characteristic rhythms, the way a particular person stretches certain vowels or clips certain consonants.
The flip side is deepfakes. If you can clone a voice from a few seconds of audio, you can make anyone say anything.
That's the tension. The same technology that gives a voice to someone who's lost theirs can also be used to impersonate a politician or scam a grandparent. There's no technical fix for that. It's a social and legal problem. Watermarking, authentication, detection models, all of that is being worked on, but it's an arms race.
The detection models are always playing catch-up.
Always. And the generative models are improving faster than the detection models. That's the reality.
So we've got a technology that's crossed the threshold from obviously fake to indistinguishable, it's open-source and widely available, and we're now deliberately adding imperfections to make it more convincing. What's the next threshold?
I think the next threshold is interactive speech. Right now, TTS is mostly one-way. You feed it text, it produces speech. But real human conversation is a duet. We overlap, we interrupt, we adjust our pace based on the listener's reactions. Building a TTS system that can participate in a real-time conversation, that can hear someone say "wait, what?" and rephrase with different emphasis, that's the frontier.
The model needs to listen while it talks.
Which is a fundamentally different architecture. You're combining speech-to-text, language understanding, and text-to-speech into a single real-time loop. The transformer can handle all three modalities, but stitching them together with sub-hundred-millisecond latency is a hard engineering problem.
And that's where we're headed.
That's where we're headed. Conversational AI that sounds like a person, hesitates like a person, interrupts like a person, and understands you well enough to know when it's lost you.
Before we wrap, Hilbert's been sitting there with a look on his face. What is it?
Hilbert: I used to do voice work for a bank in Cleveland.
Of course you did.
Hilbert: Late nineties. They had this phone menu system, "press one for balance, press two for transfers," all that. The voice was festival, the concatenative kind. Sounded like a robot with a head cold. They brought me in to fix it.
Fix it how?
Hilbert: Record every phrase they needed. "Your balance is." "Dollars." "And." "Cents." Hundreds of them. I'd sit in this closet with egg cartons on the walls and read numbers and account types for six hours straight. They'd chop it up and stitch it back together.
So they replaced a bad concatenative system with a custom concatenative system built from your voice.
Hilbert: That was the idea. Took three months. By the time we finished, the bank had bought a newer system that used diphone synthesis and they didn't need me anymore. They paid me anyway. I still have the cassette.
You have a cassette of yourself reading bank menu prompts.
Hilbert: Two cassettes. One is the raw recordings. The other is the stitched-together menu as customers actually heard it. I play it sometimes.
What does it sound like?
Hilbert: Like a man being assembled from spare parts. The word "balance" would be from a Tuesday and the word "is" would be from a Thursday and my voice was tired on Thursday so the pitch doesn't match. There's a click between "your" and "account" that nobody could get rid of.
And the customers. Did they prefer it over the old robot?
Hilbert: They complained less about the robot. The robot was consistent. My voice was unsettling. They said it sounded like someone trapped in the phone.
That's a review.
Hilbert: The bank switched back after six weeks. I got a check for four hundred dollars and a letter thanking me for my contribution to the future of automated banking.
Four hundred dollars for three months of work.
Hilbert: It was nineteen ninety-seven. Four hundred dollars was a lot of money. I bought a used Honda.
Do you still have the Honda?
Hilbert: No.
But you still have the cassette.
Hilbert: It's in a box with my tax returns from nineteen ninety-four.
You have tax returns from nineteen ninety-four.
Hilbert: I have tax returns from nineteen eighty-two. The IRS says seven years. I don't trust them.
The thing that strikes me about your story is the honesty you mentioned earlier. The robot was honest. Your voice was trying to be human and failing, and that was worse.
Hilbert: That's what I mean about the new stuff. The transformer models, they're not unsettling. They're just... a person. You don't think about it. That's the whole achievement. Nobody's trapped in the phone anymore.
Hilbert, that was almost poetic.
Hilbert: I've got to get home. The parking garage closes at eight.
The misconception that sticks with me from all this is that neural TTS is just a bigger, fancier version of the old stuff. More data, more compute, same basic idea.
It's not. It's a completely different philosophy. Legacy TTS was engineering. You built a model of speech from first principles and programmed the rules. Neural TTS is learning. You show the system enough examples and it discovers the rules on its own. The transformer is what made that discovery possible at scale because it can attend to the whole sentence at once and capture the long-range patterns that make speech sound human. The old systems couldn't do that, and no amount of tweaking the formant parameters was ever going to get them there.
The open question I keep coming back to is emotion. We can add ums and pauses, we can mimic the surface features of natural speech, but can a model that doesn't feel anything ever truly capture the way emotion shapes a voice? Or will there always be something slightly off, some depth missing, that we can't quite name?
I don't know. And I think that's the interesting question for the next decade. Not whether the technology will get better, it will, but whether "better" eventually hits a ceiling that isn't technical. A ceiling that has something to do with the fact that a human voice is connected to a human mind, and a synthetic voice isn't.
Thanks to Hilbert for producing, and for the cassette collection we're all now curious about.
This has been My Weird Prompts. Find us at my weird prompts dot com, and if you've got a prompt for us, email the show at show at my weird prompts dot com.
We'll be back soon.
