Daniel was explaining the transformer pipeline to someone the other day — tokenizer, vector space, temperature, top-K, top-P, the whole inference-as-next-token-prediction thing — and he hit a wall. He realized he couldn't answer the simplest question: if the model's entire job is predicting the next token forever, how does it ever decide to stop? Why doesn't it just keep going? He's asking whether models have some innate concept of gravity or completeness, and whether the stop mechanism actually lives in the model itself or gets handled on the front end.
This is one of those questions where the moment someone asks it, you realize you've never actually thought about it. And the answer is layered. There is no single stop mechanism. There are at least three different systems stacked on top of each other, and the one most people never hear about is the most important.
Three layers. All right, walk me through them.
Layer one is the EOS token. End-of-sequence. Every modern language model has a special token in its vocabulary — OpenAI uses something like the end-of-text marker in angle brackets, Meta's LLaMA uses the closing slash-S tag, Mistral has its own convention. But they all do the same thing. During training, every single example the model sees — every conversation, every document, every code snippet — ends with this token. The model learns that after the last word of a complete unit, the next most probable thing is EOS.
So it's not that the model knows it's done. It's that it has learned to predict a token that means nothing comes next.
And that's crucial. The model has no concept of completeness. It has a learned statistical association: in training data, when an answer looked like this, the next token was silence. At inference time, when the probability distribution says EOS is the most likely next token, the model emits it and the inference engine says we're done.
But that's probabilistic. The model can assign EOS a low probability and just... keep talking.
That's the whole rambling problem. If EOS has, say, a three percent probability and the next most likely token has four percent, and you're sampling rather than greedy decoding, the model might skip EOS entirely and start a new sentence. Then the new context makes EOS even less likely, and now you're off to the races.
So what you're describing is a system where the model's only intrinsic stop signal is a token it may or may not feel like emitting. That's not a brake. That's a suggestion.
It's worse than a suggestion — it's a token that competes with every other token in the vocabulary on equal footing. The word "however" and the EOS token are just two entries in the same probability distribution. If "however" looks more statistically appealing given the context, the model says "however."
Which explains why base models, the ones that haven't been fine-tuned for chat, will just generate forever until they hit the context limit.
Right. A base model trained on raw internet text sees documents of wildly varying lengths. It learns that EOS comes after... well, after whatever the document happened to end with. There's no consistent pattern. So at inference, it might generate for thousands of tokens before EOS looks probable. A chat-fine-tuned model, by contrast, has seen millions of examples where the assistant says something concise and then EOS immediately follows. The association is strong.
That's the instruction tuning doing the work. RLHF, DPO, all of that.
During RLHF, human raters consistently preferred answers that were complete but not padded. The model learned that emitting EOS after a sufficient answer gets rewarded. That's not an innate sense of when to stop — it's learned conversational norms, baked into the weights through preference optimization.
So Daniel's observation that models seem to have some gravity, some sense of when enough is enough — that's entirely a product of fine-tuning on data where humans demonstrated what enough looks like.
And it's fragile. Change the sampling parameters and that gravity disappears. Crank up the temperature and the probability distribution flattens out — EOS, which might have been at ninety percent, drops to forty. The model suddenly can't decide whether to stop or keep going, and you get those rambling outputs where it seems to be arguing with itself.
Temperature flattens everything. Low temperature sharpens the distribution — the highest-probability token gets even more probable. So EOS, if it was already likely, becomes nearly certain.
That's why low-temperature models feel decisive and terse. They stop early and often. High-temperature models feel creative and unhinged. The same model, same weights, same prompt — the only difference is whether you've made it easy or hard for EOS to win the probability lottery.
And top-K and top-P can just remove EOS from the candidate set entirely.
If EOS has a probability of two percent and you set top-P to ninety-five percent, EOS might not make the cut. The model literally cannot stop because the stop token has been pruned from consideration. It will generate until it hits the context window limit or you kill the request.
Which brings us to layer two. The inference-engine constraints.
Stop sequences. Every production inference system — the OpenAI API, vLLM, Hugging Face's text generation pipelines — lets you define strings that, if they appear in the output, immediately halt generation. The classic example is double-newline. Or the string "User:" so the model doesn't start roleplaying as the human. These are not part of the model at all. The inference engine is watching the output stream and pattern-matching. The moment it sees the stop string, it truncates and returns.
So the model might have been about to emit EOS on its own, or it might have been about to write another three paragraphs. Doesn't matter. The engine yanks the plug.
And this is why when you're using the OpenAI API and you set a stop parameter, you'll sometimes get responses that feel cut off mid-thought. The model was perfectly happy to continue. The engine wasn't.
Layer three is the brute-force stop. Context length.
Every model has a maximum context window. Older GPT models had four thousand ninety-six tokens. Modern ones go up to a hundred twenty-eight thousand or more. If generation reaches that limit, the inference engine either truncates the output or throws an error. This has nothing to do with the model's intent. It's a hard wall.
None of these three layers knows about the others. The model doesn't know the engine is watching for stop strings. The engine doesn't know whether the model was about to emit EOS. And the context limit is just a resource constraint.
The whole thing is held together with tape. Beautiful, effective tape, but tape.
Let me push on something. You said the EOS token is just another token in the vocabulary. But during training, the model sees EOS exactly once per example — at the very end. Doesn't that create a kind of structural asymmetry? Every other token can appear anywhere. EOS only appears in one position.
That's a really sharp observation. It does create asymmetry, and it's part of why EOS behavior can be so weird. The model learns that EOS is special not because someone told it, but because the positional distribution is completely different from every other token. EOS is the only token that, when it appears, means the sequence is over. The model has to learn that from the data structure alone.
Which means if your training data is messy — if documents sometimes have EOS in weird places, or if concatenated examples put EOS in the middle of what the model sees as a continuous sequence — the model learns corrupted stop behavior.
This was a known issue in early GPT fine-tuning. If you concatenate training examples without proper separation, the model sees EOS tokens in the middle of what it perceives as one long document. It learns that EOS doesn't really mean stop. And then at inference, it ignores EOS entirely.
So the training data hygiene around EOS placement is actually one of the most important and least-discussed parts of model training.
And it connects to something Daniel mentioned in his prompt — that he was explaining how tokens are shifting beyond text. Pixels, phonemes, code. In multi-modal models, the stop problem gets even messier.
How so?
Imagine a model that generates both text and images. It outputs some text, then an image token sequence, then more text. Where does EOS go? After the text? After the image? Does the image sequence have its own internal EOS? Different model families handle this differently, but the fundamental problem is that EOS was designed for a world where output is a single linear sequence of text tokens. Multi-modal output is not that.
And if the model is calling tools — function calling, web browsing — the stop mechanism becomes a control-flow problem.
This is where structured generation comes in. Systems like ChatGPT don't just let the model free-generate and hope EOS lands in the right place. They use grammar constraints, JSON mode, or what's sometimes called guided generation. The inference engine forces the model's output to conform to a specific schema. When the schema says the function call is complete, generation stops — regardless of what the model's probability distribution says.
So in agentic systems, the probabilistic EOS is increasingly overridden by deterministic constraints.
And that's probably the right call. If your model is controlling a robot arm or executing financial transactions, you cannot have the stop signal be a token that might or might not win a probability lottery.
The early ChatGPT bug where it would generate "As an AI language model, I..." in an infinite loop — that was an EOS failure, right?
Classic case. The training data contained many long-form disclaimers that began with that phrase and went on for paragraphs. The model learned that after "As an AI language model," the next token was rarely EOS — it was usually more disclaimer. So once it started down that path, the probability of EOS stayed low, and it just kept generating variations on the same disclaimer until it hit the context limit.
It learned a pattern where stopping wasn't the pattern.
The training data taught it that this particular phrase is followed by more text, not by silence. The model was being perfectly faithful to its training distribution. It was the distribution that was the problem.
I want to go back to something you said earlier about the EOS token being the only token that means nothing comes next. In C programming, strings are null-terminated — a zero byte marks the end. Is the EOS token essentially a learned null terminator?
That is... actually a really good analogy. In C, the null byte doesn't mean anything semantically — it's a convention that the string-processing code knows to interpret as stop. The model learns a similar convention: in this context, the convention is to emit this special token that the inference code interprets as stop. The difference is that in C, the null byte is inserted by the programmer explicitly. In a language model, the model has to learn when to insert it from examples.
And we all know how many security vulnerabilities came from null-terminated strings. Buffer overflows, off-by-one errors, null byte injection.
The parallel is uncomfortable. If the model learns to put EOS in the wrong place — too early, too late, or not at all — you get the equivalent of a buffer overflow. Too early and the response is truncated. Too late and you're generating garbage. Not at all and you've got an infinite loop.
So the stop mechanism isn't just a technical curiosity. It's a failure surface.
And as models become more autonomous, the failure surface gets larger. Think about chain-of-thought reasoning. The model is supposed to generate internal reasoning steps and then output a final answer. There's research now on what's called budget forcing — explicitly limiting how many reasoning tokens the model can generate before it must produce an answer. Without that, the model might reason forever.
Reasoning forever sounds like a philosophy problem, not an engineering one.
It's both. The model has no internal sense of having thought enough. It has learned that in training data, reasoning sequences were of a certain length and then ended with an answer. But if you change the prompt or the sampling parameters, that learned length can stretch. The model doesn't know it's going in circles. It's just predicting tokens.
Which brings us back to Daniel's original framing. He asked whether models have an innate concept of gravity or when to stop. The answer is no — they have a token that statistically correlates with the next thing in training data being nothing.
And that's a very different thing. Gravity implies some internal sense of completeness, some goal-directed behavior where the model finishes what it set out to do. The model has no goals. It has no sense of what it set out to do. It has a probability distribution over a vocabulary, and one entry in that vocabulary happens to mean stop.
The illusion of completeness is powerful though. You ask a model a question, it gives you a complete answer, and it stops. It feels intentional.
Because we're pattern-matching machines. We see a coherent answer that ends at a natural boundary and we project intentionality onto it. But what actually happened is that the model's training data contained thousands of examples where coherent answers ended at natural boundaries with an EOS token. The model learned to replicate that pattern. The intentionality is ours, not the model's.
There's something almost disappointing about that. The magic trick explained.
I find it more interesting, honestly. The fact that a purely statistical system can produce outputs that feel intentional and complete — that's remarkable. The fact that the stop mechanism is a learned convention rather than an engineered rule is what makes these models flexible. The same architecture can learn to stop after a single word or after ten thousand words, depending on what the training data demonstrates.
But it also means the stop behavior is entirely at the mercy of the training distribution. If the distribution shifts, the behavior shifts. There's no stable ground.
That's the fragility. And it's why production systems layer on so many external constraints. Stop sequences, grammar enforcement, context limits, budget forcing. The model's internal EOS mechanism is the first line of defense, but nobody trusts it to be the only line.
What about the front-end versus back-end question Daniel raised? Is the stop signal transmitted and interpreted on the front end?
It's both, and which one matters depends on the failure mode. The EOS token is generated by the model — that's back-end, intrinsic to the inference process. Stop sequences and context limits are enforced by the inference engine — that's front-end, or at least middleware. In a typical API call, the model streams tokens to the engine, the engine checks each token against stop conditions, and if any trigger, the engine closes the stream. The model might never know it was stopped.
The model might never know. That's a strange sentence.
The model doesn't know anything. It's a feed-forward network producing one token at a time. It has no memory of what it generated three tokens ago except what's in the context window. If the engine truncates the output at a stop sequence, the model's internal state doesn't register that as an event. It just... stops being asked for more tokens.
So when I'm chatting with a model and it stops mid-sentence because it hit my stop sequence, from the model's perspective nothing unusual happened. It was ready to keep going. I just stopped asking.
The model is a token-producing function. You call it, it returns a token. You call it again, it returns another token. If you stop calling it, it doesn't notice. It has no ongoing process to interrupt.
That's almost existentially bleak.
Welcome to AI alignment. The systems we're building are profoundly alien in their internals while being increasingly convincing in their outputs. The stop mechanism is a perfect microcosm of that — it looks like intentionality, it's actually statistics, and we paper over the gaps with engineering.
Let me ask you a practical question. If someone is building an application on top of a language model and they want reliable stop behavior, what should they actually do?
Don't rely on the model's internal EOS. Set explicit stop sequences. Use structured output when you can — JSON mode, function calling, grammar constraints. Set a reasonable max tokens limit as a safety net. And if you're doing anything where an infinite loop would be catastrophic, add a timeout at the application layer.
Belt and suspenders.
Belt, suspenders, and a second pair of pants. The EOS token is probabilistic. Stop sequences are pattern-matching. Context limits are a hard wall. None of them is sufficient alone. Together they mostly work.
Mostly.
There's a reason every major model provider has an incident response team. When the stop mechanism fails at scale, it fails in ways that are hard to predict and harder to debug. A model that won't stop talking is one thing. A model that stops in the middle of a function call and leaves a transaction half-complete is another.
That's the agentic future Daniel was gesturing toward. Tokens that aren't just text but actions.
And every action needs a completion signal. Did the tool call finish? Did the web search return? Did the image finish rendering? Each modality needs its own stop convention, and they all need to compose. We're very much in the early days of figuring out how to do that reliably.
Hilbert: The Hayes Smartmodem.
...Go on.
Hilbert: Nineteen eighty-one. The Hayes Smartmodem used a command set where every command string was terminated by a carriage return. The modem would buffer characters, and when it saw the carriage return, it executed the command. If the carriage return got dropped due to line noise, the modem would sit there forever, buffering, waiting for a termination signal that was never coming. We called it the silent modem bug.
So the carriage return was the EOS token.
Hilbert: Functionally identical. A special byte sequence that means the message is complete. And when it failed, the modem didn't know it had failed. It just waited. Users would stare at a blinking cursor and think the connection was down. The modem was working perfectly — it was doing exactly what it was designed to do. Wait for the stop signal.
The failure was silent.
Hilbert: All the worst failures are silent. I spent eighteen months at Hayes debugging a problem where certain terminal programs would send a line feed before the carriage return, and the modem would interpret the line feed as part of the command and reject the whole thing. The stop signal was in the wrong place. Same problem you're describing with the EOS token — the model learns to put the stop signal in the wrong place, and the system fails in a way that looks like it's working.
What did the fix end up being?
Hilbert: We added a timeout. If the modem hadn't seen a carriage return in five seconds, it flushed the buffer and reset. Crude, but it worked. Every robust system eventually adds a timeout. Your context length limit is the same idea — if the model hasn't stopped in a hundred twenty-eight thousand tokens, force-stop it and move on.
The solution space hasn't really changed in forty years. Timeouts, delimiters, and hard limits.
Hilbert: The technology changes. The problems don't. You've got a smarter null byte now, but it's still a null byte. And it still fails in the same ways — too early, too late, or not at all.
The null byte analogy keeps getting stronger. In C, if you forget to null-terminate a string, the string-processing function reads past the end of the buffer into whatever happens to be in memory. That's exactly what happens when a model fails to emit EOS — it generates past the natural endpoint into whatever statistically follows, which is often nonsense.
Hilbert: Buffer over-read. The model reads past the end of the answer into the statistical weeds.
Null byte injection — if an attacker can insert a null byte into a string, they can truncate it early. Is there an equivalent attack on EOS?
Prompt injection could potentially trick a model into emitting EOS early. If you can make the model believe the conversation is over, it might stop before completing the intended task. I don't know of specific exploits in the wild, but the attack surface is there.
Hilbert: The Hayes bug was exploitable too. You could send a carriage return in the middle of a dial string and the modem would dial a truncated number. Usually a wrong number. Sometimes a number you didn't want it to dial.
The EOS token is a null byte, the stop sequence is a carriage return, and the context limit is a timeout. We've reinvented modem protocols inside neural networks.
That's... not wrong. The abstractions are higher-level, the systems are more complex, but the fundamental control-flow problem is identical. How do you signal the end of a variable-length message in a system that only understands sequences?
The answer is always some combination of in-band signaling, out-of-band constraints, and brute-force timeouts.
Hilbert: The other thing we learned at Hayes is that the stop signal has to be unambiguous. You can't use a byte sequence that might appear in the data. If your stop sequence is something the model might naturally generate as part of a normal response, you get false positives. The model stops mid-sentence because it happened to emit your stop string.
That's why EOS is a special token that never appears in natural text. It's an unambiguous signal. The problem is that the model has to learn when to use it, and that learning is imperfect.
Hilbert: Everything's imperfect. The question is whether it's imperfect in a way you can recover from.
Right now, for language models, the answer is mostly yes. The layered approach — EOS plus stop sequences plus context limits — catches most failures. But the agentic future Daniel's pointing toward raises the stakes.
When a model is controlling something — a robot, a financial system, a medical device — a stop failure isn't just an annoying rambling response. It's a physical or financial consequence. The stop mechanism stops being a convenience feature and becomes a safety-critical component.
Which means we probably need formal verification of stop behavior. Guarantees, not probabilities.
There's active research on this. Constrained decoding with formal grammars, where the stop condition is mathematically guaranteed rather than statistically likely. The trade-off is flexibility — the more you constrain the output, the less creative the model can be. But for safety-critical applications, that's a trade-off worth making.
Daniel's question was deceptively simple. How does the model know when to stop? The answer turns out to be a window into everything that's fragile and fascinating about these systems.
It doesn't know. It guesses. And we've built a scaffolding of engineering around that guess to make it feel like knowledge. The illusion holds up remarkably well — until it doesn't.
The cutting-room floor detail I can't stop thinking about: during training, if you concatenate documents without proper EOS separation, the model learns to ignore EOS entirely. A single data preprocessing decision can break the stop mechanism across the entire model. That's how fragile this is.
It's a reminder that these models are shaped by their data in ways that are often invisible until something goes wrong. The stop behavior isn't designed — it's grown. And like anything grown, it has quirks that the gardener never intended.
The model doesn't know when to stop. It knows when it's statistically likely that the next token is nothing. And as we ask these systems to do more — reason longer, act in the world, control physical processes — the gap between those two things is going to matter more and more.
We'll be back soon. Thanks to our producer Hilbert Flumingtop for keeping this show running, and for the modem history lesson.
This has been My Weird Prompts. You can find every episode at my weird prompts dot com, or email the show at show at my weird prompts dot com.
See you tomorrow.
