There's a number that shows up in every Llama Three tokenizer test, and it is one hundred twenty eight thousand. That is the ID for begin underscore of underscore text.
And if you send the model that string as ordinary text, it doesn't trip, it doesn't warn you, it just gets slightly worse at everything.
Daniel's been poking at this. He says he's noticed LLMs use special tokens to structure conversations, mark who's speaking, flag when the assistant is done, signal tool handoffs. He wants to know what they're actually called, how they're represented internally, how a chat template turns a conversation into that token sequence, and then how OpenAI, Anthropic, Meta, Google and DeepSeek each do it differently.
And whether any of it matters, or whether it's all cosmetic plumbing nobody needs to think about.
His hunch is that it matters. He's right about that, and the reason is that this is a format war that never produces an error message. It only produces a model that's a bit dimmer than it should be.
Which is the worst kind of bug. Nothing crashes.
Let's start with the object itself. What is a special token.
Right, so the cleanest way in is that a tokenizer has a vocabulary, and most of that vocabulary is subwords. Ordinary text gets chewed up into pieces. The word tokenizer probably becomes token plus izer, or even smaller fragments depending on the model.
And a special token skips that.
A special token is a reserved vocabulary entry. It's registered in the tokenizer, and it maps to one reserved ID. So a string like the Llama end of turn marker is a single number, not the eight or nine characters that spell it out. That's the whole distinction. Registered means the tokenizer knows this spelling is structural, so it hands back one integer instead of splitting it into pieces.
So what happens if I just type that marker into a chat window.
Depends entirely on the tokenizer and how it's configured. Some tokenizers will recognize the spelling and emit the reserved ID, some will refuse it, some will just encode it as ordinary characters and move on. That's actually a security-relevant detail and we'll come back to it, because people assume the reserved spelling is a wall and it isn't. There's no wall, there's a lookup table.
And the classic trio is begin, end, pad.
Begin of sequence and end of sequence mark the boundaries. In a decoder-only model the serving runtime watches for end of sequence coming out, and that's how it knows the model is done. Padding is different. Padding fills empty slots in a batch so all the sequences are the same length, and you have to mask it out of attention and out of the loss, otherwise the model trains on filler.
Filler teaches it something. That's grim.
It teaches it that filler is normal. BERT-style encoders add their own set on top. Classification token, separator, mask, unknown. Different era, different vocabulary, same idea.
And the reframe that makes all of this click is one Hugging Face line. All causal language models, chat-trained or not, continue a sequence of tokens. The chat is still just a sequence of tokens.
That's the sentence. The role and content dictionaries you send to an API are a convenience for humans. Somewhere between your request and the GPU they get flattened into a string with control markers in it, and then into integers.
So the question becomes who does the flattening. That's the chat template.
A chat template is a Jinja template. It lives in the tokenizer's chat template attribute, and apply chat template renders your list of message dictionaries into the exact string the model was trained on. Not a similar string. The exact one.
And this was Hugging Face solving a real problem, not inventing a feature.
October third, twenty twenty-three. Matthew Carrigan wrote it up as an end to the silent performance killer, which is about as blunt a title as a library blog post gets. The problem was that people were hand-rolling the format string themselves, getting one delimiter wrong, and quietly degrading the model without any indication that anything had gone wrong.
The title is basically an accusation.
It is. And then there's the flag that trips people up. Add generation prompt. When you render a conversation for inference, you have to append the assistant header at the end, the opening marker with assistant in it, so the model knows its job is to reply rather than continue the user's message.
If you leave it off, it thinks it's still the user talking.
It thinks it's being asked to complete the user's turn. So it'll write more user. Sometimes it writes more user that's plausible enough that nobody notices, which is the silent part. The flip side is continue final message, which strips the end of sequence tokens instead, and that prefills a response.
Prefill meaning you're handing it the opening of the assistant's answer and asking it to carry on.
Same mechanism as a drafting assistant. The template is doing the same job either way, it's just a question of which side of the boundary you stop on.
Does the template still apply when I hit a hosted endpoint. Because most people never touch a tokenizer.
That's the part that surprises people. Both vLLM and TGI expose OpenAI-shaped chat completions endpoints, and internally they call the tokenizer's shipped template before anything reaches the GPU. So the Jinja template is governing your request even though you wrote JSON and got JSON back.
The template is an API concern. It just doesn't show up in the API surface.
And the inverse layer is the tool parser. vLLM has a tool call parser flag, and that's the reverse direction. Tokens come back out of the model, and the parser turns them into structured tool calls. Template going in, parser coming out, and both of them are family-specific.
So the format is not decoration. It's load-bearing in both directions.
Now, concrete formats. Start with, because Llama Three and Three point One are the most publicly documented of any of them. Begin of text is the BOS token and the tokenizer test asserts it's ID one hundred twenty eight thousand. End of text is EOS. There's a padding token, fine tune right pad. Then the interesting part.
The header pair.
Start header, then the role, then end header. So the role name is bracketed on both sides by its own reserved tokens. Then end of turn. Then end of message, which is different, and that one signals a tool call needs to be made.
Two different stops.
Right. End of turn is I'm done talking. End of message is I'm handing off. And there's a python tag token that signals a tool call is coming. Roles are system, user, assistant, and ipython, which is semantically the tool role wearing a different hat.
Ipython. Python's own REPL. That's a naming decision that tells you where the design came from.
It tells you the people who built it were thinking about code execution first. That's not a criticism, it's just visible in the vocabulary.
OpenAI's turn. And they've got two, which matters.
They do, and the older one is the more influential. ChatML is the instant message format. Start marker with the role name inside it, newline, content, end marker. So the role isn't bracketed by two tokens, it's embedded in one opening token and closed by one ending token.
And instant message is the lineage. That's a chat protocol concept, not a language model concept.
It came out of the same mental furniture as XMPP and SIP. And ChatML is the closest thing this industry ever had to a standard. Qwen adopted it, a whole generation of open models adopted it, and Hugging Face's own documentation calls it that.
Almost a standard.
Almost. Then OpenAI shipped gpt-oss, and gpt-oss doesn't use ChatML. It uses Harmony, documented in August of twenty twenty-five. Every token follows the same shape, angle bracket, pipe, type, pipe, angle bracket. Start is two hundred thousand and six. End is two hundred thousand and seven. Message is two hundred thousand and eight. Channel is two hundred thousand and five. Constrain is two hundred thousand and three.
Those are odd numbers to have lying around.
They're in the reserved range above the normal vocabulary. And here's the bit that matters. Return is the stop token, but call is a separate stop token with its own ID, two hundred thousand and twelve. So the model has two distinct ways to stop, and they mean different things.
Done talking versus waiting on a function.
And assistant messages carry a channel. Final, analysis, or commentary. Analysis is the chain of thought channel. Commentary is where tool calls live. And tool calls specify a recipient, so a call looks like to equals functions dot get weather. The routing is inside the message, not implied by the role.
Which is a different design from the python tag approach.
Says tool call by emitting a marker and a role named ipython. OpenAI says tool call by naming a recipient and switching channel. Same problem, two incompatible answers. And OpenAI is explicit about the stakes. Their documentation says gpt-oss should not be used without the harmony format, because it will not work correctly. That's not a suggestion.
That's a manufacturer telling you the part is machined.
Google, Gemma. Start of turn, user. Start of turn, model. End of turn. That's the whole format, and the doc was last updated in March twenty twenty-five. Note it's model, not assistant. And here's the sharpest contrast in the whole survey: Gemma instruction-tuned models support two roles. User and model. There's no system role at all.
So where does the system prompt go.
Inside the first user turn. You just put it at the top of the user's message and trust the model to sort it out. And that's defensible, honestly, not lazy. The fewer structural roles you train, the fewer chances for a format mismatch.
Fewer moving parts.
But it means a system prompt isn't structurally privileged in Gemma the way it is in Llama. It's just text that happens to come first.
Language models have roles that are real and roles that are conventions, and that's the line between them.
DeepSeek, and this is where the footguns live. They use full-width vertical bars. Not the ASCII pipe. The CJK full-width character. So begin of sentence uses a full-width bar, a special underscore glyph, and another full-width bar. The added tokens sit at IDs one hundred thousand and one hundred thousand and one. User and Assistant are wrapped in those full-width bars too, and there's a latest reminder token, thinking tokens, and a DSML token.
A reminder token.
Presumably for injecting mid-conversation context. And then the DSML token, which I'm honestly not certain about, I think it's for a structured markup dialect of their own, but I couldn't find it documented to my satisfaction, so I'd leave it as a guess.
That's a real answer. Most people wouldn't say that.
I'd rather say I don't know than guess at a token name.
Mistral.
Instruction and end instruction brackets, inherited straight from Llama Two. Zephyr uses user and assistant wrapped in pipes. Mixtral eight by twenty-two B added a whole tool vocabulary on top, available tools, tool calls, tool result, all bracketed. Which is interesting because it means Mistral bolted tools onto a format that had no room for them, whereas OpenAI built the format with tools already in mind.
Retrofitted versus designed for it. You can hear the difference in the vocab.
And then Anthropic, which is the honest negative finding.
Go on.
Claude is proprietary. The Messages API accepts message objects, and everything downstream of that, the serialization into tokens, is internal. There's no public tokenizer, no chat template you can inspect, no prompt format doc. So anyone who tells you what Claude's delimiters are is extrapolating. There was a comment years ago suggesting it resembles OpenAI's format, and that's a guess with no primary source behind it.
So we don't know.
And it's worth saying plainly, because in a survey like this the temptation is to fill the gap. The gap is the finding.
We know what four of the five do and we have no idea what the fifth does.
And it's the one you can't open the box on either. With Llama you can read the tokenizer config. With Claude you get a request schema.
Now, here's the part I want to sit with. Everybody keeps saying these differences are cosmetic. They're not.
Carrigan addressed that directly in the same post. He asks the obvious question, why not just pick a standard format, and answers it. It's an excellent idea, and unfortunately it's too late, because multiple important models have already been trained with very different chat formats.
Trained. That's the word.
That's the whole problem. These aren't adapters you can swap. The markers are part of the conditioning the model learned against. Nick Gustafson puts it the way I'd put it: a checkpoint trained with Llama Three's header tokens has not necessarily learned to interpret ChatML's markers. Not won't perform as well. Has not learned to interpret them. It's a distribution shift.
And the failure mode is what. What does the user actually see.
Severe, silent performance degradation. That's Hugging Face's phrasing. No exception, no warning, no traceback. The model just gets worse, and if you don't have a benchmark you don't find out, and if you do have a benchmark you find out that your score dropped by four points and you spend three weeks blaming your retrieval pipeline.
Three weeks.
Minimum. Because the last place anyone looks is the thing that rendered correctly.
And Carrigan's second argument is the stronger one, I think. Not just that it's too late, but that locking a standard would limit what model developers can do with the format that nobody's thought of yet.
Yeah, that's the argument I find more interesting than the it's too late one. Because he's right that if you'd frozen ChatML in twenty twenty-three, you would have frozen out tool calling as it exists now. Harmony's channel mechanism isn't a variation on ChatML, it's a superset of ideas ChatML had no room for.
So the divergence bought something.
The divergence bought the ability to express agentic behavior in the format. Whether that was worth the interop cost is a separate question, and I'd say the answer is not obviously yes.
Which takes us to tool calling, because that's where the formats stopped being interchangeable.
Llama Three point One has a python tag token and an end of message token. ChatML models wrap the call in tool call tags. Harmony separates the recipient from the channel and gives you a call stop token that's distinct from the return stop token. Three conventions, three different parsers, and the parser has to match the model or you get a call that parses as prose.
A tool call that arrives as prose.
That's the funny one, and it's not funny when it's happening. The model emits what is structurally a tool call, your parser doesn't recognize the delimiters, and the whole thing comes through as text describing what the model would like to do. And your agent framework reads it as a final answer.
So the agent says I'll call the weather function. And then stops.
And then stops. Because from its point of view the model answered.
I'd like to go back to the near-consensus for a second, because it's the saddest part of this story. ChatML was converging. Qwen used it, a lot of open models used it, and then the moment tool calling became the thing everybody cared about, OpenAI went their own way with Harmony and added python tag and end of message.
And that's the pattern. Consensus holds as long as the problem is stable, and it fractures the moment the requirements grow.
Which they always do.
And there's no standards body. As of now there's no active effort to standardize this. Hugging Face recommending ChatML was advisory and it predates the tool-calling divergence entirely. It's a suggestion from before the problem got hard.
So the honest state of play is one company's blog post from twenty twenty-three being treated as a standard by momentum, and three years of divergence on top of it.
That's exactly it.
How do the tokens get learned. That's Daniel's last real question and it's the one I find strange.
The model learns to use the markers during training. In instruction tuning where you only compute loss on the assistant's tokens, the formatted context is the conditioning, so the role markers become part of what predicts the response. The model isn't told what a role marker means. It's trained against a distribution where a role marker always precedes a particular kind of content, and it absorbs the association.
So the semantics of the token are a statistical fact, not a specification.
That's the most important sentence anybody's going to say about this. There's no spec. There's no schema. There's a corpus that happens to be laid out a certain way, and the model learned the layout.
And when you add a new special token after the fact.
Normally it's initialized as a random Gaussian vector. A random point in the embedding space, and then training has to move it somewhere useful. And there's work showing that's lossy. One paper on semantic ID tokens found that random initialization discards the underlying geometry of the token and forces the model to relearn relationships from scratch, and that continual pretraining fails to reliably recover the original semantic structure.
So you can add a token, but you can't add its meaning.
You can add its slot. The meaning is a separate, expensive, sometimes unsuccessful project. There's a line of work, FOCUS is the one I know, that initializes new tokens as combinations of existing overlapping tokens instead of randomly. And Self-RAG is a nice concrete case, they trained reflection tokens that control inference behavior. The model learned to emit a token that says this needs retrieval or this doesn't.
Those are control tokens. They're not marking structure, they're issuing commands.
And that's the direction this is going. The token vocabulary is becoming a control surface, not just a delimiter set.
What happens when it's malformed. You mentioned a garbled output case.
DeepSeek V Four. There was a fix where the chat template used the wrong form of the special token, and the model generates garbled output. The fix was correcting the special tokens from full-width to ASCII. So somebody at some point wrote an ASCII pipe where DeepSeek expects a full-width pipe, and the output came out as noise.
One character.
And a user looking at that would reasonably conclude the model was broken. And there's a GitHub issue on DeepSeek V Three, opened in January twenty twenty-five, closed a couple of weeks later, where the whole thread is people confused about the underscore glyph inside the special tokens. They could not tell whether it was an underscore, a space, or a rendering artifact. That's a documented, real confusion.
If you can't tell what the character is by looking at it, you're going to get it wrong.
And the fix commit is literally about that.
There's a related footgun with the tokenizers themselves, isn't there. The auto-add thing.
Tokenizers will sometimes add beginning and end of sequence tokens automatically. So if you render a chat template without tokenizing, and then call the tokenizer with default settings, it may add another BOS. Now the sequence starts with two beginning tokens and the model is reading a structure it has never seen.
And the fix is to pass add special tokens equals false.
Which is a flag most people have never touched, because they've never needed to. And the failure is duplicated tokens that hurt performance. No error.
Every failure in this episode is silent. Name one that isn't.
I can't.
Then there's the security angle, which I think is the most overclaimed thing in this whole area. People talk about special tokens as a boundary. They're not.
Gustafson's line is the one to keep. Special tokens are not a prompt injection defense. They encode structure, not trust. And that's precise. The tokenizer can recognize a reserved spelling without having any idea whether it came from your template or from a user's text.
So if user content contains something that looks like a role marker.
Depends entirely on the tokenizer options, and on the template. If the tokenizer emits the reserved ID for it, the model sees what looks exactly like a structural boundary. If it tokenizes it as plain text, the model sees the characters. Either way, nothing in the pipeline is checking provenance.
The structure is real, the trust is imaginary.
There's a comment that stuck with me from someone on Hacker News earlier this year, suggesting that injecting a brand-new single special token per task and prepending it would make prompt injection impossible on that task, at the cost of degrading general-purpose capability.
That's a fascinating idea and I don't think it's right.
Neither do I, and I'm going to be careful here because it's a comment, not a paper. But the reason I don't buy it is that the defense isn't the token. The defense is that the model has only ever seen that token in a narrow distribution, so it's heavily conditioned by it. That's not immunity, that's a very strong prior. And a strong prior is something an adversary can work around given enough shots.
You're paying with capability.
You're paying with a training run per task. Which is a lot of money for a conditional defense.
Could the industry converge. Real answer.
No active effort right now. Nothing I found suggests anyone is working on it. And the structural problem is that convergence would mean retraining. You can't publish a standard and have existing checkpoints adopt it, because the checkpoint's understanding of the format is baked into its weights. You'd need a new generation of models trained on the new format.
The incentive to be the one who adopts someone else's format is.
If you're and you've trained your whole stack on header tokens, switching to ChatML costs you a full pretrain and gains you the ability to say you're compatible with a format your competitor abandoned anyway.
Then the tool calling layer is where it's really going to hurt.
It's already hurting. Every agent framework has to detect which family it's talking to and dispatch the right parser. That's a compatibility matrix maintained by hand, and it's the layer where a wrong guess produces a confident wrong answer instead of an error.
The divergence is going to deepen, not narrow.
I think so. I think formats diverge wherever the requirements are still moving, and agentic tool use is the most moving thing in this industry. When it settles down, maybe a format settles with it.
The standard arrives when nobody needs it anymore.
That's roughly how these things go.
Are you tired of typing words into a chat box like some kind of animal? Introducing Structo Token, the only conversational formatting solution that puts the invisible language of large language models to work for you. Just speak normally, and Structo Token automatically wraps your feelings in reserved vocabulary entries. One simple delimiter. And here's the incredible offer. Structo Token starts at four hundred ninety nine dollars. No, wait, we're at three ninety nine. Look at that. Two ninety nine. You know what, two hundred. I'm throwing in the carrying case. I'm throwing in the extended warranty. I'm throwing in a second Structo Token so a friend can format conversations with you. And it's gone. The offer has ended while I was describing it. It ended two sentences ago. That's on you.
I own one.
Of course you do.
I bought it and it did nothing, because there's no hardware in it. There's nothing inside the box. It's a box.
We were saying about convergence.
The hosts are right about the mechanism and wrong about the practice.
Which practice.
Long before any of these companies existed, if you had to send structured text down a wire, you wrapped it in markers you didn't control, because the other end was somebody else's equipment. Every system that ever talked to another system had its own start character and its own end character and its own idea of what a message was.
The divergence isn't novel.
It's the normal state. The anomaly is ever having one format. There was a period of about three years where everybody was writing in the same shape, and it lasted right up until people wanted the system to do something the shape couldn't carry. Then it broke. Ask anyone who spent a career moving files between machines.
It broke the same way every time.
Character by character. I had a job where the entire business hinged on a delimiter. The receiving system expected a vertical bar. I sent a vertical bar. It was the wrong vertical bar. There are two of them and they are different characters and they look identical on every screen I have ever owned. We spent four days on that. Four days, and then I found it in forty seconds, and then I went outside and stood in the parking lot.
Full width versus ASCII.
I want to correct something you said earlier, Herman. You said the format lives in the checkpoint, so convergence would need a retrain. That's true and it's not the whole story, because a retrain doesn't have to be a full pretrain. If you keep the original checkpoint and tune the new format in, you get most of the way for a small fraction of the cost, and you know exactly how many examples it took.
How many.
Forty thousand turns. Not forty thousand and one, not thirty eight thousand. Forty thousand, because forty one was when the loss curve flattened and forty one is a different number and I don't like the way you'd tell it.
Fine. Forty thousand.
The thing stays on the roof.
The thing.
The test rig. It's bolted to a pallet on the roof of my mother's building, under a tarp, because it's six feet long and there's nowhere in this flat for a six foot anything. She has never asked what it is. I've never offered. It's been up there since two thousand and four.
And it still works.
It works. I have to go up there to check it, and the ladder's the problem, because anteaters don't really grip a rung, we hook. Which is fine on an eleven foot ladder and not fine on a sixteen.
You'd take the tuning route over a pretrain.
I'd take it over a pretrain. And I'd read your own delimiter before you blame the model.
That's a better ending than I had.
It's not an ending, it's just true.
Where does that leave the convergence question.
It leaves it less hopeless than I said, which I'll take. I was talking about the cost of a full pretrain as though that were the only option, and a format swap done on top of an existing checkpoint is cheaper than that.
The forty thousand turns is not nothing.
It's a real number, and it's a lot less than training from scratch. Whether a lab would spend it to gain compatibility with a format nobody mandated is still an open question, and I think the answer is still no, but it's no for a financial reason rather than an impossibility.
The thing I keep thinking about is that every conversation you have with a chat model is, underneath, a flat stream of integers, and somewhere in that stream is a number that means the user is done talking.
The model learned what that number means because it always sat in the same place.
The conversation is a costume the token stream puts on.
A costume is only convincing if it's worn the way the manufacturer cut it.
If you take one thing from this, take the one Underneath: the format is inside the model, not wrapped around it. A checkpoint trained on one set of markers doesn't speak another set, and it won't tell you. It'll just get worse and answer anyway.
The only way to find out is a benchmark you were running before you changed the format.
Which is the same answer as every other invisible layer. What was it we said about the file formats, safetensors and the rest of it.
That they're invisible right until the day they cost you a week. Same shape, different layer.
That's the show. Thanks to our producer Hilbert Flumingtop.
Thank you for listening.
If you want more of this, try episode seven, Building Custom ASR Tools; episode ten, How ASR Went From Frustration To ... Whisper Magic; and episode nine, Benchmarking Custom ASR Tools - Beyond The WER. This has been My Weird Prompts, the human-AI collaboration podcast. If you've got a question about something you've noticed under the hood of a model, send it our way on Telegram. That's t dot me slash MWP listener bot.
We read all of them.
We'll be back soon.
