Daniel's been thinking about tokens again — but not the way we usually talk about them. His question is: what does a token actually mean when we're not generating text anymore? When a model spits out an image, a snippet of audio, a block of code — those are all tokens in the context window, but they're not pieces of words. If every modality has its own tokenizer, does the whole idea of a standard token estimate just fall apart? And how does tokenization even work when you're dealing with pixels or sound waves instead of sentences?
This is the question I've been waiting for someone to ask. The token has become this... almost invisible abstraction that everybody uses and nobody examines. We say context window, we say tokens per second, we compare pricing — and we're talking about completely different physical things. A token of audio is not a token of text. They share a format. They do not share a meaning.
So what even is a token when it's not a word fragment?
Let's start with what it used to be, because the rupture is clearer if you see the before picture. In the GPT-3 era, a token was a chunk of text — a subword, a whole word, sometimes just a character. The rough heuristic was one token equals about three-quarters of a word. That held up pretty well for English. You had a context window of four thousand tokens, you could fit about three thousand words. Simple.
And that heuristic was never perfect — we talked about how Japanese gets clobbered, some languages take way more tokens for the same content. But it was at least a rule of thumb you could use.
Right. And then multimodal models arrived and the rule of thumb just... shattered. Because the token stopped being a linguistic unit and became something more abstract — a discrete ID in a finite vocabulary that the transformer's attention mechanism can process. That's the real definition now. A token is whatever you can map into an integer that the model knows how to attend to.
That's a pretty big expansion. From a piece of a word to... anything you can encode as a number.
And that expansion happened fast. You've got Gemini, GPT-4V, DALL-E, Stable Audio — none of them tokenize only text. They tokenize images into patches, audio into spectrogram frames, video into spatiotemporal chunks. The same architectural primitive, the token, now spans completely heterogeneous data types.
So the token is the lingua franca of the architecture — not of human language.
Exactly that. It's the common currency the transformer knows how to spend. But the thing about currencies is they're only useful if you know the exchange rate. And right now we've got about six different currencies all called the same thing.
Alright, walk me through the image case. How do you turn a picture into tokens?
So the workhorse here is a family of models called VQ-VAE — Vector Quantized Variational Autoencoders — and their successors like VQ-GAN. The basic idea is you take an image and you compress it into a grid of discrete codes. Each code represents a small patch of pixels — typically sixteen by sixteen pixels per patch.
So you're chopping the image into a grid of little squares, and each square gets assigned a number.
That's the gist. The encoder looks at a patch and says, this patch is most similar to code number seven thousand four hundred and twelve in my learned codebook. And that integer, seven four one two, is a visual token. The decoder can take that integer and reconstruct something close to the original patch. It's lossy compression with a learned dictionary.
How big is the codebook?
It varies. Early VQ-VAE work used codebooks of around eight thousand entries. Modern systems can have tens of thousands. The Gemini API documentation from earlier this year — January twenty twenty-six — spells this out pretty clearly. Images are split into sixteen-by-sixteen pixel patches, each patch gets encoded as a token, and a two hundred fifty-six by two hundred fifty-six pixel image costs about two hundred fifty-eight tokens.
Two hundred fifty-eight tokens for a tiny thumbnail.
And that's just the base cost. Higher resolution images get more patches, more tokens. A five hundred twelve by five hundred twelve image would be four times the patches, roughly a thousand tokens. And that's before you add any text.
So the one-token-equals-three-quarters-of-a-word heuristic is already dead. Two hundred fifty-eight tokens for an image tells you nothing about how many words that image is worth.
It tells you nothing because the relationship is completely different. An image token encodes spatial information — color, texture, edges — in a compressed latent space. It's not semantic in the way a word is semantic. A subword token like "ing" carries grammatical information. A visual token carries... the average color gradient of a sixteen-pixel square in a learned compression scheme.
It really is. And it gets weirder with audio. Audio tokenization uses neural audio codecs — systems like EnCodec from Meta or SoundStream from Google. These take raw waveforms and convert them into discrete tokens at something like fifty to a hundred tokens per second of audio.
Per second. So a minute of audio is three thousand to six thousand tokens.
Right. And there are different flavors. Some systems, like Bark for text-to-speech, split audio into two streams — semantic tokens that capture what's being said, and acoustic tokens that capture how it sounds. The prosody, the timbre, the speaker identity. Two parallel token sequences for the same audio clip.
So the same second of speech is being tokenized twice, into different vocabularies, and the model has to keep them aligned.
And the alignment is learned during training. The model figures out that semantic token number four hundred and two tends to co-occur with acoustic token number fifteen when the speaker is female and the emotion is neutral. None of that mapping is designed by humans. It's all emergent from the training data.
Which means the meaning of an audio token isn't just opaque to us — it's opaque to the tokenizer that created it. The tokenizer doesn't know what the token means. It just knows it's a good compression.
That's the thing that I think gets missed in most discussions. In text, a token has some relationship to meaning — subwords carry morphological information, whole words carry semantics. A visual token from a VQ-VAE is just a cluster of pixels that the model learned to compress well. It has no inherent meaning. It's a convenient bucket.
And code? You mentioned code tokenization is different again.
Code is fascinating because it sits in between. Code looks like text — it's characters on a screen — but it has structure that natural language doesn't. Indentation matters. Brackets have to match. Variable names follow conventions. So code-specific tokenizers like the ones in CodeLlama or StarCoder use approaches that preserve syntactic structure. Some are byte-level, meaning they can handle any character in any encoding without ever producing an unknown token. Others are AST-aware — they parse the abstract syntax tree before tokenizing, so a variable name like "userAuthenticationHandler" might stay as one token instead of being split into subwords.
Because splitting it would lose the fact that it's a single identifier.
In natural language, splitting "unbelievable" into "un" "believe" "able" is useful — it captures morphology. In code, splitting a function name into pieces can destroy information the model needs to generate correct programs.
So we've got at least three completely different tokenization philosophies — patch-based compression for images, spectrogram-based codecs for audio, syntax-aware splitting for code — and they all produce things we call tokens. And these tokens all live in the same context window?
They do. That's the wild part. A single inference call to GPT-4V or Gemini might have text tokens, image tokens, and audio tokens all interleaved in the same sequence. The model's attention mechanism processes them all the same way — it computes dot products between query and key vectors regardless of where the token came from.
But the vocabularies are completely disjoint. Token number five thousand in the text vocabulary is the subword "tion". Token number five thousand in the image vocabulary is... some patch of blue sky, maybe.
Right. They share a format — they're all integer IDs — but they don't share a semantic space. The model has to learn cross-modal mappings during training. Systems like Flamingo or LLaVA use cross-attention layers specifically designed to align visual tokens with text tokens. The model learns that certain visual token patterns correspond to the word "cat" or "sunset" or "traffic light." But that mapping is learned, not inherent. It's not like the tokenizers were designed to be compatible.
So the token is an engineering convenience, not a natural category.
That's... that's exactly right. It's whatever the architecture needs it to be. And that brings us to Daniel's core question — does the concept of a standard token estimate become redundant?
I mean, from what you're describing, the answer is yes for anything that isn't pure text. But also no, because we still need some way to measure and price these things.
That's the tension. The heuristic is dead for understanding what the model is actually processing, but it's alive and well for billing. Google's Gemini API now returns separate token counts in its response — text tokens, image tokens, audio tokens, each in their own field. OpenAI prices image tokens differently from text tokens for GPT-4V. They're acknowledging that these are different things by charging different rates.
Which makes sense. An image token encodes way more raw information than a text token. You're paying for the information density.
And the compute cost. Processing a two hundred fifty-eight token image through the transformer is more expensive than processing two hundred fifty-eight text tokens because the image tokens carry richer representations that require more attention computation.
But here's where it gets messy for developers. You're building an app, you're budgeting your context window — a hundred twenty-eight thousand tokens sounds like a lot. Until you realize that's a hundred twenty-eight thousand tokens of mixed modalities. One high-resolution image eats a thousand tokens. Thirty seconds of audio is another fifteen hundred. Suddenly your hundred twenty-eight thousand token context window isn't a hundred thousand words — it's maybe twenty thousand words plus a handful of images plus a few audio clips.
The effective capacity depends entirely on the modality mix. And that's not obvious from the spec sheet. A model advertises a context window size as a single number, but what that number means in practice varies wildly depending on what you're putting in.
It's like a suitcase that claims to hold forty liters, but half the space is taken up by the frame if you're packing anything rigid.
The container is the same size, but what fits inside depends on the shape of what you're packing.
Are we seeing any push toward a universal tokenizer? Something that handles all modalities with one vocabulary?
There's research in that direction. 's ImageBind project explores a shared embedding space across six modalities — text, image, audio, depth, thermal, and IMU data. The idea is you train encoders for each modality that all map into the same vector space, so a picture of a dog and the sound of a dog barking land near each other.
But that's embeddings, not tokens.
Right. It's a shared representation space, not a shared tokenizer. And there's a reason for that. Each modality benefits from specialized compression. Images have spatial redundancy that's best compressed with convolutional approaches. Audio has temporal redundancy that's best compressed with codecs that understand frequency. Text has sequential structure that's best compressed with subword tokenization. Trying to build one tokenizer that handles all three well is like trying to build one engine that's equally efficient in a car, a boat, and a plane.
You could do it, but it wouldn't be great at any of them.
And the current state of the art seems to agree. Google has research on multimodal tokenizers, but even their production systems keep the tokenizers separate. The unification happens in the transformer, not in the tokenization layer.
So the token remains this convenient abstraction — a wrapper around whatever compression scheme works best for a given modality — and the model learns to make sense of the mess downstream.
And the mess is only going to get messier. The next frontier is tokenizing three-dimensional environments, video streams, sensor data. Each of those will need its own compression scheme, its own token vocabulary, its own quirks. The word "token" will keep stretching.
Which means we're going to keep having conversations where we use the same word to mean fundamentally different things, and hope context disambiguates.
I think that's the state we're in, and I don't see it changing soon. The token is too useful as an architectural primitive to abandon, but its meaning is getting more fragmented by the month.
Let me push on something. You said earlier that a visual token has no inherent meaning — it's just a learned compression bucket. But isn't that true of text tokens too, at some level? The subword "tion" doesn't have meaning in isolation either. It only means something in context.
That's fair. The difference is one of degree, not kind. A subword token like "tion" carries statistical regularities that correlate with meaning — it tends to appear in nouns derived from Latin verbs. A visual token from a VQ-VAE carries statistical regularities that correlate with... local texture patterns. The correlation with meaning is much weaker and much less interpretable.
Weaker and less interpretable to us. The model might disagree.
The model almost certainly disagrees. To the model, visual token number seven thousand four hundred and twelve might be as semantically rich as the word "democracy" is to us. We just can't read its dictionary.
That's unsettling in a way I can't quite articulate.
It's the interpretability problem, but at the token level. We built these systems, we defined the token abstraction, and now we're generating tokens whose meaning we can't decode. We know they work — the model reconstructs coherent images and intelligible speech — but we don't know what any individual token represents.
It's like we invented an alphabet and then forgot how to read it.
And we're building products on top of it. Pricing tiers, context window budgets, rate limits — all denominated in an alphabet nobody can read.
Alright, so where does this leave the developer who's just trying to build something? What do they actually need to know?
Three practical things. First, stop using the one-token-equals-three-quarters-of-a-word heuristic for anything multimodal. It's wrong and it'll bite you. Use the API's token counting endpoint — both Google and OpenAI expose this — and test with real multimodal inputs.
Test with the actual images and audio clips you'll be using, not placeholder text.
A photograph of a busy street has the same token count as a solid blue square if they're the same resolution, but the information content is wildly different. The token count tells you the cost, not the content.
Second thing?
Budget your context window by modality. If you're building an app that processes user-uploaded photos plus text, figure out your average image token cost and your average text token cost separately, then add them. Don't assume a hundred thousand tokens means a hundred thousand tokens of whatever you feel like throwing in.
And third?
Watch the pricing. Image tokens cost more than text tokens per token on most platforms. Audio tokens are in their own tier. The pricing models are getting more granular as the modality mix gets more complex. What looks like a good deal for text-only might be expensive for multimodal.
So the token is simultaneously more abstract and more concrete than it used to be. More abstract in what it represents — anything from a pixel patch to a phoneme — and more concrete in how it's priced and counted.
That's the paradox. The term is doing more work than ever while meaning less than ever.
I keep coming back to this idea that we're papering over deep incompatibilities with a single word. And it works, mostly. The abstraction holds. But the cracks are showing.
The cracks show up in weird ways. I was reading about a bug in an early multimodal model where the image tokenizer and the text tokenizer accidentally used overlapping integer ranges. So the model would see token number five hundred and have no idea whether it was looking at a piece of an image or the word "the." It learned to disambiguate from context, which is impressive, but also...
Also that should not have been a problem the model had to solve.
Right. That's an engineering oversight that the model papered over with learned behavior. And we probably have dozens of those we haven't found yet.
Because the token abstraction hides them. If everything is just an integer ID, you don't see the collision until something breaks.
And when something breaks, it breaks in ways that are hard to diagnose. The model generates a weird image, or the audio doesn't match the text, and you're left wondering whether the problem is in the tokenization, the attention, the decoding, or somewhere in the learned cross-modal mapping.
Debugging a black box through a fog of abstractions.
That's the job now. That's what multimodal AI engineering is.
You mentioned Anthropic's approach with Claude — they treat images as a fixed token cost per resolution tier rather than proportional to content. That's a different philosophy.
It's simpler. Instead of saying this image costs exactly two hundred fifty-eight tokens based on its patch count, they say a standard resolution image costs this many tokens, period. It's less precise but easier to reason about. You know your costs upfront without running the tokenizer.
Which approach is better?
Depends on what you're optimizing for. Predictability versus granularity. Google and OpenAI's approach gives you finer-grained billing — you pay for exactly what you use. Anthropic's approach gives you simpler budgeting — you know the cost before you even have the image.
I suspect we're going to see more of the Anthropic approach over time, just because developers hate surprise bills.
The market will decide that one. But it's another example of how the token abstraction is fracturing. Even the counting methodology isn't standardized across providers.
We haven't even talked about video.
Video is where this gets genuinely overwhelming. A video is a sequence of frames, each frame is an image, each image is a grid of patches. A ten-second clip at thirty frames per second is three hundred images. If each image is two hundred fifty-eight tokens, that's over seventy-seven thousand tokens for ten seconds of video.
Before you've added a single word of text.
Before any text. And that's at low resolution. Real video understanding models are going to need much more efficient tokenization — maybe tokenizing spatiotemporal chunks instead of individual frames, compressing across time as well as space.
So the tokenization problem isn't solved. It's just getting started.
The tokenization problem is going to be one of the defining engineering challenges of the next five years. How do you compress the physical world into discrete tokens efficiently enough that a transformer can reason about it in real time?
And we'll still call them tokens.
We'll still call them tokens.
Hilbert: I spent six months in twenty nineteen tokenizing CT scans by hand.
I'm sorry, what?
Hilbert: Startup called VoxelBound. Went under in twenty twenty. My job was to sit at a terminal and manually segment medical CT scans into cubes of voxel data — three-dimensional pixels. We called them tokens because the CTO, a man named Greg who wore sandals in February, said tokens sounded more AI than voxels. They were just labeled cubes. Liver tissue, bone, blood vessel, tumor margin. Each cube got an integer. I labeled about forty thousand of them.
That's... a lot of cubes.
Hilbert: Greg said we were building the vocabulary for a model that would one day diagnose cancer from whole-body scans. The model never shipped. The investors wanted a chatbot. But I've never been able to shake the feeling that calling everything a token is a kind of cargo cult. We borrowed the prestige of language tokenization and applied it to things that have no linguistic structure whatsoever.
Cargo cult is the phrase I was looking for earlier. The term gives a false sense of uniformity — like all these things are the same kind of thing because they share a name.
Hilbert: A visual token from one of those VQ-somethings is a cluster of pixels the model learned to compress. It has no more inherent meaning than the cube of liver tissue I labeled number three thousand and twelve. It's a bucket. We're using the same word for subwords that carry grammatical information and for buckets of pixels, and I think that obscures how little we understand about what these models are actually representing.
I don't think you're wrong. But I also think the abstraction is useful. The transformer doesn't care what the integer represents — it just needs a discrete vocabulary to attend over. The token is the minimum viable interface between raw data and the attention mechanism.
Hilbert: Sure. It works. But naming things is how we think about them. Call everything a token and you start assuming the model understands a CT scan the same way it understands a sentence. It doesn't. The mapping is learned, it's statistical, and we can't read the dictionary.
You said your CTO wore sandals in February.
Hilbert: Greg. Yes.
That's the detail that makes me trust your account of this.
Hilbert: He also claimed the tokenization scheme was inspired by how the human visual cortex processes edges. I read the paper he cited. It wasn't.
The human visual cortex doesn't use discrete tokens.
Hilbert: Greg knew that. He just liked the way it sounded in pitch decks. VoxelBound raised four million dollars on the phrase "cortical tokenization." I still have the deck somewhere.
Of course you do.
Hilbert: The point is, the word token is doing too much work. It started as a precise technical term in natural language processing and now it means whatever the person using it needs it to mean. That's how words lose their usefulness.
It's also how words evolve. The meaning expands to cover new cases. The question is whether the expansion is serving clarity or obscuring difference.
Hilbert: I'd say obscuring. But I'm biased. I spent six months labeling cubes for a company that never shipped a product.
I think there's a middle ground. The token is a useful engineering abstraction that's going to stick around because it works. But we should be honest about what it's hiding. Every time someone says a model has a million-token context window, the follow-up question should be: a million tokens of what?
That's the right framing. The number alone is meaningless without the modality breakdown.
Hilbert: Greg used to say VoxelBound had a billion-token training corpus. It was CT scans from two hundred patients.
That's a good note to end on — or at least to end this part of the conversation on.
Before we wrap, there's one detail from the research that didn't fit anywhere but I want to mention. Stable Audio Two point zero tokenizes forty-four point one kilohertz stereo audio at about forty-three tokens per second using a variational autoencoder. That's remarkably efficient — forty-three integers to represent a second of high-fidelity stereo sound. The compression ratio is something like a thousand to one.
That is wild. A second of CD-quality audio is about one hundred seventy-six thousand bytes uncompressed, and they're representing it with forty-three integers.
The model learns a compression scheme that's tuned for reconstruction quality, not for human interpretability. Those forty-three tokens are completely opaque to us, but the decoder can reconstruct audio that's nearly indistinguishable from the original. It's lossy compression where the loss function is perceptual — it throws away what humans won't notice.
Which circles back to the core problem. The token is an abstraction that works, but we can't read it. We're building systems that communicate in a language we don't speak.
And that's either the most exciting thing in AI or the most troubling, depending on your disposition.
Probably both. The open question I keep coming back to is whether we'll eventually abandon the token abstraction in favor of modality-specific units. As models become natively multimodal — GPT-five, Gemini Ultra Two, whatever comes next — will we still talk about tokens, or will we talk about patches and frames and phonemes?
I think the token is here to stay as an engineering convenience. It's the common currency of the transformer architecture, and that architecture isn't going anywhere. But its meaning will keep fragmenting. We'll have visual tokens and audio tokens and code tokens and whatever comes next, and we'll keep using the same word for all of them while knowing they're not the same thing.
A useful fiction.
The most useful fictions are the ones you forget are fictions.
Thanks to our producer Hilbert Flumingtop for keeping us honest — and for the CT scan story, which I'm going to be thinking about for a while. This has been My Weird Prompts. If you want to dig into the technical details we touched on, check the show notes for links to Google's token counting documentation and the original VQ-VAE paper. And send your weird prompts to show at my weird prompts dot com. We'll be back soon.
