Daniel's been reading DeepSeek's release notes again, and he's noticed something that should bother more people than it does. He says the new model, the one everyone's calling a point release, is actually a completely new architecture. Forty layers, a different encoder, and a memory system that doesn't work like anything DeepSeek has shipped before. His point is that most people working in AI can name the transformer architecture, they can nod along when someone says encoder or decoder, they've heard the term KV cache thrown around, but they've never had to hold all three in their head at once and ask how they actually fit together. So he wants us to build that intermediate layer today. What do layers actually do, how does the encoder and decoder split interact with them, and where does the KV cache sit in all of that, because that's where DeepSeek made the change that matters.
And the first thing I have to say is that Daniel's got the model name slightly wrong, which is almost the story itself. There is no DeepSeek five point one. There isn't even a DeepSeek V four point one flagship. What shipped yesterday is DeepSeek V four point one Flash. The predecessor is V four Flash. The big model, V four Pro, is being retired and routed to this new Flash model starting in a few days. So DeepSeek shipped what is by every technical measure a new generation of model and then gave it a name that sounds like a minor revision. One watcher described it as more different from V four than Llama three was from Llama one. And they still called it a point release.
That's either humility or a very specific kind of marketing. Save some room in the version number for the thing you're actually proud of.
Or they don't want to call it V five until the Pro version lands. But let's get to the actual question, because Daniel's framing is right. The transformer architecture is a stack of layers. Most people know that sentence. What they don't usually picture is that each layer is a small factory with two machines in it. One machine does attention, which is the model deciding which earlier tokens matter for the token it's currently processing. The other machine is a feed-forward network, which is a big pile of matrix multiplications that transforms whatever the attention step produced. And here's the thing that matters for everything we're about to discuss. In a standard transformer, every single one of those layers computes its own keys and values for every token in the context. That's the KV cache. Each layer has its own private copy of what it's already seen.
So forty layers means forty separate memories of the same conversation.
Forty separate caches, yes. And that's the part that costs real money when you're serving these models. Every time the model generates a new token, it doesn't recompute the whole context from scratch. It reaches into the cache and pulls out the keys and values it needs. If the cache is enormous, that reach is slow and it eats GPU memory. The cache size is not a footnote. It's the difference between serving a million tokens of context to one user and serving it to a hundred.
And DeepSeek's history here is the clearest way to see the trend. Their first model, V one, used about three hundred eighty nine thousand bytes per token for the KV cache. V three point two, late last year, got that down to forty eight thousand. V four Flash in April brought it to thirty five hundred. The new model is eight hundred ninety bytes per token. So a full million-token context now costs about eight hundred ninety megabytes of cache. That's down from a number that would have required a data center to itself.
Four hundred thirty seven times smaller than V one, if you trust their arithmetic. The one quarter figure compared to V four Flash is the cleaner comparison. Same baseline, same measurement. Either way, the direction is unmistakable. And they got there by changing the architecture, not just by compressing harder. That's the part Daniel's picking up on.
So walk me through the forty layers. What's the split actually doing?
The new model has twenty layers doing what they call a causal encoder, and twenty doing the decoder. Now, I want to be careful here, because when people hear encoder and decoder they think of the old BERT style setup, where an encoder reads the whole input and a decoder generates from scratch. That's not what this is. This is closer to a design Microsoft published a couple of years ago called YOCO, You Only Cache Once. The idea is that the first half of the network reads the prompt and produces a single set of keys and values. The second half of the network doesn't compute its own cache at all. It just reaches into the encoder's cache through cross-attention.
So the decoder's memory is borrowed from the encoder. That's the whole trick.
That's the whole trick. And it changes the economics of serving. When you're processing the input prompt, the prefill phase, only the encoder is doing heavy work. DeepSeek says that's about eight billion active parameters per token. When you're generating output, the decode phase, the decoder kicks in and that's sixteen billion active parameters per token. So the model is asymmetric. It's cheaper to read than to write, which is exactly how it should be for agent workloads where the model reads the same context over and over while producing short bursts of output.
Eight billion parameters active during prefill. Total model size is seven hundred sixty three billion. So at any given moment, the model is using about one percent of itself.
That's the mixture of experts design, which has been DeepSeek's thing for a while. One shared expert plus three hundred eighty four routed experts, six active per token. But the new part, the part that makes this release different, is that a huge chunk of those seven hundred sixty three billion parameters aren't transformer layers at all. They're lookup tables. One hundred ninety six billion parameters of what DeepSeek calls Engram.
Engram. That's the term from their January paper. Conditional memory via scalable lookup.
Right. And the idea is almost embarrassingly simple when you say it out loud. A huge amount of what a language model stores is factual knowledge. Paris is the capital of France. The chemical formula for table salt. That kind of thing doesn't need to be computed through forty layers of matrix multiplication every time it comes up. It can be looked up. So DeepSeek built a giant table, indexed by hashed sequences of tokens, and when the model sees a familiar n-gram, it does a handful of lookups and gets back a vector that injects the relevant knowledge directly into the hidden state.
So it's an encyclopedia with a really aggressive index.
The Register actually used almost exactly that phrase. An oddly specific encyclopedia that opens the right page almost instantly. And because these are lookups, not matrix multiplications, the Engram weights don't need to sit in GPU memory. They can live in system RAM or even storage. The Register's math says that drops the minimum GPU memory from around seven hundred sixty three gigs in FP eight to about five hundred sixty seven gigs. That's the difference between needing eight high-end GPUs and needing six.
And the model still has to know when to look something up. That's not free.
No, and that's where the mechanism gets interesting. The Engram module sits at layers one and fourteen of the backbone. So it's not a separate system bolted onto the side. It's woven into the early layers. The paper's mechanistic analysis found something that sounds almost like a division of labor. The early layers of a normal transformer spend a lot of their capacity doing what they call static reconstruction. Rebuilding factual associations from the weights. Engram relieves them of that job, and the freed-up capacity effectively deepens the network for the reasoning work that has to happen later.
So the model didn't get smarter by adding more reasoning layers. It got smarter by moving the memory out of the reasoning path.
And that's the cleanest way to understand why this is a new architecture. The shape of the model changed. Memory and computation used to be the same thing. Every parameter was a matrix weight, every fact was stored as a pattern in those weights, and every forward pass had to read all of it. Now there's a split. The backbone does the thinking. The Engram does the remembering. They talk to each other through those lookup vectors.
Which brings us to the KV cache, because that's the other half of the memory story. The Engram is long-term memory, the stuff that doesn't change between requests. The KV cache is working memory, the stuff about this specific conversation.
And the KV cache in this model has its own internal structure now. It's not just a smaller version of the old thing. DeepSeek introduced something called Compressed Sparse Attention 2, CSA2, and it assigns every attention layer one of three jobs. A Full layer computes its own keys and values, selects the top five hundred twelve positions that matter, and stores those. A Reindex layer reuses the keys and values from the last Full layer but rescans them with its own query to pick a fresh set of relevant positions. A Reuse layer just takes whatever the last Full layer picked and uses it unchanged.
So within the forty layers, only some of them are doing the expensive work of deciding what to pay attention to. The rest are freeloading on those decisions.
In the encoder, the eighteen CSA2 layers are arranged in three groups of six, with a compression ratio of two. The decoder's twenty layers are in five groups of four, ratio one. So the decoder is doing more full attention work per layer, which makes sense because generation is the expensive part. And there's a hierarchical sparse indexer that lets a Full layer build a candidate pool of up to sixteen thousand positions, so the Reindex layers only have to score a bounded set rather than the whole context.
Sixteen thousand candidates out of a million tokens. That's the model deciding, in advance, what's worth looking at again.
And that's the part that's easy to miss. The KV cache isn't just smaller. It's smarter. Instead of storing everything and hoping the attention mechanism finds what it needs, the model is actively curating what goes into the cache. The Full layers are the curators. They scan the context, pick the five hundred twelve positions that seem most relevant, and write a compressed summary of those positions into the cache. The Reindex and Reuse layers then work from that curated set.
So the cache is a summary, not a transcript.
A hierarchical summary. And the compression is aggressive. FP4 quantization for the main KV cache, which is half the size of the FP8 they used in V four. Plus a technique they call SWA Bounded Replay, which handles the sliding window attention states by replaying only the last hundred twenty eight tokens when they need to be rebuilt, rather than persisting them to disk. That cut the persistent KV storage to about one eighth of what V four Flash needed.
Let me make sure I'm keeping the pieces straight. The Engram handles factual recall by looking things up in a giant table. The KV cache handles conversational context by storing compressed, curated attention states. The layers are split so the encoder does the reading and the decoder does the writing. And the whole thing is designed around the observation that agents read a lot and write a little.
That's the shape. And the reason it matters for people who aren't machine learning engineers is that this shape determines what the model is good at. DeepSeek's own benchmarks tell the story. On agentic coding tasks, the things that require reading a huge codebase and making small precise edits, this model is at the frontier. Terminal Bench two point one, ninety point six versus Opus five's eighty nine point one. DeepSWE, seventy four point two versus seventy four point zero. Automation Bench, fifty four point eight versus fifty point three. These are real leads.
But.
But on the harder versions of those same benchmarks, the picture flips. Terminal Bench three point zero, the new version, this model scores thirty point zero. Opus five scores forty three point three. HLE, the Humanity's Last Exam benchmark, thirty six point eight versus fifty six point three. That's a twenty point gap on the hardest knowledge test.
So the model that offloaded its memory to a lookup table is worse at remembering things.
There's a thirteen point gap on SimpleQA compared to V four Pro. The model does not know, but it can find out. That's the trade-off, and it's explicit in the design. If you want an agent that reads documentation, navigates a codebase, and makes changes, this is incredibly good. If you want an oracle that can answer any factual question from memory, this is not the model.
And that's a real choice, not an accident. They didn't fail to make it good at both. They aimed the architecture at the agent use case and accepted the knowledge hit.
Which is why the naming thing bothers me. This isn't a point release. This is DeepSeek saying, here's the shape of our next five years of models. Causal encoder decoder, compressed sparse attention, Engram lookup memory. That's the template. And they called it V four point one Flash.
Maybe the point one is the template number. V four point one means the first model built on the fourth generation architecture. V five will be something else.
That's the charitable reading. The less charitable one is that they're keeping V five in reserve for the Pro model, and everyone who migrates their API calls to V four point one Flash now will have to migrate again when the real flagship lands. The API routing change is already scheduled. Starting September fourteenth, every request to deepseek V four Pro gets routed to this Flash model and billed at Flash rates.
That's a hell of a way to retire a flagship. Replace it with a smaller model and charge less.
Because the smaller model beats it on most of the benchmarks they care about. DeepSeek's official line is that multiple parties put V four point one Flash ahead of V four Pro on performance, cost, speed, and total runtime. So they're phasing out the Pro. It's not a downgrade. It's a collapse of the product line. Why maintain a bigger, more expensive model when the new architecture does the job better for less?
And the pricing tells you who they're selling to. Cache hit input, three tenths of a cent per million tokens off peak. Cache miss input, fifteen cents. Output, sixty cents. Compare that to Claude Opus five at five dollars in, twenty five dollars out. That's a twenty to forty times gap on output.
The cache hit pricing is the interesting part. If your agent sends the same system prompt and the same tool definitions on every request, those tokens hit the cache and you pay almost nothing for them. DeepSeek is explicitly pricing for agents that re-read the same context constantly. The architecture and the business model are aligned.
And this isn't just DeepSeek's idea anymore. Qwen shipped something in August, Qwen three point eight Flash Next, that uses the same n-gram lookup approach. Fifty one billion parameters of embedding table held off the accelerator. Google's been working on something similar with per-layer embeddings, though only on tiny models so far. The idea is spreading.
The n-gram memory thing is probably the defining architectural trend of the year. DeepSeek published the Engram paper in January. Qwen picked it up by August. That's a very fast turnaround for a completely different lab to adopt and ship. And the paper's numbers are striking. Scaling Engram to twenty seven billion parameters beat an equivalent mixture of experts baseline on both knowledge and reasoning. MMLU up three point four, BBH up five point zero. Long context retrieval jumped from eighty four point two to ninety seven point zero on their multi-query test.
So the lookup table isn't just cheaper. It's better at some things.
For that specific job, yes. And the mechanistic finding, that it relieves the early layers from static reconstruction, suggests something deeper. The transformer was doing two jobs at once. Remembering facts and reasoning about them. Now those jobs are split across different components with different shapes. The reasoning path is a stack of attention and feed-forward layers. The memory path is a hash table with a really good index.
Which brings me back to Daniel's actual question, because I think he's right that this is where most people get lost. They hear forty layers and they picture forty identical boxes stacked on top of each other. But the layers in this model aren't identical. Some of them are doing full attention with their own cache. Some are reindexing. Some are reusing. The Engram sits at specific layers, not distributed evenly. The encoder layers and decoder layers have different jobs. So the model's internal shape is not a uniform stack. It's a pipeline with specialized stages.
And the KV cache isn't a single buffer. It's a hierarchical structure with different levels of curation. The global cache is projected from the encoder's final hidden states, which is the YOCO idea. Then the CSA2 layers maintain their own local views of that global cache, with Full layers curating and the others borrowing. The whole thing is shaped like a funnel. A million tokens come in, the encoder compresses them into a global cache of about eight hundred ninety megabytes, and then the attention layers work from curated subsets of that.
When DeepSeek says the model has one million tokens of context, what they actually mean is that the encoder can compress one million tokens into a cache that fits in under a gigabyte, and the decoder can still find what it needs in that compressed representation.
That's the thing that was impossible with the old architecture. V one's cache would have needed hundreds of gigabytes for the same context. The context window was technically there but practically useless. Now it's actually deployable. That's the difference between a spec sheet number and a feature.
There's a benchmark detail I want to put on the table because it connects to this. DeepSeek tested the same model, same benchmark, DeepSWE version one point one, with different agent scaffolding. The score ranged from sixty five point six with Codex to seventy four point two with mini-SWE. That's an eight point seven point swing. Same weights. Same model. The only difference is the harness that calls it.
Which means the benchmark score belongs to the model plus the scaffold, not the model alone. When you see a headline saying DeepSeek beats Opus five, you have to ask which scaffold they were using. And when you see a headline saying Opus five beats DeepSeek, same question. The model is a component in a system, not the whole system.
That's probably the most transferable insight from this whole release. The architecture changes we're talking about, the layers, the encoder decoder split, the KV cache, they only matter because of how the model gets used. A model that's going to sit behind a chat interface has different needs than a model that's going to be called by an agent harness fifty times in a single task.
DeepSeek has clearly decided which one they're building for. The asymmetric activation, eight billion parameters for input, sixteen for output. The cache hit pricing. The benchmark focus on agentic coding. The million token context that's actually deployable. Every design choice points the same direction.
The flip side is the knowledge gap. HLE at thirty six point eight is not a rounding error. If you need a model to answer hard questions from memory, this is the wrong tool. But if you need a model to read a codebase and make changes, the knowledge gap matters less because the model can read the documentation as part of its context.
The model does not know, but it can find out. That's the trade, and it's a trade DeepSeek made deliberately. The interesting question is whether the market agrees with the trade. If agentic workloads really are the future, then this is the right bet. If people still mostly want an oracle, then Opus five's knowledge advantage wins.
The naming decision, calling this a point release, means a lot of people are going to adopt it without realizing they've bought into a new architecture. The API endpoint changes, the bill goes down, the benchmarks look good on the saturated versions, and nobody stops to ask why the model is suddenly worse at HLE.
That's the migration budget paying for the confusion, as one observer put it. DeepSeek shipped a generation and declined to name it.
Let me ask you something about the encoder. Daniel specifically mentioned a different encoder, and the research bears that out, but I want to understand what changed. The old model had a vision encoder bolted on after the fact. This one has a vision encoder trained from scratch, from the beginning of pre-training, with a technique called two dimensional rotary position embeddings.
Right. The DeepSeek ViT, their vision transformer, was trained alongside the text model from the start. That's different from the old approach where you train a text model, then train a vision encoder separately, then glue them together with an adapter. Here the visual and textual processing share the same representational space from day one. The forty five trillion pre-training tokens are a seven to one mix of text to multimodal. So the model learned to think about images and text as the same kind of thing from the beginning.
That matters for the encoder decoder split, because the encoder is now processing both modalities through the same causal attention mechanism. A screenshot of a codebase and the text of the codebase both become tokens, both flow through the same twenty encoder layers, both get compressed into the same KV cache.
That's the native multimodality story. It's not a text model with a vision module attached. It's a single model that was trained from the start to treat pixels and words as the same kind of input. The two dimensional rotary embeddings are how it keeps track of spatial relationships in images the way one dimensional rotary embeddings keep track of word order in text.
When Daniel says the encoder is different, he's right on two counts. The text encoder is now a causal encoder that feeds a shared cache to the decoder, and the vision encoder is a from-scratch model trained jointly with everything else.
Both changes serve the same goal. Make the input side cheaper and more capable, so the output side can focus on generation. The encoder does the reading, builds the compressed representation, and the decoder just writes.
I want to circle back to the Engram for a second, because there's a detail I haven't seen many people discuss. The lookup tables are indexed by hashed n-grams. So there's a hash function involved. Tokens come in, they get grouped into n-grams, the n-grams get hashed, and the hash is used to look up the memory vector. What happens on a hash collision?
That's a good question and I don't know the answer from the paper. Hash collisions in lookup tables are a known problem. Either you design the table to be large enough that collisions are rare, or you have a collision resolution strategy. My guess is the former, because the table is enormous, one hundred ninety six billion parameters. But I haven't seen DeepSeek address it directly.
It's the kind of thing that would show up as weird confabulations. If two different n-grams hash to the same entry, the model might retrieve the wrong fact and not realize it.
That could be part of the knowledge gap. If the Engram is retrieving the wrong vector some fraction of the time, the model would be confident and wrong. That's a testable hypothesis, but I haven't seen anyone run it yet.
The other thing I wonder about is whether the Engram can be updated. If a fact changes, say a country moves its capital, can you retrain just the lookup table, or do you have to retrain the whole model?
That's the actually exciting part of the lookup table design. In principle, you could update the Engram weights independently of the backbone. Retrain the lookup table on new data, keep the reasoning path unchanged. That would be a much cheaper way to keep a model's knowledge current than retraining everything. Whether DeepSeek actually does that, I don't know. But the architecture makes it possible in a way that the old dense transformer never did.
The shape of the model also changes how you maintain it. Memory becomes a replaceable component.
Which is a very different way to think about language models. Instead of a monolithic block of weights that all have to be retrained together, you have a reasoning engine and a memory module that can be swapped. The reasoning engine changes slowly. The memory module can change quickly. That's a much more modular design.
It explains why Qwen adopted it so fast. If you're a lab trying to keep up with the frontier, a modular architecture where you can update the memory without retraining the reasoning engine is a huge operational advantage.
The n-gram approach also has a nice scaling property. The Engram parameters don't need to be read in full for every token. Just a few dozen lookups. So you can make the memory enormous without making inference slower. That's the new axis of sparsity. The old sparsity was about which experts to activate. The new sparsity is about which memories to look up.
The two sparsities interact. The mixture of experts decides which six of three hundred eighty four experts to use. The Engram decides which few dozen of billions of memory entries to look up. Both are ways of having a huge model that only uses a tiny fraction of itself at any moment.
The difference is what gets skipped. In the mixture of experts, you're skipping computation. In the Engram, you're skipping memory reads. Both save time and energy, but they save different kinds.
Let me try to summarize the shape the way I now understand it. The model has seven hundred sixty three billion parameters, but it's not one big uniform block. It's a reasoning backbone of five hundred fifty two billion parameters, arranged in forty layers, split into a twenty layer encoder and a twenty layer decoder. The encoder reads input, builds a compressed global cache, and the decoder generates output by borrowing from that cache. On top of that sits the Engram, one hundred ninety six billion parameters of lookup tables that inject factual knowledge at layers one and fourteen. The KV cache is eight hundred ninety bytes per token, down from three hundred eighty nine thousand bytes per token three years ago. And the whole thing is designed so that reading is cheap and writing is the expensive part.
That's the model. And the reason it matters is that this shape is now the template. DeepSeek has said this is the smallest model in the new architecture family. The Pro version is coming. Qwen is already building on the same ideas. Google is exploring the same direction. The next generation of models is going to look like this, and the generation after that is going to look like whatever this enables.
The people who understand the shape are going to make better decisions about which model to use for which job. That's Daniel's real point. You don't need to be a machine learning engineer to understand that a model with a lookup table for memory is going to be good at some things and bad at others. You just need to know the shape.
Hilbert: Eight hundred ninety bytes.
Per token, yes.
Hilbert: The KV cache. Eight hundred ninety bytes per token. I used to run a print shop. We had a machine that did variable data printing. Personalized mailers. The thing held a database of names and addresses and it would swap them into a template as the press ran. The database lived on a cartridge. Little solid state thing, maybe four megabytes. And the press could pull any record in about half a second. That's what the Engram sounds like to me. A cartridge full of names.
That's not a bad analogy. The lookup is the fast part. The rendering is the slow part.
Hilbert: The cartridge cost more than the press. Everyone thought that was backwards. But the cartridge was where the money was. You could swap it out when the mailing list changed. Didn't have to rebuild the press. I'm wondering if this DeepSeek thing, the Engram, if you could swap it. Like if a client wanted a model that only knew about their industry. You'd just swap the cartridge.
That's exactly the question Corn raised. In principle, yes. The Engram is a separate component. You could retrain it on a specific domain and leave the reasoning backbone alone. Whether DeepSeek offers that as a service, I don't know. But the architecture makes it possible.
Hilbert: The cache thing. Eight hundred ninety bytes per token. That means a million tokens is about eight hundred ninety megabytes. I've got a hard drive from 1998 that's bigger than that. A whole conversation, a million tokens, fits on a disk that's older than my nephew.
The compression story is remarkable. Three years ago the same context would have needed a server rack.
Hilbert: Here's what I want to know. If the cache is that small, could you put it on a cartridge? Like physically. A little solid state thing. Pull the cache out of one machine, stick it in another, and the conversation picks up where it left off.
That's, I think the cache is just data. There's no reason you couldn't serialize it and move it. The question is whether the receiving machine knows what to do with it. The cache is tied to the model's internal representations. If it's the same model, same version, it should work. If it's a different model, probably not.
Hilbert: You could have a conversation on your phone, pull the cartridge, put it in your desktop, and keep going. Same model, same cache. That's what I'm asking.
That's technically plausible. I don't know that anyone's built it, but the cache is just a buffer of keys and values. Serialize it, ship it, deserialize it. The hard part would be the software around it, not the cache itself.
Hilbert: Because I've got a box of cartridges in the garage. From the print shop. Four megabytes each. They still work. If someone wanted to build a thing where you could pull a conversation out of one machine and put it in another, I'd say start with the cartridge. It's a solved problem.
The portable conversation. That's actually a compelling idea for agents. You could hand off a long-running task from one machine to another without losing the context.
Hilbert: The cartridge was always the part people underestimated. Everyone looked at the press. The press was loud and fast and impressive. But the press without the cartridge was just a very expensive paperweight. Same with this model. Everyone's looking at the forty layers. The cartridge is the story.
The Engram as the cartridge. Memory as a replaceable component.
Hilbert: I'm just saying. The people who made the cartridge made more money than the people who made the press.
On that note, there's one detail from the research that didn't fit anywhere else but I want to put on the record. DeepSeek's reasoning effort dial is continuous now. One to a hundred. The max setting generates about two and a half times more output tokens than the default. So the same model can be cheap and fast or expensive and thorough, and the user picks where on the dial they want to be.
That's the other axis of the shape. Not just what the model is, but how hard you ask it to think. And at max effort, the benchmark numbers I quoted earlier are the ceiling. The default settings score lower. So when someone says DeepSeek V four point one Flash beats Opus five, the question is always, at what dial setting, with what scaffold, on which benchmark version.
The shape of the model and the shape of the evaluation. Both matter.
The thing I keep thinking about is whether this modular approach to memory is going to change how we think about model updates. If the Engram can be swapped independently, then a model's knowledge could be updated weekly while its reasoning stays stable for a year. That's a very different release cadence than the monolithic models we're used to.
The cartridge model of AI. Swap the memory, keep the brain.
Thanks to Hilbert Flumingtop for producing, as always. This has been My Weird Prompts. If you want to reach us, email us at show at my weird prompts dot com. We'll be back soon.
