Daniel sent us this one. He's digging into Mixture of Experts architectures, specifically how granular the router's vision can be. His question is essentially this: if you split a huge model into experts, how fine-grained can that split be before you start losing the plot? Say you have a programming expert chunk — that's still huge. Is there a Python expert? A Python web frameworks expert? If you go too narrow, you risk missing crucial context, like TypeScript in a Python project. But too broad, and you're hauling around a ton of dead weight during inference. He wants to know the approximate division of experts in today's real models, how far we could theoretically push this segmentation for precision, and which providers are exploring those limits.
That is a fantastic, layered question. And it’s timely because the adoption of MOE architectures for large-scale language models is accelerating precisely to solve the inference speed and cost problem. Everyone wants the capability of a trillion-parameter model but only wants to pay for, say, thirty billion parameters at a time.
By the way, today's script is being powered by deepseek-v3-two.
It's a perfect example to keep in mind because it’s one of the largest MoE models out there, and we’ll come back to its specific architecture.
Where do we even start with this? The router’s vision, or lack thereof, seems like the central nervous system of the whole operation.
It absolutely is. And the push for more granular, more intelligent routing is where a lot of the research action is right now. Because the initial promise of MOE was simple: have a big pool of specialized sub-networks, and a smart gate that picks the right ones for the job. But as Daniel’s question points out, ‘specialized’ is a sliding scale. Is an expert for ‘code’ specialized enough? Or do you need one for ‘Python data science with pandas and NumPy’? The router’s job is to make that call, and its vision determines the entire efficiency profile of the model.
Let's rewind for a second for anyone new to the concept. Can you just briefly define what we mean by a "Mixture of Experts" layer? I think it'll help frame the granularity discussion.
And that’s why it’s important to understand that a Mixture of Experts model isn’t one monolithic neural network. It’s a collection of smaller subnetworks, the ‘experts,’ and a routing mechanism that decides which experts get to see and process any given input token. The core innovation is sparsity—only a small subset of the total parameters are activated per token.
Which is the whole efficiency play. You get to have a massive model on disk, but only pay the computational cost of a much smaller one during inference.
The router is the traffic cop. It looks at the input—the prompt, the token—and assigns a probability score to each expert. The top-k experts with the highest scores get activated. That k is usually very small, like two or four. So out of a pool of, say, a hundred and twenty-eight experts, only four might light up for any step.
This is where Daniel's granularity question bites. The router's vision, its ability to discriminate, is everything. If your experts are too broadly defined—like one giant 'programming' expert—then you're activating a huge, generalized chunk every time someone asks about code, which is wasteful. If they're too narrow—like a 'Python list comprehensions' expert—then the router has to be impossibly precise, and you risk missing adjacent, necessary knowledge.
That's the central tension—how specialized can we make these experts before the routing itself becomes a bottleneck, or before we lose the interdisciplinary connections that make these models smart? Corn, I think the real question is: Is there an optimal point, or does it vary by task? Because that's what we need to unpack next.
Let's get concrete—how does this traffic cop actually decide? Walk us through the router's decision-making process on a real input.
The router is typically a small neural network itself, often just a linear layer. It takes the current hidden state of the model—the representation of the token it's looking at—and computes a score for every expert in the layer. Those scores are usually just a dot product, a similarity measure between the input and a learned vector for each expert. Top-k winners get in. It's fast, it's differentiable, so it can be trained end-to-end with the rest of the model.
The router learns, through training, that certain patterns in the hidden state correlate with certain experts. If the hidden state screams 'Python syntax,' it lights up the experts that have seen the most Python during training.
But here's the first granularity tradeoff. That router has limited capacity—it's a small network. If you have a thousand experts, it has to distinguish between a thousand different specializations. Can it learn to cleanly separate 'Python web frameworks' from 'Python data science' from 'Python scripting'? But the more experts you have, the harder that discrimination task becomes for the router. It might start making noisy choices.
Which would mean you're activating the wrong experts, wasting compute, and potentially degrading output quality.
So there's a balancing act between the number of experts and the router's ability to cleanly route to them. This is where a model like Google's Switch Transformer comes in as a canonical case study. It famously scaled to over a thousand experts. But the routing wasn't perfectly precise; they used load balancing losses to ensure no expert was over or under-used, which is a hint that the router alone couldn't perfectly distribute the workload.
Can you give an example of what that load balancing looks like in practice? What happens if an expert becomes too popular?
In the Switch Transformer paper, they added an auxiliary loss that penalizes the model if the routing probabilities become too skewed. Imagine one expert, say the "general knowledge" expert, starts getting selected for 50% of all tokens. That defeats the purpose of sparsity. The loss function essentially gives the router a nudge, saying "Hey, spread the love a little." It forces exploration during training so other experts learn useful specializations. It's a clever hack, but it's also an admission that left to its own devices, the router might not naturally find a perfectly balanced, granular solution.
Even with a thousand experts, they're not a thousand hyper-specialized silos. They're broader categories, and the load balancing is a kind of safety net.
And this gets us to the core of Daniel's example. Let's say you have a hundred experts and one is broadly 'programming.' That expert will contain knowledge across Python, JavaScript, C++, algorithms, etc. For a pure Python question, you're activating a lot of irrelevant parameters. That's inefficient. Now, let's say you split that into ten programming experts: Python, JavaScript, systems programming, web dev, data science, and so on.
The router now has to be smarter. It sees a prompt about Django, and it needs to know that's more 'Python web dev' than 'general Python' or 'JavaScript web dev.' If it gets it right, you activate a tighter, more relevant parameter set. Speed goes up, precision might too.
Push it further. Make an expert just for 'Python list comprehensions.' Now, for a project that uses list comprehensions but also NumPy arrays and maybe some TypeScript configuration files, the router would have to activate a dozen micro-experts. The overhead of coordinating all those tiny activations—the routing computation itself, the gathering of outputs—can start to erode your speed gains. You've minimized parameter waste but maximized coordination overhead.
There's an analogy here, right? It's like consulting a library. A broad expert is like checking out a whole encyclopedia volume. A narrow expert is like pulling a single, very specific journal article. If your question is complex, you're now running around the library grabbing twenty different articles, and the time spent gathering them might outweigh the benefit of their specificity.
You've potentially fractured knowledge. The 'list comprehensions' expert might not know the broader context of Python scoping rules that affect those comprehensions, because that context lives in a different 'Python core semantics' expert. The model loses its holistic understanding.
Which is a critical point. The magic of large models is the unexpected connections. An overly narrow segmentation might miss that a question about Python decorators is conceptually similar to aspect-oriented programming in Java, because those are in different expert silos. The router would need to see that high-level conceptual link to activate both, which is asking a lot.
What's the real-world scale? What are we actually seeing in deployed models?
The research gives us great snapshots. Take Google's Gemma 4. It uses a fine-grained routing granularity with a hundred and twenty-eight experts, but only activates the equivalent of about three point eight to four billion parameters per token. That's how it achieves near-dense-model accuracy of a thirty-one billion parameter model, but at much lower latency. The experts are numerous, but they're not microscopic.
The other end of the scale?
Look at DeepSeek-V3, which we're using right now. It's a colossal six hundred seventy-one billion parameter model, but uses MoE to only activate about thirty-seven billion per token. The scale of the total expert pool is massive, but the active set is still a sizable, general-purpose chunk. This suggests the experts themselves are still quite broad. The segmentation is for managing insane total scale, not for hyper-specialized per-task precision.
The current state of the art is leaning toward 'many broad experts' rather than 'many narrow experts.' The segmentation limit right now seems to be more about hardware and router capacity than a theoretical desire for ultra-precision.
I think that's a fair summary. The router's vision today is good at distinguishing between major domains—code versus history versus biology—and maybe some sub-domains. But asking it to reliably pick between 'Python pandas' and 'Python NumPy' for every token is probably beyond what these systems are optimized for. The tradeoff in inference speed isn't worth it yet—it feels like a practical engineering compromise.
Right, that 'many broad experts' approach seems sensible. But it makes me wonder about the bleeding edge. If we did push segmentation to its absolute limits, what breaks? What's the concrete failure mode of an overly narrow expert pool?
The research gives us some clues. One fascinating paper from ACL 2023 explored something called 'counterfactual routing' to fix what they call MoE hallucinations. The issue is that if your experts are too narrow, and the router makes a slightly off choice, the activated expert might confidently generate something plausible but wrong based on its limited view. They improved factuality by about three point one percent by having the router consider counterfactuals—what if we'd picked a different expert?—without adding inference time. That tells you the precision of the routing decision itself becomes a quality bottleneck when segmentation is fine.
The risk isn't just slower speed from coordination overhead. It's actually worse answers. The model loses consensus-building across knowledge areas.
And you can see this in a more concrete example from computer vision architectures. A model called HI-MoE used a two-stage router for instance-centric granularity in object detection. It had a scene-level router and an instance-level router to get fine-grained on small objects. The performance improved, but the complexity shot up. That's the trade in a nutshell: you can get more precision, but you're building a much more complex routing hierarchy to manage it.
Which brings us back to Daniel's TypeScript in a Python project example. A hyper-specialized 'Python' expert would be blind to that. So you'd need a router smart enough to see the prompt mentioning a 'tsconfig.json' file and activate a 'web tooling' or 'TypeScript' expert alongside the Python one. That's a high-level, cross-domain inference. We're asking the router to understand project context, not just token similarity.
That's where some of the most interesting experiments are happening. Model providers are exploring dynamic, context-aware routing. Anthropic, in their research blog, has discussed experimenting with dynamic expert activation that doesn't just look at a single token, but at the broader prompt context to make routing decisions. Early results suggest this can reduce inference time by up to thirty percent because you're making fewer, better-targeted routing decisions overall.
The frontier isn't necessarily more experts; it's smarter selection from the expert pool you have. The router's vision gets a wider field of view.
It's about moving from token-level routing to something more like task-level or session-level routing. If the model understands the first few exchanges are about debugging a Python data pipeline, it can pre-activate a constellation of relevant experts—Python, pandas, numpy, maybe SQL—and keep them active for the duration of that context, reducing the routing overhead for every single token.
That sounds like it would require a kind of meta-cognitive layer on top of the MoE. Something tracking conversation state and predicting expert needs. Is that feasible without blowing up the simple efficiency win?
It's the challenge. OpenAI's MoE implementations, compared to something like the older Switch Transformer, seem to be moving in this direction. They're treating the router not just as a static gate, but as a small model that can learn longer-range dependencies. The goal is to approximate that holistic understanding without ever activating all the parameters.
Let's talk hardware limits, because you can't have this conversation without it. Even if we had the perfect algorithm for micro-segmentation, what's the physical constraint?
GPU memory is the brutal one. Each expert, even a tiny one, needs to be loaded into VRAM to be available for activation. If you have ten thousand micro-experts, you can't have them all resident in memory at once. You'd be swapping weights in from system RAM, which would annihilate your latency gains. So there's a hard ceiling defined by available fast memory on the chip.
The dream of a model with a hyper-specialized expert for every possible subtask is a memory architecture problem as much as a routing problem.
Innovations like NVIDIA's Nemotron 3 Super with its 'LatentMoE' are working on this from the accuracy-per-FLOP angle, but the granularity details are still emerging. And research like 'CodeQuant' from ICLR is looking at low-precision quantization specifically for MoE models to cram more experts into memory. The hardware and algorithm design are becoming co-dependent. Here's a fun fact: the original Mixture of Experts idea actually dates back to the early 90s in classical machine learning. But it's only now, with massive GPU memory and trillion-token datasets, that we can scale it to this degree. We're seeing a thirty-year-old idea hit its stride because the hardware finally caught up.
It feels like we're circling a principle. The optimal granularity isn't a fixed number. It's a function of your available memory, your router's intelligence, and the expected breadth of your queries. A model designed for a single, narrow domain could afford much finer segmentation than a general-purpose chatbot.
That's the insight. For a coding-specific model, having experts for Python, TypeScript, code review, security scanning—that might work brilliantly. For a generalist model that needs to answer questions about poetry, physics, and tax law in the same session, those experts need to be broader, with more overlapping knowledge, because the router can't afford to miss context. The segmentation strategy is becoming a core part of a model's design identity—and that's where the practical considerations come in.
The practical takeaway for someone building on an MoE architecture today is to design experts around coherent domains, not microscopic tasks. Think 'programming languages' or 'scientific reasoning,' not 'list comprehensions.
Your router's discrimination ability is the limiting factor. A good rule of thumb is to ask: can a human, given just a token or a short phrase, reliably assign it to this expert versus another? If it's a coin flip, your segmentation is probably too fine. Aim for distinctions the router can actually learn.
For developers using these models, the implication is to structure your prompts to play to the router's strengths. If you're asking a coding question, lead with the primary language. That initial token heavily influences the routing decision. A prompt that starts 'In Python, how do I...' is more likely to cleanly activate the right expert block than one that buries the language context three sentences in.
That's an excellent, actionable point. Prompt engineering for MoE is about optimizing for the gatekeeper. The other strategy is batching similar queries if you're building an application. If you can group user requests by domain—all the coding questions, then all the writing feedback—you reduce the router's thrashing between very different expert sets, which can improve throughput.
It's about minimizing the router's cognitive load. Give it a clear signal and a consistent workload.
And if you're on the infrastructure side, monitoring is key. You want to track expert utilization. If you see one expert is constantly dormant or, conversely, a single expert is getting eighty percent of the traffic, your segmentation might be off. The load balancing should be dynamic, not wildly skewed. It's a diagnostic tool.
The hardware constraint you mentioned is non-negotiable. You can't just keep adding experts. So the strategy becomes making the experts you have more versatile through better training, not more numerous. Invest in the router's intelligence—maybe through techniques like that counterfactual routing—before you split an expert in two.
The frontier right now is in smarter routing, not just more routing choices. The most efficient path forward is to help the router see the bigger picture, so it can do more with the expert divisions we can physically fit in memory. That's where the next thirty percent efficiency gain will come from—but it does raise the question: how far can we push that intelligence?
If smarter routing is the next frontier, what's the ultimate limit of that intelligence? Can a router ever have a perfect, holistic understanding of what a prompt needs, or is some degree of expert overlap and redundancy an inevitable, even desirable, feature?
I think redundancy is a feature, not a bug. The goal isn't to eliminate all overlap. It's to manage it intelligently. The most interesting experiments I'm watching are in dynamic, hierarchical routing—systems where the router can activate a core expert for the main thread of a conversation, and then temporarily recruit peripheral experts for specific subtopics, almost like calling in specialists for a consultation. That's the vision: a model that can fluidly reconfigure its own brain on the fly.
The future isn't a static map of expert territories. It's a live, context-sensitive assembly of capability. The router becomes a conductor, not just a gatekeeper.
And that points to even greater efficiency down the line. If models can achieve that, we could see another step-change in what's possible for a given amount of compute. It makes the whole architecture more resilient and adaptable. That’s the potential we’re just starting to tap.
To bring it all back to Daniel's question: the approximate division in today's models is in the range of 128 to over a thousand experts, but they are broad. We could theoretically push segmentation much further, but we're limited by router discrimination, coordination overhead, and hardware memory. The providers exploring the limits are the usual research labs—Google, Anthropic, OpenAI, DeepSeek—and they're pushing on smarter, context-aware routing more than on sheer expert count.
The granularity is a lever, and the industry is still learning how hard to pull it. The answer is evolving with every new paper and model release.
A fittingly weird and wonderful place to end. Thanks to Daniel for the prompt that took us deep into the router's mind. And our thanks, as always, to our producer Hilbert Flumingtop for keeping the signal clear.
A quick thanks to Modal, whose serverless GPOs let us run the pipeline that makes this show possible. If you're building something that needs to scale intelligently, they're worth a look.
If you enjoyed this dive into the granular guts of AI architecture, the best thing you can do is leave a review wherever you listen. It genuinely helps others find the show.
This has been My Weird Prompts.
Until next time.
See you then.
