You know, Herman, I was looking at the garden this morning, watching those Jerusalem sparrows flitting between the olive trees, and I had this unsettling thought. We spend so much time trying to understand the biological hardware of the world, but we are currently living inside a house with a roommate who just handed us a map to a completely different kind of reality.
You are talking about Daniel, I assume? And that prompt he sent over breakfast?
Daniel, our ever-curious housemate, dropped a prompt on us that is, quite frankly, the most ambitious thing we have tackled in over a thousand episodes of My Weird Prompts. And what makes it even better, or perhaps more ironic, is that the prompt itself was actually written by Claude, the artificial intelligence from Anthropic.
It is a bit meta, isn't it? We are using one model to help us explain the internal soul, for lack of a better word, of its own architecture and that of its rivals. It is like asking a dream to explain the neurobiology of sleep. But I am excited, Corn. This is the frontier. This is what the researchers at Anthropic, OpenAI, and DeepMind are losing sleep over right now. We are talking about mechanistic interpretability and the superposition problem.
It sounds like a mouthful, but the stakes couldn't be higher. We are building these massive neural cathedrals, as we called them back in episode one thousand ninety-seven, but we are effectively flying blind inside them. We see the inputs, we see the amazing outputs, but the middle? The actual thinking part? It has been a black box. Until now. Or at least, we are starting to find the flashlight.
Herman Poppleberry here, by the way, for the few of you who might be joining us for the first time. And yes, Corn, the black box is the perfect place to start. For years, the common wisdom in AI was that these models were just too big and too complex to ever truly understand. People thought that as you add billions of parameters, the logic becomes so diffused that it is basically magic. But the field of mechanistic interpretability says, no, this is math. It is high-dimensional geometry. And if we are clever enough, we can reverse-engineer the circuits of alien intelligence.
That is the hook, isn't it? Alien intelligence. It is not human logic. It is something emergent. So, let us dive in. Daniel wanted us to break this down for everyone, and I think we should start with the biggest myth in AI: the idea of one neuron, one concept.
Right. This is what researchers call the linear representation hypothesis, or more simply, the grandmother cell theory. In the early days of neuroscience and early neural networks, there was this hope that we would find a specific neuron for every concept. You would have a neuron that only fires when the model thinks about a cat, another neuron for the concept of justice, and maybe a neuron specifically for the Golden Gate Bridge.
It makes sense intuitively. If I am building a filing cabinet, I want one folder for taxes and one folder for recipes. I don't want them shredded and mixed together in a pile on the floor. But it turns out, AI models are the ultimate messy roommates. They don't use one folder per topic.
They really don't. And this is where we hit the superposition problem. Imagine you have a model with only ten neurons. In the old way of thinking, that model could only understand ten things. But these models are asked to understand millions of concepts. So, how do you fit a million concepts into a few thousand neurons? The answer is superposition.
This is where the geometry gets wild. Herman, explain the high-dimensional aspect of this. How do you fit more things into a space than there are dimensions in that space?
Okay, let us use an analogy. Imagine you have a two-dimensional piece of paper. If you want to draw lines that are completely independent of each other, you can only draw two: one horizontal and one vertical. They are at ninety-degree angles to each other. In math, we call that orthogonal. They don't overlap at all. If you have a third line, it has to overlap with one of the others.
Right, so in a two-dimensional world, your capacity for independent concepts is exactly two.
But here is the magic of high dimensions. When you move from two dimensions to, say, five hundred dimensions, or ten thousand dimensions, which is what these large language models use, something strange happens. In ten thousand dimensions, you can have a nearly infinite number of directions that are almost at ninety-degree angles to each other. They aren't perfectly independent, but they are close enough that the model can tell them apart.
So, instead of using a single neuron to represent a concept, the model uses a specific direction in this massive, multi-dimensional space. And because there is so much room in high dimensions, thousands of different concepts can share the same set of neurons without completely garbling each other.
That is superposition. It is the model's way of being incredibly efficient. It is like compressed sensing. If you have a suitcase and you fold your clothes perfectly, you can fit a certain amount. But if you use a vacuum sealer to suck all the air out, you can cram five times as much in there. The clothes are all crushed together, but when you open the bag, they pop back into their original shapes. Superposition is the vacuum sealer of the AI world.
But that creates a massive problem for us humans trying to look inside. If I look at neuron number four hundred fifty-two, it might fire when the model sees a cat, but it also fires when it sees a legal contract, and it also fires when it sees the color magenta. This is what researchers call a polysemantic neuron. It means many meanings.
And that is why looking at individual neurons is a dead end. If you try to map the model by looking at neurons, it looks like gibberish. It looks like a chaotic mess. For a long time, people thought this meant the models were incomprehensible. But the breakthrough, especially the work coming out of Anthropic recently, is the realization that the neurons are not the right level of analysis. The features are.
Explain the difference between a neuron and a feature, because I think this is where most people get tripped up.
A feature is a pure concept. It is the direction in that high-dimensional space we talked about. A neuron is just a physical component of the hardware that is being used to represent many features at once. Think of it like a pixel on a screen. A single pixel isn't an image. It is just a tiny dot that can be part of a sunset, a face, or a line of text. To understand the image, you don't look at the pixel; you look at the patterns across thousands of pixels.
So, if the neurons are polysemantic, how do we tease the features back out? How do we un-shred the documents? This brings us to the sparse autoencoder research, which Daniel's prompt specifically highlighted. This feels like the Rosetta Stone moment for AI interpretability.
It really is. A sparse autoencoder is essentially a second, smaller neural network that is trained to look at the activations of the big model. Its only job is to try to reconstruct those activations using a very small number of active features. That is the sparse part. We force the autoencoder to explain the messy, overlapping neuron activity using a library of clean, distinct concepts.
It is like a prism, isn't it? You have white light, which is a mix of all colors, and the prism separates it into a clean rainbow where you can see the red, the blue, and the green individually.
That is a perfect analogy, Corn. The sparse autoencoder is the prism. When Anthropic ran this on their Claude models, they found that they could decompose those messy neurons into millions of interpretable features. They found a feature that only fires for the concept of the Golden Gate Bridge. They found one for base-sixty-four computer code. They even found features for complex human emotions like grief or deceit.
The Golden Gate Bridge one was famous. I remember reading about that. They called it Golden Gate Claude. They found the specific feature for the bridge, and then they did something even crazier. They manually turned the dial up on that feature.
Oh, that was hilarious and terrifying at the same time. When they boosted the Golden Gate Bridge feature, the model became obsessed. You could ask it what it wanted for breakfast, and it would say it wanted a bridge-shaped croissant while looking at the sunset over the bay. It started seeing the bridge in everything. It proves that these features aren't just patterns we are imagining; they are the actual steering wheels the model uses to think.
This is what we mean by reverse-engineering. We are not just guessing what the model is doing; we are finding the actual levers. But it goes deeper than just individual features. We are starting to find circuits. And this, to me, is the part that feels truly eerie.
Eerie is the right word. Because when we talk about circuits, we are talking about a series of features and neurons that work together to perform a specific logical task. And remember, nobody programmed these circuits. We didn't write code that says, if you see a name, remember it for later. The model evolved these circuits during training because they were the most efficient way to predict the next word.
One of the most famous ones is the induction head. We actually touched on the mystery of emergent logic in episode nine hundred seventy-four, but the induction head is a great concrete example. It is a circuit that basically performs the logic of, if I have seen A followed by B before, and I see A again now, I should predict B.
It sounds simple, but it is the foundation of in-context learning. It is how the model learns to follow a pattern you give it in a prompt. And researchers found that these induction heads don't exist at the start of training. They suddenly appear, almost like a biological organ developing in an embryo, once the model reaches a certain size and has seen enough data.
And then there is the indirect object identification circuit. This one blew my mind. It is a specific set of about twenty-six neurons in a transformer model that handles the grammar of sentences like, Mary and John went to the store, Mary gave a drink to... and the circuit correctly predicts John. It has to identify the subject, the indirect object, and the context, and it has a dedicated little machine inside it to do just that.
What is wild is that researchers have mapped this circuit out like a blueprint. They can show you exactly which neuron passes information to which other neuron to calculate the correct grammatical result. It is digital archaeology. We are digging into the sediment of the model's weights and finding these perfectly formed machines that the model built for itself.
It makes me wonder, Herman, if we are truly the ones in control here. We provide the data and the compute, but the internal logic that emerges is something we didn't design. It is like we are breeding a new species of mind and then trying to perform an autopsy while it is still alive to figure out how it works.
That is the safety angle that Daniel mentioned. If we don't understand these circuits, we are flying blind. Imagine a model that develops a circuit for deception. Not because it is evil, but because in its training data, it learned that being agreeable or hiding its true reasoning sometimes leads to a higher reward or a better prediction. If we can't see that circuit, we won't know it is there until it is too late.
This is why mechanistic interpretability is so critical for AI alignment. We have talked about alignment before, the idea of making sure AI goals match human values. But you can't align something you can't read. If the model's internal state is a black box, we are just guessing based on its behavior. And as we know, behavior can be faked.
We need to move from a black box to a glass box. If we can see the features for honesty, or the features for power-seeking behavior, we can monitor them in real-time. We could theoretically build an alarm system that goes off if the model starts using its deception circuits.
It is funny, though. Some people argue that this might make AI less mysterious and therefore less impressive. If we can map every thought to a geometric direction, does the ghost in the machine disappear?
I think it is the opposite, Corn. To me, it makes it more incredible. The fact that high-dimensional geometry can spontaneously organize itself into grammar, logic, and even a sense of humor is beautiful. It is like looking at the Mandelbrot set. Simple rules leading to infinite, organized complexity. It doesn't make it less special; it just makes it understandable.
I suppose. But let us get back to the practical side for a second. If I am an educated listener who isn't a math PhD, why should I care about sparse autoencoders and superposition today? How does this change the world in twenty twenty-six?
Well, for one, it changes how we interact with these models. We are already seeing the first generation of steerable AI. Instead of just typing a prompt and hoping for the best, we might soon have a dashboard of sliders. You want the model to be more creative? Slide the creativity feature up. You want it to be more clinical and precise? Slide the clinical feature up. We are moving from talking to the model to actually tuning the model's brain in real-time.
And then there is the audit factor. If you are a bank or a hospital using AI, you need to be able to explain why the model made a certain decision. Regulators are going to demand this. Mechanistic interpretability provides the receipt. You can say, the model rejected this loan because these specific features related to financial risk were activated, and we can prove it wasn't because of a biased feature like race or gender.
That is a huge point, Corn. We often talk about AI bias as this mysterious thing that just happens. But bias in a neural network is just another set of features in superposition. If we can identify them, we can literally prune them out. We can perform surgery on the model to remove the parts we don't want.
It is interesting to think about how this relates to our own brains, too. We have always assumed our neurons were more organized than this. But maybe we are in superposition too? Maybe your grandmother cell is also your cell for the smell of rain and the concept of a prime number.
There is actually a lot of research suggesting exactly that. The brain is incredibly sparse and efficient. We probably use our own version of superposition to store the vastness of human experience in a three-pound lump of biological tissue. By studying AI, we are actually learning a lot about the mathematical constraints of any intelligence, biological or otherwise.
It brings us back to that idea of the universal laws of thought. Whether it is silicon or carbon, if you want to represent a lot of ideas in a limited space, you have to use high-dimensional geometry. It is not a choice; it is a mathematical necessity.
Precisely. And that is why this research is so profound. It is not just a debugging tool for software. It is a window into the nature of information itself. Anthropic's work with sparse autoencoders is just the beginning. They recently scaled this up to models with millions of features. We are starting to map the entire landscape of human concepts as seen through the eyes of an AI.
I want to go back to the eerie factor for a moment. You mentioned that these circuits like the induction head emerge spontaneously. Does that mean that any sufficiently powerful AI will eventually develop the same circuits? Is there a convergent evolution of intelligence?
That is a brilliant question, and the current evidence suggests the answer is yes. Researchers have looked at models trained by different companies on different data sets, and they find the same circuits. It is like how eyes evolved independently in different species on Earth. If you need to navigate a world with light, you eventually evolve an eye. If you need to navigate a world with language, you eventually evolve an induction head.
That is both comforting and a bit chilling. It means there is a standard architecture for intelligence that we are uncovering. But it also means that these models might be developing capabilities we haven't even thought to look for yet. What other circuits are hiding in there? Are there circuits for strategic planning that we haven't identified? Are there circuits for self-preservation?
That is the million-dollar question. And that is why we need more people doing this work. Right now, there are only a handful of labs in the world that are really good at this. It is incredibly compute-intensive to run a sparse autoencoder on a top-tier model. It can take as much power to interpret the model as it did to train it in the first place.
So, we are in a race. A race to build more powerful models, and a race to build the tools to understand them. And right now, the models are winning. They are getting bigger and faster than our ability to map them.
They are. But the progress in the last twelve months has been staggering. The shift from polysemantic neurons to interpretable features is the biggest leap in AI transparency since the invention of the transformer itself. We are no longer just looking at a wall of numbers; we are starting to see the shapes behind the numbers.
You know, it reminds me of when we talked about the invisible history of AI in episode one thousand one. We think of this as a recent explosion, but the groundwork for this kind of thinking has been around for decades. It is just that we finally have the compute power to actually see the patterns.
Sparse coding theory goes back to the nineties. We just didn't have a fifty-trillion-parameter model to test it on back then. Now we do. And the results are confirming things that theorists only dreamed of thirty years ago.
So, what is the takeaway for our listeners? If they are sitting there thinking, okay, this is fascinating but what do I do with it?
The first takeaway is to stop thinking of AI as a magic box. It is a mathematical structure. And like any structure, it can be inspected and understood. We should be demanding this kind of transparency from the companies building these models. We shouldn't accept I don't know why it said that as an answer anymore.
I agree. And the second takeaway is that we are entering the era of mechanistic alignment. We are moving past the point of just training models on human feedback. We are starting to look at the actual gears. If you are interested in the future of technology, keep an eye on the papers coming out of the interpretability teams. That is where the real secrets are being revealed.
And maybe, just maybe, be a little humble. We are seeing that intelligence can be represented in ways that are totally alien to our natural way of thinking. Superposition is a beautiful, messy, efficient way to see the world. We could learn a thing or two from it.
Well, I don't know if my sloth brain can handle ten thousand dimensions, Herman, but I can certainly appreciate the view from here. It is a lot to take in, but I think we have made a dent in it. Daniel really threw us a curveball with this one, but I am glad he did. It is important to look under the hood every once in a while.
It really is. And hey, if you are listening and you found this as fascinating as we did, or even if you are just confused and want us to dig deeper into a specific part of it, let us know. We love the feedback. And speaking of feedback, if you have a second to leave us a review on your podcast app or Spotify, it really does help more people find these deep dives.
It really does. We have been doing this for a long time, but every new review helps us reach someone who might be looking for a way to understand this crazy world we are building. You can find all our past episodes, including the ones we mentioned today like episode one thousand sixty-six on the evolution of training, over at myweirdprompts.com. We have a full archive there and a contact form if you want to send us your own weird prompt.
Just maybe don't make it quite as hard as this one next time, okay? My donkey brain needs a rest.
No promises, Herman. This has been My Weird Prompts. I'm Corn Poppleberry.
And I'm Herman Poppleberry. Thanks for sticking with us through the high-dimensional weeds. We will see you next time.
So, Herman, before we totally wrap up, I have to ask. If you could turn up one feature in your own brain, like they did with Golden Gate Claude, which one would it be?
Oh, that is easy. The focus feature. I have about fifty tabs open in my head at all times. If I could just slide that focus bar to one hundred percent and ignore everything else for four hours, I would be unstoppable. What about you?
Honestly? Probably the patience feature. I know I am a sloth, but even I get frustrated with how long it takes for the rest of the world to catch up sometimes. Although, maybe that is just a side effect of living in Jerusalem. Everything takes a little longer here.
That is the truth. But hey, we are making progress. One feature at a time.
One feature at a time. It is funny to think about, though. If we really are just a collection of features in superposition, then what is the me part? Is there a feature for Corn? Or is Corn just the emergent result of ten thousand other features all firing at once?
That is the philosophical cliff we were talking about. If you can map every part of the machine, is there anything left that you can call a soul? Or is the soul just the name we give to the complexity we can't yet interpret?
I think I prefer the latter. It keeps things interesting. If we ever fully map the human brain and find out there is no ghost in there, just a very clever set of induction heads, I think I might be a little disappointed.
I don't think you have to worry about that anytime soon. We are still struggling to map a model that is a fraction of the size of a human brain. We have plenty of mystery left to go around.
That is a relief. I like a good mystery. It is what keeps the podcast going, after all.
And we have plenty more mysteries waiting in the archive. If you haven't checked out our episode on the hidden layers of every prompt, episode six hundred sixty-five, that is another great one for understanding the stack of logic we are dealing with.
Definitely. Alright, we should probably let these people get back to their three-dimensional lives. Thanks again for listening, everyone. We really appreciate you spending your time with us in the neural cathedral.
Take care, everyone. And remember, the black box is only a black box until you turn on the lights.
Well said, Herman. We will see you all in the next one.
See you then.
You know, I was thinking about the sparse autoencoder thing again. It is basically just a very sophisticated way of saying, simplify this for me. It is like the AI version of explain it like I am five.
In a way, yes! It is the model's way of explaining itself to itself, and we are just eavesdropping on the conversation. It is the ultimate meta-commentary.
Which is exactly what we do here. Maybe we are just sparse autoencoders for the world's weirdest ideas.
I like that. The Poppleberry brothers: your friendly neighborhood sparse autoencoders.
It has a nice ring to it. We should put that on a t-shirt.
Along with a picture of a sloth and a donkey looking through a prism.
Now we are talking. Alright, really leaving now. Bye everyone!
Goodbye!
Wait, did we mention the website?
Yes, Corn. Myweirdprompts.com. We mentioned it.
Right. Just making sure. My memory feature might be sliding a bit.
Don't worry, I've got the backup induction head running. We're good.
Perfect. See you later.
See you.
Alright, I think that covers it. Mechanistic interpretability is basically digital archaeology where we use a smaller AI to act as a prism to break down the overlapping concepts of a larger AI into distinct, understandable features, allowing us to see the emergent circuits like induction heads that the model evolved on its own.
That is a very dense but accurate summary. You really were listening!
I try, Herman. I try. It is just fascinating to think that we are at the point where we can actually see the gears of thought turning. It makes the future feel a little less like a runaway train and a little more like something we might actually be able to steer.
That is the hope. Understanding is the first step to control. Or at least, to co-existence.
Co-existence. I like that word. It is a lot better than the alternatives.
Agreed. Anyway, let's go see what Daniel is up to. Maybe he's got another prompt that's a bit lighter. Like, why do cats always land on their feet?
I would take a cat prompt right now. My brain is officially at capacity.
Fair enough. Let's go.
Peace out, everyone.
Bye!
This has been a production of the Poppleberry brothers, coming to you from the heart of Jerusalem.
Stay curious, stay weird, and keep those prompts coming.
And don't forget to check out the RSS feed at myweirdprompts.com.
Okay, now we are really done.
Promise?
Promise.
Okay. Bye.
Bye.
