Daniel's been staring into the abyss of a safetensors file and wants to know what he's actually looking at. His question: an AI model is just a giant matrix of numbers, weights and biases, sitting there on Hugging Face. You can download it, you can point a tensor visualizer at it, but you're still just seeing numbers or abstract blobs. He remembers someone claiming the entire numeric sequence of GPT-1 fit inside a single printed book, and he figures if you tried that today the paper would stack to the moon. So his real question is whether the internal logic of a trained model, once it's in this number format, is inherently indecipherable and uneditable in any conventional sense. Not fine-tuning, not incremental checkpointing — literally understanding the shape of the vector space and editing it to delete or add knowledge. And can it be visualized in a way that would make sense to a human, maybe to see what domains a mixture of experts model has learned or how knowledge flows through it?
That's a great set of questions. And the GPT-1 book thing is real, by the way. GPT-1 had about a hundred and seventeen million parameters. If you printed every weight and bias as a floating-point number at, say, ten numbers per line, sixty lines per page, you'd fill something like... I want to say around five hundred pages. A big book, but a book. GPT-4 is estimated north of one point seven trillion parameters. That same format would need roughly seven million pages. Stacked up, that's about half a mile of paper. Not quite the moon, but you'd need a very tall ladder.
Half a mile of numbers you cannot read.
Right. And that's the core of it. Let's start with what these numbers actually are, because the word "weight" makes it sound like a dial you can turn, and that's not wrong exactly, but it's deeply misleading about how.
So what are they, then?
Weights and biases are the parameters of a neural network, learned through gradient descent. During training, the model sees examples, makes predictions, computes an error, and nudges every weight a tiny amount in the direction that reduces that error. Repeat billions of times. By the end, the numbers encode everything the model knows. But here's the thing — no single weight means anything on its own. A weight is just one coefficient in a vast chain of matrix multiplications. Its meaning only exists in relation to every other weight around it.
So it's not a book, it's a... what, a recipe where every ingredient amount depends on every other ingredient amount, and if you change the salt by half a teaspoon the whole dish becomes a bicycle?
Honestly, that's closer than most analogies. The technical term is that features are distributed. A concept like "the model knows about cats" isn't stored in one neuron labeled "cat." It's spread across thousands or millions of weights, tangled up with concepts about fur and ears and four-legged things and small mammals and pets and the word "feline" and a hundred other associations. This is why the raw weight space is indecipherable in practice. You can open the file. The numbers are right there. They're deterministic. They're inspectable. And they tell you nothing.
But that distributed thing — I want to sit with that for a second, because it's so counterintuitive. If I open a database, I find a row that says "cat" and it's got fields for whiskers and purring. The information has an address. You're saying in a neural network, the concept of "cat" doesn't have an address at all.
It has millions of addresses, and none of them are useful on their own. Think of it like a hologram. If you cut a photograph in half, you lose half the image. If you cut a holographic plate in half, you still get the whole image, just at lower resolution. The information is stored in the interference pattern across the entire surface. Neural networks are similar — the knowledge is in the pattern of relationships between weights, not in any individual weight. That's also why you can prune away huge chunks of a model, sometimes thirty or forty percent of the weights, and it still works. The pattern is redundant. It's encoded everywhere.
So it's less like a filing cabinet and more like... a flock of birds where the shape emerges from thousands of individual birds following local rules, and no single bird knows the shape.
That's a beautiful analogy, actually. And just like a flock, you can watch the shape move but you can't point to the bird that's responsible for the left wing.
But people are trying to read them anyway.
They are, and the last couple of years have been genuinely exciting. The big breakthrough came from Anthropic in twenty twenty-four with something called dictionary learning. They used sparse autoencoders to extract what they call "features" from the model's internal activations.
Walk me through that. Sparse autoencoder, features — what are we actually doing here?
Okay. When a model processes input, each layer produces activations — patterns of firing across its neurons. The problem is that individual neurons are polysemantic, meaning one neuron fires for multiple completely unrelated concepts. The same neuron might light up for "cat," "the color blue," and "the concept of Tuesday."
Tuesday.
It's weird. What Anthropic did was train a second, smaller network — the sparse autoencoder — to watch those activation patterns and decompose them into a much larger set of directions, where each direction corresponds to something more like a single coherent concept. They found millions of these features. Things like "the Golden Gate Bridge," "deception," "inner monologue," "sycophancy." These aren't individual neurons. They're directions in activation space — specific combinations of many neurons firing at specific relative strengths.
So the feature isn't a pixel, it's an angle.
And once you've identified that direction, you can actually turn it up or down. Anthropic demonstrated this with a feature they called "Golden Gate Bridge." They amplified it and the model started talking about the Golden Gate Bridge in completely unrelated contexts. They asked it about the weather and it said something like "the weather is cloudy with a chance of Golden Gate Bridge."
Which is the funniest possible outcome of serious research.
It's also deeply informative. It tells us the structure is there. The vector space has shape. Concepts exist as directions, not as points, and they're recoverable. But — and this is the big but — dictionary learning works on activations, not on weights directly. It's mapping the model's behavior as it runs, not reading the static number file. The weights themselves remain a scrambled encoding of these directions.
So you can watch the model think, but you can't read its brain off the disk.
Right. And that brings us to the second part of Daniel's question: can you edit the weights to add or delete knowledge?
Because that's the fantasy, isn't it? Open the file, find the row for "bad thing the model knows," set it to zero, close the file.
And it's a fantasy that breaks immediately. Let's say you want to remove knowledge of how to make a dangerous substance. That knowledge isn't in one weight. It's distributed across the entire network, entangled with benign chemistry knowledge, with general reasoning about procedures, with language patterns. If you zero out the weights most associated with it, you don't remove the knowledge — you just damage the model. It might still know the dangerous thing but now it can't reason about chemistry at all, or it produces garbled text, or it crashes entirely on certain inputs.
What about the more surgical approaches? I've seen papers about knowledge editing.
There's a technique called ROME, Rank-One Model Editing, that came out a few years ago. The idea is clever. They use causal tracing to identify which specific layers and which specific weights are most responsible for storing a particular fact. Then they apply a targeted rank-one update to modify just those weights to change the stored fact. And it works, in a limited way. You can change "the Eiffel Tower is in Paris" to "the Eiffel Tower is in Rome" and the model will reliably answer Rome when asked.
For how long?
That's the problem. You change one fact and the model seems fine. You change ten facts and performance starts degrading in ways that are hard to predict. You change a hundred and the whole thing can fall apart. The edits don't compose well. Each edit assumes the rest of the network is static, but the network is a single coherent mathematical object. Tweaking one part creates subtle inconsistencies that accumulate.
It's like editing a novel by finding every occurrence of a character's name and changing it, but you miss the pronouns and the references and the thematic echoes, and by chapter three the story makes no sense.
Except worse, because in a novel you can at least search for the name. In a neural network, you can't even identify all the places a fact lives. Causal tracing gives you the top few layers where the fact is most strongly represented, but the fact has echoes everywhere. Every weight that was adjusted during training to accommodate that knowledge is still carrying a little piece of it.
So what you're saying is that training isn't just writing information into the model — it's weaving it into the fabric of everything else the model knows. And once it's woven in, you can't pull out one thread without unraveling the whole sweater.
And the weaving happens because gradient descent updates every weight simultaneously. When the model learns that Paris is the capital of France, it doesn't just store that as an isolated fact. It integrates it with everything else it knows about France, about geography, about how capital cities relate to countries, about the linguistic patterns around the word "capital." All those connections get reinforced or adjusted together. The fact becomes load-bearing.
So the core training process is the only robust way to change knowledge.
For now, yes. Fine-tuning and continued pretraining adjust the entire network coherently because they use the same loss signal and gradient descent that created the model in the first place. Every weight gets to adapt to every change. That's why it works and surgical editing doesn't — the network is a system, not a database.
What about the visualization side of Daniel's question? Can we actually look at this thing and see something meaningful?
We can see patterns, but we can't see meaning. There are tensor visualizers that show you the weight matrices as heatmaps or point clouds. You can see clusters, you can see structure — certain layers have weights that form recognizable patterns, like stripe formations in the early layers of vision models that correspond to edge detectors. But for a language model, the patterns are far more abstract. You're looking at a projection of a thousand-dimensional space onto a two-dimensional screen. It's like trying to understand a city by looking at its shadow.
A shadow that changes shape depending on where you shine the light.
The projection loses almost all the information. And here's the thing about high-dimensional spaces that makes this especially cruel — the curse of dimensionality means that most of the volume of a high-dimensional sphere is near its surface. Your intuitions from three-dimensional space completely break down. Two points that look close in a two-dimensional projection might be enormously far apart in the full space. You can't trust anything you're seeing.
So the visualization is almost actively misleading.
It can be, if you don't know what you're looking at. What's more useful is what Anthropic did with their feature dashboards — showing what individual features respond to, giving you a kind of visual dictionary of the model's concepts. But that's still looking at activations, not weights. It's watching the model run, not reading its source code.
And for mixture of experts models, where different parts of the model specialize in different domains?
This is actually one area where visualization does give us something real. In a mixture of experts model, the router decides which expert or experts to activate for each input token. You can visualize those routing decisions. You can see that expert three fires mostly on code, expert seven fires on mathematical notation, expert twelve fires on French text. There are papers showing these routing patterns and they're informative — you can watch the model delegate work in real time.
But the internal logic of each expert is still opaque.
Completely. You know expert three handles code, but you don't know how it represents a for-loop versus a while-loop, or what it understands about variable scoping. The expert itself is a smaller neural network with its own millions of weights, and all the same opacity problems apply. It's like knowing that a particular person in a company handles accounting — you know their function, but you have no idea what's actually happening in their brain when they're looking at a spreadsheet.
So we've got a situation where the numbers are deterministic and inspectable, but their meaning is distributed and opaque. We can map features in the activations but not read the weights. We can edit facts in a fragile, limited way but not reliably add or delete knowledge. And we can visualize routing patterns and activation clusters but not the actual logic. What does this mean for actually trusting these models?
It's the central tension of AI safety right now. We're deploying models into the world at massive scale, and we cannot look inside them and see what they know or how they reason. We can probe them with inputs and observe outputs, but that's behavioral testing, not understanding. If a model has learned something dangerous during training — a bias, a capability we didn't intend, a deceptive tendency — we might never see it in testing but it's there in the weights, latent.
And the interpretability research is trying to close that gap.
Anthropic's scaling work on dictionary learning is promising. They've shown that as they scale up the sparse autoencoders, they find more and more granular features. The long-term vision, and I should say this is still aspirational, is something like "model biology" — being able to examine a trained model the way a biologist examines an organism, identifying structures and functions and interactions.
But we're at the stage where we've just invented the microscope and we're staring at a cell going, "there's... stuff in there."
We're a little further than that. We can identify some of the stuff. We can point to specific features and say "this one handles sycophancy" or "this one tracks whether the model is in a code context." But we're a long way from a complete map. And the uncomfortable truth is that we may never get one. The vector space might be fundamentally too high-dimensional and too entangled for human comprehension. We might always be looking at projections and partial views.
Which means we might always be relying on behavioral testing for safety, knowing that behavioral testing can miss things.
Yes. And that's not a comfortable place to be, given where these models are being deployed.
Before we wrap up, Hilbert has a story that might make this all too real.
Hilbert: I typed in a million numbers once.
Sorry?
Hilbert: Nineteen ninety-three. I was working data entry for a weather prediction outfit in Boulder. They had a mainframe model — not AI, just a big numerical simulation — and someone had to type the initial condition parameters in by hand. That was me. Eight hours a day, sitting at a terminal, keying in temperature and pressure and wind speed readings from weather stations across the western United States.
A million numbers?
Hilbert: Roughly. Over the course of a summer. The supervisor, man named Gresham, he'd come by and say "those numbers know the weather." He couldn't explain how. Nobody could, really. The model ran, it produced forecasts, they were decent. But if you asked anyone what any individual number did, you'd get a blank stare.
So you were the human tokenizer.
Hilbert: I was the human something. One day I transposed two digits in a pressure reading for Salt Lake City. Should have been nine-eight-seven millibars, I typed nine-seven-eight. Tiny error. Model ran that night and produced a forecast showing a hurricane hitting Denver.
Denver doesn't get hurricanes.
Hilbert: It doesn't. Gresham traced it back to my entry. I was fired the next morning. But the thing that stuck with me — and I'm hearing it in what you two are saying — is that one wrong number out of a million broke the whole forecast. Not just a little wrong. Hurricane-in-Denver wrong. These things are sensitive in ways you can't predict until you break them.
You're the reason we don't edit weights by hand.
Hilbert: I'm the reason I don't, anyway. Gresham kept the printout of that forecast pinned to his wall for years. Called it his favorite artifact. Said it reminded him that the numbers were doing something real even if nobody understood what.
Did you ever find out what that specific pressure reading controlled?
Hilbert: Nobody knew. Probably fed into some differential equation six steps downstream and amplified. I looked at the code once, years later. Fortran. Thousands of lines. The variable names were things like X-seventeen and Q-bar. Nobody was meant to read it.
A million numbers, a million variables named X-seventeen, and one wrong digit summons a hurricane.
Hilbert: That's about the size of it.
That's a good reminder that these numbers are fragile — and that brings us to the bigger question. If we can't fully read or edit the weight space, can we ever truly trust these models?
I think the honest answer is that trust has to be earned through behavior over time, not through inspection of internals. We're building systems whose internal logic we cannot fully audit, and we're deploying them anyway. That's not unprecedented — we do it with pharmaceuticals, where we understand the mechanism imperfectly but validate through clinical trials. But the difference is that a drug's mechanism doesn't evolve. A model's behavior can shift with the input distribution, can surface latent capabilities, can fail in ways we didn't test for.
A drug doesn't learn from everyone who takes it.
Right. A model in the wild is constantly seeing new inputs, and while it's not updating its weights unless someone fine-tunes it, its behavior can still shift based on the distribution of prompts it receives. Prompting patterns can surface capabilities that were latent in the weights all along. The static file is deceptive — the same weights can produce wildly different behaviors depending on how you query them.
The interpretability work becomes a kind of safety net. Even a partial map is better than no map.
The field is moving fast. The dictionary learning results from twenty twenty-four were a real step change. The fact that we can now identify and manipulate individual conceptual features — even imperfectly — was science fiction five years ago. If that trajectory continues, we might get to something like model biology sooner than we think.
The numbers are not a book to be read, they're a landscape to be mapped. And we're only just learning to draw the map.
With a very blurry pencil.
The misconception I keep running into is that each weight or neuron corresponds to a specific concept — "this is the cat neuron." Reality is messier. Features are distributed across thousands of weights, and any given neuron fires for multiple unrelated things. You can't point to one number and say what it means.
The second one is that you can just edit a weight to delete knowledge. Naive edits break the model, and even the advanced techniques are fragile. The only robust way to change what a model knows is to train it.
I think there's a third misconception worth naming. The idea that if we just had a good enough visualization, it would all make sense. That the problem is the display, not the thing being displayed. But the truth might be that human brains aren't built to comprehend thousand-dimensional vector spaces. We might be staring at something that is fundamentally beyond our intuitive grasp, and the best we can do is build tools that translate it into terms we can work with, even if we can never truly see the whole thing.
That's humbling, and I think it's probably right. The model knows things in a way that we cannot directly understand. We built it, we trained it, the numbers are ours — but the meaning that emerges from them might always require a translator.
We'll be back soon. Thanks to our producer Hilbert Flumingtop, who has typed a million numbers and only summoned one hurricane.
That's a better record than most weather models, honestly. This has been My Weird Prompts. You can find us at my weird prompts dot com, and if you've got a question about the inside of a neural network or anything else, email the show at show at my weird prompts dot com.
See you tomorrow.
