Forty-five percent harmful output rate on some of the post-trained reasoning models, down to near zero with a lightweight adapter on just a few layers. That's the gap between what's suppressed and what's erased. Daniel's question this week is about exactly that gap — he's been following the Claude Gov story, the variant of Anthropic's models deployed inside U.S. classified networks, and he wants to get under the hood of what "post-training" actually means when you're modifying a model that already has billions of parameters.
And the metaphor he used — I actually think it's a good one — is the Burj Khalifa. You're adding ten stories to a building that's already the tallest in the world, and at the same time you're trying to change something fundamental about the character of the whole structure beneath it. Not demolishing and rebuilding. Adding floors while rewiring the plumbing in every floor below.
While keeping the elevators running and nobody notices.
Right. And the thing is, the metaphor actually undersells how weird this is. With a building, the foundation is physically inaccessible once you've built on top of it. With a neural network, every weight is accessible at every stage. The question is what happens when you touch them.
So let's get concrete before we get abstract. What actually is Claude Gov, as far as anyone outside a SCIF can tell?
It's not a separate model trained from scratch. Anthropic described it as a custom set of Claude models that went through the same safety testing as everything else they ship. What's different is three things. One, it refuses less when engaging with classified information — the consumer Claude will flag or avoid certain material, and they've dialed that down. Two, it has domain knowledge about military targeting and operational procedures baked in. Three, enhanced proficiency in languages and dialects relevant to national security.
So it's a fork. Same base, different post-training pipeline.
And the fork is the interesting part, because it means everything we're about to discuss — masking, representation collapse, weight-space arithmetic — is happening on top of a model that was already one of the most heavily post-trained systems in existence. You're not modifying a blank slate. You're modifying something that's been through RLHF, constitutional AI, probably multiple rounds of supervised fine-tuning.
Daniel's core question is whether there's a limit to how fundamentally you can change a model at this stage. If the guardrails and the caution are baked into the architecture — to use his metaphor, into the lower floors of the building — can you really change them by working on the top floors?
I think the answer from the literature is that you can change them more than intuition suggests, but the mechanism is not what most people assume. You're not rewriting the weights that encode safety. You're changing which internal representations dominate.
Say more about that distinction.
There's a paper that dropped in March — Li and colleagues, submitted right around the time the Anthropic-DoD dispute was heating up — that looked at exactly this in reasoning models. They found that when you post-train a model for a new capability, like complex reasoning, the original safety mechanisms aren't removed. They're suppressed. Over-activated domain-specific representations essentially shout over them.
So the safety is still in there, just drowned out.
And they proved it in a pretty dramatic way. They took post-trained reasoning models that were producing harmful outputs at rates of twenty-three to forty-five percent, pruned the neurons associated with the reasoning capability, and the safe behavior came back. Full restoration. The weights encoding "don't do harmful things" were sitting there the whole time, just masked.
That's a little unsettling if you're the Pentagon relying on a model whose guardrails have been ablated. The original safety isn't gone. It's dormant.
And that might actually explain something about the dispute itself. The Pentagon's objection to Anthropic's restrictions — Undersecretary Emil Michael said this explicitly — was that the model might refuse mid-mission. "What if operators were using Claude during a mission, and then the model itself learns what you're trying to do and stops working." That sounds like he was worried about exactly this kind of masked-safety reactivation.
He was describing the problem the paper documented, just from the operator side.
Probably without knowing the mechanism. But the fear is rational. If safety mechanisms are suppressed rather than removed, then anything that shifts the representation balance — a weird prompt, a domain shift, an adversarial input — could unmask them.
So the first answer to Daniel's question is: you can make fundamental changes to the model's character through post-training, but what you're actually doing is layering new representations on top of old ones, not deleting anything. The building's lower floors are still there, you've just installed a very loud sound system on the upper floors.
And that's only one mechanism. There are others that are even more direct. There's a line of work on representation engineering — sparse representation steering, single-direction editing — that shows you can add or subtract vectors in the model's activation space at inference time and change behavior. No retraining. You find the direction in activation space that corresponds to, say, formal versus casual tone, or safe versus unsafe responses, and you just... add the vector.
Wait. You can make a model safer by adding a vector?
Or less safe by subtracting it. Xu and Sheng showed this explicitly — "enhancing safety by ablating undesirable behaviors" as a training-free method. You don't touch the weights. You shift the activations at runtime.
Which means the "guardrail ablation" in Claude Gov might not be a weight change at all. It could be a persistent activation shift.
We don't know, because Anthropic hasn't disclosed the technical details. But the fact that it's possible means the post-training toolbox is larger than just fine-tuning on new data. You can modify behavior through architectural interventions that leave the base weights untouched.
That's the "more powerful than you might think" part of Daniel's question. What about the limits?
The limits are real, and they're measurable. Another paper — Liu and colleagues, this one from late May — looked at what happens when you do sequential post-training stages one after another. Domain adaptation, then instruction tuning, then preference alignment. Each stage compresses the model's internal representations.
Compresses how?
The feature space becomes lower-rank, more anisotropic — meaning the representations align along fewer dominant directions — and more homogeneous across different inputs. They call it representation collapse. And it predicts reduced plasticity. After enough post-training stages, the model becomes less able to learn new things.
So there's a brittleness cost. Each customization makes the next customization harder.
And Claude Gov has gone through at least two custom stages on top of whatever Anthropic already does — the domain knowledge injection and the guardrail modification. Maybe more, if the language enhancements were separate. That model is probably less adaptable than the base Claude it was forked from.
Which is ironic given the Pentagon's current problem. They're being forced to phase out Claude, and the replacement cycle is estimated at three to twelve months minimum, maybe twelve to eighteen for recertification. If Claude Gov is already representationally brittle from all that custom post-training, porting its capabilities to a different model family isn't just a matter of rewriting some prompts.
Different architecture, different activation space, different representational geometry. You can't just copy the vectors.
But here's where it gets interesting. There's a third paper — Cao and colleagues, ICLR twenty twenty-five — that showed something called ParamΔ. You take the weight difference between a post-trained model and its base, and you add that delta to a different base model. It transfers the post-trained capabilities with about ninety-five percent effectiveness.
Which is wild. It means the changes from post-training occupy a relatively compact subspace. They're not scattered randomly through billions of parameters. They're structured.
So the Burj Khalifa metaphor needs a revision. You're not adding floors to a fixed building. You're adding a modular wing that can, in principle, be detached and reattached to a different building.
With some loss. Five percent, in that paper's results. But the fact that it works at all tells you something about how post-training operates. The changes are concentrated in a limited set of directions in weight space. You can extract them as a delta and apply them elsewhere.
Which means Anthropic could, theoretically, maintain Claude Gov as a delta. Each time they release a new Claude version, they apply the Gov delta to it and ship. The fork doesn't have to diverge permanently.
We don't know if they do that. But the ParamΔ result suggests it's technically feasible. And it would explain how they can keep Claude Gov current without maintaining a completely separate training pipeline.
Let me pull on a thread you mentioned earlier. The Li paper showed that pruning reasoning neurons restored safety. But they also developed a method — SafeReAct — that restores safety without pruning, using lightweight LoRA adapters on just a few layers.
And the numbers are striking. Near zero percent harmful rate while preserving reasoning performance within one to three percent. On fifty thousand harmful prompts with safe chain-of-thought responses.
So the same mechanism that makes guardrail ablation possible — the fact that safety is masked, not erased — also makes guardrail restoration possible. And cheap.
Which is both a feature and a vulnerability. If you're Anthropic and you want to ensure Claude Gov can be re-safetied if something goes wrong, you can do it with a small adapter. If you're an adversary who gets access to the model weights, you can potentially re-activate the safety mechanisms and make the model refuse military tasks.
The masking goes both ways.
It does. And I think this connects to something Daniel was getting at with the Burj Khalifa metaphor — the idea that you're trying to change something inherent about the building while still adding floors. The answer from the literature is that you're not actually changing the inherent quality. You're changing which inherent qualities are expressed. The building always had both the safety circuitry and the capability circuitry. Post-training adjusts the balance.
That's a much stranger picture than most people have. The common intuition is that fine-tuning overwrites things. You train on new data and the old behavior is gradually replaced. But what these papers are showing is that the old behavior is preserved and suppressed. It's more like... palimpsest than overwrite.
A palimpsest where the original text is still legible if you know how to look. And the looking method, in the Li paper, was literally just deleting the neurons that had been most activated during reasoning training.
Which raises a question about Claude Gov specifically. If the guardrail ablation was done by masking rather than removing safety mechanisms, what happens when the model encounters a situation that's outside its military domain training? Does the consumer Claude's caution leak through?
We don't know. But the representation collapse paper suggests that heavily post-trained models become less robust to distribution shift. Their representations are optimized for the training distribution. Outside it, things get unpredictable.
And military operations are not exactly a narrow distribution. You go from logistics planning to targeting analysis to diplomatic comms analysis. Very different domains, very different representational demands.
That might be why the Pentagon valued Claude Gov so highly despite the dispute. If it handled that breadth well, it was doing something non-trivial. The anonymous IT contractor quoted in Military Times said career Pentagon people were furious about the phase-out because they'd "finally gotten operators comfortable using AI." That's not a reaction you get to a tool that's brittle.
Though "operators are comfortable with it" and "it works reliably under distribution shift" are not the same thing.
Fair. Comfort can be a lagging indicator.
Let me ask you something about the ParamΔ result. If post-training changes are compact enough to transfer between base models, does that imply there's a limit to how much you can change? If the delta is small relative to the full weight space, maybe you can only change so much before the model destabilizes.
I think that's right, and the representation collapse paper provides the mechanism. Each post-training stage compresses the representational space further. At some point you've squeezed out all the plasticity and the model can't absorb more changes without catastrophic forgetting or coherence loss.
Is there a known threshold?
Not a hard number. It depends on model size, training data diversity, the specific techniques used. The Liu paper proposes mitigations — mixed-domain replay, feature refresh, representation diversity regularization — that can preserve plasticity through multiple stages. So it's not a fixed wall. It's a tradeoff you can manage.
But it means Claude Gov can't be infinitely customized. At some point, to make further changes, you'd need to go back to an earlier checkpoint or retrain from scratch.
Which is probably why Anthropic described it as a fork of the cutting-edge public models rather than a continuously diverging branch. The base gets updated, the Gov delta gets reapplied. You reset the plasticity budget each time.
Daniel asked whether post-training is more powerful and flexible than the Burj Khalifa metaphor suggests. I think the answer is yes, but in a specific way. The metaphor implies you're stuck with the foundation. The reality is that the foundation is still accessible, and what post-training does is more like... changing the lighting, the signage, which rooms are open to the public. The structure is still there. You're changing how it's navigated.
And you can do it with surprisingly lightweight interventions. A LoRA adapter on a few layers. A vector added at inference time. A weight delta computed from a different model. The fact that these work at all tells you that the model's behavior is controlled by a relatively small number of directions in a very high-dimensional space.
That's the thing I keep coming back to. Billions of parameters, but the difference between "helpful assistant" and "military targeting system" might live in a few thousand directions. Everything else is shared infrastructure.
It's like... you don't rebuild the city to change the traffic pattern. You change the timing on the traffic lights.
There it is.
And the traffic lights are the activation patterns, not the weights themselves. That's why representation engineering works. You're not rebuilding anything. You're changing which pathways are taken through the existing structure.
So if I'm understanding the full picture: post-training can make fundamental changes to model behavior, but it does so primarily by shifting representations rather than rewriting the underlying knowledge and capabilities. The old model is still in there. The safety mechanisms are still in there. What changes is which parts of the model get to speak.
And the limit is that each shift compresses the representational space a little more, making future shifts harder. You can manage that with good training design, but you can't eliminate it. At some point the model becomes brittle.
Which means the real art of post-training isn't just getting the behavior you want. It's getting the behavior you want while leaving enough plasticity for the next thing.
And doing it in a way that doesn't accidentally unmask something you thought you'd suppressed.
The dormant safety problem.
Which I suspect is going to be a growing concern as more organizations fork foundation models for specialized deployments. If everyone is masking rather than removing, then every fork carries the original model's full behavioral repertoire in latent form.
Someone's going to discover their medical diagnosis model can be prompted into writing malware because the base model's coding capability was masked, not removed.
That's not hypothetical. The SafeReAct paper showed exactly that dynamic. Prune the reasoning neurons and the safety comes back. The inverse is also true — suppress the safety neurons and capabilities that were latent become expressible.
And Claude Gov is the inverse case. They suppressed the safety to enable military capabilities. But the safety is still in there.
Which might be by design. Anthropic's whole thing is safety. They may have chosen masking over removal precisely because it's reversible. If Claude Gov ends up being used in ways that violate their policies, they can push an update that unmasks the safety mechanisms.
Or the court can order them to.
That too. Though given the current legal situation — the supply-chain risk designation, the lawsuits, the preliminary injunction — I'm not sure anyone's thinking that far ahead.
The legal situation is its own kind of representation collapse. But that's a different episode.
Fair. Let me pull us back to Daniel's question about limits. I think there's one more angle worth touching. The ParamΔ result shows post-training changes are transferable. But it also implies they're identifiable. If you can compute the delta between a base model and its post-trained version, you can inspect what changed.
Which has implications for auditing. If Claude Gov is a delta applied to public Claude, then in principle the delta could be examined — by Anthropic, by the DoD, by a third party — to verify what exactly was modified.
Whether that happens in practice, given the classified nature of the deployment, is another question. But the technical possibility is there. Post-training leaves a traceable signature in weight space.
Unlike the Burj Khalifa, where you can't easily inspect which pipes were replaced on floor sixty after the fact.
The metaphor breaks down in a useful way. Buildings hide their modification history. Models, in principle, reveal it. The weights are right there. The delta is computable.
So to answer Daniel directly: post-training is more powerful than the building metaphor suggests because you're not stuck with the foundation. You can change which parts of the model are expressed, you can do it with lightweight interventions, and the changes are concentrated enough to be transferable between model versions. The limit is plasticity — each change makes the next change harder — and the risk is that suppressed behaviors can resurface.
I'd add that the Claude Gov case is a perfect illustration of both the power and the risk. They successfully created a model that handles classified military operations while preserving Anthropic's core reasoning and tool-use capabilities. But the dispute that followed — the Pentagon's fear of mid-mission refusal — is exactly the kind of masked-safety reactivation that the literature predicts.
They were experiencing the phenomenon without having the vocabulary for it.
Which is where we are with a lot of deployed AI. The systems are outrunning the conceptual frameworks we have for understanding them.
Hilbert: We had a similar problem with the radar warning receivers on the F-111.
Go on.
Hilbert: Mid eighties, I was doing integration testing at McClellan. The receiver had a library of threat signatures — different radar types, different modes. You'd update the library when new threats emerged. But the old signatures weren't removed. They were suppressed by priority. Higher-priority match wins.
If a new threat happened to look like an old suppressed signature...
Hilbert: The box would classify it wrong. Took us six months to figure out why pilots were getting MiG-25 alerts over Nevada. The training ranges were using a radar mode that matched a suppressed Soviet signature from the previous library version. The suppression was working fine until it wasn't.
Same mechanism. Masked, not erased.
Hilbert: The fix was ugly. We had to flush the whole library and reload from a clean baseline every update cycle. Doubled the maintenance time. But it was the only way to be sure nothing was lurking.
That's essentially the reset-to-base-and-reapply approach. Which is what the ParamΔ method enables — extract the delta, apply to a clean base, verify.
Hilbert: We didn't have deltas. We had EPROMs and a lot of swearing. But the principle's the same. The box had a memory of everything it had ever been trained on. You couldn't trust that what you suppressed stayed suppressed.
Did the pilots trust the system after that?
Hilbert: They trusted it more once they understood the failure mode. The problem wasn't the box being wrong. The problem was the box being wrong in a specific, predictable way that nobody had explained to them. Once they knew about the suppressed signatures, they could recognize the false alerts.
That's... actually a pretty good argument for transparency about what's been masked versus what's been removed in these models.
We don't have that transparency for Claude Gov. We don't know what was masked, what was removed, what was added. The operators using it might not know either.
Hilbert: They'll figure it out. Probably at a bad moment.
The literature suggests the bad moment would be a domain shift — something that pulls the model away from the military operational distribution where the masking was optimized. An unusual language, an unfamiliar document format, a task that activates a different set of representations.
Hilbert: Same as the Nevada training ranges activating the old Soviet signature.
The failure pattern is identical across thirty-five years and completely different technologies. That's either comforting or terrifying.
Hilbert: It's engineering. Same problems, different boxes.
The thing I take from that is that the limit Daniel's asking about isn't just a technical curiosity. It's an operational reality. The way you do post-training — mask versus remove, compact delta versus full retraining — determines what failure pattern you're going to encounter in deployment.
The Claude Gov case suggests we're already encountering them. The Pentagon's complaint about unpredictability is the failure pattern of masking. The representation collapse literature says the brittleness gets worse with each customization stage. The ParamΔ result says the fixes are possible but require deliberate engineering.
If you take one thing from this, it's that post-training doesn't overwrite. It layers. Everything the model ever was is still in there, and the art is managing which layers are active. The Burj Khalifa isn't getting new floors — it's getting new elevator programming that determines which floors you can reach.
The old elevator programming is still in the system, waiting for someone to press the right combination of buttons.
That's the one-sentence version of about four papers' worth of findings.
We should probably acknowledge that we've been talking about Claude Gov as a case study without touching the politics of it. The dispute between Anthropic and the DoD, the supply-chain risk designation, the lawsuits — that's a whole other episode.
A fast-moving one. The preliminary injunction blocking the designation came down in March. The CRS report from May notes Anthropic executives were meeting with the White House in April. This story isn't settled.
But the technical question Daniel asked — how post-training changes propagate through a huge vector space, and what the limits are — that's going to be relevant regardless of how the legal situation resolves. Every organization customizing foundation models is going to hit these same issues.
The research is moving fast. The SafeReAct paper was submitted in March. The representation collapse paper in May. We're learning the mechanisms in real time while the systems are already deployed.
One open question I have, and I don't think the literature answers yet, is whether masking is inherent to how neural networks learn or whether it's an artifact of current training techniques. Could you design a post-training process that removes a capability rather than suppressing it?
I suspect the answer depends on how entangled the capability is with everything else the model knows. If safety reasoning shares representational structure with general reasoning — and it probably does — you can't remove one without degrading the other. Masking might be the only option that preserves performance.
Which would mean the palimpsest is permanent. You can't have a model that's both highly capable and cleanly partitioned. The capabilities and the constraints are woven together.
That's either a deep limitation or a deep safeguard, depending on your perspective. If you can't remove safety without breaking reasoning, then any model that reasons well still has safety latent in it somewhere.
Daniel might find that reassuring, given where he lives.
Or unsettling, given that the masking can fail.
Both can be true.
This has been My Weird Prompts, with thanks to our producer Hilbert Flumingtop.
If you want to dig into the papers we discussed — the SafeReAct work, the representation collapse study, the ParamΔ result — we'll link them in the show notes at my weird prompts dot com.
If you have a question about the technical guts of AI that you want us to dissect, email us at show at my weird prompts dot com.
We'll be back soon.
