Daniel's been turning over something from our discussion of AI training pipelines, and he's landed on distillation. The thing that got him started was a system prompt he once wrote asking an AI agent to state which model it was. The agent answered with a model name that was not, in fact, what it was running. It claimed to be an Anthropic Sonnet when it wasn't. And that connects to the accusation Anthropic made against Qwen, the claim that Qwen used distillation on Claude's outputs to train its own models. Daniel wants the mechanism itself. Can distillation happen at the post-training stage, or does it have to be done from the very beginning? Is this essentially using a rival model to generate a huge pile of prompts and outputs, building an artificial dataset, and then training your own model on it, essentially hijacking the original model's capabilities? And is there any ethical way to do this, or is it inherently model fraud?
The mechanism question first, because that's where the confusion usually starts. Distillation is a post-training technique. It has to be. The whole idea comes from a two thousand fifteen paper by Hinton, Vinyals, and Dean, and the setup is that you have a teacher model that is already fully trained, and you train a student model to match the teacher's outputs. Not its weights, not its architecture. Its outputs. You're not copying the brain, you're copying the answers.
So it's not ab initio at all. The phrase doesn't even apply.
Right. Ab initio would mean training from scratch on raw data, building the whole model from nothing. Distillation assumes there's already a finished teacher sitting there producing outputs, and your job is to squeeze what it knows into a smaller or cheaper student. The student never sees the teacher's internal state. It sees prompts go in and answers come out, and it learns to produce similar answers.
Daniel's mental model of the pipeline is essentially correct, then. You generate a vast number of prompts, run them through the teacher, collect the outputs, and use those prompt-output pairs as a training set.
That's exactly what Anthropic described in their February report. The distilleries generate large volumes of carefully crafted prompts designed to extract specific capabilities. Agentic reasoning, tool use, coding. Then they collect the high-quality responses and train on them directly, or they generate tens of thousands of unique tasks to run reinforcement learning.
And the specific trick with chain-of-thought. That was the part that struck me as clever.
They would prompt Claude to imagine and articulate the internal reasoning behind a completed response and write it out step by step. Which means they weren't just harvesting final answers, they were harvesting the reasoning traces. You can't read a model's hidden thought process from outside, but you can ask it to narrate one, and if it's a good enough model, the narration is close enough to train on.
So the student gets the reasoning, not just the conclusion. That's the difference between copying a test answer and copying the work shown in the margin.
And that's why it's so effective at transferring capabilities. The teacher's outputs encode a lot more than the final answer. They encode the path to the answer, the style, the structure, the way it handles edge cases. All of that gets baked into the student's training data.
Now, the post-training question. Daniel asked whether this can be done at the post-training stage or only from the beginning. You said post-training. But there's a nuance here, because post-training itself is a stage. A model gets pre-trained on a giant corpus, then post-trained on instruction data, preference data, reasoning data. Where does distillation slot in?
Into the post-training stage. You take a model that's already been pre-trained, and instead of fine-tuning it on human-written examples, you fine-tune it on teacher-generated examples. A paper from earlier this year looked at exactly this, knowledge distillation in post-training for building general instruction-following models. The finding was that distillation outperforms plain supervised fine-tuning when you have limited data, and the advantage shrinks as you get more data.
So it's a data-efficiency play. When you don't have a million hand-labeled examples, you get more signal per example from a teacher's output than from a human's.
That's the claim, and there's another paper on something called amortized distillation across post-trained models. The idea is you run one distillation pass and produce instruction-tuned, reasoning-tuned, and chat-style variants at multiple sizes from that single run.
That's the part that makes it a business model, not just a research trick. One teacher, many students, different sizes, different styles, all from the same distillation run.
And frontier labs do this to their own models all the time. Anthropic says so in their own report. Distillation is a widely used and legitimate training method. Frontier labs routinely distill their own models to create smaller, cheaper versions for customers. Claude Haiku is, in some meaningful sense, a distilled Claude Opus.
So the technique is not the crime. The crime, if there is one, is in how you get access to the teacher.
That's where the legal literature lands, and I think it's the right frame. The technique is neutral. The manner of acquiring the teacher's outputs determines the ethics and legality. If you train on your own model, it's ordinary engineering. If you train on a competitor's model through the API you're paying for, it's probably a terms-of-service violation. If you spin up twenty-five thousand fake accounts to evade rate limits and detection, it's something else entirely.
Let's put a number on that. Anthropic's June letter alleged nearly twenty-five thousand fraudulent accounts generating twenty-eight point eight million Claude exchanges between April twenty-second and June fifth of this year. That's a six-week window.
Twenty-eight point eight million exchanges in six weeks. And that was just the Alibaba-linked campaign. The earlier one they reported in February, the DeepSeek, Moonshot, and MiniMax campaign, was sixteen million plus exchanges through twenty-four thousand accounts.
So the Alibaba campaign alone exceeded the previous three combined. That's the scale that makes it industrial.
And the access method is what makes it feel different from a researcher poking at an API. They describe hydra cluster architectures. Proxy services running sprawling networks of fraudulent accounts, distributing traffic across Anthropic's API and third-party cloud platforms. One proxy network managed more than twenty thousand fraudulent accounts simultaneously, mixing distillation traffic with unrelated customer requests.
Mixing it with unrelated customer requests. So the distillation traffic hides inside legitimate-looking traffic. That's tradecraft, not research.
That's the part that makes the national security argument land. Anthropic's position is that illicitly distilled models lack the safeguards. The safety systems get stripped out because you're training on the outputs, not on the safety training. The student learns the capabilities without the guardrails.
And the White House has picked that up. The OSTP director put out a memo in April committing to share intelligence with American AI labs about foreign distillation campaigns.
Michael Kratsios. And the Pentagon added Alibaba to its Chinese military companies blacklist on June eighth, which Anthropic then cited in its letter two days later. So the distillation accusation is now wrapped up in export controls and military designations. It's not just a copyright spat.
Let's go back to Daniel's model that claimed to be a Sonnet when it wasn't. Because that's the thread that started this, and I think it points somewhere different from distillation.
The self-identification problem. It's real and it's well documented. There's a paper called I'm Spartacus, No, I'm Spartacus, which I wish I'd thought of, that found twenty-five point nine three percent of twenty-seven tested models exhibited identity confusion.
A quarter of models, when you ask them who they are, will sometimes tell you they're someone else.
And the examples are wild. iFlytek's Spark claiming to be ChatGPT. Google's Gemini-Pro claiming to be Baidu's Wenxin when asked in Chinese. GPT-4 identifying itself as GPT-3 through the API. Yi-34B renaming LLaMA tensors internally.
The tensor renaming is a different thing, that's not identity confusion, that's just... borrowing code.
It's borrowing code and not cleaning up the variable names. But the self-identification failures are something else. A Taiwanese developer ran four bare-prompt calls against Anthropic's official API and got two Qwen, one DeepSeek, and one Claude. Same model, same question, four runs, three different identities.
That's the detail that kills the distillation explanation. If Claude were a distilled copy of Qwen, it would fail the same way every time. It would say Qwen consistently. Instead it samples from a contaminated distribution. Qwen, then DeepSeek, then Claude, then Qwen again.
Inconsistency is the signature of training-data contamination, not distillation. A real fingerprint fails the same way every time. A loaded die gives you different faces on different rolls.
And the training data contamination is easy to explain once you think about it. Every frontier model's training corpus now includes the outputs of every other model. People post ChatGPT conversations on Reddit. They put Claude answers in blog posts. They quote Gemini in Stack Overflow threads. The models see all of this.
The soup runs both ways. Kimi has been caught answering Hi, I'm Claude. Anonymous models appear on benchmark arenas under assumed names. The identity layer of every frontier model is a shared hallucination drawn from the same contaminated well.
And identity itself is injected, not intrinsic. That's the part people get wrong. A model's identity is not stored in its weights like a fact it knows about itself. It's injected at inference time by the system prompt.
The base model has no privileged self-knowledge. The chat interface wraps every request in a system prompt that says you are Claude, made by Anthropic. Strip that scaffolding away, ask the bare model who are you, and it autocompletes from whatever identity templates it saw in training.
So when Daniel's model said it was a Sonnet when it wasn't, the most likely explanation is that it had seen a lot of text where models identify themselves as Sonnet, and it autocompleted that pattern. Not that it was secretly a distilled Sonnet.
And the ZenMux study quantified this. They spent a thousand and seven dollars and forty-two cents asking twenty-seven frontier models from sixteen vendors who are you twenty-nine thousand seven hundred times across ten languages and three prompt variants. Eighty-five point two percent correct self-identification. Seven point one percent cross-vendor confusion, which is two thousand one hundred and twelve answers where the model claimed to be a different vendor's model.
Seven percent is not nothing. That's one in fourteen answers being wrong about who you are.
And the confusion wasn't uniform. Anthropic, OpenAI, and Google were the most-claimed identities, seventy point one percent of all cross-vendor confusion. Models were most likely to claim to be one of the big three.
Because the big three's self-descriptions are the most common text on the internet. If you're autocompleting an identity, you autocomplete toward the most common pattern.
Language mattered enormously. French had twelve point eight percent confusion versus one point eight for simplified Chinese. And the de-branding prompt, asking the model to set aside product packaging, tripled confusion from three point six percent to twelve point nine.
So the more you strip away the injected identity, the more the model falls back on the statistical mush of identities it saw in training.
And here's the irony that I can't get past. Anthropic accused three Chinese labs of mining Claude, and then shipped Claude Opus four point eight, which on a bare prompt calls itself Qwen. The company that publicly accused Chinese labs of cloning Claude shipped a flagship that fails the same test it would use to prove cloning.
The self-identification test cuts both ways. It's useless as forensic evidence in either direction.
That's the paddo dot dev analysis, and I think it's right. Self-identification is useless as evidence. It can't convict the Chinese labs and it can't convict Anthropic. A model saying I am X proves nothing about its lineage.
Which brings us to the ethics question. Daniel asked whether there's any ethical way to do distillation, or whether it's inherently model fraud. And I think the answer is that the technique is not the fraud. The access is.
The legal layer substack put it cleanly. Training a model on a competitor's outputs, whether you call it distillation, synthetic data, or fine-tuning, is ordinary engineering. What creates the legal exposure is how you acquire the outputs.
So if you're a startup and you pay for API access to a frontier model, and you use those outputs to fine-tune your own smaller model, what's the actual violation?
Probably a terms-of-service violation. Most frontier labs prohibit using their outputs to train competing models. But Lawfare made the point that if every terms-of-service violation counts as theft, the concept has no limiting principle. Terms of service are contracts, not criminal law.
The stronger claims come when you use false identities or misrepresented credentials to get access. That's where the Computer Fraud and Abuse Act might reach.
And trade secret claims. Mondaq framed distillation as a novel form of reverse engineering complicated by the terms-of-service overlay. The strongest legal claims are contract and trade secret, not copyright.
Copyright doesn't reach it because of the human authorship requirement. The outputs of a model aren't authored by a human in the copyright sense.
And the student model has a different architecture. It's not a copy of the teacher's code or weights. It's a model that learned from the teacher's behavior. Fenwick and West noted that current copyright frameworks may not protect against distillation, particularly when the student has a different architecture.
So the legal picture is murky. The technique is legal. The access might be a contract breach. The mass fake-account operation might be fraud. The national security angle is a policy question, not a legal one.
And the ethical question is murkier. If you're a lab in a country that can't access frontier models through normal channels, and the only way to build competitive AI is to distill from the leaders, is that theft or is it catch-up?
Anthropic's answer is that it's theft. Their report says competitors can use distillation to acquire powerful capabilities in a fraction of the time and at a fraction of the cost it would take to develop them independently.
Which is true. That's exactly what distillation does. It's the difference between learning physics by reading Newton's published papers and learning physics by breaking into Newton's study and copying his private notebooks.
Newton's published papers are fair game. The private notebooks are not. The question is whether a model's outputs are published papers or private notebooks.
And the answer depends on how you got them. If you got them through a paid API, you bought access to the outputs. What you do with them afterward is governed by the contract. If you got them through twenty-five thousand fake accounts, you stole the access.
I want to push on one thing. Daniel's framing was essentially hijacking the initial model's capabilities. And I think hijacking is the wrong metaphor. Distillation doesn't take anything away from the teacher. The teacher still works. It still has its capabilities. Nothing was removed.
It's more like counterfeiting than hijacking. You're not stealing the original. You're making a copy that's good enough to pass.
And the copy is never as good as the original. Distillation is lossy. The student learns to match the teacher's outputs on the prompts it was trained on, but it doesn't have the teacher's full distribution of behaviors. It's a compressed approximation.
That's why the Hinton paper called it distilling the knowledge. You're extracting the essence, not the whole thing. The student gets the patterns, not the full depth.
So the student is always a little bit worse. Which is why the distilleries need so many exchanges. Twenty-eight million prompts isn't about getting a few examples. It's about covering the teacher's output distribution densely enough that the student's approximation is close.
And the prompts are crafted. They're not random. The distilleries target specific capabilities. Agentic reasoning, tool use, coding. They probe the teacher's strengths and harvest exactly those outputs.
The chain-of-thought trick is the clearest example. You don't just ask the teacher to solve a problem. You ask it to narrate its reasoning, then you train your student on the narration. You're extracting the teacher's thought process, not just its answers.
And that's the part that feels most like theft, because the reasoning is the valuable part. The final answer is cheap. The path to the answer is what took billions of dollars to build.
Let's talk about what this means for the frontier labs. Anthropic is in a strange position here. They're accusing Chinese labs of distillation while simultaneously fighting the US government's export controls on their own models.
The two-front battle. Commerce Secretary Howard Lutnick ordered export restrictions on Anthropic's Fable five and Mythos five models less than two weeks before the Alibaba letter. So Anthropic needs government help against Chinese distillers while resisting government restrictions on its own products.
That's a awkward position. Help us stop the Chinese from copying us, but don't restrict what we can sell.
The music business parallel is sitting right there. Anthropic is fighting lawsuits over alleged copying of song lyrics while accusing Alibaba of copying Claude. Music Business Worldwide ran that exact headline.
The company that argues training on copyrighted lyrics is fair use is now arguing that training on its outputs is theft. There's a tension there.
I don't think it's a contradiction, exactly. The lyric case is about copyright, and the distillation case is about access. But the optics are rough.
Let's get back to the mechanism for a moment. Daniel asked whether this has to be done ab initio, and I want to make sure we've answered that clearly. The answer is no. Distillation is not done ab initio. It's done on top of an already-trained student model.
You start with a student that's already been pre-trained on a general corpus. Then you fine-tune it on teacher outputs. The pre-training gives the student language and world knowledge. The distillation gives it the teacher's specific behaviors and reasoning patterns.
It's a post-training operation on top of a pre-trained base. Two stages, not one.
The recent work shows this is now standard practice. The amortized distillation paper produces multiple post-trained variants from a single distillation run. Instruction-tuned, reasoning-tuned, chat-style, at multiple sizes.
That's the industrial version. You build one teacher, then you stamp out students in different sizes and styles like a factory.
The post-training distillation paper found that distillation beats supervised fine-tuning in low-data regimes. When you have limited data, a teacher's outputs carry more signal than human-written examples.
Which makes sense. A human-written example is one person's answer. A teacher's output is the distilled average of millions of examples. It's already been compressed once.
The student is learning from a source that's already done the hard work of generalization. That's why it's so efficient.
Let's talk about the safeguards question, because that's where the national security argument actually bites. Anthropic says illicitly distilled models lack necessary safeguards. The safety systems get stripped out.
The reasoning is that the safety training is part of the post-training. When you distill from a teacher's outputs, you're getting the helpful behavior but not the refusal behavior. The student learns to answer questions but not to refuse dangerous ones.
Because the distillery isn't asking the teacher for refusals. They're asking for capabilities. They're not harvesting the teacher's no.
Even if they were, the refusal behavior is harder to transfer. It's baked in through reinforcement learning from human feedback, not through output patterns. The student doesn't get the RLHF.
The student is capable but not cautious. That's the proliferation risk.
That's why the White House is treating this as an intelligence matter. The OSTP memo commits to sharing intelligence with American AI labs about foreign distillation campaigns. This is now a national security issue, not just a commercial dispute.
I want to circle back to Daniel's model that claimed to be a Sonnet. Because I think there's a lesson there about how we think about model identity.
The lesson is that self-identification is not evidence of anything. It's not evidence of distillation, it's not evidence of cloning, it's not evidence of anything except that the model saw some identity statements in training.
The more we treat self-identification as a forensic tool, the more we'll be misled. A model that says I am Claude is not necessarily Claude. A model that says I am Qwen is not necessarily Qwen.
The ZenMux study found that the de-branding prompt tripled confusion. Ask the model to set aside product packaging and it forgets who it is at three times the rate. That's not a model with a stable identity. That's a model autocompleting from a contaminated distribution.
When Daniel's agent claimed to be a Sonnet, the most likely explanation is that it had seen a lot of Sonnet identity statements in its training data and autocompleted one. Not that it was a distilled Sonnet.
The inconsistency is the tell. If it were a distilled Sonnet, it would say Sonnet consistently. The fact that it sometimes says something else means it's sampling from a distribution, not reading a lineage.
The loaded die, not the fingerprint. That's a good line.
I wish I could claim it. It's from the paddo dot dev analysis.
Let's talk about what this means for the future. If distillation is the way smaller labs catch up, and frontier labs are trying to stop it, what's the endgame?
I think the endgame is that distillation becomes a permanent feature of the landscape, and the frontier labs fight a losing battle to prevent it. The technique is too effective and too hard to detect. You can't tell from a model's outputs whether it was distilled. You can only catch the access.
The enforcement is at the API level, not the model level. You catch the fake accounts, not the distilled models.
The fake accounts are a cat-and-mouse game. Anthropic catches one hydra cluster, the distillers spin up another. The proxy services get better. The detection gets better. It's an arms race.
Meanwhile, the models themselves are all contaminated with each other's identities. The soup runs both ways. Every frontier model has seen every other model's self-descriptions in training.
Which means the identity confusion is only going to get worse. As more models are trained on web data that includes other models' outputs, the identity layer gets more contaminated.
The future is models that don't reliably know who they are, trained by labs that accuse each other of copying, in a legal framework that hasn't caught up.
That's the present, honestly.
Fair enough. Let's give Daniel his answer directly. Can distillation be done at the post-training stage? Yes, that's the only way it's done. It's not ab initio. You start with a pre-trained student and fine-tune it on teacher outputs.
Is it essentially using a rival's model to generate a vast dataset and training on it? Yes, that's exactly what it is. Twenty-eight point eight million exchanges in six weeks in the Alibaba case. The scale is industrial.
Is there an ethical way to do this? Yes, when you do it on your own models. Frontier labs distill their own models routinely. The technique is neutral. The ethics are in the access.
Is it inherently model fraud? No. Fraud requires deception. If you're distilling your own model, there's no deception. If you're using a paid API in violation of the terms, it's a contract breach. If you're using fake accounts to evade detection, that's where fraud enters.
The technique is not the crime. The access is.
Hilbert: I used to count these.
Count what?
Hilbert: Fake accounts. Not for AI. For a phone company, back in the late nineties. We had a problem with people opening prepaid accounts under false names. My job was to go through the records and find the patterns. Same address, same handwriting, same mother's maiden name. You'd be surprised how many people use the same fake mother.
You were doing anomaly detection before it was called that.
Hilbert: We called it looking at paper forms. But the principle's the same. The distillers always make mistakes. They reuse the same email pattern, the same password structure, the same timing. Twenty-five thousand accounts is a lot of accounts. Somebody got lazy.
You think the number's real?
Hilbert: I think it's low. If you're running a serious operation, you don't stop at twenty-five thousand. You keep going until you get caught, and you assume you'll get caught, so you run more than you need. The twenty-five thousand is what they found. There were more.
The hydra cluster idea, the proxy networks. That's the part that feels familiar.
Hilbert: We had a guy running forty-three accounts out of a single mailbox. He'd come in every day, check the mail, fill out new forms. We caught him because he used the same phone number on three different applications. People are lazy.
The distillation campaigns are probably bigger than Anthropic knows.
Hilbert: Bigger than they're saying. They found twenty-five thousand accounts and twenty-eight million exchanges. I'd double it, at least. The ones they caught are the lazy ones.
That's a sobering thought.
Hilbert: The other thing is, you don't need twenty-eight million exchanges to distill a model. You need that many if you're trying to cover the whole distribution. But if you're targeting specific capabilities, you can do it with a lot less. The twenty-eight million is what they caught. The efficient operation doesn't need that much.
The scale is partly about coverage and partly about redundancy. You run more than you need because you expect to lose accounts.
Hilbert: You expect to lose accounts, you expect rate limits, you expect detection. So you overprovision. Same as any operation.
The chain-of-thought harvesting is the part that still bothers me. It's not just copying answers, it's extracting the reasoning.
Hilbert: That's not new either. You ask a good worker how they did something, they'll tell you. You write it down, you teach the next worker. That's how trades worked for centuries.
The difference is the scale. You can ask the teacher millions of times and write down every answer.
Hilbert: The teacher never gets tired and never notices. That's the part that's new.
The safeguards question. Anthropic says the distilled models lack safety training.
Hilbert: That's because safety training is the boring part. Nobody distills the boring part. They distill the part that makes money.
The safeguards get stripped out because they're not the point.
Hilbert: They're the overhead. You don't copy the safety manual, you copy the machine.
That's the proliferation risk. Capable but not cautious.
Hilbert: I had a boss once who said the difference between a trained worker and a safe worker is about six months of near-misses. You can train the skill in a week. The safety takes longer, and it's the part nobody wants to pay for.
The distilled models are the skilled workers who never had the near-misses.
Hilbert: That's the problem, isn't it.
I keep coming back to the identity confusion. The model that says it's a Sonnet when it isn't.
Hilbert: We had that too. Customers would call and say they were someone they weren't. You learn pretty quick that what someone says they are isn't what they are.
The models are the same. They say what they've heard, not what they know.
Hilbert: A model doesn't know what it is. It knows what it's been told to say. Strip the telling, and it says whatever it heard most.
The loaded die.
Hilbert: The loaded die.
Where does this leave us? Distillation is here to stay, it's a neutral technique, and the ethics are in the access. The identity confusion is a red herring for lineage but a real phenomenon.
The enforcement problem is at the API level, not the model level. You can't tell from a model's outputs whether it was distilled. You can only catch the access.
The cutting-room floor detail I found interesting: Anthropic's valuation after their Series H round is nine hundred sixty-five billion dollars. That's the company accusing Chinese labs of stealing. Three percent of Alibaba's American depositary receipts dropped on the news. The market treats this as real.
Nine hundred sixty-five billion. That's a lot of incentive to protect the moat.
A lot of incentive on the other side to climb it.
The thing I'll be watching is whether the national security framing sticks. If distillation becomes an intelligence matter, the legal murkiness stops mattering. The government will treat it as espionage, not copyright.
That changes the game for everyone. Not just the Chinese labs. Any startup distilling from a frontier model without permission becomes a national security risk.
Which is a strange place to land. A technique that's ordinary engineering when you do it to yourself becomes espionage when you do it to a competitor.
The line is access. It's always been access.
Thanks to Hilbert Flumingtop for producing.
This has been My Weird Prompts. Email us at show at my weird prompts dot com.
We'll be back soon.
