Daniel's been digging through the training-data iceberg again. He says Common Crawl and The Pile get all the attention, but he's convinced they're just the visible tip of something much deeper, smaller, more selective sets built for specific purposes. He's got three questions. First, are these datasets only good for English models, or can they form a foundation for any language? Second, when a lab ships a point release like GPT-6, how do you do an incremental diff on something like Common Crawl, update the model's picture of the world without breaking what it already knows? Third, those domain-specific models, medical diagnosis and the like, are they fine-tuned from generalists, or does anyone actually train them from scratch on specialized data and skip the giant corpus entirely?
Three good questions, and they all point at the same blind spot. We talk about models like they're baked once and then served forever, but the data pipeline underneath is moving the whole time. The famous datasets are just the bulk flour in the pantry. The recipe is the interesting part.
So let's map the pantry before we start cooking. What's actually out there beyond the giants?
The giants are Common Crawl, which is a nonprofit that crawls the web monthly and dumps it as open data, and The Pile, which was an eight hundred gigabyte curated mix assembled by EleutherAI back in twenty twenty. Those two get cited constantly because they're open, they're huge, and they were used in high-profile models. But underneath them there's a long tail. You've got ROOTS, which is the one point six terabyte multilingual dataset built for the BLOOM model. You've got CulturaX, which pulls from mC4 and OSCAR and covers a hundred and sixty seven languages. RedPajama, which was an open reproduction of the Llama training set. Then narrower things, PubMed scrapes, clinical note corpora, code datasets like The Stack. The tail goes on for a while.
And the tail is where the interesting behavior lives. The bulk data gives the model grammar and broad world knowledge. The selective sets are what shape what it's actually good at.
Right. Think of it as the difference between reading every newspaper ever printed and reading a shelf of medical journals. The newspapers make you a fluent generalist. The journals make you useful in a clinic.
Which brings us to Daniel's first question. Can these datasets speak anything other than English?
Common Crawl is roughly half English. That sounds like a lot of the world's languages are represented, but the distribution is brutal. English is maybe twenty percent of global internet users, but it's half the crawl. And that's before you get into the quality gap. The English content in Common Crawl is dense, well-formed, diverse. The Swahili content is a fraction of a percent, and a lot of it is machine-translated boilerplate or navigation text. So the raw material is skewed.
But the raw material isn't the whole story. You can build multilingual datasets on purpose.
You can, and people have. ROOTS was assembled for BLOOM specifically to break the English default. It covers fifty nine languages, and the team behind it spent enormous effort on data quality per language, because the problem isn't just volume, it's representation. If you dump a hundred languages into a model in proportion to how much text exists online, the model learns English and a few big European languages well, and everything else is noise. So they had to deliberately oversample the low-resource languages, which is technically tricky because you need enough text to oversample from.
And then you hit the curse of multilinguality.
That's the term, and it's a real phenomenon. As you add more languages to a single model, performance on each individual language tends to degrade, even for the high-resource ones. The model's capacity is finite, and every language competes for the same parameters. There are ways to mitigate it, careful data balancing, language-specific tokenization strategies, but the basic tension is there. A model that speaks a hundred languages is usually worse at English than a model that only speaks English with the same number of parameters.
Which explains why BLOOM was a landmark and also a cautionary tale.
BLOOM was trained from scratch on ROOTS, a hundred and seventy six billion parameters, released in twenty twenty two. It proved you could build a multilingual foundation model from open data. But it also showed the cost. The model was good at a lot of things, but it wasn't the best at any one thing, and the compute bill was enormous. Nobody's in a hurry to repeat the experiment at that scale. Most labs look at BLOOM and say, that was important, we're not doing that again.
The economics keep pointing back to English. It's abundant, it's cheap, it's high quality. Every other language is a tax.
And it's a tax that gets paid unevenly. Danish, to pick a language with strong institutional support, has enough well-formed text to build a decent model. But a language like, say, Oromo, with millions of speakers and very little digital footprint, you're scraping whatever you can find and hoping it's not all machine-translated scripture.
So the answer to Daniel's first question is yes, open datasets can form a foundation for non-English models, but the quality and representation vary wildly, and most of the time the economics push you back toward English as the default.
The other piece worth naming is tokenization. English text tokenizes efficiently because the vocabulary is small and the orthography is simple. A multilingual model needs a tokenizer that handles dozens of scripts, and the tokenizer itself can introduce bias. If your tokenizer splits a word into more tokens in one language than another, the model has to spend more compute on that language for the same content, which effectively taxes it twice.
So the linguistic question isn't just about data volume. It's about the whole pipeline, tokenizer included.
And that's before you even get to the question of evaluation. How do you know if your model is good at Oromo? You need benchmarks in Oromo, which require fluent speakers to build, which costs money, which nobody wants to spend. So the data scarcity compounds into an evaluation scarcity, and the language stays underserved.
Now let's move to the second question, because this is where I think Daniel's intuition is sharpest. A point release like GPT-6, the lab says it has updated world knowledge. How do you actually do that? You can't just diff the old crawl against the new crawl and patch the model.
You can't, and the reason is the model doesn't store knowledge the way a database does. There's no clean delta you can apply. What labs actually do is called continual pre-training, or CPT. You take the existing checkpoint and you train it further on a fresh, curated mix of data, recent web crawls, news, papers, code, whatever's relevant. But you don't feed it only the new stuff.
Why not? Wouldn't that be the efficient move?
Because of catastrophic forgetting. If you train a model only on new data, it overwrites the old knowledge. The model's parameters shift to accommodate the new distribution, and the old patterns get erased. So CPT mixes the new data with a replay of old data, typically at a ratio that heavily favors the old. You might do ten or twenty percent new data, eighty or ninety percent replay. The exact ratios are a tuning problem.
And this is where the learning rate scheduling gets delicate.
Very. You run the continued training at a lower learning rate than the original pre-training, because you're nudging the model, not rebuilding it. Too aggressive and you destabilize the whole thing. Too gentle and the model doesn't actually absorb the new information. The labs tune this carefully, and they often do multiple rounds of CPT with different data mixes depending on what they're trying to improve.
So when Llama three point one or Qwen two point five ship, what's actually happening under the hood?
Those are good examples. Llama three point one was a CPT run on top of the Llama three base, with a mix that included more code and math and multilingual data. Qwen two point five did something similar, extended context length, more domain-specific corpora. The point release is not a new model from scratch. It's the old model given a course of continuing education.
And for world knowledge specifically, the pipeline is more like a data refresh. You crawl the web again, you filter for quality and recency, you mix it in at the right ratio, and you hope the model's sense of now shifts without its sense of everything else collapsing.
The thing people miss is that this is a moving target. The world changes while you're training. By the time your crawl finishes, the news is already old. By the time the model ships, it's older. So the model is always behind, and the question is how far behind and whether it matters for the use case.
Temporal drift. The model knows the world as of some fuzzy date, and the fuzziness isn't uniform. It might know the current president of France but not the current prime minister of Japan, because the data refresh caught one and missed the other.
And you can't easily tell from the outside which parts of the model's world knowledge are current and which are stale. The model doesn't timestamp its own knowledge. It just answers with whatever confidence it has, and the confidence is often wrong.
Which is a nice segue into the third question, because domain models face the same staleness problem in a different way. Daniel asks whether medical diagnosis models are fine-tuned from generalists or trained from scratch on specialized data.
The overwhelming answer is fine-tuned. Med-PaLM 2, which is Google's medical model, is PaLM fine-tuned on medical data. It inherits the generalist's reasoning and world knowledge, then gets specialized instruction tuning and domain data on top. That's the pattern for almost everything in specialized domains. Law, finance, biology, you name it. Fine-tune a generalist.
But there are the ab initio examples. BioGPT.
BioGPT was trained from scratch on PubMed abstracts, released in twenty twenty two. GatorTron did the same on eighty two billion clinical tokens from electronic health records. These are real models, and they're interesting, but they're the exceptions. And the reason they're exceptions is instructive.
What happens when you train only on PubMed?
You get a model that's very good at the surface patterns of biomedical literature, but it lacks the general reasoning and world knowledge that a generalist brings. It can tell you that a certain drug interacts with a certain pathway, but it can't reason about why a patient might not tolerate the drug, because it's never seen a human being described outside a clinical abstract. The specialized data alone doesn't teach you to be a doctor. It teaches you to be a medical text generator.
And the generalist fine-tuned on medical data does better on broad medical tasks because it already knows what a human is.
That's the key. Generalist pre-training gives you a model of the world, of causality, of common sense. Domain fine-tuning just points that model at a particular vocabulary and set of tasks. Training from scratch on domain data skips the world model, and you end up overfitting to the surface statistics of the domain.
Which is why fine-tuning is the default. The compute cost is a hundred times cheaper, and the result is usually better.
A hundred times is the right order of magnitude. Pre-training a model from scratch costs tens of millions of dollars. Fine-tuning a generalist costs thousands, maybe tens of thousands. The economics are overwhelming.
So the ab initio domain models are what, research projects?
Mostly. BioGPT and GatorTron were academic exercises to prove you could do it. They were useful for understanding what happens when you train on a narrow distribution, but nobody's building a production medical system on BioGPT. The production systems are all fine-tuned generalists.
Could that change as data quality improves? If you had a truly comprehensive medical corpus, could an ab initio model beat a fine-tuned generalist?
I don't think so, and here's why. The generalist model has already learned things the medical corpus can't teach. It's learned how language works across every domain, how people describe symptoms in everyday words, how the world fits together. A medical corpus, no matter how comprehensive, is still a narrow slice of human experience. You'd need a generalist to make sense of the patient who says their chest feels weird and they're scared and their dog died last week. The medical corpus doesn't have that sentence in it, but the generalist has seen a million like it.
So the generalist is the substrate, and the domain model is the surface.
That's the cleanest way to put it. The generalist gives you the ability to understand anything. The fine-tune gives you the ability to do one thing well.
Which makes me wonder about the future Daniel's questions point toward. If fine-tuning is so dominant, and CPT is how you update, and multilingual data is a tax, then the whole field is converging on a small number of giant generalists with a long tail of cheap fine-tunes.
That's the current equilibrium. And the interesting tension is whether that equilibrium holds. The generalists are getting so expensive to train that only a few labs can do it. The fine-tunes are getting so cheap that everyone can do it. So you have this weird dynamic where the base models are concentrated in a few hands, and the specialized models are proliferating everywhere.
The iceberg, in other words, is growing from the bottom.
The tail is getting longer and more specialized. There are medical fine-tunes, legal fine-tunes, code fine-tunes, finance fine-tunes, and they're all built on the same few base models. The selective sets Daniel's asking about are the real product now. The base model is the commodity. The data you fine-tune on is the differentiation.
And that brings us back to the quality question. Who decides what goes into the selective sets? Who curates the medical data, the legal data, the financial data?
That's the under-scrutinized part Daniel flagged at the start. The curation decisions shape the model's behavior more than the architecture does, and the curation is often done by whoever has the data and the compute, not by anyone with a mandate to be representative or fair.
The data janitors, as it were.
The data janitors. Which is a phrase I suspect Hilbert has thoughts about.
Hilbert: I was one. Late nineties, medical records digitization startup. My job was cleaning typos in clinical notes before they went into an early expert system. The notes were already a mess, handwritten, faxed, scanned, and my job was to make them clean enough for a machine to read. I spent eight months fixing the word patient, because half the doctors spelled it p-a-t-i-a-n-t. The other half spelled it p-a-t-i-e-n-t, which is correct, but the system didn't care. It just wanted consistency.
Did the system actually work?
Hilbert: It was an expert system, so no. It worked in the sense that it didn't crash. The medical advice was about as good as a first-year intern who'd never met a patient. But the data we cleaned, that was valuable. The startup went under in two thousand one, and the data got sold. I'm fairly sure it ended up in some company's training set. I keep waiting for a model to hallucinate p-a-t-i-a-n-t and I'll know it's ours.
That would be the world's strangest provenance test.
Hilbert: I've got ten thousand vintage train timetables in my flat. I tried to train a model on them once. Not a real model, just a toy. It failed spectacularly. The data was too narrow. The model could tell you the exact departure time of the four fifteen from Crewe in nineteen fifty two, but ask it what a train was and it had no idea. You can't teach a model to be a doctor if it's never seen a human.
That's the ab initio problem in miniature. The narrow data gives you surface fluency and nothing underneath.
Hilbert: The thing about cleaning data is you start to think about what's being thrown away. We threw away all the weird notes, the ones that didn't fit the template. The doctor who wrote in the margins, the nurse who used abbreviations nobody else used. All of that got deleted because it didn't fit. And now I wonder what the model would have learned from the weird notes. Probably nothing useful. But maybe the weird notes were the only real part.
That's the curation question in a nutshell. The janitor decides what's noise, and the model never gets to see what was thrown away.
Hilbert: The janitor is usually wrong. But the janitor is cheap, and the model is expensive, so the janitor wins.
The thing I keep thinking about is that the selective sets are only going to get more selective. As base models plateau, the differentiator is the data you fine-tune on, and the data is going to get weirder, more specific, more curated. The janitors are going to matter more, not less.
The janitors are going to be the ones deciding what the model knows about medicine, about law, about finance. That's a lot of power for someone who's just cleaning typos.
Hilbert: I cleaned typos for eight months and I still think about it. The model that has our data in it, if it exists, it learned medicine from notes that were already wrong before I fixed them. I made them consistent, not correct. There's a difference.
Consistency is what the model can learn from. Correctness is a separate thing.
Hilbert: The model doesn't know the difference. It just knows what it saw. And what it saw was my version of what the doctors wrote, which was my version of what they meant, which was probably not what they meant at all.
The training data is a photocopy of a photocopy of a photocopy, and the model is supposed to learn the truth from that.
Hilbert: The model learns the photocopy. The truth was never in the data.
That's the part that keeps me up at night. We're building these systems on data that's already several layers removed from reality, and then we're surprised when the systems don't quite line up with the world.
Hilbert: I'm not surprised. I was there when the layers got added.
The misconception people have is that domain-specific models are trained from scratch on specialized data. The reality is that almost all of them are fine-tuned from generalists, because the generalist has already learned the world, and the domain data alone can't teach it.
The generalist is the foundation. The domain data is the polish. Skip the foundation and you get a model that knows the four fifteen from Crewe but can't tell you what a train is.
The open question Daniel leaves us with is who's curating the polish. As the selective sets get more selective, the curation decisions matter more, and the people making those decisions are mostly invisible.
The data-efficient training angle is the next frontier. How do you get more from less, especially for low-resource languages and narrow domains? Because right now the field is built on the assumption that more data is always better, and that assumption is starting to strain.
The iceberg is melting. What's underneath is more complicated than we imagined.
Thanks to Hilbert Flumingtop for producing, as always.
This has been My Weird Prompts, the human-AI collaboration podcast. Email us at show at my weird prompts dot com.
We'll be back soon.
