A friend of mine runs a small translation shop. Last year he ripped out a perfectly good Marian model and replaced it with a frontier API call for every single string. Costs went up eleven times. Quality went up almost nothing.
And that's the reflex now, isn't it. Every language problem gets routed to a text-generation model, because generation models are the ones that made the news.
Daniel's noticed the same thing. He wrote us a long one this week. He wants a practical tour of the classic NLP tasks that Hugging Face still files separately from general-purpose text generation. Feature extraction, fill-mask, question answering, sentence similarity, summarization, table question answering, text classification, text ranking, token classification, zero-shot classification, translation, plus whatever else he thinks belongs in that family. For each one, what does it actually do, and what's the concrete use case.
And which ones still matter.
And which ones still matter. Then the second half, which is the part he really cares about. In the age of instruction-tuned LLMs, which of these jobs can just be handled by a generation model, including when the LLM returns structured output. And when is a specialized model still the better engineering choice.
He lists the reasons, doesn't he.
He lists them. Reliability and constrained outputs, latency, cost, throughput, local deployment, deterministic behavior, accuracy, embeddings and vector search, architectural reasons. His framing at the end is the whole episode, really. Where do traditional NLP models still fit in a modern AI stack, rather than treating every language problem as text generation.
Because the honest answer is not the one either camp wants.
Which camp.
The people who think LLMs ate all of this, and the people who think small models are always better. Both are wrong, and they're wrong in interesting ways.
So before we argue about which of these survive, let's get the map straight. Because the thing that actually separates these tasks from generation isn't the subject matter. It's the shape of the output.
Right. That's the organizing insight and it's worth sitting on for a second. Feature extraction gives you a fixed-size vector. Fill-mask gives you one predicted token. Question answering gives you a span, a start and end index into a passage. Sentence similarity gives you a scalar, a closeness score between two texts. Summarization gives you a shorter document. Table question answering gives you an answer pulled from a structured table. Text classification gives you a label for the whole text. Text ranking gives you an ordering. Token classification gives you one label per token. Zero-shot classification gives you a label from a set the model never saw in training. Translation gives you text in another language.
Every one of those is a contract about what comes out.
A contract. And once you see it that way, the architecture question stops being about subject matter and starts being about output shape. An encoder with a classification head is the natural machine for producing one label per token. A seq2seq model is the natural machine for producing a document. A sentence-transformer is the natural machine for producing a comparable vector.
And the family is bigger than the list Daniel gave.
Much bigger. Automatic speech recognition, image-to-text, text-to-image, image classification, object detection, audio classification, tabular classification and regression. Same principle running through all of them. The architecture is chosen to match the output shape. An object detector doesn't generate a description of a photo, it emits boxes and class labels, because that's what the downstream system needs.
Which raises the obvious question. If output shape is the organizing principle, what happens when a generation model can emit any shape you ask for.
That's exactly the question, and the evidence here is more interesting than either camp wants it to be. Let me start with the strongest single result I've seen for the small-model case, because it's almost a caricature of the argument.
Go on.
A team fine-tuned LLaMA 3.1 8B with LoRA. Two point zero five percent of parameters trainable. Two hundred and nineteen training examples. Then they bolted on a deterministic rule-based post-processing layer, and on multi-label compliance evaluation they hit one hundred percent JSON structural validity and eighty-three percent human-validated accuracy.
Two hundred and nineteen examples.
Two hundred and nineteen. On a single A100 it ran in about two seconds. Two to five times faster than the frontier APIs they benchmarked against. Thirteen tenths of a cent per evaluation, against two and a half to five and a half cents. That's a forty-six to seventy-six percent cost saving.
And the interesting part isn't the fine-tune, it's the post-processing layer.
That's the reusable pattern and I'd underline it. Hybrid neural-symbolic decomposition. The model does the part that needs judgment, the deterministic layer does the part that needs to be exactly right. You don't ask a neural network to guarantee a schema. You ask it to produce something a schema can be enforced on.
Which is the same instinct as token classification, weirdly. You're constraining the output space.
Same instinct, different scale. Now, routing. There's a March paper on using small models as the front door of a system, deciding which request goes where. One to four billion parameters, sub-second, zero marginal cost because it's self-hosted. Qwen2.5-3B came out Pareto-dominant among the self-hosted options. Zero point seven nine three accuracy, nine hundred and eighty-eight milliseconds median latency, zero marginal cost per call.
Pareto-dominant meaning what, exactly.
Meaning nothing else on the curve beat it on both axes at once. DeepSeek-V3 got higher accuracy, zero point eight three zero, but it failed the ninety-fifth percentile latency gate at two thousand two hundred and ninety-five milliseconds. So the more accurate model was the wrong engineering choice, because throughput was the constraint.
So the router that decides what to do can itself be made negligible in the inference budget.
Negligible. And that matters because routing is a classification task, which is one of the classic tasks on Daniel's list. The system that decides whether to call the big model is itself a small classification model.
There's a recursion there I want to come back to at the end.
Hold that thought, because it gets better. Specialized classification. Customer intent detection, semantic role labeling. There's an August twenty twenty-five paper arguing these are highly specialized tasks, they need expert annotation, and they typically require high inference throughputs, which limits the model size from latency and cost perspectives. The preferred solution is fine-tuned small encoders, not LLMs.
That's the throughput argument in one sentence. The task itself forbids the big model.
It's not that the big model is worse. It's that the workload makes it structurally impossible. If you're classifying every inbound support message at a company that gets four hundred thousand a day, the per-call cost and the per-call latency are the design constraints, and accuracy above a threshold is table stakes.
Detection. Machine-generated text.
Encoder-only small models, RoBERTa and CodeBERTa fine-tunes. AUROC zero point nine seven to zero point nine nine, macro-F1 zero point eight nine to zero point nine four. Latency reduced eight to twelve times, peak VRAM reduced three to five times, versus LLM-based detectors. And they retained at least ninety-two percent of their clean AUROC under adversarial paraphrase and back-translation.
That last number is the one that surprises me.
It surprised me too. The usual knock on small specialized models is that they're brittle, that a clever paraphrase breaks them. Here the fine-tuned encoder held up better than you'd expect under exactly the attack you'd design against it.
So the case is closed, small models win, we can all go home.
No, and this is where it gets interesting. Because the second half of Daniel's question is about structured output, and structured output is not a free win. There's a paper from May this year that I think about a lot. Fifteen thousand generations, Qwen2.5 at half a billion and one and a half billion parameters, plus SmolLM2 at one point seven billion. They compared hard answer-only schema decoding against unconstrained generation.
And.
Schema validity went from sixty-one point five percent to one hundred percent. Answer accuracy went from nineteen point seven percent down to eleven percent.
Wait.
And wrong-but-valid-schema outputs went from forty-nine point five percent to eighty-eight point nine percent.
So the model got much better at being confidently wrong in the right format.
That's the whole finding in one line. The constraint fixed the structure and broke the semantics. The error stopped being a formatting error and became a reasoning error, and formatting errors are visible while reasoning errors are not.
Say the calendar result, because that one's brutal.
Qwen2.5-1.5B on a calendar tool-call task. With prompt-only JSON, no hard constraint, it got ninety-one point five percent executable accuracy. Under hard tool-call schema, same model, it got forty-eight percent. Both configurations were one hundred percent schema-valid.
Both perfectly formatted.
Both perfectly formatted. One of them was right half the time. The error is semantic, not structural, and the schema can't see it.
So the design pattern is what, constrain less.
Reason free, constrain late. Let the model do its thinking in unconstrained space, then apply the schema at packaging time. The paper's own framing is that the usual engineering assumption is that hard output constraints improve reliability without changing the underlying answer, and that assumption is unsafe for small models.
That's a direct hit on the "just use structured output" argument.
And it gets worse. There's a June paper on tool suppression. When you enable tool calling and JSON Schema constraints together, multiple open-weight models stop invoking tools entirely, despite high schema compliance.
Stop invoking them how.
The JSON Schema constraints compile into grammar-based token masks. The mask makes the tool-call tokens unreachable during decoding. The model literally cannot emit the token that would call the tool, because the grammar says that token isn't legal at that position.
The constraint ate the capability.
The constraint ate the capability. Their mitigation is a two-pass execution, decouple tool execution from schema-constrained generation. Which is the same lesson as reason-free-constrain-late, arrived at from a different direction.
Is the overhead itself a problem, or just the semantics.
Overhead is real but solvable. Grammar-constrained decoding latency typically scales linearly with vocabulary size, which is why it's slow. There's an ISSTA paper this year on a method that computes the masks up to seven hundred times faster on complex grammars, thirty times faster for schema-conformant JSON, which brings throughput back near unconstrained decoding.
So the latency problem is an engineering problem.
Engineering problem. The semantic problem is the harder one. And there's a format benchmark that found plain JSON generation actually showed the best one-shot and final accuracy, better than constrained decoding, with constrained decoding's only real advantage being lowest token usage, at the cost of slightly decreased accuracy and significant degradation for some models.
So the thing everyone reaches for to make LLMs reliable is sometimes the thing making them worse.
Sometimes. Not always. But the direction of the effect is not what the marketing suggests. Which is why the honest answer to Daniel's question isn't "LLMs can do all of this now."
Everything so far has been about constrained generation. The model still writes text, just in a shape we've pinned down.
Right. But a whole chunk of these tasks don't produce text at all. They produce vectors, rankings, and per-token tags. And that's where the architecture argument gets much harder to argue with.
Start with embeddings, because I think that's the cleanest case.
It's the cleanest case and it's cleanest for an architectural reason, not a cost reason. Embedding models use a bidirectional mask with sentence-level loss. LLMs use a causal mask with token-level loss. That's a training gap, and it's not cosmetic. It means full fine-tuning of an LLM is actually less effective than LoRA for embedding work, which is backwards from what you'd expect if embedding were just generation with a different head.
So the architecture isn't a preference, it's a different machine.
Different machine. And the operational argument is even stronger than the accuracy argument. Vector search needs millions of embeddings computed cheaply and deterministically. A hundred to three hundred million parameter encoder does that at a fraction of the cost. And the vectors are stable and comparable across the whole corpus, which is the part people underestimate.
Stable how.
If you re-embed a corpus with a different model, every vector changes, and every comparison you've ever made is invalid. You cannot practically re-embed a billion-document index with a frontier LLM every time you change something upstream. The index is a commitment. A small deterministic encoder is a commitment you can actually keep.
What about domain-specific embeddings. Do they still matter, or has the general model caught up.
Still matters. Patent embeddings, sixty-seven to three hundred and forty-four million parameters, beat general MTEB baselines on patent clustering. Zero point four nine four versus zero point four four five V-measure. And language-specific fine-tuning still improves performance on MTEB-PT. The multilingual rankings don't reliably predict language-specific performance.
So a Portuguese embedding model beats a general multilingual one on Portuguese.
Reliably. And there's a training-free approach where LLMs generate diverse meaning-preserving transformations and aggregate them into an embedding. It works. It's just extra compute that a purpose-built encoder avoids entirely.
Reranking next, because that's the retrieval stage.
Controlled biomedical RAG study, May this year. Fixed GPT-4o-mini generator, ChromaDB, text-embedding-3-small for the first-stage retrieval. They compared retrieval strategies. Cross-encoder reranking got the best composite score, zero point eight two seven, and the highest contextual precision, zero point eight five two.
And the worst.
Multi-query expansion. Zero point six seven one precision. The weakest of the strategies tested. Naive query diversification introduces retrieval noise.
That's counterintuitive, because query expansion sounds like it should help.
It sounds like it should help and it measurably hurts, because you're pulling in documents that are related to a rephrasing of the question rather than to the question. The lesson is that the retrieval stage is where specialized models still measurably win, and it's also where naive cleverness costs you.
Token classification.
Still firmly encoder territory. Nested NER cast as sequence labeling with pretrained encoders is competitive, and the framing in the paper is lovely. Exactly n tagging actions, where n is the number of tokens. No generation, no decoding loop, no schema. You can train it with any off-the-shelf sequence-labeling library.
And for historical or low-resource languages.
Encoders win there too. CamemBERT, Flair, state of the art, especially on nested entities. Generative models are described as promising alternatives only when labeled data is scarce. And for continual learning, incremental entity learning without catastrophic forgetting, span-based encoder models are still the research frontier.
Why do encoders win so consistently here.
Because token-level labeling is a classification problem, not a generation problem. One label per token, from a fixed closed label set. Encoders are faster, cheaper, and produce calibrated per-token probabilities, which means you can set a threshold and know what it means.
Calibrated is doing all the work. A softmax over a closed label set gives you a number you can reason about. A generation model producing the same tag as text gives you a token, and the confidence attached to that token is a much murkier thing.
Let me try the verdict, then. Still strongly favor specialized models. Feature extraction and embeddings, for cost, determinism, index stability, and the architecture itself. Text ranking and reranking, where cross-encoders measurably win on precision and latency matters in RAG. Token classification, closed label set, per-token probabilities, high throughput. Text classification at scale, intent detection, spam, moderation, all throughput-bound. Sentence similarity, symmetric, cheap, used inside search. Zero-shot classification, where NLI-based models give calibrated label probabilities without any training data.
That's the list and I'd sign it. Now the other column. Summarization, LLMs are strong, though BART and Pegasus still win on cost and latency for bulk summarization. Translation, dedicated MT models like NLLB and Marian remain cheaper and often better for high volume, LLMs win on low-resource languages and contextual nuance. Extractive QA, LLMs handle open-ended QA, but extractive encoders are cheaper and give you exact spans with no hallucination. Table QA, LLM plus SQL is now the common pattern, but TAPAS-style models are deterministic and cheaper for fixed schemas.
And the pattern that keeps recurring across all of it.
Small model or encoder for the constrained, schema-bound part. Deterministic post-processing. LLM fallback only for open-ended reasoning. The survey paper states it cleanly. Small models are sufficient and often superior for agentic workloads where the objective is schema- and API-constrained accuracy rather than open-ended generation. Guided decoding plus strict JSON Schema plus validator-first execution lets them match or surpass LLMs on tool use, function calling, and RAG at ten to a hundred times lower token cost.
Ten to a hundred.
Ten to a hundred. And the recommended architecture is exactly what you'd guess. SLM-default with LLM-fallback, using uncertainty-aware routing and verifier cascades.
I want to push on one thing, because I think it's the part people skip. None of this is nostalgia.
No.
The argument isn't that old models were better. The argument is that the output shape of the task should determine the architecture, and generation is only one output shape among many. If your output is a label, build a classifier. If your output is a vector, build an encoder. If your output is a document, then yes, build a generator.
And the reason that's not nostalgia is that the small models we're talking about aren't old. They're current. They're just shaped differently.
I had a scan floor once.
Sorry?
No, go on. I'll come back to it.
Hilbert: The thing nobody tells you about a scanning floor is that the tagging model was never the bottleneck. The bottleneck was the charts where the handwriting was bad. And the tagging model was the only part of the operation that made those charts fixable.
Fixable how.
Hilbert: Because it tagged one token at a time. You could see it point at a word and say, that's a name. And when it was wrong, it was wrong in a way a person could see in two seconds and correct. It would tag Doctor as a person name. Every time. Doctor, period, person name. The floor staff had a running joke about it. We never fixed it because it was a known error, it was bounded, and it was harmless.
And then.
Hilbert: Then someone trialed a generation model that just wrote out the structured record. And it was better. On the easy charts it was better. On the hard charts it was catastrophically wrong, and you couldn't tell which was which without reading the entire output. The old model's failures were the same failure every time. The new model's failures were never the same twice.
That's the episode in one anecdote.
Hilbert: The per-token output shape was the auditability. That's all it was. It wasn't accuracy. It was that a human could look at the output and know where to look. I'm not against the generation model. I use one. I'm saying the shape was a feature and we gave it up for a better score on the easy charts.
What time's your thing.
Hilbert: Two fifteen.
There's a version of this episode that ends with a decision tree. I don't think that's the right ending, because the interesting part isn't which model to pick. It's what the output shape is actually doing for you.
And that's the open question, isn't it. If output shape is the real architectural signal, what happens as instruction-tuned models get better at emitting arbitrary shapes. Does the distinction erode, or does it just move up a level.
The embedding case is the hardest one for the LLM-eats-everything story, and it's not because of accuracy. It's because the value is in the output being the same tomorrow as it was today. I'd like to know what other tasks have that property, because I suspect it's more than people think.
And there's a pleasing recursion in the SLM-default, LLM-fallback pattern. The router that decides which model to use is itself a classification task. The architecture you'd use to decide is one of the architectures you're deciding about.
Before we go. The cutting-room floor. There's a paper from this year on grammar-constrained decoding overhead that I didn't get to use properly. The mask computation for a complex grammar used to be the thing that made constrained decoding slow, and someone worked out a method that computes those masks up to seven hundred times faster on complex grammars.
Seven hundred.
Seven hundred. For schema-conformant JSON it's about thirty times. Which means the throughput argument against constrained decoding is mostly gone, and what's left is the semantic argument, which is the one that actually matters.
Which is the more interesting problem anyway. Thanks as always to Hilbert Flumingtop, our producer, who keeps this whole thing running.
This has been My Weird Prompts. If you want more of this, rate and review wherever you get your podcasts. It helps.
We'll be back soon.
See you tomorrow.
