Here's the question that's been rattling around my head all week. If the tell disappears the moment you name it, what exactly are you hunting for?
That's the one. And it came from Daniel, who has been staring at our own archive and realizing it's not an archive anymore.
Right. Daniel's pitch, in his words, is roughly this. We've looked before at small models for fixing things that get overstated in a particular kind of generated text. He's noticed specific turns of phrase, "failure mode" being his example, that are wildly overrepresented in our scripts. And the reason he can name that one is that he's listened to thousands of episodes, so it eventually occurred to him. But he suspects there are other little things he hasn't noticed, and possibly things that older models leaned on that newer ones have quietly dropped as they got more capable. The missing link, he says, between having the scripts and actually pinpointing the overuse is text corpus analysis. So question one: are there tools specifically built to explore a corpus like ours? And question two, which is the one I find more interesting: is labeling item types in the episode text, exclamations, idioms, ticks, actually a classifier-model job, or is the right workflow a classifier and a traditional corpus tool working in conjunction?
Two questions tangled together. What tooling exists, and who does the labeling.
And the second one is where the real methodological fight is.
It is. So let's define the thing first, because "corpus analysis" gets used loosely. Text corpus analysis is the study of a body of text as a body. Not reading document one, then document two. You look at frequency, dispersion, which is how evenly a term is spread across the corpus, collocation, which words keep showing up next to which other words, concordance, which is keyword-in-context, and keyness.
Keyness being the one that matters most here.
Keyness is the whole episode, honestly. It's a signed two-by-two association score. Originally implemented in a tool called WordSmith, and the idea is dead simple: you have a target group and a reference group, and keyness tells you which words are unusually frequent in the target relative to the reference. Signed, so you get both directions. That is literally the question "what does model X overuse compared to model Y."
Which is exactly what Daniel's metadata gives us. Each script carries the model that generated it.
And that's the asset most corpus studies don't have. But before we get to the good news, let me deliver the honest negative finding early, because I went looking and I want to save everyone the trouble.
Go on.
There is no dedicated LLM tick detector. There's no package called overused-expression-finder. Searches on arXiv for "LLM overused phrases detection corpus" return zero results. "Overused phrases LLM writing style," zero results. This workflow is something you assemble. It does not ship as a product.
I find that reassuring, actually. It means Daniel isn't missing an obvious tool. The gap is real.
The gap is real, and it's interesting rather than disappointing, because it means the interesting part is the assembly. So the concept is clear enough. The question is what you'd actually point at a corpus this size.
Let's walk the tooling.
Two families. Family one is the general NLP stack. NLTK, spaCy, gensim, which is topic modelling, sixteen thousand four hundred stars on GitHub, and textacy, which is built on top of spaCy and describes itself as focusing primarily on the tasks that come before and after core NLP.
Meaning what, exactly?
Meaning spaCy does the parsing, and textacy handles the cleanup before and the analysis after. It's a convenience layer. Useful, but it doesn't know anything about keyness.
So family two.
Family two is purpose-built corpus linguistics, and this is where it gets good. quanteda, in R, is the reference implementation for keyness. The function is textstat_keyness, and it gives you a choice of measures: chi-squared, exact, log-likelihood, and pointwise mutual information. If you want to know what one group of texts overuses relative to another, that function is the canonical answer.
And in Python?
corpkit. It parses, it interrogates, it does concordances and keywords. But here's the feature that made me sit up. corpkit lets you build structured corpora with speaker ID labels, and restrict your searches to subcorpora.
Say why that matters for us.
Because if every script carries the generating model as metadata, you can slice the corpus by model and run the identical analysis on each slice. That turns "has it improved?" from a hunch into a comparison. You're not eyeballing it. You're running the same keyness calculation on the model-A slice and the model-B slice and reading the difference off the chart.
That's the difference between a feeling and a measurement.
It's the whole difference. Now, scale. Two tools worth knowing. ELFEN, Efficient Linguistic Feature Extraction for Natural Language Datasets, is a Python package for pulling linguistic features out of text datasets at scale. Built on polars for large datasets, with spaCy or stanza underneath, and it's explicitly designed to analyze NLP model outputs. That last part is not nothing. Somebody built it with our exact use case in mind.
And the second?
EleutherAI's tokengrams. It computes n-gram statistics over large pre-tokenized corpora using a suffix-array index, and it computes n-gram counts on the fly for any n. So if you want to find overrepresented phrases, not just single words, across millions of script tokens, that's the scalable path. You don't have to pre-decide which n-grams you care about.
So you can ask "what three-word sequences show up more than they should" without knowing in advance what you're looking for.
Exactly that.
Now the one you're excited about.
pycorpdiff. And I want to be honest about its maturity, because this is not a mature tool. It positions itself as the missing comparative layer between quanteda, the closed-source SketchEngine, and the fragmented Python stack. Three verbs. Compare A to B. Track a term in a corpus. Compare before and after an event.
Track a term.
And this is the killer feature for our episode. You can take a corpus, name a term, and ask for its trajectory over time, then ask for changepoints. The changepoint detection uses something called PELT, and there's a burstiness measure using a Kleinberg hidden Markov model. So you could plot exactly when a phrase like "failure pattern" spiked, and exactly when it faded.
Across model generations.
And when you get a ranked list of terms, there's an explain function that returns the keyword-in-context concordances behind any ranked term. So you don't just get "this word is overused." You get the actual sentences.
That's the part that would convince me. A number alone is an accusation. The concordance is the evidence.
And the status. Alpha. Version zero point one point zero alpha thirty-five. MIT licensed. Created June sixteenth of this year. Two GitHub stars.
Two.
Two. I'm not going to pretend it's battle-tested. But the design is pointed at exactly this problem.
What else is out there, briefly?
Unified Corpus Explorer, dockerized, open source, presented as a NAACL demo last year. KonText, which is Charles University's web front-end for the Manatee-open search engine. Corpus sense, from Moreno-Ortiz, in Applied Corpus Linguistics. TALL, Text Analysis for All, an R Shiny app, code-free, version one point zero released in April. montre, an embeddable corpus query engine. And Context-Fabric, which is memory-efficient storage and querying for annotated corpora, claiming ninety-two percent less memory.
Ninety-two percent less memory is the kind of number that makes a dataset this size actually tractable.
It's the unglamorous part that decides whether the project happens.
Here's the mechanism point I want to land before we move on. Keyness is a two-group comparison. The entire method depends on having a reference group.
Correct. You cannot compute keyness against nothing. You need a target and a reference.
And our metadata is what supplies it. Model A as target, model B as reference. Or all models as target and a human-written baseline as reference.
And that second one is the more interesting comparison, because it asks not "is model A different from model B" but "is any of this different from a person."
Which is the question Daniel is actually asking.
It is. That's the machinery. Now here's the part that surprised me. What the machinery finds when you point it at real text.
Let's hear it.
Start with the literature that directly answers "have they improved," because it contains a genuine surprise. There's a study in Science Advances that tracked excess words across fifteen point one million scientific abstracts before and after ChatGPT. They estimated at least thirteen and a half percent of all 2024 abstracts had been through a model. Roughly two hundred thousand papers. For computation papers out of China, the figure hit forty-one percent.
Forty-one percent of a country's computation abstracts.
And a separate team tracked twelve terms across six bibliographic databases from 2015 to 2024. Between 2022 and 2024, "delve" rose about fifteen hundred percent. "Underscore" about a thousand. "Intricate" about seven hundred. In PubMed Central full texts, the share of papers using "underscore" six or more times grew over ten thousand percent between 2022 and 2025.
Ten thousand percent. That's not a trend, that's a phase change.
And a third team, looking at nine hundred fifty thousand papers, estimated up to seventeen and a half percent of computer science arXiv abstracts had been model-modified by February of 2024.
So the phenomenon is real, it's large, and it's measurable. Now give me the surprise.
The surprise is that the famous words are not the story. There's a corpus study from June that measured eleven million words of paired human and ChatGPT answers. Fifty-eight thousand human answers, twenty-six thousand machine answers, using a log-odds-ratio method with a Dirichlet prior, the Fightin' Words approach. And "delve," "tapestry," "intricate," "realm" all fall below the counting threshold.
They don't even register.
They don't register. And "robust" runs the other way. Zero point four seven times. Humans used "robust" more than the machines did.
So the word everyone jokes about is a human word.
In that corpus, yes. The actual signature is a hedged, list-building textbook register. "Overall" at about nine and a half times. "Important" about eight point two. "Including" about seven point seven. "Factors" about six point seven. "Help" about five point seven. "May" about four point six. "Such" about three point three. "Also" about one point eight. "Other" about one point seven.
Those are the emptiest words in the language.
They're the connective tissue. And that validates Daniel's instinct in one direction and undermines it in another. He's right to trust his ear over a word list. But the actual overuse is boring words, which is much harder to hear by ear. You cannot notice "overall" being nine times too frequent. Nobody can. That's precisely why you need the corpus tool.
You can hear "failure pattern." You cannot hear "including."
And that's the argument for the whole project.
Now answer his improvement question directly.
Here's the finding I'd put on the wall. "Delve" and "intricate" fell in arXiv abstracts soon after they were publicly named as tells in early 2024. Meanwhile bland markers like "significant" kept climbing. Model use didn't drop. The conspicuous words did.
The fingerprint migrates.
So Daniel's "failure pattern" example may already be a former tick. The tooling should be hunting the next one, not the last one. A detector or a house style built on a fixed word list is dated from the day it ships.
Which is a slightly bleak conclusion for anyone building a blacklist.
It is. And it raises the question of why the overuse exists at all, because the answer reframes everything.
Go.
A team tested the obvious explanations for twenty-one of these words. Training data. Architecture. Decoding settings. None of them accounted for the overrepresentation. What survived was consistent with reinforcement learning from human feedback.
So it's not the internet's fault.
Not primarily. Their follow-up is the clean experiment. They took identical prompts, derived from PubMed, and put them to Llama Base and Llama Instruct. "Nuanced" appeared eight thousand three hundred forty-two percent more often from the instruct model. "Firstly," four thousand seven hundred ninety-four percent more.
Same base model. Same prompts. The only difference is the tuning.
And here's the part that makes it click. They had four hundred raters choose between the high-overuse variant and the low-overuse variant. The raters picked the high-overuse one fifty-two point four percent of the time, against forty-seven point six.
A five-point preference.
Five points. And another team named the mechanism. Typicality bias. The rater's judgment equals true quality plus a coefficient times the typicality of the wording. And that coefficient came out at about zero point five seven. Familiarity pulls more than half as hard as being right.
Five points of preference, collected in bulk, optimized against without mercy.
And it comes out the far end as a verbal habit anyone can spot. That's the whole mechanism. It's not a bug in the data. It's the training objective working exactly as designed.
Which means it won't disappear. It'll relocate.
It'll relocate. Now, the classifier question, because Daniel asked it precisely and the literature answers it precisely.
This is the part I want to slow down on.
The answer is both, in conjunction, and the literature says so explicitly. Start with the classifier side. There's a 2026 paper in Expert Systems with Applications using tree-based models, decision trees and LightGBM, on top of a stylometric feature set plus n-gram features. They got up to point eight seven Matthews correlation coefficient in a seven-class problem, and point nine eight accuracy on a binary Wikipedia-versus-GPT-4 task.
And the interpretability?
That's the good part. They ran SHAP on it, and SHAP pinpointed features characteristic of the encyclopaedic text type, individual overused words, and a greater grammatical standardization in the model text.
So the classifier, when you open it up, hands you overused words.
It hands you overused words. That is the exact hybrid Daniel is asking about, and it already exists in the literature. The classifier finds and labels. The corpus tool quantifies, tracks over time, and shows you the concordance evidence. Neither one alone gives you "which tick, how much, since when, and in what context."
Push the architecture further. What does the state of the art actually look like?
Two papers combine discrete stylistic indicators with continuous stylistic representations from semantic embeddings. Discrete features plus embeddings. That's classifier plus feature extraction, fused. And there's a study from April that analyzed stylistic variation across eleven models, eight genres, and four decoding strategies, using Douglas Biber's lexicogrammatical and functional features, which is the canonical corpus-linguistics feature set.
Findings?
Three. The key differentiators are robust to prompting. Genre exerts stronger influence than source. And model has a larger effect on style than decoding strategy.
That last one is operational for us.
It's the most operational finding in the whole episode. If model has a bigger effect on style than decoding settings, then model-generation slicing should be the primary axis of our analysis. Don't slice by temperature. Slice by which model wrote it.
And the metadata angle closes the loop.
It does. There's work showing LLMs have distinct and consistent stylistic fingerprints. An ensemble classifies which of Claude, Gemini, Llama, or OpenAI generated a text with precision zero point nine nine eight eight and a false-positive rate of zero point zero zero zero four.
That's essentially perfect attribution.
It's essentially perfect. And because our scripts carry the generating model, we can train a classifier and validate it against known labels. That's a luxury most corpus studies don't have. They're guessing at provenance. We know it.
So state the division of labor plainly, because this is Daniel's actual question.
The classifier finds and labels. The corpus tool quantifies, tracks over time, and shows concordance evidence. You want to know which tick, how much, since when, and in what context. No single tool gives you all four.
And name the genuine gap, because I think it's the most interesting thing we've said.
No published study tracks a single outlet's own scripts across model generations to show tick improvement over time. The closest analogues are the aggregate arXiv and PubMed fingerprint-migration studies. Nobody has done the per-publisher longitudinal version.
We have the dataset.
Hilbert: Four thousand two hundred and eleven.
What?
Hilbert: That's the number of words on the laminated card. The banned list. I did a stretch as a night-shift proofreader for a regional newspaper chain, and they had a style guide, and the style guide had a banned-words card, and I counted the words on it once out of boredom. Four thousand two hundred and eleven words on a card that fit in a shirt pocket, both sides, six-point type.
Six-point.
Hilbert: You'd hold it up to the light. Anyway. The card got updated twice while I was there. And the reason both times was the same. A word would get banned, the columnists would stop using it, and within a month every one of them would be leaning on a different word that nobody had thought to ban yet.
So the card was always behind.
Hilbert: The card was never wrong. That's the thing I want to say. Every word on it deserved to be on it. It was just late. It was a record of the last war. And I kept my own tally in the back of a notebook, because I wanted to see how long the lag was.
How long was it?
Hilbert: About a month. Sometimes five weeks. And I never got to say the thing I wanted to say to the editor, which is that you can't ban your way out of this. You can only get faster at noticing. Bans are a list. Noticing is a process. He wanted a list.
And you think the corpus tool is the first thing that could actually be faster at noticing.
Hilbert: I think it's the first thing I've heard of that isn't a card. That's why I've been sitting on the notebook detail. The card is a snapshot. What you two are describing is a camera.
Can you show us the notebook?
Hilbert: I can describe the pattern. A word gets named, it dies, a cousin word rises. I could show you the exact month it happened. Twice. Two different words, in a paper nobody has ever heard of.
Is it legible?
Hilbert: Parts of it.
Parts.
Hilbert: I've left the garage light on at home. It's the kind that gets hot.
Right. So the thing that keeps nagging at me is whether the words migrate but the shape stays put.
That's the thread I'd leave hanging. The conclusion from that corpus study was that word lists age instantly, but the durable signal is the distributional shape. Which raises the question: is this a permanent arms race, or is there a stable shape underneath the migrating words?
Whether that shape is stable is a question you can only answer with a dataset that spans model generations.
Which we happen to have. Whether it's worth doing is a question for Daniel, not a recommendation from us.
Thanks as always to Hilbert Flumingtop, who produces this show and counts words on cards.
This has been My Weird Prompts.
If you're enjoying the show, a review helps more than you'd think. We'll be back soon.
