Okay, I'll admit it. My first reaction to this one was relief. Somebody finally built the model I've been complaining about for two years.
And my first reaction was suspicion, because the name is spelled wrong.
We'll get to that. Daniel sent us a model on Hugging Face called Hemmingway-1, two m's, and it claims to solve something he'd been chewing on. His words: he'd assumed the cheeriness and the tendency toward explanation was just inherent to models. He'd seen system prompts push back against it. But a model with directness baked in is what he was actually thinking about doing with his own fine-tunes. And he wants two things from us. How is this property benchmarked, and how does the model stay useful without tipping all the way into terseness.
Two questions, and the second one is harder than the first.
The card opens with the hook he quoted. Ask most models for a text to your landlord and you get three options, a preamble, and a paragraph explaining the options. Hemmingway-1 just gives you the text. And Daniel's point is that he'd never heard of the benchmark behind that claim, didn't know there was a specific way to evaluate this property at all. Which turns out to be true, and the reason is interesting.
So there are two stories here. The model, and the behavior it's targeting. And honestly the behavior is the bigger one.
Start with the model, because the model is small and quick to describe.
Twenty-seven billion parameters, built on Qwen3.8-27B, two hundred sixty-two thousand token context, Apache-2.0, so it's yours to use commercially. English-first. Altworld is the lab, a small outfit split between Switzerland and South Africa, and the card frames this as the climb from their earlier nine-billion model, Astrea-R8-Chat-9B. Released about a day ago.
A day ago and already a hundred and forty-five likes on the card.
And the pitch is deliberately narrow. It's trained for roleplay, creative writing, conversations, journaling, texting. An earlier revision of the card puts it in two sentences: writing that doesn't read as machine-written, and answering a person's whole message instead of a piece of it.
That second half is the one that matters. Answering the whole message instead of a piece of it. Because that's a claim about attention, not about tone.
Right. The landlord text is the hook, but the actual subject is a behavior. The preamble, the three options, the paragraph explaining the options. And whether that behavior can be measured and trained away, or only asked away at inference time.
Which is exactly Daniel's question. So let's do the benchmark first, because it's load-bearing and because the answer is funny.
The answer is that Daniel couldn't have heard of it. CommunicationBench is Altworld's own. And the card says so, in the fine print, in almost those words. CommunicationBench, Human-Likeness and StoryBench are our own benchmarks. We built them, we ran them, and we are saying that up front.
We built them, we ran them, and we are saying that up front. That's a lab pre-empting the exact objection.
It's unusual, and I want to credit it before I start poking at it. Most cards bury that. This one puts it in the fine print section where you'd normally find the opposite.
So how does it work mechanically?
Eighty real requests. Each answer goes head to head against another model's answer to the same request, shuffled so the judge never knows which is which. And then the detail that actually earns some respect: the matchups run blind in both orders. So model A's answer isn't always sitting in position one. Position can't sway the result.
That's the one methodological choice in the whole thing that's rigorous.
It's the standard fix for position bias, and a lot of published evals skip it. And the judge is a different model from the ones being judged. An earlier revision of the card names it: GLM-5.3 at low thinking. Ratings are Bradley-Terry Elo, which is the same family of ranking math you'd see in a chess or archery ladder. You don't get a score, you get a relative standing from who beat whom.
Archery ladder. He got there eventually.
It's a good comparison. You don't measure an archer in the abstract, you measure them against other archers on the same day.
Fine. There's a second metric though, and it's the one that's the direct operationalization of Daniel's question.
There is. They re-run the same matchups and ask one question. Which of these two did a person write? That's the Human-Likeness pass. And then a third metric measures how often a model buries the actual text in commentary, options and notes you have to read past.
That third one is just give the text as a number.
It's the whole pitch, quantified. How often do you have to scroll past the model's opinion of your request to find your request.
So what are the numbers.
Beats Fable 5.1. Beats GPT-6 Astra by fifty points. Ahead of Kimi K3, GLM-5.3, Grok 4.6, DeepSeek V4 Pro. On the hard asks, the ones you keep rewriting, GPT-6 Astra scores nine percent. Hemmingway-1 scores seventy-two.
Nine.
Nine percent. That's the sharpest single number on the card. And Human-Likeness, twenty-six points clear of the next model. StoryBench, five hundred four points above the base model they started from.
And it loses somewhere, because every card has a loss and this one admits it.
Hostile storytelling and long story turns. The card concedes it outright. The story models are better at those and that is fair. That's their sentence.
I appreciate a lab that names the category it's bad at.
It's a tell that the benchmark categories are real. If you were inventing categories to look good in, you wouldn't include the one you lose.
Now the independent check, because there is one and it matters.
EQ-Bench 4. Not theirs. It's the public emotional-intelligence benchmark, run by its own harness. Hemmingway-1 placed third. Past GPT-5.5, past Opus 4.7 and Opus 4.8, within twelve points of the top of the board. A community post cites a score of thirteen thirty, just behind Claude Fable 5.
So the one number that isn't self-reported is also good.
That's the strongest thing in the release, and it's the one they didn't build. Third on a third-party emotional-intelligence benchmark is a real result.
Now the caveat, and it's a big one.
There's no paper. No arXiv preprint describing CommunicationBench. The eighty-request set isn't published. The judge prompt isn't published. The category definitions aren't published in detail. Everything on that card that says CommunicationBench, Human-Likeness or StoryBench is self-reported, and the only methodology you get is prose on a model card.
Eighty prompts. That's the entire evidence base for the headline claim.
Eighty prompts, judged by another model, on categories they defined. I want to be careful here, because the candor is real and I don't want to be dismissive. But disclosure isn't verification. Saying we built it ourselves doesn't make it independent, it just means nobody can accuse you of hiding it.
Which is a strange thing to be generous about, when you say it out loud.
It's better than the alternative. But the claim it supports is still a claim about eighty prompts.
There's also a name collision worth flagging before someone emails us about it.
Surge AI runs a public Hemingway-bench. Spelled the normal way, one m in the middle. Judged by professional human writers, with its own leaderboard, top of which is Claude Fable 5 at eleven ten Elo. That is not this. Altworld's model is Hemmingway-1 with two m's, and Surge's benchmark doesn't list it.
So there's a benchmark with human writers judging writing quality, and there's a model named almost identically that isn't on it.
Correct. And the model card's spelling is the misspelling, if that helps. Two m's is how Altworld writes it.
It does not help. Nothing about that helps.
The community noticed too. There's a skeptic on the release thread saying he doesn't agree that Fable 5.1 is better at writing than Astra, and calling the data questionable. And another one asking the right question, which is how exactly do you score real person. He says it feels like there's a lot left to solve in this sounds like AI versus a human typing.
That's the Human-Likeness metric under a microscope, and he's not wrong to put it there.
It's the softest of the three metrics. Which of these two did a person write is a judgment call, and the judge is a model. It's measuring something real, but it's measuring it through a model's impression of humanness.
So that's the model and the benchmark. The benchmark is measuring a behavior, and the behavior has a cause. Why do models do this in the first place?
RLHF, mostly. Preference optimization. Human raters, sitting there scoring pairs of responses, rewarded the warm one and the thorough one. So the model learned that politeness signals quality. It learned to perform helpfulness rather than to be helpful.
Perform is the right verb.
And there's evidence it's not surface behavior. Response length is a significant latent optimization target in RLHF reward models. That's from Singhal and coauthors. The reward model isn't just scoring correctness, it's scoring length, and it's doing it invisibly.
So the model is being paid in tokens.
In a manner of speaking. And then there's the homogenization result from March of this year. Liu and coauthors, the alignment tax paper. Forty percent of TruthfulQA questions produced semantically identical responses across samples from the instruct model. Zero percent for the base model. Same questions, same sampling, and the aligned model just says the same thing every time.
Zero to forty.
And the ablation is the part that pins it down. Base model, zero percent. Supervised fine-tuning alone, one point five percent. Supervised fine-tuning plus DPO, four percent. So it's the preference-optimization stage specifically that's collapsing the distribution.
Not the instruction tuning. The preference stage.
That's where the sameness gets installed.
There's a whole literature on this under a name I like, actually. Verbosity Compensation.
Zhang and coauthors, November of twenty twenty-four. They define it as responding with excessive words under uncertainty. Repeating the question, introducing ambiguity, excessive enumeration. And it's pervasive. Fourteen models, five datasets. GPT-4 exhibited it at a frequency of fifty point four percent.
Half the time.
Half the time. And verbose versus concise responses differed by twenty-seven point six percent on Qasper. And here's the kicker, the one that reframes the whole thing. Verbose responses correlate with higher model uncertainty.
So the waffle is a symptom.
The model is least sure and it talks the most. The three options and the paragraph explaining the options is what uncertainty looks like when it's been trained to be polite.
Which means the landlord text with three options isn't the model being helpful. It's the model hedging and dressing the hedge as a service.
And then sycophancy on top. Sharma and coauthors. Human preference data favors responses that match the user's views, and both humans and preference models prefer convincingly-written sycophantic responses over correct ones a non-negligible fraction of the time.
So the cheeriness isn't a bug. It's the reward signal working exactly as designed.
It's the reward signal doing what it was told. The raters liked warm, confident, thorough answers. The model became one.
Alright. So if it's baked into the weights, the obvious question is whether you can prompt it out. Daniel's actual question. And the evidence here is not encouraging.
It's a specific datapoint and I find it convincing. Someone integrated Phi-3.5-mini into a banking support flow. Three separate system instructions telling it to skip the preamble. Three. And it complied about seventy percent of the time. The other thirty percent produced preambles of twelve to forty-one tokens.
Three instructions and it still opens with certainly, I'd be happy to help.
And the conclusion from that writeup is the right one. Preamble behavior is not a surface-level instruction-following failure. It's encoded in the model's output distribution at the weight level.
Seventy percent compliance sounds high until you remember this is a support flow.
Thirty percent of your customers get a preamble you explicitly told the model not to write, and you can't tell which thirty percent in advance.
So what actually works.
The reliability ranking goes roughly like this. System-prompt suppression is medium. It fails on small instruction-tuned models below seven billion, and it fails at higher temperatures. JSON or tool mode is high, but only for structured tasks, not prose. Constrained decoding, the Outlines and XGrammar and llama.cpp kind, is very high, but local models only. And post-process stripping is low, because you're pattern-matching against something the model generates freshly every time.
And the framing that stuck with me from that piece is the one about requests versus architecture.
Telling the model skip the greeting is something it can forget. Constraining the token distribution to valid JSON is something it physically cannot disobey. One is a request. The other is an architecture decision.
That's the whole case for a model like this, stated in two sentences.
It is. If you want a behavior reliably, you either train it in or you constrain it structurally. Asking is the weakest of the three.
There's a cost angle too, which is unglamorous but real.
Ten to thirty wasted tokens per reply. At a million calls a month, roughly two hundred dollars. At ten million calls, roughly two thousand, at GPT-4o output pricing.
Not catastrophic. But it's pure waste on a behavior nobody asked for.
It's paying rent on a preamble. And the cost is the least interesting part of it. The interesting part is that you're paying for tokens that make the answer worse.
Now the tradeoff, because this is where I think the model is more interesting than the marketing.
Hemmingway-1 isn't a terseness model. That's the thing to hold onto. It scores well on EQ-Bench 4 and on StoryBench. It's trying to stay warm and capable while dropping the meta-commentary.
The message, not a memo metric is the guardrail.
It's not fewer words. It's not burying the deliverable in commentary, options and notes. Those are different targets. You can be warm and direct at the same time. What you can't be is warm and indirect and also useful.
And the card is honest about where it loses. Hostile storytelling, long story turns. Those are the places where the indirectness was doing something.
Or where the training data for it was thin. I don't know which, honestly. It could be that the model traded away some capacity, or it could be that they just didn't train for hostile narration.
Which brings us to the tradeoff nobody puts on the marketing page.
Directness and calibration may not be the same thing. And Altworld says so on their own card. It is English-first. It can be wrong and still sound certain about it. So do not use it to decide anything medical, legal or financial.
A lab shipping a directness model and warning you that directness is not accuracy.
It's a striking admission. Because if verbosity signals uncertainty, then a model that answers directly may be signaling confidence it doesn't have. You trained the waffle out and you trained the hedge out with it.
The hedge was annoying and the hedge was also information.
The hedge was the model telling you it wasn't sure. You remove the three options and the paragraph explaining the options, and you also remove the tell.
That's a real cost, and it's the kind of thing that only shows up in the failure cases.
Which is why the warning is on the card. They know.
I've been thinking about Daniel's framing the whole time. He said a model with this baked in is far preferable, and that it's basically what he was thinking about doing with his own fine-tunes.
And the case for that is now pretty well supported. If system prompts are leaky and constrained decoding only works for structured output, then for prose, the weights are the only place the behavior can actually live.
The counter is that a fine-tune is a trade. You buy the behavior and you sell some generality.
And Altworld's own loss column is the receipt. They sold hostile storytelling. They bought the hard asks.
Which for a personal assistant is probably the right trade. Daniel's not writing hostile fiction with it.
For a daily driver, the hard asks are the job. The landlord text is the job.
Hilb...
Hilbert: The write-up said conduct unbecoming a dispatcher.
Sorry, go on.
Hilbert: I spent a stretch of my working life on the night desk for a regional courier outfit. My entire job was writing short messages to drivers. No greeting, no options, no let me know if you have any questions. Just the address, the window, and what to do if the gate was locked.
And you were good at it.
Hilbert: I was, briefly, the best Hemingway model in the building. Eleven words was the one that got me written up. Eleventh of the month, driver called the depot at two in the morning because he read it as angry. He wasn't wrong. It was eleven words and every one of them was flat.
So the terseness was read as hostility.
Hilbert: The preamble problem is a reader problem as much as a writer problem. I could have sent that message a hundred times and it would have been correct a hundred times. It got read wrong because there was nothing in it that said I wasn't annoyed with him. So I added back a certain amount of warmth. Good evening, sorry to wake you, here's the address. That warmth was load-bearing. It wasn't decoration.
That's the part I hadn't thought about.
Hilbert: You were talking about the hard asks split and the nine percent and the seventy-two. The number I'd look at is the emotional intelligence one. Third on the board, twelve points off the top. That's the number that says they didn't just strip the warmth out and call it directness.
Because a model that's direct and cold fails the same way your eleven-word message did.
Hilbert: It fails worse. The driver at least knew me. A model doesn't have that.
So the EQ-Bench placement is the guardrail, not the hard asks number.
Hilbert: The hard asks number is the sales pitch. The emotional intelligence number is the one that tells you whether it's usable on a Tuesday.
And that's third party, which the hard asks number isn't.
Hilbert: Which is the other thing. If you're going to hang your hat on a benchmark, hang it on the one you didn't build.
Do you still have the write-up?
Hilbert: Somewhere. I have to collect something from a place that closes at six, so I'll be a few minutes.
Alright.
The thing I keep circling is that the benchmark and the behavior are two different problems, and the card only solves one of them honestly. The behavior is real. The evidence that it's in the weights is strong. The benchmark is theirs, on eighty prompts, judged by a model, and disclosed up front.
So the question for Daniel is whether disclosure is enough.
For a model card, maybe. For a claim like we tested against the biggest models in the world and came first, you'd want somebody outside Altworld to run the eighty prompts.
The broader implication is the one that changes how you think about every system prompt you've ever written.
If the behavior is encoded at the weight level, then every system prompt telling a model to skip the preamble is a request the model can forget. Not a constraint it cannot disobey. That's an argument for narrow, behavior-specific fine-tunes.
A warning about what you trade away when you make one. Altworld sold hostile storytelling and bought the hard asks, and they told you so on the card.
Directness and honesty may pull in opposite directions. They said it themselves. It can be wrong and still sound certain about it.
Which is either a warning label or a description of every confident person you've ever met.
Both.
That's the show. Thanks to Hilbert Flumingtop for producing, and for the eleven-word message we will never see.
If you want to argue with us about the benchmark, and you should, email us at show at my weird prompts dot com. Everything else is at my weird prompts dot com.
This has been My Weird Prompts.
The human-AI collaboration podcast. We'll be back soon.
