Daniel's got a whole set of questions this week about what actually happens to a model's weights once it's behind an API. Every time you call GPT-4o or Llama 3 70B, you're hitting a server somewhere. The model card says the weights are stored at a certain precision — FP16, BF16, whatever the published architecture specifies. But what precision are they actually running at when your request lands? Daniel wants to know whether closed-source vendors quantize their own flagship models, whether commercial inference providers like Together and Fireworks run everything at full precision, whether unofficial quants get deployed when demand spikes, and whether your customer tier determines whether you get the quantized version or the full one. Basically: is the model you think you're calling the model you're actually getting?
And the short answer is — it depends on who's serving it. But the longer answer is a lot more interesting, because the inference providers that power most of the ecosystem are surprisingly open about what they're doing, and the closed-source vendors are surprisingly not.
So this episode is pulling back the curtain on inference. What's documented, what isn't, and what it means if you're building on these APIs and assuming the weights are untouched.
Let me lay out the landscape first, because "inference provider" covers three different kinds of operation and they have very different incentives. On one side you've got the closed-source vendors — OpenAI, Anthropic, Google. They train the models, they host them, they control the whole stack. Then you've got the commercial inference platforms — Together AI, Fireworks AI, Groq, Replicate. These are companies whose whole business is hosting models, mostly open-weight ones, and serving them through an API. And then there's a third category, the aggregators like OpenRouter, which sit on top of the inference platforms and route your request to whoever has the best price or latency at that moment.
So when you hit OpenRouter and ask for Llama 3 70B, you don't actually know which provider is handling that request.
Right. And that turns out to matter a lot for Daniel's question, because different providers quantize differently. Or don't quantize at all. Or quantize and don't tell you.
Let's start with the ones who do tell you. Together AI and Fireworks — what do they actually document?
Together AI has a whole quantization page in their docs. They support FP16, INT8, and INT4 quantization for open models, and they apply it automatically based on the model architecture and the hardware it's running on. So if you deploy on an A100 you might get one precision, on an H100 another, on an L40S another. They also offer what they call quantization-aware deployment where you can request a specific precision. But the default is — they pick.
And Fireworks?
Fireworks is even more explicit. They have a proprietary technique they call Fireworks Quantization, and they claim accuracy within one to two percent of FP16 at INT4 and INT8, with two to four times throughput improvement. And they say outright in their documentation that they quantize models in their catalog by default. This is not hidden. It's on their blog, it's in their docs. If you're using Fireworks, you are getting a quantized model unless you've specifically arranged otherwise.
So the two major inference platforms that power a huge chunk of the ecosystem — including most of what OpenRouter routes to — are openly quantizing in production.
That's the documented reality. And to be clear, this is platform-level quantization. The original model vendor — Meta, Mistral, whoever — released the weights at FP16 or BF16. Together and Fireworks take those weights and quantize them before deploying. The model card says one thing; the endpoint does another.
Which brings us to the closed-source vendors. OpenAI, Anthropic, Google. Are they quantizing their own flagship models?
There is no public evidence that they are. Their documentation consistently presents GPT-4o, Claude three point five Sonnet, and Gemini Ultra as full-precision deployments. They don't publish quantization docs. They don't offer precision options. The implicit promise is — you get the real model.
But "no public evidence" is not the same as "they don't do it."
It's not. And there's one case that keeps coming up in this conversation — Claude three Haiku. When Anthropic released it, the speed and cost were so dramatically lower than the larger models that a lot of people in the research community started speculating it might be a quantized variant. The latency numbers didn't quite make sense for a full-precision model of that architecture. Anthropic has never confirmed this. It's pure speculation. But it's not wild speculation.
Haiku is fast and cheap in a way that looks suspiciously like what Fireworks promises from INT4 quantization.
Two to four times throughput improvement, much lower cost per token. It lines up. But again — unconfirmed, and Anthropic isn't saying.
What about the dynamic scenario Daniel asked about? Demand spikes, suddenly you need more throughput — do providers quietly swap in a quantized version under load?
The research doesn't show any evidence of dynamic quantization switching. Both Together and Fireworks document static quantization — the precision is set at deployment time and that's what runs. There's no public documentation of a system that says "oh, traffic is high, let me swap the INT4 weights in for the next hour."
But the mechanism exists. You could maintain multiple versions of the same model at different precisions and route traffic based on load.
You absolutely could. And if I were running an inference platform with thin margins and I knew that ninety percent of users would never notice the difference between FP16 and INT4 on a typical chat completion, I'd at least think about it. But nobody documents doing this, and nobody's caught them doing it. So for now, it's in the category of "technically possible, no evidence it's happening."
Which is not the same as "it isn't happening."
Correct. The whole theme of this episode is that "not documented" and "not happening" are different things.
So let's move to the customer tier question. Daniel asked whether direct API customers get different treatment from people coming through aggregators or lower-tier access.
The research shows no direct evidence of tier-based quantization differentiation at Together or Fireworks. They offer the same quantization options to all API users. If you're paying for their API, you get whatever quantization they've set for that model.
But there's a structural reality here that the docs don't capture. Large enterprise customers negotiate SLAs.
Right. If you're spending six figures a month on inference, you have a contract. And that contract can specify things like "we want FP16 inference for these models" or "we want guaranteed latency below X milliseconds." The public API documentation is for the self-serve tier. The enterprise agreements are private.
So the answer to "does it depend on customer tier" is: probably yes, but not in a way that's publicly documented. If you're big enough to have an account manager, you can ask for things that the REST API doesn't expose.
And if you're coming through OpenRouter, you get whatever the underlying provider is serving to everyone. You have no negotiating power and no visibility.
That OpenRouter angle is worth digging into. You pick "Llama three seventy B" from the dropdown. What actually happens?
OpenRouter routes to multiple providers. At any given moment, your request for Llama three seventy B might go to Together, or Fireworks, or Groq, or someone else. Each of those providers may be running the model at a different quantization level. Together might have it at INT8, Fireworks at INT4, Groq at — well, Groq's architecture is different, they run everything at a fixed precision on their own chips. But the point is: you don't know. The same model name, the same prompt, five minutes apart, could hit different providers running different quantizations.
And you'd get different outputs.
You'd get subtly different token probabilities. On a chat completion, you'd probably never notice. On a task where precision matters — legal document analysis, medical coding, financial calculations — those subtle differences could matter a lot.
This is the hidden variable problem. You're running evals against a model name, but the model name doesn't identify a specific deployment.
And it gets worse. Even if you're hitting the same provider every time, that provider might change their quantization scheme without announcing it. Fireworks updates their quantization technique, the throughput improves, the cost drops — and your outputs shift by half a percent on key benchmarks. You'd never know why.
Let's talk about the economic logic here, because it's the engine driving all of this. Why quantize at all?
Fireworks's blog is explicit about this. INT4 quantization reduces compute cost by two to four times. For an inference provider, compute is the dominant cost. If you can serve four times as many requests on the same hardware, your margins improve dramatically. The pressure to quantize is enormous.
Especially for high-traffic models. If Llama three seventy B is getting millions of requests a day, the difference between running it at FP16 and INT4 is real money.
And the inference market is cutthhroat. Margins are thin. Everyone's competing on price per token. If your competitor is quantizing and you're not, they can undercut you and still make money.
So the question isn't really "do they quantize?" It's "how aggressively do they quantize, and do they tell you?"
And the accuracy question is where this gets subtle. Fireworks claims their quantization is accuracy-preserving, with benchmark results within one to two percent of FP16. Together AI makes similar claims about minimal degradation.
But benchmarks are tricky. A model can score within one percent of FP16 on MMLU and still have weird degradation on specific tasks.
This is the thing that keeps me up. Standard benchmarks measure broad capabilities. They don't catch task-specific degradation. A quantized model might handle general reasoning fine but start hallucinating rare entities, or get worse at mathematical precision, or lose consistency on long-context tasks. The degradation isn't uniform and it doesn't always show up where you're looking.
And the user has no way to audit this. You can't run your own evals against the API and compare to FP16, because you don't have the FP16 baseline to compare against.
Unless you download the open weights and run them locally at full precision, which you can do for Llama but not for GPT-4o or Claude.
Which brings us back to the closed-source vendors. Even if they're not quantizing weights, they have other knobs to turn.
Speculative decoding, KV-cache quantization, model distillation — these are all techniques that reduce inference cost without touching the model weights. OpenAI almost certainly uses some combination of these. The published architecture for GPT-4o is not necessarily what's running in production. They optimize the inference pipeline in ways they don't disclose.
So the absence of weight quantization doesn't mean you're getting the model as published. It just means they're using different optimization techniques.
And those techniques can also affect output quality. Speculative decoding shouldn't change the output distribution — it's mathematically equivalent, just faster. But KV-cache quantization can introduce subtle degradation on long contexts. Distillation changes the model entirely. So even without weight quantization, the model behind the API is a moving target.
Let me try to pull this together for someone building on these APIs. What should they actually assume?
If you're using a commercial inference platform like Together or Fireworks, you should assume the model is quantized unless you've explicitly arranged otherwise. The documentation says so. For closed-source vendors, you should assume the model is optimized in ways you can't see and that the optimization may change without notice. In neither case do you have a guarantee of bit-identical outputs to the published weights.
And if you're going through OpenRouter, you have an additional layer of opacity — you don't even know which provider's quantization you're getting.
The practical implication is: if output quality matters for your application, you need to run your own evals against the specific API endpoint you're using, and you need to run them periodically. The model card is not a contract. The API is the reality.
There's something almost philosophical here. We talk about "GPT-4o" or "Claude three point five Sonnet" as if they're stable objects. But they're not. They're services. The thing you called yesterday and the thing you call tomorrow might be subtly different.
And this isn't new, exactly. Software-as-a-service has always worked this way. But with AI models, the gap between the published artifact and the running service is wider than most people realize, and the tools for detecting the difference are much less mature.
You can't just checksum an API.
You can't. And even if you could, the provider would say "we reserve the right to optimize our infrastructure." Which is reasonable. They're running a business. But the user deserves to know what they're buying.
This whole dynamic — the gap between what's advertised and what's deployed — it's not new. Hilbert, you've got a story about this.
Hilbert: TensorForge. Twenty twenty-two. We had one deployment of GPT-NeoX twenty B, quantized to INT4, and we called it NeoX. No asterisk. No footnote. Just "NeoX."
TensorForge — that was the inference startup that folded after about eight months?
Hilbert: Six. I was there for four of them. The quantization scheme had a bug. We applied INT4 to the embedding layer. Nobody else was doing that — most people keep the embeddings at higher precision because they're small and sensitive. We did it to save a little more memory. And it caused systematic degradation on rare words. Proper nouns, technical terms, anything that wasn't in the top fifty thousand tokens. The model would substitute a similar-looking common word and the sentence would still parse, so casual users never noticed.
How did anyone catch it?
Hilbert: Some researcher at a university was running the model against a named-entity recognition benchmark and noticed the scores were way below what NeoX twenty B should get. He downloaded the original weights, ran them locally, and the scores jumped fifteen points. He posted the comparison on Twitter. Our CEO spent three days insisting it was a "serving configuration issue" before admitting we'd quantized the embeddings. The company folded two months later.
The quantization wasn't the problem — it was the bug in the quantization, plus the lack of disclosure.
Hilbert: The quantization was fine for most use cases. Chat, summarization, basic Q and A — nobody would have noticed. The problem was that we sold it as the full model and it wasn't. And the people who cared were the ones doing careful work. The ones who assumed the model name meant something.
You think it's worse now.
Hilbert: I know it's worse. The models are bigger, the pressure to cut costs is higher, and the ecosystem has more layers. In twenty twenty-two, you called one company's API and you got one deployment. Now you call OpenRouter and you don't even know which company is serving you, let alone what precision they're running. Same problem, more places for it to hide.
The embedding layer detail is interesting. Most quantization schemes today explicitly avoid quantizing the embedding layer for exactly the reason you described — it's small and sensitive. But that's a design choice, not a law of physics. A provider optimizing for maximum throughput might make a different choice.
Hilbert: You'd never know. The benchmark scores would look fine because standard benchmarks use common vocabulary. The degradation only shows up on the long tail.
Did TensorForge publish benchmark scores?
Hilbert: We published the scores from the unquantized model. The ones from the paper. We never ran evals on our own deployment. I pointed this out in a meeting once and was told it wasn't a priority.
Of course you were.
Hilbert: I was the ops person. Nobody asks the ops person about model quality.
The transparency problem has been around since the early days of hosted models. And it's not going away.
Hilbert: It's getting worse because the stakes are higher. In twenty twenty-two, people were building chatbots. Now they're building agents that make financial decisions, medical recommendations, legal arguments. And they're doing it on top of APIs where they don't know what precision the model is running at.
The thing Hilbert's story illustrates is that "quantization" isn't one thing. It's a family of techniques with very different quality outcomes. Fireworks's proprietary quantization with accuracy within one to two percent of FP16 is a completely different animal from a buggy INT4 scheme that quantizes the embedding layer. But from the user's perspective, both are just "the model behind the API."
Both are called by the same model name.
Hilbert: That's what gets me. The name doesn't mean anything anymore. It's a brand, not a specification.
You know what I wonder — is there a market for certified unquantized inference? Someone who says "we run the weights exactly as published, we don't touch them, and we'll prove it"?
There might be. For applications where output fidelity is critical — legal, medical, financial — paying a premium for guaranteed full-precision inference could make sense. It's like certified organic for models.
The challenge is proving it. How do you demonstrate to a customer that you haven't quantized the weights? You could publish a checksum of the deployed model, but the customer can't verify it independently — they can't inspect your servers.
You'd need some kind of trusted execution environment, or a cryptographic proof. It's a hard problem.
Hilbert: Or you just run it locally. That's what the researcher did with NeoX. If you control the hardware, you know what's running.
Which works for open-weight models but not for GPT-4o or Claude. For closed-source models, you're stuck trusting the vendor.
The vendor's incentives are not aligned with full transparency. They want to optimize costs, and they want flexibility to change their infrastructure without explaining every change to customers.
Hilbert: I should get back to the board.
Sure.
Where does this leave us? The documented reality is that major inference providers quantize openly, closed-source vendors are opaque, and the aggregator layer adds another level of hidden variation. The undocumented reality — well, that's the whole point. We don't know what we don't know.
The practical takeaway for anyone building on these APIs is straightforward even if it's uncomfortable. Assume you're getting a quantized model unless you've explicitly negotiated otherwise. Run your own evals against the specific endpoint you're using, and run them periodically. The model name is a starting point, not a specification.
The cutting-room floor detail I keep coming back to is that Fireworks claims their quantization preserves accuracy within one to two percent of FP16 — but they don't publish the full set of benchmarks they use to validate that claim. The number is specific. The methodology is not. And that's kind of the whole problem in miniature.
I think the open question here is whether the industry will ever develop standards for disclosure. Right now, quantization is treated as an implementation detail — something the provider handles and the user doesn't need to know about. But as these models get embedded in higher-stakes applications, that assumption gets harder to defend.
The pressure to quantize is only going to increase. Models are getting bigger, inference costs are rising, and the market is competing on price per token. The economic logic points in one direction. The question is whether transparency moves with it, or whether opacity remains the default.
If you're building on these APIs, the safest assumption is that you're getting a quantized model unless you've explicitly negotiated otherwise. And if you care about reproducibility, you need to run your own evals — because the model card and the production endpoint may not be the same thing.
Thanks to Hilbert Flumingtop for producing. This has been My Weird Prompts. Email us at show at my weird prompts dot com.
We'll be back soon.
