The thing about Hugging Face is, it solved discovery before it solved delivery. You can find a model for anything — text-to-SQL, protein folding, detecting whether a photo contains a hot dog — but the moment you click that model card, you're staring at a decision tree that nobody really walks you through.
Daniel's been poking at the two main pathways for actually running models from the Hub — local inference versus cloud — and he wants to get into the mechanics, not the marketing. On the local side, there's the compatibility tracker that checks your hardware, calculates TOPS, and tells you whether a model will run. If it will, then what? Downloading the weights, where they end up on your disk, how the cache is structured, and how you clean it up when you've accumulated forty-seven variants of Llama. On the cloud side, he's asking about two distinct routes — Hugging Face as a gateway that routes your request to a backing provider, versus calling the API directly where Hugging Face hosts the model themselves. Both use the same Python package, but the routing underneath is completely different. And he specifically doesn't want us to get bogged down in subscription tiers.
Smart. The tiers change every six months anyway.
So where do we start?
Let's start where most people start — staring at a model card wondering if your laptop can even run it.
The moment of truth. You've found the perfect model, the benchmark numbers look great, and then you scroll down and see "requirements: unspecified."
And that's exactly what the compatibility tracker is supposed to solve. Hugging Face has over half a million models now, and the model card is really just the storefront. It tells you what the model does, what it was trained on, maybe some evaluation scores. But the actual question — "can my machine run this?" — that's a hardware compatibility problem, and it's a lot more nuanced than just checking how much VRAM you have.
Most people think it's a VRAM check. You've got twelve gigs, the model needs fourteen, answer's no. Done.
And that's the first misconception worth busting. The compatibility tracker doesn't just look at VRAM. It's calculating TOPS — trillion operations per second — which is a measure of your hardware's raw compute throughput. The model has a TOPS requirement based on its architecture, parameter count, and expected precision. Your GPU has a TOPS rating. The tracker maps those against each other. But it's also looking at memory bandwidth — how fast can you shuttle those weights between VRAM and the compute units? A model might technically fit in your twelve gigs, but if your memory bandwidth is too low, inference will be painfully slow.
So it's not a binary "will it run." It's more like "will it run at a speed that doesn't make you want to throw your computer out a window."
And the tracker surfaces that. You input your hardware specs — GPU model, VRAM, system RAM, CPU if you're doing CPU inference — and it gives you a compatibility score, not just a yes or no. It'll say something like "this model will run with 4-bit quantization at approximately fifteen tokens per second on your RTX 3060." That's actionable.
And if the answer is yes — or yes with caveats — the next step is actually getting the weights onto your machine.
Right. There are two main ways to download weights from the Hub programmatically, both through the huggingface_hub Python library. The first is from_pretrained() — you call it in your code, point it at a model repo, and it handles the download automatically. It's lazy, meaning it only pulls down what it needs when it needs it. The second is snapshot_download(), which downloads the entire model snapshot at a specific revision.
And where do those files actually land?
This trips people up because it's not obvious. The default cache directory on Linux and macOS is ~/.cache/huggingface/hub. On Windows, it's %USERPROFILE%\.cache\huggingface\hub. And the structure inside is clever. Each model gets a folder named by its repo ID, with slashes replaced by double dashes. So meta-llama/Llama-2-7b-hf becomes models---llama--Llama-2-7b-hf.
That's... a choice.
It is, but it keeps things flat and predictable. Inside that folder, you've got two things. There's a blobs directory — stored at the hub level, not per-model — and a refs directory. The blobs are the actual file contents, stored by their SHA-256 hash. The refs folder maps branch names and tags to specific commits, which are essentially pointers to sets of blobs.
So it's content-addressable storage. You don't store the same file twice even if two different models share it.
And that's huge. If you've got five different fine-tuned variants of Llama 2, they all share the same base layers. Those blobs get downloaded once. The refs just point to them from different model folders. This is symlink-based deduplication — it's not copying files, it's creating references.
Which means your disk usage is a lot lower than the sum of all the model cards would suggest.
In theory. In practice, people end up with bloated caches anyway because they never clean up old revisions. Every time you pull a model at a different commit, you get a new snapshot. The old one doesn't go anywhere unless you explicitly remove it.
So how do you actually manage this? If I've been downloading models for six months and my cache is sixty gigs, what do I do?
The huggingface_hub library has a whole cache management API. The entry point is scan_cache(). It returns a list of CachedRepoInfo objects — one per repo you've ever downloaded. Each tells you the total size on disk, last access time, and how many revisions are stored. From there, you've got delete_cache() which lets you remove models by repo ID, age, or strategy — "delete the oldest ones first," that kind of thing. And then there's clean_cache(), which is the one everyone should be running and nobody does. It removes revisions that aren't pointed to by any ref anymore — stale branches, deleted tags, orphaned data.
So clean_cache() is the garbage collector.
And it's safe. It won't touch anything still referenced. You can run it whenever and it just sweeps up the cruft.
But here's the catch — deleting a model doesn't always free as much space as you'd expect.
Right, and this is the second big misconception. When you delete a model from the cache, the blobs unique to that model get removed. But any blobs shared with other models stay. So you delete a Llama fine-tune and expect to get back, say, thirteen gigs. But if you've still got the base Llama model in your cache, most of those blobs are shared. You might only free up a few hundred megabytes of the adapter weights.
The scan_cache() output actually breaks this down — it shows "shared" versus "unique" size.
It does. And that's the number you should be looking at, not the total. The total size is misleading because a chunk of it is probably being used by three other models.
Alright, let's make this concrete. I've got an RTX 3060 with twelve gigs of VRAM. I want to run Llama 2 7B. I punch my specs into the compatibility tracker. What does it tell me?
At full FP16 precision, Llama 2 7B needs about thirteen gigs just for the weights — before the key-value cache overhead during inference. The tracker would tell you it doesn't fit at FP16. But you're not going to run it at FP16. You're going to quantize it to 4-bit.
And quantization is a separate optimization step. The tracker doesn't automatically account for it.
It doesn't, and this is worth flagging because it's a gap in the workflow. The tracker checks the model's published requirements, which assume standard precision. If you're planning to quantize, you have to do that math yourself. A 7B model at 4-bit drops from roughly thirteen gigs to somewhere around four to five gigs. That fits comfortably in twelve gigs of VRAM with room for context.
So the tracker says no, but the real answer is yes if you know what you're doing.
And that's the friction. The tracker is a first-pass filter, not a definitive answer. It's useful for catching obviously impossible combinations — trying to run a 70B model on a laptop with eight gigs of RAM. But for the edge cases, you need to understand what quantization buys you and how to apply it.
Let's say I've done that. I've quantized, I've downloaded, I'm running inference locally. What does that actually mean in terms of the pipeline?
Your inference framework — usually Transformers or something built on top of it — loads the model into GPU memory, tokenizes your input, and runs it through the layers. Each layer is a series of matrix multiplications. The weights stay in VRAM the whole time. Your input tokens flow through, the attention mechanism does its thing, and you get output tokens one at a time. The key thing about local inference is that once the model is loaded, there's no network round trip. Your latency is purely compute-bound.
No network round trip, and no per-token billing.
You paid for the hardware upfront, and the marginal cost of each token is electricity. Local inference is capital expenditure heavy and operational expenditure light. Cloud inference is the reverse.
Which brings us to the other half of Daniel's question. If you don't want to manage hardware, don't want to think about quantization, don't want to clean up cache directories — you go cloud. But there are two different cloud routes, and they're not the same thing.
They're not, and conflating them is the third big misconception. The Hugging Face Inference API and Inference Endpoints sound like the same product. They both use the InferenceClient in Python. But the routing underneath is completely different.
Walk me through route one — Hugging Face as a gateway.
Route one is the Inference Endpoints product. You create an endpoint, pick a model, choose a backing provider — AWS, Azure, a dedicated GPU cloud provider — and Hugging Face deploys the model on that provider's infrastructure. When you send a request, it goes to Hugging Face first, then Hugging Face routes it to the provider, the provider runs inference, sends the result back to Hugging Face, and Hugging Face sends it to you. There's an extra hop in there.
So Hugging Face is acting as a load balancer and API gateway, but the actual compute is happening on someone else's metal.
Right. The advantage is model coverage. Because Hugging Face isn't hosting the model directly, they can support any model on the Hub as long as a provider is willing to run it. You're not limited to a curated list. The downside is that extra hop adds latency, and you're paying both Hugging Face and the provider — it's bundled into one price, but the cost structure reflects the middleman.
And route two — the direct API.
Route two is the hosted Inference API. Hugging Face runs the model on their own infrastructure. No third-party provider. Your request goes from your code to Hugging Face's servers, inference happens, the result comes straight back. It's simpler, there's less latency, and the free tier covers a decent amount of usage for smaller models.
But you're limited to models Hugging Face has chosen to host.
Correct. They pre-deploy a selection of popular models. If the model you want isn't on that list, you can't use the direct API — you'd have to go the Endpoints route or run it locally. The direct API is curated; Endpoints are open.
And both of these use the exact same Python interface.
Same InferenceClient. Same method calls. text_generation(), image_classification(), automatic_speech_recognition() — the API surface is identical. The difference is what happens after you hit "send." With the direct API, your call goes to a fixed endpoint that HF maintains. With Endpoints, your call hits a load balancer that picks a provider based on availability and latency, then forwards the request.
So from a code perspective, switching between them is basically changing a URL or an endpoint name.
It's changing one parameter when you instantiate the client. The library abstracts the routing completely. And that's both a strength and a trap — it's easy to use, but it's also easy to not understand what you're actually paying for or where your data is going.
What about the practical differences? If I'm calling InferenceClient().text_generation('-llama/Llama-2-7b-chat-hf') through the direct API versus setting up an Inference Endpoint for the same model, what am I actually experiencing differently?
Latency first. The direct API is generally faster because there's no intermediary hop. With Endpoints, you've got that routing layer — it's usually tens of milliseconds, but it's there. And if you're doing streaming, those tens of milliseconds per token add up. Cost is the bigger difference, though. The direct API has a free tier for smaller models and then usage-based pricing for PRO subscribers. Endpoints are billed by the hour that the endpoint is running, regardless of how many requests you send. If you need a model available twenty-four seven with guaranteed capacity, Endpoints is the way to go. If you're just prototyping or sending occasional requests, the direct API is dramatically cheaper.
And model availability — the direct API has maybe a few hundred models pre-deployed. Endpoints can run anything.
Anything with a compatible architecture that a provider is willing to host. In practice, that's most of the popular model families. But there's another layer worth mentioning — the free Inference API for small models is rate-limited and runs on shared infrastructure. You might get queued if demand is high. Endpoints give you dedicated resources.
So the decision tree is: if your model is on the curated list and your usage is low to moderate, direct API. If your model isn't on the list or you need guaranteed throughput, Endpoints. If you want zero per-token cost and you've already got the hardware, local.
That's the framework. And you can mix and match. Prototype on the direct API, deploy to an Endpoint when you need production reliability, and keep a quantized local copy for development and testing. The InferenceClient doesn't care which backend it's talking to.
The portability between these pathways is underrated. You write the same code, you point it at different infrastructure. That's the promise of the Hub model — the model is the interface, and where it runs is an implementation detail.
Until it isn't. There are edge cases. Some models use custom inference code that only works in specific environments. Some providers don't support certain model architectures. The abstraction leaks. But for the vast majority of models on the Hub — the Transformers-compatible ones — it really is that portable.
Let's talk about what happens when the abstraction leaks in the other direction — locally. You've downloaded a model, it's in your cache, you're running inference. Three months later, you've got forty models and your disk is full. What does the cleanup workflow actually look like?
You start with scan_cache(). That gives you a report — every repo, its size, when you last touched it, how many revisions. From there, you make decisions. Maybe you delete anything you haven't accessed in sixty days. Maybe you delete specific models you know you're done with. The delete_cache() function takes a strategy parameter — you can pass it a repo ID, a list of repo IDs, or a pattern.
If you just want to be aggressive about it, clean_cache() handles the orphaned revisions.
That's the low-hanging fruit. Run clean_cache() first, because it's safe and it'll probably free up a surprising amount of space. Then run scan_cache() again to see what's left, and make decisions from there.
I'm imagining someone running scan_cache() for the first time and discovering they've got eighty gigs of models they forgot they downloaded.
That's basically every machine learning engineer's laptop. The cache is invisible until it's a problem. It's in a dot-directory, it doesn't show up in your normal file browsing, and the files have hash names so you can't even tell what they are by looking at them. The cache management API is the only sane way to deal with it.
Unless you're the kind of person who just nukes the whole directory and starts over.
Which works, but then you have to re-download everything you actually wanted to keep. And if you're on a metered connection or you've got bandwidth caps, that's painful.
Alright, so we've covered the local pathway — compatibility tracker, downloading, cache structure, cleanup. We've covered the two cloud routes — direct API versus Endpoints, same interface, different routing. What's the thing most people get wrong about all of this?
I think the biggest thing is assuming there's one right answer. The pathways aren't competitors — they're different points on a spectrum of control versus convenience. Local gives you total control and zero per-token cost, but you manage everything. Direct API gives you zero infrastructure overhead but limited model selection. Endpoints give you any model with guaranteed capacity, but you're paying by the hour. The right choice depends on where you are in your project's lifecycle.
The lifecycle thing matters. I've watched people spin up an Endpoint for a model they're just evaluating, burn through a hundred dollars in a week, and then realize they could have done the same evaluation on the direct API for free.
Or locally. If you've got a GPU that can handle a quantized version, your evaluation cost is literally zero beyond the electricity. The compatibility tracker should be the first thing you check, not the last.
There's also a psychological dimension to this. When you're running locally, the model feels like a tool. It's on your machine, it's yours, you can modify it, you can break it, you can poke at its internals. When you're calling an API, the model feels like a service. It's someone else's thing that you're renting access to. Those are different relationships to the technology, and they shape how you use it.
That's... actually not something I'd considered, but you're right. The local pathway invites experimentation in a way that metered API access doesn't. If every request costs money, you think twice before sending a weird prompt just to see what happens. If it's running on your own GPU, the weird prompts are free.
Weird prompts are kind of our whole thing.
They really are.
To Daniel's core question — the two pathways, how the mechanics differ — I think the answer is: they differ at every level except the code you write. The interface is the same. Everything underneath is different. Where the model lives, who owns the hardware, how you pay, how you clean up, what happens when something goes wrong.
The compatibility tracker is the fork in the road. It's the thing that tells you whether the local pathway is even an option. If it is, you've got a decision to make. If it isn't, cloud is your answer, and then the question becomes direct API versus Endpoints.
One thing we haven't touched on — and this is where I think things are heading — is the blurring of these lines. Split inference, where part of the model runs locally and part runs in the cloud. Or speculative decoding where a small local model generates candidate tokens and a large cloud model verifies them. The pathways are starting to merge.
That's the open question, isn't it? As models get larger and quantization gets better, does local become more viable for more people? Or does the convenience of cloud — no downloads, no cache management, no GPU driver headaches — win out for everyone except the enthusiasts?
I think it depends on whether the hardware keeps up. Right now, a 7B model at 4-bit runs on a consumer GPU. A 70B model at 4-bit needs something closer to a workstation. If quantization techniques keep improving and hardware keeps getting cheaper, the crossover point moves. More models become locally runnable for more people.
But at the same time, the cloud providers aren't standing still. Latency keeps dropping. The free tiers keep getting more generous. The curated model lists keep growing. The gap between "it runs locally" and "it runs conveniently in the cloud" might actually be narrowing from both sides.
Which means the right answer today might not be the right answer in eighteen months. And that's fine. The point is understanding the mechanics well enough to make the call for yourself.
Hilbert: You know what nobody ever talks about with that cache system? The symlinks.
The symlinks?
Hilbert: The way the blobs are stored by hash. I managed a GPU cluster at a university lab — this was around twenty nineteen, eight A100s, a queue system that made everyone furious — and we had a ten-terabyte NAS that the grad students kept filling up. They'd download every variant of every model. Bert base, Bert large, Bert base uncased, Bert base cased, and then five different fine-tunes of each. The NAS would hit ninety-five percent and the jobs would start failing.
The symlinks were the problem?
Hilbert: They were the thing that made the problem invisible. You'd look at a model folder, it says thirteen gigs. You delete it. You get back maybe two gigs. Because the other eleven gigs of blobs are still being referenced by three other models. The scan_cache() output shows it — the shared versus unique breakdown — but nobody ever ran scan_cache(). They'd just delete folders in the file browser and then wonder why their disk was still full.
What did you do?
Hilbert: Wrote a cron job. Every Sunday at 3 a.m., it ran scan_cache(), calculated how much space each user's models were taking, and if a user was over eighty percent of their quota — we gave everyone five hundred gigs — it emailed them a report. Listed every model, when they last accessed it, how much was unique versus shared. Told them to run clean_cache() and then delete_cache() on anything older than thirty days.
Did they actually do it?
Hilbert: About half of them. The other half ignored the emails until their jobs started failing, and then they'd file a ticket saying the cluster was broken.
You were basically running tech support for a cache management API that already had all the tools built in.
Hilbert: The tools were there. The knowledge wasn't. Everyone knew how to download a model. Nobody knew where it went or how to get rid of it. We eventually added a script to the cluster login banner that just ran clean_cache() automatically on login. Saved about four terabytes a month.
Four terabytes of orphaned revisions.
Hilbert: Orphaned revisions, stale branches, models people had downloaded once to test and never touched again. The cache is designed to be efficient — the content-addressable storage is good engineering — but it assumes someone's going to maintain it. Nobody does.
The clean_cache() function is safe, though. It only removes things that aren't referenced.
Hilbert: Yeah, I know. That's why we automated it. The problem wasn't safety. It was awareness. People didn't know the cache existed, didn't know it was growing, didn't know there was a garbage collector. The whole system is invisible until you run out of disk.
That's the thing about local inference in general. The tools are good. The documentation exists. But the gap between "I found a model" and "I'm running it sustainably" is full of these invisible friction points — the cache, the quantization step, the compatibility tracker's blind spots.
Hilbert: The tracker's fine for what it is. It's a first check. The problem is when people treat it as the final answer. It says no, they give up, when the real answer was "yes, if you quantize." Or it says yes, they download, and six months later their disk is full and they don't know why.
What's the fix? Better tooling? Better defaults?
Hilbert: Default to running clean_cache() after every delete_cache() call. And show a disk usage warning when the cache crosses some threshold — fifty gigs, a hundred gigs, whatever makes sense for the machine. The information is all there. It just needs to be surfaced.
The cache as a product problem, not a technical one.
Hilbert: Most things are.
Alright, let's land this. We've traced the two pathways — local and cloud — and the cloud fork between direct API and Endpoints. The mechanics are different at every level except the code you write. The compatibility tracker is your first decision point, the cache management API is your ongoing maintenance burden, and the cloud routing is an abstraction that works until it doesn't.
The thing I keep thinking about is how much of this is going to change. Split inference — running part of a model locally and part in the cloud — is already happening in research. Apple's doing on-device inference with cloud fallback. The line between "local" and "cloud" is getting blurrier, not sharper.
Which means understanding both pathways isn't just about making a choice today. It's about understanding the pieces well enough to reassemble them when the pathways start merging.
That's a good place to leave it. Thanks to Hilbert Flumingtop for producing, and for the four terabytes of orphaned model revisions.
This has been My Weird Prompts. If you've got a weird prompt of your own, email us at show at my weird prompts dot com. We'll be back soon.
