Here's the question that's been rattling around my head since I read Daniel's email: can you make a bucket watch a model repo and pull new weights automatically, the way Git users pull upstream?
And the answer is no. Well... not as a checkbox.
Right. So let me put Daniel's actual prompt on the table, because it's a good one. He's writing in about Xet, the storage backend Hugging Face swapped in across the Hub. His framing is that model weights are huge, Git was never built for files that big, and Git LFS was the useful middle ground for years. But Xet is interesting because it's tightly coupled to Hugging Face's own bucket product, so you can copy weights straight into your own storage. What he wants to look at is how to use this new storage effectively. And then the real question: if you're integrating other people's models, you want a reliable way to update weights from the upstream lab. Is there a mechanism where a bucket you created for specific weights can be configured to pull automatically whenever new variants of the same release show up?
Five questions in one paragraph. That's a Daniel special.
It is. So where do we start?
With the two nouns. Xet and Buckets. Everything else hangs off those.
Then start with the scale problem, because that's what forced all of this.
Hub repos hold files in the range of gigabytes and above. There are terabytes-scale files sitting in repositories right now. And they're binary. Safetensors, Parquet, GGUF. Plain Git is hopeless there, and not for the reason people assume. It's not that Git can't store a big file. It's that a clone retrieves the entire history, every revision, all of it. You'd be downloading four generations of a forty-gigabyte model to get the current one.
Which is the same reason Parquet took over from row-based formats in data pipelines. The storage model has to match the access pattern, or you pay for it forever.
Exactly that mismatch. So Git LFS was the workaround. It keeps a small pointer file in Git. SHA256, pointer size, remote file size. The actual bytes live in remote storage, typically S3. The repo stays small, the Git workflows stay fast, and for a long time that was good enough.
Good enough until it wasn't.
File-level deduplication. That's the limitation. When a file changes, LFS re-uploads and re-stores the whole file. A forty-gigabyte Safetensors shard where one tensor moved means forty gigabytes of traffic for a few megabytes of difference.
And that's the sentence that makes Xet make sense.
Xet was built specifically for this. Chunk-level deduplication, byte-level, smaller uploads, faster downloads, local disk caching. It came out of the XetHub acquisition in August 2024, a Seattle startup. The xet-core library is adapted from xetdata/xet-core, which already had deep Git integration. Deployed January 2025, available on the Hub that March, and default for new users and organizations since May 2025.
And the second noun.
Storage Buckets. Launched publicly on the tenth of March this year. S3-like object storage powered by the same Xet engine, but with a completely different contract. Non-versioned and mutable.
So that's the two nouns. Now let's get into why the old system couldn't keep up, and what LFS was actually doing for us.
The pointer file is the part people get wrong. It looks like a stub, but it's doing real work. Git tracks the pointer, so your history, your diffs, your branches all behave normally. The pointer says: this file is this big, here's its hash, go fetch it from the LFS server. And for a world where models were uploaded once and downloaded many times, that's a perfectly good design.
Where does it break?
Anywhere the file changes often. Fine-tuning runs, quantization sweeps, LoRA merges, anything that produces a new checkpoint from a mostly-identical previous checkpoint. The Hub is full of that. You have mradermacher with forty-two thousand repos, most of them quantizations of the same handful of base models. Under LFS, every one of those is a full copy of the weights.
Forty-two thousand full copies.
Not literally, because LFS does deduplicate at the file level. If the bytes are identical, it stores one copy. But the moment a single byte differs, it's a new object. And in practice, quantization changes every weight in the file, so you get almost no reuse.
Which means the Hub was storing the same model over and over in slightly different numeric precision.
At petabyte scale. That's the problem Xet was built to solve. Content-defined chunking.
Define it properly, because this is the mechanism the whole episode rests on.
The client breaks a file into chunks based on the content itself, not at fixed offsets. It uses gearhash-based boundary detection, so the chunk boundaries are determined by what's in the bytes. Those chunks go into a content-addressed store. On download, the store returns chunk ranges the client asks for, and the client reconstructs the file. The docs put the distinction cleanly: unlike Git LFS, which deduplicates at the file level, Xet-enabled repositories deduplicate at the level of bytes.
Why does content-defined matter versus just cutting the file into fixed blocks?
Because fixed blocks are brittle. If you insert a single byte at the front of a file, every fixed-size block after that point shifts by one, and now none of them match the previous version. Every chunk is new. Content-defined boundaries are stable, because the boundary is determined by local content, not by position. Insert bytes at the front, and the chunks after the insertion point still hash the same.
So a re-quantized model shares almost all of its chunks with the original.
In the ideal case, yes. Chunk size in practice is around sixty-four kilobytes, per the community write-ups. So a forty-gigabyte shard is roughly six hundred thousand chunks, and a quantization pass might change a fraction of them rather than all of them.
The TRL delta-weight-sync example is the cleanest illustration of this. Per-step payload on Qwen3-0.6B drops from 1.2 gigabytes to somewhere between twenty and thirty-five megabytes. That's the same principle applied to training checkpoints.
Same principle, different layer. That's the storage engine. Now the part that explains why nobody noticed the swap.
The bridge.
The Git LFS Bridge. It reconstructs Xet files and returns a single presigned URL that mimics the LFS protocol. So any client that doesn't know Xet exists, which is basically every client in early 2025, just sees LFS. It asks for a file, it gets a URL, it downloads. Nothing breaks.
And uploads?
Non-Xet uploads land in LFS first, then a background migration moves them to Xet. The migration blog is explicit about the design intent: there would be no hard cut-over from Git LFS to Xet.
That's the sentence I'd frame. Because the numbers that follow are absurd.
Five hundred thousand repositories and twenty petabytes migrated in six months. More than a million users on Xet. And the blog's own characterization is that this was perhaps the quietest migration of that magnitude, with only a few dozen GitHub issues, forum threads, and Discord messages across the whole thing.
Twenty petabytes and a few dozen complaints.
That's the payoff of keeping the old protocol alive as a compatibility layer. You never force anyone to change their tooling on your schedule. You change the floor underneath them and let them keep walking.
What did the throughput look like?
bartowski's migration sustained around thirty-five gigabits per second. mradermacher and RichardErkhov peaked at roughly three hundred gigabits per second while still serving about forty gigabits per second of everyday load. The baseline was just shy of a hundred gigabits per second as of July last year.
Three hundred gigabits. That's not a migration, that's a firehose.
Largest single migrations: mradermacher moved 6.1 petabytes across forty-two thousand repos. RichardErkhov 1.7 petabytes across twenty-five thousand. bartowski about five hundred terabytes across two thousand repos. And the batching was a thousand files or five hundred megabytes, whichever came first.
The batching detail is the one that tells you how careful they were being. Small batches, background, no user-visible pause.
Right. Which is why the migration story is a good piece of engineering rather than a marketing claim. The bridge plus background migration plus small batches equals a backend swap that most users experienced as nothing at all.
That's the storage engine. But the question Daniel actually asked is about distribution, and that lives in a different product, Storage Buckets.
Buckets are where the architectural consequence shows up. And the contract is the opposite of a repository in almost every way that matters.
Lay out the comparison.
Repositories have full Git history, pull requests, model and dataset cards, git push and pull. Buckets have none of that. No versioning, overwrite in place, no pull requests, no cards. What they have is S3-like sync, copy, remove, and listing. Both use Xet chunk-level dedup, so the storage engine is shared and the semantics are not.
So a bucket is not a repo with a different name. It's a different category of object.
It's a working layer. That's the way to think about it. Buckets live at hf colon slash slash buckets slash username slash my-bucket. The CLI is hf buckets create, sync, cp, rm, list. Python exposes create_bucket, sync_bucket, batch_bucket_files, download_bucket_files, list_bucket_tree. JavaScript support through the hub package from version 2.10.5, Python since huggingface_hub 1.5.0.
And then the copy that makes Daniel's question interesting.
Server-side copy. hf buckets cp from a dataset or model repo into a bucket, or the API equivalent copy_files. Only Xet-tracked files are copied server-to-server, and the changelog's framing is that even very large files are copied instantly thanks to chunk-level deduplication. Terabytes in just a few seconds.
Instant because nothing is actually moving.
Nothing is moving. The bucket is getting a reference to chunks that already exist in the content-addressed store. That's why it's instant. It requires the same storage region, and small non-Xet files like configs and READMEs get downloaded and re-uploaded the old way, but the weights themselves never travel.
Now the asymmetry.
Repo to bucket is instant and server-side. Bucket to repo is not yet available, but is on the roadmap. Those are the docs' words. The launch blog confirms the plan is direct transfers between buckets and repos in both directions.
So right now a bucket is a working layer, not a publishing layer.
Which is exactly the constraint that shapes everything Daniel is asking about. Because if you want to distribute weights, you have to publish from a repo. And if you want to consume weights into your own storage, a bucket is the natural landing zone. The flow is one-directional by design at the moment.
So let's answer the central question directly. Can a bucket be configured to automatically pull new upstream variants?
No. There is no built-in auto-pull feature. And the reason is structural, not a missing checkbox. Buckets are explicitly non-versioned and mutable. They have no refs, no tags, no commits. There is no native concept of tracking an upstream release, because there is no native concept of a release at all. I went through the storage-buckets docs, the webhooks docs, the jobs docs, the launch blog, the changelog. Nothing.
So the answer is no.
The answer is no as a setting. Yes as an architecture. Those are different things and the distinction is the whole episode.
Then build the architecture.
Hub Webhooks. You can register a webhook on any repo, including repos you don't own, watching for repo.content events. New commits, new tags, new branches. When something lands, the Hub posts a JSON payload to an endpoint you control.
And you don't need to be the repo owner.
You don't. The event scopes include repo, repo.content, repo.config, discussion, discussion.comment. So you can point a webhook at meta-llama slash whatever and get told every time they push.
Then what?
Then you run hf buckets cp from that model repo into your bucket. Server-side, effectively instant, because of the chunk reuse. Or you trigger an HF Job to do it. That is the closest analog to the Git distribution paradigm. It's assembled by the user, not configured as a bucket setting.
So the pattern is: webhook on the upstream repo, copy on the event.
That's it. And that's reliable, in the sense that it fires on the event. What it isn't is a native feature with a support contract.
What breaks?
Your webhook handler is now the reliability layer. The rate limit is a thousand triggers per twenty-four hours per webhook. Payloads are delivered asynchronously with no ordering guarantee. Failing webhooks get auto-suspended. And bucket payloads truncate above ten thousand entries in updatedFiles.
So retries, idempotency, ordering, all of that becomes your problem.
All of it. If the upstream lab pushes three commits in ten seconds, you get three payloads in an order nobody promised you. If your endpoint is down, you get suspended rather than queued indefinitely. Those are the properties you inherit.
There's a detail in the bucket payload worth naming, because it changes how you write the handler.
For buckets, repo.content fires on files added or deleted, and the payload uses updatedFiles with xetHash and size rather than updatedRefs. And overwriting a file reports as an add.
Overwriting reports as an add.
Which means you can't distinguish a new file from a replaced file by the event type. You have to look at the hash.
That's the kind of detail that eats an afternoon.
It eats a week if you find it in production.
What about the scheduled alternative?
HF Jobs support scheduling and webhook automation, so a cron-style hf buckets sync or cp is possible. davanstrien's blog on using Storage Buckets as a working layer for data pipelines documents exactly this pattern. Buckets as intermediate storage, HF Jobs for the scheduling. It's the same outcome by a different route, and it's arguably more robust if you don't need event-level latency.
Polling instead of pushing.
Polling instead of pushing, with the tradeoff that you find out about a new release on your schedule rather than theirs.
And the third-party evidence.
backblaze-labs hf-cache-sync syncs the local Hugging Face cache to Backblaze B2 or S3-compatible storage with LRU eviction. The community is building its own sync tooling rather than waiting for a native bucket feature. That's usually a signal.
A signal of what?
Either that the native feature is coming, or that a third-party ecosystem is about to form around the gap. Both happen. Sometimes both at once.
The second-order implication is the part I keep circling.
Say it.
Because buckets have no refs and no versioning, reliable update from the upstream lab has to be reconstructed from webhooks plus server-side copy. And the reliability properties you get are the reliability properties of your handler. That's a real gap, not a missing checkbox. It's an architectural fact about what a bucket is.
It's the same reason Git gave you tags in the first place. Tags are a promise that a name points at a specific set of bytes forever. A bucket can't make that promise, because overwrite-in-place is the whole contract.
There's a billing tension worth touching too.
Chunk-level dedup is global. If a chunk exists anywhere on the Hub, it's stored once. But users still see full logical size against their quotas. A community comment on the migration blog noted exactly that, and the author's reply was that dedup is global and billing is based on the logical size of each new object, for predictability and fairness.
So you pay for the bytes you asked for, not the bytes that were actually new.
Enterprise is billed on the deduplicated footprint, and there's a headline figure of twelve dollars per terabyte floating around for the standard tier. But for the ordinary user, the quota math is logical size. Which is defensible, and also means the dedup savings don't show up on your bill.
Defensible and slightly annoying are not mutually exclusive.
They rarely are.
There's one more thing in the research that's worth a line, because it tells you where the roadmap sits.
Wauplin on cold storage tiers. We've discussed such feature internally and we might offer it at some point, but it's not in our short-term roadmap.
Same posture could apply to native upstream tracking.
It could. Which is why I'd bet on the community tooling arriving before the setting does.
Hilbert: The tapes were the index.
What?
Hilbert: The index tapes. I worked at a medical imaging archive for a while. Tapes and a clipboard, that was the job. Every scan was stored under a hash of its contents. Two copies of a file anywhere in that building were the same file, and that was the entire disaster recovery plan. Nobody called it content-addressed storage. It was just how you kept four hundred thousand scans from eating the building.
So the dedup idea isn't new.
Hilbert: It's not new. The chunking is a good answer to a real problem. I'll give them that. But the interesting part isn't the chunking. It's that they kept the old pointer protocol alive so nobody had to change anything. That's the engineering. Anybody can write a new storage engine. Keeping ten years of tooling working through the swap is the hard part.
That's the LFS Bridge.
Hilbert: Whatever you call it. The bridge is the achievement. The chunking is arithmetic.
There's a parallel in the migration numbers. Twenty petabytes and a few dozen complaints. That's the bridge doing its job.
Hilbert: Twenty petabytes. We moved four hundred terabytes over a summer and I still have the tapes in the garage. Box of them. My wife has asked me three times to throw them out.
Are they readable?
Hilbert: No idea.
You've never checked.
Hilbert: Never checked.
So the disaster recovery plan is a box of tapes in a garage that nobody has verified.
Hilbert: The disaster recovery plan was a box of tapes in a garage. The tapes being readable was always somebody else's department. There's a phone call I'm expecting. Won't take it in here.
The tapes are a problem for another day. Let's leave people with the open questions.
The one I keep coming back to is the roadmap. If bucket-to-repo transfer ships, and the launch blog says both directions are planned, then the working-layer and publishing-layer split softens. You could promote a bucket back into a versioned repo server-side, and suddenly the asymmetry that shapes every pipeline design right now just... stops being the constraint.
Which would change the architecture, not just the convenience.
It would change what a bucket is for. Right now a bucket is a place data lands. If it can be promoted, it becomes a place data stages before it ships.
And the other open question is whether Hugging Face ever ships a native upstream-tracking setting. The cold storage answer suggests the posture: discussed internally, not in the short-term roadmap.
The community is already building the missing sync layer. hf-cache-sync exists. That's usually the signal that either a native feature is coming or a third-party ecosystem forms around the gap. Either way, the gap closes. The question is who closes it.
Thanks to Hilbert Flumingtop for producing. This has been My Weird Prompts, the human-AI collaboration podcast. If you're enjoying the show, a review helps more than you'd think. Everything lives at my weird prompts dot com, including the RSS feed and the archive.
We'll be back soon.
See you tomorrow.
