Hugging Face Launches Storage Buckets for ML Production Workloads
Key Facts
- What: Hugging Face introduced Storage Buckets, a new S3-like mutable object storage service on the Hub built on its Xet chunk-based backend
- When: Announced March 10, 2026
- Purpose: Designed specifically for intermediate ML artifacts such as checkpoints, optimizer states, processed data shards, logs and traces that change frequently and do not require Git versioning
- Key Technology: Xet enables chunk-level deduplication across related files, reducing bandwidth, speeding transfers and lowering storage costs for Enterprise users
- Features: Browser-based viewing, hf CLI integration, Python SDK support, pre-warming to bring data closer to compute in AWS and GCP regions
Lead paragraph
Hugging Face has released Storage Buckets, a new class of non-versioned object storage on its Hub platform tailored for the high-velocity, mutable data generated during machine learning training and data processing. Unlike traditional Model and Dataset repositories that excel at publishing final, version-controlled artifacts, Buckets provide fast, S3-compatible storage for intermediate files that are frequently overwritten or produced by distributed jobs. The service, powered by Hugging Face’s Xet storage technology acquired in 2024, uses intelligent chunk-based deduplication to make storage and transfer of related ML artifacts significantly more efficient.
Why Hugging Face Built Buckets
The company’s engineering team recognized that Git-based workflows quickly become cumbersome for production ML environments. Training clusters continuously write checkpoints and optimizer states, data pipelines iteratively process raw datasets, and AI agents generate traces, memory states and knowledge graphs. These workloads share a common requirement: the ability to write quickly, overwrite as needed, sync directories efficiently, and remove stale files without the overhead of version history.
According to the official announcement, “A Bucket is a non-versioned storage container on the Hub. It lives under a user or organization namespace, has standard Hugging Face permissions, can be private or public, has a page you can open in your browser, and can be addressed programmatically with a handle like hf://buckets/username/my-training-bucket.”
This design fills a clear gap between temporary local storage and the permanent, versioned repositories that have defined the Hugging Face Hub since its inception as the “GitHub for ML.”
The Role of Xet in Making Buckets Efficient
Buckets are built on Xet, Hugging Face’s modern chunk-based storage backend. Rather than storing files as monolithic blobs, Xet breaks content into smaller chunks and automatically deduplicates them across files and uploads.
This architecture delivers major benefits for typical ML workloads. When a processed dataset shares most content with the raw version, or when successive model checkpoints contain large frozen sections, Xet reuses existing chunks instead of re-uploading identical data. The result is reduced bandwidth usage, faster transfers and more efficient storage.
For Enterprise customers, billing is calculated on deduplicated storage, meaning shared chunks directly lower costs. This makes the system particularly attractive for organizations running large-scale training jobs that generate families of related artifacts.
The transition to Xet began after Hugging Face acquired XetHub in August 2024. While Git LFS remains supported for backward compatibility, Xet has become the preferred backend for new storage features due to its superior handling of AI/ML data patterns.
Pre-warming and Multi-Cloud Compute Integration
One of the more advanced capabilities is pre-warming, which addresses the challenge of data locality in distributed training. Although Buckets provide global storage by default, large-scale workloads often require data to be physically close to compute resources to maximize throughput.
Pre-warming allows users to declare target cloud providers and regions so that hot data is proactively moved closer to where training jobs will run. This eliminates repeated cross-region data transfers during critical training phases. Hugging Face is initially partnering with AWS and Google Cloud Platform, with additional cloud providers planned for the future.
Getting Started with Buckets
Developers can create and use a Bucket in under two minutes using the updated Hugging Face CLI. After installing the CLI and authenticating with hf auth login, users run:
hf buckets create my-training-bucket --private
Directory synchronization is handled with the hf buckets sync command, which supports familiar patterns such as --dry-run for previewing changes, generating sync plans for review, and applying plans later. Individual file operations are available through hf buckets cp for copying and hf buckets remove for cleanup.
The service also offers full programmatic access from Python, allowing seamless integration into training scripts and data pipelines. Each Bucket has a dedicated browser page at https://huggingface.co/buckets/username/my-training-bucket for visual inspection.
Impact on the ML Ecosystem
The introduction of Storage Buckets represents a significant expansion of the Hugging Face Hub beyond its original focus on open-source model and dataset sharing. By providing production-grade mutable storage with the same permissions, namespace and web interface users already know, Hugging Face is positioning the Hub as a more complete platform for the entire ML lifecycle.
This move is particularly relevant as AI development shifts from experimental notebooks to sophisticated, distributed production systems. Organizations can now keep intermediate artifacts on the same platform where they publish final models, simplifying workflows and reducing the need to manage separate storage solutions.
The deduplication capabilities of Xet also align with growing industry attention to storage efficiency and cost control in large-scale AI training. As models continue to grow in size and training runs become more expensive, technologies that reduce unnecessary data transfer and storage become increasingly valuable.
What’s Next
Hugging Face has indicated that additional cloud provider integrations for pre-warming are in development. The company is also expected to expand the Buckets feature set based on user feedback from the initial release.
The launch is part of a broader update that includes CLI improvements, Space hot-reload capabilities, and enhancements for AI coding agents, according to the huggingface_hub GitHub releases.
Enterprise customers on paid plans already receive substantial storage allocations — including 200TB of public storage for organizations releasing open-source models — and Buckets are expected to integrate into these existing storage offerings.
Sources