Introducing Storage Buckets on the Hugging Face Hub

Building Mutable ML Artifact Storage with Hugging Face Buckets

Storage Buckets on the Hugging Face Hub let you manage mutable, S3-like object storage for intermediate ML files (checkpoints, optimizer states, processed shards, logs) using Xet’s chunk-based deduplication, accessible via hf://buckets/ handles, the hf CLI, or Python.

Why this matters for builders

Until now, the Hub was excellent for final, versioned artifacts (models and datasets) but felt wrong for the constant stream of intermediate files that training runs and data pipelines produce. Buckets solve exactly that gap: they are non-versioned, mutable containers living under your namespace, with standard HF permissions, a browsable web page, and native Xet backend that deduplicates overlapping chunks across related files.

This unlocks a clean separation in production ML workflows: use Buckets for everything that changes often and comes from many jobs, then publish only the polished artifacts to traditional repos. For Enterprise users, billing is based on deduplicated storage, so you directly benefit from the similarity between successive checkpoints or raw vs processed data.

When to use it

• Training jobs that write frequent checkpoints and optimizer states
• Data pipelines producing iterative processed shards or parquet files
• Multi-job agents that need to store traces, memory graphs, and derived summaries
• Any workflow where you need fast write/overwrite + directory sync without Git history overhead
• Scenarios requiring pre-warming of data to specific cloud regions (AWS/GCP initially) for training throughput

The full process

1. Define the goal

Start by writing a short spec. Good example:

“I want a training orchestration script that saves every checkpoint and optimizer state to a private Bucket called llama3-fine-tune-run-42. It should support dry-run planning, resume from partial syncs, pre-warm the bucket to us-east-1 before launching distributed training on AWS, and expose a simple Python API that my training loop can call after each epoch. Only the final best checkpoint should be pushed to a versioned Model repo.”

This spec keeps you honest about scope and makes prompting AI coding tools much more effective.

2. Shape the spec / prompt your coding assistant

Use this starter prompt with Claude, Cursor, or any strong coding model:

You are an ML infrastructure engineer. Implement a Python module `hf_bucket_training.py` with the following requirements:

- Use huggingface_hub >= 0.28 (the version that ships Buckets support)
- Provide a class `TrainingBucket` that takes org/user name and bucket name
- Support creating the bucket if it doesn't exist (private by default)
- Method `sync_checkpoints(local_dir: str, remote_prefix: str = "checkpoints")` that:
  - First runs --dry-run and logs the plan
  - Supports saving the plan to JSONL and applying it later
  - Uses hf buckets sync under the hood via subprocess or the new Python API
- Method `prewarm(region: str)` that triggers data locality for AWS us-east-1 or GCP
- Method `promote_best_checkpoint(local_best_path: str, repo_id: str)` that copies the best model to a versioned repo
- Include proper error handling and logging
- Add a CLI entrypoint using typer or fire

Focus on reliability and observability. Do not invent APIs — only use what is documented in the official announcement and huggingface_hub.

3. Scaffold

First install the latest CLI and library:

curl -LsSf https://hf.co/cli/install.sh | bash
pip install --upgrade huggingface_hub hf_xet
hf auth login

Create the bucket:

hf buckets create llama3-fine-tune-run-42 --private

Verify it exists and is browsable at https://huggingface.co/buckets/YOURUSER/llama3-fine-tune-run-42.

4. Implement

Here is a minimal reliable implementation pattern (adapt to your AI tool’s output):

import subprocess
from pathlib import Path
from typing import Optional
import json
import logging

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)

class TrainingBucket:
    def __init__(self, namespace: str, bucket_name: str):
        self.handle = f"hf://buckets/{namespace}/{bucket_name}"
        self.namespace = namespace
        self.bucket_name = bucket_name
    
    def create_if_not_exists(self, private: bool = True):
        try:
            subprocess.run(
                ["hf", "buckets", "create", self.bucket_name, "--private" if private else ""],
                check=True,
                capture_output=True
            )
            logger.info(f"Bucket {self.bucket_name} ready")
        except subprocess.CalledProcessError as e:
            if "already exists" not in e.stderr.decode():
                raise
    
    def sync(self, local_dir: str, remote_prefix: str = "checkpoints", dry_run: bool = False, plan_file: Optional[str] = None):
        cmd = ["hf", "buckets", "sync", str(local_dir), f"{self.handle}/{remote_prefix}"]
        
        if dry_run:
            cmd.append("--dry-run")
        if plan_file:
            cmd.extend(["--plan", plan_file])
            
        result = subprocess.run(cmd, capture_output=True, text=True)
        logger.info(result.stdout)
        if result.stderr:
            logger.warning(result.stderr)
        return result.returncode == 0
    
    def apply_plan(self, plan_file: str):
        return subprocess.run(["hf", "buckets", "sync", "--apply", plan_file], check=True)

Extend this with pre-warm and promotion logic using hf buckets cp for the final model.

5. Validate

Run these checks before trusting the system:

• Dry-run a sync and inspect the plan file
• Confirm chunk deduplication by uploading two similar checkpoints and checking storage size in the Hub UI
• Test pre-warming (check the bucket page or CLI for region status)
• Validate permissions: try accessing from a different account
• Simulate failure: kill a sync and resume with the saved plan
• Measure bandwidth and time compared to naive S3 upload of the same data

6. Ship safely

1. Make the bucket private during development
2. Add a clear naming convention (project-run-id-timestamp)
3. Set up a cleanup policy for old checkpoints (use hf buckets remove)
4. Only promote final artifacts to versioned repos using huggingface_hub snapshot or hf buckets cp
5. Document the bucket handle in your training repo’s README
6. For teams: use organization-level buckets with proper HF roles

Pitfalls and guardrails

### What if the sync command is missing in my hf CLI?
Update immediately: hf --version should show a version that includes Buckets (post March 2026). Reinstall with the install script.

### What if I accidentally make the bucket public?
You can change visibility later via the Hub UI or CLI. Treat intermediate buckets as private by default.

### What if two jobs write to the same prefix simultaneously?
Xet handles concurrent writes safely at chunk level, but use unique prefixes per job (checkpoints/job-001) if you need strict isolation.

### What if my training cluster is on-prem?
Pre-warming is currently cloud-focused. Sync to the Bucket first, then let your on-prem jobs pull from the nearest region.

### How do I avoid filling the bucket with stale checkpoints?
Add a simple retention script that runs hf buckets remove for any checkpoint older than N days, keeping only the latest 5 per run.

What to do next

• Add monitoring: write a daily script that reports bucket size and top-level prefixes
• Integrate with your experiment tracker (Weights & Biases, MLflow) by logging the bucket handle
• Build a small dashboard Space that shows live sync status for all active training buckets
• Experiment with pre-warming in a multi-region training setup
• Extract the final model and push it to a proper Model repo with full card and evaluation results

Storage Buckets close a major gap between research and production on the Hub. By treating mutable intermediate storage as a first-class citizen backed by intelligent deduplication, Hugging Face has made large-scale ML experimentation significantly more practical.

Sources

• Introducing Storage Buckets on the Hugging Face Hub — official announcement, March 10, 2026
• Hugging Face Hub storage documentation and huggingface_hub release notes
• Xet storage backend documentation

(Word count: 982)