Hugging Face: Create Private Bucket Guide

Q: Why is my sync taking forever?

First, run with `--dry-run`. Large directories with millions of small files can be slow. Consider archiving logs or older checkpoints before syncing. Xet performs best with larger files that have overlapping content.

How to Use Storage Buckets on the Hugging Face Hub for ML Artifacts

TL;DR

Create a mutable S3-like bucket on the Hub in under 2 minutes using the hf CLI and sync local directories containing checkpoints, logs, or processed data.
Leverage Xet’s chunk-based deduplication for faster uploads/downloads and lower storage costs on related ML files.
Access buckets via browser, CLI, or Python SDK — ideal for intermediate training artifacts that don’t need Git versioning.

Storage Buckets solve a common pain point in production ML: Git repositories work great for final models and datasets, but they become inefficient for the constant stream of changing intermediate files generated during training, data processing, and agent workflows.

Prerequisites

Before you begin, make sure you have:

Hugging Face account (free or Enterprise)
hf CLI installed (v0.5.0 or later)
Basic familiarity with terminal commands
Local directories containing ML artifacts (checkpoints, optimizer states, processed shards, logs, etc.)
For private buckets: appropriate organization or user permissions

Install the CLI (if you haven’t already):

curl -LsSf https://hf.co/cli/install.sh | bash

Then authenticate:

hf auth login

Step 1: Create Your First Bucket

Use the CLI to create a new bucket under your username or organization.

hf buckets create my-training-bucket --private

# Create a public bucket
hf buckets create my-public-bucket

What happens:
A new non-versioned storage container is created at the address hf://buckets/your-username/my-training-bucket.
You can view it immediately in your browser at:
https://huggingface.co/buckets/your-username/my-training-bucket

Step 2: Sync Local Directories to a Bucket

The most common workflow is syncing a local folder (e.g. ./checkpoints) into the bucket.

# Sync a directory
hf buckets sync ./checkpoints hf://buckets/your-username/my-training-bucket/checkpoints

Dry-run first (highly recommended for large jobs):

hf buckets sync ./checkpoints hf://buckets/your-username/my-training-bucket/checkpoints --dry-run

Save and apply a sync plan (great for CI/CD or reviewing large transfers):

hf buckets sync ./checkpoints hf://buckets/your-username/my-training-bucket/checkpoints --plan sync-plan.jsonl

# Later, execute the plan
hf buckets sync --apply sync-plan.jsonl

Step 3: Manage Individual Files

For one-off operations, use cp and remove:

# Copy a single file
hf buckets cp ./latest-model.pth hf://buckets/your-username/my-training-bucket/models/latest-model.pth

# Remove stale files
hf buckets remove hf://buckets/your-username/my-training-bucket/old-checkpoints/

Step 4: Inspect and Browse Buckets

List contents from the CLI:

hf buckets list your-username/my-training-bucket -h

Or simply open the bucket page in your browser for a familiar file explorer view.

Using Buckets from Python

The huggingface_hub library provides full programmatic access. Install the latest version:

pip install -U huggingface_hub

Basic Python usage:

from huggingface_hub import HfApi, Bucket

api = HfApi()

# List buckets
buckets = api.list_buckets()
print([b.name for b in buckets])

# Upload a file
api.upload_file(
    path_or_fileobj="checkpoint_1000.pth",
    path_in_repo="checkpoints/checkpoint_1000.pth",
    repo_id="your-username/my-training-bucket",
    repo_type="bucket"
)

# Download a file
api.download_file(
    repo_id="your-username/my-training-bucket",
    filename="checkpoints/checkpoint_1000.pth",
    local_dir="./downloaded_checkpoints",
    repo_type="bucket"
)

Syncing directories from Python (using the new bucket utilities):

from huggingface_hub import sync_bucket

sync_bucket(
    local_dir="./checkpoints",
    bucket_id="hf://buckets/your-username/my-training-bucket/checkpoints",
    dry_run=True
)

Pre-warming Data for Training Jobs

For production workloads, bring data closer to your compute:

# Pre-warm to a specific cloud region
hf buckets prewarm hf://buckets/your-username/my-training-bucket \
    --region us-east-1 \
    --provider aws

Currently supported: AWS and GCP. More providers are coming soon.

This feature is particularly valuable for distributed training where latency between storage and compute nodes matters.

Tips and Best Practices

Use descriptive bucket names that reflect the project or experiment (e.g. llama3-fine-tune-run-42).
Organize with prefixes inside the bucket: checkpoints/, logs/, processed_data/, traces/.
Run dry-run before large syncs — especially important when dealing with terabytes of checkpoints.
Combine with versioned repos: Keep final, publishable artifacts in regular HF repos. Use buckets for everything that changes frequently.
Monitor storage usage on the bucket page or via CLI for Enterprise accounts (billed on deduplicated size thanks to Xet).
Leverage Xet deduplication: Successive model checkpoints and similar processed datasets will share chunks, dramatically reducing upload time and cost.

Common Issues

Why am I getting "Bucket not found" error?

Make sure you’re using the correct namespace. The handle should be hf://buckets/username-or-org/bucket-name. Check spelling and that you created the bucket in the right account.

Why is my sync taking forever?

First, run with --dry-run. Large directories with millions of small files can be slow. Consider archiving logs or older checkpoints before syncing. Xet performs best with larger files that have overlapping content.

Can I use buckets with Git LFS repos?

Yes. Many teams keep their final model in a standard repo and use a linked bucket for intermediate checkpoints during training.

"Permission denied" when creating bucket?

If you’re part of an organization, you may need specific IAM-like permissions. Check your role in the organization settings.

How do I delete a bucket?

Currently buckets can be deleted via the web UI or CLI (check official docs for the exact hf buckets delete syntax as it may still be evolving).

Next Steps

After setting up your first bucket:

Integrate bucket sync into your training scripts or CI/CD pipelines.
Experiment with pre-warming for multi-region or multi-cloud training.
Explore combining buckets with Hugging Face Agents for storing traces and memory graphs.
Try syncing processed dataset shards to see Xet deduplication in action.
Monitor the huggingface_hub releases for new bucket commands and Python API improvements.

Storage Buckets represent a major shift toward production-grade ML infrastructure on the Hub. They give you fast, mutable object storage with the same permissions and discoverability as the rest of the platform — while the Xet backend makes them especially efficient for the kinds of related files ML teams produce every day.

Create a private bucket