Implementing Falcon-H1 Hybrid Architecture in NVIDIA Megatron Core

Implementing Falcon-H1 Hybrid Architecture in NVIDIA Megatron Core

Why this matters for builders

Falcon-H1 Hybrid Architecture lets you combine Transformer attention heads with Mamba-2 (SSM) heads in parallel inside the same mixer block, then concatenate their outputs before the final projection. NVIDIA Megatron Core now natively supports this hybrid design, giving you the long-context efficiency of state-space models while retaining the strong in-context learning of attention—all within the same high-performance training framework used to build frontier models.

This is the first time Megatron Core officially expands beyond pure Transformers to hybrid attention + SSM architectures. The change unlocks new workflows: training models that match or beat 70B-class performance at lower compute cost, experimenting with tunable attention-to-SSM ratios, and iterating on hybrid designs without rewriting the entire parallelism stack from scratch.

When to use it

• You are training or fine-tuning models longer than 32k–128k tokens and want linear-time memory scaling for the SSM portion.
• You want to explore hybrid architectures where attention and Mamba-2 heads run concurrently and their outputs are merged.
• You already have a Megatron Core training pipeline and want to test whether replacing some attention heads with Mamba-2 improves throughput or quality.
• You are a researcher or startup building the next generation of efficient foundation models and need production-grade parallelism (tensor, pipeline, sequence, context parallelism).

The full process

1. Define the goal

Start by writing a one-paragraph spec. Example:

“I want to train a 7B-class Falcon-H1-style hybrid model using Megatron Core. The model should use 8 attention heads and 8 Mamba-2 heads in parallel inside each hybrid mixer layer. Attention heads use standard scaled-dot-product, Mamba-2 heads use the latest Mamba-2 formulation. Outputs are concatenated and passed through a shared projection. I will train on 128k context with sequence parallelism + selective activation checkpointing. Target throughput > 180 TFLOPS/GPU on H100.”

Keep this spec visible. Every prompt you give to Cursor, Claude, or Grok should reference it.

2. Scope and scaffold

Create a new branch in your fork of NVIDIA/Megatron-LM.

git clone https://github.com/NVIDIA/Megatron-LM.git
cd Megatron-LM
git checkout -b falcon-h1-hybrid-experiment

Run the new hybrid model registration path (added in recent Megatron Core):

from megatron.core.models.hybrid import HybridConfig, HybridModel

Use your AI coding assistant with this starter prompt:

“Using the official NVIDIA Megatron Core implementation of Falcon-H1 hybrid architecture, create a HybridConfig class for a 7B model. Set num_attention_heads=8, num_mamba_heads=8, hidden_size=4096, num_layers=32. Use Mamba2 layer type. Enable sequence_parallel and context_parallel. Output the complete config and the exact command to register the model in the training script.”

Copy the generated config into examples/falcon_h1/config.py (or equivalent).

3. Implement the hybrid mixer

The core change lives in the mixer block. Prompt your AI assistant again:

“Implement the Falcon-H1 hybrid mixer block as described in the NVIDIA blog. Inside each layer we run standard Attention heads and Mamba-2 heads in parallel. Concatenate the outputs along the hidden dimension before the final linear projection. Respect existing Megatron Core parallelism (tensor parallel, sequence parallel). Provide the complete HybridMixer class with forward pass and the necessary sharding annotations.”

Validate the generated code against the official pattern:

• Attention and Mamba heads must run concurrently (not sequentially).
• Concatenation happens before the output projection.
• The ratio of attention to Mamba heads must be configurable at the config level.

4. Validate locally

Before launching a full training job, run a forward-backward pass test:

python -m torch.distributed.launch --nproc_per_node=2 \
    pretrain_gpt.py \
    --config-path examples/falcon_h1 \
    --config-name 7b_hybrid \
    --mock-data \
    --train-iters 10 \
    --seq-length 8192 \
    --micro-batch-size 1

Check logs for:

• No shape mismatch between attention and SSM branches
• Correct tensor-parallel and sequence-parallel sharding
• Memory usage scaling linearly with sequence length on the SSM portion

Use megatron/core/utils memory profiling helpers to confirm activation checkpointing is applied selectively to the attention branch.

5. Ship it safely

Follow this checklist before pushing to main or opening a PR:

• All new hybrid code is behind a model_type == "falcon_h1_hybrid" flag
• Unit tests added for hybrid mixer with 2 attention + 2 mamba heads
• Training script example updated with both pure attention and hybrid configs
• Documented the exact attention-to-SSM ratio used and why
• Measured and recorded tokens-per-second and TFLOPS on at least 8×H100
• Confirmed convergence on a small 10B-token subset (use the Pile or FineWeb)

Create a clean PR with title: “Add Falcon-H1 hybrid attention + Mamba-2 support to Megatron Core”.

Copy-paste prompts (starter templates)

Prompt for config generation:

“Generate a complete HybridConfig for Falcon-H1 7B following the exact pattern in the NVIDIA Megatron Core Falcon-H1 blog post. Use hidden_size=4096, num_layers=32, num_attention_heads=8, num_mamba_heads=8, mamba_layer_type='mamba2'. Enable sequence_parallel=True, context_parallel=True, and selective activation checkpointing for attention only.”

Prompt for mixer implementation:

“Write the HybridMixer class that runs Attention and Mamba2 heads in parallel, concatenates outputs, then applies the final projection. Follow Megatron Core conventions for parallel_state, tensor_parallel, and sequence_parallel. Include type hints and docstring matching existing code style.”

Pitfalls and guardrails

### What if the attention and SSM outputs have different dimensions?
They must match before concatenation. Force both branches to output hidden_size (or a multiple of TP degree). Adjust the Mamba-2 expansion factor if needed.

### What if training becomes unstable after switching to hybrid?
Start with a 1:1 attention-to-Mamba ratio. Lower the learning rate by 20–30% for the first 1k steps. Use the same initialization scale for both branches.

### What if sequence parallelism breaks with long contexts?
Make sure you are using the latest Megatron Core that includes the updated Ring Attention and Mamba-2 sequence-parallel kernels. Check the release notes for the exact commit that added Falcon-H1 support.

### What if my PR gets rejected?
The Megatron team prefers small, well-tested incremental changes. First submit the config + forward-pass test, then the training recipe in a follow-up PR.

What to do next

1. Train a 1.5B hybrid model for 50B tokens and compare perplexity vs. pure Transformer baseline on 32k context.
2. Sweep the attention-to-Mamba ratio (8:0, 6:2, 4:4, 2:6, 0:8) and log throughput vs. quality.
3. Add hybrid support to your downstream fine-tuning and inference stack (vLLM or TensorRT-LLM).
4. Share your config and throughput numbers with the Megatron Core community on GitHub.

The hybrid era of foundation models is here. Megatron Core now gives you the tools to experiment at scale without reinventing parallelism.

Sources

• Implementing Falcon-H1 Hybrid Architecture in NVIDIA Megatron Core
• Megatron Core official page
• Falcon-H1 Technical Report & Blog
• Hugging Face Falcon-H1 announcement
• NVIDIA/Megatron-LM GitHub repository (current main)