- What: P-EAGLE (Parallel EAGLE), a new framework for parallel speculative decoding in LLM inference.
- Performance: Delivers up to a 1.69x speedup over the previous state-of-the-art EAGLE-3 method.
- Hardware: Benchmarked on NVIDIA B200 GPUs using real-world workloads.
- Availability: Integrated into vLLM (v0.16.0+) with pre-trained models available on HuggingFace for GPT-OSS and Qwen3-Coder.
Amazon has unveiled P-EAGLE, a major advancement in speculative decoding that significantly accelerates Large Language Model (LLM) inference by generating multiple draft tokens in a single forward pass. By eliminating the sequential bottleneck inherent in traditional autoregressive drafting, P-EAGLE achieves up to a 1.69x speedup over the industry-standard EAGLE-3 on NVIDIA B200 GPUs. The technology is now natively supported in the vLLM serving framework, allowing enterprise developers to slash latency on high-demand reasoning models with a single configuration update.
Breaking the Autoregressive Bottleneck
Speculative decoding has become the primary strategy for reducing LLM latency. The process typically involves a small "drafter" model predicting several future tokens, which a larger "target" model then verifies in parallel. While the EAGLE framework is widely considered the state-of-the-art for this task—regularly delivering 2x to 3x speedups over standard decoding—it relies on an autoregressive drafting process.
According to Amazon’s technical announcement, this sequential nature creates a "hidden bottleneck." To produce $K$ draft tokens, a standard drafter must perform $K$ sequential forward passes. As developers attempt to increase the "speculation depth" to get more tokens per cycle, the overhead of these multiple passes begins to consume the performance gains. P-EAGLE solves this by transforming the drafting phase from a sequential process into a parallel one.
The Architecture of P-EAGLE
P-EAGLE maintains the core advantages of the original EAGLE framework—utilizing the target model’s internal hidden states to guide the drafter—but modifies the architecture to enable simultaneous token generation.
The process involves two primary steps:
- Prefilling: The target model processes the prompt and generates a new token. During this phase, P-EAGLE captures internal hidden states ($h_{prompt}$ and $h_{context}$), which encode the target model's knowledge at each position.
- Parallel Drafting: Unlike standard models that wait for one token to be predicted before starting the next, P-EAGLE constructs inputs for all $K$ positions simultaneously.
For the first token (Next-Token-Prediction), the drafter uses the actual embeddings from the target model. For subsequent positions (Multi-Token-Prediction), where actual embeddings do not yet exist, P-EAGLE utilizes two specific learnable parameters: a shared mask token embedding ($emb_{mask}$) and a shared hidden state ($h_{shared}$). These act as neutral placeholders, allowing the drafter to predict tokens $t1$ through $t4$ in one unified forward pass through the transformer layers.
Benchmarks and Performance Gains
In head-to-head testing on NVIDIA B200 hardware, P-EAGLE demonstrated substantial improvements over vanilla EAGLE-3 across several standard benchmarks, including MT-Bench, HumanEval, and SpeedBench.
When applied to the GPT-OSS 20B model, the parallel approach yielded speedups ranging from 1.05x to 1.69x. These gains are particularly relevant for modern reasoning models, which tend to produce longer outputs. Amazon's research highlighted that models like GPT-OSS 120B often generate sequences with a median length of nearly 3,900 tokens, with the 90th percentile reaching 10,800 tokens. In these long-sequence scenarios, the efficiency of the drafter becomes the deciding factor in overall system throughput.
Implementation in vLLM
The P-EAGLE integration is now live in vLLM starting from version 0.16.0 (via PR#32887). Developers using the vLLM serving pipeline can enable the feature by setting parallel_drafting: true within their configuration.
To facilitate immediate adoption, Amazon has released pre-trained P-EAGLE "heads" on HuggingFace for three major model families:
- GPT-OSS 120B
- GPT-OSS 20B
- Qwen3-Coder 30B (specifically the A3B-Instruct variant)
A sample command to deploy a model with P-EAGLE in vLLM looks like this:
vllm serve openai/gpt-oss-20b \
--speculative-config '{"method": "eagle3", "model": "amazon/gpt-oss-20b-p-eagle", "num_speculative_tokens": 5, "parallel_drafting": true}'
Impact for Developers and the Industry
For developers, P-EAGLE represents a "free" performance upgrade. By shifting the computation of draft tokens from sequential to parallel, the framework allows for a higher number of speculative tokens without the usual latency penalty. This translates directly to lower costs for API providers and faster response times for end-users.
"By turning sequential drafting into a parallel operation, P-EAGLE finally allows developers to scale speculation depth without hitting a latency wall," the Amazon team noted in their technical breakdown.
In the broader competitive landscape, this move reinforces Amazon's commitment to the open-source AI ecosystem, specifically through its contributions to vLLM. As models grow more complex and inference costs remain a primary barrier to scale, techniques that maximize the utilization of high-end silicon like the NVIDIA B200 are critical for maintaining a competitive edge.
What’s Next
The release of P-EAGLE also addresses the challenge of training draft models for long-context sequences. Amazon noted that parallel drafting increases memory requirements during the training phase, and future iterations of the toolset are expected to further optimize this memory footprint.
As the vLLM community adopts these parallel speculators, it is likely that more pre-trained heads for models like Llama 3 and Mistral will emerge on HuggingFace, further cementing parallel speculative decoding as the standard for high-performance LLM deployment.