What Actually Happens During Speculative Decoding in LLMs

Autoregressive language model generation is notoriously slow. When you run a 70-billion parameter model on an enterprise GPU, you might get 20 to 30 tokens per second.

The intuitive assumption is that the GPU compute cores are sweating under massive matrix multiplications. But if you profile the GPU during single-batch generation, you find the exact opposite: the GPU's compute cores are sitting 95% idle.

The true bottleneck is memory bandwidth. Speculative decoding is an inference optimization that exploits this hardware reality to double or triple token generation speed with zero loss in mathematical output quality.

Here is what actually happens inside the GPU, the transformer layers, and the sampling math during speculative decoding.

1. The Real Bottleneck: Memory Bandwidth vs Arithmetic Intensity

To understand why speculative decoding works, you first have to look at the GPU memory bus.

During autoregressive generation (decoding phase), an LLM generates tokens sequentially: one token per forward pass. To generate a single token for a batch size of 1:

1. The GPU must load every single weight parameter from High Bandwidth Memory (HBM / VRAM) into the on-chip SRAM / registers.
2. It performs a vector-matrix multiplication between the single input token embedding and the model weights.
3. It writes the updated Key-Value (KV) cache entry back to VRAM.

Let us do the concrete math for a 70B parameter model in FP16 running on an Nvidia H100 GPU:

• Model size in VRAM: 70 billion parameters * 2 bytes = 140 GB.
• H100 Memory Bandwidth: ~3,350 GB/s.
• H100 Tensor Core Compute: ~1,979 TFLOPs (FP16).

Time spent streaming weights:

$$\text{Time}_{\text{memory}} = \frac{140\text{ GB}}{3,350\text{ GB/s}} \approx 41.8\text{ ms}$$

Time spent computing math:

Generating 1 token requires roughly $2 \times 70 \times 10^9 = 1.4 \times 10^{11}$ FLOPs.
$$\text{Time}_{\text{compute}} = \frac{1.4 \times 10^{11}\text{ FLOPs}}{1,979 \times 10^{12}\text{ FLOPs/s}} \approx 0.07\text{ ms}$$

The ratio is brutal: the GPU spends 41.8 ms loading weights from memory and only 0.07 ms doing arithmetic. The arithmetic intensity (FLOPs per byte transferred) is nearly 1, far below the GPU's saturation threshold (which is around 150 FLOPs/byte).

Traditional Autoregressive Decoding (Batch Size = 1):
[ Load 140 GB Weights ] ---> ( Compute 1 Token: 0.07 ms ) ===> Takes 41.8 ms
[ Load 140 GB Weights ] ---> ( Compute 1 Token: 0.07 ms ) ===> Takes 41.8 ms
[ Load 140 GB Weights ] ---> ( Compute 1 Token: 0.07 ms ) ===> Takes 41.8 ms
Result: 3 forward passes = ~125 ms for 3 tokens

2. The Core Insight: Parallel Verification is "Free"

What happens if, instead of 1 token, you feed 5 candidate tokens into the 70B target model simultaneously?

The model still loads the 140 GB weights from VRAM exactly once (41.8 ms). But now the Tensor Cores multiply a $[5 \times d]$ matrix against the weights instead of a $[1 \times d]$ vector.

The compute time rises from 0.07 ms to 0.35 ms, but both numbers are dwarfed by the 41.8 ms memory transfer time.

Evaluating 1 Token:  41.8 ms (memory) + 0.07 ms (compute) = 41.87 ms
Evaluating 5 Tokens: 41.8 ms (memory) + 0.35 ms (compute) = 42.15 ms

Verifying 5 tokens in parallel takes virtually the exact same wall-clock time as generating 1 token.

Speculative decoding exploits this asymmetry by splitting generation into two distinct roles:

1. A small, lightweight Draft Model guesses $K$ tokens sequentially at high speed.
2. The heavy Target Model verifies all $K$ tokens simultaneously in one parallel forward pass.

3. The Speculative Decoding Execution Loop

Here is the exact step-by-step pipeline executed on every speculative decoding iteration:

                  +-----------------------------------+
                  |  Prompt / Current Prefix Sequence  |
                  +-----------------+-----------------+
                                    |
                                    v
                  +-----------------------------------+
                  |      Draft Model (e.g. 1B)        |
                  |  Generates K tokens sequentially  |
                  |  Tokens: [x1, x2, x3, x4, x5]     |
                  +-----------------+-----------------+
                                    |
                                    v
                  +-----------------------------------+
                  |      Target Model (e.g. 70B)      |
                  |  Parallel Forward Pass on prefix  |
                  |  + all K speculative tokens       |
                  +-----------------+-----------------+
                                    |
                                    v
                  +-----------------------------------+
                  |    Speculative Rejection Sampler  |
                  |  Accepts x1, x2, x3               |
                  |  Rejects x4                       |
                  |  Samples corrected replacement x4'|
                  |  Discards x5                      |
                  +-----------------+-----------------+
                                    |
                                    v
                  +-----------------------------------+
                  |  Append Accepted Tokens to Prefix |
                  |  Total yield: 4 tokens in 1 step! |
                  +-------------------+---------------+

Step 1: The Draft Phase

A small model (such as a 1B parameter model paired with a 70B target) generates $K$ candidate tokens $(\hat{x}_1, \hat{x}_2, \dots, \hat{x}_K)$ autoregressively.

Because the draft model is tiny (only ~2 GB in FP16), streaming its weights takes under 0.6 ms per token. Generating 5 draft tokens takes roughly $5 \times 0.6\text{ ms} = 3.0\text{ ms}$.

For each generated token, the draft model records its predicted probability distribution $q(x)$.

Step 2: The Target Verification Phase

All $K$ draft tokens are appended to the input sequence and evaluated by the 70B target model in one single forward pass.

Because the attention mask allows each token at position $i$ to attend to all prior tokens, the target model produces the true output probability distributions $p_1(x), p_2(x), \dots, p_K(x)$ for all $K$ positions simultaneously.

Step 3: Speculative Sampling (Rejection & Correction)

We iterate through the draft tokens from $i = 1$ to $K$:

1. For token $\hat{x}_i$, compare target probability $p(\hat{x}_i)$ against draft probability $q(\hat{x}_i)$.
2. Draw a uniform random number $r \sim U[0, 1]$.
3. Acceptance Condition: If $r \le \min\left(1, \frac{p(\hat{x}_i)}{q(\hat{x}_i)}\right)$, accept token $\hat{x}_i$ and advance to $i+1$.
4. Rejection Condition: If the condition fails at position $j$, reject $\hat{x}j$. Sample a replacement token from the residual distribution: $$p'(x) = \frac{\max(0, p(x) - q(x))}{\sum{y} \max(0, p(y) - q(y))}$$ Immediately discard all subsequent draft tokens $(\hat{x}_{j+1}, \dots, \hat{x}_K)$, and conclude this iteration.
5. Bonus Token: If all $K$ tokens are accepted, sample one extra $(K+1)$-th token directly from the target model's final logits $p_{K+1}(x)$.

4. The Python Implementation

Here is a clean, dependency-free reference implementation of Leviathan & Chen's Speculative Sampling algorithm:

import numpy as np

def speculative_sample_step(draft_probs_list, target_probs_list, draft_tokens):
    """
    draft_probs_list: list of K numpy arrays, each shape (vocab_size,) representing q(x)
    target_probs_list: list of K+1 numpy arrays, each shape (vocab_size,) representing p(x)
    draft_tokens: list of K integers drafted by the small model

    Returns: list of accepted and sampled tokens
    """
    accepted_tokens = []
    K = len(draft_tokens)

    for i in range(K):
        token = draft_tokens[i]
        q_prob = draft_probs_list[i][token]
        p_prob = target_probs_list[i][token]

        # Calculate acceptance probability
        acceptance_threshold = min(1.0, p_prob / (q_prob + 1e-10))
        r = np.random.uniform(0.0, 1.0)

        if r <= acceptance_threshold:
            # Token accepted
            accepted_tokens.append(token)
        else:
            # Token rejected: sample from modified residual distribution
            residual = np.maximum(0.0, target_probs_list[i] - draft_probs_list[i])
            residual_sum = np.sum(residual)

            if residual_sum > 0:
                p_prime = residual / residual_sum
            else:
                p_prime = target_probs_list[i]

            replacement_token = int(np.random.choice(len(p_prime), p=p_prime))
            accepted_tokens.append(replacement_token)

            # Discard all remaining draft tokens
            return accepted_tokens

    # All K tokens were accepted! Sample bonus (K+1)-th token directly from target
    bonus_token = int(np.random.choice(
        len(target_probs_list[K]), 
        p=target_probs_list[K]
    ))
    accepted_tokens.append(bonus_token)
    return accepted_tokens

5. Why the Output Distribution is 100% Lossless

A common misconception among developers is that speculative decoding is an approximation or lossy distillation (like quantization or pruning).

It is mathematically exact. The probability of generating any sequence under speculative decoding is identical to generating directly from the large target model.

Here is the simple algebraic proof:

Let $x$ be the token candidate at position $i$. What is the total probability that token $x$ is emitted?

1. Probability $x$ was drafted and accepted:
   $$P(\text{drafted } x) \times P(\text{accepted} \mid x) = q(x) \times \min\left(1, \frac{p(x)}{q(x)}\right) = \min(p(x), q(x))$$

2. Probability another token was rejected and $x$ was sampled from $p'(x)$:
   The total rejection probability across all possible tokens is:
   $$1 - \alpha = 1 - \sum_{y} \min(p(y), q(y)) = \sum_{y} (p(y) - \min(p(y), q(y))) = \sum_{y} \max(0, p(y) - q(y))$$

When rejection happens, the probability of choosing $x$ from the normalized residual is:
$$p'(x) = \frac{\max(0, p(x) - q(x))}{1 - \alpha}$$

Therefore, the joint probability of rejection and choosing $x$ is:
$$(1 - \alpha) \times \frac{\max(0, p(x) - q(x))}{1 - \alpha} = \max(0, p(x) - q(x)) = p(x) - \min(p(x), q(x))$$

1. Total Probability: $$\text{Total } P(x) = \min(p(x), q(x)) + [p(x) - \min(p(x), q(x))] = p(x)$$

The draft distribution $q(x)$ completely cancels out. Whether the draft model has 50% accuracy or 90% accuracy affects only speed, never correctness.

6. Modern Variants: Beyond Standalone Draft Models

While standalone draft models (like Llama-3-8B drafting for Llama-3-70B) work well, the open-source community and inference engines (vLLM, SGLang, llama.cpp) have developed even faster variations:

A. Medusa (Multi-Head Self-Speculative Decoding)

Instead of maintaining an entire second model in VRAM, Medusa attaches multiple lightweight feed-forward decoding heads to the target model's final hidden state. Head 1 predicts token $t+1$, Head 2 predicts token $t+2$, Head 3 predicts token $t+3$. This eliminates the draft model footprint entirely.

B. EAGLE (Extensible Autoregressive Generation)

Standard draft models operate on text tokens, losing the rich contextual embeddings computed by the target model. EAGLE passes the top-layer feature vectors of the target model into the draft head. This boosts token acceptance rates from ~60% to over 80-85%.

C. Tree-Based Attention (SpecInfer)

Linear drafting guesses a single path $[x_1, x_2, x_3]$. If $x_1$ is rejected, everything else is wasted. Tree attention constructs a tree of multiple speculative branches (e.g. top-2 candidates at each depth). The target model verifies all candidate branches in parallel using a custom 2D attention mask, guaranteeing a higher average accepted token yield per step.

Linear Speculation (Fragile):
[Root] -> x1 -> x2 -> x3 (If x1 fails, x2 and x3 are dead)

Tree Attention Speculation (Resilient):
               +--> x1a --> x2a
[Root] --------|
               +--> x1b --> x2b
(If x1a fails, x1b might still be accepted!)

7. When Speculative Decoding Fails (Production Gotchas)

Speculative decoding is not a magic bullet for every deployment. Here are the 4 scenarios where it actually hurts performance:

1. High Batch Sizes (Compute-Bound Saturation)

Speculative decoding relies on spare GPU compute. When serving hundreds of concurrent requests (batch size 32, 64, or 128), your GPU is already 100% compute-bound. Adding draft forward passes creates compute contention and reduces overall throughput. Speculative decoding shines primarily at batch sizes 1 to 8 (low-latency streaming).

2. High Temperature and Creative Entropy

When sampling with high temperature ($T > 1.0$) on highly creative tasks, the output entropy is high. The overlap $\alpha = \sum \min(p, q)$ shrinks dramatically, causing frequent early rejections. Speculative decoding works best on structured output, code generation, and deterministic reasoning ($T \le 0.7$).

3. Tokenizer and Vocabulary Mismatches

The draft model and target model must share the exact same tokenizer vocabulary and special token mappings. If token IDs do not map to identical string representations, the logits cannot be compared directly.

4. VRAM Overhead on Edge Devices

Running a draft model requires reserving additional GPU VRAM for its weights and its own separate KV cache. On memory-constrained GPUs (e.g. 16 GB or 24 GB consumer cards), this extra memory might push the target model into slower quantized formats or offload buffers.

Summary & Mental Model

• The Problem: Single-token decoding is memory-bandwidth bound. 99% of GPU latency is spent streaming weights from VRAM to compute cores.
• The Workaround: Evaluating $K$ tokens in parallel takes the same memory bandwidth time as evaluating 1 token.
• The Execution: A tiny draft model quickly proposes $K$ tokens. The target model verifies all $K$ tokens in a single parallel forward pass.
• The Guarantee: Leviathan's speculative sampling algorithm mathematically recovers the exact target model probability distribution with zero quality loss.
• The Sweet Spot: Low batch sizes, low temperature, deterministic coding, and agent tool-calling loops.