Today's LLMs are autoregressive: they generate one token at a time, and each new token depends on every token that came before it. For each token, the model runs a full forward pass, appends the result to the input and runs the entire cycle again to generate the next token. That is what slows things down.
One solution to this delay is key-value caching. As the model processes each token, it calculates internal representations called keys and values. Recalculating these from scratch every time wastes effort; instead, the model stores them in a cache and reuses them. But caching only goes so far. The model still requires one full computation cycle per token; that remains the bottleneck.
Enter speculative decoding. This technique works in two steps: draft and verify. First, a smaller, faster model guesses several tokens ahead. That's the draft. Then the target LLM checks every guess in a single pass. That's verification. When the guesses are accurate, you save real compute and time. But when the guesses miss, the advantage disappears: you still pay for the verification check without gaining any speed.
More methods
Cache isn't the only way to get a draft. Some methods train a model to predict it, like EAGLE. This approach, known as parameter-based speculative decoding, requires extra parameters and dedicated training.
Another approach is parameter-free speculative decoding, which retrieves text the model already has. It works well for structured and agent-based tasks, pulling text from the prompt or from other sources. But there is a catch. The main LLM still has to check that draft, and this is where things can go wrong. A matching start doesn't guarantee a matching finish. You pay for the full check regardless, and you only learn how well the draft held up after you've paid for it. Hand the model a 20-word draft and watch it accept just two, and you've paid full price for a bargain that didn't deliver. The "free" draft turns out to be expensive to verify, for very little in return.
Another technique is adaptive speculation, which adjusts speculative decoding while the model is running. Speculation works better with runtime control over draft length, tree shape, budget or verification rules. But it still can't answer the core question: once a cache match is found, is it even worth verifying?
To address this, we developed the hybrid verified decoding method (Figure 1). Before checking a draft, we estimate how many tokens in the parameter-free draft the model will accept. That prediction drives the next step. If the estimate is high, we send the cache draft for checking. If it’s low, we switch to a model-based drafter instead. The target model still makes every final call. Our predictor just picks which draft gets checked.
We trained the predictor on trace replay using real decoding runs to teach the system what pays off and what doesn't. Then we tested it across three LLMs and 16 datasets, evaluating the full pipeline: input structure, cache opportunity, verification payoff, draft selection, computational overhead and sensitivity to prompt templates.
In brief:
- We turn draft selection into a payoff question: how many tokens will the model accept?
- We developed hybrid verified decoding to predict payoff, choosing cache- or model-based drafting before checking.
- We train the predictor using past generation traces, so it learns without needing heavy supervision.
- We test it on three LLMs, 16 datasets and four other methods for comparison.
- We study cache use, payoff, predictor choices, cost and prompt sensitivity, then validate it in a real vLLM deployment.
Model details
We'll now go through the technique step-by-step.
The question we asked was, “before checking can we guess how good this draft would be?” Hybrid verified decoding answers this question, but let’s see how.
Step 1: How draft-and-check works in general
We call the starting text the prefix. The drafter inspects this prefix and guesses the next few tokens, producing a draft. The target LLM then verifies that draft, accepting the valid tokens in sequence and rejecting the rest. The number of accepted tokens is called the accepted length. If the count is high, we save significant compute and time. If it’s low, we pay for verification in vain. That accepted length is the decisive metric: it tells us whether a draft was worth the effort, and it raises the core question of how we decide which draft to verify in the first place.
Step 2: Cache-based drafting (the reuse method)
This parameter-free approach reuses previously generated text and requires no additional training. It works especially well on repetitive content like code or agentic workflows. Instead of generating a draft from scratch, the system inspects the prompt and the tokens generated so far, looks for patterns where the same text appeared earlier and retrieves the text that followed that match. This mechanism, known as a suffix cache, stores recurring patterns for fast lookup. It’s a lightweight, efficient technique, but it doesn't guarantee the draft will be accepted.
Step 3: Teaching the model to predict the payoff
We can predict how many tokens the model will accept before checking the cache draft. This is a metric we call payoff. Using a lightweight predictor model, we estimate this payoff based on signals we already have: draft length, recent cache performance or pattern match quality. If the predicted payoff is high, verifying the cache draft is worth the compute; if it’s low, it isn't.
Because we only know the true payoff after verification, we train the predictor on historical decoding traces. We ask the cache what it would have drafted at each step of a past run, then compare that draft to what the LLM actually generated next. The count of matching tokens gives us the ground truth. Repeating this process creates the training examples our predictor needs to make reliable real-time calls.
Step 4: Hybrid verified decoding (complete method)
Here’s how the hybrid verified decoding workflow operates in practice:
- Inspect the current prefix and query the cache for a draft.
- Ask the predictor: How many tokens will the target model accept?
- Make the routing decision:
- If the predicted payoff is high enough, send the cache draft for verification.
- If not, generate a draft with the model-based drafter and send that draft instead.
Verify the chosen draft using the target model, then append the accepted tokens to the output.
Update the cache and generation history with the newly accepted tokens.
Repeat from step 1 until generation is complete.
Hybrid verified decoding algorithm
Now we’ve been through the flow, let’s have a deeper look at the hybrid verified decoding algorithm. Below we provide a line by line explanation.
Target model M, model-based drafter Dₘ, suffix cache 𝒞, payoff predictor gθ, threshold τ.
S No. | Steps | Description |
| 1 | Initialize the current prefix χ with the input prompt | Start with the prompt. Warm up the cache. Start tracking recent history. |
| 2 | Initialize C from cache-warmup traces and the input prompt | -do- |
| 3 | Initialize the recent-history summary h | -do- |
| 4 | While generation is not finished do | Keep looping until the text is fully generated |
| 5 | Dc <- Dc (χ ; 𝒞) | Ask the cache for the draft |
| 6 | ϕ ← Φ ( χ, dᶜ, 𝒞 , h), ŷ ← gθ (ϕ) | Turn what's happening right now into a set of numbers, then use those numbers to guess how many tokens of this draft the real model will accept |
| 7 | If ŷ >= τ then | -do- |
| 8 | d ← dᶜ | If the predicted payoff is high enough, use the cache draft. Otherwise, fall back to the trained drafter. |
| 9 | else | -do- |
| 10 | d ← Dm(χ) | -do- |
| 11 | End if | -do- |
| 12 | a, z ← Verify (χ, d; M) | Send the chosen draft to the real model for checking. Get back how many tokens were accepted (a) and the committed tokens (z). |
| 13 | Append the committed block z to χ | Add the new tokens to the output. |
| 14 | Update 𝒞 and h with z | Update the cache and history so the next loop has fresh information. |
| 15 | End while | Repeat until done |
Our experiments
To see how this technique works and performs in practice, we ran a number of experiments.
Setup
We tested our method on three target models: Qwen3-8B, Qwen3-4B and Llama 3.1-8B. Each one pairs with its own EAGLE3 checkpoint as the fallback drafter. We run all main timing tests on H100 GPUs, with deterministic decoding.
We evaluate on 16 datasets, across six workload types.
| Workload family | Dataset | Source |
| Agentic / workflow editing | Delegate-52 | microsoft/delegate52 |
| InstructEdit/FineEdit | YimingZeng/FineEdit_bench | |
| Code / repository | RepoBench | tianyang/repobench_python_v1.1 |
| SWE-bench OpenHands | SWE-bench/SWE-bench_Verified | |
| Magicoder | ise-uiuc/Magicoder-Evol-Instruct-110K | |
| MBPP | google-research-datasets/mbpp | |
| Structured generation | BFCL | llamastack/bfcl_v3 |
| Spider | NBAmine/xiangai-spider-with-context | |
| Multi-hop QA | HotpotQA | hotpotqa/hotpot_qa |
| MuSiQue | dgslibisey/MuSiQue | |
| 2WikiMultiHopQA | Alab-NII/2wikimultihop | |
| Long-context | InfiniteBench | xinrongzhang2022/InfiniteBench |
| CNN/DailyMail | abisee/cnn_dailymail | |
| GovReport | ccdv/govreport-summarization | |
| Open-ended instruction / chat | Alpaca | tatsu-lab/paca |
| MT-Bench | Official EAGLE GitHub repository (MT-Bench prompts) |
These datasets vary in input length, structure, context reuse and output format.
We run everything inside the official EAGLE repository's generation loop, using each model's own chat template. We measure throughput as tokens generated per second. Speedup is simply that throughput divided by the baseline's throughput.
Baselines
We compare our method against four baselines.
Greedy decoding: This is the plain reference. No drafter, no cache. Just the target model, running alone.
EAGLE3: A strong, trained drafter. It makes drafts using its own checkpoint. The target model checks them.
SuffixDecoding: The cache-only baseline. It builds drafts from a suffix cache, then sends them for checking.
Frequency-based hybrid decoding: A rule-based hybrid. Same setup as ours: try cache first, fall back to EAGLE3. But the switch rule is simpler. It just uses SuffixDecoding's frequency score to decide.
Evaluation metrics
We ran all main results on H100 GPUs, about 50 GPU-hours total, covering trace prep, predictor training and all timing runs.
For each run, we track three raw numbers: tokens generated (N), wall-clock time (T) and decoding cycles (C). From these, we derive four metrics: throughput, speedup, cycles per 1K tokens and time per cycle.
Analysis
We ran into some interesting patterns along the way. Here's what the data told us:
What makes a task cache-friendly?
Cache opportunities aren't the same everywhere. Agentic and coding workflows feature frequent repetition across file paths, entities and structural patterns. Structured generation relies on rigid schemas and tool formats, while long-context tasks carry extended, recurring reference text. Open-ended chat has the least repetition of all.
The pattern is straightforward: more repetition creates more cache opportunity. But knowing where a cache draft comes from isn't the same as knowing if it's any good; its actual payoff still depends on whether it fits the current generation state.
When does a cache draft become worth verifying?
Agentic, coding and long-context tasks produce more high-payoff drafts with longer accepted lengths. Structured and multi-hop QA tasks lean toward low or marginal payoff. Open-ended tasks still produce some high-payoff drafts, just far fewer of them.
This is the key distinction: cache opportunity and payoff aren't the same thing. A cache match gives you a candidate, but its real value depends on how many tokens the target model actually accepts. High-payoff drafts earn back their verification cost; low-payoff drafts simply waste it. That's why we predict payoff before verifying; routing only the drafts with high expected payoff through to the model.
Can the predictor actually spot high-payoff drafts?
We tested this directly. We collected cache drafts from test prompts, then compared their actual accepted length against the target model's output. Drafts with an accepted length of six or more counted as oracle high-payoff; a retrospective benchmark used strictly for evaluation.
We then evaluated our predictor against that benchmark using precision and recall. The predictor successfully identified most high-payoff opportunities before verification occurred. In other words, it reframes the selection into a question of state (is this the right moment for this draft?) and it answers that question reliably.
How does this turn into real speedup?
The mechanics are simple: fewer verification cycles, similar cost per cycle and more tokens committed per check.
On Qwen3-8B with InfiniteBench, we're 1.59x faster than EAGLE3 on RTX 5090 and 1.73x faster on H100. Cycles per 1,000 tokens drop from 260 to 153, and from 257 to 135. On B200, the gain is smaller (1.06x), since cycles only drop from 257 to 230. Our method helps most on consumer and standard server GPUs, where cutting cycles makes the biggest difference.
We also tested it inside vLLM, a real serving system rather than just an offline benchmark. It held up: On MT-Bench, we're 2.31x faster than EAGLE3 and 1.21x faster than SuffixDecoding. On CNN/DailyMail, we're 3.56x and 1.16x faster, respectively.
Does prompt format matter?
We tested this on Llama 3.1-8B with MT-Bench, changing only the prompt template. Without the official Llama template, EAGLE3 struggled, managing just 1.20 accepted tokens and 455 cycles per 1,000 tokens. Adding the template back improved performance sharply: 4.21 accepted tokens and 192 cycles.
A trained drafter, it turns out, is sensitive to exactly how the prompt is built. Hybrid verified decoding doesn't have that weakness. Even without the template, it still reaches 7.32 accepted tokens and just 120 cycles. When the trained drafter loses its footing, cache drafts quietly pick up the slack.
Results
Cache drafts save time, but only when accepted. Low-payoff drafts need filtering early, or they just add cost.
Across 48 model-dataset pairs (16 datasets × three models), our method beats greedy decoding in 47 cases and SuffixDecoding in 46, with average speedups of 2.97x and 2.64x.
The gains are clearest in agentic workloads (Delegate-52, InstructEdit/FineEdit), where we beat both EAGLE3 and frequency-based hybrid decoding across all three models. These tasks repeat structure, making long cache drafts easy to find, even though details often shift. Our payoff predictor catches that shift before verifying.
Overall, the method works best when cache opportunities are frequent.
We tested two model families: Qwen and Llama. Our hybrid method picks between just two draft sources: a suffix cache and EAGLE3. We haven't tested other model families or other draft-source combinations, such as multi-head drafters.
Conclusion
We introduced hybrid verified decoding around a simple idea: before checking a cache draft, predict how well it will perform. Then use that prediction to choose between a cache draft and a model-based drafter.
Across models and tasks, this approach delivers tangible gains. It can drive up throughput by directing verification effort where it actually pays off.
The broader takeaway is clear: speculative decoding doesn't just need stronger drafters. It needs runtime systems that know which drafts are worth checking in the first place.
Disclaimer
In addition to the original paper, we used Claude (operating within the Thoughtworks enterprise workspace) and Gemini (via the Thoughtworks on-brand — global brand voice pilot) to assist with diagram design and alignment with the Thoughtworks writing guide. Both are official AI models approved and licensed for use by Thoughtworks.