Every agent already runs a loop. Loop engineering adds a loop around the agent itself, enabling it to evaluate its output, try again when the work falls short, and refine its instructions when the same mistakes recur. Today, you perform that role: reviewing the work, diagnosing what went wrong, and prompting the agent again. This article shows how to automate that process with a working example, while exploring where human judgment still belongs. A 70-billion-parameter model requires reading roughly 140 GBs of weights out of the GPU memory. On a modern data center GPU, this transfer can take tens of milliseconds. The actual calculation applied to these weights takes a fraction of that time. This means that the processor’s math units are unused for most of the time taken by the token generation step. Speculative decoding is a technique that converts this unused capacity into output. A second, much smaller model produces several candidate tokens in advance. The large model evaluates all of them in a single forward pass instead of one pass per token, resulting in 2-3 times faster generation. To make things better, the text produced remains statistically identical to the output of the large model running alone. In this article, we will look at how speculative decoding works. Here’s what we will cover:
- Why token generation runs one step at a time
- What a GPU spends its time on during generation
- How several candidate tokens are evaluated in a single pass
- The accept and reject loop, and what happens when a candidate is wrong
- Why output quality is preserved exactly
- Acceptance rate, and why it varies by workload
- The four places a draft can come from
- When speculative decoding stops helping Disclaimer: This post is based on publicly shared details from various sources. References at the end. Please comment if you notice any inaccuracies. Autoregressive Decoding Text generation works one token at a time. The model reads everything produced so far, computes a...