Eagle3

Eagle3 is the baseline feature-space draft architecture and the method AngelSpec inherits from TorchSpec. It predicts tokens autoregressively and is trained with test-time training (TTT).

How it works

Eagle3 fuses the target model’s hidden states into the draft input (an fc layer concatenates the token embedding with the projected hidden states) and then predicts the next token with a small transformer. During training it runs a TTT loop: it predicts a token, feeds its own prediction back as the next input, and repeats for a fixed number of steps (ttt_length). This teaches the draft model to stay accurate over the multi-token horizon it will speculate at serving time.

The loss is forward KL divergence against the target model’s distribution, so the draft model learns to match the target’s next-token probabilities rather than only the argmax token.

Model variants

Eagle3 draft models follow the architecture of the target model family:

• Llama-family targets use the Llama-based Eagle3 draft (LlamaConfig).

• DeepSeek-family targets, which use multi-head latent attention (MLA), use the MLA-based Eagle3 draft (DeepseekV3Config).

The draft model shares the target’s embedding and lm_head (frozen), and supports vocabulary pruning to shrink the lm_head to a smaller token set. See Checkpoint Conversion for pruning.

When to use it

Eagle3 is a strong, well-understood default. If you want fewer forward passes per drafted block, consider the block-parallel members of the family (DFlash, DFlare, DSpark).