The Draft-Model Family

The Draft-Model Family#

AngelSpec supports several draft architectures behind a common training pipeline. They differ in how they predict tokens (autoregressive vs. block-parallel), how they consume the target model’s hidden states, and what loss they optimize. All are selected by the draft model’s config, so switching architectures is a config change, not a code change.

Comparison#

Architecture

Prediction

How it uses target hidden states

Eagle3

Autoregressive (test-time training)

Input fusion — hidden states concatenated into the draft input

DFlash

Block-parallel

Dual-source KV — multi-layer hidden projected into attention K/V

DFlare

Block-parallel

DFlash, with a learnable per-layer mix of the captured target layers

DFly

Block-parallel

DFlash FC context plus DFlare’s per-layer fusion as a residual (DFlareV2), with optional hidden-state correction

DSpark

Block-parallel + autoregressive head

DFlash backbone, plus a Markov bias and an accept-rate head

MTP

Single-head, test-time training

A full MoE decoder layer consuming the target’s last hidden state

  • Autoregressive drafters (Eagle3, MTP) predict one token at a time, feeding each step’s output back in — accurate per token, but N sequential forward passes to draft N tokens.

  • Block-parallel drafters (DFlash, DFlare, DFly, DSpark) predict a whole block of tokens in a single forward pass using a block-causal attention mask, trading some per-token accuracy for far fewer forward passes.

Shared training pipeline#

All architectures reuse the same infrastructure — the disaggregated pipeline, FSDP2 training, the data loader, and evaluation. The block-parallel models (DFlash, DFlare, DFly, DSpark) additionally share one training wrapper: DFlare, DFly, and DSpark are built on the DFlash backbone and attach their differences through a small set of extension hooks, rather than reimplementing the forward pass.

Draft and target also share weights where it helps: the draft model’s token embedding and lm_head are loaded from the target model and frozen, so they add no trainable parameters.

Selecting an architecture#

The draft config’s architectures field (and, for the DFlash family, a model_arch field) picks the model. The trainer dispatches on the config type:

DFlashConfig   → DFlash        (model_arch="dflare" → DFlare)
DSparkConfig   → DSpark        (model_arch="dflare" → DFlare-backbone variant, model_arch="dfly" → DFly)
MTPConfig      → MTP
LlamaConfig    → Eagle3 (Llama family)
DeepseekV3Config → Eagle3 (DeepSeek MLA family)

Architecture pages#