Docs / Guides

Speculative decoding

Let Neutrino-0.6B propose tokens while Neutrino-8B verifies every token that reaches the user.

Enable the canonical draft

shell

fermion generate "Explain speculative decoding." --draft auto
fermion chat --draft auto
fermion serve --draft auto

--draft auto downloads and caches FermionResearch/Neutrino-0.6B. You can also pass a local model directory, container path, or Hugging Face repository id. The draft must share the target tokenizer and vocabulary.

What changes and what does not

The small model proposes a short run of tokens. Neutrino-8B checks those proposals against its own greedy choices. Rejected proposals are replaced by the target model. The draft changes the amount of work, not the target model.

Native drafting is greedy-only. If temperature is above zero, the native runtime prints a note and runs without the draft. A resident session is also disabled while a draft is active.

Prove identity on your machine

shell

fermion verify --draft auto
fermion verify --draft auto --prompts 12 --max-new 32

verify runs the target alone and with the draft, then compares token streams position by position. It defaults to float32 because the product identity guarantee is CPU, greedy, float32.