Docs / Guides

Chat and generate

Use the interactive session for conversation and the one-shot command for deterministic scripts.

Interactive chat

shell

fermion chat
fermion chat --model FermionResearch/Neutrino-0.6B-Chat

On the native backend, a resident process holds the model and KV cache for the session. Each turn can reuse the existing prefix instead of loading the model and replaying the full conversation. Set FERMION_SESSION=0 to use one process per turn.

One-shot generation

shell

fermion generate "Summarize the main argument." --max-new 128
fermion generate --raw "Once upon a time" --max-new 80

--raw bypasses the chat template. The completion goes to standard output. The final statistics line separately reports wall throughput and steady-state decode throughput.

Why the defaults differ

Surface	Temperature	Top p	Repetition penalty	Max new
chat	0.01	1.0	1.05 over 256 tokens	1024
serve	0.01	1.0	1.05 over 256 tokens	1024
generate	0.0	1.0	1.0, disabled	256

Chat and serve use the model’s evaluated conversational settings. Generate stays greedy with repetition penalty disabled so command-line scripts are repeatable. Every setting remains available as a flag.

Sampling controls

shell

fermion chat --temperature 0.7 --top-p 0.8 --rep-penalty 1.1
fermion generate "List three ideas" --temperature 0 --rep-penalty 1.05