--- title: "The Neutrino Engine" description: "An A100 can move the weights for one measured decoder step in 104 microseconds. The step takes 694. The Neutrino Engine is built around the gap." canonical: "https://www.fermionresearch.com/research/the-neutrino-engine/" source: "Fermion Research" --- # The Neutrino Engine An A100 can move the weights for one measured decoder step in 104 microseconds. The step takes 694. The Neutrino Engine is built around the gap. On an NVIDIA A100, moving the weights for one measured decoder step took 104 microseconds. The full step took 694. Roughly 590 microseconds remained in kernel dependencies and about 120 execution boundaries. Replacing the graph with one fused kernel did not recover that time; software barriers recreated much of the same wait. That gap is the starting point for the Neutrino Engine. The runtime executes packed weights directly, keeps the KV cache bounded as context grows, and verifies that CUDA, Apple silicon, and x86 emit the same greedy token stream. A 596M draft model can change the execution schedule, but it never gets to change the 8B model's answer. ## A 104 microsecond floor inside a 694 microsecond step Single-stream decode reads almost every model weight for each token. Each of the 36 decoder layers applies its attention and feed-forward matrices to the current activations. At batch size one, a weight usually serves one multiply-accumulate before the next token needs it. Weight bytes divided by memory bandwidth set a lower limit on token latency. A smaller artifact lowers this limit on every backend. Neutrino-1 8B uses 3.88 GB for the serving artifact, compared with about 16 GB at fp16. This reduces the bandwidth limit by about four times. Plain decode reaches 396 tokens per second on an H100 80 GB. At that rate, the engine moves about 1.5 TB of weight traffic each second. To separate bandwidth from execution overhead, we instrumented the same engine path on a 24-layer model and an NVIDIA A100. Pure DRAM work took 104 microseconds per token. The layers used another 590 microseconds across about 120 kernel boundaries and their dependency chains. Each launch cost about one microsecond. A single fused kernel did not recover the 590 microseconds. Its software barriers replaced the graph launches with similar waits. The engine therefore focuses on memory traffic, not only kernel count. The output projection behaves differently. It maps the final hidden state to the full vocabulary and reaches the DRAM limit. This work creates 43 percent of DRAM traffic in each step. Backend limits differ. The Apple M5 CPU path reaches 24.9 tokens per second and moves about 97 GB/s. That rate is close to the laptop memory limit, so fewer weight bytes translate directly into more tokens. GPUs retain more launch and dependency overhead. Every backend reads the packed weights directly. The engine keeps no float shadow copy. It does not convert weights during decode or read a tensor twice for one token. 24-layer model, NVIDIA A100, same runtime lane, batch one. The vertical rules inside the second segment are the kernel boundaries. Fig. 1One decoder step on a microsecond axis, separated into DRAM time and additional launch or dependency latency. ## The file is not the traffic File bytes and per-token bytes differ because the engine does not read every tensor in the same way. The 3.88 GB file contains 2.60 GB of ternary projection weights. This 67.2 percent lane includes the query, key, value, output, gate, up, and down matrices in all 36 layers. Each token streams this full lane. Two untied int8 vocabulary tables use another 1.24 GB, or 32.1 percent. Each table has 151,936 rows and 4,096 columns. The engine reads selected input rows but scores the full output table. Metadata, row scales, and normalization vectors use 26 MB. Each layer has a 72,351,744-byte weight slab. Attention uses 21.7 percent, and the feed-forward block uses 78.3 percent. At short context, feed-forward matrices create most per-token traffic. Vocabulary takes 47.5 percent of the 0.6B file, so model scale changes this balance. Fig. 2Byte counts from the released artifact, with projection weights separated by decoder layer. ## At 26K tokens, the cache becomes larger than the model Model weights have a fixed memory cost. The KV cache grows with context and joins every later attention step. Neutrino-1 8B has 36 layers, 8 key-value heads, and 128 dimensions per head. One cached position uses 147,456 bytes at 16 bits per element. This equals 144 KiB of keys and values per token. Grouped-query attention keeps only 8 key-value heads. Caching all 32 query heads would use four times more memory. Cache use grows linearly. It reaches 0.60 GB at 4,096 tokens and 2.42 GB at 16,384. It reaches 4.83 GB at 32,768 and 6.04 GB at 40,960. After about 26,000 tokens, the cache exceeds the 3.88 GB model file. Runtime planning must therefore include cache memory and traffic. Fig. 3KV-cache allocation by context length, with the 3.88 GB model artifact shown for comparison. ## The same weights take four execution paths The same file is read by the CUDA, MLX, native arm64, and desktop x86 backends. Each backend reads the same bytes. The release has no platform-specific model export or quality tier. Evaluation uses the same artifact that users run. Greedy decode uses identical stored values and produces identical tokens. Execution still differs by device. Datacenter GPUs, integrated GPUs, and CPU thread pools have different launch costs. They also expose different parallelism and memory systems. H100-class GPUs use HBM. Apple silicon uses unified memory. Desktop x86 uses DDR behind deep cache hierarchies. The loader arranges the same values for each device at load time. This takes 1.9 seconds for the 8B and 0.3 seconds for the 0.6B on a datacenter GPU. Each backend has separate kernels, tiling, and instructions. Measured rates span about thirty times. Neutrino-1 8B reaches 763 tokens per second with drafting and 396 without it on an H100 80 GB. The optimized Apple M5 path reaches 33.7, and its CPU path reaches 24.9. Our llama.cpp fork reaches 30.7 on an NVIDIA L4 in 4.68 GiB of VRAM. Neutrino-1 0.6B exceeds 200 tokens per second on the same laptop CPU. All five measurements use the released 8B weights. The drafted H100 result additionally runs Neutrino-1 0.6B as the proposal model. Fig. 4Single-stream decode rates on a logarithmic scale, from 24.9 tok/s on an M5 CPU to 763 tok/s on a drafted H100 path. ## One changed token fails the release The release gate compares tokens. It does not score speed. A candidate build decodes a fixed prompt set with greedy decoding. The test compares each token with a stored full-precision reference. One different token fails the build, regardless of speed. A stack that emits different tokens serves a different model. The original quality results no longer apply to it. We apply this gate at every format boundary. Engine builds compare with the full-precision reference. Drafted decode compares with plain decode on the same artifact. Converted packs compare with the engine. Imported artifacts compare with their reference engines. We do not publish speed for a path that fails identity. ## The 0.6B model does not get to decide Speculative decoding changes the execution path more than any other engine feature. Neutrino-1 0.6B first proposes several tokens. The controller sets the count from recent acceptance. The 8B scores all proposed positions in one verify pass. The engine keeps the longest prefix that matches the 8B. At the first mismatch, it emits the 8B token and starts a new draft. Every emitted token therefore comes from the 8B greedy sequence. 1. 01The 0.6B draftsOne to seven candidate tokens, length set from recent acceptance. 2. 02The 8B verifies onceEvery drafted position scored in a single batched forward pass. 3. 03The agreed prefix is keptThe longest run where the draft chose what the 8B would have chosen. 4. 04The 8B closes the cycleThe first disagreement is replaced by the 8B's own token. Fig. 5The four stages of one speculative-decoding cycle, including bytes moved, verification passes, and accepted tokens. The draft file uses one twelfth of the 8B bytes. Six draft steps and one verify pass move about one fifth of the bytes used by seven plain 8B steps. The cycle saves time when the 8B accepts enough proposed tokens. Its minimum cost is one 8B step plus the small draft cost. We certified drafted decode before timing it. The shipped container produced 27,648 consecutive tokens with zero differences from plain decode. Every tested prompt class then exceeded the plain rate. Text determines acceptance length. It measures how many next tokens the 596M model predicts before the 8B chooses a different token. Counting and enumeration have the most predictable continuations. They accept 100 percent of six-token drafts. A 512-token counting run used 74 verify passes and emitted about seven tokens per 8B pass. It reached 763 tokens per second against 395.9 without drafting. Factual recall accepts 96.5 percent, or 5.8 of six tokens per pass, and reaches 613. Prose reaches 532. Chat and code accept fewer tokens but still exceed plain decode. Rings mark 200, 400, 600 and 800 tok/s; the dashed ring is 395.9 tok/s of undrafted decode on the same H100 80 GB. Clockwise from counting at the top. Fig. 6Drafted decode rate and acceptance by prompt class, compared with undrafted throughput. The controller adjusts draft length from recent acceptance. Short factual completions accept 97 to 99 percent across tested lengths. The controller proposes 5.5 tokens per pass there and keeps 5.49. Code and adversarial counting match for only one or two tokens. The controller uses shorter drafts for them. Their 512-token runs use 181 and 200 verify passes. One rule controls all nine text classes, and each class remains faster than plain decode. Fig. 7Draft length committed against tokens surviving the pass, nine classes, 512-token probe, 2026-07-25. ## The laptop gain is smaller, and still real The MLX draft-and-verify path keeps both models in one process under a 6 GiB limit. All six tested prompts matched plain decode token for token. On factual prompts, the rate rises from 22.00 to 25.71 tokens per second. This is a 1.168-times gain at 74.4 percent acceptance. These values are medians from three 512-token greedy runs with a four-token draft. Peak memory is 4.3 GiB. Each verify pass emits 3.98 tokens on average. Each step is one verify pass. The dashed line is the same machine without a draft model, 512-token greedy runs under a 6 GiB cap. Fig. 8Every step is one verify pass on the M5; the dashed line is the same machine decoding one token at a time. ## The engine also runs weights we did not train We also test the engine with public artifacts from other projects. Both engines run on the same machine in one session. We interleave rounds so heat and clock drift affect both sides. Token identity must pass before timing starts. The importer verifies every translated tensor against the source file before the first run. These measurements use weights that we did not train. A public 27B ternary model reaches 105.15 tokens per second in our engine. Its reference engine reaches 97.80. Our engine won all six interleaved rounds, and its slowest round exceeded the reference maximum. Our container also stores the same weights in a file 19 percent smaller than the published artifact. A public 2B ternary model reaches 102.4 tokens per second in our engine on an Apple M5. Its reference engine reaches 89.0 in three interleaved rounds. On one A100, our kernels run the same model 1.77 times faster than the lab GPU kernels. On desktop x86, they run 2.12 times faster than a published lookup-table engine on its target instruction set. At batch size one, our matrix-vector kernel runs 12 to 13 percent faster than an int4 matrix-vector kernel. Six interleaved rounds on one machine: reference range 97.59 to 98.12 tok/s; Neutrino Engine range 102.65 to 105.45 tok/s. The imported container is 19 percent smaller than the published artifact. Fig. 9Six interleaved rounds of the 27B measurement, both stacks on one rate axis. Six interleaved rounds on one machine; measured ranges do not overlap. The imported container is 19 percent smaller than the published artifact. Three same-moment rounds on one Apple M5 with thermal conditions shared between stacks. The weights are theirs and unchanged; the traversal is ours. Measured on the instruction set that stack was written for. The batch size decode actually lives at, on the operation that dominates it. Fig. 10Five measurements on other stacks' artifacts, each interleaved on one machine and identity-checked before scoring. ## What ships The engine installs from pip and reads the released artifacts directly. The model weights, manifest, GGUF pack, MLX implementation, and llama.cpp fork are public under their stated licenses. Fermion keeps the optimized kernels closed. The GGUF pack uses the FV5 tensor type and requires the Fermion Research llama.cpp fork. Stock llama.cpp, Ollama, and LM Studio do not load it. With full GPU offload, the fork reaches 30.7 tokens per second on an NVIDIA L4. It uses 4.68 GiB of VRAM at 4,096 tokens of context. The MLX pack serves Apple GPUs. Both converted formats must match the shipping model token for token before release. ## More from the lab ![An amber cellular orbit gathering around a compact central form](https://www.fermionresearch.com/images/backgrounds/amber-orbit.webp) AnnouncementJuly 27, 2026 ### Introducing the Neutrino-1 models Three open-weight models trained for a compact ternary format and released with CUDA, Apple-silicon, and x86 runtimes. ![A cobalt cellular membrane becoming progressively finer](https://www.fermionresearch.com/images/backgrounds/cobalt-cell.webp) ResearchJuly 27, 2026 ### Intelligence at one-eighth the bits One-shot two-bit conversion lands near chance. Training inside the constraint produces 72.1 MMLU from a 3.88 GB artifact. This is what changed inside the weights. Source: [https://www.fermionresearch.com/research/the-neutrino-engine/](https://www.fermionresearch.com/research/the-neutrino-engine/)