Documentation.Documentation.

$ pip install fermion-research
$ fermion chat --model fermionresearch/Neutrino-8B

The package is fermion-research on PyPI and the default artifact is fermionresearch/Neutrino-8B. The first fermion chat --model fermionresearch/Neutrino-8B runs a fixed sequence before the prompt appears. Every later run skips straight to the load.

1. 1

   Fetch

   The downloader pulls the 2.56 GB coded transport for Neutrino-1 8B from the release repository into the standard Hugging Face cache and shows its progress.

2. 2

   Expand

   The transport decodes losslessly to the 3.88 GB serving artifact. One file; there is nothing to assemble.

3. 3

   Verify

   The engine computes the artifact's SHA-256 and compares it against the digest published with the release. A mismatch aborts the load with both digests printed.

4. 4

   Bind a backend

   The engine probes the host and binds the prebuilt native runtime published for it, compiled CPU binaries for macOS arm64 and Linux x86-64, with a bit-exact torch reference path underneath. fermion info names the backend in use, and --backend pins it.

Then the prompt opens and tokens stream. At the end of each session the engine prints its steady-state decode rate for that machine.

What the engine maps is almost entirely two lanes: the ternary weight lane holding every projection of the 36 decoder layers, uniform at 72.4 MB per layer, and the int8 token-embedding table. Scales, norms, and the header account for the remaining 0.7 percent. The 2.56 GB transport is this same file, coded tighter for the wire.

The pack ships in the 8B repository under gguf/. Our fork is the runtime for it.

The pack uses the FV5 tensor type: stock llama.cpp, Ollama, and LM Studio cannot load it. The fermion-fv5 branch is the CPU/CUDA build and requires -DGGML_METAL=OFF. For Apple-GPU kernels, use the live fermion-fv5-metal branch and omit that flag.

$ git clone https://github.com/fermionresearch/llama.cpp && cd llama.cpp
$ git checkout fermion-fv5
$ cmake -B build -DCMAKE_BUILD_TYPE=Release -DLLAMA_CURL=OFF -DGGML_METAL=OFF
$ cmake --build build -j --target llama-completion

Then run the pack with every layer offloaded:

$ ./build/bin/llama-completion -m neutrino-8b-fv5.gguf -ngl 99 -c 4096 -p "Explain why the sky is blue." -n 256 --temp 0 -no-cnv

-ngl 99 offloads all layers; -c 4096 is the context length the published number is measured at. In that configuration the 8B pack decodes at 30.7 tokens per second on an NVIDIA L4 and holds 4.68 GiB of VRAM, comfortably inside an 8 GB card.

Decode rates are measured single stream: one prompt, one reply, batch of one, greedy decoding, steady state over at least 512 generated tokens. The engine prints exactly this number at the end of every session, so a session on your machine is already a run of the protocol.

On the GGUF path, the fork carries llama.cpp’s own benchmark tool, which separates prefill from decode:

$ ./build/bin/llama-bench -m neutrino-8b-fv5.gguf -ngl 99 -p 512 -n 128

The published L4 number is the tg128 row, the generation row. Prefill rates run far higher and are not comparable; quoting one as the other is the most common way speed claims go wrong.

For the accuracy battery, each entry on the model card states the suite version, prompt format, shot count, and answer extraction rule. Those four parameters define the number; run any public evaluation suite with the same four against the shipping artifact.