Docs / Getting started

Choose and verify a backend

Fermion selects an optimized native CPU runtime where available and a PyTorch reference path elsewhere.

Platform matrix

This table is for the language models (fermion chat, fermion generate, and a Neutrinofermion serve). Speech backends are different: on Apple silicon, Phonon runs on the MLX engine; on Linux (x86-64 and 64-bit ARM), x86-64 Windows, and on a Mac without the MLX runtime or withFERMION_DEVICE=cpu, it runs on the CPU engine. See Speech platforms below.

Platform	Default path	Status
macOS arm64	Native CPU	Optimized path and local transport expansion
Linux x86-64	Native CPU	Optimized path
Linux arm64	PyTorch	Reference path
Windows x86-64	PyTorch	Reference path
CUDA or MPS device	PyTorch	Reference path

See what your machine selected

shell

fermion info
fermion bench --tokens 64

info names the active backend and explains why. Published throughput figures describe the native path. bench measures the exact path active on the current machine.

Pin or require a backend

shell

fermion chat --backend native
fermion generate "Hello" --backend torch --dtype float32

--backend native fails rather than silently using a slower path. Use --native-bin or FERMION_NATIVE_BIN to select a specific runner.

Native thread settings

The native runner uses one fewer thread than the detected CPU count, capped at 16. Set FERMION_THREADS to choose a count, and raise the ceiling deliberately with FERMION_MAX_THREADS.

shell

FERMION_THREADS=12 fermion bench
FERMION_MAX_THREADS=24 FERMION_THREADS=24 fermion bench

Speech platforms

Phonon speech models run on a Mac (via MLX), an NVIDIA GPU, or an ordinary CPU. On a Mac, the pip package is the complete runtime. On NVIDIA hardware, the Docker image runs the same weights:

shell

docker run --rm --gpus all ghcr.io/fermionresearch/phonon-cuda:latest \
  transcribe recording.wav

On Linux (x86-64 and 64-bit ARM) and x86-64 Windows, the same commands run on the CPU, no GPU required:

shell

pip install fermion-research torch safetensors soundfile scipy zstandard
fermion transcribe recording.wav

On Linux, installing torch from its CPU wheel index (--index-url https://download.pytorch.org/whl/cpu) skips the much larger GPU build. On Linux x86-64 the CPU engine needs glibc 2.34 or newer (Ubuntu 22.04, Debian 12, RHEL 9 and later) and about 7 GB of free RAM. On Windows the plain torch wheel already is the CPU build; a clean machine may also need Microsoft's vc_redist.x64.exe (the fix when import torch fails with WinError 126). A CPU image runs the same weights anywhere Docker does, Docker Desktop included:

shell

docker run --rm ghcr.io/fermionresearch/phonon-cpu:latest \
  transcribe recording.wav

Every container serves the same endpoint as fermion serve, so a client written against one works unmodified against the others. The Neutrino backend and sampler flags have no meaning for a speech model; if you set one, the server says so once at startup and ignores it.