Docs / Guides

Transformers, GGUF, and MLX

Use the same Neutrino release through the library path, the Fermion llama.cpp fork, or Apple GPU code.

Load from a local directory in Transformers

python

import fermion  # registers the trtc_v4 model type
from transformers import AutoModelForCausalLM, AutoTokenizer

path = "/data/models/Neutrino-8B"
tokenizer = AutoTokenizer.from_pretrained(path)
model = AutoModelForCausalLM.from_pretrained(path)

Run the GGUF pack

The 8B repository contains a 4.09 GB GGUF file for the Fermion llama.cpp fork. Its FV5 data types are not available in upstream llama.cpp.

shell

git clone -b fermion-fv5 https://github.com/fermionresearch/llama.cpp
cd llama.cpp
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release -j

./build/bin/llama-completion \
  -m /path/to/neutrino-8b-fv5.gguf \
  -ngl 99 -c 4096 -p "Explain why the sky is blue." -n 256 --temp 0 -no-cnv

Run on Apple GPU with MLX

Each model repository includes an mlx/ package that runs the same container on Apple silicon. It is a separate integration and is not installed by pip install fermion-research.

shell

git lfs clone https://huggingface.co/FermionResearch/Neutrino-8B
cd Neutrino-8B/mlx
pip install -r requirements.txt