Docs / Guides

Run an OpenAI-compatible local server

Expose Neutrino on localhost for OpenAI SDKs and agent frameworks, with streaming, tools, and persistent KV sessions.

Start the server

shell

fermion serve
# Listening on http://127.0.0.1:8000

The default bind is localhost on port 8000. The server loads one resident model and handles one request at a time. Additional requests wait behind the active generation.

Make a request

bash

curl http://127.0.0.1:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "neutrino",
    "messages": [{"role": "user", "content": "Hello"}],
    "temperature": 0,
    "max_tokens": 128
  }'

python

from openai import OpenAI

client = OpenAI(
    base_url="http://127.0.0.1:8000/v1",
    api_key="local",
)

response = client.chat.completions.create(
    model="neutrino",
    messages=[{"role": "user", "content": "Hello"}],
)
print(response.choices[0].message.content)

Bind and authentication

shell

fermion serve --host 127.0.0.1 --port 8000 --api-key "$FERMION_API_KEY"
fermion serve --cors

Sessions and context capacity

The native server keeps a resident session by default and can reuse a matching prompt prefix across requests. --session-ctx defaults to 8192 positions. Requests that do not fit fall back to one-shot mode and are counted on /health.

shell

fermion serve --session-ctx 16384 --max-new-ceiling 2048
fermion serve --no-session

Leave enough room for both the prompt and generated tokens. A high token ceiling against a small session context can force every request onto the one-shot path.