Docs / Guides

Run an OpenAI-compatible local server

Expose Neutrino on localhost for OpenAI SDKs and agent frameworks, with streaming, tools, and persistent KV sessions.

Start the server

shell

fermion serve
# Listening on http://127.0.0.1:8000

The default bind is localhost on port 8000. The server loads one resident model and handles one request at a time. Additional requests wait behind the active generation.

Make a request

bash

curl http://127.0.0.1:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "neutrino",
    "messages": [{"role": "user", "content": "Hello"}],
    "temperature": 0,
    "max_tokens": 128
  }'

python

from openai import OpenAI

client = OpenAI(
    base_url="http://127.0.0.1:8000/v1",
    api_key="local",
)

response = client.chat.completions.create(
    model="neutrino",
    messages=[{"role": "user", "content": "Hello"}],
)
print(response.choices[0].message.content)

Bind and authentication

shell

fermion serve --host 127.0.0.1 --port 8000 --api-key "$FERMION_API_KEY"
fermion serve --cors

Sessions and context capacity

The native server keeps a resident session by default and can reuse a matching prompt prefix across requests. --session-ctx defaults to 8192 positions. Requests that do not fit fall back to one-shot mode and are counted on /health.

shell

fermion serve --session-ctx 16384 --max-new-ceiling 2048
fermion serve --no-session

Leave enough room for both the prompt and generated tokens. A high token ceiling against a small session context can force every request onto the one-shot path.

Speech

shell

fermion serve --model phonon

Serving a speech model mounts the audio routes instead of the chat routes. POST /v1/audio/transcriptions accepts a multipart file upload in the OpenAI audio shape and returns the transcript:

bash

curl -s http://127.0.0.1:8000/v1/audio/transcriptions \
  -F file=@clip.wav
# {"text": "The transcript."}

GET /v1/audio/stream carries live transcription over a standard WebSocket. Send one JSON text frame naming the sample rate and PCM format, then binary frames of raw audio. The server answers with partial, final, and done JSON frames, and each partial is the whole current hypothesis. Finish with {"type":"end"} or a clean close; one live stream runs at a time.

Streaming API covers the full protocol, and Transcribe and dictate covers response formats and limits.