---
title: "Run an OpenAI-compatible local server"
description: "Expose Neutrino on localhost for OpenAI SDKs and agent frameworks, with streaming, tools, and persistent KV sessions."
canonical: "https://www.fermionresearch.com/docs/serve/"
source: "Fermion Research"
---

Docs / Guides

# Run an OpenAI-compatible local server

Expose Neutrino on localhost for OpenAI SDKs and agent frameworks, with streaming, tools, and persistent KV sessions.

## Start the server

```text
fermion serve
# Listening on http://127.0.0.1:8000
```

The default bind is localhost on port 8000. The server loads one resident model and handles one request at a time. Additional requests wait behind the active generation.

## Make a request

```text
curl http://127.0.0.1:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "neutrino",
    "messages": [{"role": "user", "content": "Hello"}],
    "temperature": 0,
    "max_tokens": 128
  }'
```

```text
from openai import OpenAI

client = OpenAI(
    base_url="http://127.0.0.1:8000/v1",
    api_key="local",
)

response = client.chat.completions.create(
    model="neutrino",
    messages=[{"role": "user", "content": "Hello"}],
)
print(response.choices[0].message.content)
```

## Bind and authentication

```text
fermion serve --host 127.0.0.1 --port 8000 --api-key "$FERMION_API_KEY"
fermion serve --cors
```

## Sessions and context capacity

The native server keeps a resident session by default and can reuse a matching prompt prefix across requests. `--session-ctx` defaults to 8192 positions. Requests that do not fit fall back to one-shot mode and are counted on `/health`.

```text
fermion serve --session-ctx 16384 --max-new-ceiling 2048
fermion serve --no-session
```

Leave enough room for both the prompt and generated tokens. A high token ceiling against a small session context can force every request onto the one-shot path.

## Speech

```text
fermion serve --model phonon
```

Serving a speech model mounts the audio routes instead of the chat routes. `POST /v1/audio/transcriptions` accepts a multipart file upload in the OpenAI audio shape and returns the transcript:

```text
curl -s http://127.0.0.1:8000/v1/audio/transcriptions \
  -F file=@clip.wav
# {"text": "The transcript."}
```

`GET /v1/audio/stream` carries live transcription over a standard WebSocket. Send one JSON text frame naming the sample rate and PCM format, then binary frames of raw audio. The server answers with `partial`, `final`, and `done` JSON frames, and each partial is the whole current hypothesis. Finish with `{"type":"end"}` or a clean close; one live stream runs at a time.

[Streaming API](https://www.fermionresearch.com/docs/speech-streaming/) covers the full protocol, and [Transcribe and dictate](https://www.fermionresearch.com/docs/speech/) covers response formats and limits.

Source: [https://www.fermionresearch.com/docs/serve/](https://www.fermionresearch.com/docs/serve/)