Docs / Reference

Server API reference

Routes, request fields, streaming frames, errors, context clamps, and operational health data.

Endpoints

Method	Route	Purpose
POST	/v1/chat/completions	Chat completions with streaming and tools.
POST	/v1/completions	Legacy single-string text completion.
GET	/v1/models	The resident model card.
GET	/v1/models/{id}	One model card, or model_not_found.
GET	/health, /v1/health, /healthz	Runtime, model, session, tool, and integrity status.
GET	/	Service name, version, model id, and endpoint list.
OPTIONS	any route	CORS preflight when --cors is enabled.

Chat-completion fields

Field	Behavior
model	Accepted for client compatibility. One model is resident, and the response uses the configured served id.
messages	Required, non-empty. system, developer, user, assistant, and tool roles. Text content only.
tools / tool_choice	OpenAI function schemas and auto, none, required, or a named function.
functions / function_call	Legacy fields are accepted and converted to the current tools form.
temperature	0 through 2. Explicit values override server profiles.
top_p	0 through 1.
repetition_penalty	Extension from 0.01 through 2. A value of 1 disables it.
max_tokens / max_completion_tokens	Positive integer, clamped to server and model context limits.
stop	String or up to four strings, enforced server-side and streaming-safe.
n	Only 1 is accepted.
stream	Boolean. Uses server-sent events.
stream_options.include_usage	Adds a trailing usage chunk with empty choices.

Unsupported fields

The server warns once per process when it sees unsupported sampling fields.

presence_penalty, frequency_penalty, top_k, min_p
logit_bias, seed, logprobs, top_logprobs
response_format and constrained JSON output are not implemented.

Streaming format

Streaming uses real SSE with Content-Type: text/event-stream and chunked transfer. The sequence is role delta, content deltas, an optional indexed tool-calls delta, finish chunk, optional usage chunk, then [DONE].

text

data: {"object":"chat.completion.chunk","choices":[{"delta":{"role":"assistant"}}]}

data: {"object":"chat.completion.chunk","choices":[{"delta":{"content":"Hello"}}]}

data: [DONE]

If generation fails after the stream starts, the server sends an in-band error frame followed by [DONE].

Errors and request limits

Errors use {"error":{"message", "type", "param", "code"}}. Unknown routes return an OpenAI-shaped 404. The request-body limit is 32 MB, and chunked request bodies are rejected.

Client token requests are clamped rather than rejected. The server applies --max-new-ceiling, the model’s 40,960-position window, and the remaining positions after prompt tokenization.

Health data

/health reports the model id and kind, container SHA-256 and path, byte size, package version, device and dtype, backend, native runner and session state, sampler defaults, tool profile, stuck-detector configuration and counters, container header, context, and KV reuse statistics.