Fermion ResearchFermion Research

Deploy/facts checked 13 November 2026

Replace expensive API model calls with a small model built for the task

A frontier model is the right tool for the open-ended ten percent of requests. The other ninety percent are classification, extraction, transcription and routing, and a model a hundredth the size can do them on hardware you already own.

Talk to uscontact@fermionresearch.com

What usually happens

A team ships a feature on a frontier API because it is the fastest way to get it working. The feature succeeds, traffic grows, and the API line becomes the largest cost in the product. The calls are the same narrow job repeated millions of times: label a ticket, pull five fields from a document, transcribe a call, decide which flow a request belongs to.

What a task-specific small model changes

  • Size: a model of 0.1 to 9 billion parameters, or a 156 MB speech model, in place of a hosted model you cannot run yourself.
  • Where it runs: your own cloud, a CPU box, or the user's device. Data stays inside your environment.
  • Latency: milliseconds per decision instead of seconds per generation.
  • Cost: hardware you own at high utilisation, instead of a per-token price.

A public example: Phonon-2.5

Phonon-2.5 is our open speech recognition model: 25 languages from a 156 MB download. On the FLEURS test sets it averages 11.71 % word error, against 13.26 % for Parakeet TDT 0.6B v3, a 2.5 GB model covering the same languages: a sixteenth of the size and more accurate on average. Its published throughput (Phonon-2's figures, shared architecture) is 143 times realtime on eight AMD Zen 5 cores and 6,680 times realtime batched on an H100: an eight-core CPU box clears more than a hundred hours of audio per hour.

The same method, train a small model for exactly the task and measure it on public test sets, is what we apply to a customer's workload.

How an engagement runs

  • Week one: we take a sample of your real requests and the API outputs you already have, and agree the metric and the target.
  • We train and compress a model for the task and report accuracy against the API on a held-out set, with the latency and cost per request on the hardware you choose.
  • You deploy it in your cloud, on-premises or on devices, with an OpenAI-compatible endpoint, and keep the frontier API for the cases the small model abstains on.

Questions

How much can a small model save compared with an API?
It depends on volume and hardware; the saving is the per-token API price replaced by owned hardware at high utilisation. We report the measured cost per request on your chosen hardware before you commit.
Will a small model be as accurate as the frontier model?
On a narrow task with enough examples, often yes, and we measure it: Phonon-2.5, 16 times smaller than its source model, is more accurate on average across 25 languages on FLEURS. Where it falls short, the model abstains and the request goes to the larger model.
Can it run on-premises or on devices?
Yes. Phonon runs on CPUs, Apple silicon, NVIDIA GPUs and Android; Detta runs speech recognition on the Mac's Neural Engine.
Do you keep our data?
Deployments run in your environment. Terms for data handling are agreed per engagement; write to contact@fermionresearch.com.

Related

Sources and method

  • Phonon-2.5 numbers: the Phonon-2.5 release post (2026-10-22); throughput: Phonon-2 model page, published figures.
  • No price or contract term is stated on this page; none is confirmed.

Canonical: https://www.fermionresearch.com/deploy/replace-api-models-with-small-models/