---
title: "Replace expensive API model calls with a small model built for the task"
description: "Most production LLM calls do one narrow job. A small model trained for that job runs on your own hardware or on the device, answers in milliseconds, and cuts the per-call cost. How Fermion builds them."
canonical: "https://www.fermionresearch.com/deploy/replace-api-models-with-small-models/"
source: "Fermion Research"
---

Deploy/facts checked 13 November 2026

# Replace expensive API model calls with a small model built for the task

A frontier model is the right tool for the open-ended ten percent of requests. The other ninety percent are classification, extraction, transcription and routing, and a model a hundredth the size can do them on hardware you already own.

[Talk to us](https://www.fermionresearch.com/deploy/)contact@fermionresearch.com

## What usually happens

A team ships a feature on a frontier API because it is the fastest way to get it working. The feature succeeds, traffic grows, and the API line becomes the largest cost in the product. The calls are the same narrow job repeated millions of times: label a ticket, pull five fields from a document, transcribe a call, decide which flow a request belongs to.

## What a task-specific small model changes

- Size: a model of 0.1 to 9 billion parameters, or a 156 MB speech model, in place of a hosted model you cannot run yourself.
- Where it runs: your own cloud, a CPU box, or the user's device. Data stays inside your environment.
- Latency: milliseconds per decision instead of seconds per generation.
- Cost: hardware you own at high utilisation, instead of a per-token price.

## A public example: Phonon-2.5

Phonon-2.5 is our open speech recognition model: 25 languages from a 156 MB download. On the FLEURS test sets it averages 11.71 % word error, against 13.26 % for Parakeet TDT 0.6B v3, a 2.5 GB model covering the same languages: a sixteenth of the size and more accurate on average. Its published throughput (Phonon-2's figures, shared architecture) is 143 times realtime on eight AMD Zen 5 cores and 6,680 times realtime batched on an H100: an eight-core CPU box clears more than a hundred hours of audio per hour.

The same method, train a small model for exactly the task and measure it on public test sets, is what we apply to a customer's workload.

## How an engagement runs

- Week one: we take a sample of your real requests and the API outputs you already have, and agree the metric and the target.
- We train and compress a model for the task and report accuracy against the API on a held-out set, with the latency and cost per request on the hardware you choose.
- You deploy it in your cloud, on-premises or on devices, with an OpenAI-compatible endpoint, and keep the frontier API for the cases the small model abstains on.

## Questions

## Related

- [Deploy with Fermion](https://www.fermionresearch.com/deploy/)
- [Phonon-2.5](https://www.fermionresearch.com/models/phonon-2-5/)

## Sources and method

- Phonon-2.5 numbers: the Phonon-2.5 release post (2026-10-22); throughput: Phonon-2 model page, published figures.
- No price or contract term is stated on this page; none is confirmed.

Canonical: https://www.fermionresearch.com/deploy/replace-api-models-with-small-models/

Source: [https://www.fermionresearch.com/deploy/replace-api-models-with-small-models/](https://www.fermionresearch.com/deploy/replace-api-models-with-small-models/)
