The short answer
On-device speech recognition means the model that turns speech into text runs on the hardware where the audio is: a laptop, phone, desktop or a server you control. The audio does not have to travel to a remote service to be transcribed, and after the model is downloaded it works without a network connection.
It became practical as models got small enough to fit in ordinary memory. Phonon-2.5 is a 156 MB download for 25 languages and runs on a plain laptop CPU with no GPU.
On-device, self-hosted server or cloud API
The three ways to run speech recognition differ mainly in where the audio goes and who pays for the compute.
| On-device | Self-hosted server | Cloud API | |
|---|---|---|---|
| Where the audio goes | Stays on the device | To a machine you run | To the provider |
| Network needed to transcribe | No, after the model download | Only your own network | Yes |
| Cost model | The device you already have | Your GPU or CPU hours | Price per audio hour |
| Model size limit | What fits in the device's memory | What fits on your server | None for you |
| Updates | New model file shipped to devices | You redeploy | Provider changes it |
| Typical uses | Dictation, offline apps, field devices | Batch transcription, internal tools | Low volume, many languages, no ops |
What makes it fast enough
Speech recognition on a device runs on whatever compute is there: the CPU, the GPU, or a neural processing unit such as Apple's Neural Engine. The same model can differ several times in speed between them. Phonon-2's published figures on one M5 MacBook Air show the spread: 40 times realtime on its CPU, 174 times realtime on its GPU through MLX, and 606 times realtime on the Neural Engine through Core ML.
Phonon-2.5 runs on Apple silicon through MLX, on plain CPUs on Linux (x86-64 and arm64) and x86-64 Windows, on NVIDIA GPUs, and on Android through an arm64 package. A Core ML package for the Neural Engine is public for Phonon-2.
pip install fermion-research
fermion transcribe phonon-2.5 file.wavTrade-offs and what on-device does not promise
- Coverage: a small model covers fewer languages than the largest cloud models. Phonon-2.5 covers 25 European languages; if you need Arabic, Hindi or Japanese, it is not the right model.
- Accuracy: small does not have to mean worse, but the lowest English error rates on the Open ASR Leaderboard belong to larger models and closed APIs. In the 2026-09-25 snapshot the lowest seven-set average was 3.68 % (zoom/scribe_v2_pro, a closed API); Phonon-2.5 English scores 4.82 % on the same sets.
- Privacy: running the model locally means the transcription step needs no upload. Whether audio or text leaves the device is still decided by the app around the model, so check its sharing settings.
- Hardware spread: on phones and older laptops the same model runs slower, so measure on the slowest device you support.
Questions
- What is on-device speech recognition?
- Speech-to-text where the model runs on the device that recorded the audio, such as a laptop or phone, instead of on a remote service. Once the model is downloaded, transcription needs no network.
- Does on-device speech recognition need the internet?
- Only to download the model. After that the transcription itself runs offline, though the app around it may still use the network for other things.
- Is on-device speech recognition less accurate than cloud APIs?
- Not necessarily. Phonon-2.5 averages 11.71 % word error on FLEURS across its 25 languages, lower than the 2,508 MB Parakeet TDT 0.6B v3 at 13.26 % on the same scoring, but some larger models and closed APIs score lower on English.
- Does on-device mean my audio is private?
- It means the transcription step does not need to upload audio. Whether recordings or text are sent anywhere depends on the app, so read its settings and policy.
- What hardware do I need?
- A recent laptop CPU is enough for Phonon-2.5. Apple silicon Macs can also use the GPU through MLX, and NVIDIA GPUs and Android arm64 devices are supported.
Related
Sources and method
- Phonon numbers: Phonon-2.5 release post (/research/phonon-2-5/), 2026-10-22; Fermion Research measurement, full test sets, one scoring code for every model in a table.
- Speed figures: Phonon-2's published figures (/models/phonon-2/ and /research/phonon-2/), M5 MacBook Air, AMD Zen 5, Google Axion and H100 hosts as stated.
- Lowest seven-set average: Open ASR Leaderboard, seven English sets, snapshot 2026-09-25.
- Phonon command line and server behaviour: Fermion Research docs (/docs/cli-reference/, /docs/server-reference/), read 2026-11-13.
Canonical: https://www.fermionresearch.com/glossary/on-device-speech-recognition/