Fermion ResearchFermion Research

Speech recognition/facts checked 21 November 2026

How Fermion Research measures speech models

The rules behind every word error rate on this site: which test sets, which scoring code, which normaliser, which rows we measured, and what we do not claim.

Phonon-2.5 downloads and model cardOpen weights on Hugging Face.

The short answer

Fermion Research scores speech models on full public test sets, with one scoring code for every model in a table. English results use the Open ASR Leaderboard's own scoring code and English text normaliser; FLEURS results use one code for all models, with diacritics kept. Rows we measured are marked apart from rows taken from a published leaderboard.

Averages are plain means of the per-set or per-language word error rates, each set weighted equally, and every table says which sets its average covers.

The rules at a glance

Each rule exists so that a row in one of our tables can be compared with every other row in the same table.

RuleWhat we doWhy
Test setsThe full test split of each public set, never a hand-picked subsetSamples are noisier and easy to cherry-pick
Scoring codeOne code for every model in a tableDifferent scorers give different numbers for the same transcripts
English normaliserThe Open ASR Leaderboard's code and its English text normaliser (the one published with Whisper)Our English numbers line up with the leaderboard's
Multilingual normaliserFLEURS scored with one code for all models, diacritics keptA missing accent is a real error in Czech, French or Greek
DecodingGreedy decoding, as the released command line doesThe number describes the model as you run it
Our rows vs published rowsLeaderboard rows are labelled with the snapshot date; our measurements say soReaders can tell who ran what
AveragesPlain mean over the named sets or languagesA seven-set and a nine-set average are different numbers

English

We run the Open ASR Leaderboard's evaluation code on the full test sets. The classic seven sets are LibriSpeech test-clean and test-other, AMI, Earnings-22 (the leaderboard's chunked set), GigaSpeech, SPGISpeech and VoxPopuli. The leaderboard's current protocol adds URGENT24 and Monsoon en-IN, for nine public sets, and we publish both averages with the sets named.

When a model already has a row on the leaderboard, we quote that row and mark it. Numbers for models we submitted are our own run with the leaderboard's code until the maintainers publish their official run; we label them "submitted, pending the maintainers' run".

Comparisons with other models use the leaderboard's published seven-set averages from a dated snapshot (currently 2026-09-25). We compare our seven-set averages only with seven-set averages.

Multilingual (FLEURS)

For the 25 languages of Phonon-2.5 we score the full FLEURS test set of each language. Every model in the table, ours and the comparison models, is run and scored with the same code and the same text normalisation, which keeps diacritics.

This makes our FLEURS rows comparable with each other, but not with FLEURS figures published elsewhere: a paper that strips diacritics or uses a different reference text will get different numbers for the same model on the same audio. We do not mix such figures into our tables.

Paired comparisons, and what we do not claim

When a comparison uses a sample rather than full test sets, for example a head-to-head on clips we can run both models on, both models transcribe the same clips, and we report a paired bootstrap 95 % confidence interval on the difference. A difference inside the interval is reported as a tie. Such samples are labelled as samples, not as the official protocol.

  • No speed figures for Phonon-2.5: none are published. Speed figures on this site are Phonon-2's, and say so.
  • No accuracy claims for your audio: read speech (FLEURS) and the leaderboard's sets are not your calls or meetings. Test a sample of your own recordings.
  • No estimates of other models' accuracy, speed or price: we quote a source with its date, or leave the number out.
  • Where another model scores lower than ours on a set or a language, the table shows it.

Questions

How does Fermion Research benchmark speech models?
On full public test sets, with one scoring code for every model in a table: the Open ASR Leaderboard's code and English normaliser for English, and one code with diacritics kept for FLEURS.
Does Fermion Research use the Open ASR Leaderboard's normaliser?
Yes, for English: results are scored with the leaderboard's code and its English text normaliser, the one published with Whisper.
Are diacritics removed when scoring FLEURS?
No. Our FLEURS scoring keeps diacritics, so a missing or wrong accent counts as an error, for every model in the table.
Can I compare your FLEURS numbers with numbers from papers?
Only with care. Papers often use different normalisers or references, so we compare models only inside one table scored with one code.
How are averages computed?
As the plain mean of the per-set or per-language word error rates, each set weighted equally. Every table states which sets its average covers.
Which numbers are yours and which are the leaderboard's?
Rows we measured say so; rows from the Open ASR Leaderboard are marked and carry their snapshot date. Submitted numbers stay labelled as ours until the maintainers publish their run.

Related

Sources and method

  • Seven-set English numbers: Phonon-2.5 release post Table 2, 2026-10-22; Open ASR Leaderboard scoring code on the full test sets (Earnings-22 = the leaderboard's chunked set); † = the Open ASR Leaderboard's published row.
  • Nine-set English numbers: Fermion Research measurement, 2026-10-28, Open ASR Leaderboard code, full test sets; Phonon-2 = its leaderboard row (removed from the board 2026-11-20). Official board rows: Open ASR Leaderboard version 1.16.2, read 2026-11-21.
  • FLEURS numbers: Fermion Research measurement, published 2026-10-22, full FLEURS test set per language, one scoring code and normalisation for all three models, diacritics kept.
  • Other models, seven English sets: Open ASR Leaderboard, seven English sets, snapshot 2026-09-25.
  • Open ASR Leaderboard protocol: github.com/huggingface/open_asr_leaderboard README, read 2026-11-13.

Canonical: https://www.fermionresearch.com/speech/benchmark-methodology/