The short answer
Fermion Research scores speech models on full public test sets, with one scoring code for every model in a table. English results use the Open ASR Leaderboard's own scoring code and English text normaliser; FLEURS results use one code for all models, with diacritics kept. Rows we measured are marked apart from rows taken from a published leaderboard.
Averages are plain means of the per-set or per-language word error rates, each set weighted equally, and every table says which sets its average covers.
The rules at a glance
Each rule exists so that a row in one of our tables can be compared with every other row in the same table.
| Rule | What we do | Why |
|---|---|---|
| Test sets | The full test split of each public set, never a hand-picked subset | Samples are noisier and easy to cherry-pick |
| Scoring code | One code for every model in a table | Different scorers give different numbers for the same transcripts |
| English normaliser | The Open ASR Leaderboard's code and its English text normaliser (the one published with Whisper) | Our English numbers line up with the leaderboard's |
| Multilingual normaliser | FLEURS scored with one code for all models, diacritics kept | A missing accent is a real error in Czech, French or Greek |
| Decoding | Greedy decoding, as the released command line does | The number describes the model as you run it |
| Our rows vs published rows | Leaderboard rows are labelled with the snapshot date; our measurements say so | Readers can tell who ran what |
| Averages | Plain mean over the named sets or languages | A seven-set and a nine-set average are different numbers |
English
We run the Open ASR Leaderboard's evaluation code on the full test sets. The classic seven sets are LibriSpeech test-clean and test-other, AMI, Earnings-22 (the leaderboard's chunked set), GigaSpeech, SPGISpeech and VoxPopuli. The leaderboard's current protocol adds URGENT24 and Monsoon en-IN, for nine public sets, and we publish both averages with the sets named.
When a model already has a row on the leaderboard, we quote that row and mark it. Numbers for models we submitted are our own run with the leaderboard's code until the maintainers publish their official run; we label them "submitted, pending the maintainers' run".
Comparisons with other models use the leaderboard's published seven-set averages from a dated snapshot (currently 2026-09-25). We compare our seven-set averages only with seven-set averages.
Multilingual (FLEURS)
For the 25 languages of Phonon-2.5 we score the full FLEURS test set of each language. Every model in the table, ours and the comparison models, is run and scored with the same code and the same text normalisation, which keeps diacritics.
This makes our FLEURS rows comparable with each other, but not with FLEURS figures published elsewhere: a paper that strips diacritics or uses a different reference text will get different numbers for the same model on the same audio. We do not mix such figures into our tables.
Paired comparisons, and what we do not claim
When a comparison uses a sample rather than full test sets, for example a head-to-head on clips we can run both models on, both models transcribe the same clips, and we report a paired bootstrap 95 % confidence interval on the difference. A difference inside the interval is reported as a tie. Such samples are labelled as samples, not as the official protocol.
- No speed figures for Phonon-2.5: none are published. Speed figures on this site are Phonon-2's, and say so.
- No accuracy claims for your audio: read speech (FLEURS) and the leaderboard's sets are not your calls or meetings. Test a sample of your own recordings.
- No estimates of other models' accuracy, speed or price: we quote a source with its date, or leave the number out.
- Where another model scores lower than ours on a set or a language, the table shows it.
Questions
- How does Fermion Research benchmark speech models?
- On full public test sets, with one scoring code for every model in a table: the Open ASR Leaderboard's code and English normaliser for English, and one code with diacritics kept for FLEURS.
- Does Fermion Research use the Open ASR Leaderboard's normaliser?
- Yes, for English: results are scored with the leaderboard's code and its English text normaliser, the one published with Whisper.
- Are diacritics removed when scoring FLEURS?
- No. Our FLEURS scoring keeps diacritics, so a missing or wrong accent counts as an error, for every model in the table.
- Can I compare your FLEURS numbers with numbers from papers?
- Only with care. Papers often use different normalisers or references, so we compare models only inside one table scored with one code.
- How are averages computed?
- As the plain mean of the per-set or per-language word error rates, each set weighted equally. Every table states which sets its average covers.
- Which numbers are yours and which are the leaderboard's?
- Rows we measured say so; rows from the Open ASR Leaderboard are marked and carry their snapshot date. Submitted numbers stay labelled as ours until the maintainers publish their run.
Related
Sources and method
- Seven-set English numbers: Phonon-2.5 release post Table 2, 2026-10-22; Open ASR Leaderboard scoring code on the full test sets (Earnings-22 = the leaderboard's chunked set); † = the Open ASR Leaderboard's published row.
- Nine-set English numbers: Fermion Research measurement, 2026-10-28, Open ASR Leaderboard code, full test sets; Phonon-2 = its leaderboard row (removed from the board 2026-11-20). Official board rows: Open ASR Leaderboard version 1.16.2, read 2026-11-21.
- FLEURS numbers: Fermion Research measurement, published 2026-10-22, full FLEURS test set per language, one scoring code and normalisation for all three models, diacritics kept.
- Other models, seven English sets: Open ASR Leaderboard, seven English sets, snapshot 2026-09-25.
- Open ASR Leaderboard protocol: github.com/huggingface/open_asr_leaderboard README, read 2026-11-13.
Canonical: https://www.fermionresearch.com/speech/benchmark-methodology/