---
title: "How Fermion Research measures speech models"
description: "Full public test sets, one scoring code for every model in a table, the leaderboard's English normaliser, diacritics kept for FLEURS, and our rows marked."
canonical: "https://www.fermionresearch.com/speech/benchmark-methodology/"
source: "Fermion Research"
---

Speech recognition/facts checked 21 November 2026

# How Fermion Research measures speech models

The rules behind every word error rate on this site: which test sets, which scoring code, which normaliser, which rows we measured, and what we do not claim.

[Phonon-2.5 downloads and model card](https://www.fermionresearch.com/models/phonon-2-5/)Open weights on Hugging Face.

## The short answer

Fermion Research scores speech models on full public test sets, with one scoring code for every model in a table. English results use the Open ASR Leaderboard's own scoring code and English text normaliser; FLEURS results use one code for all models, with diacritics kept. Rows we measured are marked apart from rows taken from a published leaderboard.

Averages are plain means of the per-set or per-language word error rates, each set weighted equally, and every table says which sets its average covers.

## The rules at a glance

Each rule exists so that a row in one of our tables can be compared with every other row in the same table.

| Rule | What we do | Why |
| --- | --- | --- |
| Test sets | The full test split of each public set, never a hand-picked subset | Samples are noisier and easy to cherry-pick |
| Scoring code | One code for every model in a table | Different scorers give different numbers for the same transcripts |
| English normaliser | The Open ASR Leaderboard's code and its English text normaliser (the one published with Whisper) | Our English numbers line up with the leaderboard's |
| Multilingual normaliser | FLEURS scored with one code for all models, diacritics kept | A missing accent is a real error in Czech, French or Greek |
| Decoding | Greedy decoding, as the released command line does | The number describes the model as you run it |
| Our rows vs published rows | Leaderboard rows are labelled with the snapshot date; our measurements say so | Readers can tell who ran what |
| Averages | Plain mean over the named sets or languages | A seven-set and a nine-set average are different numbers |

## English

We run the Open ASR Leaderboard's evaluation code on the full test sets. The classic seven sets are LibriSpeech test-clean and test-other, AMI, Earnings-22 (the leaderboard's chunked set), GigaSpeech, SPGISpeech and VoxPopuli. The leaderboard's current protocol adds URGENT24 and Monsoon en-IN, for nine public sets, and we publish both averages with the sets named.

When a model already has a row on the leaderboard, we quote that row and mark it. Numbers for models we submitted are our own run with the leaderboard's code until the maintainers publish their official run; we label them "submitted, pending the maintainers' run".

Comparisons with other models use the leaderboard's published seven-set averages from a dated snapshot (currently 2026-09-25). We compare our seven-set averages only with seven-set averages.

## Multilingual (FLEURS)

For the 25 languages of Phonon-2.5 we score the full FLEURS test set of each language. Every model in the table, ours and the comparison models, is run and scored with the same code and the same text normalisation, which keeps diacritics.

This makes our FLEURS rows comparable with each other, but not with FLEURS figures published elsewhere: a paper that strips diacritics or uses a different reference text will get different numbers for the same model on the same audio. We do not mix such figures into our tables.

## Paired comparisons, and what we do not claim

When a comparison uses a sample rather than full test sets, for example a head-to-head on clips we can run both models on, both models transcribe the same clips, and we report a paired bootstrap 95 % confidence interval on the difference. A difference inside the interval is reported as a tie. Such samples are labelled as samples, not as the official protocol.

- No speed figures for Phonon-2.5: none are published. Speed figures on this site are Phonon-2's, and say so.
- No accuracy claims for your audio: read speech (FLEURS) and the leaderboard's sets are not your calls or meetings. Test a sample of your own recordings.
- No estimates of other models' accuracy, speed or price: we quote a source with its date, or leave the number out.
- Where another model scores lower than ours on a set or a language, the table shows it.

## Questions

## Related

- [FLEURS results in 25 languages](https://www.fermionresearch.com/speech/fleurs-results-25-languages/)
- [English benchmark results](https://www.fermionresearch.com/speech/english-benchmark-results/)
- [Text normalisation for ASR](https://www.fermionresearch.com/glossary/text-normalisation-for-asr/)
- [Word error rate (WER)](https://www.fermionresearch.com/glossary/word-error-rate/)
- [The Open ASR Leaderboard](https://www.fermionresearch.com/glossary/open-asr-leaderboard/)

## Sources and method

- Seven-set English numbers: Phonon-2.5 release post Table 2, 2026-10-22; Open ASR Leaderboard scoring code on the full test sets (Earnings-22 = the leaderboard's chunked set); † = the Open ASR Leaderboard's published row.
- Nine-set English numbers: Fermion Research measurement, 2026-10-28, Open ASR Leaderboard code, full test sets; Phonon-2 = its leaderboard row (removed from the board 2026-11-20). Official board rows: Open ASR Leaderboard version 1.16.2, read 2026-11-21.
- FLEURS numbers: Fermion Research measurement, published 2026-10-22, full FLEURS test set per language, one scoring code and normalisation for all three models, diacritics kept.
- Other models, seven English sets: Open ASR Leaderboard, seven English sets, snapshot 2026-09-25.
- Open ASR Leaderboard protocol: github.com/huggingface/open_asr_leaderboard README, read 2026-11-13.

Canonical: https://www.fermionresearch.com/speech/benchmark-methodology/

Source: [https://www.fermionresearch.com/speech/benchmark-methodology/](https://www.fermionresearch.com/speech/benchmark-methodology/)
