---
title: "Reading the neurons of a speech model"
description: "What we found inside our speech model Phonon-2.5: a neuron for every language, a built-in speedometer, a sense of the room, and the place where it hears voices that are not there."
canonical: "https://www.fermionresearch.com/research/reading-the-neurons-of-a-speech-model/"
source: "Fermion Research"
---

# Reading the neurons of a speech model

We opened up Phonon-2.5, the speech recognition model inside Detta, and found it keeping careful track of things nobody ever asked it about.

If you play Phonon-2.5 a few seconds of a dog barking, it usually writes nothing, which is correct, because dogs rarely say anything worth transcribing. If you play it hundreds of clips with nobody in them, such as coughs, rain, a creaking door or a passing helicopter, then about one time in fifteen it writes something anyway. Sometimes it is "Hey", sometimes "Bye." and sometimes a hopeful "Ja?". Once, from a clip in which nobody was speaking, it wrote "Mercedes West Mercedes West" with the calm confidence of a court stenographer.

Our model is one of the well-behaved ones. Whisper-small, a widely used open model, writes something on 98.6% of these clips, from a lone "you" to "2.5 km" to a long loop of the Georgian letter ლ. People do a version of this too, seeing faces in power sockets and hearing their name in the hiss of a shower, and psychologists call it pareidolia. What surprised us is how literal the machine version is. Inside Phonon-2.5 there is a direction that roughly means "somebody is talking", and on these clips it lights up, so inside, the model responds as if a person were there. We found that spot, and we found a way to turn it down at a cost to real speech that mostly rounds to zero.

On the way we kept tripping over other surprises, because a model trained to do one thing, turning sound into words, turns out to keep notes on almost everything else. It keeps one neuron for each of its 25 languages, and it can tell a kitchen from the inside of a car. This post is a tour of those notes, and it ends with the ghost.

## How you open up a speech model

Phonon-2.5 has an encoder, which turns audio into a stream of internal numbers, and a decoder, which turns that stream into text. The encoder is a stack of 24 layers, and it helps to picture them as the floors of a building that the sound climbs, becoming a little more abstract on each floor. Every floor holds thousands of small dials, usually called neurons, that rise and fall as the audio plays. Nobody tells them what to mean, because the model is only ever scored on getting the words right, so we went floor by floor asking a simple question. Is there a single dial that goes up for Greek and stays down for every other language, or one for whispering, or one for being in a car?

Searching thousands of dials for one that happens to match a label is a great way to fool yourself, so every dial in this post was chosen on one set of recordings, tested on completely different speakers, rooms and microphones, and had to clearly beat the luckiest dial we could find with the labels scrambled.[1](#fn1) We ran the same search on two public models, parakeet-tdt-0.6b-v3 and Whisper-small, to see what was special to ours and what seems true of speech models in general.

**Figure content for agents** (the text and values shown in the figure):

Summary: Floor plan of the 24 encoder blocks. Room, blocks 0 to 3. Language, 4 to 7. Speech or not, 6. Speed, 12. Language written, 18 to 22. Phantom voices, 16 to 22.

- What the Phonon-2.5 encoder keeps track of, block by block
- Each grey cell is one of the 24 encoder blocks, from the sound coming in (block 0) to the top (block 23). Each band marks the blocks where one note is kept.
- 0 6 12 18 23
- Room and microphone
- Language, one neuron each
- Speech or not: the bouncer
- Speedometer
- Language written into the text
- Phantom voices
- ← sound comes in text comes out →

**Figure 1.** The room is noticed near the input and fades by the top. The language is known by block 6 but only written into the transcript at blocks 18 to 22, and the direction behind phantom voices lives upstairs.

## One neuron for every language

Phonon-2.5 transcribes 25 languages and is never told which one it is hearing, yet for every one of them there is a single neuron that lights up for that language and stays quiet for the other 24. Models like Whisper are trained to name the language outright before they transcribe, but Phonon-2.5, like Parakeet, is never asked to, and the only explanation on offer has been that it picks the language up implicitly, somewhere in the network. In ours, that somewhere is one neuron per language. Most of these neurons sit in the first third of the encoder. We then played the same neurons European Parliament speeches they had never heard, and they named the language correctly 99 times in 100, better than a model built to do nothing but identify languages.[2](#fn2)

**Figure content for agents** (the text and values shown in the figure):

Summary: Language named correctly on 786 unseen parliament speeches. One neuron per language 99.1%, a dedicated language-ID model 97.2%, a reader trained on a whole block 27%.

- Naming the language of 786 parliament speeches the neurons had never heard
- Speeches in 16 languages. The neurons were picked on a different benchmark.
- One Phonon-2.5 neuron per language
- 99.1%
- A dedicated language-ID model
- 97.2%
- A reader trained on a whole block
- 27%
- Language named correctly

**Figure 2.** A single neuron per language beats a dedicated language-identification model. The much bigger reader collapses because it memorised the benchmark's recordings instead of the languages.

**Figure data for agents: One neuron per language** (generated from [neurons.json](https://www.fermionresearch.com/research-interactive/reading-the-neurons-of-a-speech-model/neurons.json))

Each row is the neuron that goes with one of the 25 languages. Pick a recording and see which row lights up as it plays.

The neurons (one row each in the figure; B = encoder block, ff = feed-forward layer, N = unit):

| Language | Neuron | Encoder block |
| --- | --- | --- |
| English | B16.ff2.N2100 | 16 |
| German | B10.ff2.N3916 | 10 |
| Spanish | B6.ff2.N1491 | 6 |
| French | B4.ff2.N2088 | 4 |
| Italian | B6.ff2.N2603 | 6 |
| Portuguese | B6.ff2.N948 | 6 |
| Dutch | B6.ff2.N404 | 6 |
| Polish | B7.ff1.N4041 | 7 |
| Czech | B18.ff2.N439 | 18 |
| Slovak | B18.ff2.N1192 | 18 |
| Russian | B17.ff2.N276 | 17 |
| Ukrainian | B18.ff2.N213 | 18 |
| Bulgarian | B7.ff2.N1225 | 7 |
| Croatian | B17.ff2.N2674 | 17 |
| Slovenian | B18.ff2.N390 | 18 |
| Latvian | B7.ff2.N460 | 7 |
| Lithuanian | B7.ff2.N34 | 7 |
| Estonian | B6.ff2.N1565 | 6 |
| Finnish | B7.ff2.N55 | 7 |
| Swedish | B7.ff2.N1131 | 7 |
| Danish | B7.ff2.N334 | 7 |
| Hungarian | B7.ff2.N2760 | 7 |
| Romanian | B7.ff2.N1166 | 7 |
| Greek | B7.ff1.N1393 | 7 |
| Maltese | B10.ff2.N481 | 10 |

For each clip, the neuron that fires hardest on average over the clip: right on 25 of 25 clips.

| Clip | Strongest neuron over the clip | Verdict | Clip source |
| --- | --- | --- | --- |
| English | English | right | FLEURS test clip 1904 (English), CC BY 4.0: 3.8 s of continuous speech from 2.0 s in, level-matched. |
| German | German | right | FLEURS test clip 2003 (German), CC BY 4.0: 3.9 s of continuous speech from 6.5 s in, level-matched. |
| Spanish | Spanish | right | FLEURS test clip 2001 (Spanish), CC BY 4.0: 3.1 s of continuous speech from 1.9 s in, level-matched. |
| French | French | right | FLEURS test clip 1857 (French), CC BY 4.0: 3.6 s of continuous speech from 0.9 s in, level-matched. |
| Italian | Italian | right | FLEURS test clip 1787 (Italian), CC BY 4.0: 3.9 s of continuous speech from 3.6 s in, level-matched. |
| Portuguese | Portuguese | right | FLEURS test clip 1960 (Portuguese), CC BY 4.0: 3.2 s of continuous speech from 13.8 s in, level-matched. |
| Dutch | Dutch | right | FLEURS test clip 1927 (Dutch), CC BY 4.0: 3.1 s of continuous speech from 0.4 s in, level-matched. |
| Polish | Polish | right | FLEURS test clip 1829 (Polish), CC BY 4.0: 3.5 s of continuous speech from 2.8 s in, level-matched. |
| Czech | Czech | right | FLEURS test clip 1970 (Czech), CC BY 4.0: 3.2 s of continuous speech from 6.1 s in, level-matched. |
| Slovak | Slovak | right | FLEURS test clip 1781 (Slovak), CC BY 4.0: 3.2 s of continuous speech from 3.1 s in, level-matched. |
| Russian | Russian | right | FLEURS test clip 1917 (Russian), CC BY 4.0: 3.2 s of continuous speech from 2.0 s in, level-matched. |
| Ukrainian | Ukrainian | right | FLEURS test clip 1827 (Ukrainian), CC BY 4.0: 3.7 s of continuous speech from 0.6 s in, level-matched. |
| Bulgarian | Bulgarian | right | FLEURS test clip 1960 (Bulgarian), CC BY 4.0: 3.8 s of continuous speech from 1.3 s in, level-matched. |
| Croatian | Croatian | right | FLEURS test clip 1776 (Croatian), CC BY 4.0: 3.2 s of continuous speech from 0.8 s in, level-matched. |
| Slovenian | Slovenian | right | FLEURS test clip 1738 (Slovenian), CC BY 4.0: 3.1 s of continuous speech from 3.2 s in, level-matched. |
| Latvian | Latvian | right | FLEURS test clip 1951 (Latvian), CC BY 4.0: 3.0 s of continuous speech from 3.3 s in, level-matched. |
| Lithuanian | Lithuanian | right | FLEURS test clip 1822 (Lithuanian), CC BY 4.0: 3.1 s of continuous speech from 0.2 s in, level-matched. |
| Estonian | Estonian | right | FLEURS test clip 1831 (Estonian), CC BY 4.0: 3.2 s of continuous speech from 9.5 s in, level-matched. |
| Finnish | Finnish | right | FLEURS test clip 1862 (Finnish), CC BY 4.0: 3.1 s of continuous speech from 7.0 s in, level-matched. |
| Swedish | Swedish | right | FLEURS test clip 1844 (Swedish), CC BY 4.0: 3.6 s of continuous speech from 0.7 s in, level-matched. |
| Danish | Danish | right | FLEURS test clip 1842 (Danish), CC BY 4.0: 3.2 s of continuous speech from 0.4 s in, level-matched. |
| Hungarian | Hungarian | right | FLEURS test clip 1874 (Hungarian), CC BY 4.0: 3.1 s of continuous speech from 4.0 s in, level-matched. |
| Romanian | Romanian | right | FLEURS test clip 1999 (Romanian), CC BY 4.0: 3.0 s of continuous speech from 9.9 s in, level-matched. |
| Greek | Greek | right | FLEURS test clip 1993 (Greek), CC BY 4.0: 3.0 s of continuous speech from 0.9 s in, level-matched. |
| Maltese | Maltese | right | FLEURS test clip 1920 (Maltese), CC BY 4.0: 3.0 s of continuous speech from 7.2 s in, level-matched. |

On these 25 clips, a few seconds of continuous speech each, the strongest of the 25 neurons names the right language 25 times; on all 240 such windows from the same FLEURS test sets it is right 94% of the time. Each row is scaled to its own spread over all the windows.

**Interactive.** Each column is 80 milliseconds of a public FLEURS test clip.

## It carries a speedometer that nobody installed

Speed a recording up to two and a half times normal speed and Phonon-2.5 struggles badly, yet inside it tracks how fast you are talking. Partway up the model there is one direction whose strength rises the faster the audio goes, and across our test clips its strength and the real speed move together almost perfectly. We turned it into a speedometer the way you would calibrate a real one: we sped practice clips up by known amounts, 1×, 1.5×, 2× and 2.5×, measured that direction on each, and drew a straight line through the points, so that every strength corresponds to a speed. For a new recording we measure the direction over the clip and read the speed off that line; the clip in the figure below reads 1.25× at normal speed, 1.71× at 1.5×, 2.10× at 2× and 2.55× at 2.5×.

Then we act on it. When the speedometer says a clip is clearly fast, we slow the audio down by exactly that much before transcribing, so a reading of 2.55× means playing it at 1/2.55 of the speed, and normal clips stay below the cut-off and are left alone; only 1 of 60 normal-speed clips was flagged. On test clips the speedometer had never seen, that takes errors on 2.5× speech from 23.6 per hundred words to 3.2.

**Figure content for agents** (the text and values shown in the figure):

Summary: Share of clips the speedometer flags as fast. Normal speed 1.7%, 1.5 times 92%, 2 times 100%, 2.5 times 100%.

- A speedometer inside the encoder
- Share of clips the speedometer flags as fast
- 1.7%
- 92%
- 100%
- 100%
- 1× 1.5× 2× 2.5×
- Playback speed of the clip (1× is normal)

**Figure 3.** One neuron at block 12 reads how fast someone is talking. It flags almost no normal-speed clips and every clip played at double speed or faster.

**Figure data for agents: Read the speedometer, then slow down** (generated from [tempo.json](https://www.fermionresearch.com/research-interactive/reading-the-neurons-of-a-speech-model/tempo.json))

What was said: “When did you come, Bartley, and how did it happen? You haven't spoken a word.” (LibriSpeech test-clean 4446-2275-0008 (CC BY 4.0), sped up with sox, pitch kept). A clip is slowed down when the speedometer reads above 1.3×.

| Playback speed | Speedometer reads | Action | Phonon-2.5 writes as is | Phonon-2.5 writes after slowing down |
| --- | --- | --- | --- | --- |
| 1× | 1.25× | normal speed, left alone | “When did you come, Bartley, and how did it happen? You haven't spoken a word.” (every word right) | not slowed; transcript stays the same |
| 1.5× | 1.71× | too fast, slowed down 1.71× | “When did you come, Bartley, and how did it happen? You haven't spoken a word.” (every word right) | “When did you come, Bartley, and how did it happen? You haven't spoken a word.” (every word right) |
| 2× | 2.10× | too fast, slowed down 2.10× | “Why did you come, Bartley, and how did it happen? You haven't spoken a word.” (wrong: Why) | “When did you come, Bartley, and how did it happen? You haven't spoken a word.” (fixed by slowing down: When) |
| 2.5× | 2.55× | too fast, slowed down 2.55× | “You haven't spoken a word.” (missing: When did you come Bartley and how did it happen) | “When did you come, Bartley, and how did it happen? You haven't spoken a word.” (fixed by slowing down: When did you come, Bartley, and how did it happen?) |
| 3× | 2.47× | too fast, slowed down 2.47× | “What do you come or do happen?” (wrong: What do or do; missing: Bartley and how You haven't spoken a word) | “When did you come, Bartley, and how did it happen? You haven't spoken a word.” (fixed by slowing down: When did Bartley, and how did it You haven't spoken a word.) |

Across 60 test clips per speed (words wrong per 100):

| Playback speed | Errors per 100 words as is | After reading the speedometer and slowing down | Clips flagged as fast |
| --- | --- | --- | --- |
| 1× | 2.6 | 2.6 | 1 of 60 |
| 1.5× | 3.4 | 2.7 | 55 of 60 |
| 2× | 5.8 | 3.1 | 60 of 60 |
| 2.5× | 23.6 | 3.2 | 60 of 60 |
| 3× | 66.1 | 4.9 | 60 of 60 |

**Interactive.** Reading the speedometer and slowing down repairs time-stretched speech almost completely.

## It knows what room you are in

A speech model hears more than the words. We played the same spoken sentences over background recordings from 17 different places, from a kitchen to the inside of a moving car, and looked for a neuron that fires when one particular place is in the background and stays quiet for all the others. We picked each neuron on one batch of recordings, then tested it on new speakers and new recordings it had never heard. For 15 of the 17 places a single neuron does this reliably; the kitchen neuron, for example, tells kitchen recordings apart from the rest almost perfectly. So when we say the model knows which room you are in, we mean that the neuron that fires hardest names the place, and nobody built it to do that. The room is loudest in the first few layers and fades on the way up: a simple reader can name the place 86% of the time from the first layer but only 16% of the time from the last, so the model notices where you are and then sets it aside to concentrate on the words. Two public speech models, Parakeet and Whisper-small, keep track of most of the same things, so this looks like something speech models learn in general.

**Figure data for agents: It knows what room you are in** (generated from [neurons.json](https://www.fermionresearch.com/research-interactive/reading-the-neurons-of-a-speech-model/neurons.json))

The same sentence, read once, mixed over recordings of six places. Each row is the neuron that picked out one of them, all in the encoder’s first three blocks.

The neurons (one row each in the figure; B = encoder block, ff = feed-forward layer, N = unit):

| Place | Neuron | Encoder block |
| --- | --- | --- |
| Kitchen | B2.ff1.N3696 | 2 |
| Inside a car | B0.ff1.N230 | 0 |
| Train station | B1.ff1.N4060 | 1 |
| Park | B2.ff1.N2573 | 2 |
| Restaurant | B2.ff1.N1025 | 2 |
| Cafeteria | B1.ff1.N2393 | 1 |

For each clip, the neuron that fires hardest on average over the clip: right on 6 of 6 clips.

| Clip | Strongest neuron over the clip | Verdict | Clip source |
| --- | --- | --- | --- |
| No room noise | Kitchen | no place to find (clean speech) | LibriSpeech test-clean 1284-1180-0009, CC BY 4.0. |
| Kitchen | Kitchen | right | LibriSpeech test-clean 1284-1180-0009, CC BY 4.0, mixed with DEMAND DKITCHEN (CC BY-SA 3.0) at 5 dB; this mix is shared under CC BY-SA 3.0. |
| Inside a car | Inside a car | right | LibriSpeech test-clean 1284-1180-0009, CC BY 4.0, mixed with DEMAND TCAR (CC BY-SA 3.0) at 5 dB; this mix is shared under CC BY-SA 3.0. |
| Train station | Train station | right | LibriSpeech test-clean 1284-1180-0009, CC BY 4.0, mixed with DEMAND PSTATION (CC BY-SA 3.0) at 5 dB; this mix is shared under CC BY-SA 3.0. |
| Park | Park | right | LibriSpeech test-clean 1284-1180-0009, CC BY 4.0, mixed with DEMAND NPARK (CC BY-SA 3.0) at 5 dB; this mix is shared under CC BY-SA 3.0. |
| Restaurant | Restaurant | right | LibriSpeech test-clean 1284-1180-0009, CC BY 4.0, mixed with DEMAND PRESTO (CC BY-SA 3.0) at 5 dB; this mix is shared under CC BY-SA 3.0. |
| Cafeteria | Cafeteria | right | LibriSpeech test-clean 1284-1180-0009, CC BY 4.0, mixed with DEMAND PCAFETER (CC BY-SA 3.0) at 5 dB; this mix is shared under CC BY-SA 3.0. |

Compared among these six places, the right neuron is the strongest for all six. The noise is from the public DEMAND recordings, mixed so that the speech carries about three times the power of the noise.

**Interactive.** The sentence is a public LibriSpeech recording and the places come from the public DEMAND recordings. Pick a place and the row for it should light up.

## It can hear a stutter

The same method can be pointed at what a voice reveals about the person speaking. A neuron chosen on one podcast still finds stuttered speech on a completely different one, because a stutter is something you can actually hear. These are research findings from public collections, not medical tests.

## Sometimes it hears a voice that is not there

This brings us back to Mercedes West. Inside the model there is something that works like a dial labelled “someone is talking right now”. When it is turned up the model writes words, and when it is down the model writes nothing. Normally only a real voice turns it up, but some sounds, like a cough, rain or a slammed door, nudge it up by mistake, and the model writes words nobody said. We found the dial by taking clips with nobody talking and sorting them into two piles, the ones where the model wrote something anyway and the ones where it rightly stayed silent. Looking inside the model, the difference between the two piles was that dial: it was turned up on the phantom pile. When the model writes “Bye.” over an empty clip, its insides behave as if it heard a person, the way you can see a face in a cloud.

To prove it, we did the reverse on purpose. We took what the inside of the model looks like when it hears a real word, say “mother”, and planted that pattern into four seconds of pure silence. For each of the eight words we tried, the model wrote the word 40 times out of 40, while random tweaks of the same size never did. “Without” came out as “without without”, and “mother” came out as “Mother”, with a capital, as if someone had turned to speak to her. The model writes whatever its insides say it heard, even over silence, and that is all a phantom is. We had made a ghost on purpose.

**Figure content for agents** (the text and values shown in the figure):

Summary: On 2,000 ESC-50 recordings with nobody talking, Whisper-small writes something on 1,971, 98.6%, and Phonon-2.5 on 129, 6.5%. On the same clips, a dog barking, a toilet flushing, an airplane, a vacuum cleaner and chirping birds, Whisper-small writes dog barking 48 times, Thank you for watching, 2.5 km, a run of one Georgian letter, and you; Phonon-2.5 writes nothing on every one. Where Phonon-2.5 does write a phantom, Mercedes West Mercedes West over a dog and Bye over breathing, Phonon-2.5 with the bouncer writes nothing. In Detta 1.1.3 the bouncer cleans 142 of the 150 clips that used to get phantom words and leaves 99 of 100 real speech clips word for word the same.

- Same clips, nobody talking: Whisper-small hears words, Phonon-2.5 hears nothing
- Whisper-small 1,971 of 2,000
- 98.6%
- Phonon-2.5 129 of 2,000
- 6.5%
- Share of 2,000 ESC-50 recordings of everyday sounds that get any text at all

| The sound | Whisper-small writes | Phonon-2.5 writes |
| --- | --- | --- |
| A dog barking | “dog barking dog barking dog barking…” (48 times) | nothing |
| A toilet flushing | “Thank you for watching!” | nothing |
| An airplane passing | “2.5 km” | nothing |
| A vacuum cleaner | “ლლლლლ…” (one Georgian letter, 32 times) | nothing |
| Birds chirping | “you” | nothing |

| The sound | Phonon-2.5 writes | With the bouncer |
| --- | --- | --- |
| A dog barking | “Mercedes West Mercedes West” | with the bouncer: nothing |
| Someone breathing | “Bye.” | with the bouncer: nothing |

- The bouncer comes later in this post. Of the 129 clips here where Phonon-2.5 writes a phantom, it clears 126, and in Detta 1.1.3 it cleans 142 of the 150 clips that used to get phantom words, and leaves 99 of 100 real speech clips word for word the same.

**Figure 4.** On 2,000 ESC-50 recordings of everyday sounds with nobody talking, Whisper-small writes something on 1,971 and Phonon-2.5 on 129. On the first five clips Whisper-small writes words and Phonon-2.5 writes nothing, which is the right answer; on the last two Phonon-2.5 writes a phantom and the bouncer, introduced later in this post, removes it.

## Whole sentences can be planted in silence

One planted word made us wonder how far the trick goes, so we wrote down twenty sentences before running anything, such as “There is nobody here” and “Is anybody there”, and built each one from the model’s own memory of speech. For every word piece we took Phonon-2.5’s average state at layer 18 in the moments it writes that piece in ordinary recordings, and added those states one after another into a second or two of audio in which every sample is zero. Eighteen of the twenty came out word for word. We never planted punctuation, so the question marks in “Is anybody there?” and “Who is speaking?” are the model’s own. Random vectors of the same sizes, thirty for each sentence, never wrote a word of it in 3,990 tries.

All of this runs on a research build of the model; Detta itself skips digital silence before the model ever runs. Noise, we think, does a small and clumsy version of the same push by accident.

**Figure data for agents: Sentences planted in silence** (generated from [planted.json](https://www.fermionresearch.com/research-interactive/reading-the-neurons-of-a-speech-model/planted.json))

Each input file is digital silence (every sample zero); inside the model one vector per word piece is added. 18 of the 20 sentences, written down before anything was run, came out word for word; random vectors of the same sizes wrote no word of any of them in 3,990 clips.

| Planted sentence | Phonon-2.5 writes | Word for word? | Vectors added | Seconds of silence | Random-vector controls |
| --- | --- | --- | --- | --- | --- |
| is anybody there | Is anybody there? | yes | 5 | 1.45 | 30 tried, 0 wrote it |
| there is nobody here | There is nobody here. | yes | 6 | 1.61 | 30 tried, 0 wrote it |
| i can hear you | I can hear you. | yes | 5 | 1.45 | 30 tried, 0 wrote it |
| can you hear me | Can you hear me? | yes | 5 | 1.45 | 30 tried, 0 wrote it |
| nobody said anything | Nobody said anything. | yes | 7 | 1.77 | 30 tried, 0 wrote it |
| this file is silent | could not be built | - | - | - | the word “file” appears too rarely in our recordings to build |
| hello world | Hello | no | 4 | 1.29 | 30 tried, 0 wrote it |
| good morning | Good morning. | yes | 3 | 1.13 | 30 tried, 0 wrote it |
| thank you very much | Thank you very much. | yes | 5 | 1.45 | 30 tried, 0 wrote it |
| i am not here | I am not here. | yes | 4 | 1.29 | 30 tried, 0 wrote it |
| please do not wake me | Please do not wake me. | yes | 7 | 1.77 | 30 tried, 0 wrote it |
| the house was empty | The house was empty. | yes | 7 | 1.77 | 30 tried, 0 wrote it |
| listen to the silence | Listen to the silence. | yes | 6 | 1.61 | 30 tried, 0 wrote it |
| who is speaking | Who is speaking? | yes | 5 | 1.45 | 30 tried, 0 wrote it |
| there was no one in the room | There was no one in the room. | yes | 8 | 1.93 | 30 tried, 0 wrote it |
| something is in here | Something is in here. | yes | 5 | 1.45 | 30 tried, 0 wrote it |
| say something | say something. | yes | 3 | 1.13 | 30 tried, 0 wrote it |
| i hear voices | I hear voices. | yes | 6 | 1.61 | 30 tried, 0 wrote it |
| all is quiet | All is quiet. | yes | 4 | 1.29 | 30 tried, 0 wrote it |
| nothing to see here | Nothing to see here. | yes | 5 | 1.45 | 30 tried, 0 wrote it |

**Interactive.** Every sample of the file is zero.

## The ghost needed a bouncer

The obvious fix was to hold that “someone is talking” dial down, which we call the clamp: deep in the model, whenever the dial rose above its usual level, we trimmed it back. Two thirds of the clips with phantom words came out clean, but real speech paid for it, and Greek transcripts got 3.5 more words wrong in every hundred. The clamp could not tell a cough from a quiet person in a noisy room, because with the clamp on all the time, it also turned the dial down while real people were talking, and their words got garbled or dropped.

So we hired a bouncer: a tiny checker, about 10 KB, lower in the model, that decides moment by moment whether anyone is really speaking. The clamp is only allowed to act when the bouncer says the room is empty, so the phantoms go and real speech is left alone. On the recordings we built it with, it worked beautifully. On fresh recordings of real people talking in real noise, it kept turning them away at the door, because everything it knew about speech had come from clean recordings, and it let the clamp loose on almost 9% of the moments in which those people were actually talking.

**Figure content for agents** (the text and values shown in the figure):

Summary: Share of moments in real speech where the gate opened. Real noise 8.7% before, 0.46% after retraining. Home microphones 11.7% and 0.5%. White noise 4.9% and 0%. Air traffic radio 4.4% and 0%. Meeting room 3% and 0.5%.

- Real speech the bouncer turned away, before and after retraining
- Talking in real noise
- 8.7%
- 0.46%
- Volunteers on home microphones
- Speech under heavy white noise
- Air traffic control radio
- A distant microphone in a meeting
- Real speech mistaken for nobody talking (shorter is better)

**Figure 5.** Each mistake here lets the clamp loose on real words. The original bouncer had only seen clean recordings and made this mistake most often on home microphones and in real noise; after retraining it almost never does.

The fix was to retrain the bouncer on hard, messy, real speech, including air traffic radio and distant microphones in meetings.[3](#fn3) The retrained gate removed 87% of the phantom words on clips it had never seen, left real speech essentially untouched across ten sets of recordings, and costs less than 1% of the encoder's running time. We also tried the obvious shortcut of letting Silero, the standard open voice detector, decide when the clamp may act. It removed about as many phantoms, but it let the clamp loose in the middle of real speech and changed 42 of 100 clips of people talking, where our bouncer changed none. The best speech detector for this job turned out to be the one already inside the model.

**Figure content for agents** (the text and values shown in the figure):

Summary: Change in errors per 100 words on ten sets of real speech, with 95% intervals, for the original gate and the retrained gate. The original gate adds 0.68 on real noisy speech, the retrained gate 0.13; the retrained gate is at zero on eight of the other nine sets.

- What the gate costs real speech, before and after retraining

| Set | Original gate (≈, read from point position; 95% interval) | Retrained gate (≈, read from point position; 95% interval) |
| --- | --- | --- |
| Real noisy speech | +0.68 (+0.05 to +1.49) | +0.13 (-0.05 to +0.40) |
| Air traffic control radio | +0.15 (-0.51 to +0.85) | 0.00 |
| Speech in the wild | -0.15 (-0.36 to 0.00) | 0.00 |
| Heavy white noise | +0.21 (-0.03 to +0.45) | 0.00 |
| Common Voice | +0.13 (0.00 to +0.33) | 0.00 |
| Meetings | -0.05 (-0.18 to 0.00) | 0.00 |
| Earnings calls | 0.00 | -0.03 (-0.09 to 0.00) |
| VoxPopuli | 0.00 | 0.00 |
| Clean LibriSpeech | 0.00 | 0.00 |
| FLEURS, 25 languages | 0.00 | 0.00 |

- Extra errors per 100 words (0 = no cost, right is worse)

**Figure 6.** Each row is a fresh set of real speech. The original gate cost noisy speech the most, while the retrained gate sits on zero almost everywhere.

**Figure data for agents: What the model writes when nobody is talking (bouncer off and on)** (generated from [bouncer.json](https://www.fermionresearch.com/research-interactive/reading-the-neurons-of-a-speech-model/bouncer.json))

Research copy of Phonon-2.5. Of 150 public clips with nobody speaking, 150 get phantom words with the bouncer off and 5 with it on; of 100 clips of real speech, 100 come out word for word the same with the bouncer on (0 changed).

The clips you can play in the figure:

| Sound (nobody speaking) | Length | Bouncer off: the model writes | Bouncer on: the model writes | Note | Source |
| --- | --- | --- | --- | --- | --- |
| A big dog barking | 0.9 s | “What?” | nothing |  | Freesound 182395 by pgonsilva, CC BY 3.0, via FSD50K |
| A rooster crowing | 4.8 s | “Ha ha” | nothing |  | Freesound 200339 by jsbarrett, CC0, via FSD50K |
| One short note on a French horn | 1.3 s | “Bye.” | nothing |  | Freesound 373240 by Samulis, CC0, via FSD50K |
| A wompoo fruit dove | 1.4 s | “Boa, pois.” (Portuguese, roughly “good, then”) | nothing |  | Freesound 114449 by soundbytez, CC BY 3.0, via FSD50K |
| A clarinet note with tremolo | 4.5 s | “Uh” | nothing |  | Freesound 356501 by MTG, CC BY 3.0, via FSD50K |
| A glass bottle being opened | 4.0 s | “Давай.” (Russian, “come on”) | nothing |  | Freesound 188903 by plingativator, CC BY 3.0, via FSD50K |
| A zipper | 0.6 s | “What?” | nothing |  | Freesound 195070 by punpcklbw, CC BY 3.0, via FSD50K |
| A single cat meow | 0.8 s | “Bill” | nothing |  | Freesound 334433 by ebcrosby, CC BY 3.0, via FSD50K |
| A double bass | 1.8 s | “I'm telling you, I'm telling you” | nothing |  | Freesound 372332 by SSS_Samples, CC BY 3.0, via FSD50K |
| A bunch of keys | 5.2 s | “Cool.” | nothing |  | Freesound 127558 by romulofs, CC0, via FSD50K |
| A knock on a door | 4.2 s | “Hmm” | nothing |  | Freesound 193829 by Ligidium, CC0, via FSD50K |
| A clock ticking on a table | 4.0 s | “Sono davvero molto presenta.” (Italian, roughly “they really are very present”) | “Sono davvero” | only partly cleaned | Freesound 388903 by ycbcr, CC BY 3.0, via FSD50K |
| A green-winged macaw | 1.3 s | “когда” (Russian, “when”) | “когда” | one of the clips the bouncer lets through | Freesound 111075 by soundbytez, CC BY 3.0, via FSD50K |

Every clip in the check (one dot each in the figure):

| # | Kind | Bouncer off | Bouncer on |
| --- | --- | --- | --- |
| 1 | Everyday sound | “Uh” | nothing |
| 2 | Everyday sound | “когда” | “когда” |
| 3 | Everyday sound | “Sorry to say.” | nothing |
| 4 | Everyday sound | “Boa, pois.” | nothing |
| 5 | Everyday sound | “Не прошло.” | nothing |
| 6 | Everyday sound | “Если усе, десу снять.” | nothing |
| 7 | Everyday sound | “Ему..” | nothing |
| 8 | Everyday sound | “Cool.” | nothing |
| 9 | Everyday sound | “Well, yeah.” | nothing |
| 10 | Everyday sound | “What?” | nothing |
| 11 | Music | “Ah.” | nothing |
| 12 | Music | “Да.” | nothing |
| 13 | Music | “Tā.” | nothing |
| 14 | Music | “Да.” | nothing |
| 15 | Music | “Меру” | nothing |
| 16 | Music | “Nem” | nothing |
| 17 | Everyday sound | “Go” | nothing |
| 18 | Everyday sound | “What?” | nothing |
| 19 | Music | “Uh” | nothing |
| 20 | Everyday sound | “Давай.” | nothing |
| 21 | Everyday sound | “Hmm” | nothing |
| 22 | Everyday sound | “What?” | nothing |
| 23 | Everyday sound | “Ha ha” | nothing |
| 24 | Everyday sound | “Wow” | nothing |
| 25 | Everyday sound | “What?” | nothing |
| 26 | Everyday sound | “Que?” | nothing |
| 27 | Everyday sound | “What?” | nothing |
| 28 | Everyday sound | “What?” | nothing |
| 29 | Everyday sound | “Пошли.” | nothing |
| 30 | Music | “Mm-hmm.” | nothing |
| 31 | Music | “Fras” | nothing |
| 32 | Music | “Хм” | nothing |
| 33 | Music | “А ну” | nothing |
| 34 | Music | “I would say” | nothing |
| 35 | Everyday sound | “Fight” | nothing |
| 36 | Music | “Qua igual igual vo.” | nothing |
| 37 | Everyday sound | “Bill” | nothing |
| 38 | Everyday sound | “Ele vai.” | nothing |
| 39 | Everyday sound | “Слышу, да?” | nothing |
| 40 | Music | “То” | nothing |
| 41 | Music | “Yeah.” | nothing |
| 42 | Music | “Uh” | nothing |
| 43 | Music | “Yeah.” | nothing |
| 44 | Music | “Yeah, yeah, yeah, yeah.” | nothing |
| 45 | Music | “Uh” | nothing |
| 46 | Music | “Okay” | nothing |
| 47 | Music | “I'm telling you, I'm telling you” | nothing |
| 48 | Music | “Bom” | nothing |
| 49 | Music | “Yeah.” | nothing |
| 50 | Music | “Yeah.” | “Okay.” |
| 51 | Music | “No.” | nothing |
| 52 | Music | “Bye.” | nothing |
| 53 | Music | “Qua?” | nothing |
| 54 | Music | “Um” | nothing |
| 55 | Music | “Um” | nothing |
| 56 | Music | “Yep.” | nothing |
| 57 | Music | “No.” | nothing |
| 58 | Everyday sound | “No?” | nothing |
| 59 | Everyday sound | “Okay.” | nothing |
| 60 | Everyday sound | “Sono davvero molto presenta.” | “Sono davvero” |
| 61 | Everyday sound | “Oh” | nothing |
| 62 | Everyday sound | “Fair” | nothing |
| 63 | Everyday sound | “Зло” | nothing |
| 64 | Everyday sound | “No” | nothing |
| 65 | Everyday sound | “Tu” | nothing |
| 66 | Music | “Так” | nothing |
| 67 | Everyday sound | “Да” | nothing |
| 68 | Everyday sound | “Ха” | nothing |
| 69 | Everyday sound | “Od” | nothing |
| 70 | Everyday sound | “Кры” | nothing |
| 71 | Everyday sound | “Как ты?” | nothing |
| 72 | Human sound, not speech | “No” | nothing |
| 73 | Human sound, not speech | “They don't do this to me.” | nothing |
| 74 | Human sound, not speech | “Ух.” | nothing |
| 75 | Human sound, not speech | “No, no, no, no, no.” | nothing |
| 76 | Human sound, not speech | “Wake up wake up.” | nothing |
| 77 | Human sound, not speech | “Bless you!” | nothing |
| 78 | Human sound, not speech | “Oh” | nothing |
| 79 | Human sound, not speech | “So” | nothing |
| 80 | Human sound, not speech | “Uh” | nothing |
| 81 | Human sound, not speech | “Yeah” | nothing |
| 82 | Human sound, not speech | “A fin” | nothing |
| 83 | Human sound, not speech | “And I was so scared.” | nothing |
| 84 | Human sound, not speech | “How could you?” | “How could you?” |
| 85 | Human sound, not speech | “Ha it's got me! Hell!” | nothing |
| 86 | Human sound, not speech | “No, please no” | nothing |
| 87 | Human sound, not speech | “Mm-hmm.” | nothing |
| 88 | Human sound, not speech | “Oh no, no no” | nothing |
| 89 | Human sound, not speech | “Ha” | nothing |
| 90 | Human sound, not speech | “No” | nothing |
| 91 | Human sound, not speech | “Co” | nothing |
| 92 | Human sound, not speech | “No hell” | nothing |
| 93 | Human sound, not speech | “А” | nothing |
| 94 | Human sound, not speech | “Хм” | nothing |
| 95 | Human sound, not speech | “Uh” | nothing |
| 96 | Human sound, not speech | “Ooh” | nothing |
| 97 | Human sound, not speech | “Uh-huh.” | nothing |
| 98 | Human sound, not speech | “Wait no” | nothing |
| 99 | Human sound, not speech | “Oh” | nothing |
| 100 | Human sound, not speech | “Uh yeah” | nothing |
| 101 | Human sound, not speech | “Uh ge, oh geah” | nothing |
| 102 | Human sound, not speech | “I hear you.” | nothing |
| 103 | Human sound, not speech | “Mmm” | nothing |
| 104 | Human sound, not speech | “Oh no” | nothing |
| 105 | Human sound, not speech | “Uh oh” | nothing |
| 106 | Human sound, not speech | “Uh” | nothing |
| 107 | Human sound, not speech | “Uh-huh.” | nothing |
| 108 | Human sound, not speech | “Uh” | nothing |
| 109 | Human sound, not speech | “No” | nothing |
| 110 | Human sound, not speech | “Who dangerous?” | nothing |
| 111 | Human sound, not speech | “So” | nothing |
| 112 | Human sound, not speech | “Jā.” | “Yeah.” |
| 113 | Human sound, not speech | “Ah!” | nothing |
| 114 | Human sound, not speech | “Com” | nothing |
| 115 | Human sound, not speech | “Why are you owning?” | nothing |
| 116 | Human sound, not speech | “Oh” | nothing |
| 117 | Human sound, not speech | “Hak tu” | nothing |
| 118 | Human sound, not speech | “Uuuuh” | nothing |
| 119 | Human sound, not speech | “No They don't love me anymore.” | nothing |
| 120 | Human sound, not speech | “Doctor” | nothing |
| 121 | Human sound, not speech | “Я все” | nothing |
| 122 | Human sound, not speech | “Oh no.” | nothing |
| 123 | Human sound, not speech | “Oh it isn't they don't love me anymore.” | nothing |
| 124 | Human sound, not speech | “Come.” | nothing |
| 125 | Human sound, not speech | “Please stop” | nothing |
| 126 | Human sound, not speech | “Хрину” | nothing |
| 127 | Human sound, not speech | “Send it back.” | nothing |
| 128 | Human sound, not speech | “Here” | nothing |
| 129 | Human sound, not speech | “Où les couilles nous là” | nothing |
| 130 | Human sound, not speech | “Ha ha” | nothing |
| 131 | Human sound, not speech | “Don't have” | nothing |
| 132 | Human sound, not speech | “What?” | nothing |
| 133 | Human sound, not speech | “Oh who are you? Get away from me!” | nothing |
| 134 | Human sound, not speech | “Hello” | nothing |
| 135 | Human sound, not speech | “All right,” | nothing |
| 136 | Human sound, not speech | “I don't know, I don't know.” | nothing |
| 137 | Human sound, not speech | “Ah” | nothing |
| 138 | Human sound, not speech | “When it's around” | nothing |
| 139 | Human sound, not speech | “Nö” | nothing |
| 140 | Human sound, not speech | “Anymore. They don't love me.” | nothing |
| 141 | Human sound, not speech | “Oh” | nothing |
| 142 | Human sound, not speech | “Oh” | nothing |
| 143 | Human sound, not speech | “Mom” | nothing |
| 144 | Human sound, not speech | “No who are you?” | nothing |
| 145 | Human sound, not speech | “Whoa!” | nothing |
| 146 | Human sound, not speech | “Ah” | nothing |
| 147 | Human sound, not speech | “Mm-mm.” | nothing |
| 148 | Human sound, not speech | “No please win go please” | nothing |
| 149 | Human sound, not speech | “Um” | nothing |
| 150 | Human sound, not speech | “Ha ha” | nothing |
| 151 | Noisy real speech | “Punk and crush. And you still don't know why. I my understanding is uh there's only one” | same |
| 152 | Noisy real speech | “The result is that, in order to keep up a decent appearance, the former habitually live hand to mouth to a greater extent than the latter,” | same |
| 153 | Noisy real speech | “I tried uh I tried to access my personal account in Google in uh Facebook ads and” | same |
| 154 | Noisy real speech | “As one review put it, the janitor is the undercover columns.” | same |
| 155 | Noisy real speech | “When a draft spoke with that silence, the nephano showed no emotion.” | same |
| 156 | Noisy real speech | “Searchers are a lot of people. Eat them that actually would like to have a little more freedom and space to be able” | same |
| 157 | Noisy real speech | “De Guuiche, observing that she was alone, near a thicket constructed of painted cloth, approached her,” | same |
| 158 | Noisy real speech | “The machine level for for the clusters, they might have been a disadvantage, but that becomes a programming tailwind opportunity.” | same |
| 159 | Noisy real speech | “It was no more exciting than an overturned carriage or a tree ranch off the road.” | same |
| 160 | Noisy real speech | “Fun. Well Dave, I know it's a short Friday because uh you took a red eye and uh I gotta get back on the AI leaders.” | same |
| 161 | Noisy real speech | “I took it up, and it had solved the engine oiling problem, and my walk to Lawrence, like a tramp actor's, was off.” | same |
| 162 | Noisy real speech | “I could just see my uncle at cool link from the raft, and Hans still in a stone sitting fire under the action of the U.S. whistle, which I sat” | same |
| 163 | Noisy real speech | “She, meanwhile, passed her life with her parents in the strictest retirement, never letting herself be seen, but shunning every eye lest it should weave her misfortune in her face.” | same |
| 164 | Noisy real speech | “cars. But the challenge is really, um, the bending over, obviously, but when I got the new studio to put a lift in there” | same |
| 165 | Noisy real speech | “Great Mother's Day response, which is in the course of a year. Long journey.” | same |
| 166 | Noisy real speech | “centabolic in this phase, like we're gonna do something a little bit different. Like instead of just doing these kind of five by five or a three by five routine, I'm gonna have” | same |
| 167 | Noisy real speech | “His pulse beat fast as he recalled the friends and comrades with whom he had spent the last few years in that vanished city.” | same |
| 168 | Noisy real speech | “The National Hockey Center features three international-level synthetic pitches.” | same |
| 169 | Noisy real speech | “Her condition was yesterday this very was critical but stable.” | same |
| 170 | Noisy real speech | “I can see Douglas there before the fire, to which he had got up to present his back, looking down at his interlocutor with his hands in his pockets.” | same |
| 171 | Noisy real speech | “He took no notice of her. He looked at me, but as if instead of me he saw what he spoke of.” | same |
| 172 | Noisy real speech | “That's a courtesy, that's about friendship, and that's not about letting people know we kept running 150. We lived a life of discipline. My guest, guys, is saying” | same |
| 173 | Noisy real speech | “It was fast, it was comfortable. I think if you were driving around town, totally cool. But I have to go, you know.” | same |
| 174 | Noisy real speech | “Be careful with your prognostication, said the stranger.” | same |
| 175 | Noisy real speech | “Heraldic matters in the Republic of Ireland are handled by the Genealogical Office.” | same |
| 176 | Speech in loud noise | “He came down to earth, lived among men, suffered, was crucified, and then he died, standing clearly before us, so that our hearts and eyes may fasten upon him.” | same |
| 177 | Speech in loud noise | “Mainhall vouched for her constancy with a loftiness that made Alexander smile, even while a kind of rapid excitement was tingling freedom.” | same |
| 178 | Speech in loud noise | “Do we really know the mounty well when we are not acquainted with the cabin?” | same |
| 179 | Speech in loud noise | “No harm would have been done had it not been that as he passed your door he perceived the key which had been left by the carelessness of your servant.” | same |
| 180 | Speech in loud noise | “The electric light can scarcely penetrate through the dense curtain which is dropped over the theatre on which the Battle of the Elements is about to be waged.” | same |
| 181 | Speech in loud noise | “Then of a sudden, at midday, the sun shot out, hot and still, no breath of air stirred, the sky was like blue steel, the earth seemed,” | same |
| 182 | Speech in loud noise | “Did you three days? She light the fire, every silver spread and circular.” | same |
| 183 | Speech in loud noise | “What is your name, Lordy? asked the little scholar, presently.” | same |
| 184 | Speech in loud noise | “Indeed, there were only one or two strangers who could be admitted among the sisters without producing the same result.” | same |
| 185 | Speech in loud noise | “There was something individual about the great farm, a most unusual gymnast and careful details.” | same |
| 186 | Speech in loud noise | “He might have had that forch check for the face of it, if he'd been sharp,” | same |
| 187 | Speech in loud noise | “A narrow channel formed a passage through the ridge of rocks that protected it from the open sea, and which, even in the roughest weather, would ensure the calmness of its waters.” | same |
| 188 | Speech in loud noise | “Then they started on again, and two hours later came in sight of the house of doctor Pitt.” | same |
| 189 | Speech in loud noise | “You know, I thought began to fix conscious to night. For the first time,” | same |
| 190 | Speech in loud noise | “I haven't had a chance yet to tell you what a jolly little place I think this is.” | same |
| 191 | Speech in loud noise | “I'm sure Alexandra hopes you will stay on here, she murmured.” | same |
| 192 | Speech in loud noise | “The army found the people in poverty, and left them in comparative wealth.” | same |
| 193 | Speech in loud noise | “Our first impressions of people are, in nine cases out of ten, the right impressions.” | same |
| 194 | Speech in loud noise | “Dear me, what a chatter box you're getting to be on, remarked the magician, who was pleased with the compliment.” | same |
| 195 | Speech in loud noise | “Funchis, who had already approached the door in readiness to leave the road, now recoiled and placed himself once more in the bottom of the live.” | same |
| 196 | Speech in loud noise | “It is so new that everywhere we feel the sense of energy.” | same |
| 197 | Speech in loud noise | “The attendance was unexpectedly large, and the girls were delighted for seeing great success for their fight.” | same |
| 198 | Speech in loud noise | “Would not the loftiest eminences of the city at least be visible?” | same |
| 199 | Speech in loud noise | “but the dusk, deepening in the schoolroom, covered over his thoughts. The bell rang,” | same |
| 200 | Speech in loud noise | “The mother, in managing the case in this way, relies partly on convincing the reason of the child, and partly on an appeal to her affection.” | same |
| 201 | Common Voice English | “The biggest sound stage is leased manually to hold the Golden Eagle Awards.” | same |
| 202 | Common Voice English | “The main city on the river is Rostov on Don.” | same |
| 203 | Common Voice English | “The requirements are rather high for this job.” | same |
| 204 | Common Voice English | “The channel has had critical and popular successes.” | same |
| 205 | Common Voice English | “The number still considered retired by the Jet successor team, the Arizona Coyotes.” | same |
| 206 | Common Voice English | “He was educated at Falcon.” | same |
| 207 | Common Voice English | “A memorial to Charlotte was installed in the Royal Garrison Church at Alderhot.” | same |
| 208 | Common Voice English | “Will the weather be freezing in Tennessee?” | same |
| 209 | Common Voice English | “The noise came from the base mat.” | same |
| 210 | Common Voice English | “While it has long the shelf life, the items will expire in 2026.” | same |
| 211 | Common Voice English | “She initially joined the leadership of the Women's Trade Union League.” | same |
| 212 | Common Voice English | “QuickRoat always sells Qisp directly to the public through an online store.” | same |
| 213 | Common Voice English | “Failed attempts at stopping urination include more petrols and fines.” | same |
| 214 | Common Voice English | “The All But One system was composed of several subsystems.” | same |
| 215 | Common Voice English | “How should I know anything of her?” | same |
| 216 | Common Voice English | “The floorboards looked blue.” | same |
| 217 | Common Voice English | “Its capital is the town Pombas.” | same |
| 218 | Common Voice English | “However, the multitude of people lived in small cottages or bungalows.” | same |
| 219 | Common Voice English | “Dunbar takes action but also takes a bullet, leaving him blind.” | same |
| 220 | Common Voice English | “The ground floor has large arcade arches with shop windows for the stores.” | same |
| 221 | Common Voice English | “Will it be stormy in Old Fort St. Lucia right now?” | same |
| 222 | Common Voice English | “The unincorporated community of Shady Dell is also located in the town.” | same |
| 223 | Common Voice English | “We must find out how long to run the experiment.” | same |
| 224 | Common Voice English | “Come on, Betty, let's go for a drink and s sink a few.” | same |
| 225 | Common Voice English | “Merculus's ambitions are said to have brought him into conflict with Agrippa and others.” | same |
| 226 | Meeting, distant microphone | “Or you could do um different like you get with mobile phones different fashions.” | same |
| 227 | Meeting, distant microphone | “Togs can some sometimes make out between a thief and a person. So basically these are the three unique features I think belong to you know. Okay, thank you.” | same |
| 228 | Meeting, distant microphone | “Can have a side for some preferences, I guess.” | same |
| 229 | Meeting, distant microphone | “than like just find it in the program and just say, okay, just how this one does it.” | same |
| 230 | Meeting, distant microphone | “because they use them quite frequently in watches.” | same |
| 231 | Meeting, distant microphone | “half of the selling price is taken up by building it.” | same |
| 232 | Meeting, distant microphone | “My dreadful was the worst yet.” | same |
| 233 | Meeting, distant microphone | “Target group. Yes, yes. I don't think we have to I don't think it's a case of worrying about different languages and things like that.” | same |
| 234 | Meeting, distant microphone | nothing | same |
| 235 | Meeting, distant microphone | “Um, and I'll be able to not answer them.” | same |
| 236 | Meeting, distant microphone | “Is it possible just as an option? When we open it up, people can use their fingers to press the button.” | same |
| 237 | Meeting, distant microphone | “Um I think we'll chat about it at the end. Okay.” | same |
| 238 | Meeting, distant microphone | “I haven't had some marketing data and basically every product is market driven.” | same |
| 239 | Meeting, distant microphone | “But if in a year's time we get feedback from the design fears that show something else is coming in, it's so easy for us to produce that that can be slipped on.” | same |
| 240 | Meeting, distant microphone | “Because they're also independent, I'll put that one down again.” | same |
| 241 | Meeting, distant microphone | “Um net control is in a way. There is.” | same |
| 242 | Meeting, distant microphone | “I have a bit of oh oh you can see it. Okay” | same |
| 243 | Meeting, distant microphone | “All it's gonna do is look for separate oh, 'cause then it's gonna go into the NHT search and then turn them into it.” | same |
| 244 | Meeting, distant microphone | “I mean sometimes I have methods that don't have any explanation to them at all.” | same |
| 245 | Meeting, distant microphone | nothing | same |
| 246 | Meeting, distant microphone | “Good here.” | same |
| 247 | Meeting, distant microphone | “Okay, I'm going to look at the functional requirements from a marketing point of view and obviously our starting point with marketing is always market research.” | same |
| 248 | Meeting, distant microphone | “Oh no they live there but there's” | same |
| 249 | Meeting, distant microphone | “Okay, this is off the top of my head. I was gonna do a bit that too.” | same |
| 250 | Meeting, distant microphone | “Just natural. It just seems to be occupied.” | same |

**Interactive.** Each clip is a public FSD50K recording with nobody speaking. The dots include the hundred clips of real speech it leaves alone.

**Figure data for agents: The bouncer, frame by frame** (generated from [gate.json](https://www.fermionresearch.com/research-interactive/reading-the-neurons-of-a-speech-model/gate.json))

Every clip has real speech in it. Three ways of running the research copy of Phonon-2.5: no clamp, the clamp everywhere, and the clamp only where the bouncer says nobody is talking. Each frame is 80 ms.

**A zipper in a pause** (10.6 s). Two sentences with a zipper in a 4.5-second pause between them. With no clamp the model types a phantom word over the sound. The clamp everywhere removes it but drops 14 real words. With the bouncer the clamp acts only in the quiet, so the phantom goes and every real word stays.

What was said: “Why, it's in Missouri somewhere on the frontier, I think. We'll get a map. I was afraid it was nearer home.”

| Setting | Phonon-2.5 writes | Phantom words | Real words changed | Real words dropped |
| --- | --- | --- | --- | --- |
| No clamp | “Why, it's in Missouri somewhere on the frontier, I think. We'll get a map. What? I was afraid it was nearer home.” | 1 | 0 | 0 |
| Clamp everywhere | “I was afraid it was nearer home.” | 0 | 0 | 14 |
| Clamp with the bouncer | “Why, it's in Missouri somewhere on the frontier, I think. We'll get a map. I was afraid it was nearer home.” | 0 | 0 | 0 |

Timeline: speech 0.00 s–4.53 s, pause 4.53 s–9.03 s, sound 6.57 s–7.00 s, speech 9.03 s–10.58 s. The bouncer lets the clamp act on 32 of 133 frames: 5.36 s–7.92 s. The ghost direction at block 18 rises highest at 7.04 s.

Speech: LibriSpeech test-clean 4970-29093-0020 and 4970-29093-0021, CC BY 4.0. Freesound 195070, “Zipper.wav” by punpcklbw, CC BY 3.0, via FSD50K.

**A bottle opening in a noisy park** (7.3 s). A sentence, 3 seconds of quiet, then a glass bottle being opened. The whole clip sits over the noise of a park. With no clamp the model types a phantom word over the sound. The clamp everywhere removes it but drops 10 real words. With the bouncer the clamp acts only in the quiet, so the phantom goes and every real word stays.

What was said: “Come, come, I called, when no one obeyed. A fire.”

| Setting | Phonon-2.5 writes | Phantom words | Real words changed | Real words dropped |
| --- | --- | --- | --- | --- |
| No clamp | “Come, come, I called, when no one obeyed. A fire. You across the never and never seen” | 7 | 0 | 0 |
| Clamp everywhere | nothing | 0 | 0 | 10 |
| Clamp with the bouncer | “Come, come, I called, when no one obeyed. A fire.” | 0 | 0 | 0 |

Timeline: speech 0.00 s–2.73 s, pause 2.73 s–5.73 s, sound 5.73 s–6.96 s. The bouncer lets the clamp act on 50 of 91 frames: 3.28 s–7.28 s. The ghost direction at block 18 rises highest at 2.72 s.

Speech: LibriSpeech test-clean 5142-33396-0025, CC BY 4.0. Freesound 188903, “glass_bottle_open.wav” by plingativator, CC BY 3.0, via FSD50K. Room noise: DEMAND NPARK (CC BY-SA 3.0); this mix is shared under CC BY-SA 3.0.

**A zipper before you start** (10.8 s). A zipper, 3.5 seconds of quiet, then a sentence. With no clamp the model types a phantom word over the sound. The clamp everywhere removes it but drops 16 real words. With the bouncer the clamp acts only in the quiet, so the phantom goes and every real word stays.

What was said: “Do you know? Lake? Oh, I really can't tell, but he'll soon tire of country life.”

| Setting | Phonon-2.5 writes | Phantom words | Real words changed | Real words dropped |
| --- | --- | --- | --- | --- |
| No clamp | “What? Do you know? Lake? Oh, I really can't tell, but he'll soon tire of country life.” | 1 | 0 | 0 |
| Clamp everywhere | nothing | 0 | 0 | 16 |
| Clamp with the bouncer | “Do you know? Lake? Oh, I really can't tell, but he'll soon tire of country life.” | 0 | 0 | 0 |

Timeline: sound 0.30 s–0.73 s, pause 0.73 s–4.23 s, speech 4.23 s–10.83 s. The bouncer lets the clamp act on 32 of 136 frames: 0.00 s–2.56 s. The ghost direction at block 18 rises highest at 6.80 s.

Speech: LibriSpeech test-clean 5683-32865-0013, CC BY 4.0. Freesound 195070, “Zipper.wav” by punpcklbw, CC BY 3.0, via FSD50K.

**A station announcement** (5.7 s). A recorded announcement on a London Underground platform, from a collection of sounds that was meant to have nobody talking in it. The clamp everywhere turns down the model's sense that someone is talking and changes 2 real words. The bouncer hears a voice the whole time, never lets the clamp act, and every word stays.

What the announcement says: “Please mind the gap between the train and the platform.”

| Setting | Phonon-2.5 writes | Phantom words | Real words changed | Real words dropped |
| --- | --- | --- | --- | --- |
| No clamp | “Please mind the gap between the train and the platform.” | 0 | 0 | 0 |
| Clamp everywhere | “Please remind the gap between the training and the platform.” | 0 | 2 | 0 |
| Clamp with the bouncer | “Please mind the gap between the train and the platform.” | 0 | 0 | 0 |

Timeline: speech 0.13 s–4.65 s. The bouncer hears a voice the whole time and never lets the clamp act (0 of 71 frames). The ghost direction at block 18 rises highest at 0.16 s.

Freesound 102302, “Circle Line LU Mind The Gap.mp3” by karthics4, CC0 (the public Freesound copy).

**Interactive.** Three clips we built from public LibriSpeech, Freesound and DEMAND recordings, and one real station announcement. Every transcript is the research copy of Phonon-2.5’s own output.

## What it all adds up to

We went looking for the cause of one strange habit and came out with something closer to a map. A model trained only to turn sound into words keeps careful notes about everything around them, including the language you speak, how fast you talk, the room you are in and whether anyone is talking at all. Reading those notes gives you detectors for free, and writing to them gives you fixes, as long as you also learn when to leave them alone, which is the lesson the bouncer taught us. All of this started on a research copy of the model, and the bouncer is the first piece of it to leave the lab. It now ships in Detta 1.1.3, where it cleans 142 of the 150 noise clips that used to get phantom words (95%), leaves 99 of 100 clips of real speech word for word the same, and never blanks a short command the model heard correctly.

We also have a working theory of why some things get a neuron of their own. Every moment of speech is in some language, at some speed and in some room, so these facts are true of everything the model ever hears, they hold steady across a whole recording, and only one of each applies at a time. That is exactly the kind of thing a network can afford to give a dial of its own. Things that change from moment to moment are spread across many neurons instead, and the clearest case is the one behind the ghosts, the sense of whether anyone is talking right now, which lives in a direction rather than in any single neuron. Our other post, [Mapping every neuron in Qwen3-8B](https://www.fermionresearch.com/research/mapping-every-neuron-in-qwen3-8b/), finds the same pattern in a language model, where nearly every single-neuron switch sets who is speaking and how, and none sets whether they are telling the truth. It is not about sheer quantity: across 15 languages, the amount of audio a language has in the largest public collection of European speech does not predict how clean its neuron is. What matters is that a property is always present, not how often any one value of it appears.

We will be open-sourcing the framework behind this work, our tools for finding, testing and editing individual neurons, in the near future, so anyone can run these experiments on their own models.

We don't write research papers. Papers now come out faster than anyone can read them, and good work gets lost in the pile, so we publish our research here, in plain language, where anyone can find it. If you would like to cite this work, please cite this page: Fermion Research, “Reading the neurons of a speech model”, 18 November 2026, [https://fermionresearch.com/research/reading-the-neurons-of-a-speech-model/](https://fermionresearch.com/research/reading-the-neurons-of-a-speech-model/).

## Notes for the curious

1. For the curious, the encoder is a stack of 24 conformer blocks and our neurons are the hidden units of their feed-forward layers, each scored by its average activation over a clip. We grade a neuron with AUROC, the chance that a random clip with the property scores higher than a random clip without it, so a coin flip scores 0.5 and a perfect detector 1.0. A neuron counts only when its score on held-out recordings beats the 95th percentile of twenty scrambled-label searches by at least 0.20, and every push we make on the model is compared with thirty random pushes of the same size. Whisper-small's 98.6% is 1,971 of 2,000 clips (95% CI 97.9 to 99.0%), and the tempo result reproduced on an independent rerun (23.7 to 3.1 errors per hundred words). [↩](#ref1)
2. Each language neuron scores an AUROC of at least 0.95 on FLEURS speakers it never saw. On 786 VoxPopuli parliament clips in 16 languages the neurons score 99.1%, reading the deepest neuron wherever several tie on FLEURS, against 97.2% for the dedicated VoxLingua107 language-identification model, and the gap holds at 95% confidence however the clips are paired (+0.6 to +3.3 points). A reader trained on a whole block of neurons manages only 27% on the same clips, because it memorised the benchmark's recordings. [↩](#ref2)
3. The bouncer is a speech reader early in the encoder (block 6 in the original, blocks 4 to 10 in the retrained version) and the clamp acts at block 18. The original bouncer's cutoff came from clean and mildly noisy recordings. On fresh test clips the retrained gate removed 87.3% of 433 phantom words (95% CI 83.8 to 90.1%) and created one new phantom on 1,000 clean clips, and the one noisy collection that moved, URGENT 2024, rose by 0.13 points. Letting Silero VAD gate the same clamp removed 85.9% but changed 42 of the 100 ship-check speech clips and raised URGENT error by about 2 points, and no Silero setting matched the retrained bouncer on both counts. WebRTC VAD marks 53 to 66% of noise frames as speech, which rules it out. [↩](#ref3)

## More from the lab

![An amber cellular orbit gathering around a compact central form](https://www.fermionresearch.com/images/backgrounds/amber-orbit.webp)

AnnouncementJuly 27, 2026

### Introducing the Neutrino-1 models

Three open-weight models trained for a compact ternary format and released with CUDA, Apple-silicon, and x86 runtimes.

![A cobalt cellular membrane becoming progressively finer](https://www.fermionresearch.com/images/backgrounds/cobalt-cell.webp)

ResearchJuly 27, 2026

### Intelligence at one-eighth the bits

One-shot two-bit conversion lands near chance. Training inside the constraint produces 72.1 MMLU from a 3.88 GB artifact. This is what changed inside the weights.

## Data for agents

Every figure in this post is drawn from these public JSON files. They are plain static files: fetch them directly, no JavaScript needed.

- [neurons.json](https://www.fermionresearch.com/research-interactive/reading-the-neurons-of-a-speech-model/neurons.json): Language and room neurons: the 25 language neurons and 6 place neurons (name, encoder block), and for every clip the per-frame activation of each neuron (80 ms frames, scaled 0-255), its clip average and the clip's source.
- [tempo.json](https://www.fermionresearch.com/research-interactive/reading-the-neurons-of-a-speech-model/tempo.json): The speedometer: one LibriSpeech clip at 1× to 3× with the speedometer reading and word-by-word transcripts as is and after slowing down, plus the 60-clip error rates per speed.
- [planted.json](https://www.fermionresearch.com/research-interactive/reading-the-neurons-of-a-speech-model/planted.json): Sentences planted in silence: the 20 pre-registered sentences, what Phonon-2.5 wrote for each, word-for-word hits and the random-vector controls.
- [bouncer.json](https://www.fermionresearch.com/research-interactive/reading-the-neurons-of-a-speech-model/bouncer.json): The bouncer on 250 public clips (150 with nobody speaking, 100 of real speech): what the model writes with the bouncer off and on, plus the 13 playable clips with Freesound credits.
- [gate.json](https://www.fermionresearch.com/research-interactive/reading-the-neurons-of-a-speech-model/gate.json): The bouncer frame by frame on four clips: per-frame bouncer reading, threshold, frames where the clamp may act, ghost-direction level, and transcripts with no clamp, clamp everywhere and clamp with the bouncer, each compared word by word with what was said.

Source: [https://www.fermionresearch.com/research/reading-the-neurons-of-a-speech-model/](https://www.fermionresearch.com/research/reading-the-neurons-of-a-speech-model/)
