Fermion ResearchFermion Research

Glossary/facts checked 13 November 2026

Transducer (RNN-T) vs CTC vs attention encoder-decoder

The three families of end-to-end speech recognition models, what each one assumes, and why the choice matters for streaming, speed and errors.

Phonon-2.5 benchmarks and downloadsOpen weights on Hugging Face.

The short answer

A transducer, often called RNN-T, is a speech recognition architecture with three parts: an encoder that reads the audio, a prediction network that reads the tokens emitted so far, and a joint network that combines the two to emit either the next token or a blank that moves on to the next audio frame. Alex Graves introduced it in "Sequence Transduction with Recurrent Neural Networks" (arXiv 1211.3711, November 2012).

Its two main alternatives are CTC, which emits one token or blank per frame with no memory of earlier tokens, and attention encoder-decoders such as Listen, Attend and Spell or Whisper, which write the transcript token by token while attending over the whole audio.

The families side by side

"RNN" in the name is historical: modern transducers usually use Conformer or Transformer encoders. The family is defined by how it aligns audio with text.

CTCTransducer (RNN-T)TDTAttention encoder-decoder
PaperGraves et al., ICML 2006Graves, arXiv 1211.3711 (2012)Xu et al., arXiv 2304.06795 (2023)Chan et al., LAS, arXiv 1508.01211 (2015)
Uses earlier tokensNoYesYesYes
Alignment to audioMonotonic, frame by frameMonotonic, frame by frameMonotonic, skips framesLearned by attention, not forced to be monotonic
StreamingNaturalNaturalNaturalNeeds chunking or special variants
Decoding costOne pass over framesJoint network per frame and tokenFewer joint steps than RNN-TOne decoder step per output token
Typical failureSpelling and context errors without an external language modelErrors stay tied to audio frames; no free-running text generationSame as RNN-TCan repeat or invent text on silence or long audio if alignment drifts

How a transducer decodes

The encoder turns audio into a sequence of frames. Starting at the first frame, the joint network looks at that frame and at the prediction network's summary of the tokens so far, and outputs a distribution over tokens plus a blank. If it picks a token, the token is appended and the prediction network updates; if it picks blank, decoding moves to the next frame. Several tokens can be emitted at one frame, and none at others.

Because decoding walks forward through the audio and never jumps back, a transducer can emit words while the speaker is still talking. That is why the family is common in dictation and live captioning.

How the alternatives differ

CTC (Graves, Fernández, Gomez and Schmidhuber, ICML 2006) adds a blank symbol so a network can label unsegmented audio directly, but it treats each frame's output as independent of the others given the audio. It is fast and simple, and often paired with an external language model for context.

Attention encoder-decoders, introduced for speech by Listen, Attend and Spell (Chan, Jaitly, Le and Vinyals, 2015), decode like a translation model: each output token attends over all encoder frames. Whisper (Radford et al., arXiv 2212.04356, 2022) is the best-known example. The approach handles translation and long context well, and it can generate text that does not match the audio when the attention loses its place.

TDT (Token-and-Duration Transducer) is a 2023 extension of the transducer that also predicts how many frames each token covers, so decoding can skip frames.

Questions

What is RNN-T in speech recognition?
The recurrent neural network transducer: an encoder for audio, a prediction network for previous tokens and a joint network that emits a token or a blank per step. Alex Graves introduced it in 2012 (arXiv 1211.3711).
What is the difference between CTC and RNN-T?
CTC predicts each frame's output independently of earlier tokens; RNN-T adds a prediction network so each output depends on what was already emitted. Both align monotonically with the audio.
Is Whisper a transducer?
No. Whisper is an attention encoder-decoder (sequence-to-sequence) model that writes text token by token while attending over the whole audio window.
Which architecture is best for streaming?
CTC and transducers stream naturally because they move forward through the audio frame by frame. Attention encoder-decoders need chunking or special variants to stream.
Do transducers still use RNNs?
Usually not. The name is historical; modern transducers typically use Conformer or Transformer encoders, and the prediction network can be small.

Related

Sources and method

  • Graves, Sequence Transduction with Recurrent Neural Networks, arXiv 1211.3711 (2012-11-14), read 2026-11-13.
  • Graves, Fernández, Gomez, Schmidhuber, Connectionist Temporal Classification: Labelling Unsegmented Sequence Data with Recurrent Neural Networks, ICML 2006 (cs.toronto.edu/~graves/icml_2006.pdf), read 2026-11-13.
  • Chan, Jaitly, Le, Vinyals, Listen, Attend and Spell, arXiv 1508.01211 (2015-08-05), read 2026-11-13.
  • Radford et al., Robust Speech Recognition via Large-Scale Weak Supervision, arXiv 2212.04356 (2022-12-06), read 2026-11-13.
  • Xu, Jia, Majumdar, Huang, Watanabe, Ginsburg, Efficient Sequence Transduction by Jointly Predicting Tokens and Durations, arXiv 2304.06795 (2023-04-13, revised 2023-05-29), read 2026-11-13.

Canonical: https://www.fermionresearch.com/glossary/transducer-rnn-t/