The fragmentation problem

By 2019, NLP had accumulated a collection of task-specific architectures, pre-training objectives, and fine-tuning procedures. BERT used a bidirectional encoder. Autoregressive models used decoders only. Summarization used sequence-to-sequence models. This fragmentation made it hard to isolate which design choices actually mattered and to reuse components across tasks. T5 resolves this by converting every NLP task into the same form: a text-to-text problem where both input and output are always strings processed by an encoder-decoder transformer.

The text-to-text framing

Every task is expressed as a string transformation. Summarization becomes “summarize: {article}” mapping to “{summary}”. Translation becomes “translate English to German: {text}” mapping to “{translation}”. Sentiment classification becomes “sst2 sentence: {text}” mapping to “positive” or “negative”. Question answering becomes “question: {q} context: {c}” mapping to the answer span. A task-specific prefix in the input tells the model which behavior to activate. The model is trained with cross-entropy on the output string regardless of task. This single framing allows direct comparison of architecture variants, pre-training objectives, and dataset choices across every task without changing anything else.

Span corruption and the C4 dataset

T5 replaces BERT-style masked language modeling with span corruption. Contiguous spans of tokens are removed from the input and replaced with a single sentinel token per span. The model predicts only the removed content, not the full sequence. This is more compute-efficient than MLM because fewer tokens need to be predicted per training step. Pre-training uses C4 (Colossal Clean Crawled Corpus), a 750GB dataset built from Common Crawl by filtering for English, deduplicating content, and keeping only lines ending in terminal punctuation. The paper’s ablations show that data quality significantly affects downstream task performance.

Ablation findings

The systematic ablations are as important as the model itself. Span corruption outperforms BERT-style MLM in compute efficiency without sacrificing quality. More pre-training compute consistently improves results. Model scale matters more than most architectural variations. Multi-task pre-training helps when task mixture proportions are tuned carefully. T5 was released at five sizes ranging from 60M parameters (Small) to 11B parameters.

Results and impact

T5 simultaneously achieved state of the art on GLUE, SuperGLUE, SQuAD, and CNN/DailyMail at release. The text-to-text framing proved highly influential: Flan-T5 extended it with instruction tuning, mT5 applied it multilingually, and the pattern of expressing diverse tasks as string-to-string transformations is a direct ancestor of instruction-following language models. The ablation methodology is the standard reference for empirical pre-training design.