The central question

GPT-2 asked a radical question: if a language model is large enough and trained on sufficiently diverse text, can it perform downstream tasks without any fine-tuning at all? The hypothesis is that diverse text on the internet contains examples of virtually every task humans care about, written in natural language. A model that truly understands language should be able to perform a task when prompted with a natural language description, the same way a person would.

WebText and scale

The training corpus is WebText, constructed by scraping all outbound links from Reddit posts with at least three upvotes. Upvote filtering acts as a quality proxy: content that humans found worth sharing tends to be coherent and informative. WebText contains roughly 40GB of text across 8 million documents, substantially more diverse than BooksCorpus.

The model itself is a 10x scale-up from GPT-1: 1.5B parameters across 48 layers. Two architectural changes accompanied the scale-up. Layer normalization was moved to the input of each sub-block rather than the output, and an additional layer normalization was added after the final self-attention block. Both changes improve training stability at scale.

Zero-shot task performance

Without any fine-tuning, GPT-2 performs tasks by prompting in natural language. Translation is elicited with “English: [sentence] French:”. Summarization is elicited with “TL;DR:” appended to an article. Question answering works by simply posing a question after a context passage. These tasks appear naturally in pre-training text, so the model implicitly learns their structure rather than being explicitly taught it.

A critical finding is that capabilities emerge with scale in ways not predictable from smaller models. GPT-1 showed no zero-shot translation ability; GPT-2 shows meaningful performance. This non-linear relationship between scale and capability became a central theme in subsequent research.

Results and impact

GPT-2 achieved state-of-the-art perplexity on seven of eight language modeling benchmarks in zero-shot evaluation. It demonstrated that zero-shot task performance is possible at scale, shifting the research question from “how do we fine-tune?” to “how large does the model need to be?”. OpenAI staged the release of model weights, citing misuse concerns, which itself generated significant public and academic debate. GPT-2 directly set up GPT-3 by showing that scale alone, without fine-tuning, produces qualitatively new capabilities.