Scale and training data

GPT-3 contains 175B parameters trained on a filtered mix of Common Crawl, WebText2, Books1, Books2, and English Wikipedia. Common Crawl was filtered aggressively using a classifier trained to distinguish high-quality documents from low-quality ones, with fuzzy deduplication applied across the full corpus. The model uses the same decoder-only transformer architecture as GPT-2, scaled to 96 layers, 12,288-dimensional embeddings, and 96 attention heads. Alternating dense and banded sparse attention patterns are used in some layers to manage compute at this scale.

In-context learning

GPT-3 is evaluated under three settings. Zero-shot provides only a natural language task description with no examples. One-shot provides one demonstration example alongside the description. Few-shot provides up to roughly 50 examples, all within the context window, with no weight updates of any kind.

In-context learning is fundamentally different from fine-tuning. No gradients are computed and no weights change. The examples in the prompt shift the model’s completion distribution by demonstrating the expected format and task structure. The model recognizes patterns in the context and continues them. This is purely inference-time behavior, not learning in the gradient-based sense.

The scaling law finding is sharp: larger models are dramatically better few-shot learners, and the relationship is smooth across several orders of magnitude. A 175B model is not just incrementally better than a 13B model at few-shot tasks; it is categorically better.

Strengths and limitations

GPT-3 excels at tasks that can be framed as pattern completion: TriviaQA, translation, arithmetic on small numbers, word unscrambling, and analogy tasks. It struggles on tasks requiring consistent multi-step reasoning, where errors early in a chain of thought compound and the model cannot reliably self-correct.

Results and impact

GPT-3 achieved state-of-the-art or near-state-of-the-art on dozens of NLP benchmarks in few-shot settings. Its commercial release via API changed how AI was deployed, moving from model-per-task systems to a single general-purpose model accessed through prompting. It established in-context learning and scaling laws as central research areas. The API-first release model and the question of emergent capabilities at scale defined the research agenda for the following several years.