Framing image generation as sequence modeling

Prior text-to-image approaches using GANs struggled with compositional prompts and rarely generalized to concept combinations not seen during training. DALL-E takes a different approach: treat image generation as a next-token prediction problem. If both text and images can be represented as sequences of discrete tokens, a transformer trained to predict the next token in a text-then-image sequence can generate images from a text prefix, using the same machinery that generates text.

The dVAE image tokenizer

Directly predicting pixels autoregressively is intractable for 256x256 images, which contain 196,608 pixel values. DALL-E first trains a discrete variational autoencoder to compress each 256x256 image into a 32x32 grid of discrete tokens drawn from a codebook of 8,192 visual token types. Each image is reduced to 1,024 integer codes. The dVAE is trained to reconstruct images from these codes, so the codebook learns to capture the visual content of images in a compact discrete form that a transformer can model as a sequence.

The transformer and CLIP reranking

The core model is a 12B parameter decoder-only transformer trained on 250 million image-text pairs. The input sequence consists of up to 256 BPE-encoded text tokens followed by 1,024 image tokens, for a maximum context of 1,280 tokens. Text tokens attend causally and image tokens see all preceding text tokens. The model is trained to predict each next token autoregressively. At inference, multiple candidate images are generated by sampling image token sequences from the text prefix, then decoded back to pixels via the dVAE decoder. A CLIP model re-ranks the candidates by cosine similarity between each image embedding and the text prompt embedding, and the top-ranked image is returned. CLIP reranking substantially improves both visual quality and prompt alignment compared to returning the first sample.

Results and impact

DALL-E produces convincing zero-shot compositional generation across concept combinations that rarely appear together in training data, including “an armchair in the shape of an avocado” and “a snail made of harp.” This demonstrated genuine generalization rather than memorization of training examples. The paper established that scale and autoregressive modeling were sufficient for creative image synthesis without task-specific architectures. DALL-E 2 and DALL-E 3 followed with diffusion-based generation that substantially improved realism and prompt fidelity, building directly on the tokenization and reranking ideas introduced here.