Flamingo, a family of visual language models from DeepMind described in an April 2022 paper by Jean-Baptiste Alayrac and 26 co-authors, showed that a model could learn a new image or video task from just a handful of examples placed in its prompt. The largest version, Flamingo-80B, set a new few-shot state of the art on all 16 image and video benchmarks the authors tested, and beat fine-tuned models on 6 of them while seeing only up to 32 task examples. The paper was published at NeurIPS 2022, a peer-reviewed conference. Its central trick, bolting new cross-attention layers onto a frozen language model, remains one of the two main ways multimodal models are built today. We read the paper; we did not rerun its experiments.
The problem Flamingo set out to solve
Before Flamingo, the usual way to get a vision model to do a new task, such as answering questions about photos or captioning videos, was to fine-tune it on thousands of labeled examples. Contrastive models like CLIP could classify images zero-shot, but they could not write free-form answers.
Large language models had meanwhile shown in-context learning: give GPT-3 a few input-output examples in the prompt and it would continue the pattern. The Flamingo team asked whether the same could work for images. Could a model read a prompt like “image of a cat: a cat sleeping on a sofa; image of a dog: a dog catching a ball; new image: …”, and complete it?
How Flamingo works
Flamingo has three main parts, and two of them are frozen, meaning their weights do not change during training.
A frozen vision encoder. The authors trained a NormalizerFree ResNet (NFNet-F6) image encoder with a CLIP-style contrastive objective, then froze it. For video, frames are sampled at one per second and encoded separately, with learned time embeddings added.
A Perceiver Resampler. An image produces a large and variable number of features. The resampler, inspired by DeepMind’s earlier Perceiver work, uses a fixed set of learned queries that attend to those features and output exactly 64 visual tokens, whatever the image size or number of video frames. This keeps the cost of the next step predictable.
A frozen language model with new gated layers. The text side is a pretrained Chinchilla language model. Between its frozen layers, the authors inserted new blocks, called GATED XATTN-DENSE, that let text tokens cross-attend to the visual tokens. Each new block’s output is multiplied by tanh of a learnable scalar that starts at zero. At the start of training the new layers therefore contribute nothing, and the model behaves exactly like the original language model. As training proceeds, the gates open and visual information flows in. The ablations show this zero-initialized gating improves both stability and final performance.
There were three sizes, built on 1.4B, 7B and 70B Chinchilla models: Flamingo-3B, Flamingo-9B and Flamingo-80B.
Handling many images in one prompt
To support interleaved prompts with several images, Flamingo uses a masking rule: each text token cross-attends only to the visual tokens of the image that came immediately before it. Earlier images still influence the output indirectly through the language model’s own self-attention. Because of this, the model was trained with at most five images per sequence but could use up to 32 image-text examples at test time.
What it was trained on
The few-shot ability depended on training data where images and text are interleaved, as on real web pages. DeepMind built MultiModal MassiveWeb (M3W) from the HTML of about 43 million web pages, keeping the positions of images relative to the text. This was mixed with three paired datasets: ALIGN, with 1.8 billion images paired with alt text; LTIP, 312 million image-text pairs with longer descriptions; and VTP, 27 million short videos, about 22 seconds on average, each paired with a sentence. None of these datasets were released.
What the paper reported
The authors evaluated on 16 benchmarks spanning visual question answering, captioning, video understanding and multiple-choice tasks. With only a few examples in the prompt, Flamingo outperformed previous zero-shot and few-shot methods on all 16, in some cases with as few as four examples. On 6 of the 16 tasks, Flamingo-80B with 32 examples also beat the best fine-tuned models, which had been trained on orders of magnitude more task-specific data. Larger models and more shots both helped.
How Flamingo compares with BLIP-2 and LLaVA
Flamingo, BLIP-2 and LLaVA all reuse pretrained vision and language models, but they connect them differently.
| Model | Year | How vision reaches the language model | What is trained |
|---|---|---|---|
| Flamingo | 2022 | New gated cross-attention layers inside the frozen LM, fed by 64 resampled visual tokens | Resampler and cross-attention layers |
| BLIP-2 | 2023 | A Q-Former turns image features into soft tokens placed at the start of the prompt | Q-Former and a projection layer |
| LLaVA | 2023 | A projection maps image features into the LM’s input embedding space | Projection, then the LM during instruction tuning |
The cross-attention route used by Flamingo keeps the language model’s input sequence short, since images do not occupy prompt positions, but it requires changing the model’s internals. The input-token route used by BLIP-2 and LLaVA is simpler to bolt onto any language model and has become more common in open models. Both designs are still in use.
Limitations
The authors list several weaknesses. Because Flamingo inherits a language model’s priors, it can hallucinate or make ungrounded guesses. On plain image classification it trails contrastive models like CLIP, which are trained directly for image-text matching. In-context learning is sensitive to which examples are chosen and in what order, and its compute cost grows with the number of shots. The training data were scraped from the web and not released, and the models themselves were never made publicly available, which limited outside scrutiny. The OpenFlamingo project later released open 3B to 9B replications, which averaged 80 to 89 percent of the corresponding Flamingo performance on seven datasets.
What to watch next
- Whether new multimodal models favor cross-attention designs, which keep long video out of the prompt, or the simpler token-injection designs.
- Open interleaved image-text datasets that make Flamingo-style training reproducible.
- Few-shot evaluation of current omni models on the same benchmarks Flamingo used.
Key takeaways
- Flamingo, from DeepMind, was published at NeurIPS 2022 and introduced few-shot learning for visual language models.
- It links a frozen image encoder to a frozen Chinchilla language model through a Perceiver Resampler and gated cross-attention layers.
- Zero-initialized tanh gates let training start from the unchanged language model.
- Flamingo-80B beat fine-tuned models on 6 of 16 benchmarks using at most 32 examples.
- Its training data and weights were not released; OpenFlamingo is an open replication.
Frequently asked questions
What is the Flamingo model?
Flamingo is a family of visual language models from DeepMind, published in 2022, that can answer questions about images and videos and learn new visual tasks from a few examples in the prompt, without fine-tuning.
What is a Perceiver Resampler?
It is a small transformer module in Flamingo that takes a variable number of image or video features and compresses them into a fixed set of 64 visual tokens, using learned queries that attend to the features.
Can I download Flamingo?
No. DeepMind did not release the Flamingo weights. Open replications such as OpenFlamingo, which offers models from 3B to 9B parameters, follow the same architecture and are publicly available.
