Many of today’s multimodal models can look at an image and answer questions about it, and a large share of them trace a design idea to one 2023 paper: BLIP-2, from Salesforce Research. Instead of training a huge model on images and text from scratch, its authors asked a cheaper question: can we keep a strong image encoder and a strong language model frozen, and train only a small module that translates between them? This explainer covers what that module does, how it was trained, what the paper reported and where the approach falls short. The paper, by Junnan Li, Dongxu Li, Silvio Savarese and Steven Hoi, was published at ICML 2023, a peer-reviewed conference, and the arXiv version is the preprint of that work. We read the paper itself; we did not rerun its experiments.

The problem BLIP-2 tackles

Vision-language pretraining was getting expensive. End-to-end training of large models on image-text pairs requires substantial compute, and each new, better language model would require starting over. At the same time, two kinds of models were already very good at their own jobs: image encoders such as the one in CLIP, and large language models trained on enormous text corpora.

The difficulty is that a language model only understands text tokens, and an image encoder produces visual features that mean nothing to it. Freezing both, to save compute and avoid the language model “forgetting” its text abilities, means something has to connect the two. Earlier work such as DeepMind’s Flamingo inserted new layers into a frozen language model. BLIP-2 chose a different, lighter connection.

The Q-Former

The paper’s central component is the Querying Transformer, or Q-Former. It is a small transformer, around 188 million trainable parameters according to the paper, that sits between the frozen image encoder and the frozen language model. It holds a fixed set of learnable “query” vectors, 32 in the paper’s setup. Those queries interact with the image features through cross-attention and with each other through self-attention, and the result is a short, fixed-length summary of the image regardless of how large the image encoder’s output is.

Two things make this useful. First, the summary is compact, so the language model receives only a few visual tokens instead of hundreds. Second, the queries learn to pull out the visual information that is most relevant to text, acting as a bottleneck that filters what the language model sees. A small bridge replaces the expensive job of retraining the giants on either side.

Two stages of training

BLIP-2 trains the Q-Former in two stages, so that it first learns what in an image matters for language and then learns to speak to a specific language model.

Stage one: vision-language representation learning. With only the image encoder attached, the Q-Former is trained on image-text pairs with three objectives borrowed from earlier BLIP work: matching images to captions contrastively, deciding whether a caption and image match, and generating the caption conditioned on the image. This forces the queries to capture text-relevant visual detail.

Stage two: vision-to-language generative learning. The Q-Former output is passed through a linear layer and prepended to the input of a frozen language model as a soft visual prompt. The model is trained to generate text from it. The paper tried decoder-only language models (OPT) and encoder-decoder models (FlanT5), and image encoders including a CLIP vision transformer and EVA-CLIP’s ViT-g. Because the language model is frozen, it keeps its text knowledge, and swapping in a newer language model is a matter of retraining the small bridge.

For training data, the authors used about 129 million images from sources including COCO, Visual Genome, Conceptual Captions and a filtered subset of LAION-400M, and applied a captioning-and-filtering method called CapFilt to generate synthetic captions and remove noisy web text.

What the paper reported

The headline claim is efficiency. According to the paper, BLIP-2 outperformed Flamingo80B on zero-shot VQAv2, a visual question answering benchmark, by 8.7 percent while using 54 times fewer trainable parameters. The authors also reported state-of-the-art or competitive results on image captioning, image-text retrieval and other visual question answering tasks, and showed zero-shot image-to-text generation that could follow natural-language instructions, such as writing a story about a photo or answering questions about it.

Treat these numbers as the authors’ reported results on the benchmarks and model versions available in early 2023. Newer systems have since surpassed them, and benchmark scores do not by themselves tell you how a model performs on your images.

Limitations the authors acknowledge

The paper is candid that the method did not give the language model in-context learning from images: showing the model several image-text examples in the prompt did not improve its output as it does for text. The authors attribute this to the pretraining data containing only single image-text pairs, not interleaved sequences. They also note that the model inherits the weaknesses of its frozen parts, including possible inaccurate knowledge, unwanted language output and the risk of leaking private information from the language model’s training data. Image-text datasets scraped from the web carry their own biases.

Why it still matters

The Q-Former idea, a small trainable connector between frozen specialist models, shaped a family of later designs. LLaVA, published a few months later, made a simpler choice, using a projection layer and visual instruction tuning, and the trade-off between compact bridges and simpler ones is still debated. For practitioners, the lesson is about cost: you can often get a capable multimodal system by composing strong parts and training the glue, which lowers the barrier for research groups without frontier-scale compute. Readers new to the area may find our glossary entries on multimodal models and vision-language models useful background.

What to watch next

  • Whether compact connectors like the Q-Former keep their place as native multimodal training scales up.
  • Better evaluation of visual hallucination, where a model describes things not in the image.
  • Interleaved image-text training that restores in-context learning with images.
  • Efficient adaptation methods that let teams swap in newer language models cheaply.
  • Open reproductions and independent benchmark comparisons.

Key takeaways

  • BLIP-2 keeps the image encoder and language model frozen and trains only a small Q-Former between them.
  • The Q-Former uses 32 learned queries to produce a short visual summary for the language model.
  • Training happens in two stages: representation learning, then generative learning.
  • The authors report beating Flamingo80B on zero-shot VQAv2 by 8.7 percent with 54 times fewer trainable parameters.
  • The paper notes limits, including no gain from in-context image examples and inherited model biases.

Frequently asked questions

Is BLIP-2 peer reviewed?

Yes. It was published at ICML 2023. The arXiv version is the preprint of the paper.

What does “frozen” mean here?

The weights of the image encoder and the language model are not updated during training. Only the Q-Former and a small projection layer learn.

Does BLIP-2 generate images?

No. It takes images as input and produces text, such as captions and answers to questions about a picture.