In early 2023, proprietary assistants were beginning to accept images as input, but open alternatives lagged far behind. A paper from researchers at the University of Wisconsin–Madison, Microsoft Research and Columbia University, Visual Instruction Tuning, introduced LLaVA (Large Language and Vision Assistant). Its recipe was strikingly simple and inexpensive, and it became the template for a large family of open multimodal models.

The idea: connect, don’t rebuild

Training a multimodal model from scratch is costly. LLaVA instead connects two existing models:

  • a pre-trained vision encoder — CLIP’s ViT-L/14 — which turns an image into a grid of feature vectors;
  • a pre-trained language model — Vicuna, an instruction-tuned model built on LLaMA.

Between them sits a small projection layer: in the original version, a single linear layer that converts image features into vectors the language model can read as if they were word tokens. The image becomes a sequence of “visual tokens” placed before the user’s question.

The data trick: AI-generated instructions

To teach the combined model to follow instructions about images, the team needed many examples of image-based conversations, and few existed. Their solution was to use a text-only language model, GPT-4, to generate the training data.

They gave the model text descriptions of images from the COCO dataset — captions and the coordinates of objects — and asked it to produce three types of examples:

  1. Conversations: questions and answers about the image.
  2. Detailed descriptions.
  3. Complex reasoning: questions requiring inference beyond what is literally visible.

The result was about 158,000 instruction-following samples, created without paying people to write them.

Two-stage training

  1. Feature alignment. Train only the projection layer on image–caption pairs, so visual features land in the language model’s input space. The vision encoder and language model stay frozen.
  2. Visual instruction tuning. Train the projection layer and the language model on the generated conversations, keeping the vision encoder frozen.

The whole process required modest compute by the standards of large models, which made it reproducible by academic labs.

Results

LLaVA showed impressive conversational abilities about images — explaining memes, describing scenes in detail, reasoning about what was happening. On a synthetic benchmark comparing its answers with those of GPT-4 given text descriptions of the images, LLaVA achieved a relative score of about 85%. Fine-tuned on ScienceQA, a multimodal science question benchmark, the combination of LLaVA and GPT-4 set a new state of the art at the time.

Limitations

  • Hallucinations: like other vision-language models, LLaVA sometimes described objects that were not in the image.
  • Low resolution: the CLIP encoder worked on small input images, limiting the reading of fine text or detailed charts.
  • Inherited biases from both the language model and the generated data.
  • Evaluation: using another AI model as the judge, as the paper partly did, has known weaknesses.

The legacy

The LLaVA design — vision encoder, projector, language model — became the standard pattern for open multimodal models. Follow-up versions improved the projector, raised image resolution and used better data, and many later open models adopted the same architecture with stronger components. Equally influential was the data idea: using strong models to generate training data for weaker ones is now common practice across AI.

Key takeaways

  • LLaVA connects a frozen CLIP vision encoder to an open language model through a small projection layer.
  • Its training conversations were generated by a text-only GPT-4 from human-written captions and object annotations.
  • Training happens in two cheap stages: align visual features, then fine-tune on visual instructions.
  • The “encoder + projector + LLM” design became the standard template for open multimodal models.
  • Like other vision-language models, it can still describe objects that are not in the image.

Frequently asked questions

What is a projection layer?

A small trainable network that converts the vectors produced by the vision encoder into vectors the language model understands as input tokens. It acts as a translator between two models that were trained separately.

Why freeze the vision encoder?

Freezing preserves the visual knowledge CLIP already learned from hundreds of millions of images and makes training cheaper. The model only needs to learn how to pass that knowledge to the language model.

Is training on AI-generated data a problem?

It can be. Generated data can contain errors and biases of the generating model, and models trained heavily on synthetic data may lose diversity. LLaVA showed it can work well when the data is grounded in real annotations, as with the COCO captions and object boxes used here.