Multimodal assistants can describe a photo, read a chart, transcribe and answer spoken questions or generate images from text. Some process all modalities natively in one network; others connect specialized encoders to a language model.
Multimodality enables applications such as visual assistance for blind users and voice conversations in real time.