On this page
Read
The useful mental model
Each modality is encoded into model-readable representations, with modality-specific processing and limits. A multimodal model or coordinated system relates those representations within one task, then produces only the output types it explicitly supports.
In plain English
It can work with more than one kind of information at once, such as reading a question while inspecting an image or listening to audio.
Where builders use it
- 01Document and screenshot analysis
- 02Audio or video summarization
- 03Voice and visual assistants
Why it matters
Many real tasks arrive as mixed evidence rather than clean text. Combining the relevant modalities can preserve relationships between a written request, a visual region, a sound, or a moment in a video that separate pipelines might lose.
Why now
Current developer APIs can accept combinations of text, images, audio, and video for analysis or generation. Support still varies by model, direction, resolution, duration, latency, and cost, so modality capabilities must be checked rather than assumed.
A common misconception
“A multimodal model automatically accepts and produces every kind of media.”
Multimodal is not a universal capability label. Each model documents which modalities it accepts, which it can generate, and the limits and quality tradeoffs for each direction.
See
See the mental model
The visual is generated from the same structured knowledge as the written explanation.
A text question, screenshot, and voice note enter supported multimodal analysis, which relates their evidence and returns a diagnosis linked to visual regions and transcript timestamps.
Watch
Watch the explainer
A short visual explanation built from the same reviewed knowledge object.
Multimodal AI: relate supported forms of evidence
See a written question, screenshot, and voice note converge into one evidence-linked diagnosis while the system keeps its supported capability boundary explicit.
Read transcript
A learner sends a question, a screenshot of an error, and a voice note. Each is a different kind of input, called a modality. A text-only tool sees just the question. It misses the error on screen and the detail in the voice note. A multimodal system turns each input into something the model can read. Then it relates them in one task. The answer points to the exact part of the screenshot, and the moment in the voice note. That gives a diagnosis you can check against each source. But no model takes every kind of input or makes every kind of output. Limits differ in size, quality and cost. Multimodal means more than one kind of input can take part. The documented limits decide what is possible.
Builder example
Diagnose an automation issue from mixed evidence
A learner submits a text question, a screenshot of a failed automation, and a voice note describing what happened immediately before the error.
Failure
A text-only workflow sees only the typed question, misses the error shown in the screenshot, and loses the timing detail contained in the voice note.
Intervention
Use a model or coordinated pipeline that supports the three input modalities, preserve source references, and request a structured diagnosis with screenshot regions and transcript timestamps.
Observable result
The response identifies the visible error and the preceding spoken action, while each diagnostic claim points back to the modality that supports it.
Sources
Go deeper
- Google AI for Developers - Image understanding
- Google AI for Developers - Audio understanding
- Google AI for Developers - Video understanding
Last researched 11 October 2026