AI Vocabulary · 03 Working with AI
What is Multimodal?
A model that works with more than text: images, audio, video, documents.
A multimodal model can take in, and sometimes produce, more than one kind of data. Many current LLMs can read photos, screenshots, charts, scanned PDFs and handwriting, and some can listen and speak or generate images.
This opens practical uses: reading a whiteboard photo into notes, extracting data from invoices, describing a chart, checking a design screenshot. The same caution applies as with text: the model can misread small numbers or details, so verify what matters.
Why it matters for you
Many everyday tasks are not pure text. Before buying a separate tool for scanning, transcription or image description, check whether the assistant you already use can do it.
Own it
Images and recordings often contain more personal data than you expect, such as faces, voices and documents in the background. Decide what may be uploaded, and where it is processed.
Where it comes from
"Multimodal" has long been used in research for systems that combine several kinds of data, such as text, images and sound. Two milestones brought it to everyday AI: OpenAI's CLIP in 2021 learned to connect images with text, and GPT-4 in 2023 accepted images as input in a mainstream assistant. Today, reading images and documents is a standard feature of most large models.
Related
Your AI Learning Hub
Learn the whole AI Vocabulary, step by step, and build with it. Build it. Own it.