--- title: "19 — Multimodal AI: Images, Video, Audio, and the APIs Behind Them" uid: multimodal-ai tags: ["multimodal", "stt", "tts", "audio", "dall-e", "vision", "video", "image-generation", "roadmap:ai-engineer", "whisper"] excerpt: "'The model can see pictures' undersells it. Multimodal capability reduces to four moves — understand, generate, transcribe, synthesize — across text, image, audio, and video." date: 2026-08-13T03:28:35+0000 source: https://www.aveshina.my.id/en/blog/multimodal-ai --- The sprawling "multimodal" category looked like a dozen unrelated capabilities to me until I found the primitive count. The frame that collapsed it: **multimodal AI is the integration of multiple data types — text, image, audio, video — into one reasoning system, and almost every capability reduces to one of four moves: understand an input, generate an output, transcribe audio to text, or synthesize text to audio.** [1] Once I saw those four primitives, the sprawling "multimodal" category collapsed into a small set of operations I could reason about clearly. The framing that finally landed is a two-by-two. On one axis: _consume_ an input versus _produce_ an output. On the other: the modality in question — visual (image, video) or auditory (audio, speech). Image understanding, image generation, speech-to-text, text-to-speech each occupy one quadrant, and almost every multimodal feature I'd build is some combination of these four moves, often piped into a text-based LLM for the reasoning. ```figure ``` ## Why multimodal: the use cases The reason to integrate modalities is that some tasks are fundamentally impoverished without them. **Multimodal AI powers visual question answering, content moderation, enhanced search, smarter virtual assistants, and interactive AR** — combining text, images, and audio for richer experiences across e-commerce, accessibility, and entertainment [2]. A support bot that can read a screenshot of the user's error, a search engine that finds images by their content, an accessibility tool that narrates what's on screen — none of these work with text alone. The pattern: whenever the user's problem is naturally expressed in a non-text modality, forcing it through text loses information that multimodal handling preserves. ## Image understanding: consume the visual **Image understanding** is the consume-visual quadrant — taking an image and producing semantic output about it [3]. The model recognizes objects, scenes, and actions, and can generate captions, answer questions about the image, or extract text (OCR). The OpenAI Vision API is the canonical example: it identifies objects, recognizes text, and interprets visual content, integrating image processing with natural language so I can ask "what's in this image and is it safe?" in one call [4]. For an AI Engineer, image understanding is the input side of any feature that reasons about user-uploaded images — moderation, classification, visual search, accessibility (describing images for visually impaired users), and document extraction. The model produces text or structured data from the image, and from there the rest of my text-based pipeline takes over. ## Image generation: produce the visual **Image generation** is the produce-visual quadrant — creating new images from prompts or existing data [5]. The generative models behind this (DALL-E, Stable Diffusion, and others) take a textual description and produce a corresponding image. The DALL-E API exposes this as a developer-facing tool: send a descriptive prompt, get back a generated image, opening up design, advertising, content creation, and art applications [6]. The honest tradeoff: generated images are cheap and fast but inconsistent. Getting a _specific_ image — consistent characters, precise composition, brand-aligned style — is much harder than getting _an_ image, which is why production image-generation features usually layer additional control (reference images, inpainting, style tuning) on top of the raw prompt-to-image call. The newer multimodal APIs (like Google's image generation and the NanoBanana API) increasingly bundle some of this control [7]. ## Video understanding: consume over time **Video understanding** extends image understanding along the time axis, and it usually involves analyzing both visual and audio content together [8]. The use cases — video summarization (extracting key scenes), content moderation (detecting inappropriate visuals or audio), video indexing for search and retrieval — all require reasoning about a sequence of frames and the accompanying audio, not a single still image. The engineering challenge is scale: a video is hundreds or thousands of frames, each potentially an image-understanding call, plus a contiguous audio track. Practical video-understanding features sample strategically (keyframes, not every frame), lean on the audio track for cheap signal, and use models designed to handle temporal context. It's the most computationally expensive of the four quadrants, which is why video features are the last to ship and the first to need cost optimization. ## Speech-to-text: consume the auditory **Speech-to-text (STT)** converts spoken language into written text, and it's the consume-auditory quadrant [9]. The payoffs are transcription, captions, and voice command — any feature where a user speaks and my system needs the words. **Whisper** is OpenAI's speech recognition model, available as an API and as open weights, supporting multiple languages and accents, and handling both real-time and pre-recorded audio [10]. For most speech features, Whisper (via API or self-hosted) is the default starting point. STT is also the bridge that makes audio addressable by the rest of a text-based pipeline. Once the audio is text, a text LLM can reason about it — summarize the meeting, extract action items, classify the support call. That "audio → text → reasoning" pattern is one of the most common multimodal architectures, because it lets me reuse all my text tooling on spoken input. ## Text-to-speech: produce the auditory **Text-to-speech (TTS)** is the produce-auditory quadrant — converting written text into natural-sounding spoken audio [11]. The applications are virtual assistants, educational tools, accessibility (reading content aloud), and any feature where the system needs to speak back. Cloud TTS services (AWS Polly is the canonical example) make this a simple API call: send text, receive an audio file. The quality bar has risen sharply — modern TTS produces natural, expressive speech, not the robotic output of a decade ago. For most features, the cloud TTS APIs are good enough that I'd never build TTS myself; the decision is which vendor's voice quality and language coverage fits the product. ## Audio processing: the broader auditory layer Beyond the two speech quadrants, **audio processing** covers the analysis of sound more generally — meeting transcription paired with visual analysis, voice-controlled assistants interpreting commands alongside on-screen context, multimedia content analysis for moderation or indexing [12]. The same bridge pattern applies: audio is processed into either text (via STT) or embeddings (via an audio embedding model), and from there it joins the text-based reasoning pipeline. Audio embeddings are the less-discussed but powerful piece — they let me do similarity search over audio the same way text embeddings do over text. ## How I use this The four-quadrant model gives me a design shortcut for any multimodal feature. I ask which quadrants are involved: does the feature consume images, generate images, consume audio, or produce audio? Each quadrant maps to specific APIs and models, and most features combine two or more (a voice assistant consumes audio via STT, reasons with a text LLM, and produces audio via TTS). I default to the hosted APIs — Vision API, DALL-E, Whisper, cloud TTS — because the quality and breadth are hard to match self-hosted, and I reach for open options (Whisper weights, Stable Diffusion) only when cost, privacy, or customization demand it. The unifying discipline is to remember that the text LLM is still the reasoning core; the other modalities are inputs to and outputs from that core, and the integration architecture — how each modality converts to or from text — is where the actual design work lives. ## References [1] Hugging Face, "A Multimodal World," 2024. [Online]. Available: [https://huggingface.co/learn/computer-vision-course/en/unit4/multimodal-models/a_multimodal_world](https://huggingface.co/learn/computer-vision-course/en/unit4/multimodal-models/a_multimodal_world) [2] Google Cloud, "Multimodal AI," 2024. [Online]. Available: [https://cloud.google.com/use-cases/multimodal-ai](https://cloud.google.com/use-cases/multimodal-ai) [3] OpenAI, "Low or High Fidelity Image Understanding," 2024. [Online]. Available: [https://platform.openai.com/docs/guides/images](https://platform.openai.com/docs/guides/images) [4] OpenAI, "Vision," 2024. [Online]. Available: [https://developers.openai.com/api/docs/guides/images-vision](https://developers.openai.com/api/docs/guides/images-vision) [5] AssemblyAI, "How DALL-E 2 Actually Works," 2024. [Online]. Available: [https://www.assemblyai.com/blog/how-dall-e-2-actually-works/](https://www.assemblyai.com/blog/how-dall-e-2-actually-works/) [6] OpenAI, "Image Generation," 2024. [Online]. Available: [https://platform.openai.com/docs/guides/images](https://platform.openai.com/docs/guides/images) [7] NanoBanana API, 2025. [Online]. Available: [https://nanobananaapi.ai/](https://nanobananaapi.ai/) [8] "Awesome LLMs for Video Understanding," GitHub, 2024. [Online]. Available: [https://github.com/yunlong10/Awesome-LLMs-for-Video-Understanding](https://github.com/yunlong10/Awesome-LLMs-for-Video-Understanding) [9] AWS, "What is Speech to Text?," 2024. [Online]. Available: [https://aws.amazon.com/what-is/speech-to-text/](https://aws.amazon.com/what-is/speech-to-text/) [10] OpenAI, "Whisper," 2024. [Online]. Available: [https://openai.com/index/whisper/](https://openai.com/index/whisper/) [11] AWS, "What is Text-to-Speech?," 2024. [Online]. Available: [https://aws.amazon.com/polly/what-is-text-to-speech/](https://aws.amazon.com/polly/what-is-text-to-speech/) [12] Appwrite, "The State of Audio Processing," 2024. [Online]. Available: [https://appwrite.io/blog/post/state-of-audio-processing](https://appwrite.io/blog/post/state-of-audio-processing) ```quiz Q: Multimodal AI is best understood as… - a single new kind of model that replaces text LLMs - the integration of multiple data types (text, image, audio, video) into one reasoning system, with four core moves correct: 1 explain: Multimodal isn't a new model class; it's integration. Most capabilities reduce to understand, generate, transcribe, or synthesize across modalities. Q: The four core multimodal primitives are… - search, retrieve, rank, generate - image understanding, image generation, speech-to-text, text-to-speech correct: 1 explain: These four cover consume/produce across visual/auditory. Most multimodal features combine them around a text-LLM reasoning core. Q: Why is the "audio → text → reasoning" pattern so common? - because audio can't be reasoned about directly - because STT converts speech to text, letting you reuse your entire text-based pipeline on spoken input correct: 1 explain: STT is a bridge. Once audio is text, summarization, classification, and all your text tooling apply without modification. Q: Why is video understanding the most computationally expensive of the four quadrants? - because video APIs don't exist - because a video is many frames (each a potential image call) plus a contiguous audio track, requiring strategic sampling correct: 1 explain: Practical video features sample keyframes, lean on the audio track for cheap signal, and use temporally-aware models — not every frame. Q: A practical default for most multimodal features is to… - self-host every modality model for cost reasons - use hosted APIs (Vision, DALL-E, Whisper, cloud TTS) and reach for open options only when cost, privacy, or customization demand it correct: 1 explain: Hosted APIs offer quality and breadth that's hard to match self-hosted. Open options earn their place on specific constraints, not by default. ```