19 — Multimodal AI: Images, Video, Audio, and the APIs Behind Them

The sprawling "multimodal" category looked like a dozen unrelated capabilities to me until I found the primitive count. The frame that collapsed it: multimodal AI is the integration of multiple data types — text, image, audio, video — into one reasoning system, and almost every capability reduces to one of four moves: understand an input, generate an output, transcribe audio to text, or synthesize text to audio. [1] Once I saw those four primitives, the sprawling "multimodal" category collapsed into a small set of operations I could reason about clearly.

The framing that finally landed is a two-by-two. On one axis: _consume_ an input versus _produce_ an output. On the other: the modality in question — visual (image, video) or auditory (audio, speech). Image understanding, image generation, speech-to-text, text-to-speech each occupy one quadrant, and almost every multimodal feature I'd build is some combination of these four moves, often piped into a text-based LLM for the reasoning.

Why multimodal: the use cases

The reason to integrate modalities is that some tasks are fundamentally impoverished without them. Multimodal AI powers visual question answering, content moderation, enhanced search, smarter virtual assistants, and interactive AR — combining text, images, and audio for richer experiences across e-commerce, accessibility, and entertainment [2]. A support bot that can read a screenshot of the user's error, a search engine that finds images by their content, an accessibility tool that narrates what's on screen — none of these work with text alone. The pattern: whenever the user's problem is naturally expressed in a non-text modality, forcing it through text loses information that multimodal handling preserves.

Image understanding: consume the visual

Image understanding is the consume-visual quadrant — taking an image and producing semantic output about it [3]. The model recognizes objects, scenes, and actions, and can generate captions, answer questions about the image, or extract text (OCR). The OpenAI Vision API is the canonical example: it identifies objects, recognizes text, and interprets visual content, integrating image processing with natural language so I can ask "what's in this image and is it safe?" in one call [4].

For an AI Engineer, image understanding is the input side of any feature that reasons about user-uploaded images — moderation, classification, visual search, accessibility (describing images for visually impaired users), and document extraction. The model produces text or structured data from the image, and from there the rest of my text-based pipeline takes over.

Image generation: produce the visual

Image generation is the produce-visual quadrant — creating new images from prompts or existing data [5]. The generative models behind this (DALL-E, Stable Diffusion, and others) take a textual description and produce a corresponding image. The DALL-E API exposes this as a developer-facing tool: send a descriptive prompt, get back a generated image, opening up design, advertising, content creation, and art applications [6].

The honest tradeoff: generated images are cheap and fast but inconsistent. Getting a _specific_ image — consistent characters, precise composition, brand-aligned style — is much harder than getting _an_ image, which is why production image-generation features usually layer additional control (reference images, inpainting, style tuning) on top of the raw prompt-to-image call. The newer multimodal APIs (like Google's image generation and the NanoBanana API) increasingly bundle some of this control [7].

Video understanding: consume over time

Video understanding extends image understanding along the time axis, and it usually involves analyzing both visual and audio content together [8]. The use cases — video summarization (extracting key scenes), content moderation (detecting inappropriate visuals or audio), video indexing for search and retrieval — all require reasoning about a sequence of frames and the accompanying audio, not a single still image.

The engineering challenge is scale: a video is hundreds or thousands of frames, each potentially an image-understanding call, plus a contiguous audio track. Practical video-understanding features sample strategically (keyframes, not every frame), lean on the audio track for cheap signal, and use models designed to handle temporal context. It's the most computationally expensive of the four quadrants, which is why video features are the last to ship and the first to need cost optimization.

Speech-to-text: consume the auditory

Speech-to-text (STT) converts spoken language into written text, and it's the consume-auditory quadrant [9]. The payoffs are transcription, captions, and voice command — any feature where a user speaks and my system needs the words. Whisper is OpenAI's speech recognition model, available as an API and as open weights, supporting multiple languages and accents, and handling both real-time and pre-recorded audio [10]. For most speech features, Whisper (via API or self-hosted) is the default starting point.

STT is also the bridge that makes audio addressable by the rest of a text-based pipeline. Once the audio is text, a text LLM can reason about it — summarize the meeting, extract action items, classify the support call. That "audio → text → reasoning" pattern is one of the most common multimodal architectures, because it lets me reuse all my text tooling on spoken input.

Text-to-speech: produce the auditory

Text-to-speech (TTS) is the produce-auditory quadrant — converting written text into natural-sounding spoken audio [11]. The applications are virtual assistants, educational tools, accessibility (reading content aloud), and any feature where the system needs to speak back. Cloud TTS services (AWS Polly is the canonical example) make this a simple API call: send text, receive an audio file.

The quality bar has risen sharply — modern TTS produces natural, expressive speech, not the robotic output of a decade ago. For most features, the cloud TTS APIs are good enough that I'd never build TTS myself; the decision is which vendor's voice quality and language coverage fits the product.

Audio processing: the broader auditory layer

Beyond the two speech quadrants, audio processing covers the analysis of sound more generally — meeting transcription paired with visual analysis, voice-controlled assistants interpreting commands alongside on-screen context, multimedia content analysis for moderation or indexing [12]. The same bridge pattern applies: audio is processed into either text (via STT) or embeddings (via an audio embedding model), and from there it joins the text-based reasoning pipeline. Audio embeddings are the less-discussed but powerful piece — they let me do similarity search over audio the same way text embeddings do over text.

How I use this

The four-quadrant model gives me a design shortcut for any multimodal feature. I ask which quadrants are involved: does the feature consume images, generate images, consume audio, or produce audio? Each quadrant maps to specific APIs and models, and most features combine two or more (a voice assistant consumes audio via STT, reasons with a text LLM, and produces audio via TTS). I default to the hosted APIs — Vision API, DALL-E, Whisper, cloud TTS — because the quality and breadth are hard to match self-hosted, and I reach for open options (Whisper weights, Stable Diffusion) only when cost, privacy, or customization demand it. The unifying discipline is to remember that the text LLM is still the reasoning core; the other modalities are inputs to and outputs from that core, and the integration architecture — how each modality converts to or from text — is where the actual design work lives.

References

[1] Hugging Face, "A Multimodal World," 2024. [Online]. Available: https://huggingface.co/learn/computer-vision-course/en/unit4/multimodal-models/a_multimodal_world

[2] Google Cloud, "Multimodal AI," 2024. [Online]. Available: https://cloud.google.com/use-cases/multimodal-ai

[3] OpenAI, "Low or High Fidelity Image Understanding," 2024. [Online]. Available: https://platform.openai.com/docs/guides/images

[4] OpenAI, "Vision," 2024. [Online]. Available: https://developers.openai.com/api/docs/guides/images-vision

[5] AssemblyAI, "How DALL-E 2 Actually Works," 2024. [Online]. Available: https://www.assemblyai.com/blog/how-dall-e-2-actually-works/

[6] OpenAI, "Image Generation," 2024. [Online]. Available: https://platform.openai.com/docs/guides/images

[7] NanoBanana API, 2025. [Online]. Available: https://nanobananaapi.ai/

[8] "Awesome LLMs for Video Understanding," GitHub, 2024. [Online]. Available: https://github.com/yunlong10/Awesome-LLMs-for-Video-Understanding

[9] AWS, "What is Speech to Text?," 2024. [Online]. Available: https://aws.amazon.com/what-is/speech-to-text/

[10] OpenAI, "Whisper," 2024. [Online]. Available: https://openai.com/index/whisper/

[11] AWS, "What is Text-to-Speech?," 2024. [Online]. Available: https://aws.amazon.com/polly/what-is-text-to-speech/

[12] Appwrite, "The State of Audio Processing," 2024. [Online]. Available: https://appwrite.io/blog/post/state-of-audio-processing