--- title: "15 — LLM Integration Patterns — Providers, Streaming, Structured Output, and Function Calling" uid: llm-integration-patterns tags: ["structured-outputs", "streaming", "llm", "roadmap:backend", "anthropic", "openai", "ai", "gemini", "function-calling"] excerpt: "Every LLM integration is four decisions in sequence — which provider, streamed or not, structured or freeform, callable tools or not. Pick one of each and the pipeline assembles itself." date: 2026-08-13T03:28:24+0000 source: https://www.aveshina.my.id/en/blog/llm-integration-patterns --- Providers, streaming, structured outputs, function calling — each looked like a separate API to learn until I noticed the same four questions behind every integration. The composition: **every LLM integration is four decisions, in sequence — which provider, streamed or not, structured or freeform, callable tools or not.** [1] Pick one of each and the pipeline assembles itself. The frame that helped is that the providers all expose the same underlying capability (next-token prediction, from the earlier fundamentals post) with different surface APIs. The differences that matter for backend work aren't the model internals — they're the integration shape. Does the response arrive whole, or token-by-token as it generates? Does the model return freeform text, or JSON conforming to a schema I declared? Can the model decide to call a function I exposed, or only emit text? Those four choices define the engineering of any LLM feature. ```figure ``` ## The providers: OpenAI, Anthropic, Gemini Three providers dominate the backend LLM landscape, and they're more similar than different at the API level [2][5][6]: - **OpenAI** — the suite of models (GPT family) accessed via a hosted API, with text generation, function calling, structured output, embeddings, and image/multimodal capabilities. OpenAI's API set the de facto shape that others imitate; the SDK ergonomics are widely cloned. - **Anthropic** — the AI safety company behind Claude, accessed via API with similar capabilities (text generation, tool use, structured output). Anthropic's differentiators are a focus on safety and steerability (Constitutional AI), long context windows, and the Claude Code agentic CLI ecosystem. - **Gemini** (Google) — a family of multimodal models handling text, code, images, audio, and video, accessed via Google's API. Gemini's pitch is native multimodality and integration with Google's ecosystem. The practical observation: the provider choice is usually decided by model quality for the specific task, pricing, and ecosystem alignment (which SDK, which agent framework, which features matter), not by dramatic API differences. A text-generation call looks structurally similar across all three. The patterns below — streaming, structured output, function calling — exist in all three, with slightly different names. ## Streamed or unstreamed: who waits The first decision is whether the response arrives **whole** (the model finishes generating, then returns the complete text) or **streamed** (tokens are sent to the client as they're generated) [3]. - **Unstreamed** is simpler — one request, one response, easy to cache, easy to log. The cost is that the user waits for the whole answer before seeing anything, which feels slow for long outputs. - **Streamed** sends tokens as they're produced, so the user sees the answer grow in real time. The cost is implementation complexity (Server-Sent Events or chunked HTTP, partial JSON parsing, harder logging). The trade-off is UX versus simplicity. For chat-like features where the answer is several paragraphs, streaming materially changes perceived speed — the first token arriving in 500ms feels faster than a complete answer in 5 seconds, even though the total time is similar. For backend-to-backend calls where a human isn't watching, unstreamed is usually right. The portfolio pattern is streamed for user-facing chat, unstreamed for internal data extraction. ## Structured output: forcing parseable shape The second decision is whether the response is **freeform text** or **structured** — JSON conforming to a schema I declare [4]. This is the single most important pattern for backend LLM work, because freeform text is fragile to consume. The problem with freeform: the model might return "Yes, the user is over 18" or "The user's age category is adult: true" or "``\ntrue\n``" — all valid answers, none parseable without heuristics. Structured output fixes this by constraining the model to emit JSON matching a schema I provide: ``` { "type": "object", "properties": { "is_over_18": { "type": "boolean" }, "age_category": { "type": "string", "enum": ["minor", "adult", "senior"] } }, "required": ["is_over_18", "age_category"] } ``` The provider's structured-output mode (OpenAI, Anthropic, Gemini all support variants) constrains decoding so the output validates against the schema. The payoff is that the model's output becomes reliable program input — I parse it with the same JSON parsing I'd use for any API. The failure mode to know: structured output reduces (not eliminates) hallucination; the schema constrains shape, but the values can still be wrong. Validation of the *contents* is still on me. ## Function calling: the model decides to act The third decision is whether I expose **callable functions** (tools) to the model [7]. Without function calling, the model can only emit text. With it, I declare a set of functions (name, description, argument schema), and the model can decide — mid-conversation — to "call" one by emitting a structured request instead of plain text. The flow: 1. I send the prompt *plus* a list of available functions with their schemas. 2. The model either responds in text, or emits a function call: { "name": "get_order_status", "arguments": { "order_id": "42" } }. 3. My code executes the real function (query the database, call an API), gets the result. 4. I feed the result back to the model, which uses it to continue the response. The model decides *when* to call a function and *which arguments* to pass; my code actually runs it. This is the foundation of tool-using agents — function calling is the primitive that lets a model reach outside its training data and act on real state. It's also where structured output and function calling compose: the function-call arguments are themselves structured (a JSON object matching the function's schema), so they're reliably parseable. The payoff over the model writing raw code or natural-language commands: structure means fewer mistakes. The model emits a parseable request; my code does the deterministic work; the model reads the result. The boundary between probabilistic (the model) and deterministic (the function) is exactly where it should be. ## Composing the four choices The integration patterns emerge from composing these four decisions. A few common shapes: - **Chat feature** → any provider, streamed, freeform (or lightly structured for safety), function calling for tools (search the docs, look up the order). This is the agent pattern. - **Classification/extraction pipeline** → any provider, unstreamed, structured output (JSON schema), no function calling. Pure data transformation. - **Autonomous coding agent** → Anthropic/OpenAI, streamed for UX, structured for tool calls, function calling heavily (read file, run tests, edit). This is Claude Code and its peers. The same four decisions describe every LLM feature I've built or used. The provider is the least interesting choice; the other three define the engineering. ## How I use this Three rules capture the practical takeaway: - **Default to structured output.** Freeform text is fragile; JSON matching a schema is reliable. I reach for structured output for any backend LLM call where the result feeds code. - **Stream only when a human is watching.** Streaming adds complexity; I reserve it for user-facing chat where perceived latency matters. Internal calls stay unstreamed. - **Use function calling when the model needs real data.** If the answer depends on something the model can't know (current inventory, user-specific state), expose a function. The model decides when to call; my code returns the truth. The framing — *provider, streamed, structured, callable* — is the design language of LLM integration. Every feature I ship is one composition of these four, and naming them turns "build an AI feature" from an open-ended research problem into a sequence of engineering choices. ## References [1] "Emerging Patterns in Building GenAI Products," Martin Fowler, 2024. [Online]. Available: [https://martinfowler.com/articles/gen-ai-patterns/](https://martinfowler.com/articles/gen-ai-patterns/) [2] OpenAI, "OpenAI Platform." [Online]. Available: [https://platform.openai.com/docs/overview](https://platform.openai.com/docs/overview) [3] "Streaming Responses in AI: How AI Outputs Are Generated in Real Time," dev.to. [Online]. Available: [https://dev.to/pranshu_kabra_fe98a73547a/streaming-responses-in-ai-how-ai-outputs-are-generated-in-real-time-18kb](https://dev.to/pranshu_kabra_fe98a73547a/streaming-responses-in-ai-how-ai-outputs-are-generated-in-real-time-18kb) [4] OpenAI, "Structured model outputs." [Online]. Available: [https://platform.openai.com/docs/guides/structured-outputs](https://platform.openai.com/docs/guides/structured-outputs) [5] Anthropic, "Anthropic." [Online]. Available: [https://www.anthropic.com/](https://www.anthropic.com/) [6] Google, "Google Gemini." [Online]. Available: [https://gemini.google.com/](https://gemini.google.com/) [7] "A Comprehensive Guide to Function Calling in LLMs," The New Stack. [Online]. Available: [https://thenewstack.io/a-comprehensive-guide-to-function-calling-in-llms/](https://thenewstack.io/a-comprehensive-guide-to-function-calling-in-llms/) ```quiz Q: What are the four decisions that define any LLM backend integration? - Which provider, streamed or not, structured or freeform, function calling or not - Which IDE to use, which database, which frontend framework, which CDN correct: 0 explain: Every LLM integration is one composition of four choices: provider (OpenAI/Anthropic/Gemini), streaming (whole or token-by-token), output shape (structured JSON or freeform text), and whether callable functions are exposed. The IDE/database/CDN choices are orthogonal. Q: Why is streaming usually preferred for user-facing chat, and unstreamed for internal calls? - Streaming sends tokens as generated, so the user sees the answer grow in real time, improving perceived latency; internal calls have no human watching, so the simplicity of one response wins - Streaming is always faster end-to-end than unstreamed correct: 0 explain: Streaming doesn't make the total generation faster, but it improves perceived latency because the first token appears quickly. For human-facing chat that perception matters; for backend-to-backend calls where no one watches, the implementation simplicity of one whole response wins. Q: What problem does structured output solve, and what does it NOT solve? - It forces the response to conform to a declared JSON schema, making it reliably parseable; it does NOT eliminate hallucinated values — shape is constrained, contents can still be wrong - It eliminates all hallucination; the output is guaranteed factually correct correct: 0 explain: Structured output constrains the shape (the JSON validates against the schema), which makes parsing reliable. But the values inside can still be hallucinated. You still validate the contents; structured output only solved the parsing fragility, not factual accuracy. Q: In function calling, who decides when to call a function and who actually runs it? - The model decides when to call and which arguments; your code actually executes the function and returns the result - Your code decides when to call; the model runs the function correct: 0 explain: The model emits a structured function-call request when it decides one is needed, with arguments matching the schema. Your code executes the real function (deterministic) and feeds the result back to the model. The boundary between probabilistic (model) and deterministic (your function) is exactly where it should be. Q: What is the main benefit of function calling over letting the model emit raw code or natural-language commands? - Structure — a parseable function-call request is more reliable than freeform code or text, reducing mistakes - It is faster than generating text correct: 0 explain: A function call is a structured request your code parses and executes deterministically. Compared to the model writing raw code (which might be wrong) or natural-language commands (which need interpretation), the structured call is reliably parseable and the deterministic function returns ground truth. Fewer mistakes, clearer boundary. ```