15 — LLM Integration Patterns — Providers, Streaming, Structured Output, and Function Calling

Providers, streaming, structured outputs, function calling — each looked like a separate API to learn until I noticed the same four questions behind every integration. The composition: every LLM integration is four decisions, in sequence — which provider, streamed or not, structured or freeform, callable tools or not. [1] Pick one of each and the pipeline assembles itself.

The frame that helped is that the providers all expose the same underlying capability (next-token prediction, from the earlier fundamentals post) with different surface APIs. The differences that matter for backend work aren't the model internals — they're the integration shape. Does the response arrive whole, or token-by-token as it generates? Does the model return freeform text, or JSON conforming to a schema I declared? Can the model decide to call a function I exposed, or only emit text? Those four choices define the engineering of any LLM feature.

The providers: OpenAI, Anthropic, Gemini

Three providers dominate the backend LLM landscape, and they're more similar than different at the API level [2][5][6]:

• OpenAI — the suite of models (GPT family) accessed via a hosted API, with text generation, function calling, structured output, embeddings, and image/multimodal capabilities. OpenAI's API set the de facto shape that others imitate; the SDK ergonomics are widely cloned.
• Anthropic — the AI safety company behind Claude, accessed via API with similar capabilities (text generation, tool use, structured output). Anthropic's differentiators are a focus on safety and steerability (Constitutional AI), long context windows, and the Claude Code agentic CLI ecosystem.
• Gemini (Google) — a family of multimodal models handling text, code, images, audio, and video, accessed via Google's API. Gemini's pitch is native multimodality and integration with Google's ecosystem.

The practical observation: the provider choice is usually decided by model quality for the specific task, pricing, and ecosystem alignment (which SDK, which agent framework, which features matter), not by dramatic API differences. A text-generation call looks structurally similar across all three. The patterns below — streaming, structured output, function calling — exist in all three, with slightly different names.

Streamed or unstreamed: who waits

The first decision is whether the response arrives whole (the model finishes generating, then returns the complete text) or streamed (tokens are sent to the client as they're generated) [3].

• Unstreamed is simpler — one request, one response, easy to cache, easy to log. The cost is that the user waits for the whole answer before seeing anything, which feels slow for long outputs.
• Streamed sends tokens as they're produced, so the user sees the answer grow in real time. The cost is implementation complexity (Server-Sent Events or chunked HTTP, partial JSON parsing, harder logging).

The trade-off is UX versus simplicity. For chat-like features where the answer is several paragraphs, streaming materially changes perceived speed — the first token arriving in 500ms feels faster than a complete answer in 5 seconds, even though the total time is similar. For backend-to-backend calls where a human isn't watching, unstreamed is usually right. The portfolio pattern is streamed for user-facing chat, unstreamed for internal data extraction.

Structured output: forcing parseable shape

The second decision is whether the response is freeform text or structured — JSON conforming to a schema I declare [4]. This is the single most important pattern for backend LLM work, because freeform text is fragile to consume.

The problem with freeform: the model might return "Yes, the user is over 18" or "The user's age category is adult: true" or "``\ntrue\n``" — all valid answers, none parseable without heuristics. Structured output fixes this by constraining the model to emit JSON matching a schema I provide:

{
  "type": "object",
  "properties": {
    "is_over_18": { "type": "boolean" },
    "age_category": { "type": "string", "enum": ["minor", "adult", "senior"] }
  },
  "required": ["is_over_18", "age_category"]
}

The provider's structured-output mode (OpenAI, Anthropic, Gemini all support variants) constrains decoding so the output validates against the schema. The payoff is that the model's output becomes reliable program input — I parse it with the same JSON parsing I'd use for any API. The failure mode to know: structured output reduces (not eliminates) hallucination; the schema constrains shape, but the values can still be wrong. Validation of the contents is still on me.

Function calling: the model decides to act

The third decision is whether I expose callable functions (tools) to the model [7]. Without function calling, the model can only emit text. With it, I declare a set of functions (name, description, argument schema), and the model can decide — mid-conversation — to "call" one by emitting a structured request instead of plain text.

The flow:

1. I send the prompt plus a list of available functions with their schemas.
2. The model either responds in text, or emits a function call: { "name": "get_order_status", "arguments": { "order_id": "42" } }.
3. My code executes the real function (query the database, call an API), gets the result.
4. I feed the result back to the model, which uses it to continue the response.

The model decides when to call a function and which arguments to pass; my code actually runs it. This is the foundation of tool-using agents — function calling is the primitive that lets a model reach outside its training data and act on real state. It's also where structured output and function calling compose: the function-call arguments are themselves structured (a JSON object matching the function's schema), so they're reliably parseable.

The payoff over the model writing raw code or natural-language commands: structure means fewer mistakes. The model emits a parseable request; my code does the deterministic work; the model reads the result. The boundary between probabilistic (the model) and deterministic (the function) is exactly where it should be.

Composing the four choices

The integration patterns emerge from composing these four decisions. A few common shapes:

• Chat feature → any provider, streamed, freeform (or lightly structured for safety), function calling for tools (search the docs, look up the order). This is the agent pattern.
• Classification/extraction pipeline → any provider, unstreamed, structured output (JSON schema), no function calling. Pure data transformation.
• Autonomous coding agent → Anthropic/OpenAI, streamed for UX, structured for tool calls, function calling heavily (read file, run tests, edit). This is Claude Code and its peers.

The same four decisions describe every LLM feature I've built or used. The provider is the least interesting choice; the other three define the engineering.

How I use this

Three rules capture the practical takeaway:

• Default to structured output. Freeform text is fragile; JSON matching a schema is reliable. I reach for structured output for any backend LLM call where the result feeds code.
• Stream only when a human is watching. Streaming adds complexity; I reserve it for user-facing chat where perceived latency matters. Internal calls stay unstreamed.
• Use function calling when the model needs real data. If the answer depends on something the model can't know (current inventory, user-specific state), expose a function. The model decides when to call; my code returns the truth.

The framing — provider, streamed, structured, callable — is the design language of LLM integration. Every feature I ship is one composition of these four, and naming them turns "build an AI feature" from an open-ended research problem into a sequence of engineering choices.

References

[1] "Emerging Patterns in Building GenAI Products," Martin Fowler, 2024. [Online]. Available: https://martinfowler.com/articles/gen-ai-patterns/

[2] OpenAI, "OpenAI Platform." [Online]. Available: https://platform.openai.com/docs/overview

[3] "Streaming Responses in AI: How AI Outputs Are Generated in Real Time," dev.to. [Online]. Available: https://dev.to/pranshu_kabra_fe98a73547a/streaming-responses-in-ai-how-ai-outputs-are-generated-in-real-time-18kb

[4] OpenAI, "Structured model outputs." [Online]. Available: https://platform.openai.com/docs/guides/structured-outputs

[5] Anthropic, "Anthropic." [Online]. Available: https://www.anthropic.com/

[6] Google, "Google Gemini." [Online]. Available: https://gemini.google.com/

[7] "A Comprehensive Guide to Function Calling in LLMs," The New Stack. [Online]. Available: https://thenewstack.io/a-comprehensive-guide-to-function-calling-in-llms/