--- title: "10 — Workers AI: Inference at the Edge" uid: workers-ai tags: ["workers-ai", "text-generation", "llm", "image", "cloudflare", "inference", "roadmap:cloudflare", "speech"] excerpt: "Workers AI runs inference on Cloudflare's network — a Worker calls a model through a binding, with the computation happening near the user, not in a distant GPU region." date: 2026-08-13T03:28:17+0000 source: https://www.aveshina.my.id/en/blog/workers-ai --- "AI on the edge, somehow" was my Workers AI summary, and the "somehow" was doing all the work. The model that pinned it: **Workers AI runs model inference — asking an already-trained model to produce an answer, rather than training it — on Cloudflare's global network, so a Worker invokes a model through a binding the same way it does any other Cloudflare service — the inference happens near the user, not in a distant GPU region.** [1] That placement is the whole pitch, and it shapes everything about how it feels to use. The framing that landed is the contrast with the default AI integration pattern. The conventional approach: a Worker receives a request, calls a third-party AI API over the public internet, waits for a faraway GPU farm to run the inference, and returns the result. The latency is dominated by the network round trip and the queue at the provider; the cost is per-call to an external service. Workers AI inverts the geography: the inference runs on GPUs at Cloudflare's edge, in-network, with the model exposed as a binding [1][2]. The Worker doesn't leave the network to get a prediction. ```figure ``` ## Text generation The headline capability is text generation — large language models running at the edge [3]. A Worker calls a model through the binding: ``` const result = await env.AI.run("@cf/meta/llama-3-8b-instruct", { prompt: "Summarize this article in one sentence: " + article, }); ``` The shape is the same as any other binding call. The model is addressed by a versioned identifier, the input is a prompt (or a chat-style messages array), and the output is the generated text. The model catalog is curated — Cloudflare picks and hosts a set of open models, so I'm choosing from a list rather than bringing my own weights. The performance characteristic worth internalizing: because the model runs at the edge, the time-to-first-token is dominated by the inference itself, not the network. For streaming responses, this means the first tokens reach the user quickly, which is what makes an LLM-powered feature feel responsive rather than laggy. ## Choosing a text generation model The reference points at a real decision point: which model [3]. The catalog has several, with different trade-offs: - **Size versus latency.** Larger models are more capable but slower per token. An 8B-parameter model is fast; a 70B model is smarter but the per-token latency is higher. - **Task fit.** Instruction-tuned models (the -instruct suffix) are tuned for following directions. Base models are tuned for completion. Chat models expect a messages array. - **Licensing.** The catalog is open-weight models, which means the licensing terms of the underlying model apply — relevant if usage is commercial or high-volume. The guidance I use: start with a mid-size instruct model for prototyping (it's the most general), measure the latency for my actual prompts, and move up or down the size curve based on whether the quality or the speed is the bottleneck. ## Image processing Workers AI also runs vision models — image classification, object detection, and image generation [4]. The binding shape is the same; the input is image bytes (or a text prompt for generation) and the output is predictions or a generated image. The use cases at the edge are the ones that benefit from running near the user: real-time moderation of user uploads, on-the-fly tagging, generating thumbnails or variants. ## Speech recognition Speech-to-text — the Whisper family — is available as a binding [5]. Audio in, transcription out. The edge-native characteristic matters most for real-time use cases: live captioning, voice commands, transcription of short clips where the user is waiting. The pattern is the same as text generation: stream the audio to the model binding, get text back. ## AI model integration The unifying idea across all the modalities is **integration shape consistency** [6]. Every model — text, image, speech, embeddings — is called the same way: env.AI.run(modelId, input). The mental load is low because there's one API surface, and swapping models is a one-line change to the identifier. That consistency is what makes it reasonable to prototype with one model and ship with another. ## Where Workers AI fits (and where it doesn't) The honest framing is what Workers AI is _not_. It's not a hosted version of every model — the catalog is curated, focused on open weights. It's not fine-tuning of arbitrary large proprietary models (the GPT-4-class models live elsewhere, and Cloudflare's AI Gateway, covered separately, is the layer for routing to those). It's not training — only inference. Where it fits: - **Latency-sensitive AI features.** Anything user-facing where the round trip to a distant API would dominate the experience. - **Open-model workloads.** When the model I need is in the catalog (Llama, Mistral, Whisper, etc.), Workers AI is the lowest-friction place to run it. - **Edge-native composition.** When the inference is one step in a Worker that also touches KV, D1, R2 — keeping it all in-network avoids the egress and the round trips. Where it doesn't: when I need a specific proprietary model that isn't in the catalog, or when the workload is training rather than inference. Those go through AI Gateway to external providers. ## How I use this The pattern I keep: prototype the feature against a mid-size instruct model from the catalog, measure the actual end-to-end latency (not just the model's reported latency), and treat the model identifier as a swap point — easy to change as the catalog evolves. For anything user-facing where responsiveness matters, the edge placement of Workers AI is the actual feature; for batch or background work where latency is less critical, I weigh it against the cost characteristics of alternatives rather than assuming it's always the right call. ## References [1] Cloudflare, "Workers AI — Cloudflare Docs," Cloudflare Docs, 2024. [Online]. Available: [https://developers.cloudflare.com/workers-ai/](https://developers.cloudflare.com/workers-ai/) [2] Cloudflare, "Models · Cloudflare Workers AI," Cloudflare Docs, 2024. [Online]. Available: [https://developers.cloudflare.com/workers-ai/models/](https://developers.cloudflare.com/workers-ai/models/) [3] Cloudflare, "Choose the right text generation model," Cloudflare Workers AI Tutorials, 2024. [Online]. Available: [https://developers.cloudflare.com/workers-ai/tutorials/how-to-choose-the-right-text-generation-model/](https://developers.cloudflare.com/workers-ai/tutorials/how-to-choose-the-right-text-generation-model/) [4] Cloudflare, "How to build an image generator using Workers AI," Cloudflare Workers AI Tutorials. [Online]. Available: [https://developers.cloudflare.com/workers-ai/tutorials/image-generation-playground/](https://developers.cloudflare.com/workers-ai/tutorials/image-generation-playground/) [5] Cloudflare, "Whisper — Cloudflare Workers AI," Cloudflare Docs. [Online]. Available: [https://developers.cloudflare.com/workers-ai/models/whisper/](https://developers.cloudflare.com/workers-ai/models/whisper/) [6] Cloudflare, "Cloudflare + AI," ai.cloudflare.com, 2024. [Online]. Available: [https://ai.cloudflare.com/](https://ai.cloudflare.com/) ```quiz Q: How does a Worker invoke a model in Workers AI? - Over the public internet to a third-party API endpoint - Through a binding — env.AI.run(modelId, input) — with inference running on edge GPUs in-network - By importing the model weights as an npm package correct: 1 explain: Models are exposed as bindings, the same shape as KV, R2, or D1. The call stays in-network and the inference runs on GPUs at Cloudflare's edge. Q: What dominates latency for a streaming text-generation call on Workers AI? - The network round trip to a distant GPU farm - The inference itself — the network is local, so time-to-first-token is inference-bound - The cost of the per-call charge correct: 1 explain: Because inference runs at the edge near the user, the network is not the bottleneck. Time-to-first-token is dominated by the model's inference speed, which is why streaming feels responsive. Q: You need a specific large proprietary model (GPT-4-class) that isn't in the Workers AI catalog. What's the right approach? - Wait — Cloudflare will eventually add it to the catalog - Route to the external provider through AI Gateway, since Workers AI only hosts curated open-weight models - Fine-tune an equivalent open model inside Workers AI correct: 1 explain: Workers AI is inference for a curated set of open models, not every model. For proprietary models, AI Gateway (covered separately) is the routing layer to external providers. Q: A Worker calls an image-classification model and a text-generation model. What's true of the two calls? - They use completely different API surfaces - They use the same shape — env.AI.run(modelId, input) — only the model identifier and input differ - Image models must be called via a REST endpoint, text via binding correct: 1 explain: Integration shape consistency is the unifying idea. Every modality — text, image, speech, embeddings — uses the same binding call, which makes swapping models a one-line change. Q: Which workload is the BEST fit for Workers AI? - Training a new foundation model from scratch - A user-facing feature that classifies an uploaded image with low latency - Running a proprietary model hosted only in one distant region correct: 1 explain: Latency-sensitive, user-facing inference on catalog-available models is Workers AI's sweet spot. Training isn't supported (only inference), and proprietary distant-region models go through AI Gateway. ```