10 — Workers AI: Inference at the Edge

"AI on the edge, somehow" was my Workers AI summary, and the "somehow" was doing all the work. The model that pinned it: Workers AI runs model inference — asking an already-trained model to produce an answer, rather than training it — on Cloudflare's global network, so a Worker invokes a model through a binding the same way it does any other Cloudflare service — the inference happens near the user, not in a distant GPU region. [1] That placement is the whole pitch, and it shapes everything about how it feels to use.

The framing that landed is the contrast with the default AI integration pattern. The conventional approach: a Worker receives a request, calls a third-party AI API over the public internet, waits for a faraway GPU farm to run the inference, and returns the result. The latency is dominated by the network round trip and the queue at the provider; the cost is per-call to an external service. Workers AI inverts the geography: the inference runs on GPUs at Cloudflare's edge, in-network, with the model exposed as a binding [1][2]. The Worker doesn't leave the network to get a prediction.

Text generation

The headline capability is text generation — large language models running at the edge [3]. A Worker calls a model through the binding:

const result = await env.AI.run("@cf/meta/llama-3-8b-instruct", {
  prompt: "Summarize this article in one sentence: " + article,
});

The shape is the same as any other binding call. The model is addressed by a versioned identifier, the input is a prompt (or a chat-style messages array), and the output is the generated text. The model catalog is curated — Cloudflare picks and hosts a set of open models, so I'm choosing from a list rather than bringing my own weights.

The performance characteristic worth internalizing: because the model runs at the edge, the time-to-first-token is dominated by the inference itself, not the network. For streaming responses, this means the first tokens reach the user quickly, which is what makes an LLM-powered feature feel responsive rather than laggy.

Choosing a text generation model

The reference points at a real decision point: which model [3]. The catalog has several, with different trade-offs:

• Size versus latency. Larger models are more capable but slower per token. An 8B-parameter model is fast; a 70B model is smarter but the per-token latency is higher.
• Task fit. Instruction-tuned models (the -instruct suffix) are tuned for following directions. Base models are tuned for completion. Chat models expect a messages array.
• Licensing. The catalog is open-weight models, which means the licensing terms of the underlying model apply — relevant if usage is commercial or high-volume.

The guidance I use: start with a mid-size instruct model for prototyping (it's the most general), measure the latency for my actual prompts, and move up or down the size curve based on whether the quality or the speed is the bottleneck.

Image processing

Workers AI also runs vision models — image classification, object detection, and image generation [4]. The binding shape is the same; the input is image bytes (or a text prompt for generation) and the output is predictions or a generated image. The use cases at the edge are the ones that benefit from running near the user: real-time moderation of user uploads, on-the-fly tagging, generating thumbnails or variants.

Speech recognition

Speech-to-text — the Whisper family — is available as a binding [5]. Audio in, transcription out. The edge-native characteristic matters most for real-time use cases: live captioning, voice commands, transcription of short clips where the user is waiting. The pattern is the same as text generation: stream the audio to the model binding, get text back.

AI model integration

The unifying idea across all the modalities is integration shape consistency [6]. Every model — text, image, speech, embeddings — is called the same way: env.AI.run(modelId, input). The mental load is low because there's one API surface, and swapping models is a one-line change to the identifier. That consistency is what makes it reasonable to prototype with one model and ship with another.

Where Workers AI fits (and where it doesn't)

The honest framing is what Workers AI is _not_. It's not a hosted version of every model — the catalog is curated, focused on open weights. It's not fine-tuning of arbitrary large proprietary models (the GPT-4-class models live elsewhere, and Cloudflare's AI Gateway, covered separately, is the layer for routing to those). It's not training — only inference.

Where it fits:

• Latency-sensitive AI features. Anything user-facing where the round trip to a distant API would dominate the experience.
• Open-model workloads. When the model I need is in the catalog (Llama, Mistral, Whisper, etc.), Workers AI is the lowest-friction place to run it.
• Edge-native composition. When the inference is one step in a Worker that also touches KV, D1, R2 — keeping it all in-network avoids the egress and the round trips.

Where it doesn't: when I need a specific proprietary model that isn't in the catalog, or when the workload is training rather than inference. Those go through AI Gateway to external providers.

How I use this

The pattern I keep: prototype the feature against a mid-size instruct model from the catalog, measure the actual end-to-end latency (not just the model's reported latency), and treat the model identifier as a swap point — easy to change as the catalog evolves. For anything user-facing where responsiveness matters, the edge placement of Workers AI is the actual feature; for batch or background work where latency is less critical, I weigh it against the cost characteristics of alternatives rather than assuming it's always the right call.

References

[1] Cloudflare, "Workers AI — Cloudflare Docs," Cloudflare Docs, 2024. [Online]. Available: https://developers.cloudflare.com/workers-ai/

[2] Cloudflare, "Models · Cloudflare Workers AI," Cloudflare Docs, 2024. [Online]. Available: https://developers.cloudflare.com/workers-ai/models/

[3] Cloudflare, "Choose the right text generation model," Cloudflare Workers AI Tutorials, 2024. [Online]. Available: https://developers.cloudflare.com/workers-ai/tutorials/how-to-choose-the-right-text-generation-model/

[4] Cloudflare, "How to build an image generator using Workers AI," Cloudflare Workers AI Tutorials. [Online]. Available: https://developers.cloudflare.com/workers-ai/tutorials/image-generation-playground/

[5] Cloudflare, "Whisper — Cloudflare Workers AI," Cloudflare Docs. [Online]. Available: https://developers.cloudflare.com/workers-ai/models/whisper/

[6] Cloudflare, "Cloudflare + AI," ai.cloudflare.com, 2024. [Online]. Available: https://ai.cloudflare.com/