08 — Queues: Asynchronous Message Processing at the Edge

"Async Workers, somehow" was my Queues summary, and it didn't explain when or why to reach for one. The model that did: a queue is a buffer between a producer and a consumer that decouples them — it absorbs traffic spikes, retries failures, and paces work to whatever rate the consumer (or the backend behind it) can handle. [1] That decoupling is the entire point, and it's the answer to a class of problems that synchronous request/response can't solve.

The framing that landed is the failure mode it removes. When a Worker handles everything synchronously, two things break under load: a sudden spike of expensive work (image processing, email sending) ties up the request and either times out or burns the CPU budget; and a downstream dependency that's briefly unavailable fails the whole request, even though the work itself was valid. A queue fixes both. The producer enqueues a message in milliseconds and returns; the consumer processes at its own pace; a failed message retries instead of failing the user's request [1][2].

Message processing: producer, consumer, at-least-once

The mechanics are straightforward. A producer (usually a Worker) sends messages to a queue with send or sendBatch. A consumer (another Worker, bound to the queue) receives batches of messages and processes them [2][3]. The delivery guarantee is at-least-once: a message will be delivered to the consumer at least once, even if there are failures. The consumer acknowledges successful processing to remove a message from the queue; if it crashes or throws before acknowledging, the message is redelivered.

That "at-least-once" wording is the part I had to take seriously, because it has a consequence: consumers must be idempotent. If the same message can arrive twice, processing it twice must not corrupt anything. A payment Worker that charges a card on each message is broken under at-least-once delivery; one that checks "has this message ID been processed yet?" before charging is correct. Designing for redelivery is the discipline that makes queues safe.

Background jobs

The canonical use case is moving expensive work out of the request/response cycle [4]. A user uploads an image; the upload Worker stores the raw file and enqueues a "process image" message; the user gets an immediate "uploaded, processing" response; a separate consumer Worker processes the image (resize, optimize, store derivatives) at its own pace. The user doesn't wait for the processing, the upload Worker stays within its CPU budget, and a spike of uploads becomes a deeper queue rather than a timeout storm.

Any task that's slow, bursty, or non-critical-to-the-current-request is a candidate: email sending, PDF generation, data export, webhook fanout, thumbnail creation. The decision rule: if the work doesn't need to block the response, it probably shouldn't.

Rate limiting

The less obvious use is using a queue to pace work [5]. Instead of hammering a downstream API or a backend directly, a producer enqueues every request; the consumer pulls from the queue at a controlled rate — say, 50 messages per second — matching the downstream's capacity. The queue absorbs the spikes; the downstream sees a steady, survivable load.

This is rate limiting as architecture rather than as rejection. A traditional rate limiter rejects requests over the limit; a queue-based one absorbs them and lets them through at the limit. The choice depends on whether the work needs to happen eventually (queue it) or right now (reject if over).

Dead letter queues

Not every message can be processed. A message whose payload is malformed, or that consistently throws in the consumer, would block the queue forever if there were no escape valve. That's the dead letter queue (DLQ) [6].

The pattern: after a configurable number of failed delivery attempts, a message is moved to a separate DLQ instead of being retried indefinitely. The main queue keeps flowing; the poison messages are quarantined for investigation. I then inspect the DLQ to find the underlying bug, fix the consumer, and reprocess or discard the messages deliberately.

The DLQ turns "a single broken message breaks the whole pipeline" into "a single broken message sits in a side queue waiting for me to look at it." That's the difference between a queue that survives production and one that doesn't.

How I use this

The shape I've settled on: any work that's slow, bursty, or downstream-of-something-fragile goes through a queue. The producer is thin (validate, enqueue, acknowledge), the consumer is idempotent (designed for at-least-once from day one), and every queue has a DLQ configured from the start — not as an afterthought. The single habit that's saved me the most pain is writing the consumer as if every message will arrive twice, because eventually one will.

References

[1] Cloudflare, "Cloudflare Queues," Cloudflare Docs, 2024. [Online]. Available: https://developers.cloudflare.com/queues/

[2] Cloudflare, "How does Queues work?," Cloudflare Docs, 2024. [Online]. Available: https://developers.cloudflare.com/queues/reference/how-queues-works/

[3] StackOverflow, "Difference between stream processing and message processing," community discussion. [Online]. Available: https://stackoverflow.com/questions/41744506/difference-between-stream-processing-and-message-processing

[4] Dispatched, "Background jobs for serverless applications," 2024. [Online]. Available: https://dispatched.dev/

[5] Cloudflare, "Cloudflare Queues — Queues & rate limits," Cloudflare Docs. [Online]. Available: https://developers.cloudflare.com/queues/tutorials/handle-rate-limits/

[6] AWS, "What is a dead-letter queue?," AWS Docs. [Online]. Available: https://aws.amazon.com/what-is/dead-letter-queue/