Message Queues (Deep Workflow)#

Queues decouple producers and consumers—and introduce duplicates, ordering surprises, poison messages, and lag. Make delivery semantics and failure handling explicit.

When to Offer This Workflow#

Trigger conditions:

• Choosing Kafka vs RabbitMQ vs SQS vs Pub/Sub
• Consumer lag, DLQ growth, reprocessing needs
• “Exactly-once” expectations—need alignment on reality

Initial offer:

Use six stages: (1) delivery semantics, (2) topology & partitions, (3) message contract, (4) consumers & retries, (5) ops & scaling, (6) failure drills). Confirm cloud and ordering requirements.

---

Stage 1: Delivery Semantics#

Goal: Choose at-most-once, at-least-once, or effective-once via idempotency.

Questions#

1. Can duplicate processing break invariants?
2. Is ordering global, per-entity, or unnecessary?
3. Latency vs durability trade-offs

Exit condition: One paragraph per pipeline stating semantics.

---

Stage 2: Topology & Partitions#

Goal: Throughput and ordering align—ordering only within partition when using Kafka-style partitions.

Practices#

• Partition key often equals business key (e.g., user id)
• Watch hot partitions

---

Stage 3: Message Contract#

Goal: Versioned events or commands with schema registry.

Practices#

• Envelope: id, type, version, timestamp
• Payload size limits; reference blobs by id

---

Stage 4: Consumers & Retries#

Goal: Exponential backoff + jitter; DLQ with reason; replay tooling owned.

Pitfalls#

• Retries can reorder unless single-threaded per partition

---

Stage 5: Ops & Scaling#

Goal: Lag metrics, consumer offset health, rebalance awareness (Kafka).

---

Stage 6: Failure Drills#

Goal: Kill consumer mid-batch; duplicate publish intentionally; validate idempotency.

---

Final Review Checklist#

• Delivery semantics and idempotency explicit
• Partitioning/ordering strategy documented
• Versioned message contract
• Retry, DLQ, replay documented
• Lag metrics and alerts; capacity plan

Tips for Effective Guidance#

• Exactly-once end-to-end is rare—design for at-least-once + idempotent handlers.
• Challenge global ordering requirements—they cost scale.
• Visibility timeouts (SQS) differ by product—read the vendor docs.

Handling Deviations#

• Transactional outbox when you need DB + queue consistency without dual writes.