How do you handle asynchronism, message queues and backpressure at scale?

Step 1: Use Cases, Constraints, Assumptions, Back-of-Envelope

Use cases

We'll scope to handle only the core flows that expose the bottlenecks; everything else is deferred.

• User triggers async work — POST returns 202, job enqueued (email, thumbnail, analytics), polled or webhook on done.
• Service consumes — workers pull with prefetch, ack on success, DLQ on poison, retry with backoff.
• Service handles backpressure — bounded queues, shed/slow producers, scale consumers on lag.
• Service guarantees — at-least-once vs exactly-once via idempotency keys and deduplication.

Out of scope

• Specific broker tuning (Kafka partition math beyond overview), exactly-once transactions deep dive, stream processing (Flink) beyond queue.

State assumptions

• Queue depth 1M messages, each 1 KB → 1 GB buffered; throughput 10k msg/s.
• Consumer lag target 1000.
• Backpress