Error taxonomy, retries, and DLQ

Audience: Anyone configuring worker retries. Goal: Retry what heals; fail fast on poison; never infinite loops.

TL;DR

Contents

1. Taxonomy
2. Retry policy
3. Timeouts
4. Dead-letter queues
5. Pseudocode
6. Alerting
7. Anti-patterns

---

1. Taxonomy

Encode classes as exceptions or error codes your middleware understands.

2. Retry policy

Recommended defaults (tune with data):

max_attempts: 5
backoff: exponential
base: 1s
cap: 10m
jitter: full or equal jitter

# PSEUDOCODE
delay = min(cap, base * 2 ** attempt) * random(0.5, 1.5)

Do not retry non-idempotent permanent side effects without a guard.

3. Timeouts

Every external call needs a timeout:

• HTTP client: connect + read
• DB statement timeout
• Soft/hard task time limits (Celery/Dramatiq)

A task without timeouts becomes a stuck consumer (prefetch blocked).

4. Dead-letter queues

RabbitMQ

• Queue with DLX → jobs.dlq
• Reject/nack without requeue after max attempts
• Redrive tool for operators

Kafka

• Error topic billing.events.errors
• Or stop-the-world + alert on poison (document choice)

5. Pseudocode

# PSEUDOCODE
async def handle(job_id: str):
 try:
 await run(job_id)
 await mark_succeeded(job_id)
 except PermanentError as e:
 await mark_failed(job_id, str(e))
 # ack / no retry
 except TransientError as e:
 attempts = await bump(job_id, str(e))
 if attempts >= max_attempts:
 await mark_failed(job_id, str(e))
 await publish_dlq(job_id)
 return
 raise Retry(delay=backoff(attempts))
 except Exception as e:
 attempts = await bump(job_id, "unknown:" + str(e))
 if attempts >= max_attempts:
 await mark_failed(job_id, str(e))
 await publish_dlq(job_id)
 await alert("poison_or_bug", job_id)
 return
 raise Retry(delay=backoff(attempts))

6. Alerting

7. Anti-patterns

• except Exception: retry forever
• No jitter (thundering herd)
• Retrying after non-idempotent charge without guard
• Silent drop of poison messages
• Using DLQ as a black hole with no runbook

Related: 03 Lifecycle · 10 Testing