Designing Reliable Background Jobs

Background jobs fail. Networks blip, dependencies rate-limit you, and processes get killed mid-flight. This post collects the patterns I reach for to make that failure boring instead of catastrophic.

🔗Retries with backoff

Naïve retries stampede. Add exponential backoff with full jitter so retries spread out instead of synchronising into a thundering herd.

Here’s the whole helper, included straight from the source file that lives next to this post — no copy-paste drift:

import time
import random


def with_retries(fn, *, attempts=5, base=0.2, cap=10.0):
    """Call fn, retrying with exponential backoff and full jitter."""
    for attempt in range(1, attempts + 1):
        try:
            return fn()
        except TransientError:
            if attempt == attempts:
                raise
            backoff = min(cap, base * 2 ** (attempt - 1))
            time.sleep(random.uniform(0, backoff))  # full jitter

A fenced block, for comparison, gets full token colouring:

result = with_retries(lambda: charge_card(order), attempts=5)

🔗Idempotency

Every job must be safe to run twice, because at-least-once delivery means it will run twice eventually.

• Assign each job a stable idempotency key
• Make writes upserts where possible
• Add a dead-letter queue for poison messages

🔗The shape of it

Treat the queue as the source of truth, not the worker’s memory. If a worker dies, another must be able to pick up exactly where it left off.¹

That’s the core. Retry with jitter, stay idempotent, and make every step observable enough to debug at 3am.