Error Handling That Survives Contact With Production

There is a particular kind of log line that shows up during every incident:

ERROR: request failed

It is technically correct. It contains no information. Somebody wrote it while thinking about the happy path, and it has been quietly failing to help ever since.

Good error handling is not about catching more exceptions. It is about making sure that when something goes wrong, the person reading about it can act.

An error message has exactly one job

It has to let a reader decide what to do next. That decision needs three things:

1. What was being attempted - the operation, with its identifying inputs.
2. What happened instead - the underlying failure, not a paraphrase.
3. What is now true - did the operation partially apply? Is a retry safe?

Compare:

raise RuntimeError("Failed to sync")

with:

raise SyncError(
    f"Failed to sync account {account_id} at cursor {cursor}: "
    f"upstream returned {response.status}. No records were written; "
    f"retry is safe from this cursor."
) from exc

The second is longer, and the length is the point. Every clause answers a question the reader would otherwise have to answer by reading source code during an outage.

If an error message does not tell the reader whether it is safe to retry, they will guess. Half of them will guess wrong.

Preserve the cause, always

The most expensive habit in error handling is catching an exception, logging it, and raising a new one that discards the original.

# Loses the stack trace and the original type.
except HTTPError as exc:
    raise SyncError("upstream failed")

# Keeps both. `from` is not decoration.
except HTTPError as exc:
    raise SyncError(f"upstream failed for account {account_id}") from exc

Every mainstream language has the equivalent - from in Python, %w in Go’s fmt.Errorf, cause in JavaScript’s Error options, initCause in Java. Using it costs nothing and preserves the only reliable link between the symptom and the line that caused it.

Distinguish the three kinds of failure

Almost every failure falls into one of three categories, and they want opposite handling.

The common mistake is handling all three the same way - usually by catching broadly, logging at ERROR and continuing. That turns an unrecoverable config problem into a slow trickle of alerts nobody reads, and an unexpected bug into corrupted state.

A useful heuristic: catch narrowly, at the level that can actually do something about it. If the function catching the exception cannot choose a different course of action, it should not be catching.

Retries need a boundary and a budget

Retries are the most common way a small failure becomes an outage. Three rules keep them safe:

• Only retry what is idempotent. A retried POST that already succeeded creates duplicates. If you cannot make the operation idempotent, use an idempotency key so the server can.
• Bound the total, not the attempts. “Three attempts” behaves very differently against a 30-second timeout than a 300 ms one. Budget in wall-clock time.
• Add jitter. Synchronised retries across a fleet reproduce exactly the load spike that caused the failure.

def with_retries(fn, *, budget_s=10.0, base=0.2):
    deadline = time.monotonic() + budget_s
    attempt = 0

    while True:
        try:
            return fn()
        except TransientError:
            attempt += 1
            delay = min(base * 2**attempt, 2.0) * (0.5 + random.random())
            if time.monotonic() + delay > deadline:
                raise
            time.sleep(delay)

Note that TransientError is a specific type. Retrying every exception retries the bugs too, which turns a deterministic crash into a slow one.

Log once, at the boundary

A failure that is logged at every level of the stack produces five entries for one event, all with different wording, and makes the error rate metric meaningless.

Pick a boundary - the request handler, the job runner, the CLI entry point - and log there, once, with the full chain. Everywhere below it, add context to the error and re-raise.

except Exception as exc:
    raise SyncError(f"while processing batch {batch_id}") from exc

The result is a single log entry containing the whole causal chain: what the user asked for, which batch, which account, which upstream call, and the original socket error at the bottom. That entry is a diagnosis. Five entries are a search problem.

The test that is worth writing

Assert on your error paths the way you assert on your success paths. Not just that an exception is raised - that the message contains the identifiers someone would need.

def test_sync_error_identifies_the_account():
    with pytest.raises(SyncError) as err:
        sync(account_id="acc_123", cursor="c_9")

    assert "acc_123" in str(err.value)
    assert "c_9" in str(err.value)
    assert err.value.__cause__ is not None

It looks like a trivial test. It is the reason that, in six months, the incident takes four minutes instead of forty.