Error handling and building fault tolerant systems

The worst error I have shipped did not crash anything. A discount was applied twice on a subset of orders, shipping came out negative on a handful of them, and the endpoint returned 200 every single time. No alert fired because nothing was broken in the sense that monitoring understands broken. It ran for weeks.

That is the category worth starting with, because it sets the shape of everything else. Errors are not a single thing you catch at the top of a handler. They differ in how they are detected, whether you control them, and what you can do once they happen.

The five kinds, ranked by how much they hurt

Logic errors are first because they are the ones that cost money. The code runs, the response is 200, the result is wrong. They come from misread requirements, an algorithm implemented slightly off, or an edge case nobody thought about. Nothing in your stack will tell you about them. You find out from a user, or from a business metric moving in the wrong direction, or not at all.

Database errors are next and split three ways. Connection errors — pool exhausted, network down, server overloaded — take your whole app with them, because the backend has nothing to serve without the data. Constraint violations are different: unique email, foreign key pointing at an author that does not exist. Some of these are validation-layer failures you should have caught earlier; some, like uniqueness, are structurally impossible to catch earlier, because only the database knows whether that email is taken. Those are not bugs, they are outcomes, and your job is to translate them. Then query errors: malformed SQL, a typo'd table name, a query that times out, or a deadlock where two transactions wait on each other in a circle.

External service errors are the ones you do not control at all. Payment processors, email providers, S3, Redis, Clerk or Auth0 for auth. You cannot stop depending on them — rebuilding auth from scratch is not the cheaper option — so every one is a failure point you own the consequences of but not the fix. Network timeouts and DNS failures, credential and token expiry, rate limits (a 429 because your own traffic spiked or your own retry loop went feral), and the plain outage where a cloud region falls over and half of Twitter notices at the same time.

Input validation errors are the easiest kind you will ever handle, and it is worth saying why: you already know the rules. Format, range, required fields — all of it is knowable at the entry point, before anything downstream is touched. This is the only error category where the correct answer is completely determined in advance.

Configuration errors show up when you move between dev, staging, and production. You added OPENAI_API_KEY to your local .env, the PR merged, and nobody added it to the production environment. Two things can happen next, and only one of them is acceptable.

Fail at boot, not at 3am

If you validate required config before the server starts, a missing key kills the process immediately. With blue-green deployment, the old version is still running, the new one never takes traffic, and the operator gets a message naming the variable. Nothing user-facing happens at all.

If you do not, the deploy succeeds. The key is read lazily, inside the handler, the first time someone actually hits the feature. That could be four hours later. Now it is a 500 in production, on a live user, in a code path nobody is watching.

startup:
  for each required config key:
    if missing or malformed: log the name, exit non-zero

Crashing at boot is the good outcome. Design for it.

The general form: the best error handling happens before the error. Health checks that only return 200 when the process is alive tell you almost nothing. Check that the database answers a representative query, and how long it took — a query that went from 500ms to 4s is a failure that has not happened yet.

Recoverable vs. not

Once an error is in front of you, the response depends on one question: can this succeed later, unchanged?

Sending an email can. It is not on a millisecond budget, so retry with exponential backoff — wait 1 minute, then 2, then 4. Same for a temporarily exhausted connection pool. The rule to keep in mind is that your retry logic is running against a system that is already under stress. Retrying hard is how you turn a blip into an outage.

A hard outage cannot. There is no delay that fixes an object store being down. The strategy there is containment and graceful degradation: serve stale cache, disable the non-essential feature, fall back to in-memory caching if Redis is gone, and keep the paths that actually matter — payments, order processing — alive. You are shrinking the blast radius, not fixing anything.

The separation matters because applying the wrong one is worse than doing nothing. Retrying an unrecoverable error is just a slower failure with more load attached.

One place to handle all of it

Layers in a typical backend: routing picks a handler, the handler validates and binds, the service orchestrates, repository methods each run one query. Errors can arise at any of them.

Do not handle them where they arise. Bubble them up — throw in JavaScript or Python, return the error up the chain in Go — and catch everything in one global error handler in middleware, where you have access to the request and the response.

Take a book-management API. POST /books with a 700-character name fails validation in the handler. POST /books with a duplicate name fails at the repository with a unique constraint violation. GET /books/123 for a book that does not exist fails at the repository with no rows. POST /books with a nonexistent author_id fails with a foreign key violation. Four different failures, four different layers, one place that decides what the user sees:

validation error        → 400, field-level messages
unique violation        → 400, "book already exists"
no rows                 → 404, "book 123 not found"
foreign key violation   → 404, "author does not exist"
anything unrecognised   → 500, "something went wrong"

Two things fall out of this. You stop forgetting conditions — if unique-violation handling lives in each repository method, the one method where you forgot it returns a 500 saying nothing instead of a useful 400. And you stop duplicating the same five branches after every query in the codebase.

What the error message is allowed to say

That final 500 branch is a security boundary. If it forwards whatever bubbled up, you are handing users your table names, index names, and constraint names — the exact reconnaissance an attacker wants before trying anything more targeted. The 500 case means you failed to recognise the error, which means you do not know what is in the string. Return a fixed message.

The same logic applies to login. If a wrong email returns "no user with that email" and a wrong password returns "incorrect password", you have built an account enumeration oracle: spray emails with one password, collect the ones that change their answer, then attack only those with a password list. Return invalid email or password for both branches. The OWASP cheat sheets cover the rest of this ground.

And logs count as output. Companies leak logs — they are hundreds of gigabytes a day flowing through several third-party storage and analysis services, and those get breached. So log the user ID and a correlation ID, never the email, password, API key, or card number. Anything you write on an authentication failure is one vendor incident away from being public.