Task queues and background jobs

A background job is any code that runs outside the request/response lifecycle. That's the whole definition. The interesting question is which code belongs there, and the answer is: anything the client doesn't need an answer about right now.

The canonical example is signup. User posts their email and password, your handler validates it, hashes the password, writes a row, generates a verification code, and then calls Resend to send the email. That last step is a network call to a machine you don't own.

POST /signup  ->  validate  ->  INSERT user  ->  POST api.resend.com  ->  201

Everything before the fourth arrow is under your control and takes single-digit milliseconds. The fourth arrow is a stranger's uptime, and your response is behind it.

What synchronous costs you when it breaks

Resend goes down for ninety seconds. Two things can happen, and both are bad.

If you didn't wrap the email call, the exception propagates and POST /signup returns a 500. The user's account was already written to the database, but as far as they can tell signup failed. They'll retry and hit a unique constraint on the email column.

If you did wrap it, signup returns 201, the frontend renders "check your inbox", and no email exists. Now the user waits, finds nothing, clicks resend, which is a fresh request against the same dead service. You are asking a human to poll a third-party outage by hand.

The problem in both cases isn't error handling. It's that a task with no deadline was placed on the critical path of a task with one.

The queue version

Same handler, one change at the end. Instead of calling the email provider, serialize everything the email needs — recipient, template name, verification code, user's first name — into JSON and push it onto a queue. Then return 201.

{
  "task": "send_verification_email",
  "to": "user@example.com",
  "template": "verify",
  "code": "839217"
}

The handler is now entirely made of things you control. Response time is your database write plus a queue push.

On the other side, a consumer — a separate process, possibly a separate deployment — watches that queue, dequeues the task, deserializes the JSON back into a native dict or struct, and calls the handler you registered for send_verification_email. In practice this happens milliseconds after the enqueue. Verification codes are valid for fifteen minutes; nobody notices a two-second gap.

The producer/broker/consumer split is the entire architecture. Your application code is the producer and only ever enqueues. The broker — Redis, RabbitMQ, SQS — stores tasks until someone is ready. The consumer dequeues and executes. A to-do list for your backend, with workers picking items off it.

Celery in Python, BullMQ in Node, asynq in Go. Different names, same three boxes.

Failure becomes a retry instead of a 500

Now Resend is down and the consumer's handler throws. Nothing user-facing happens, because there is no user attached to this execution anymore. The task goes back in the queue with a delay, typically exponential backoff: retry after 1 minute, then 2, then 4, then 8, up to a configured max attempts.

A real outage at a real email provider lasts seconds. Your task fails once, maybe twice, and succeeds on the third attempt. The email arrives late and the user never knew there was an incident.

The mechanism underneath is acknowledgement. When a consumer finishes a task it acks the broker, and the broker deletes it. If no ack arrives within the visibility timeout — because the worker crashed, or the process was OOM-killed mid-execution — the broker makes the task visible again for another consumer to pick up. Without that, every crashed worker would silently swallow whatever it was holding.

What actually gets offloaded

Four families cover most of it:

One-off: triggered by a request. Verification emails, password resets, push notifications. Note that push notifications are the same shape as email — your backend can't reach a phone directly, it calls APNs or FCM with a stored device token, so it's another external API call and another thing to keep off the request path.

Recurring: cron-shaped. Weekly PDF reports at Sunday midnight. Deleting session rows that haven't been touched in three months.

Chained: parent/child dependencies. A user uploads a video to an LMS. Encode it to multiple resolutions first. Only after that succeeds do thumbnail generation and audio transcription start — those two don't depend on each other, so they run in parallel, and thumbnail resizing hangs off thumbnail generation.

Batch: one task fanning out into many. Delete-account is the one to think about. A user's data is spread across projects, assets, ownership rows, maybe several shards. Deleting it inline takes forty seconds you can't hold a connection open for. So: enqueue delete_account, return 200, log them out. The worker fans out into per-resource deletions and the user gets a confirmation email at the end.

Design constraints that bite

Idempotency, first and by a wide margin. Retries mean every task will eventually run more than once, so a task must be safe to execute repeatedly. If delete_account deleted half a user's projects and then hit an error, the retry starts from zero — and it had better find a consistent state to start from. Wrap the mutations in a transaction so a failure rolls back to the state the retry expects.

Small and focused. One task, one unit of work. If a task does five things and step four fails, steps one through three run again on every retry, burning CPU on work that already succeeded. Split it and chain it — then a child failing is a child failing, not the whole pipeline.

Rate limiting. Your workers scale horizontally, which is the point, but the external service they hammer has its own quota and its own bill. Twenty consumers pulling from an email queue can exceed a provider's rate limit far more efficiently than one synchronous handler ever could.

Monitoring. You need queue depth, success and failure counts, and failure reasons — external service vs. internal error — as metrics, with alerts on queue length and worker health. A background job that dies quietly is worse than an endpoint that returns 500, because the 500 at least has someone watching it.