Making long-running jobs survive failures

A job that takes a few seconds rarely fails in interesting ways. A job that runs for twenty minutes will, sooner or later, be interrupted by a deploy, a timeout or a partner API having a bad day. The goal is not to prevent failure. It is to make failure cheap.

Run it on a queue

Long work should never run inside a web request. Put it on a queue, process it in a worker, and let the user check the status. That alone removes timeouts and lets you scale workers separately from the web app.

Retry, but with a strategy

Some failures are temporary: a network blip, a rate limit, a locked row. For those, retry. Two rules keep retries from making things worse:

• Back off. Wait longer between each attempt instead of hammering a struggling service.
• Limit attempts. After a few tries, stop and surface the failure so someone can look at it.

Do not retry errors that will never succeed, such as invalid input. Fail those immediately with a clear message.

Make jobs safe to run twice

Retries and restarts mean a job may run more than once. Design each step to be idempotent: running it again produces the same result, not a duplicate invoice or a second email. A unique key per operation is often enough.

Control concurrency

Two workers handling the same record, or fifty jobs hitting the same third-party API at once, create their own failures. Use locks or per-resource limits so only as many jobs run together as the system can handle.

Make it restartable

For long jobs, work in small batches and record progress as you go. If the job stops at item 4,000 out of 10,000, a restart should continue from there, not start over. This turns a failure from a lost afternoon into a short delay.

Watch it

Log what each job does, track failures and durations, and alert on stuck queues. A reliable system is one where you hear about problems before your users do.

If your product has long-running processes that need to be dependable, this is work I enjoy, and I would be happy to help.