Sidekiq Retries and Idempotent Jobs for Ecommerce: Stop Sending Duplicate Orders
Sidekiq is a wonderful tool, and I use it with Redis on most of the Rails projects I work on. It also makes a promise that surprises people the first time they read it carefully. Your job will run at least once. It does not promise exactly once.
For a job that resizes an image, that difference does not matter. For a job that creates an order in your warehouse system, charges a card or emails a customer, it matters a great deal. This article is about writing background jobs for ecommerce so that running them twice is harmless.
How a job can run more than once
There are several ordinary ways it happens:
- The job fails and Sidekiq retries it. By default it retries many times, with growing delays between attempts, spread over a few weeks. That is helpful for temporary errors, and dangerous if a failure happens after the job already did something.
- The process is killed halfway. A deploy, a memory limit or a server restart can stop a worker in the middle of a job. Sidekiq sees an unfinished job and runs it again.
- The same event is enqueued twice. A webhook delivered twice, a double click, or a reconcile task that overlaps with the real event, can all put duplicate jobs in the queue.
None of these are bugs in Sidekiq. They are the price of not losing work, which is the better trade for most businesses.
The dangerous moment
Look at a typical job: fetch the order, call the warehouse API to create a shipment, save the warehouse’s ID on the order. Suppose the API call succeeds and then your server crashes before the save. The retry runs from the top and creates a second shipment.
The risky window is always the gap between an outside action and your record of it. You cannot close that gap completely, but you can make the repeat safe.
Pass identifiers, not objects
Give jobs small, simple arguments, such as an order ID. Do not pass a whole order as a hash that might be stale by the time the job runs. When the job starts, load the current record and decide what to do based on its present state. This one habit prevents a surprising number of bugs, because a job that waits in the queue for ten minutes should act on today’s data, not the data from when it was enqueued.
Check before you act
The first line of most of my jobs reads the record and asks whether the work is already done. If the order already has a warehouse ID, stop. If the email is already marked as sent, stop. Returning quietly from a job counts as success, and it is the cheapest form of idempotency you can have.
For this to work, store the outcome of every outside action somewhere you can check it. A column such as the warehouse shipment ID, a timestamp for the sent email, or a status field with clear stages all do the job.
Use the other system’s idempotency features
Many APIs let you make a request safely repeatable. Stripe supports idempotency keys, and ShipStation lets you update an order that has the same order key instead of creating another. When you can, derive the key from your own stable data, such as your order ID, so a retry sends the same key. I describe this for payments in Handling Stripe Webhooks in Rails.
When the other system has no such feature, search for the record first, using a reference you wrote into it earlier. Look up the shipment by your order number before creating a new one. It is an extra call, and it is usually worth it.
Use the database to enforce it
Code checks can race against each other. Two workers can both see “not sent yet” at the same instant. A unique index in the database settles the question, because only one of them can insert the row. A common pattern is a small table that records “this action for this order” with a unique constraint, and a job that must insert into it before doing the work. The loser of the race gets an error, treats it as “someone else has it”, and exits.
Decide which errors deserve a retry
Not every failure should be retried. Sort them into two groups:
- Temporary problems, such as timeouts, rate limit responses and server errors. Raise the error and let Sidekiq retry later.
- Permanent problems, such as a missing SKU or an address the carrier rejects. Retrying a hundred times will not fix them. Catch the error, record a clear reason on the order, notify a human, and finish the job without raising.
Mixing these up creates two classic problems. Permanent errors that raise fill your retry set with noise, and temporary errors that are swallowed lose orders quietly.
You can also tune retries per job. A job that talks to a flaky partner might get a longer, gentler backoff. A job that sends a time sensitive text message might get only a couple of attempts, because a message that arrives three days late is worse than none.
Respect rate limits
If your jobs call APIs with limits, put the limit in one place instead of hoping the retries sort it out. Limit how many workers handle that queue, or use a rate limiter, and honor the wait time an API gives you in a 429 response. My notes on this are in Shopify API Rate Limits During Black Friday.
Separate queues for separate priorities also help. Order creation should never sit behind a pile of catalog updates.
Watch the dead set
When a job runs out of retries, Sidekiq moves it to the dead set, where it waits for a person. Many teams never look there. Make sure a human does:
- Add a handler that fires when retries are exhausted and sends an alert with the job name and the order ID.
- Review the dead set on a schedule, and replay or discard each job on purpose.
- Track the count over time. A slowly growing number means something systematic is wrong.
A job that dies quietly is a lost order. The goal is that someone knows within hours, not weeks.
A short checklist
Before you ship a job that touches money, stock or customer email, ask:
- Can I run this twice and get the same result?
- What state do I record after each outside action, and do I check it first?
- Is there a unique index protecting the dangerous step?
- Which errors are temporary, and which are permanent?
- Who gets told when retries run out?
If you can answer all five, you are in better shape than most production systems I look at. For a wider look at keeping integrations calm under pressure, my notes on order syncs that survive retries pair well with this article.
If you have a Rails app where background jobs occasionally double up or vanish, email kevin@kevinhq.com with a description of the job and what it touches. These problems are very fixable once you can see them clearly.