technical
Three Ways to Make Go Handlers Idempotent
Retries are unavoidable, so handlers must be safe to run twice. Three techniques in Go: safe writes, a processed-messages table, and idempotency keys.
You’ve probably seen this message on a checkout page: “Please don’t refresh or press back, or you may be charged twice.” The people who built that page know the same request can reach their server twice, and they’re counting on you to prevent it.
Compare that with an elevator button. Press it once and an elevator is on its way. Press it five more times and nothing changes: one elevator is still coming. Nobody asks you to press it only once, because pressing it again is harmless.
That property is called idempotency: doing something twice has the same effect as doing it once.
Most duplicates don’t come from impatient people, though. They come from software doing the right thing. A mobile app sends a payment, the connection drops before the response arrives, and the app retries, because a timeout says nothing about whether the payment went through. A webhook provider resends an event because your server answered too slowly. A queue redelivers a message because the consumer crashed before confirming it. Retrying is correct in all of these cases, so the receiver has to behave like the elevator button.
Here are three ways to get there in Go, from the simplest to the most involved. The examples use Postgres with pgx and need Go 1.25 for sync.WaitGroup.Go in the test. Other databases with transactions and unique constraints can do the same, but check how yours handles two inserts of the same key at once.
1. Make the Write Safe to Repeat
Some writes are already safe to repeat. Setting a value is safe. Adding to it is not:
-- Safe: running this twice leaves the same result.
UPDATE orders SET status = 'shipped' WHERE id = $1;
-- Not safe: running this twice adds 20 twice.
UPDATE accounts SET balance = balance + 20 WHERE id = $1;When a write isn’t safe, you can often make it safe by only changing the row if it’s still in the state you expect:
func (s *Store) MarkOrderPaid(ctx context.Context, orderID string) (bool, error) {
tag, err := s.db.Exec(ctx,
`UPDATE orders SET status = 'paid', paid_at = now()
WHERE id = $1 AND status = 'pending'`,
orderID,
)
if err != nil {
return false, err
}
// Zero rows means the order was already paid (or doesn't exist).
return tag.RowsAffected() == 1, nil
}A second call finds nothing to update. The return value tells the caller which case it hit, for example to avoid sending a second receipt. It does lump “already paid” together with “cancelled” and “missing”, so if the caller needs to tell those apart, read the row back.
The state check does one more thing. Safe to repeat is not the same as safe to reorder: a delayed 'shipped' update that arrives after 'delivered' would move the order backwards. Naming the expected state in the WHERE clause stops that too.
For inserts, INSERT ... ON CONFLICT DO NOTHING does the same job: it adds the row only if no row with that key exists yet, and otherwise does nothing and reports zero rows affected. The next two sections build on it.
This costs almost nothing, so it’s the first thing I check. It only works when there’s a row to find and a state to check against, though. Creating a shipment for an order that was just paid has neither: there’s no shipment row yet.
2. Record What You’ve Already Processed
Messages and webhooks usually carry a unique ID that stays the same across retries. You can store the IDs you’ve handled in a table with a unique key:
CREATE TABLE processed_messages (
consumer TEXT NOT NULL,
message_id TEXT NOT NULL,
processed_at TIMESTAMPTZ NOT NULL,
PRIMARY KEY (consumer, message_id)
);The consumer column lets each service track the same message separately, so the shipping service doesn’t skip a message just because the email service already handled it.
The record and the work must go into the same transaction, so they either both happen or neither does:
func (h *ShippingHandler) HandleOrderPaid(ctx context.Context, msg Message) error {
tx, err := h.db.Begin(ctx)
if err != nil {
return err
}
defer tx.Rollback(ctx)
tag, err := tx.Exec(ctx,
`INSERT INTO processed_messages (consumer, message_id, processed_at)
VALUES ('shipping', $1, now())
ON CONFLICT (consumer, message_id) DO NOTHING`,
msg.ID,
)
if err != nil {
return err
}
if tag.RowsAffected() == 0 {
return nil // already processed
}
if err := createShipment(ctx, tx, msg.OrderID); err != nil {
return err // rollback also removes the record, so a retry can try again
}
return tx.Commit(ctx)
}This also works when two copies arrive at the same moment. Postgres makes the second INSERT wait, because it can’t tell whether the key is taken until the first transaction commits or rolls back. If the first commits, the second insert affects zero rows and returns early. If the first rolls back, the second does the work.
That’s the behaviour at Postgres’s default isolation level, READ COMMITTED. Under REPEATABLE READ or SERIALIZABLE the second insert still waits, but then fails with a serialization error (SQLSTATE 40001) instead of returning zero rows. If you run at those levels, retry the whole transaction on that error.
The ID has to identify the operation, not the delivery. If the producer can retry its own publish, or the broker gives you no stable ID, key on something from the domain instead: one shipment per order, with a unique constraint on the order ID in the shipments table. That is technique 1 again, and it’s stronger than a message ID.
The table will grow forever unless you clean it up. Keep each record for as long as a duplicate could still show up, such as your queue’s retention period or the webhook provider’s retry window, then delete older rows on a schedule.
3. Use Idempotency Keys for APIs
The mobile payment from the opening is harder. A plain HTTP request has no message ID, and the client needs more than “already done”: it needs the original response, such as the ID of the payment that was created.
The usual solution, popularized by Stripe, is an idempotency key. The client generates a key for each operation and sends it in a header:
POST /payments HTTP/1.1
Idempotency-Key: 9b2f6c1e-7d4a-4f0b-a3c8-5e1d2f7a9b04The server stores the key with the response, and returns the stored response when the same key comes back:
CREATE TABLE idempotency_keys (
user_id TEXT NOT NULL,
key TEXT NOT NULL,
request_hash BYTEA NOT NULL,
response_status INT,
response_body BYTEA,
created_at TIMESTAMPTZ NOT NULL,
PRIMARY KEY (user_id, key)
);Keys are scoped to the user, so one user can’t fetch another user’s stored response by guessing or reusing that user’s key. request_hash is a hash of the method, the path and the parsed request body. It catches a client reusing a key for a different request; more on that below.
The handler follows the same pattern as before, plus storing the response:
func (s *Server) createPaymentOnce(
ctx context.Context,
userID, key string,
requestHash []byte,
req PaymentRequest,
) (int, []byte, error) {
tx, err := s.db.Begin(ctx)
if err != nil {
return 0, nil, err
}
defer tx.Rollback(ctx)
tag, err := tx.Exec(ctx,
`INSERT INTO idempotency_keys (user_id, key, request_hash, created_at)
VALUES ($1, $2, $3, now())
ON CONFLICT (user_id, key) DO NOTHING`,
userID, key, requestHash,
)
if err != nil {
return 0, nil, err
}
if tag.RowsAffected() == 0 {
// We've seen this key before, so replay the stored response.
// status is never NULL here: the insert and the update below commit together.
var storedHash, body []byte
var status int
err := tx.QueryRow(ctx,
`SELECT request_hash, response_status, response_body
FROM idempotency_keys WHERE user_id = $1 AND key = $2`,
userID, key,
).Scan(&storedHash, &status, &body)
if err != nil {
return 0, nil, err
}
if !bytes.Equal(storedHash, requestHash) {
return 0, nil, errKeyReused // the HTTP handler maps this to 400, as Stripe does
}
return status, body, nil
}
payment, err := insertPayment(ctx, tx, userID, req)
if err != nil {
return 0, nil, err
}
body, err := json.Marshal(payment)
if err != nil {
return 0, nil, err
}
_, err = tx.Exec(ctx,
`UPDATE idempotency_keys SET response_status = $3, response_body = $4
WHERE user_id = $1 AND key = $2`,
userID, key, http.StatusCreated, body,
)
if err != nil {
return 0, nil, err
}
if err := tx.Commit(ctx); err != nil {
return 0, nil, err
}
return http.StatusCreated, body, nil
}insertPayment only writes to our own database. Nothing inside this transaction talks to another service, and the next section explains why that matters.
The request hash catches a client reusing a key for a different request, like abc for a $20 payment and then abc again for $50. Returning the $20 response would hide that bug, so the server rejects it instead. Hash the parsed struct, not the raw bytes, or a client that re-encodes its JSON on retry gets rejected on the one request you wanted to accept. Include the method and path too, or abc on /payments followed by abc on /refunds replays the payment.
Concurrent duplicates work the same way as in the previous section. A second request with the same key waits on the INSERT until the first transaction finishes, then replays the stored response. A failed attempt leaves nothing behind: if insertPayment fails, the rollback removes the key too, and the client’s retry gets a fresh attempt instead of a stored error.
Waiting is simple, but each waiting duplicate holds a database connection for as long as the first attempt runs. If first attempts can be slow, set a short lock_timeout on the insert and answer 409 “request in progress” instead, as Stripe does.
Like the processed-messages table, this one needs pruning. Stripe prunes keys after at least 24 hours. Pick a window longer than your own clients’ longest retry. A mobile app with an offline queue can retry days later, and a retry that arrives after its key was pruned looks exactly like a new request.
When the Work Leaves Your Database
A database transaction can’t cover a call to a card processor or an email provider. That’s why the handler above only writes to its own database. Put the processor call inside that transaction and the “failed attempt leaves nothing behind” rule turns against you: the charge goes through, the client times out and disconnects, the request context is cancelled, the transaction rolls back, the key disappears, and the retry charges again.
If the external service accepts idempotency keys, pass one along, derived from your own key, so your retries are safe on their side too.
If it doesn’t, no option is fully safe, so pick the failure you can live with. To never do the work twice, commit a record in its own transaction before making the call, with a status that says the call is in flight. Then make the call, and store the result in a second transaction. A crash in between leaves a record for work that may never have happened, and the retry sees it and stops. To never skip the work, make the call first and record it after, and a crash in between does the work twice. For a payment, not charging is better than charging twice. For a welcome email, a rare duplicate is fine.
With an in-flight record, a replay can find a row that has no response yet. Answer “still in progress” with a 409, as Stripe does, rather than treating the empty response as an error.
Mistakes That Look Correct
Checking first, then acting.
if alreadyProcessed(ctx, msg.ID) {
return nil
}
process(ctx, msg)
markProcessed(ctx, msg.ID)Two copies arriving together can both pass the check, and both do the work. Reading first and writing second leaves a gap between the two. The unique key inside the transaction has no gap, which is why the earlier sections insert first and treat the insert itself as the check. A SELECT ... FOR UPDATE on the parent row would also close the gap, but a plain SELECT never will.
There’s a second bug in that snippet: markProcessed runs outside any transaction, so a crash right after process does the work again on redelivery.
Generating a new key on every retry. The key identifies the operation, not the attempt:
// Wrong: every attempt looks like a new payment.
for range 3 {
req := newPaymentRequest(body)
req.Header.Set("Idempotency-Key", uuid.NewString())
// ...
}
// Right: one key, reused by every attempt.
key := uuid.NewString()
for range 3 {
req := newPaymentRequest(body) // a fresh request each time; the body is consumed after one send
req.Header.Set("Idempotency-Key", key)
// ...
}Deduplicating by content. Two genuine $20 payments to the same person look identical. Only the caller knows whether a request is a retry, so the caller has to say so with a key.
Testing It
Send the same request several times at once, check that the work happened once, and check that every caller got the same answer:
func TestCreatePaymentIsIdempotent(t *testing.T) {
srv := newTestServer(t) // a real Postgres, with a pool of at least two connections
req := PaymentRequest{To: "bob", AmountCents: 2000}
hash := hashRequest(t, req)
const attempts = 10
results := make([]string, attempts)
var wg sync.WaitGroup
for i := range attempts {
wg.Go(func() {
status, body, err := srv.createPaymentOnce(t.Context(), "alice", "key-123", hash, req)
if err != nil {
t.Errorf("createPaymentOnce: %v", err)
return
}
results[i] = fmt.Sprintf("%d %s", status, body)
})
}
wg.Wait()
if n := countPayments(t, srv.db); n != 1 {
t.Fatalf("expected 1 payment, got %d", n)
}
for _, r := range results[1:] {
if r != results[0] {
t.Fatalf("responses differ: %q vs %q", results[0], r)
}
}
}Ten goroutines don’t guarantee an overlap, so this test is probabilistic. Run it a few times with -count, and keep the boring sequential version too: one call, then a second, same response. The same shape works for the processed-messages handler.
Use a real database, for example with testcontainers. All of this depends on how the database handles concurrent transactions, and a mock won’t tell you that. A pool with a single connection would serialize the goroutines and let the check-then-act bug pass every time.
Picking One
Use the simplest option that fits. Make the write safe to repeat if you can. Use a processed-messages table for messages and webhooks. Save idempotency keys for APIs where clients need the original response back.
And in code review, ask: “What happens if this runs twice?” Sooner or later, it will.