What is Idempotency? — Safe Retries in Go APIs and Services

Definition

An operation is idempotent if executing it multiple times produces the same result as executing it once. The term comes from mathematics, where an idempotent operation is one where f(f(x)) = f(x). In distributed systems, idempotency is a correctness property that makes retries safe.

The need for idempotency arises from the fundamental uncertainty of distributed communication. When a client sends a request and doesn’t receive a response, it has no way to know whether the server received the request and processed it, received it and failed before responding, or never received it at all. The correct response is to retry — but if the operation isn’t idempotent, retrying creates duplicates: a payment charged twice, an order created twice, an email sent twice.

Idempotency keys solve this. The client generates a unique key for each logical operation and sends it with every retry of the same operation. The server records the key with the result on first processing. Subsequent requests with the same key return the cached result without re-executing the operation.

How It Works

Client → POST /orders  Idempotency-Key: k_abc123  {items: [...]}
Server:
  1. Check: has k_abc123 been processed before?
  2. No → process order → store result with k_abc123 → return 201
  3. Yes → return stored result → return 201 (same response, no duplicate)

Client (retry after timeout) → POST /orders  Idempotency-Key: k_abc123  {items: [...]}
Server: k_abc123 exists → return stored result → 201 (no new order created)

The idempotency key storage needs to be durable (survives restarts) and consistent (no race conditions on concurrent requests with the same key). Redis is a common choice for its atomic SET NX operation.

In Go

verikt’s idempotency capability scaffolds middleware that extracts the key, checks storage, and caches responses:

// Middleware checks and caches
func IdempotencyMiddleware(store IdempotencyStore) func(http.Handler) http.Handler {
    return func(next http.Handler) http.Handler {
        return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
            key := r.Header.Get("Idempotency-Key")
            if key == "" {
                next.ServeHTTP(w, r)
                return
            }
            if cached, ok := store.Get(r.Context(), key); ok {
                w.WriteHeader(cached.StatusCode)
                w.Write(cached.Body)
                return
            }
            // capture response and store with key
        })
    }
}

How verikt Supports It

The idempotency capability requires http-api and suggests redis for the key store. verikt warns when retry is used without idempotency — retrying non-idempotent operations (payments, order creation) causes duplicates. See the Capabilities Matrix.

Saga Pattern

Saga compensation steps are retried on failure — idempotency is essential for safe retry behavior. Saga Pattern

Outbox Pattern

Outbox provides at-least-once delivery — consumers need idempotency to handle duplicate events. Outbox Pattern

Circuit Breaker

When the circuit reopens and retries resume, idempotency prevents duplicate side effects. Circuit Breaker