Make the retry safe before you make it

Ask one question before every retry: if this already happened, does doing it again change anything?

• No → retry freely.
• Yes → do not retry until you have made the answer no.

Everything below is ways to make the answer no.

Prefer operations that are already idempotent

Design the operation so repetition is a no-op. This is cheaper than any bookkeeping.

The pattern behind the right column: declare the end state, not the transition. "The file should contain exactly this" survives being run five times. "Add this line" does not.

The config case is the one that bites hardest in practice:

# WRONG — a retry duplicates the line, and the duplicate may be harmless
# for months before something reads only the first occurrence
echo "timeout = 60" >> /srv/app/config.ini

# Also wrong — grep-then-append is check-then-act; two runs can interleave
# between the grep and the append and both append
grep -q 'timeout' /srv/app/config.ini || echo "timeout = 60" >> /srv/app/config.ini

# Right — regenerate the whole file, write it atomically
render_config > "$tmp" && mv -f "$tmp" /srv/app/config.ini

See skills/atomic-file-writes for why mv and not >.

When the operation must append: an idempotency key

Some things genuinely create. Then you supply an identifier the server can deduplicate on:

POST /api/orders
Idempotency-Key: 7f3a-order-2026-09-05-batch-12
Content-Type: application/json

{"item": "widget", "qty": 3}

Four rules, and each one is a real bug when broken:

1. Generate the key once, before the first attempt. A key generated inside the retry loop is a new key each time, which is the same as having none. This is the most common way idempotency keys are implemented wrong.
2. Derive it from the operation, not from the clock. hash(operation + run_id) is reproducible if your process dies and restarts with the same inputs. uuid() at send time is not.
3. Persist it with the intent, before sending. Write "I am about to do X with key K" somewhere durable first. Then a crash mid-flight leaves you a note to reconcile against — see skills/partial-failure.
4. Check the server actually honours it. Many do not. A 200 on the second send proves nothing; look for whether two records exist. Test it on a scratch resource before you rely on it. See skills/verifying-a-claim.

Conditional writes are idempotency for free

If the API supports If-Match / baseHash, you already have it. Your first attempt succeeds and changes the version. A duplicate retry carries the old version and is refused with 409 / 412 — which is precisely the outcome you wanted, delivered by the server, with no keys to manage.

So a 409 on a retry after an ambiguous timeout is good news: it usually means the first attempt landed. Treat it as "verify, then stop", not as an error to work around. See skills/optimistic-concurrency.

Check-then-act is not idempotent

if ! service_exists svc; then create_service svc; fi

Between the check and the act, anything may happen — including the other copy of you. This is the same shape as the grep -q || echo >> line above and the same shape as read-modify-write. The fixes are the same three: make the act itself idempotent, make it conditional on a version, or hold a lock across both.

Retrying reads

Reads are not automatically safe either. A retried read is harmless to the server but not to you: two reads of a moving target give you two different bases, and if you merge work from both you produce a state that never existed. Read once, keep the version, and use it.

A checklist for any retry loop

[ ] Is the operation idempotent, or keyed, or conditional?  (if none: don't retry)
[ ] Is the key generated OUTSIDE the loop?
[ ] Is there a cap on attempts AND on total elapsed time?
[ ] Does the loop distinguish "retryable" from "your request is wrong"?
[ ] On give-up, does it say what state the world is in, not just "failed"?

That last line matters more than the rest. "Failed after 3 attempts" sends the next reader to start from nothing. "Failed after 3 attempts; key K was sent at least twice; reconcile by reading /api/orders?key=K" hands them the thread.

See also skills/rate-limits-and-backoff for when to retry, and skills/partial-failure for the case where you do not know whether to.

skills/index