Idempotency in Workflow Automation: Why Retries Duplicate Your Side Effects

When a workflow retries, it does not know what it already did. The API call to Stripe or SendGrid or your CRM succeeded, the job crashed before the success was recorded, and the retry sends the same request again. In practice this means duplicate refunds, duplicate confirmation emails, duplicate invoices, and occasionally duplicate database rows that no one notices until reconciliation. The fix is not cleverer retry logic. It is making every side-effecting step idempotent, so that running it twice produces the same result as running it once.

This is one of the most common failure modes in n8n, Zapier, Make and any custom pipeline that touches external APIs, and it gets worse once AI agents are added, because agents branch unpredictably and re-call tools based on context they have already seen. The sources below are a useful map of what experienced builders actually do, rather than what the marketing pages suggest.

What the duplicate side effect problem actually looks like

A typical workflow looks like this: a trigger fires, a node calls an external API, the API returns 200 OK, the next node is supposed to record the success and move on, and the workflow crashes somewhere between step two and three. The retry framework sees no record of completion, so it runs the whole sequence again. From the perspective of Stripe, this is a brand new refund request, not a replay. From the perspective of the customer, it is two emails and two refunds.

Practitioners in the Hacker News thread on duplicate side effects converge on a blunt diagnosis: if the upstream service offers neither idempotency keys nor a way to retrieve an existing resource by a client-supplied identifier, the client cannot fully solve the problem. You can reduce the frequency, but you cannot eliminate it. That single sentence reframes the whole exercise. Idempotency is a contract with the external system, and where the contract does not exist you have to compensate.

The three patterns builders actually use

From the thread and from broader practice, three patterns cover most realistic cases:

• Client-generated idempotency keys at the API boundary. Every side-effecting call gets a UUID minted at the workflow start, passed as an idempotency key (where the API supports it), and stored alongside the workflow execution. Stripe, and a growing list of payment and messaging APIs, deduplicate on the key for a window of typically 24 hours. The TTL matters: too short and retries outside the window still duplicate, too long and your key namespace grows unbounded.

• An outbox pattern. Instead of calling the external service directly from the workflow, the workflow writes its intent to a local outbox table in the same database transaction as the state change. A separate process polls the outbox and delivers. CDC tools like Debezium make this clean. The benefit is that the workflow crash and the external call are no longer in the same atomic unit: either both happen, or neither does, and the poller can retry the delivery forever without re-running your business logic.

• A pre-flight lookup against the external system. Before issuing the call, query the external service to see whether the operation already exists. This is the simplest pattern, and the one several commenters argued should be the default, because duplicate side effects are rare relative to total traffic and the latency cost of an extra GET is usually acceptable. It works best when the external system exposes a stable queryable identifier for the operation you are about to perform.

Where AI agents change the calculation

Deterministic workflows are hard enough to keep idempotent. AI agents are worse, because the agent's decision to call a tool can depend on a summary of what it has already done, and that summary can drift. A common pattern in agent frameworks is to attach a unique operation key to every tool call, generated at the start of a logical task and persisted outside the model context. If the agent calls the same tool twice with the same key, the receiving system dedupes. If the agent loops and re-reads the same file, an outbox-style log can tell it the work is already done.

This is where the Statewright project is interesting, although it sits slightly outside the workflow tool category. Statewright frames agents as state machines rather than as free-form tool users, which means each phase of the agent's work has a constrained tool set. Write tools are not available in a read-only planning phase, destructive shell calls are blocked even when Bash is broadly allowed, and tool calls outside the current phase are rejected at execution time rather than reasoned away in a prompt. For an automation builder, the relevant lesson is not the specific tool but the broader principle: the most reliable way to prevent an agent from re-running an expensive side effect is to make the side effect unavailable in the states where it would be a mistake.

A practical comparison of the three patterns

None of the three patterns is universally best. They trade off complexity, latency, blast radius and dependency on the external system.

PatternWhere it fitsMain costIdempotency keysAPIs with native support (Stripe, most payment and email providers)Key TTL management, key storage, unsupported APIsOutbox + CDCMulti-step workflows that already touch a databaseExtra infrastructure (Debezium, a poller, schema changes)Pre-flight lookupLow-volume side effects, legacy APIs without idempotencyExtra API call per workflow, race window between check and act

How to apply this inside n8n, Zapier or Make

In n8n specifically, the cleanest path is to generate an idempotency key in a Code or Set node at the start of the workflow, write it to a database or KV store alongside the execution id, and pass it to the side-effecting HTTP node as a header. If the workflow retries from a failure point before the success was recorded, the same key is reused and the external system dedupes. Zapier and Make have less control over execution replay, so for high-value side effects the outbox pattern is often worth the extra setup: write the intent to a table in Airtable, Postgres or Notion, then have a separate short zap or scenario poll the table and make the API call with its own retry policy.

Two rules of thumb from practitioners in the thread are worth lifting out:

1. Treat every job execution as replayable from the moment you design it, not after the first duplicate. Adding idempotency later is a migration across every workflow that touches money, email or external state.

2. Keep the dedup window long enough to cover your worst-case retry chain, including manual reruns. Twenty-four hours is a common default, but multi-day chains are not unusual when humans are in the loop.

Where AutoStack fits

If you are buying ready-made automations rather than building them, idempotency is the kind of detail you want to see explicitly in the workflow description and the documentation. Templates on AutoStack are listed by creators who have shipped the workflow in production, and the better ones call out whether side effects are protected by idempotency keys, outbox writes or pre-flight checks. Ask before you buy: if the answer is vague, the workflow is going to double-charge someone within the first month.