API & Webhooks

Webhooks That Lie: Building Defences for the Automation Stack You Cannot See

September 3, 2026
Webhooks That Lie: Building Defences for the Automation Stack You Cannot See

Webhook-driven workflows break in quiet ways: delayed retries, dropped events, silent duplicates. Here is how builders harden against gaps they only see later.

Most workflow outages are not dramatic. A webhook arrives late, an event arrives twice, a retry gets reordered against a state that no longer exists. The automation runs, the CRM gets the wrong tag, the customer sees an invoice that does not match reality, and nobody notices until the support inbox fills up. The lesson, repeated across every builder's career, is that a webhook is a promise the upstream system might or might not keep.

What follows are the durable patterns for dealing with that uncertainty. They apply whether you build in n8n, wire flows in Make, sketch Zaps in Zapier, or compose logic inside GoHighLevel. The tool does not save you from bad assumptions about timing, ordering, and delivery. Your architecture does.

Why webhooks fail in ways that workflows do not expect

A polling workflow is a controlled read. You ask, you receive, you decide. A webhook is an uncontrolled push. The sender decides when it goes out, what it carries, how many times it retries, and whether your endpoint was reachable when it mattered. The implication is brutal: every webhook handler is, structurally, a distributed systems problem disguised as an integration.

Three failure modes cover most incidents:

  • Delivery gaps. The sender's network blips, your endpoint returns a 502 while a deploy is rolling, and the event is never re-sent. Some platforms retry aggressively, others give up after one attempt.

  • Duplicates. Retries mean the same event arrives twice. If your workflow credits a customer wallet, charges a usage meter, or fires an email, that duplication is a real-world consequence.

  • Reordering. Event A arrives after event B because A was retried from a different region. If your workflow assumes "created" precedes "updated", you have just lost the customer.

None of these are exotic. They are the default behaviour of webhook plumbing at scale, and they are exactly what an open-source payment project called Flowglad set out to remove by replacing webhooks with reactive SDK reads of a shared source of truth.

The three defences every webhook workflow needs

You cannot make a webhook reliable. You can make your workflow tolerant of an unreliable webhook. Three habits do almost all of the work.

1. Idempotency keys on every side effect

An idempotency key is a string you attach to a write operation so the receiving system can detect a repeat and discard it. Stripe popularised the pattern for payments, but it generalises. If your workflow fires "send welcome email" from a webhook, derive a key from the upstream event ID plus the action type, store it, and short-circuit on the second arrival.

In n8n this becomes a Set node plus a database or KV lookup. In Make it is a router with a filter on a seen-events store. In Zapier it is a Code step hitting a tiny table. The cost is small. The benefit is that retries stop being a liability.

2. A pull-based fallback for critical events

For anything that touches money, identity, or compliance, treat push as a hint, not a source of truth. Add a scheduled poll that reconciles state from the upstream system's read API on a cadence shorter than your tolerance for lag. If the webhook arrives, great, use it. If it does not, the poll catches up before anyone notices.

This is the architectural difference between a workflow that reacts and one that stays correct. The GitHub webhooks incident in May 2026, in which Issues and Webhooks degraded simultaneously, is a useful reminder that the upstream you depend on is also a system that can wobble. If your entire downstream is waiting for a single push channel, your recovery time is bounded by theirs.

3. Explicit ordering and freshness checks

Never assume "later" means "more recent". Webhooks from different regions, or from a retry queue, can arrive out of order. Two cheap patterns cover this:

  • Include a version or timestamp in the payload, and ignore events older than the last accepted version.

  • Use a conditional update at the destination. Update only if the incoming record's updated_at is newer than what is stored. Most CRMs and databases make this a one-liner.

DefenceWhat it preventsWhere it lives in the stackIdempotency keyDuplicate side effects from retriesWorkflow + destination writeScheduled reconciliationLost or delayed webhooksSeparate scheduled flow reading the source APIFreshness checkOut-of-order events overwriting newer stateUpdate node or SQL conditionDead-letter queueSilent failures on permanent errorsSeparate workflow that logs and alerts

Designing the receiving side: webhooks inside an automation builder

Most builders do not run their own HTTP server. They use the webhook trigger node inside n8n, Make, Zapier, or GoHighLevel, and accept whatever URL the platform hands them. That is fine for the happy path. It hurts when you need to test, replay, or hand off.

A few practical adjustments help:

  • Generate the webhook URL once and pin it. Re-creating trigger URLs on every save is a habit that quietly breaks integrations. Capture the production URL and treat it like any other external dependency.

  • Wrap the trigger in a verification node. Most platforms sign their webhooks. Verify the signature before doing anything destructive. Unverified payloads should land in a quarantine workflow, not in your main path.

  • Persist the raw payload first, parse second. Drop the body into a "raw events" table before any logic runs. If a downstream node misinterprets a field, you have the original to replay. This is the cheapest insurance you will ever buy.

For teams that have outgrown visual builders, the same shape applies with a thin custom receiver: a small function that authenticates, validates, persists the body, and enqueues work. The automation tool then becomes the consumer, not the entry point.

When the gap is the product: API-only services and AI agents

An emerging category of tools treats webhooks as legacy. Flowglad's pitch, in its open-source release, is that you should be able to read billing state from an SDK without standing up a webhook endpoint at all. The same logic is showing up in agent platforms. Tracecat, an open-source security automation project, exposes its connectors and workflows over MCP and gRPC so an AI agent can call them directly, rather than waiting for a webhook to fire.

For buyers, the practical question is whether your stack can route around a missing webhook. If a vendor only offers push delivery, you have one integration shape. If the same vendor offers an API, an MCP server, or a streaming export, you have options when the push channel falters. Treat that flexibility as part of the purchase decision.

A short checklist before you ship a webhook workflow

  1. Can the workflow survive the same event arriving twice?

  2. Can the workflow survive the event arriving never?

  3. Can the workflow survive the event arriving after a newer one?

  4. Is there a scheduled reconciliation that catches up state if the push channel is down?

  5. Is the raw payload stored before any business logic runs?

  6. Is there a dead-letter path that surfaces permanent failures?

If you can answer yes to all six, you have built something close to production-grade. If you cannot, you have built a demo that happens to work today. The difference shows up the first time the upstream has a bad day, and the only people who notice are your customers.

For teams that want to skip the rebuild, vetted workflow templates with these patterns baked in are available on the AutoStack marketplace. If you would rather commission one, how it works explains the install, support, and escrow model.