Reliable Webhook Integrations for Small Teams: From “It Fired” to “It Worked”

Webhooks are one of the fastest ways for small teams to connect tools: the billing platform tells your app a subscription changed, your form sends a lead into your CRM, your store notifies your warehouse when an order is paid.

But many webhook projects fail in a predictable way: you see the incoming request in logs and assume the integration worked, then later discover missing records, duplicates, or silent failures during a brief outage.

This post turns webhooks into a reliable system. The goal is not “we received the webhook” but “the right business action happened exactly once, and we can prove it.”

What webhooks are good for (and what they are not)

A webhook is an event notification delivered by HTTP. It is best when you want near real time updates without polling, and when an upstream system can push events whenever they occur.

Webhooks are a strong fit for:

• Event-driven integrations: “customer created,” “invoice paid,” “ticket closed.”
• Fan-out: one event triggers multiple internal actions (analytics event, email, CRM update).
• Lightweight automation: small teams that want one endpoint instead of a complex ETL job.

They are a poor fit for:

• Bulk history: “send me all orders since 2019” is better as a sync job or export.
• Guaranteed ordering needs: if the business rule depends on strict ordering, you must design for reordering or use another mechanism.
• High consequence actions without review: irreversible operations should have additional safeguards.

The reliability basics: receive, verify, queue, process

Most reliability problems come from doing too much work inside the webhook request. The upstream system expects a quick acknowledgment; if your handler takes too long, it may time out and retry, which can cause duplicates.

A durable approach splits the work into two phases:

1. Ingress (fast): validate and store the event, then return a success response quickly.
2. Processing (slow, retriable): run the business logic from the stored event with controlled retries.

Verify authenticity and integrity

At minimum, confirm the request is from the expected sender. Common patterns include a shared secret signature, basic auth, or a known token header. Your handler should reject missing or invalid signatures before writing anything to your system.

Also protect against accidental duplication by requiring a stable event identifier. If the sender does not provide one, derive one carefully (for example, a hash of a canonical payload) and treat it as “best effort,” not perfect.

Define a message shape you can live with

Even if you store the raw webhook payload, you will thank yourself later if you define a small, consistent envelope for internal processing. It becomes your contract across retries, reprocessing, and audit logs.

Keep it boring: a unique event id, an event type, a timestamp from the sender, a source, and the raw body. Add an “attempt count” and “status” for operations.

{
  "eventId": "evt_12345",
  "eventType": "order.paid",
  "source": "storefront",
  "occurredAt": "2026-01-01T12:34:56Z",
  "receivedAt": "2026-01-01T12:34:59Z",
  "payload": { "...raw webhook body..." },
  "processing": { "status": "queued", "attempts": 0 }
}

This structure is intentionally generic. It supports both “do something now” and “replay later,” which is the difference between a fragile integration and an operational system.

Implementation checklist you can copy

Use this as a build checklist or as a review rubric before calling an integration “done.” Small teams can implement all of it without heavy infrastructure.

• Endpoint hygiene
  • Dedicated URL path per provider or per integration domain (billing, CRM, store).
  • Strict method handling (usually POST only) and size limits to prevent abuse.
  • Request signature or token validation before processing.
• Ingress storage
  • Write the envelope to durable storage (database table, queue with persistence, or object store plus index).
  • Store the raw body and headers needed for debugging and verification.
  • Capture receivedAt and a correlation id for logs.
• Deduplication and idempotency
  • Enforce uniqueness on eventId (or a derived key) at the database level.
  • Ensure the downstream “business action” can be executed safely more than once (for example, upsert by external id).
• Processing and retries
  • Async worker pulls queued events and marks status transitions (queued, processing, succeeded, failed).
  • Retry on transient failures with backoff (network errors, 429s, timeouts).
  • Stop retrying on permanent failures and route to a review queue (bad payload, missing mapping).
• Observability
  • Log one line per status change with eventId, eventType, and result.
  • Basic metrics: events received, events succeeded, events failed, retry count, oldest queued age.
  • Alert on “backlog age” and “failure rate,” not on every single error.
• Operational controls
  • Admin view or simple internal page to search by eventId and see state.
  • A safe “replay” button that re-queues a failed event after you fix the cause.
  • Document which event types are supported and how to test them in a non-production environment.

A concrete example: orders to fulfillment without duplicates

Imagine a small ecommerce business. When an order is paid, the store sends an order.paid webhook. Your system must create a shipment request in a fulfillment tool and then email the customer a receipt.

Here is the failure mode you want to avoid: your webhook handler calls fulfillment directly. The fulfillment API times out after 10 seconds. Your endpoint times out at 15 seconds. The store retries the webhook. Now you create two shipment requests and the customer gets two receipts.

A reliable implementation looks like this:

1. Ingress: verify signature, parse the payload, and write it to webhook_events with a unique constraint on the provider event id.
2. Acknowledge: respond with 200 quickly (or the provider’s expected success code) once the write succeeds.
3. Process: a worker picks up the event and checks if “shipment request already created” using an idempotency key like orderId or eventId.
4. Side effects: create shipment request, write the returned shipment id to your database, then send the email.
5. Retry safely: if fulfillment is down, the worker retries. Because you record “shipment already created,” the second attempt does not double-ship.

Note the ordering: you record the “what we did” state before triggering the next side effect whenever possible. That turns retries from scary to routine.

Common mistakes (and how to avoid them)

• Doing heavy work in the request thread: keep webhook responses fast. Move API calls and complex logic to an async processor.
• No database-level dedupe: relying on “we check in code” fails under concurrency. Enforce uniqueness with a constraint or atomic write.
• Assuming providers never resend: resend happens for timeouts, maintenance, and network hiccups. Design for it upfront.
• Logging only errors: you also need lifecycle logs. A missing event is harder than a failing event.
• Infinite retries: endless retries can amplify outages. Use max attempts and send hard failures to a human review queue.
• Conflating “accepted” with “processed”: a 200 response means “we received it,” not “we completed the business action.” Track both states explicitly.

When NOT to use webhooks

Webhooks are not a universal answer. Consider alternatives when:

• You need guaranteed completeness over long history: use a periodic sync job that compares state, then treat webhooks as an accelerator.
• The sender cannot provide stable identifiers: without an event id or a stable business id, deduplication becomes fragile.
• You require strict ordering for correctness: you can build ordering controls, but it may be simpler to consume a sequence-based API or use a batch process.
• The action is irreversible and high risk: add a manual approval step or run in a “create draft” mode first.

A healthy pattern is hybrid: webhooks for speed, periodic reconciliation for confidence.

FAQ

What should my webhook endpoint return?

Return success as soon as you have verified the request and durably recorded the event. Do not wait for downstream API calls or long processing. If verification fails, return an error quickly and log enough context to debug safely.

How do I handle duplicate webhook deliveries?

Use a unique key (ideally the provider’s event id) and enforce uniqueness at storage. Then make downstream actions safe to repeat by using upserts and recording “action already performed” state keyed by a stable id like order id.

Do I need a queue, or can I just use a database table?

For many small teams, a database table with a status field works well: it is searchable, durable, and easy to replay. A dedicated queue can help at high volume, but the reliability principles stay the same: acknowledge fast, process async, retry with limits, and keep a clear audit trail.

What should I monitor first?

Start with three signals: oldest queued event age, percentage failed in the last hour, and total processing rate. These tell you when you are falling behind, when something is broken, and whether the system is keeping up.

Conclusion

A reliable webhook integration is less about clever code and more about simple, durable mechanics: verify, store, process, and observe. If you can replay any event, explain what happened to it, and safely retry it, your integrations stop being fragile and start behaving like infrastructure.

If you want more posts on building dependable automation systems, browse the Archive or subscribe via RSS.