Reliable Webhook Handling for Small Teams: A Practical Pipeline

Webhooks are one of the simplest ways to connect tools: when something happens in System A, it sends an HTTP request to System B. That simplicity is exactly why teams underestimate them. A webhook integration that works in testing can still fail in production due to retries, timeouts, out-of-order events, and transient outages.

A small team does not need enterprise infrastructure to make webhooks dependable. You do need a few deliberate design choices: respond quickly, store events durably, process them safely even if they repeat, and provide a way to recover when something goes wrong.

This post walks through a pipeline pattern you can apply to almost any webhook provider, whether you are connecting a payment tool to your CRM, a form service to your ticketing system, or your own product to downstream systems.

What Webhooks Are and Why They Fail in the Real World

A webhook is a push notification delivered as an HTTP request. The sender typically expects your endpoint to respond with a success status quickly. If it does not, the sender often retries. Those retries are well intentioned, but they create the most common class of failures: duplicates.

Beyond duplicates, webhooks fail in a few predictable ways:

• Timeouts: your server takes too long to respond, so the sender retries.
• Temporary errors: brief database outages, deploys, DNS hiccups, or rate limits.
• Out-of-order delivery: you might receive “updated” before “created”, especially across distributed systems.
• Missing events: the sender drops an event or you accidentally reject it during an incident.
• Schema drift: payload shapes change, optional fields disappear, new event types show up.

The key mindset shift is this: treat every webhook as an at-least-once message with uncertain ordering. Once you accept that, the solution becomes a set of straightforward guardrails rather than a fragile pile of special cases.

A Minimal Architecture That Scales: Receive, Queue, Process, Confirm

A reliable webhook pipeline separates concerns. Your webhook endpoint should do the minimum work needed to validate and persist the event, then respond quickly. Everything else happens asynchronously.

Four stages, one responsibility each

1. Receive: authenticate the request (signature or shared secret), check it is well formed, and capture basic metadata.
2. Queue: store the event durably and put it on a work queue (or mark it pending for a worker). This can be a database table plus a background job system.
3. Process: perform the business action (create a record, update a customer, trigger an email), using idempotency safeguards.
4. Confirm: mark success or failure, store errors, and decide whether to retry.

Conceptually, the data you keep can be very small. A single table or collection is often enough:

{
  event_id: "provider_event_id",
  received_at: "timestamp",
  type: "event.type",
  payload: "{...raw JSON...}",
  status: "pending|processing|succeeded|failed",
  attempts: 0,
  last_error: "string or null"
}

This pattern scales down nicely. “Queue” can be a database row plus a scheduled worker. It scales up too: the same table can feed a dedicated queue, and the processing step can fan out into multiple workers.

Step-by-Step: Making Processing Idempotent Without Overengineering

Idempotency means that processing the same event twice has the same effect as processing it once. Since most webhook providers retry on timeouts or non-2xx responses, idempotency is your primary defense against double-charging, duplicate records, and “phantom” updates.

1) Choose a stable idempotency key

Most providers include a unique event identifier. Use that as your primary idempotency key. If the provider does not supply one, derive a key from a combination of fields that should be unique, such as type + object_id + timestamp. Avoid hashing the entire payload unless you have to, because minor payload differences can break deduping.

2) Record “event seen” before side effects

When a worker starts processing, it should claim the event and ensure it cannot be claimed by another worker at the same time. Then it should write a record that the event is being processed, before calling external APIs or writing dependent records. This reduces the chance that a crash causes the same event to be processed concurrently later.

3) Make each business action safe to repeat

Deduping at the event level helps, but it is not always sufficient. A worker can crash after partially completing work. Aim to make downstream operations repeatable too:

• Upserts instead of inserts: create-or-update for records keyed by a stable external ID.
• Natural uniqueness constraints: enforce uniqueness on fields like external_invoice_id.
• State transitions: only allow transitions forward (for example, pending → paid), and ignore repeats.

4) Retry with boundaries

Retries are good when failures are transient and bad when failures are permanent. Keep your retry policy simple:

• Retry network errors and rate limits, up to a small maximum attempt count.
• Do not endlessly retry validation errors or missing required data.
• Separate “failed, needs human review” from “failed, will retry automatically”.

If you only implement one thing beyond “store the payload”, implement idempotency. It is the difference between an annoying incident and a costly one.

Operational Checklist: Logs, Alerts, and Backfills

A webhook pipeline is not complete until you can answer two questions quickly: “What happened?” and “How do we recover?” The best time to add operational hooks is before you need them.

A copyable checklist

• Persist raw payloads (or a sanitized version) so you can replay without asking the provider to resend.
• Store processing status with timestamps for received, started, and finished.
• Capture errors in a searchable way (message plus a short code or category).
• Expose a small admin view listing failed events, with a manual “retry” action.
• Add a dead-letter concept: events that exceeded retries move to “failed” and stop retrying.
• Set alerts on symptoms: sudden spike in failures, growing backlog, or no events received for an expected integration.
• Define a backfill process: select events by date range or status, then replay them in order.

Backfills deserve special attention. Even if your provider offers a “resend last N events” feature, you still want your own replay ability. A practical approach is to support replay by time window and by status, then run the same processing logic used by the worker.

Common Mistakes (and How to Avoid Them)

Most webhook incidents are caused by a few repeat offenders. Fixing them early saves you from paging yourself later.

• Doing real work in the webhook request: if you call third-party APIs or run heavy business logic before responding, you will time out under load. Solution: respond after you persist and enqueue.
• Assuming “exactly once” delivery: treating retries as anomalies rather than normal behavior leads to duplicates. Solution: idempotency keys and unique constraints.
• Throwing away payloads: logging only “success/failure” forces you to guess what the provider sent. Solution: store raw payload plus headers you need for auditing.
• No separation between transient and permanent failures: endless retries hide real issues and can create provider rate limit storms. Solution: cap retries and route permanent failures to review.
• Not planning for schema changes: if you hard-fail when an unknown field appears or a field becomes optional, you will break unexpectedly. Solution: tolerate extra fields and validate required fields defensively.

A good integration is boring: when something breaks, it breaks visibly, recoverably, and without corrupting data.

When Not to Build a Pipeline Like This

This pipeline pattern is useful, but it is not always necessary. Consider keeping it simpler when:

• You control both sides and can use a synchronous API call with a clear response contract.
• The consequence of duplicates is harmless, such as writing to an append-only analytics log.
• The integration is strictly internal and can be re-run cheaply from a primary database export.

Even then, borrowing a few pieces is usually worth it. For example, “store the raw event and respond fast” is almost always a win.

A Concrete Example: Booking App to Accounting System

Imagine a small studio that takes appointments in a booking tool. When a customer pays, the booking tool sends a payment.succeeded webhook. The studio wants invoices created in an accounting system and customer records updated in a CRM.

Here is how the pipeline plays out:

1. Receive: the webhook endpoint validates the signature and extracts event_id, type, and the booking tool’s payment_id.
2. Queue: it writes a row to webhook_events with status pending and enqueues a job to process that row.
3. Process (idempotent):
   • The worker attempts to “claim” the event by marking status processing.
   • It upserts the customer in the CRM keyed by external_customer_id.
   • It creates an invoice in accounting, but only if an invoice with external_payment_id does not already exist.
4. Confirm: on success, it marks the event succeeded. On a transient API error, it increments attempts and schedules a retry. On a permanent error (missing required customer email), it marks failed with a clear error reason.

Two weeks later, the accounting system has an outage. Events accumulate as pending or failed. After recovery, you run a backfill: replay all events from the outage window. Because the processing is idempotent, replaying is safe, and you do not create duplicate invoices.

This is the practical promise of the pipeline: failures become backlog, not data corruption.

Conclusion

Reliable webhooks are not about fancy infrastructure. They are about acknowledging that delivery is messy and designing for it: accept fast, store durably, process idempotently, and make recovery routine.

If you are building more than one integration, consider standardizing on this pattern so every new webhook has the same life cycle, the same states, and the same operational tools. For more posts like this, browse the Archive or learn how the site is built on the About page.

FAQ

Should my webhook endpoint always return 200 OK?

Return a success response when you have authenticated the request and durably stored the event for processing. If authentication fails, return an error. Avoid returning errors for downstream processing failures, because that couples the provider’s retry behavior to your internal incident state.

Do I need a message queue, or is a database enough?

A database is often enough for small teams: store events in a table and use a background worker to pull pending rows. A dedicated queue can help at higher volumes, but the core idea is durability and controlled processing, not the specific tool.

How do I handle out-of-order events?

Prefer designing handlers that do not require strict ordering. Use upserts and state transitions that only move forward. When ordering matters, store the latest known version and ignore older updates, or delay processing until prerequisites exist.

How long should I keep webhook payloads?

Keep them long enough to support debugging and backfills, which for many teams is weeks or months. If payloads contain sensitive data, store a minimized or redacted version and apply retention limits aligned with your internal policies.

What is the smallest “good” implementation?

Validate authenticity, store the raw event with a unique provider event ID, respond quickly, and process asynchronously with idempotency checks. Even that minimal set eliminates most duplicate and timeout-related failures.