Webhooks are deceptively fragile. A provider sends an event to your receiver, you process it, life goes on. Except the provider's retry logic kicks in because your server returned a 500, so the event hits you twice. Or your deploy window drops the request entirely. Or a flaky third-party API call inside your handler fails and the event vanishes. The provider thinks it delivered; you never got it.
Velocity X runs production webhook handlers for HubSpot, Stripe, and custom integrations. Here's the pattern that actually works.
Three Webhook Failure Modes
Mode 1: Duplicate Events on Provider Retry. You receive a contact.update event, process it, but your response times out. The provider's retry logic fires and sends the same event again 30 seconds later. You've now created two update records instead of one. If you're counting event occurrences (billing webhooks, trigger counts), you've just billed the customer twice.
Mode 2: Lost Events During Deploy. Your server is rolling out a new version. A webhook arrives mid-deploy, hits a stale process, and gets dropped. Your health checks pass. No error is logged. The provider marked it as delivered. You never knew.
Mode 3: Failed Processing, No Retry. Your handler calls an external API (rate limit, timeout, service down) and crashes. The webhook HTTP response never goes out, so the provider will retry — but by then your database may have been partially updated, leading to inconsistent state. Retry logic inside handlers is fragile and breeds bugs.
Idempotency Key: The Foundation
The fix starts before processing. Every webhook event has a unique ID from the provider (HubSpot uses `objectId` + `timestamp`, Stripe uses `id`). We store this as an idempotency key in the database:
CREATE TABLE webhook_events (
id UUID PRIMARY KEY,
provider TEXT NOT NULL,
idempotency_key TEXT NOT NULL UNIQUE,
payload JSONB,
processed BOOLEAN DEFAULT FALSE,
processed_at TIMESTAMP,
error TEXT
);
-- Unique constraint prevents duplicate processing
CREATE UNIQUE INDEX webhook_events_provider_key
ON webhook_events(provider, idempotency_key);On every webhook POST, insert the event with its idempotency key. If the provider sends it again, the UNIQUE constraint fires — we skip processing. No duplicate charge, no double-update. Dead simple, bulletproof.
Dead-Letter Queue: The Safety Net
Idempotency stops duplicates, but what about failures? Instead of processing webhooks synchronously, insert them into the queue table and return HTTP 202 (Accepted) immediately. A background job dequeues and processes:
-- Handler receives webhook
INSERT INTO webhook_events (provider, idempotency_key, payload)
VALUES ('hubspot', 'contact_123_1717939211', payload)
ON CONFLICT (provider, idempotency_key) DO NOTHING;
RETURN 202 Accepted;
-- Background job (runs every 5 seconds)
SELECT * FROM webhook_events WHERE processed = FALSE
AND (error_count < 5 OR next_retry_at < NOW())
LIMIT 100;
-- Process each, update processed = TRUE or error_count++The background job retries failed events hourly for 5 attempts, then stops and alerts the ops team. Zero events are lost; they live in the queue until you fix the root cause. Contrast that with synchronous handlers: if your API call times out, the webhook is gone forever.
Retry Cadence and Exponential Backoff
Velocity X retries on a schedule: 30 seconds, 2 minutes, 10 minutes, 1 hour, 24 hours. That's 5 attempts over a day. If a third-party API is flaky, it recovers within an hour. If it's permanently broken, you catch it on the first alert and investigate. Exponential backoff also prevents retry storms — you're not hammering a downed service every 5 seconds.
On the 5th failure, we log the event to a Slack channel with full context: provider, idempotency key, error message, and a link to re-trigger. The ops team can fix the root cause (missing API key, rate-limit bump, schema change) and manually replay the event without rebuilding it.
Frequently Asked Questions
What if the same event is processed by two job runs simultaneously?
Lock the row during processing with a `processing_started_at` timestamp. Before dequeuing, set `processing_started_at = NOW()` in the same query. If another job tries to lock the same row, the query returns zero rows and moves on. Postgres row-level locking is built-in.
Should I process webhooks in the request or in a background job?
Background job, always. Request handlers should insert into the queue and return 202. If you process synchronously, a slow API call in your handler stalls the webhook response, provider sees timeout, and retries. You're fighting the provider's retry logic instead of working with it.
What if I need to replay all webhooks from a specific time?
Query the queue table for events with `created_at > X` and `processed = TRUE`. Set `processed = FALSE` for those rows. The background job picks them up and reprocesses. This is how you recover from a schema change or bug fix without manual intervention.
The Bottom Line
Webhook reliability is table stakes for any integration. Three patterns — idempotency keys, dead-letter queue, exponential backoff — ship together. No duplicates, no lost events, no silent failures. Velocity X uses this for HubSpot, Stripe, and custom event sources; it's survived production for 2+ years. If you're building webhook handlers, implement this first. See pricing for how Velocity X handles integrations at scale, or deep-dive into OAuth bidirectional sync.