# Delivery and security > How Kustomer delivers outbound webhooks. HMAC-SHA256 signatures, the retry schedule, at-least-once delivery, ordering, timeouts, automatic disabling and how to debug failed deliveries. Source: https://developer.kustomer.com/docs/events/delivery-and-retries Kustomer delivers each event as an HTTPS `POST` and retries failures for about an hour. Your receiver should do three things: verify the signature, return `2xx` quickly, and tolerate duplicates and reordering. ## The request ```http POST /kustomer HTTP/1.1 Host: hooks.example.com Content-Type: application/json X-Kustomer-Webhook-Event: kustomer.conversation.update X-Kustomer-Webhook-Signature: 3f1c9a…e07b X-Source: kustomer {"id":"6717c2b1e5b1f00012ab3511","name":"kustomer.conversation.update",…} ``` | Header | Value | | ------------------------------ | --------------------------------------------------------------------------- | | `Content-Type` | `application/json` | | `X-Kustomer-Webhook-Event` | The event name, so you can route before parsing | | `X-Kustomer-Webhook-Signature` | Lowercase hex HMAC-SHA256 of the raw body, keyed with the webhook's `token` | | Your custom headers | Whatever you set in `headers` when creating the webhook | ## Verify signatures The `token` returned when you create the webhook is the signing secret. You can also read it later with [`GET /v1/outbound-webhooks/{id}`](https://developer.kustomer.com/api/apps-platform/get-outbound-webhook), or in **Settings › Platform › Outbound webhooks › Edit › Header information**. ```text signature = hex( HMAC_SHA256( key = token, message = raw request body ) ) ``` > **Sign the raw bytes:** Compute the HMAC over the body exactly as received. Parsing and re-serializing JSON changes whitespace and key order, and the signature won't match. In Express, use `express.raw()` on this route rather than `express.json()`. ```ts title="Node.js" import crypto from 'node:crypto' export function verifyKustomer(rawBody: Buffer, signature: string | undefined, token: string): boolean { if (!signature) return false const expected = crypto.createHmac('sha256', token).update(rawBody).digest('hex') return signature.length === expected.length && crypto.timingSafeEqual(Buffer.from(signature), Buffer.from(expected)) } ``` ```python title="Python" import hashlib, hmac def verify_kustomer(raw_body: bytes, signature: str | None, token: str) -> bool: if not signature: return False expected = hmac.new(token.encode(), raw_body, hashlib.sha256).hexdigest() return hmac.compare_digest(expected, signature) ``` ```go title="Go" func VerifyKustomer(rawBody []byte, signature, token string) bool { mac := hmac.New(sha256.New, []byte(token)) mac.Write(rawBody) return hmac.Equal([]byte(hex.EncodeToString(mac.Sum(nil))), []byte(signature)) } ``` ```ruby title="Ruby" require "openssl" def verify_kustomer(raw_body, signature, token) return false if signature.nil? expected = OpenSSL::HMAC.hexdigest("SHA256", token, raw_body) Rack::Utils.secure_compare(expected, signature) end ``` ```php title="PHP" function verify_kustomer(string $rawBody, ?string $signature, string $token): bool { if ($signature === null) return false; $expected = hash_hmac('sha256', $rawBody, $token); return hash_equals($expected, $signature); } ``` Always compare in constant time (`timingSafeEqual`, `compare_digest`, `hmac.Equal`). A plain `===` leaks how many leading characters matched. ### Rotate the token There's no rotate endpoint. To rotate, create a second webhook with the same URL and events, update your receiver to accept either token, then delete the old webhook. ## Respond fast Kustomer waits up to **30 seconds** for a response. Return `200` as soon as the signature checks out, then do the work from a queue. Slow responses tie up deliveries and count as failures when they time out. ```mermaid sequenceDiagram participant K as Kustomer participant R as Your receiver participant Q as Queue participant W as Worker K->>R: POST event (signed) R->>R: verify signature R->>Q: enqueue raw body R-->>K: 200 OK (milliseconds) Q->>W: deliver W->>W: dedupe on event id, apply ``` ## Retries A delivery fails on a timeout, a connection error, or a status of `400` or higher. Kustomer retries a failed event up to **8 attempts in total, over about 64 minutes**: | Attempt | Delay before it | Time since the first attempt | | ------- | --------------- | ---------------------------- | | 1 | — | 0 | | 2 | 1 min | 1 min | | 3 | 1 min | 2 min | | 4 | 2 min | 4 min | | 5 | 5 min | 9 min | | 6 | 10 min | 19 min | | 7 | 15 min | 34 min | | 8 | 30 min | 64 min | Delays vary by about ±10% so retries from many events don't arrive in lockstep. If all 8 attempts fail, Kustomer drops that event. It isn't redelivered later, so plan to [reconcile](#reconcile-after-an-outage) after an outage. A delivery whose host name doesn't resolve (a DNS failure) isn't retried at all. > **Use status codes deliberately:** Return `2xx` for anything you've accepted, even if you'll ignore it. Return `5xx` only when you want a retry. A `4xx` is also retried, so don't use it to reject events you never want; acknowledge and drop them instead. _A timeline of up to 8 delivery attempts for one event, at 0, 1, 2, 4, 9, 19, 34 and 64 minutes, with the waits of 1, 1, 2, 5, 10, 15 and 30 minutes labelled between them and a to-scale strip showing the same attempts on a 70-minute clock. You pick how your endpoint responds: a 200 is delivered on the first attempt; a 500, a 401 or a timeout after 30 seconds fails every attempt, so the event is dropped after 64 minutes and isn't redelivered, and an endpoint down that long gets the webhook disabled; a 503 that recovers on a chosen attempt is delivered then; a 301 counts as delivered even though redirects aren't followed. Every attempt re-sends the same event id, and the latest transaction shows status failed with a nextRetry timestamp, or null once the event succeeds or runs out of attempts._ ## Duplicates and ordering Delivery is **at least once**. The same event can arrive twice, for example if your `200` was lost on the way back. Deduplicate on the envelope's `id`: ```ts title="Node.js" async function handle(event: KustomerEvent) { const firstTime = await redis.set(`kustomer:event:${event.id}`, '1', { NX: true, EX: 60 * 60 * 24 }) if (!firstTime) return await apply(event) } ``` Order isn't guaranteed either: a retried event can arrive after a newer one for the same record. Compare `data.attributes.updatedAt` with what you stored and skip older snapshots: ```sql title="Postgres" INSERT INTO conversations (id, status, updated_at, doc) VALUES ($1, $2, $3, $4) ON CONFLICT (id) DO UPDATE SET status = EXCLUDED.status, updated_at = EXCLUDED.updated_at, doc = EXCLUDED.doc WHERE conversations.updated_at < EXCLUDED.updated_at; ``` ## Automatic disabling If your endpoint keeps failing, Kustomer turns the webhook off so it stops sending to a dead destination. This is based on failures **sustained over time**, not a short blip: a brief outage or a burst of errors on a busy webhook won't disable it, but an endpoint that stays down for tens of minutes will. Successful deliveries count in your favor. When a webhook is disabled: - `enabled` becomes `false` and `isError` becomes `true`. - Admins see an error notification in Kustomer. - Events that happen while it's off are not queued. Fix the endpoint, then re-enable it with [`PATCH /v1/outbound-webhooks/{id}`](https://developer.kustomer.com/api/apps-platform/update-outbound-webhook) and `{ "enabled": true }`, or with the toggle in **Settings › Platform › Outbound webhooks**. Re-enabling resets the failure history. ## Debug deliveries List the 25 most recent attempts for a webhook: `GET /v1/outbound-webhooks/{webhookId}/transactions` ([reference](https://developer.kustomer.com/api/apps-platform/list-transactions-for-webhook)) ```json title="200 OK" { "data": [ { "type": "outbound_webhook_transaction_log", "id": "5f2c62067aba570013e64bb1", "attributes": { "webhookId": "5f0f572d3919cc00123fd52e", "eventName": "kustomer.conversation.update", "sentAt": 1729522931273, "status": "failed", "nextRetry": "1729522991000" } } ] } ``` `status` is `succeeded` or `failed`. `nextRetry` is a millisecond timestamp for the next attempt, or `null` once the event succeeded or ran out of attempts. Common causes of failures: | Symptom | Likely cause | | ----------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------ | | Every delivery fails with `401` | Your receiver's signature check is hashing parsed JSON instead of the raw body | | Timeouts | Doing work before responding; move it to a queue | | `403` or no requests arrive | A firewall, WAF or bot protection is blocking Kustomer; allow the webhook path | | Deliveries show `succeeded` but never reach your code | The URL redirects (for example to add a trailing slash). Redirects aren't followed but count as delivered, so point the webhook at the final URL | | Some events never arrive | They exhausted their retries during an outage; reconcile | ## Reconcile after an outage Webhooks are for low latency, not completeness. After an outage longer than about an hour, or after re-enabling a webhook, fetch what changed with [search](https://developer.kustomer.com/docs/guides/search-recipes#incremental-sync) using an `updated_at` window that starts before the outage began. Because your handler [ignores older snapshots](#duplicates-and-ordering), overlapping with events you already processed is harmless. ## Network and security checklist - [x] HTTPS endpoint with a valid certificate - [x] Signature verified over the raw body, in constant time - [x] Token stored in a secret manager, not in code - [x] `200` returned within a few seconds; work done asynchronously - [x] Deduplication on event `id` - [x] Older snapshots ignored using `updatedAt` - [x] Alerting on non-2xx rates and on the webhook being disabled - [x] A reconciliation job for outages