Key Takeaways
- Set connection timeouts between 1-3 seconds to fail fast on unreachable endpoints
- Use exponential backoff (2x multiplier) to avoid overwhelming recovering services
- Target P99 latency under 10 seconds for real-time webhook integrations
- Include jitter (random delay) to prevent thundering herd problems
- Always implement idempotency to safely handle duplicate deliveries
Understanding Webhook Latency
Webhook latency is the total time from when a webhook event is triggered to when the receiving endpoint successfully acknowledges the delivery. This includes network time, connection establishment, request processing, and response transmission.
Proper latency planning ensures your webhooks are delivered reliably without timing out, while avoiding unnecessary retries that could overwhelm your systems or exceed rate limits.
Components of Webhook Latency
- DNS Resolution: 1-50ms typically (consider caching)
- TCP Connection: 20-200ms depending on geographic distance
- TLS Handshake: 50-200ms for HTTPS connections
- Request Transmission: Depends on payload size and bandwidth
- Server Processing: Varies based on endpoint implementation
- Response Transmission: Usually minimal for acknowledgments
Total Latency = Connection Time + Request Time + Processing Time + Response Time
Retry Strategy Comparison
Exponential Backoff
The most recommended approach. Each retry waits exponentially longer: 1s, 2s, 4s, 8s, etc. This gives services time to recover and prevents overwhelming them with requests.
Linear Backoff
Delays increase by a fixed amount: 1s, 2s, 3s, 4s. Good for predictable recovery patterns but may be too aggressive for cascading failures.
Fixed Interval
Same delay between each retry. Simple but can cause thundering herd issues when multiple webhooks retry simultaneously.
Pro Tip: Add Jitter
Add random jitter (10-30% variance) to retry delays. This prevents synchronized retries across multiple webhook deliveries, which can overwhelm your endpoints when recovering from an outage.
Webhook Delivery Best Practices
- Return 2xx immediately: Acknowledge receipt quickly, process asynchronously
- Implement idempotency: Use delivery IDs to handle duplicate webhooks safely
- Set reasonable timeouts: 5-30 seconds is standard for most providers
- Use dead letter queues: Store failed webhooks for manual retry or analysis
- Monitor delivery metrics: Track success rates, latency percentiles, and retry counts
- Test with failures: Verify your retry logic handles various failure modes
Frequently Asked Questions
Most providers use 5-30 second timeouts. For real-time integrations, keep it under 10 seconds. For batch processing webhooks, longer timeouts (30-60s) may be acceptable. Always return a 2xx response quickly and process the webhook payload asynchronously.
3-5 retries with exponential backoff is standard. This provides resilience against transient failures while avoiding infinite retry loops. After exhausting retries, move the webhook to a dead letter queue for manual investigation.
Retry on: 408 (Timeout), 429 (Rate Limited), 500-599 (Server Errors). Don't retry on: 2xx (Success), 400-499 except 408/429 (Client Errors). For 429, respect the Retry-After header if provided.
Use HMAC signatures to verify webhook authenticity. Check timestamps to prevent replay attacks (reject webhooks older than 5 minutes). Always use HTTPS. Consider IP allowlisting for additional security.
Idempotency means processing the same webhook multiple times produces the same result. This is critical because retries may deliver duplicate webhooks. Use a unique delivery ID to deduplicate - store processed IDs and skip duplicates.