Mulberry | Docs
Home

Webhooks

Webhooks let you receive real-time notifications when crawl events occur. Instead of polling the API, Mulberry pushes events to your endpoint.

Creating a Webhook Endpoint

Create webhook endpoints from the dashboard or via the API: 

curl -X POST \
  -H "Authorization: Bearer sk_your_api_key" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://your-app.com/webhooks/mulberry",
    "events": ["crawl.completed", "crawl.failed"]
  }' \
  https://your-server/api/webhook_endpoints

Available Events

Subscribe to specific events or use wildcards:

Crawl Events

  • crawl.started - Crawl began processing
  • crawl.completed - Crawl finished successfully
  • crawl.failed - Crawl encountered an error
  • crawl.* - All crawl events (wildcard)

Wildcard Patterns

Use wildcards to subscribe to groups of events:

{
  "events": ["crawl.*"]  // All crawl events
}

Webhook Payload

When an event occurs, Mulberry sends a POST request to your endpoint with a JSON payload: 

{
  "id": "evt_xyz789",
  "type": "crawl.completed",
  "created_at": "2024-01-15T10:35:00Z",
  "data": {
    "crawl_id": "crawl_abc123",
    "url": "https://docs.example.com",
    "status": "completed",
    "pages_crawled": 120,
    "started_at": "2024-01-15T10:30:00Z",
    "completed_at": "2024-01-15T10:35:00Z"
  }
}

Verifying Webhook Signatures

All webhook requests include a signature header for verification. This ensures the request came from Mulberry and wasn't tampered with.

Signature Header

The X-Mulberry-Signature header contains the signature: 

X-Mulberry-Signature: t=1705315200,v1=abc123...

The header contains:

  • t - Unix timestamp when the signature was generated
  • v1 - HMAC-SHA256 signature of the payload

Verification Steps

  1. Extract the timestamp and signature from the header
  2. Construct the signed payload: {timestamp}.{request_body}
  3. Compute HMAC-SHA256 using your webhook signing secret
  4. Compare the computed signature with the received signature
  5. Verify the timestamp is within an acceptable window (e.g., 5 minutes)

Example Verification (Node.js)

const crypto = require('crypto');

function verifyWebhook(payload, signature, secret) {
  const [tPart, v1Part] = signature.split(',');
  const timestamp = tPart.split('=')[1];
  const receivedSig = v1Part.split('=')[1];

  const signedPayload = `${timestamp}.${payload}`;
  const expectedSig = crypto
    .createHmac('sha256', secret)
    .update(signedPayload)
    .digest('hex');

  return crypto.timingSafeEqual(
    Buffer.from(receivedSig),
    Buffer.from(expectedSig)
  );
}

Handling Webhook Requests

Your endpoint should:

  1. Respond with a 2xx status code within 30 seconds
  2. Verify the signature before processing
  3. Process the event asynchronously if needed
  4. Handle duplicate events idempotently
Tip: Return a 200 response immediately, then process the event in a background job. This prevents timeouts.

Retry Behavior

If your endpoint fails to respond with a 2xx status, Mulberry retries with exponential backoff:

  • 1st retry: 1 minute
  • 2nd retry: 5 minutes
  • 3rd retry: 30 minutes
  • 4th retry: 2 hours
  • 5th retry: 24 hours

After 5 failed attempts, the webhook is marked as failed and you'll receive a notification.

Managing Webhooks

List Endpoints

curl -H "Authorization: Bearer sk_your_api_key" \
  https://your-server/api/webhook_endpoints

View Delivery History

curl -H "Authorization: Bearer sk_your_api_key" \
  https://your-server/api/webhook_endpoints/wh_xxx/deliveries

Delete Endpoint

curl -X DELETE \
  -H "Authorization: Bearer sk_your_api_key" \
  https://your-server/api/webhook_endpoints/wh_xxx

Webhook Signing Secret

Your webhook signing secret is available in Settings → Webhooks. You can regenerate it at any time, but this will invalidate signatures for any in-flight requests.

Next Steps