# Guide to KoboToolbox Webhooks: Features and Best Practices

KoboToolbox is an open-source platform for field data collection, used for surveys, assessments and monitoring by humanitarian organisations, researchers and NGOs. Enumerators collect data on the KoboCollect app or web forms, often offline, and submissions sync to the server when a connection is available. REST Services are how those submissions reach your own systems as soon as the server receives them.

This guide covers what a REST Service sends and when, how to authenticate it with Basic credentials, how KoboToolbox retries, and the practices that keep a receiver reliable in production.

## What are KoboToolbox webhooks?

KoboToolbox calls its webhooks REST Services. Each one belongs to a project and sends every new submission to a URL you register, in JSON or XML. KoboToolbox's documentation is short, so some details in this guide come from the open-source `kpi` code that implements the sender.

A REST Service fires on one thing: a new submission. Data is sent only on creation, and nothing is sent when a submission is edited. There is no event type and no event field, because there is only one event.

KoboToolbox does not sign requests. The documented authentication is HTTP Basic, with a username and password you set on the service. The `kpi` sender hands them to Python's `requests` library, which sends the `Authorization` header with the first request rather than waiting for a challenge.

## KoboToolbox webhook features

| Feature | Details |
| --- | --- |
| Configuration | A REST Service in the project's settings, one per endpoint |
| Fires on | A new submission only. Edits send nothing |
| Format | JSON or XML, chosen per service |
| Options | A subset of fields, and (JSON only) a custom wrapper |
| Method | `POST` |
| Authentication | HTTP Basic credentials on the service. Custom headers are also possible. No signature |
| User-Agent | `KoboToolbox external service` followed by the server's domain and the service's ID |
| Timeout | 30 seconds, in the `kpi` sender |
| Retries | 3, after 60, 600 and 6,000 seconds. `kpi` retries `502`, `503`, `504` and timeouts |
| Failure reports | Optional email notifications |
| Source IPs | Not published. Self-hosted servers send from wherever they run |

## What a request looks like

With the JSON format, the body is the submission itself: one key per question in your form, using the question names, plus KoboToolbox metadata fields such as `_id` and `_uuid`. Groups and repeat groups change how the keys nest, and the subset-of-fields option sends only the questions you list. The exact shape depends on your form, so log a real submission before you write your parser.

The custom wrapper wraps the submission in a JSON template of your own. KoboToolbox replaces the placeholder `%SUBMISSION%` with the submission's JSON, so a template such as:

```json
{ "source": "household-survey", "submission": %SUBMISSION% }

```

lets one endpoint tell several projects apart. With the XML format, the body is the submission's XML and the `Content-Type` is `application/xml`.

> **Inspect KoboToolbox submissions as they arrive.** Point a REST Service at a [Hookdeck Inspect](https://inspect.hookdeck.com) URL to see the exact headers and body your form produces, with no account or setup required.

## Setting up KoboToolbox webhooks

Pick a username and generate a long random password, for example with `openssl rand -hex 32`, and store both on your receiver.

1. Open the project in KoboToolbox and go to its **Settings**, then **REST Services**.
2. Register a new service with a name and your endpoint URL.
3. Choose **JSON** or **XML**.
4. Choose basic authentication and enter the username and password.
5. Optionally list a subset of fields, or (JSON only) add a custom wrapper, and enable email failure notifications.
6. Save, then submit a test record from a web form or KoboCollect.

To put Hookdeck in front of your service, create a Hookdeck source with the **KoBoToolbox** type, use its URL as the endpoint, and enter the same **Username** and **Password** on the source. The source accepts `POST`, which is the only method KoboToolbox uses.

For local development, `hookdeck listen 3000 kobotoolbox --path /webhooks/kobotoolbox` gives you a public URL that forwards to your machine.

## Securing KoboToolbox webhooks

Each request carries `Authorization: Basic base64(username:password)`. Decode it, split on the first colon, and compare the username and password with your stored values in constant time, evaluating both before you decide.

```javascript
const crypto = require('crypto');
const express = require('express');

function safeEqual(a, b) {
  // timingSafeEqual throws on unequal lengths, so check first.
  return a.length === b.length && crypto.timingSafeEqual(a, b);
}

function verifyBasicAuth(header, username, password) {
  if (!username || !password) return false; // never accept when unconfigured
  const match = /^basic\s+([A-Za-z0-9+/]+={0,2})\s*$/i.exec(header || '');
  if (!match) return false;
  const decoded = Buffer.from(match[1], 'base64');
  const colon = decoded.indexOf(0x3a); // split on the first colon: passwords may contain ':'
  if (colon === -1) return false;
  const userOk = safeEqual(decoded.subarray(0, colon), Buffer.from(username, 'utf8'));
  const passOk = safeEqual(decoded.subarray(colon + 1), Buffer.from(password, 'utf8'));
  return userOk && passOk; // evaluate both before deciding
}

const app = express();

app.post('/webhooks/kobotoolbox', express.raw({ type: '*/*' }), (req, res) => {
  const ok = verifyBasicAuth(
    req.get('authorization'),
    process.env.KOBOTOOLBOX_WEBHOOK_USERNAME,
    process.env.KOBOTOOLBOX_WEBHOOK_PASSWORD
  );
  if (!ok) return res.sendStatus(401);

  if (req.is('application/json')) {
    let submission;
    try {
      submission = JSON.parse(req.body.toString('utf8'));
    } catch {
      return res.sendStatus(400);
    }
    // Deduplicate on submission._uuid, then store the answers.
  } else {
    // XML format: parse req.body with an XML parser.
  }

  res.sendStatus(200);
});

```

The same check in Python:

```python
import base64
import binascii
import hmac
import os
import re

BASIC_PATTERN = re.compile(r"^basic\s+([A-Za-z0-9+/]+={0,2})\s*$", re.IGNORECASE)

def verify_basic_auth(header, username, password) -> bool:
    if not username or not password:
        return False  # never accept when unconfigured
    match = BASIC_PATTERN.match(header or "")
    if not match:
        return False
    try:
        decoded = base64.b64decode(match.group(1), validate=True)
    except (binascii.Error, ValueError):
        return False
    received_user, sep, received_password = decoded.partition(b":")  # first colon only
    if not sep:
        return False
    # Compare bytes: compare_digest raises TypeError on a non-ASCII str.
    user_ok = hmac.compare_digest(received_user, username.encode("utf-8"))
    password_ok = hmac.compare_digest(received_password, password.encode("utf-8"))
    return user_ok and password_ok

# verify_basic_auth(request.headers.get("authorization"),
#                   os.environ.get("KOBOTOOLBOX_WEBHOOK_USERNAME"),
#                   os.environ.get("KOBOTOOLBOX_WEBHOOK_PASSWORD"))

```

Basic credentials prove the sender knows the password. They don't cover the body and carry no timestamp, so a modified or replayed request still passes. Survey data is often personal or sensitive, so use an HTTPS endpoint, a long random password, and rotate it if it might have leaked.

> **Make KoboToolbox submissions production-ready.** [Hookdeck Event Gateway](/event-gateway) checks the Basic credentials, stores every submission, and keeps retrying delivery to your service long after KoboToolbox's three retries would have run out.

## KoboToolbox webhook limitations and pain points

### Three retries, then nothing

**The Problem:** If your endpoint is down for more than about two hours, submissions collected in the field never reach your system.

**Why It Happens:** KoboToolbox retries a failed delivery 3 times, after 60, 600 and 6,000 seconds. In the `kpi` code, only `502`, `503`, `504` and timeouts count as retriable, so a `500` or a `4xx` fails at once.

**Workarounds:**

* Enable email failure notifications on the service.
* Reconcile by pulling recent submissions through the KoboToolbox API, since the data stays on the server.
* Return `503` rather than `500` for a temporary problem in your own code, so KoboToolbox will retry.

**How Hookdeck Can Help:** Hookdeck accepts and stores each submission as it arrives, then delivers to your service with [retries](/docs/retries) you configure. [Issues](/docs/issues) alert you when deliveries fail, and you can replay any submission once your service is back.

### Edits never arrive

**The Problem:** A submission corrected after review in KoboToolbox stays wrong in your system.

**Why It Happens:** REST Services send data only on creation. Nothing is sent when a submission is edited, and there is no other webhook to subscribe to.

**Workarounds:**

* Treat the REST Service as the fast path, and sync edits on a schedule through the KoboToolbox API.

### A static credential instead of a signature

**The Problem:** Anyone with the credentials can send submissions your endpoint will accept.

**Why It Happens:** Basic authentication is the only documented security option, and nothing is computed over the body.

**Workarounds:**

* Use a long random password over HTTPS, and a separate credential per project.
* Validate submissions against the form you expect before storing them.

**How Hookdeck Can Help:** Hookdeck's KoboToolbox source checks the Basic username and password on every request. Hookdeck still answers with a `200`, but a request with missing or wrong credentials is recorded as failed verification and never reaches your destination.

### Offline collection means bursts

**The Problem:** A team returning from the field syncs a day's submissions at once, and your endpoint receives them all within minutes.

**Why It Happens:** KoboCollect stores submissions offline and uploads them when it has a connection, and each one triggers its own REST Service request.

**Workarounds:**

* Keep the handler fast: authenticate, store, return `200`.
* Size the endpoint for peak uploads, not the daily average.

**How Hookdeck Can Help:** Hookdeck absorbs the burst and delivers to your service at a [max delivery rate](/docs/destinations#set-a-max-delivery-rate) you set.

## Best practices

### Deduplicate on `_uuid`

Retries can deliver the same submission more than once. Store `_uuid` (or `_id`) and skip repeats. See our [guide to webhook idempotency](/webhooks/guides/implement-webhook-idempotency).

### Use the custom wrapper to label projects

If several projects post to one endpoint, wrap each one's submissions with a project label, so routing doesn't depend on guessing from question names.

### Expect your form to change

Adding, renaming or regrouping questions changes the keys in the body. Parse defensively, and keep the raw submission so you can reprocess it after a form change.

### Acknowledge within the timeout

The `kpi` sender waits 30 seconds, and a timeout counts as a retriable failure. Store the submission and return `200`, then process it from a queue. See [why to process webhooks asynchronously](/webhooks/guides/why-implement-asynchronous-processing-webhooks).

## Conclusion

KoboToolbox REST Services send each new submission once, as JSON or XML, authenticated with Basic credentials and retried only three times over about two hours. Compare the credentials in constant time over HTTPS, deduplicate on `_uuid`, sync edits separately, and keep the endpoint fast enough to take a team's offline uploads at once.

[Hookdeck Event Gateway](https://hookdeck.com) checks the credentials and holds each submission until your service has processed it, so a longer outage on your side costs latency instead of field data.

[Get started with Hookdeck](https://dashboard.hookdeck.com/signup) for free and handle KoboToolbox webhooks reliably in minutes.