Guide to KoboToolbox Webhooks: Features and Best Practices

KoboToolbox is an open-source platform for field data collection, used for surveys, assessments and monitoring by humanitarian organisations, researchers and NGOs. Enumerators collect data on the KoboCollect app or web forms, often offline, and submissions sync to the server when a connection is available. REST Services are how those submissions reach your own systems as soon as the server receives them.

This guide covers what a REST Service sends and when, how to authenticate it with Basic credentials, how KoboToolbox retries, and the practices that keep a receiver reliable in production.

What are KoboToolbox webhooks?

KoboToolbox calls its webhooks REST Services. Each one belongs to a project and sends every new submission to a URL you register, in JSON or XML. KoboToolbox's documentation is short, so some details in this guide come from the open-source kpi code that implements the sender.

A REST Service fires on one thing: a new submission. Data is sent only on creation, and nothing is sent when a submission is edited. There is no event type and no event field, because there is only one event.

KoboToolbox does not sign requests. The documented authentication is HTTP Basic, with a username and password you set on the service. The kpi sender hands them to Python's requests library, which sends the Authorization header with the first request rather than waiting for a challenge.

KoboToolbox webhook features

Feature	Details
Configuration	A REST Service in the project's settings, one per endpoint
Fires on	A new submission only. Edits send nothing
Format	JSON or XML, chosen per service
Options	A subset of fields, and (JSON only) a custom wrapper
Method	POST
Authentication	HTTP Basic credentials on the service. Custom headers are also possible. No signature
User-Agent	KoboToolbox external service followed by the server's domain and the service's ID
Timeout	30 seconds, in the kpi sender
Retries	3, after 60, 600 and 6,000 seconds. kpi retries 502, 503, 504 and timeouts
Failure reports	Optional email notifications
Source IPs	Not published. Self-hosted servers send from wherever they run

What a request looks like

With the JSON format, the body is the submission itself: one key per question in your form, using the question names, plus KoboToolbox metadata fields such as _id and _uuid. Groups and repeat groups change how the keys nest, and the subset-of-fields option sends only the questions you list. The exact shape depends on your form, so log a real submission before you write your parser.

The custom wrapper wraps the submission in a JSON template of your own. KoboToolbox replaces the placeholder %SUBMISSION% with the submission's JSON, so a template such as:

{ "source": "household-survey", "submission": %SUBMISSION% }

lets one endpoint tell several projects apart. With the XML format, the body is the submission's XML and the Content-Type is application/xml.

Inspect KoboToolbox submissions as they arrive. Point a REST Service at a Hookdeck Inspect URL to see the exact headers and body your form produces, with no account or setup required.

Setting up KoboToolbox webhooks

Pick a username and generate a long random password, for example with openssl rand -hex 32, and store both on your receiver.

1. Open the project in KoboToolbox and go to its Settings, then REST Services.
2. Register a new service with a name and your endpoint URL.
3. Choose JSON or XML.
4. Choose basic authentication and enter the username and password.
5. Optionally list a subset of fields, or (JSON only) add a custom wrapper, and enable email failure notifications.
6. Save, then submit a test record from a web form or KoboCollect.

To put Hookdeck in front of your service, create a Hookdeck source with the KoBoToolbox type, use its URL as the endpoint, and enter the same Username and Password on the source. The source accepts POST, which is the only method KoboToolbox uses.

For local development, hookdeck listen 3000 kobotoolbox --path /webhooks/kobotoolbox gives you a public URL that forwards to your machine.

Securing KoboToolbox webhooks

Each request carries Authorization: Basic base64(username:password). Decode it, split on the first colon, and compare the username and password with your stored values in constant time, evaluating both before you decide.

const crypto = require('crypto');
const express = require('express');

function safeEqual(a, b) {
  // timingSafeEqual throws on unequal lengths, so check first.
  return a.length === b.length && crypto.timingSafeEqual(a, b);
}

function verifyBasicAuth(header, username, password) {
  if (!username || !password) return false; // never accept when unconfigured
  const match = /^basic\s+([A-Za-z0-9+/]+={0,2})\s*$/i.exec(header || '');
  if (!match) return false;
  const decoded = Buffer.from(match[1], 'base64');
  const colon = decoded.indexOf(0x3a); // split on the first colon: passwords may contain ':'
  if (colon === -1) return false;
  const userOk = safeEqual(decoded.subarray(0, colon), Buffer.from(username, 'utf8'));
  const passOk = safeEqual(decoded.subarray(colon + 1), Buffer.from(password, 'utf8'));
  return userOk && passOk; // evaluate both before deciding
}

const app = express();

app.post('/webhooks/kobotoolbox', express.raw({ type: '*/*' }), (req, res) => {
  const ok = verifyBasicAuth(
    req.get('authorization'),
    process.env.KOBOTOOLBOX_WEBHOOK_USERNAME,
    process.env.KOBOTOOLBOX_WEBHOOK_PASSWORD
  );
  if (!ok) return res.sendStatus(401);

  if (req.is('application/json')) {
    let submission;
    try {
      submission = JSON.parse(req.body.toString('utf8'));
    } catch {
      return res.sendStatus(400);
    }
    // Deduplicate on submission._uuid, then store the answers.
  } else {
    // XML format: parse req.body with an XML parser.
  }

  res.sendStatus(200);
});

The same check in Python:

import base64
import binascii
import hmac
import os
import re

BASIC_PATTERN = re.compile(r"^basic\s+([A-Za-z0-9+/]+={0,2})\s*$", re.IGNORECASE)


def verify_basic_auth(header, username, password) -> bool:
    if not username or not password:
        return False  # never accept when unconfigured
    match = BASIC_PATTERN.match(header or "")
    if not match:
        return False
    try:
        decoded = base64.b64decode(match.group(1), validate=True)
    except (binascii.Error, ValueError):
        return False
    received_user, sep, received_password = decoded.partition(b":")  # first colon only
    if not sep:
        return False
    # Compare bytes: compare_digest raises TypeError on a non-ASCII str.
    user_ok = hmac.compare_digest(received_user, username.encode("utf-8"))
    password_ok = hmac.compare_digest(received_password, password.encode("utf-8"))
    return user_ok and password_ok


# verify_basic_auth(request.headers.get("authorization"),
#                   os.environ.get("KOBOTOOLBOX_WEBHOOK_USERNAME"),
#                   os.environ.get("KOBOTOOLBOX_WEBHOOK_PASSWORD"))

Basic credentials prove the sender knows the password. They don't cover the body and carry no timestamp, so a modified or replayed request still passes. Survey data is often personal or sensitive, so use an HTTPS endpoint, a long random password, and rotate it if it might have leaked.

Make KoboToolbox submissions production-ready. Hookdeck Event Gateway checks the Basic credentials, stores every submission, and keeps retrying delivery to your service long after KoboToolbox's three retries would have run out.

KoboToolbox webhook limitations and pain points

Three retries, then nothing

The Problem: If your endpoint is down for more than about two hours, submissions collected in the field never reach your system.

Why It Happens: KoboToolbox retries a failed delivery 3 times, after 60, 600 and 6,000 seconds. In the kpi code, only 502, 503, 504 and timeouts count as retriable, so a 500 or a 4xx fails at once.

Workarounds:

• Enable email failure notifications on the service.
• Reconcile by pulling recent submissions through the KoboToolbox API, since the data stays on the server.
• Return 503 rather than 500 for a temporary problem in your own code, so KoboToolbox will retry.

How Hookdeck Can Help: Hookdeck accepts and stores each submission as it arrives, then delivers to your service with retries you configure. Issues alert you when deliveries fail, and you can replay any submission once your service is back.

Edits never arrive

The Problem: A submission corrected after review in KoboToolbox stays wrong in your system.

Why It Happens: REST Services send data only on creation. Nothing is sent when a submission is edited, and there is no other webhook to subscribe to.

Workarounds:

• Treat the REST Service as the fast path, and sync edits on a schedule through the KoboToolbox API.

A static credential instead of a signature

The Problem: Anyone with the credentials can send submissions your endpoint will accept.

Why It Happens: Basic authentication is the only documented security option, and nothing is computed over the body.

Workarounds:

• Use a long random password over HTTPS, and a separate credential per project.
• Validate submissions against the form you expect before storing them.

How Hookdeck Can Help: Hookdeck's KoboToolbox source checks the Basic username and password on every request. Hookdeck still answers with a 200, but a request with missing or wrong credentials is recorded as failed verification and never reaches your destination.

Offline collection means bursts

The Problem: A team returning from the field syncs a day's submissions at once, and your endpoint receives them all within minutes.

Why It Happens: KoboCollect stores submissions offline and uploads them when it has a connection, and each one triggers its own REST Service request.

Workarounds:

• Keep the handler fast: authenticate, store, return 200.
• Size the endpoint for peak uploads, not the daily average.

How Hookdeck Can Help: Hookdeck absorbs the burst and delivers to your service at a max delivery rate you set.

Best practices

Deduplicate on _uuid

Retries can deliver the same submission more than once. Store _uuid (or _id) and skip repeats. See our guide to webhook idempotency.

Use the custom wrapper to label projects

If several projects post to one endpoint, wrap each one's submissions with a project label, so routing doesn't depend on guessing from question names.

Expect your form to change

Adding, renaming or regrouping questions changes the keys in the body. Parse defensively, and keep the raw submission so you can reprocess it after a form change.

Acknowledge within the timeout

The kpi sender waits 30 seconds, and a timeout counts as a retriable failure. Store the submission and return 200, then process it from a queue. See why to process webhooks asynchronously.

Conclusion

KoboToolbox REST Services send each new submission once, as JSON or XML, authenticated with Basic credentials and retried only three times over about two hours. Compare the credentials in constant time over HTTPS, deduplicate on _uuid, sync edits separately, and keep the endpoint fast enough to take a team's offline uploads at once.

Hookdeck Event Gateway checks the credentials and holds each submission until your service has processed it, so a longer outage on your side costs latency instead of field data.

Get started with Hookdeck for free and handle KoboToolbox webhooks reliably in minutes.