# Apache Kafka

Publish events to an Apache Kafka topic. Authentication uses SASL (PLAIN, SCRAM-SHA-256, or SCRAM-SHA-512) and TLS is enabled by default.

## Creating a Kafka Destination

```sh
curl 'https://api.outpost.hookdeck.com/2025-07-01/tenants/<TENANT_ID>/destinations' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer <API_KEY>' \
--data '{
  "type": "kafka",
  "topics": ["orders"],
  "config": {
    "brokers": "broker1.example.com:9092,broker2.example.com:9092",
    "topic": "events",
    "sasl_mechanism": "scram-sha-256"
  },
  "credentials": {
    "username": "outpost",
    "password": "<SASL_PASSWORD>"
  }
}'

```

## Configuration

### Config

| Field | Type | Required | Description |
| --- | --- | --- | --- |
| `config.brokers` | string | Yes | Comma-separated list of Kafka broker addresses (e.g., `broker1:9092,broker2:9092`) |
| `config.topic` | string | Yes | The Kafka topic to publish messages to |
| `config.sasl_mechanism` | string | Yes | SASL mechanism: `plain`, `scram-sha-256`, or `scram-sha-512` |
| `config.tls` | string | No | Enable TLS: `true` or `false` (default: `true`) |
| `config.partition_key_template` | string | No | JMESPath expression for the message partition key (default: the event ID) |

### Credentials

| Field | Type | Required | Description |
| --- | --- | --- | --- |
| `credentials.username` | string | Yes | SASL username |
| `credentials.password` | string | Yes | SASL password |

## Message Format

Events are published as Kafka messages:

* Key: The partition key — evaluated from `partition_key_template`, falling back to the event ID
* Value: The event's `data` field as raw JSON (no wrapper)
* Headers: `content-type: application/json` plus system metadata (`event-id`, `topic`, `timestamp`) and any event metadata from the published event

Messages are routed using a hash balancer over the key, so events with the same key land on the same partition.

### Example

Publishing this event:

```json
{
  "topic": "orders",
  "data": { "order_id": "123", "status": "created" },
  "metadata": { "source": "checkout-service" }
}

```

Results in:

Key: `evt_123`

Value:

```json
{"order_id": "123", "status": "created"}

```

Headers:

| Header | Value |
| --- | --- |
| `content-type` | `application/json` |
| `event-id` | `evt_123` |
| `topic` | `orders` |
| `timestamp` | `1704067200` |
| `source` | `checkout-service` |

## Partition Key

By default, the event ID is used as the partition key. Customize it with a JMESPath expression evaluated against the event payload (`metadata` and `data`):

```json
{
  "config": {
    "brokers": "broker1.example.com:9092",
    "topic": "events",
    "sasl_mechanism": "scram-sha-256",
    "partition_key_template": "data.customer_id"
  }
}

```

If the template is empty, fails to evaluate, or returns an empty value, the event ID is used as the fallback key.

## SASL Authentication

Outpost supports three SASL mechanisms, set via `config.sasl_mechanism`:

* `plain` — SASL/PLAIN
* `scram-sha-256` — SASL/SCRAM with SHA-256
* `scram-sha-512` — SASL/SCRAM with SHA-512

The `username` and `password` credentials are used for all mechanisms.

## TLS

TLS is enabled by default. Set `config.tls` to `"false"` to connect without TLS (for example, against a local broker):

```json
{
  "type": "kafka",
  "topics": ["orders"],
  "config": {
    "brokers": "localhost:9092",
    "topic": "events",
    "sasl_mechanism": "plain",
    "tls": "false"
  },
  "credentials": {
    "username": "admin",
    "password": "admin-secret"
  }
}

```