# Decompress Gzip and Zip Payloads with a Transformation Some providers send compressed payloads: a gzip file with `Content-Type: application/gzip`, a JSON body sent with `Content-Encoding: gzip`, or a `.zip` export. A [Transformation](/docs/transformations) receives these bodies as a `Buffer`. It can decompress them with the built-in `node:zlib` module and replace the body with JSON, so the [Event](/docs/events) becomes searchable and filterable and your [Destination](/docs/destinations) receives plain JSON. This guide covers gzip, zip archives, and the reverse: compressing a payload before delivery. The gzip and zip transformations have no dependencies, so you can paste them straight into the transformation editor. ## Decompress gzip ### Minimal version This transformation decompresses a gzip body that contains a JSON document: ```js const { gunzipSync } = require('node:zlib'); addHandler('transform', (request) => { request.body = JSON.parse(gunzipSync(request.body).toString('utf8')); request.headers['content-type'] = 'application/json'; delete request.headers['content-encoding']; return request; }); ``` It handles both ways providers send gzip: * A gzip file, sent with `Content-Type: application/gzip`. The transformation sets the content type to `application/json`. * A compressed JSON body, sent with `Content-Type: application/json` and `Content-Encoding: gzip`. The transformation removes the `content-encoding` header, because the body it returns is no longer compressed. Send a gzip file: ```bash gzip -c orders.json | curl -X POST "https://hkdk.events/src_xxxxxxxx" \ -H "Content-Type: application/gzip" \ --data-binary @- ``` Or a compressed JSON body: ```bash gzip -c orders.json | curl -X POST "https://hkdk.events/src_xxxxxxxx" \ -H "Content-Type: application/json" \ -H "Content-Encoding: gzip" \ --data-binary @- ``` In both cases, your destination receives `Content-Type: application/json` and the decompressed document: ```json { "orders": [{ "id": 1, "total": 99.5 }, { "id": 2, "total": 12 }] } ``` ### Handle any content and cap the size A small gzip file can decompress to gigabytes. This version caps the decompressed size with `maxOutputLength`, and accepts a JSON document, newline-delimited JSON, or plain text: ```js const { gunzipSync } = require('node:zlib'); // Largest size the payload may decompress to (protects against gzip bombs). const MAX_BYTES = 10 * 1024 * 1024; addHandler('transform', (request) => { const text = gunzipSync(request.body, { maxOutputLength: MAX_BYTES }).toString('utf8'); let body; try { body = JSON.parse(text); // a JSON document } catch { const lines = text.split('\n').filter((line) => line.trim()); try { body = lines.map((line) => JSON.parse(line)); // newline-delimited JSON } catch { body = { text }; // anything else: wrap the text } } request.body = body; request.headers['content-type'] = 'application/json'; delete request.headers['content-encoding']; return request; }); ``` | Decompressed content | Body your destination receives | | --- | --- | | A JSON document | The document | | Newline-delimited JSON | An array with one item per line, such as `[{"id":1},{"id":2}]` | | Anything else | The text wrapped in an object, such as `{"text":"plain log line"}` | If the content decompresses to more than `MAX_BYTES`, `gunzipSync` throws `RangeError: Cannot create a Buffer larger than 10485760 bytes` and the transformation fails, which opens a [transformation issue](/docs/transformations#transformation-issues). ## Unzip an archive A zip archive can hold several files. This dependency-free transformation reads the archive's central directory, decompresses each entry with `inflateRawSync` from `node:zlib`, and returns the files as JSON. JSON files are parsed, other UTF-8 files become strings, and binary files are base64-encoded. ```js const { inflateRawSync } = require('node:zlib'); const { isUtf8 } = require('node:buffer'); // Largest size an entry may decompress to (protects against zip bombs). const MAX_ENTRY_BYTES = 10 * 1024 * 1024; // Reads a .zip archive (stored or deflated entries) into [{ name, data }]. function unzip(zip) { // The end-of-central-directory record sits at the end of the archive. let eocd = zip.length - 22; while (eocd >= 0 && zip.readUInt32LE(eocd) !== 0x06054b50) eocd--; if (eocd < 0) throw new Error('Not a zip archive'); const count = zip.readUInt16LE(eocd + 10); let offset = zip.readUInt32LE(eocd + 16); const files = []; for (let i = 0; i < count; i++) { if (zip.readUInt32LE(offset) !== 0x02014b50) throw new Error('Corrupt zip directory'); const method = zip.readUInt16LE(offset + 10); const compressedSize = zip.readUInt32LE(offset + 20); const nameLength = zip.readUInt16LE(offset + 28); const extraLength = zip.readUInt16LE(offset + 30); const commentLength = zip.readUInt16LE(offset + 32); const localHeader = zip.readUInt32LE(offset + 42); const name = zip.toString('utf8', offset + 46, offset + 46 + nameLength); offset += 46 + nameLength + extraLength + commentLength; if (name.endsWith('/')) continue; // folder entry const start = localHeader + 30 + zip.readUInt16LE(localHeader + 26) + zip.readUInt16LE(localHeader + 28); const raw = zip.subarray(start, start + compressedSize); let data; if (method === 0) data = raw; else if (method === 8) data = inflateRawSync(raw, { maxOutputLength: MAX_ENTRY_BYTES }); else throw new Error(`Unsupported compression method ${method} for ${name}`); files.push({ name, data }); } return files; } addHandler('transform', (request) => { const files = unzip(request.body); request.body = { files: files.map(({ name, data }) => { let content; if (name.endsWith('.json')) content = JSON.parse(data.toString('utf8')); else if (isUtf8(data)) content = data.toString('utf8'); else content = { base64: data.toString('base64') }; return { name, size: data.length, content }; }), }; request.headers['content-type'] = 'application/json'; return request; }); ``` Send an archive: ```bash curl -X POST "https://hkdk.events/src_xxxxxxxx" \ -H "Content-Type: application/zip" \ --data-binary @export.zip ``` For an archive with a JSON file, a text file and a binary file, your destination receives: ```json { "files": [ { "name": "report.json", "size": 70, "content": { "order_id": 1234, "total": 99.5, "items": [{ "sku": "abc", "qty": 2 }] } }, { "name": "notes/readme.txt", "size": 72, "content": "héllo from the archive\nhéllo from the archive\nhéllo from the archive\n" }, { "name": "raw/logo.bin", "size": 10, "content": { "base64": "AP+JUE5HDQoaCg==" } } ] } ``` The reader supports the two compression methods zip files use in practice, stored and deflate. It doesn't support encrypted entries or ZIP64 archives; those throw an error and fail the transformation. ### Shorter alternative with fflate If you'd rather use a library, [fflate](https://github.com/101arrowz/fflate) reads zip archives in one call. Install it, write the transformation as an ES module, and [bundle it](/docs/transformations#bundle-dependencies) with esbuild: ```bash npm install fflate npm install --save-dev esbuild ``` ```js import { unzipSync, strFromU8 } from 'fflate'; export default { transform(request) { const files = unzipSync(request.body); request.body = { files: Object.keys(files), orders: JSON.parse(strFromU8(files['orders.json'])), }; request.headers['content-type'] = 'application/json'; return request; }, }; ``` ```bash npx esbuild unzip.js --bundle --format=esm --platform=node --outfile=dist/unzip.js ``` For an archive containing `orders.json` and `notes/readme.txt`, your destination receives: ```json { "files": ["orders.json", "notes/readme.txt"], "orders": [{ "id": 1, "total": 99.5 }] } ``` Use fflate's synchronous functions, such as `unzipSync` and `gunzipSync`. Its callback and async functions run on worker threads, which transformations don't support. `unzipSync` has no limit on the decompressed size, so an oversized archive runs into the transformation's memory limit. If senders aren't trusted, prefer the dependency-free reader above, which caps each entry with `maxOutputLength`. ## Compress a payload before delivery To send a compressed payload to your destination, return the gzip bytes and set `content-encoding`: ```js const { gzipSync } = require('node:zlib'); addHandler('transform', (request) => { request.body = gzipSync(JSON.stringify(request.body)); request.headers['content-encoding'] = 'gzip'; return request; }); ``` For a JSON request such as `{"order_id":1042,"total":99.5}`, your destination receives the gzip bytes with `Content-Type: application/json` and `Content-Encoding: gzip`, Make sure your destination decompresses request bodies sent with `Content-Encoding: gzip`. Because the transformation returns bytes, Hookdeck stores and delivers the event as a binary payload, byte for byte. The event's body is no longer searchable as JSON, so put any [filters](/docs/filters) before this transformation. ## Limits and pitfalls * Body size. Binary bodies larger than 20 MiB arrive as `null` and are delivered unchanged. Check for `null` if you might receive large files. * Decompressed size. A single decompression call returns at most 32 MiB, and each execution has 128 MB of memory. Set `maxOutputLength` to a lower limit that fits your data, as in the examples above. * Compression input. A single compression call such as `gzipSync` accepts up to 8 MiB. * Execution time. Transformations must finish within 1 second. Decompressing a few megabytes takes milliseconds. For the other built-in modules transformations can use, see [Transformation Node.js compatibility](/docs/transformations#nodejs-compatibility).