Decompress Gzip and Zip Payloads with a Transformation

Some providers send compressed payloads: a gzip file with Content-Type: application/gzip, a JSON body sent with Content-Encoding: gzip, or a .zip export. A Transformation receives these bodies as a Buffer. It can decompress them with the built-in node:zlib module and replace the body with JSON, so the Event becomes searchable and filterable and your Destination receives plain JSON.

This guide covers gzip, zip archives, and the reverse: compressing a payload before delivery. The gzip and zip transformations have no dependencies, so you can paste them straight into the transformation editor.

Decompress gzip

Minimal version

This transformation decompresses a gzip body that contains a JSON document:

const { gunzipSync } = require('node:zlib');
addHandler('transform', (request) => {
  request.body = JSON.parse(gunzipSync(request.body).toString('utf8'));
  request.headers['content-type'] = 'application/json';
  delete request.headers['content-encoding'];
  return request;
});

It handles both ways providers send gzip:

• A gzip file, sent with Content-Type: application/gzip. The transformation sets the content type to application/json.
• A compressed JSON body, sent with Content-Type: application/json and Content-Encoding: gzip. The transformation removes the content-encoding header, because the body it returns is no longer compressed.

Send a gzip file:

gzip -c orders.json | curl -X POST "https://hkdk.events/src_xxxxxxxx" \
  -H "Content-Type: application/gzip" \
  --data-binary @-

Or a compressed JSON body:

gzip -c orders.json | curl -X POST "https://hkdk.events/src_xxxxxxxx" \
  -H "Content-Type: application/json" \
  -H "Content-Encoding: gzip" \
  --data-binary @-

In both cases, your destination receives Content-Type: application/json and the decompressed document:

{ "orders": [{ "id": 1, "total": 99.5 }, { "id": 2, "total": 12 }] }

Handle any content and cap the size

A small gzip file can decompress to gigabytes. This version caps the decompressed size with maxOutputLength, and accepts a JSON document, newline-delimited JSON, or plain text:

const { gunzipSync } = require('node:zlib');

// Largest size the payload may decompress to (protects against gzip bombs).
const MAX_BYTES = 10 * 1024 * 1024;

addHandler('transform', (request) => {
  const text = gunzipSync(request.body, { maxOutputLength: MAX_BYTES }).toString('utf8');

  let body;
  try {
    body = JSON.parse(text); // a JSON document
  } catch {
    const lines = text.split('\n').filter((line) => line.trim());
    try {
      body = lines.map((line) => JSON.parse(line)); // newline-delimited JSON
    } catch {
      body = { text }; // anything else: wrap the text
    }
  }

  request.body = body;
  request.headers['content-type'] = 'application/json';
  delete request.headers['content-encoding'];
  return request;
});

If the content decompresses to more than MAX_BYTES, gunzipSync throws RangeError: Cannot create a Buffer larger than 10485760 bytes and the transformation fails, which opens a transformation issue.

Unzip an archive

A zip archive can hold several files. This dependency-free transformation reads the archive's central directory, decompresses each entry with inflateRawSync from node:zlib, and returns the files as JSON. JSON files are parsed, other UTF-8 files become strings, and binary files are base64-encoded.

const { inflateRawSync } = require('node:zlib');
const { isUtf8 } = require('node:buffer');

// Largest size an entry may decompress to (protects against zip bombs).
const MAX_ENTRY_BYTES = 10 * 1024 * 1024;

// Reads a .zip archive (stored or deflated entries) into [{ name, data }].
function unzip(zip) {
  // The end-of-central-directory record sits at the end of the archive.
  let eocd = zip.length - 22;
  while (eocd >= 0 && zip.readUInt32LE(eocd) !== 0x06054b50) eocd--;
  if (eocd < 0) throw new Error('Not a zip archive');

  const count = zip.readUInt16LE(eocd + 10);
  let offset = zip.readUInt32LE(eocd + 16);
  const files = [];
  for (let i = 0; i < count; i++) {
    if (zip.readUInt32LE(offset) !== 0x02014b50) throw new Error('Corrupt zip directory');
    const method = zip.readUInt16LE(offset + 10);
    const compressedSize = zip.readUInt32LE(offset + 20);
    const nameLength = zip.readUInt16LE(offset + 28);
    const extraLength = zip.readUInt16LE(offset + 30);
    const commentLength = zip.readUInt16LE(offset + 32);
    const localHeader = zip.readUInt32LE(offset + 42);
    const name = zip.toString('utf8', offset + 46, offset + 46 + nameLength);
    offset += 46 + nameLength + extraLength + commentLength;
    if (name.endsWith('/')) continue; // folder entry

    const start =
      localHeader + 30 + zip.readUInt16LE(localHeader + 26) + zip.readUInt16LE(localHeader + 28);
    const raw = zip.subarray(start, start + compressedSize);
    let data;
    if (method === 0) data = raw;
    else if (method === 8) data = inflateRawSync(raw, { maxOutputLength: MAX_ENTRY_BYTES });
    else throw new Error(`Unsupported compression method ${method} for ${name}`);
    files.push({ name, data });
  }
  return files;
}

addHandler('transform', (request) => {
  const files = unzip(request.body);
  request.body = {
    files: files.map(({ name, data }) => {
      let content;
      if (name.endsWith('.json')) content = JSON.parse(data.toString('utf8'));
      else if (isUtf8(data)) content = data.toString('utf8');
      else content = { base64: data.toString('base64') };
      return { name, size: data.length, content };
    }),
  };
  request.headers['content-type'] = 'application/json';
  return request;
});

Send an archive:

curl -X POST "https://hkdk.events/src_xxxxxxxx" \
  -H "Content-Type: application/zip" \
  --data-binary @export.zip

For an archive with a JSON file, a text file and a binary file, your destination receives:

{
  "files": [
    { "name": "report.json", "size": 70, "content": { "order_id": 1234, "total": 99.5, "items": [{ "sku": "abc", "qty": 2 }] } },
    { "name": "notes/readme.txt", "size": 72, "content": "héllo from the archive\nhéllo from the archive\nhéllo from the archive\n" },
    { "name": "raw/logo.bin", "size": 10, "content": { "base64": "AP+JUE5HDQoaCg==" } }
  ]
}

The reader supports the two compression methods zip files use in practice, stored and deflate. It doesn't support encrypted entries or ZIP64 archives; those throw an error and fail the transformation.

Shorter alternative with fflate

If you'd rather use a library, fflate reads zip archives in one call. Install it, write the transformation as an ES module, and bundle it with esbuild:

npm install fflate
npm install --save-dev esbuild

import { unzipSync, strFromU8 } from 'fflate';

export default {
  transform(request) {
    const files = unzipSync(request.body);
    request.body = {
      files: Object.keys(files),
      orders: JSON.parse(strFromU8(files['orders.json'])),
    };
    request.headers['content-type'] = 'application/json';
    return request;
  },
};

npx esbuild unzip.js --bundle --format=esm --platform=node --outfile=dist/unzip.js

For an archive containing orders.json and notes/readme.txt, your destination receives:

{ "files": ["orders.json", "notes/readme.txt"], "orders": [{ "id": 1, "total": 99.5 }] }

Use fflate's synchronous functions, such as unzipSync and gunzipSync. Its callback and async functions run on worker threads, which transformations don't support. unzipSync has no limit on the decompressed size, so an oversized archive runs into the transformation's memory limit. If senders aren't trusted, prefer the dependency-free reader above, which caps each entry with maxOutputLength.

Compress a payload before delivery

To send a compressed payload to your destination, return the gzip bytes and set content-encoding:

const { gzipSync } = require('node:zlib');

addHandler('transform', (request) => {
  request.body = gzipSync(JSON.stringify(request.body));
  request.headers['content-encoding'] = 'gzip';
  return request;
});

For a JSON request such as {"order_id":1042,"total":99.5}, your destination receives the gzip bytes with Content-Type: application/json and Content-Encoding: gzip, Make sure your destination decompresses request bodies sent with Content-Encoding: gzip.

Because the transformation returns bytes, Hookdeck stores and delivers the event as a binary payload, byte for byte. The event's body is no longer searchable as JSON, so put any filters before this transformation.

Limits and pitfalls

• Body size. Binary bodies larger than 20 MiB arrive as null and are delivered unchanged. Check for null if you might receive large files.
• Decompressed size. A single decompression call returns at most 32 MiB, and each execution has 128 MB of memory. Set maxOutputLength to a lower limit that fits your data, as in the examples above.
• Compression input. A single compression call such as gzipSync accepts up to 8 MiB.
• Execution time. Transformations must finish within 1 second. Decompressing a few megabytes takes milliseconds.

For the other built-in modules transformations can use, see Transformation Node.js compatibility.