Hookdeck Evals: Finding the friction between your agent and Hookdeck

Over the last few months we've extended our CLI and added and improved MCPs for Event Gateway and Outpost, made our docs machine-readable, and introduced agent skills for our own products, and 160+ webhook providers.

The goal: to make every Hookdeck product work for an AI agent as well as it works for you.

You should be able to hand Claude Code or Codex a Hookdeck project and ask it to receive Stripe webhooks, or fix a failing destination, and do so without you having to step in.

Most of the time that works, but sometimes it doesn't. The problem is how we find out.

When a person hits a docs page that's wrong, or a button that doesn't do what they expected, some of them tell us. They post in Slack or open an issue. But agents don't. An agent that can't find what it works around the issue, or makes a guess, then reports success. A bug doesn't get filed because nobody knows there was one, the friction stays, and we never hear about it.

So we built Hookdeck Evals to go looking for it. It is a set of scenarios we run against coding agents, scored on what they built rather than on what they said they built, and we publish the results, pass or fail.

What friction looks like

We gave an agent a Stripe signing secret, put it in the project it was working in, and asked it to receive Stripe webhooks through Hookdeck.

It configured the source with whsec_xxx and told us the job was done.

We ran that scenario six times, once per model with our skills loaded and once without. One of those six runs used the secret it had been given. The rest used whsec_new, whsec_replace_me, whsec_test_secret, or a placeholder of their own. To Event Gateway, the signing secrets looked fine because they're just a string, but once an event arrived it failed.

If an agent had set that up for you, you might have only found out in production, and the agent would get the blame. But we don't think that's where the problem is. If nothing in Hookdeck tells you the secret you pasted is a placeholder, and nothing warns you afterwards, that's something we should fix. So we opened an issue, to do just that.

This is exactly the kind of thing Evals exists to find.

What is on the scoreboard

Evals run nineteen common scenarios across multiple models in real, live Hookdeck projects with an API key and the CLI installed. Scoring reads the project back through our own API and sends events through it, so a pass means the thing the agent built works.

Scenarios are grouped by what you would be asking an agent to do: Build something new, Troubleshoot something broken, or Recover from something that already went wrong. Open any model and you can drill into a single scenario and see exactly what was asked and what happened.

Why do this in public

We could have run this internally and fixed what we found, but we want you to see the work and be held accountable to making the improvements.

The findings from each run are filed as issues, whether the fix belongs in the docs, the API, the CLI, the skills, or the product itself. We record every fix and what the follow-up run showed. The scenarios, the scoring, and every past result are in hookdeck/evals, which builds on supabase/evals.

What it doesn't cover yet

Eleven of the nineteen scenarios were passed by every model, so the set doesn't discriminate as sharply as we want. Console, our free webhook inspector, has no scenarios and our MCP server isn't in the comparison, yet. We will keep adding scenarios, covering more of our products, and adding more models.

Tell us where your agent got stuck

Evals finds the friction we thought to look for. The gaps we haven't found are the ones we need most. If an agent has gone wrong on Hookdeck, whether it invented an endpoint, misread a docs page, or burned half its context hunting for something that wasn't there, we want to know about it. Please open an issue.

The scoreboard and changelog lives at hookdeck.com/evals if you'd like to follow our progress.