How Rotate keeps 4 ecommerce projects on track by managing webhooks with Hookdeck

Company
Studio Rotate
Location
England
Industry
Development Agency
Size
11-50
Hookdeck's monitoring and error recovery of webhooks gives me confidence to successfully tackle multiple projects at the same time.
Ben Cooper
Tech Lead

Studio Rotate is a premier development agency studio with a focus on ecommerce. Jim Tattersall, co-founder and CTO, is committed to leading projects at the cutting edge, working with the latest stack, and pushing the boundary of digital experiences. Brands such as Chilly’s, Big Green Egg, Wild Cosmetics, and Bamford all trust Studio Rotate to create their online experience and deliver beyond what they themselves could imagine.

That being said, being at the cutting edge has its own set of challenges. One challenge, creating an outstanding experience, is built on top of many integrations, often 10 to 12 per ecommerce for flows such as orders or fulfillment. This leaves them exposed to greater risk of webhook failure, which would cause issues for downstream events.

Visibility instills confidence that everything is working at all times.
Project budget and time are spent working on new development instead of fixing fires.
Simple to use so that team members from different units and technical proficiency can work together to manage webhooks.
THE PROBLEM

Project time spent fixing instead of building

"Fixing issues caused by webhook failures takes away project's precious development time."

Time spent fire-fighting not building

At the mercy of the webhook provider

Rotate relies onCommerce Layer as part of the his composable commerce stack. However, their webhook behavior and rules can cause lots of problems.

For example, after a certain number of failed events Commerce Layer will do a “short-circuit” and stop everything. Imagine if a service went down during the night causing webhooks to fail. Suddenly their flows stops, order fulfillment doesn’t go out the door until a fix is deployed, and Ben ends up in a scenario where he has to manually push data over a few hours to reconcile the problem.

That’s a full day of work spent fixing a scenario that he can’t do anything about.

Fixing fires

Fires are the priority. Ben has to deal with issues before doing anything else, only to realize that dropping everything in order to fix things leaves him no time to work on project features.

A webhook error can take a lot of time to recover (as we saw earlier, up to 8 hours) because of a lack of visibility that makes it hard to troubleshoot. His workflow to solve the errors included first finding the error, then troubleshooting, and starting a reconciliation by rerunning all the orders manually from his local machine. And the pains don't end there, now he has to think about throttling… it’s unsustainable.

Impact compound when managing with 3-4 projects

Spending his time fixing errors on one project is hard enough, imagine if he has 3 projects to maintain success. In the back of his mind he is always thinking about the stack to create the best solution for their client as well as deliver the project on schedule, and solve errors as part of their maintenance and retainer contracts.

To keep projects on schedule, he believed webhook problems were something valuable worth solving. He tried a few solutions to help reduce the fire-fighting. He invested time to learn and use AWS SQS/EventBridge but realized that except for the person setting it up, it was difficult for others to grasp what was happening and take advantage of it.

The alternative was to use RabbitMQ, but he quickly shied away from the solution because he would have the overhead of maintaining the services.

So what now? Ben kept looking for a solution to manage webhooks that gives them visibility and that is easy to use.

THE SOLUTION

How Hookdeck keeps projects on track

"I feel Hookdeck gives me 10x reliability."

On Ben’s search for a solution that would allow him to spend less time on webhook failures and more time delivering more value, as well as ability to be transparent during incidents with clients, he created a wish list. It consisted of:

Ease of use

  • Set up is quick and painless
  • Able to onboard devs as fast as possible and get project out of the door
  • Can be owned by the whole team, including operation and support

Usefulness

  • Reduce errors that impact downstream events
  • Alerts to help to recover in a timely manner
  • Logging to communicate what is happening to clients

Ben came across a Hookdeck blog post and shared the solution with Jim. After getting his green light and talking with Alex to validate that Hookdeck is the solution to manage his webhooks, he then proceeded to give Hookdeck a shot.

Webhook visibility

When I notice a problem, if it’s not Hookdeck, then I have a lot confidence that it’s on the webhook provider’s end.

As soon as he started using Hookdeck, Ben realized how visibility was a game changer for him and his team. No need for an external logging system anymore. His workflow to finding answers boils down to:

  • “what happened with order_id
  • Search the order_id in Hookdeck
  • Check the status of all the calls to see if there was any problem

Now when we talk about managing over 10 integrations, the visibility allows all his team to see the same “schema” and understand what happened with each customer workflow from start to finish.

Workflow recovery from errors is frictionless

Studio Rotate often has a retainer contract on their ongoing project to fix bugs that they have to solve within 12 hours. They use Sentry to capture small issues but rely on Hookdeck to tackle big integration issues.

Recovering from errors went from big headaches to an approachable process

  1. Receive an alert when webhook fails
  2. Use the dashboard to identify the problem and work on a fix
  3. Once it’s deployed, use the bulk retry to reconcile everything

A 8-hour process now takes minutes. It went from having no visibility and manually doing every step of troubleshooting to taking action before it impacts the customer.

Gradually using Hookdeck in projects: The staged approach

Hookdeck is so easy to adopt. No code. Low risk. Incremental.

Ben reassures his clients by suggesting to take one step at the time:

  1. Add Hookdeck to a non-mission critical workflow or new integration
  2. Show how Hookdeck’s observability and error recovery is beneficial
  3. Add new integrations on Hookdeck one by one

At the end of the process, everyone is happy. Ben and his team are solving clients’ problems quicker than ever while the clients get their time’s worth with more new developments now that there’s a lot less fire-fighting.

THE RESULT

Customers enjoy what Studio Rotate does best: Outstanding ecommerce

"Now when there’s a project with webhooks, we strongly suggest our clients have Hookdeck, and for the big projects, we list Hookdeck as a requirement."

Build confidence with their clients through transparency

With Hookdeck, Ben and his team are able to show data such as time of events and reasons for failures to the clients when they have questions and requests, providing clarity and building trust.

The whole team is contributing

Hookdeck dashboard makes managing webhooks is accessible and enable everyone from the team to move the needle.

  1. Devs do not run into the AWS wall and can start contributing to projects, great for onboarding juniors.
  2. Support has an intuitive way to find the information and either replay to fix it or report to customers what’s happening.
  3. Lead can follow the progress of the integrations and monitor the status of webhooks.

Laser focus on digital experience and business value

Ben noticed that he got his head out of the water and doesn’t deal with fire-fighting anymore. He is able to be more productive by tackling the long list of new developments that’s been slowly piling up and delivering value where he knows Studio Rotate shines.

Read more customer stories

Start developing with Hookdeck