Phillip Malboeuf is the Lead Developer at Dispatch Coffee, which means he is in charge of everything related to technology. He is the one being called upon, whether it's to build internal tooling to scale operations or to improve the website and other digital touchpoints that delight customers with a top-notch experience.
"There's a lot of things to build, the team and I have many projects in mind. That's why when I build something, I care about reliability but speed is of the essence"
He relies on webhooks for many of the workflows he has built. For example:
- The subscription order workflow that integrates Shopify and ReCharge's webhooks
- The gift subscription management that integrates with Recharge webhooks.
When there's an error, often it is the customer support that will notify Phil as they are the first person a client contacts when something goes wrong. A great example of this is when a customer contacts support when they don't receive or have an issue with their gift card.
The issues were often relatively simple to fix. It took Phil on average between 15 to 30 minutes to troubleshoot. Sometimes it could take longer, but overall it didn't feel like a reason to sound the alarm. "The number of 30 minutes I spent troubleshooting issues was starting to add up as we were scaling, but it remained very manageable".
By browsing the logs (when he was lucky enough that the API provider provided them), small issues were easy to deal with - but sometimes, the issue was out of his control. Going back to his subscription order workflow, ReCharge imposes a rate-limiting which causes a lot of webhook to fail. It's a good thing that Shopify has a retry logic, as this problem always happened on Thursday at midnight... Not the ideal time to be getting an error.
"Troubleshooting wasn't a big issue. It could be annoying that some of the providers didn't have logs and they all have different logic but at the end of the day they are very useful and I could still keep track of them."
Phil used to work with our co-founder, Alex, as a freelance at Rachel. Even though their project was finished, they still kept in touch regularly over the years. During one of their many interesting discussions, they ended on the topic of webhook failures. Alex shared with Phil how he was in a similar situation and that unless he builds a queuing system, he can expect to do more troubleshooting because errors starts to come up more frequently as the company continues to scale and he continues to introduce new integrations.
Alex mentioned how he was working on Hookdeck, a product that will solve the exact problems they were discussing. He offered Phil the opportunity to test it out with a non-mission-critical webhook. At that point, Phil was curious so he decided to test Hookdeck and see how a webhook infrastructure could help him. He quickly realized he was able to troubleshoot much quicker among many other things (see below). For him, just the troubleshooting was enough to convince him to add all of his remaining webhooks on Hookdeck.
"I didn't realize that having a webhook infrastructure could save me this much time and headaches. Hookdeck really made handling and managing webhooks integrations super simple."
With Hookdeck, Phil was able to decide the retry logic and failure alert logic for each webhook connection. He would define his logic based on the importance of the webhook and its use case. It went something along the lines of every minute up to 10 times, or every 30 minutes up to 5 times. This was extremely helpful in dealing with the subscription order workflow. Instead of dropping all of the events and solely relying on Shopify to retry all his webhooks, he was able to set a very generous retry logic in Hookdeck. Hookdeck would accept all the events from Shopify and deliver and retry them based on Phil's opinion on what is better retry logic for his system.
"After setting the custom logic for each of my webhook connections, I felt at ease knowing that everything was handled the way I wanted and if there was an error, I would be notified in a very timely manner"
With Hookdeck, Phil was able to set up an event filter (on the header and/or the request) for each of his webhook connection by writing a simple JSON snippet. After monitoring his webhooks for two weeks on Hookdeck, he realized he was receiving a lot (99%) of webhooks that were not useful for his workflows. For example, he could be waiting for a product inventory= 0 to trigger the sold-out SKU workflow, but he would receive a webhook from Shopify's event "inventory/level" for each inventory change for each SKU. That ends up being a lot of webhooks...
"It was very simple to set an event filter for each of my webhook connections. Now when I troubleshoot using the dashboard, I only see the useful webhook events"
Since putting all his webhooks connections in Hookdeck, Phil has already added five new webhook connections. Spending his energy on building his applications rather than fixing production-related issues is starting to pay off. Being the sole owner of the tech makes him invaluable to the progress of Dispatch Coffee. For every additional feature, he can get out of the door, his impact on the business is multiplied.
"I can't quantify the number of hours it saved me but I know it's a lot because since I've been using Hookdeck, I've been able to tackle more projects with confidence. "
Hookdeck is an integral part of Phil's workflow when he is working with webhooks. He knows that all his integrations will end up on Hookdeck and can piggyback on its reliability to make it work. "The gift subscriptions and referral program are features that rely on Hookdeck to work."
"I genuinely look forward to seeing what Hookdeck is doing. Every time they release a feature, it helps me. I'm going to need more help as the business continues to scale!"