Handle big webhooks
Fun fact: GoCardless only allow you 10s to process a webhook request. By design, apparently.
If you take longer, they assume it failed (even though your server may chug through and complete the processing). Guess what they do then? They keep sending it all the time!
In my experience (there are over 200 organisations using this integration) webhooks are usually processed in much less than 10s, however, sometimes you can get a rush of them at once, e.g. I've had a webhook that's 80kB with hundreds of events in it. This takes over 10s to process.
There are two solutions to this problem
the sensible solution
GC implement a per-webhook config option: "Max events in one webhook". Setting that to a suitable number would ensure the requests could be handled within 10s.
This solves the problem for this and other integrations and means we can all still use the handy manage.gocardless.com interface for monitoring webhooks.
I have requested this feature.
the ugly solution
We have to accept-and-queue the webhook events ourselves, and then asynchronously process our queue. This means:
- We ideally need a way to monitor that queue, reprogramming much of the functionality in manage.gocardless.com ourselves.
- manage.gocardless.com will likely just show 'success' by all the webhooks, but it just means that they were accepted onto the local queue
If anyone is interested in funding this work, it's about 1-2 days worth, you can contact me (or ping me on MM chat @artfulrobot). But as I say, a far more sensible solution would be for GC to implement some sensible, configurable limits on the data they send.