Looking for input for our Webhooks implementation

Warren Sherliker

Hi, 


Just to backup Steve's point here.


Please don't do anything outside the norm for these things. We also use modsecurity on our own servers as do loads of providers - it's standard. If what you are doing prevents/makes difficult those users from consuming webhooks then it is not a big step to assume you need not to do so.


Cheers


Warren

Ben Wong

As always, thanks for your feedback. We know we'll need to make changes to the Webhook implementation to help developers be successful. We added an endpoint last week for testing the event payload based on your feedback.

 

The intent isn't to go outside the norm. In fact, that's the reason why we want to adhere to the CloudEvents specification so that we can follow a common path for publishing events.

 

For now, we're going to do some more research into the user-agent setting and get more feedback from others about the content-type.

 

Bobby Earl

Hey all. I've been closely watching this thread and the great feedback you're providing.


In regards to some of that feedback, I wanted to highlight my recent announcement. Specifically a couple of our new entires in the App Showcase. While developing the NodeJS and PHP webhook demos, I also encountered the need for accommodating the `content-type`.


As it relates to a few of you in this thread, I did want to call out our testing on several shared and dedicated hosting providers with a wide range of ModSecurity setups. As an example of a quick win, I did have a really good experience reaching out to HostGator and having them whitelist the new content-type without issue while other hosts expose the ability to adjust accepted content-types without having to completely disable ModSecurity.


Showing that it's possible to stick with open source standards without sacrificing security, all while using real-world hosting providers was important to us.


Thanks again for the great conversation.


Bobby

Warren Sherliker

Hi,


Sorry but I don't agree. Making barriers to adoption like having to contact hosting providers makes no sense whatsoever. There are a lot of standards out there that you could follow but the basics are that you want people/developers/clients to be able to use your solutions easily. If you deliberately follow a standard that makes that difficult i.e. they have more hoops to go through then you are putting up barriers to adoption... Which quite obviously is against your own interest and that of your clients.


Please use a normal application/json header... the client loses nothing in terms of functionality... I am at a loss to what you think is gained by the more specific content-type other than some internal dev teams being more happy that they could implement some newer standard that the industry is not ready for.


Cheers


Warren

Bobby Earl

Warren,


To help keep this discussion real and not purely academic, what was your personal experience in setting up webhook subscriptions with your hosting provider?


I do not disagree, having to contact a hosting provider to enable a new content-type is not an ideal solution.  To make sure we're correctly scoping the problem though, this situation only applies to those using the (current) default ModSecurity rules (CRS) applied with PHP on a hosting provider that does not allow direct editing of any of the ModSecurity rules.


To be honest, that seems like very narrow subset of consumers where conditions aren't, again currently, ideal for adoption.  Anyone using .NET, Java, NodeJS, Go, Ruby, Rust, Python, or PHP (with editable ModSecurity rules) should not experience any difficulty.  Using CloudEvents has lots of benefits and includes an ever growing list of contributors and adopters.


As adoption continues to grow, I am excited to see `cloudevents+json` and `cloudevents-batch+json` make their way in to the several default ModSecurity rulesets available.  As an example, look at all the other content-type changes in CRS.

Steven Cinquegrana

I’m with Warren 100% here, Bobby. And I find it quite "reverso-world" that you say "… keep this discussion real and not purely academic …" as I feel that not going with a standard option is actually the more academic approach. And, in context, you’re providing 5 or 6 strings of non-sensitive information. You really need a dedicated schema for that?


And remember: this issue wasn’t highlighted by Blackbaud, it was something we discovered after several hours of faffing about trying to get this to work. As I mentioned, getting other providers’ webhooks to work takes no-time and no hosting provider liaison, but we’re in week 4 of trying to get yours to work.


I’d love to know the wide range of hosts you have actually tested with. You can be sure that none of the thousands out there slipped through the cracks?


And what are you basing your "…very narrow subset of consumers …" comment on? Stats please.


In fact, one of the hosts you mention, Hostgator, is one of the largest, most commonly-used hosts in the world. And they are renowned for making global "updates" to hosting accounts, without any forewarning, which breaks things. They overwrite working settings, revert others, etc. So asking them to accommodate a "special" and expecting it to persist is somewhat fanciful.


To me, this looks like oAuth2 all over again: “Wow, oAuth2 looks great, let’s do that!”, forgetting - or ignoring - the repercussions for non-web applications; headless/unattended and some desktop applications suffer immensely from a dynamic authorization method over, say, HTTP Basic Auth. oAuth2-authorized applications are virtually guaranteed to lose authorization at some point! And perhaps the single most common topic on these Developer Community pages regards problems with authorization. And not even the originators of oAuth like it! In fact, at least one of them is vehemently anti-oAuth2! Yet, despite all of this, there is no flexibility in providing an alternative means of authorization as other providers such as Mailchimp do.


The topic of this thread is "Looking for input for our Webhooks implementation". Please don’t waste our time asking if you’re not willing to listen.


Parting shot: To paraphase the old lead singer joke: How many Blackbauds does it take to change a lightbulb? One: Blackbaud holds the bulb and the world revolves around it.

 

Steven Cinquegrana

From the Webhook Troubleshooting page: "In the case of the 406, we contacted customer support for our well-known PHP hosting provider (who was able to implement the change across their entire server fleet within minutes)".


If you're referring to Hostgator, this change has certainly not migrated to our servers; I've just spent 4+ hours with various support personnel who have not managed to fix this issue, or really even to grasp it. They did, however, manage to change our PHP version three times without notice, breaking a couple of our PHP 7+ sites and requiring two more support calls to rectify. (This is what I mean when I say, expecting ModSecurity settings to persist is a bit fanciful ...)


So, currently, we have no means to get Blackbaud webhooks working until (if?) our Hostgator support case is attended to. There is no guarantee that this will happen as they don't action cases about 30% of the time. (Yes, we could - should - change hosts but the level of pain is, as you might be aware, quite high. Though I'm getting close, I think.)


I'm leaving this until there is better support all 'round as I simply can't waste any more time on it. [Serious eye-roll and head-shake.]

UPDATE


It seems that HostGator has completely trashed our php.ini rendering our Datawise webhook processor dead in the water. Really lovin' this cloudevents stuff!

 

Warren Sherliker

Just so it is in this discussion I saw from the 'What's new digest email':


For many consumers of ModSecurity (a web application firewall often used in Apache and PHP hosting environments to enforce OWASP security suggestions), they rely on a set of core rules. We made a contribution to the OWASP ModSecurity Core Rule Set (CRS) repo in GitHub to add application/cloudevents+<wbr />json and application/<wbr />cloudevents-batch+json to their default list of accepted values for the content-type header. Our contribution is tentatively planned as part of their CRS v3.3.0 release. We'll provide an update when this is live!


So hopefully a release there will sort things long term. Given hosting providers never work off the 'latest' code though it might be some time till this solves the problems that people are seeing with ModSecurity blocking the events.


If you are going to insist on these content types how about having a page setup to help providers with the changes needed with current ModSecurity setups to allow the content types through? or do you already have that?


As before given the lack of data you are providing in the webhook payload and the barrier to customers I still don't see the need for these custom content types though. It seems you are doing all sorts of extra work but for very little reason - just use application/json... or maybe make that an option on the webhook setup or something... even if until the ModSecurity v3.3.0 patch is actually live and in general use which might be a year away...