Big Increase in Blocked Requests

Dan Bonville

Starting around 9/17, we saw a huge uptick in blocked calls, which seemed to manifest in a lot more authentication errors on our end with our various client integrations. Over the last few days, it looks like ~75% of our calls to the Constituent API are being blocked (when I look in the developer portal > analytics). As far as I can tell we are nowhere near our rate limits, and I can't find any definition of why calls would be blocked.

• Is there any way I can get more details on why calls would be blocked?
• Are there any known issues on the Blackbaud side dealing with authentication? I've also noticed some flaky login behavior on the developer portal, which maybe is just coincidence or maybe not.
• Is anyone else experiencing any degraded API performance?

I'd send in a support request about this, and please advise me if that is prudent (@Ray Bergman ) but I figured I would start here first in case I am missing some obvious documentation on what “blocked” calls can be caused by.

Thanks!

Dan Bonville

These errors seem to have ceased, at least temporarily. We last saw an error at ~7:15 am EST.

Dan Bonville

…..well, maybe I spoke too soon. They seemed to start again over the night. I will reach out to support about this.

Daniel Leonard

Good morning Dan,

I am sorry you are having issues with SKY API. The developer portal is the best way for you to see if calls are being blocked and support will generally be your most consistent option for this type of issue. We are not aware of any existing authentication issues on the Blackbaud side and I am not currently aware of anyone else having this issue.

Generally, an HTTP status code of 401 would indicate a token related authentication issue. In this case, the tokens that are failing appear to be expired. Please verify that your token refresh policy is correct and the tokens being used are not expired.

We are not currently aware of any issues with the developer portal login process. Any details you would be able to provide around the problems you are experiencing there would be greatly appreciated.

Dan Bonville

Thanks Daniel,

I have reached out to support as well, per Ray's advice.

We are indeed seeing 401s at a regular clip. Would those be presented as “Blocked” or “Failed” calls in the analytics in our developer account?

Comparing week over week, last Monday and Tuesday we saw 350 blocked calls while today and yesterday we already have over 6k. It could certainly be something on our end, but we've made no changes since well before this started occurring.

Appreciate the help,

Dan

Daniel Leonard

Requests with invalid or missing tokens will show up in the "Blocked calls" column of the Analytics reports in the developer portal

Steven Cinquegrana

Hi Dan,

Is there a chance that your token set is being used by more than one application or instance at a time?

In this case, one instance can request a new Access Token (and associated Refresh Token) then, before it's used, another instance does the same, thus invalidating the previous tokens which causes an error when the first instance uses the replaced token.

Just a thought. You would likely need to use the preserve_refresh_token option in cases such as this and then schedule a Refresh Token refresh every so often in isolation before it expires (ie, before the 365 days validity is up).

Cheers,

Steve Cinquegrana | CEO and Principal Developer | Protégé Solutions

Dan Bonville

Thanks Steve and Daniel,

We shouldn't be using more than one token per environment concurrently…that being said, we did see an issue with our queuing mechanism that may have caused at least some situations where that was untrue (our developers are investigating that right now).

I mostly reached out to see if others may have been experiencing this because it was so “all of a sudden” without us having done anything (seemingly) to cause it (no updates, no huge uptick in usage of the integration, etc.).

I also opened a support ticket to see if perhaps the logs exposed anything or at least to provide a more documented definition of failed vs. blocked calls. Thank you for your clarification there, Daniel.

Hopefully we find the bug on our end and we can call it a day!

Dan Bonville

After finding some obscure behaviors in one of our libraries, we were able to identify this issue and a fix is forthcoming.

Thanks for the help all,

Dan