API Errors

Steven Cinquegrana
Steven Cinquegrana
Tenth Anniversary Kudos 2 Name Dropper Participant
in SKY Developer General Discussions

We're suddenly seeing a lot of SKY API errors.

Most of them are related to the Constituent, Communication Preference, Fundraising, Gift, List and Opportunity APIs and look like this:

{"status":500,"statusCode":500,"title":"Internal Server Error","message":"The SKY API service is currently unavailable - failed to resolve the environment audience.","detail":"The SKY API service is currently unavailable - failed to resolve the environment audience."}

We're also seeing the same error using the console, as well as an occasional 503 error:

9ba85b64a7dc3577a63c3a1dcbbd3f8b-huge-im
Console Response

For the Gift Batch API, we're seeing:

{"type":"urn:blackbaud:unexpected","title":"An error has occurred.","status":500,"trace_id":"e520e5b993a847699905e927baf4c3a5","span_id":"23e0fa128606f6f9"}

For the Statistical Unit API we're seeing:

{"statusCode":404,"message":"Resource not found","status":404,"title":"Resource not found"}

It appears that the base URL has been changed, though I can’t find a reference to the change, from

https://api.sky.blackbaud.com/statisticalunit to

https://api.sky.blackbaud.com/statistical-unit (note the hypen)

UPDATE

For the Event API we're seeing:

{403 Forbidden}

This is the same via the console.

Thanks.

Steve Cinquegrana | CEO and Principal Developer | Protégé Solutions

Comments

  • Warren Sherliker
    Warren Sherliker
    Tenth Anniversary Kudos 2 Name Dropper Participant

    Yep, seems to go in spates this one…

    We get a load of 50x, 404 and 53 (SSL) errors - I suspect during maintenance periods for the underlying products. Then for a while no issues. You just have to retry again after the issues go away and make all your coding resilient to these issues which is tricky given you don't know which calls are going to fail.

    I suspect BB should really have an API to tell you when the specific APIs (constituent, events etc.) are not going to be working so we can query that to ensure we expect responses or delay if not i.e. response constituent API will be in maintenance period in 20 minutes… Maybe that is overengineering.

    Whatever happens though this is not normal with any other APIs out there that we have worked with. They just don't have such outages.

  • Ben Lambert
    Ben Lambert
    Ninth Anniversary Kudos 5 Name Dropper Participant

    Hi Steve, we are investigating and will report back asap.

  • Steven Cinquegrana
    Steven Cinquegrana
    Tenth Anniversary Kudos 2 Name Dropper Participant

    @Warren Sherliker Yes, we see the same thing, and have a whole swag of error/outage management code included in our SKYLib.NET code library explicitly to mitigate this. No API is perfect of course. Good thought on the Outage API, perhaps part of a broader Utilities API providing such things as quota usage, test end points, etc. Upcoming outage info could even be provided via a webhook.

    @Ben Lambert Thanks, I appreciate it. I think there're a few separate issues here after delving a bit deeper at our end. One might be account-related as many of the problematic APIs are of the “RE” variety, while the “FE” ones seem fine (albeit slow, but that's normal for the dev sandpit, I think).

    The SU one does appear to be an (unannounced?) change to the base URL.

    The others - Event and Gift Batch - are the outliers and I don't have any further information about these.

    There's also the issue - yes, I know it's a touchy topic - that the Constituent API appears to have some changes to the API Definition where collection models have been changed from, for example, ApiCollectionSearchResultRead to SearchResultReadApiCollection. I can't see how this could be anything other than cosmetic or accidental; most of the other APIs stick to either ApiCollectionModelName or ModelNameCollection (or don't use a specifier at all). Changes to API definitions impact code generated from them and effectively result in a breaking change to code that consumes that code. I've posted more info here. I know Blackbaud doesn't consider OpenAPI changes to be breaking changes, but the end result to our SKYLib.NET customers is that they do constitute breaks as they have to change their own code to accommodate them. I don't expect anything to change, I'd just prefer not to have to work around it if at all possible, especially if the change serves no purpose.

  • Ben Lambert
    Ben Lambert
    Ninth Anniversary Kudos 5 Name Dropper Participant

    Hi all, I wanted to provide an update on these issues:

    Regarding the base URL change of the Statistical Unit SKY API...It took some time to research, but we've tracked this down to an unexpected side effect from a configuration change made back in mid-February.

    Obviously this was a breaking change, and something we would have preferred to avoid. I can appreciate the level of disruption this will have caused, and our team will be discussing this in detail to better understand how it happened and what we could have done to prevent it. I want to apologize and reassure you this is not something we take lightly.

    Given that this change happened more than 2 months ago, and the affected API is relatively new and has very little traffic (relative to other SKY APIs), we've decided to leave the change in place rather than revert it (and risk breaking recent code that uses the new base URL).

    We'll be updating our documentation accordingly, as well as adding a new changelog item and Community announcement to raise awareness.

    Steve, thank you for bringing this to our attention and once again I apologize for the hassle I know this will have caused you.

    ---

    Regarding the "failed to resolve the environment audience" message...

    This error happens deep in our API processing pipeline for a variety of reasons - sometimes due to a configuration issue on our end, sometimes due to an unexpected outage, sometimes due to a customer initiated migration that is not yet completed, etc.

    In this case, the root cause isn't clear yet based on the description here. It could also be due to trying to call an API associated with RENXT in the context of an environment that doesn’t actually contain RENXT. In other words, the environment selected by the consenting user doesn’t actually contain Raiser’s Edge NXT.

    Because this could happen for a variety of reasons, and due to the location in which it happens, we consider it a 500 Internal Server Error, which of course external developers should never encounter. We do have plans to introduce additional logic upstream in the pipeline that will allow us to return a better error message (no ETA yet, but something we are actively working on).

    Steve, if you can let me know the environment ID in which you are seeing this error, I'll be happy to get more details.

    ---

    Regarding the 403 Forbidden error you're seeing calling the Event SKY API...this could be due to a few reasons, but the most likely is that the consenting user had no permission to access Event data or perform the event-specific operation. If you can provide me with any details about the environment in which you're seeing this issue, I can investigate further.

    Feel free to message me directly in Community, or you can reach me via email at ben (dot) lambert (at) blackbaud (dot) com.

    Thanks again for bringing these issues to our attention!

  • Steven Cinquegrana
    Steven Cinquegrana
    Tenth Anniversary Kudos 2 Name Dropper Participant

    Hi @Ben Lambert. Thanks for your detailed response.

    I can confirm that nothing our end has changed with regard to environment/authorization; we have a standard suite of tests we run prior to releasing a new version of SKYLib.NET and this is where the breakages showed up.

    I'll message you privately regarding the info you need to resolve these issues for us.

    Cheers and thanks, Steve

Categories