Multiple SKY APIs Failing

Lindsey Bryant · November 12

We've been receiving multiple 503 Service Unavailable errors today starting around 1:40 PM. We pull data out of BBEM multiple times a day to update our data warehouse. We can attempt to run them again through Postman without receiving an error, but then we get them again the next time our schedule runs.

Anyone else having issues?

Brian Gray · November 12

Yes - several of us have seen them for the past hour or so.

Johan Guzman · November 12

Yup same here.

Lindsey Bryant · November 12

@Stephen Boyle I did submit a support ticket but my guess is that it's not getting routed. I'm sure you're already aware but thought I would at least tag you here just in case. Thanks!

Bob Raymond · November 13

yes same problem multiple fails. all on school API - esp advance lists. started 11/12

Brian Gray · November 13

@Stephen Boyle - any update you can provide will be helpful.

The villagers are gathering pitchforks and torches this morning. They may turn into an angry mob at any minute.

Seth Battis · November 13

We're now at about 18 hours of this, with no update on the status page and no meaningful response from customer service. Could we at least get some sort of status report on what's broken/is there mitigation going on/any sort of an ETA? Or failing that, an acknowledgement that we're seeing a real thing?

Lindsey Bryant · November 13

This was the response I got from support yesterday:

After a thorough review, we’ve identified that the issue stems from a product-related concern. Our engineering team is now actively investigating to isolate the root cause and develop a fix. This process involves careful testing to ensure the solution is stable and effective before it can be included in a future release.

This is extremely unfortunate as we (and assuming several of you) rely heavily on APIs to support our business processes. I hope we will be notified it's resolved soon. I didn't see an API release updates this week, which makes me question what "product-related" concern might have caused this.

Johan Guzman · November 13

Looks like we might be back up?

Bob Raymond · November 13

this morning it seems intermittent. about half requests generating 503

Erik Leaver · November 13

I urge everyone encountering this problem to log a support case & reference Bug 3622889. Having the count of affected customers is helpful in prioritizing triage.

Nick Marchese · November 13

Impacted as well. Just created case and referenced the bug.

Seth Battis · November 13

Currently can't authorize a new access token either — stalling on the green dot after I click authorizze

Stephen Boyle · November 13

https://community.blackbaud.com/discussion/comment/308584#Comment_308584

The team diagnosing this (and, rest assured, they've been on it since yesterday) has implemented an update to address the issue. They will continue to monitor the affected service. Support will also contact folks that called in about the outage and the resolution.

Lindsey Bryant · November 13

@Stephen Boyle thanks for the update. Moving forward (and I hate to put you on the spot) it seems like we get more traction on community post verse support. Yesterday, my support agent suggested that I try to filter data down because I might be pulling to much data back in a single API call.

We know your team is doing everything they can and we appreciate the quick turn around and I think we would all agree if we had more insight knowing that the issue is confirmed and is actively being worked, we could probably eliminate a lot of pressure for everyone!

In the future, is there a better way to escalate these issues? Is it really everyone submitting a support ticket or posting on a forum like this? How can we stay up to date on these issues verse responding back to a thread and feeling left in the dark?

Stephen Boyle · November 13

I think there will be an incident review on this one. So, while, in general, the ability of Support to respond to API and integration issues is still an improving process, this specific issue isn't a good yardstick. Things like this get to us by several avenues (including this community) and the team doesn't wait to begin work.

From our Director of Software Development (in a discussion with our Client Advisory Board):

We still have additional root cause analysis to do to fully understand what happened here but did just push out a change that is likely accounting for why it appears fixed now.
When this was first reported yesterday evening it looked more like an individual defect (intermittent and only had a couple reports), then an Incident. That said, we will discuss if in hindsight an Incident should have been opened and if so will handle retroactively.
Thanks for letting us know and sorry for the disruption.

Brian Gray · November 13

@Stephen Boyle - thanks for the info. Several of my programs that failed yesterday are running correctly now.

By the way… when I re-read my comment that you quoted I realized it could have sounded like a threat. The villagers I referred to were the ones at my school coming for me - not for you.

Eric Eskildsen · November 13

@Stephen Boyle Thanks for the updates. Just a heads up, the affected API (or rather the one that affected me) was green on status.blackbaud.com throughout. If these could surface there as well, that would be helpful as we go there for a first check when something is wrong.

Ruben Perez · November 14

Yep, thx everyone all green lights over here too.

Multiple SKY APIs Failing

Answers

Categories