We've been getting 503 “no healthy upstream” errors for various SKY education endpoints over the past 24 hours, from different networks and computers.
Example calls:
https://api.sky.blackbaud.com/school/v1/users/extended?base_role_ids=14
https://api.sky.blackbaud.com/school/v1/lists/advanced/123456789
Some calls work, like /v1/roles.
Same here on the users extended. Weird the token refresh works but not the others.
@Eric Eskildsen our team is working through an issue with our advanced list feature that is likely causing the issues for you today. Please see
@Todd DeSchuiteneer I'm seeing this on and off at night (Eastern Time). The jobs have retry with exponential backoff, so we try to hit the endpoints multiple times, waiting longer between each time.
Example call that errored out ~1 AM Eastern today:
GET /v1/years
This is happening again this morning.
@Johan Guzman we're experiencing this as well. We're also seeing other inconsistencies with the Get User Extended by Role API. We opened a ticket as it appears we're getting a 500 error on every other scheduled time.
@Todd DeSchuiteneer - I added details to case 019871148 this morning. With a particular Advanced List, I'm getting a 500 error when I call V1ListsAdvancedByList_idGet with a page number that has no records. It takes about 20-30 seconds to return a page of 1000 records.
@Brian Gray thank you for reporting through support. I see we have most of the details we need through the ticket to research this and I will be keeping an eye on the ticket to make sure it progresses in timely manner.
@Lindsey Bryant I am also looking into your ticket regarding this issue. I have our agent reaching out to confirm one last bit of information. I will be sure to monitor it closely.
@Johan Guzman I apologize for the issue. The problem that caused the issues you were seeing was listed on our status page and resolved.
@Todd DeSchuiteneer Just FYI, I still get this on SKY Education endpoints not too infrequently.
Hi @Eric Eskildsen, are you seeing the errors on the same endpoints you listed in the original post still? Is this in a particular environment or you see it across all environments you work with?
@Todd DeSchuiteneer I haven't noticed a particular environment….I'll try to start noting when and where. This morning it was all endpoints for a particular school I was querying.
In our batch jobs, I'm not sure how often it's happening, as there's retry and I'm not sure it's being logged.
@Todd DeSchuiteneer Saw this for another school just now. This time it was the advanced lists endpoint.
@Todd DeSchuiteneer Okay, you probably don't need my play-by-play at this point, but I'm seeing it for a third school, this time for users/extended.
@Eric Eskildsen thank you for the information. I have alerted my team and have someone looking into this now.
@Eric Eskildsen I have opened a ticket for us on this issue and will be reaching out shortly regarding the issue.
