Multiple SKY endpoints: 503 "no healthy upstream" errors

Eric Eskildsen

We've been getting 503 “no healthy upstream” errors for various SKY education endpoints over the past 24 hours, from different networks and computers.

Example calls:

https://api.sky.blackbaud.com/school/v1/users/extended?base_role_ids=14

https://api.sky.blackbaud.com/school/v1/lists/advanced/123456789

Some calls work, like /v1/roles.

Evan Cox

Same here on the users extended. Weird the token refresh works but not the others.

Todd DeSchuiteneer

@Eric Eskildsen our team is working through an issue with our advanced list feature that is likely causing the issues for you today. Please see

Eric Eskildsen

@Todd DeSchuiteneer I'm seeing this on and off at night (Eastern Time). The jobs have retry with exponential backoff, so we try to hit the endpoints multiple times, waiting longer between each time.

Example call that errored out ~1 AM Eastern today:

GET /v1/years

Johan Guzman

This is happening again this morning.

Lindsey Bryant

@Johan Guzman we're experiencing this as well. We're also seeing other inconsistencies with the Get User Extended by Role API. We opened a ticket as it appears we're getting a 500 error on every other scheduled time.

Brian Gray

@Todd DeSchuiteneer - I added details to case 019871148 this morning. With a particular Advanced List, I'm getting a 500 error when I call V1ListsAdvancedByList_idGet with a page number that has no records. It takes about 20-30 seconds to return a page of 1000 records.

Todd DeSchuiteneer

@Brian Gray thank you for reporting through support. I see we have most of the details we need through the ticket to research this and I will be keeping an eye on the ticket to make sure it progresses in timely manner.

Todd DeSchuiteneer

@Lindsey Bryant I am also looking into your ticket regarding this issue. I have our agent reaching out to confirm one last bit of information. I will be sure to monitor it closely.

Todd DeSchuiteneer

@Johan Guzman I apologize for the issue. The problem that caused the issues you were seeing was listed on our status page and resolved.

Eric Eskildsen

@Todd DeSchuiteneer Just FYI, I still get this on SKY Education endpoints not too infrequently.

Todd DeSchuiteneer

Hi @Eric Eskildsen, are you seeing the errors on the same endpoints you listed in the original post still? Is this in a particular environment or you see it across all environments you work with?

Eric Eskildsen

@Todd DeSchuiteneer I haven't noticed a particular environment….I'll try to start noting when and where. This morning it was all endpoints for a particular school I was querying.

In our batch jobs, I'm not sure how often it's happening, as there's retry and I'm not sure it's being logged.

Eric Eskildsen

@Todd DeSchuiteneer Saw this for another school just now. This time it was the advanced lists endpoint.

Eric Eskildsen

@Todd DeSchuiteneer Okay, you probably don't need my play-by-play at this point, but I'm seeing it for a third school, this time for users/extended.

Todd DeSchuiteneer

@Eric Eskildsen thank you for the information. I have alerted my team and have someone looking into this now.

Todd DeSchuiteneer

@Eric Eskildsen I have opened a ticket for us on this issue and will be reaching out shortly regarding the issue.