Odd List Single errors: HTTP status 429, "Too many requests", "Limited by error rate"

Eric Eskildsen

Had an odd issue pop up with advanced lists. List Single suddenly started returning errors for all advanced lists for a school.

• Content body: Limited by error rate
• HTTP status code: 429
• Reason: Too Many Requests

I'm baffled. My API usage is well below my daily quota. Other API endpoints return 200 OK for this school.

List of Lists says I have access to the list. Its response looks like this (anonymized):

{

"count": 1,

"value": [

{

"id": 123456789,

"name": "MyList",

"type": "Advanced",

"description": "",

"category": "",

"created_by": "MyUsername",

"created": "2022-08-10T18:59:53.61+00:00",

"last_modified": "2024-03-27T20:59:01.827+00:00"

}

]

}

My call to https://api.sky.blackbaud.com/school/v1/lists/advanced/123456789 results in:

StatusCode: 429, ReasonPhrase: 'Too Many Requests', Version: 1.1, Content: System.Net.Http.HttpConnectionResponseContent, Headers:
{
Cache-Control: no-store, must-revalidate, no-cache
X-Content-Type-Options: nosniff
Strict-Transport-Security: max-age=31536000; includeSubDomains
Date: Tue, 02 Apr 2024 14:01:28 GMT
Content-Length: 23
Content-Type: application/json; charset=utf-8
}
"Limited by error rate"

Has anyone else been having List Single issues?

Alex Wong

@Eric Eskildsen
have you tried again making this call “alone” from any other potential calls?

Blackbaud API not only have a 25K API call limits per 24hours, it also has a 10 (I think it's 10) call per seconds limit. If you have any other process running that's making calls at the same time, that might be the issue.

Eric Eskildsen

@Alex Wong Yes, it seems to be this specific call. I can consistently call other API endpoints for the school and get 200 OK (e.g., List of Lists). Then I switch to calling any List Single endpoint, and it immediately returns this error.

Eric Eskildsen

And now, the endpoints are suddenly all returning 200 OK, and the content is the JSON. Hmm.

Eric Eskildsen

@Alex Wong I do have scheduled tasks fetching API data, so I suppose it's possible that the rate is (/was, optimistically) the culprit like you were saying. It's odd that no other calls were failing except those list calls, though….It was consistent over several days of testing.

Maybe it was just really unlucky timing on my part…I don't know. I'll keep an eye on it and update this thread if anything else emerges.

Alex Wong

@Eric Eskildsen
error 429 is rate limit error for sure, however, if you run the api and everytime it is 429, then there may be an error in the error code returned too, that's going to be a support ticket.

Jared Harbour

@Eric Eskildsen
I wanted to jump in and add some context. SkyAPI has global rate limiting in place as @Alex Wong described, but on top of that we have implemented separate rate limiting for the Get List endpoint in School API. Here is how it works.

• Any successful calls made to the GET List Single endpoint will have no restrictions
• Any call resulting in an error to the GET List Single endpoint will trigger a temporary time-out on the Environment and BBID of the calls origin. This time-out/hold will be for 60 seconds. After that calls may resume.
• If calls are made to the endpoint while the time-out is in place the response back will be Status 429 too many requests and an error of "Limited by error rate" will appear.

99% of the time the list requested is either a very complicated query or returning a lot of data and therefore hits a timeout, and that state causes an error which in turn causes a rate limit for 60 seconds. This only currently applies to the Get List endpoint.

I will be adding the above bullet points to the Get List documentation later today.

Hope this helps!

Eric Eskildsen

Thanks @Jared Harbour—this is great context!

Michael Conan

@Jared Harbour

I’m experiencing a similar issue with the OneRoster API I’ve outlined here.
Is this related?