OneRoster API User Endpoint 429 Errors

Michael Conan
Michael Conan
Kudos 1 Name Dropper Participant First Anniversary
in Education Management APIs

We have an application that is using the Blackbaud SKY OneRoster API.

Our application uses many of the API endpoints, which occasionally return "429 Too Many Requests" responses indicating that we've exceeded the rate limit, at which point our application implements an exponential backoff retry strategy. Wherever possible, we batch requests together and limit the data to be returned using query parameters to filter data, specify required fields and sort / paginate.

Starting earlier today, we began observing a much larger volume of 429 responses from /users endpoints, including /students and /teachers. Even after a long waiting period, these endpoints are returning this status, while other endpoints are working as expected.

This error is causing most of the key functionality of our application to fail.

Comments

  • Michael Conan
    Michael Conan
    Kudos 1 Name Dropper Participant First Anniversary

    @Stephen Boyle @Todd DeSchuiteneer do you have any insight on this?

  • Michael Conan
    Michael Conan
    Kudos 1 Name Dropper Participant First Anniversary

    @Eric Eskildsen

    It seems I hit a similar issue to the one your reported previously here…
    ?

  • Eric Eskildsen
    Eric Eskildsen
    Sixth Anniversary Kudos 1 Name Dropper Participant

    @Michael Conan I've been seeing this too lately. It's happening to OneRoster calls for me too.

  • Eric Eskildsen
    Eric Eskildsen
    Sixth Anniversary Kudos 1 Name Dropper Participant

    Last night around 2 AM ET, one of our jobs got 429. Another got 403 Forbidden.

    When I reran in the day, no issues with the same credentials.

  • Todd DeSchuiteneer
    Todd DeSchuiteneer Blackbaud Employee
    Tenth Anniversary Kudos 2 Name Dropper Participant

    @Michael Conan our team had put some rate limiting in place on the /users, /students, and /teachers endpoints which is why you were receiving some 429 errors when calling. For now, we have loosened the restrictions on the throttle. You should be able to make your calls to these endpoints normally now.

  • Michael Conan
    Michael Conan
    Kudos 1 Name Dropper Participant First Anniversary

    @Todd DeSchuiteneer thanks it’s looking good now.

    I was only calling the endpoints a few times a day, hours apart, outside of development so I was surprised to see these errors, maybe they weren’t being enforced properly or I misunderstood what would trigger them?

  • Todd DeSchuiteneer
    Todd DeSchuiteneer Blackbaud Employee
    Tenth Anniversary Kudos 2 Name Dropper Participant

    @Michael Conan I'm not sure on your specific situation but the triggers set were similar to those set on our GET Lists single endpoint where if there was an error on the endpoint, it could not be called immediately. If tried the 429 was served up.

  • Michael Conan
    Michael Conan
    Kudos 1 Name Dropper Participant First Anniversary

    @Todd DeSchuiteneer ok interesting, I did get an error testing different filters right before the issue began, but then was receiving 429 for 4 days until it was turned off as you mentioned

  • Eric Eskildsen
    Eric Eskildsen
    Sixth Anniversary Kudos 1 Name Dropper Participant

    @Todd DeSchuiteneer What are those limits set to in terms of time between and number of requests?

    Multiple school jobs have been failing with 429 at night for us, each running a few minutes apart. The calls to /users are just the complete data set with pagination for each.

  • Todd DeSchuiteneer
    Todd DeSchuiteneer Blackbaud Employee
    Tenth Anniversary Kudos 2 Name Dropper Participant

    @Eric Eskildsen currently there are no additional rate limits in place for OneRoster besides the standard 10 calls per second. Before yesterday there were some additional in place on the /users, /students, /teachers endpoints but those have been lifted for the time being.

    Are you still seeing the 429 error coming up as of last night? Can you share the endpoints you received that error on?

  • Eric Eskildsen
    Eric Eskildsen
    Sixth Anniversary Kudos 1 Name Dropper Participant

    @Todd DeSchuiteneer It looks like the 429s last night were right after a 500. Attached.

    I guess the timeouts could be related to my page size? I haven't seen them before. I've been at that size for a couple years.

    Errors.csv

Categories