List Execution Jobs Throwing "OutOfMemory" Exceptions

Omer Boratav
Omer Boratav
Second Anniversary Name Dropper Facilitator 1
in Education Management APIs

I should start this off by saying that I know list execution jobs are in preview, and not yet fully supported. This is more feedback/suggestion than anything else.

List execution jobs created through the List Execution Job (Create) endpoint have a habit of failing and returning System.OutOfMemoryException exceptions after running for an hour or so. This mostly happens when running larger advanced lists (70k to 80k rows), but it's not exclusive to large lists.

In the rare case when they do succeed, the largest list is about 10-15MB at the most; so they're not super massive. The same advanced lists tend to succeed more frequently when requested page-by-page through the List Single endpoint.

The ability to use list execution jobs to run larger lists would be extremely helpful, as currently we're having to run 90 or so requests to paginate through some lists, and it's cumbersome. The list execution jobs don't need to be lightning-fast, as long as they're reliable.

Anybody else experiencing similar System.OutOfMemoryException exceptions with list execution jobs?

Answers

  • Bryce Howard
    Bryce Howard
    Fourth Anniversary First Reply Kudos 1 Name Dropper

    Other users have had similar issues with no useful reply, so the whole "submit job, poll, download via sas uri" preview is likely just something blackbaud needs to fix

    • https://community.blackbaud.com/discussion/83028/getjob-query-api-giving-status-failed

    The System.OutOfMemoryException is more than likely coming from a .net process, which seems like the worker is buffering the entire result in memory before pushing it to a sas uri. If it were streaming, you'd see timeouts or storage errors, not an oom. This is also likely why they are successful when you are doing it page by page because blackbaud's worker isn't running out of headroom while building the result. I think the only fix is to just wait till preview is developed more since it is a server side issue.

    Maybe there is room for improvement on pagination being cumbersome if you want to expand on that some more until it gets fixed.

  • Omer Boratav
    Omer Boratav
    Second Anniversary Name Dropper Facilitator 1

    Thanks @Bryce Howard, I thought the same but hadn't seen that other post. Glad to see that someone else wrote about it too.

    I don't know if you relate to any of these, but here are some issues I see with the pagination thing:

    1. Paginating through a list can take a really long time. So long that it sometimes makes me think that somewhere in the backend, Blackbaud is producing the full list and then throwing away all but the requested page, every time we request a page. And then we have to do 60, 70, 80 pages of that; and about 10% of those times, a request will fail at a place like page 59. The maintenance of the retry logic around that just feels… unnecessary.
    2. There's uncertainties around how pagination works if a change is made to an advanced list while we're on page 16 of 39. Say it's a list of grades, and a new grade gets entered (or modified, or deleted):
    • Would an addition on page 13 cause rows to shift and make the last row on page 16 appear once again on page 17?
    • What about a deletion on page 8, where the first row of page 17 would move into page 16 (which we'd already fetched) and potentially get missed?

    3. There's basically no guarantee of list accuracy or integrity when a list takes 30 minutes to fetch end-to-end, and the only means of pagination we have is a page number (as opposed to something like an opaque token tied to an instance of a list at a point in time).

    List execution jobs are definitely a step in the right direction, I just hope that they start working soon.

  • Nick Marchese
    Nick Marchese
    Kudos 1 Name Dropper Facilitator 1

    Hey @Omer Boratav. I'm also experiencing this for larger lists and haven't been able to find any true solution. @Stephen Boyle Any idea about this one?

  • Stephen Boyle
    Stephen Boyle Blackbaud Employee
    Tenth Anniversary Kudos 5 First Reply Name Dropper
    https://community.blackbaud.com/discussion/comment/313754#Comment_313754

    Thanks for this info. We have a case we are looking into for this - I'll add this to the discussion.

  • Nick Marchese
    Nick Marchese
    Kudos 1 Name Dropper Facilitator 1

    Thanks @Stephen Boyle! Would you be able to post updates here, if there are any in the future?

Categories