Retrieving more than 500 records in one call

Jonathan Bird
Jonathan Bird
Second Anniversary
in SKY Developer General Discussions

Hello,

I need to extract the full list of cons info for a database of many 10's of thousands of records. The issue I have is that the API limit the number of records I can pull at any one time. I see you can set a limit which appears to allow you to change it to 5k however it still only ever pulls 500 records at any one time. I need to access full list of cons actions and gifts so it's not just bio info.

I currently use the db view and query/export however this is something that I trying to move away from as it doesn't work very well when exporting large amounts of data so I'm keen to get it working via the API.

Any ideas or workarounds?

Comments

  • Ben Lambert
    Ben Lambert
    Ninth Anniversary Kudos 5 Name Dropper Participant

    @Jonathan Bird can you say which endpoint(s) you're calling where the limit query parameter isn't being respected (only returning 500 records)? That would certainly be a bug, and if confirmed I'd recommend that you contact Blackbaud Support to get a ticket filed and routed to the appropriate engineering team.


  • Lucien Piekut
    Lucien Piekut
    Third Anniversary Name Dropper Participant Facilitator 1

    @Jonathan Bird

    Hi Jonathan, the endpoint you are retrieving does have a 500 maximum limit, even if you key in 2000 it will retrieve 500 records PER CALL.

    To pull larger datasets what you'll need your code to do is pull 500 records, then pull the next 500 records and append, then pull the next 500 records and append. And so on until you hit whatever dataset + filters/paramters you've keyed in.

    Endpoint APIs usually keep a parameter that let's you know what the next entry is and your follow-up call will start from that entry.

    Lucien Piekut
    txtstring.com Data Development Services

Categories