Exporting large amount of data through API

Nick McGinnis

I am working on syncing our CRM data into an in house application we have. I was using data list, the accessing through API, for the majority of the data, 2 of them have over 1.2 million records though. Data lists have a limit for records that cannot be bypassed. I tried to change them to query views and query them through the API and it will only return 32767 records. A workaround would be to just use an export definition and download the file and have our ETL developer handle them manually but we want to automate this process as much as possible. Is there any feature or spec that we can use to be able to access all 1.2 million records through the API?

Dennis Ladnier

Nick, what CRM do you use?  Thanks

Nick McGinnis

We are currently on service pack 24 if that is what you mean?

Steven Finnell

Have you tried using the paging feature of the API (GetMoreRows)? Because it could be that it's limiting it to 32,767 because it's just far too much data to stuff into a single response.

Chris Geady

Nick is using BBCRM 4.0 with SP24. Have you looked at the read-only connection for BBCRM? I hear it's supposed to be released (GA) sooner (like in 2021). I don't have specifics on the timeline. This may help.

Nick McGinnis

Steven, I will have to loop back and try the paging, I thought I had looked at it before but I cannot remember why I didn't use it. 


Chris, I will need to ask about the read-only connection, that would be extremely helpful if they finally allow for that.

Joseph Styons

Nick,

I think you're going to have trouble getting any OOB APIs to deliver that much data reliably.

You can automate the exports via an API call though.

Here is an old bbdevnetwork article that explains how - there's a download link in there with source code.
https://www.bbdevnetwork.com/blogs/interacting-with-crm-exports-from-code/

-J.R. Styons

https://www.styonssoftware.com