Slow pulling data

Question

Could you clarify how RE handles high volumes of GET requests? Specifically, is there any delay or throttling mechanism that might be affecting synchronization performance? We are invoking 9 API endpoints per constituent, and it takes approximately 2 hours to update 500 users. We are retrieving only biographical data—none of the requests are gift-related. Do you have any insight into potential causes for this slowdown?

Alex Wong · Answer

Your developer is going about this completely the wrong way. I thought you were making updates call into RE NXT and have to update multiple different properties of constituent therefore the 9 calls.

It sounds like you are trying to data warehouse the RE NXT data, which you can do in bulk using various other means (i.e. list api, query api, etc). there are only VERY few situation where you must use "single get" api (i.e. pledge installment and payment).

I do data warehouse for my org and can sync ~70 tables of data (i.e. >500K constituent, >2M gifts, etc) every 4 hours in under 1 hour. Your developer's methodology needs to be revisited.

Brandy Hulvey · Answer

We are not using PowerAutomate. Our developer is using https://tray.ai/, which he said is a standard tool.

Here is the list of calls he sent us.

Get constituents: /constituents

Get constituent by ID: /constituents/{id}

Get aliases: /constituents/{id}/aliases

Get phones: /constituents/{id}/phones

Get email addresses: /constituents/{id}/emailaddresses

Get addresses: /constituents/{id}/addresses

Get custom fields: /constituents/{id}/customfields

Get educations: /constituents/{id}/educations

Get relationships: /constituents/{id}/relationships

We are trying to get the data that has been updated in Raiser's Edge since the last API sync to move it to Hivebrite.

Alex Wong · Answer

Can you expand on what you are trying to do? I do not understand why you need to do 9 API calls per constituent. Also, are you using Power Automate, or are you using some custom application you created.

If I have to make some estimate:

Generally, 1s to 3s per API call from my experience. So let's say 2s average, 9 API calls will be 18s. Let's round it to 20s per constituent. 500 x 20s = 10K seconds = 166 minutes. Then your experience looks reasonable, assuming you are running sequentially.

Client API limit is 10 API calls per second, so if you do 2 concurrent threads, that reduce the time to ~83 minutes, 4 concurrent threads ~42 minutes. I wouldn't do more than 5 threads though.