Slow pulling data

Brandy Hulvey
Brandy Hulvey
Tenth Anniversary Name Dropper Facilitator 1 Loyalty Badge
in SKY Developer General Discussions

Could you clarify how RE handles high volumes of GET requests? Specifically, is there any delay or throttling mechanism that might be affecting synchronization performance? We are invoking 9 API endpoints per constituent, and it takes approximately 2 hours to update 500 users. We are retrieving only biographical data—none of the requests are gift-related. Do you have any insight into potential causes for this slowdown?

Answers

  • Alex Wong
    Alex Wong Community All-Star
    Tenth Anniversary Kudos 5 Facilitator 4 bbcon 2025 Attendee Badge

    Can you expand on what you are trying to do? I do not understand why you need to do 9 API calls per constituent. Also, are you using Power Automate, or are you using some custom application you created.

    If I have to make some estimate:

    Generally, 1s to 3s per API call from my experience. So let's say 2s average, 9 API calls will be 18s. Let's round it to 20s per constituent. 500 x 20s = 10K seconds = 166 minutes. Then your experience looks reasonable, assuming you are running sequentially.

    Client API limit is 10 API calls per second, so if you do 2 concurrent threads, that reduce the time to ~83 minutes, 4 concurrent threads ~42 minutes. I wouldn't do more than 5 threads though.

  • Brandy Hulvey
    Brandy Hulvey
    Tenth Anniversary Name Dropper Facilitator 1 Loyalty Badge
    edited February 20

    We are not using PowerAutomate. Our developer is using https://tray.ai/, which he said is a standard tool.
    Here is the list of calls he sent us.

    Get constituents: /constituents
    Get constituent by ID: /constituents/{id}
    Get aliases: /constituents/{id}/aliases
    Get phones: /constituents/{id}/phones
    Get email addresses: /constituents/{id}/emailaddresses
    Get addresses: /constituents/{id}/addresses
    Get custom fields: /constituents/{id}/customfields
    Get educations: /constituents/{id}/educations
    Get relationships: /constituents/{id}/relationships

    We are trying to get the data that has been updated in Raiser's Edge since the last API sync to move it to Hivebrite.

  • Alex Wong
    Alex Wong Community All-Star
    Tenth Anniversary Kudos 5 Facilitator 4 bbcon 2025 Attendee Badge

    Your developer is going about this completely the wrong way. I thought you were making updates call into RE NXT and have to update multiple different properties of constituent therefore the 9 calls.

    It sounds like you are trying to data warehouse the RE NXT data, which you can do in bulk using various other means (i.e. list api, query api, etc). there are only VERY few situation where you must use "single get" api (i.e. pledge installment and payment).

    I do data warehouse for my org and can sync ~70 tables of data (i.e. >500K constituent, >2M gifts, etc) every 4 hours in under 1 hour. Your developer's methodology needs to be revisited.

Categories