Best strategy for large volume automation?

Andrew Hill
Andrew Hill
Sixth Anniversary Facilitator 1
in SKY Developer General Discussions

if i want to insert 10,000 donations this appears to require a very chatty connection, as constituent, address, phone number. gift, soft credit, custom fields, etc are all updated per record.
updating one person could have 10 to 20 api calls needed, meaning that a file of 10k records could use most of an entire days quota all by itself.

Is there a best way to update multiple people at one time, or at least have the capability to “insert, ignoring the record if it already exists”, otherwise i need to do a search + insert or update for every attribute.

Comments

  • Ben Regier
    Ben Regier
    Sixth Anniversary Kudos 5 Name Dropper Participant

    I don't believe SKY API has the ability to do batch updates or inserts. There are also no “insert, ignoring the record if it already exists” endpoints to my knowledge. I think you are correct that this could be a chatty process to set up.

    If it were me, I'd first check to see if this insert could be done with an Import rather than using the API. The options you have there might be better for this scenario. If you need to use the API, I'd break the data up into several chunks and process one per day for several days.

  • Lucien Piekut
    Lucien Piekut
    Third Anniversary Name Dropper Participant Facilitator 1

    I'd say don't underestimate what the API can do. In the end, the processor in the other end is waiting for work. 10,000 donations seems like a lot, but if the machine does it over night it's still just one night.

    When it comes to the job of comparing, you always have the option to pull down from production, compare local/in memory, then post after compare.

Categories