AWS AppFlow Incremental Loads

Michael Rzonca
Michael Rzonca
Name Dropper First Anniversary Facilitator 1
in Blackbaud Raiser's Edge NXT® APIs

We're trying to create an integration using AWS AppFlow and while we are able to execute full data loads of subject areas, the incremental load supported by AppFlow is grayed out/unavailable to select.

According to AppFlow documentation, incremental loads should be able to be configured once a Schedule-triggered flow option is selected, however with the Raiser's Edge NXT source we are unable to do so.

I've reached out AWS support and they said the following:

"I have received the updates from internal team and they have informed me that the availability of the Incremental transfer option depends on whether the custom connector's definition and metadata provide the necessary support for it.

Referring to the developer docs for entity metadata - https://github.com/awslabs/aws-appflow-custom-connector-java/blob/main/README.md :

"isTimestampFieldForIncrementalQueries - Boolean indicating if this field can be used for scheduled flows to pull incremental data from last successful execution time. This property indicates that the field contains timestamp values (either as Date or DateTime) which can be used by AppFlow to configure incremental data pulls. If no such field exists then scheduled flows cannot be configured to pull incremental data (all executions will result in a full data pull)."


As such the custom connector implementation must ensure that for the metadata, there are valid fields for capturing incremental changes. The custom connector must be designed to identify and utilize specific metadata fields for tracking incremental changes. This ensures efficient data synchronization by transferring only the modified or new data, reducing resource and time requirements.

Internal team has also requested to verfiy the utilized connector's implementation and have requested to reach out to connector's owner for further clarification and troubleshooting support as this Connector is not owned by AWS and is third party, providing support for this would be difficult."

Basically we're trying to see what needs to be done in order to load incremental data using AppFlow?

Thanks!

Some useful links:

Comments

  • Michael Tims
    Michael Tims Blackbaud Employee
    Seventh Anniversary Kudos 2 Name Dropper Participant

    Hi @Michael Rzonca - I spoke with internal Blackbaud staff, and it appears the Blackbaud Raiser's Edge NXT connector for Amazon AppFlow was built and is owned/maintained by Amazon. Any support or usage questions should be directed to the AWS team at Amazon.

    Internal team has also requested to verfiy the utilized connector's implementation and have requested to reach out to connector's owner for further clarification and troubleshooting support as this Connector is not owned by AWS and is third party, providing support for this would be difficult.

    There's likely just a disconnect on Amazon's end, not realizing who built and owns this connector.

  • Michael Rzonca
    Michael Rzonca
    Name Dropper First Anniversary Facilitator 1

    Hi @Michael Tims, thanks for the reply. I'll follow up with the case I have with AWS with this information. Thank you!

Categories