Sharing SKY Bridge - an Airbyte Connector for NXT

Sure! I respect the work being done with power automate - the official Blackbaud connectors and all the creative flows developed and shared by the community are fantastic for workflow automation. From my perspective, I would actually argue that power automate and airbyte are complementary tools for solving different problems.

TL;DR: I see Airbyte best for moving the data, while Power Automate is best for creating workflows from the data.

I think power automate excels at:
-Workflow automation (gift acknowledgement, notifications, action creation, etc.)
-Event-driven processes for human interaction

I think Airbyte is probably a better choice for:
-High-volume data handling (millions of records)
-'warehouse loading' (Snowflake, BigQuery, etc.)
-Complex transformations and CDC (if necessary - haven't really gone down that road yet but the tools look juicy sweet)

I like how Power Automate handles scenarios like "when a gift comes in, trigger an acknowledgment."

However, I strongly prefer Airbyte for a job like "sync our entire donor database to the data warehouse for analytics."

I'd be interested to hear about your Power Automate implementations for data movement/mirroring though. Perhaps you've got more efficient / less complex setups than what I've experienced.

My inspiration for creating this Airbyte connector came from my experience with the Power Automate setup I inherited. The architecture involves Power Automate flows exporting query data as CSVs to SharePoint, which then supplements data gaps from the Power BI connector - all requiring a gateway VM just to maintain scheduled refreshes for dashboards. That's a lot of moving parts just to join data and keep reports fresh in Power BI (which has its own adoption challenges internally, but that's a totally separate issue). While this might work for some scenarios, for me, Airbyte simplifies it to a single pipeline directly to our data warehouse.

Thanks for sharing! Yes, I think we're in agreement here - your setup isn't 'beginner friendly' in that sense. But in practicality, you achieve the same end result. So no, you don't 'need' the airbyte connector. However…

About your question: airbyte handles this! This is exactly where airbyte excels. In my current connector configuration, we use pagination to divide records into individual API calls, 100 at a time. Short calls, managed tracking **

For large calls, the records/page should be updated to a much larger number. I made the mistake of attempting a full constituent sync at 100/page: my quota was reached after pulling 2,500,000 records (2.5 million / 100 records per page which is a single api call = 25,000 api calls).

I don't know what the appropriate records per page is (sounds like you're saying up to 5,000? In this case, we can configure a lot more records per call to make this much more efficient — trying this now!)

But in short, no time-based max, and with pagination, it automatically handles chunking the call according to your settings.

**with respect to managed tracking: right now, it works fine as I've built it - but it still needs improvement. Here's the situation according to my understanding

When using incremental sync with last_modified parameter on SKY API endpoints, the API prevents duplicate records by adding a sort_token parameter to the next_link URL.
However, this token is embedded in the URL string, not returned as a separate field.

The problem:

• Constituents endpoint: Returns sort_token inside the next_link URL (e.g.,
  https://api.sky.blackbaud.com/constituent/v1/constituents?limit=100&offset=200&sort_token=abc123)
• Actions endpoint: Returns continuation_token as a direct response field that can be easily extracted and used

Why it matters:

• Without using sort_token, offset-based pagination with last_modified causes duplicate records (as records modified during pagination can appear multiple times)
• Airbyte's declarative connector can easily handle continuation_token (direct field) but I still need to figure out how to extract sort_token from a URL string

Current workaround:
Using Airbyte's built-in deduplication, but this doesn't solve the root cause and still processes unnecessary duplicate records during sync.

The question for the community:
How can we extract and use the sort_token from the next_link URL in a declarative Airbyte connector, similar to how continuation_token works for the actions stream?

@Alex Wong awesome - thanks so much! This set me on the right path, and I figured it out. Resolved the issue AND resolved my other warnings I was getting about unique ID's. With this fix for the sort token and recent error handling, i can now submit this to Airbyte for placement in their catalog … simplifies installation greatly, and opens it up to providing downstream updates to the connector so people don't need to fork it and run their own copy.