API usage: Call count

Bryce Howard

I'm currently tracking API usage via my own config file. My scripts capture and store the usage when they're done. I know applications like Postman can assist, but I'm essentially doing it myself already.

I was hoping my scripts could pull these metrics directly from Blackbaud. Is there a way to get API usage without having to navigate through the Developer Account > Analytics page on the web?

Alex Wong

@Bryce Howard
Blackbaud does not have an API to get API usage.

Though I am not sure why you have to track it. 25K per 24 hours period should normally be enough for most org use-case.

I sync SKY API data every 4 hours in a day and also have many automation that calls SKY API and have not had situation where I would run out.

The only time I had issue was running SKY API in the “single” endpoint for getting each pledge's installment and each pledge's payment details. Had a total of 38K pledge record, so 25K per 24hours wasn't “enough” to do 76K calls. But since then, I only need to do iterative calls to get changed.

Bryce Howard

@Alex Wong Over Christmas break, I'm syncing data across 90 different databases using a few CRMs. They’re sending about 490k 'New' records, but RE has most of these marked as existing.

Using Apache Airflow, the script runs 'New' records through the endpoint '/constituent/v1/constituents/duplicatesearch'. The script processes about 20 records per minute. Given the 25k per 24-hour limit and usage from other scripts, this will take about a month to complete. I set up a config to track usage a few months ago when I was just getting into APIs, but I haven’t revisited it till now.

We did this last couple years purely using script I wrote in python. My plan is to let this run until all records are screened while I focus on testing other methods and scripts to prep for the sync. I’m relatively new to using SKY API, but trying add what I can to speed the process up this year, not sure if there’s a more efficient approach.

Alex Wong

@Bryce Howard
In your situation, I would not have use SKY API for the initial sync.

Depending on data that is available, generally you match by name, email, phone, address, you would be better off just exporting all of RE's constituent data point that is used for matching and the 490K record you mentioned from other platform, and do the “match” off RE. (excel formula, power query, load to a sql database table, etc).

You mention you are “syncing data across 90 different databases using a few CRMS". I don't understand what you mean by that. Are you planning to sync every year/few months? What industry and platforms are you using that you have so many customer/constituent data spread across so many db/crm? You are going to need a much better strategy if you are going to keep doing this over and over.

Bryce Howard

@Alex Wong
These 90 databases are parishes that operate independently. The diocese uses their data for diocesan-wide appeals.

Since taking on this role, I've made several changes to the process whenever possible. My predecessors had no technical experience and relied solely on ImportOmatic to sync once a year.

Currently, my ETL process involves importing all data into SQLite for processing. Records that fail my data integrity scripts are rejected and sent back to the respective parish. Existing records are compared to identify changes to update records done by the diocesan data team via a database view import. "New" records are screened, and an existing ConsID is applied to matches. Low-scoring "New" records are processed through ImportOmatic. I use ImportOmatic because the SQLite data is static, and this syncing process is not the only way records are added. Truly new records are returned to their respective parish and added to my SQLite database with an assigned ConsID. I'm summarizing here, but this is the strategy in place that I am improving.

Right now, I want to improve the process by automating the screening of "new" records. This will remove the dependency on ImportOmatic completely. Since SQLite is static, I want to add a method to make an API call when my scripts returns a low score for a record. Python's fuzzywuzzy, ImportOmatic, and SKY’s duplicate search generally use Levenshtein Distance in their matching algorithms. ImportOmatic's application of Levenshtein Distance is not very good and the manual process slows the data team down. SKY’s application is much better and on my par with my current application where I'm using fuzzywuzzy.

Alex Wong

@Bryce Howard
There is a lot of specific terminology you used that is unfamiliar to me, so I can't really comment much on it (i.e. parish, diocese, diocesan).

while running 20 per minute is slow, you can increase that with multi-threaded calls to SKY API. SKY API allows for 10 API calls per seconds, but you will more quickly reach the 25K per 24hr limit. However, you can “pay" to get more if you just have this need to do this project on an annual basis. what is the exact cost and how much increase is something you can ask your account rep at BB. @Erik Leaver or @Ben Wong may be able to give you some ball parks numbers.

Good luck.

Bryce Howard

@Alex Wong
Those are Catholic terms and not really important. If you're curious though:

• 90 parishes: Catholic church entities - Each operate their own database.
• Diocese/diocesan: Collection of parishes - It's the central hub for all parish data.

I should have been more clear about the 20 records per minute metric. It includes processing in python, then API call for low scoring records. The methods doing the processing in python are multi-threaded, but I took no time to rework it and just added the API call at the end. Definitely room for improvements to be made. Even though you answered the initial question right away, I appreciate all the input. Thanks, Alex.