Advanced lists - row limit rolled back

Jessi Walters

Thanks to additional feedback from users like you, we're taking a second look at advanced lists that have very large results. Recently, we announced that we'd limit the results to 20,000 rows, and that change was withdrawn Wednesday morning.

To help us better support your school's needs for very large lists, it would help to have your answers to the following questions:

• What are your top 4 biggest lists – with the List name and/or with your estimated number of rows you typically retrieve?
  • Alternatively, what Objects are you using when you create massive lists?
• What is the average number of results sent to a third-party reporting tool (like PowerBI) per list?
• What is the highest number of recipients in a single Pushpage job
• What are you doing with 50K plus results? What context is lost with fewer rows? Does it help to break it down by school year or term?
• If we (purely hypothetically) re-introduce a row limit at 50k, how many additional lists would you have to build?
  

If you're willing to provide feedback, please either reply to this discussion or email me at jessica dot walters at blackbaud dot com. Also, please complete the Advanced List and Pushpage survey (published in the blue alert banner in-product) if you have not yet done so.

Thank you for your input!

Lauren Henderson

@Jessi Walters

Our school did not experience a breaking change with the row limit, HOWEVER, as we potentially look to BBEM as a repository for all historical academic data to then be pushed out for analysis and research, row limits would make me rethink this strategy.

Marshall Chaney

@Jessi Walters

We have 6000 students with 8-9 enrollments each. A full enrollment export for 1 term is 48,000 records+

We also export emergency contact info and authorized pickup info into our visitor management platform, This can be 4-16 records per student.

Mostly in Advanced lists we’re combining enrollment course details like teacher and building along with constituent relationship data like parent cell number.

A push page may go out to 10,000 prospective parents in addition to 10,000 enrolled parents.

Seth Battis

@Jessi Walters

Echoing Lauren's thoughts: we too treat Blackbaud as our system of record for historical queries. Without getting into thinking about queries related to alumni, I observe that, with 385 students taking 5-6 courses a term, 3 terms a year, for 5 years: just querying their enrollment history puts us up around 34k rows. Attendance history for a single term is even worse.

I discussed this with our database manager, who spoke with knowledge of how our development office uses the tool, and we think that the biggest lists we pull are the ones that I use to generate our course planning tool (which does require pulling that enrollment history, as noted above).

The current iteration of the course planning tool is pulling an enrollment history listing that hovers around 20k rows (not the worst case scenario), with some clever filtering. I'm pulling that data via API, to make a bridge between Blackbaud and the actual course plans for each student (so… 385 course plans). These get updated several times a week at various points over the term (maybe 3 dozen times a year), so automation is key, not just for convenience, but for maintaining clean data.

IF I could parameterize an Advanced List, I could pull shorter lists (e.g. grouping by class year or term or whatnot). In general, parameterization would be great. Without it, I either have to maintain multiple copies of almost-identical lists to be able to pull via API (a recipe for errors) or I have to manually pull the data while manually tweaking the list between runs (an other recipe for errors). Similarly, IF I could edit or create Advanced Lists via the API, I could also pull smaller lists.

I took a look at the Push Page and Advanced Lists survey and I just can't speak to exactly which data I need that isn't in the SKY Lists, because that's not exactly the problem. I will try, every time, to create the list I need in SKY Lists, but the starting point of making it a “kind” of list limits the data available in ways that aren't helpful. I usually need to cross-reference across the list kinds ("it's a student list, but I need to know which teachers they had last year"). Honestly, being able to just write a SQL query would blow my hair back, but I know that I'm in the minority here.

Brian LeBlanc

@Seth Battis:
Honestly, being able to just write a SQL query would blow my hair back, but I know that I'm in the minority here.

This, right here, is the crux of the issue. Almost everyone who uses advanced lists does so because they need something that is specific to their use case and their use case only. SKY lists, for all their user friendliness, will likely never be able to offer the bespoke customization that an advanced list can come close to providing, so it isn't an issue of just saying “what can an advanced list do that a SKY list can't?” Every customer has a different answer, by definition. That's the whole point of using an advanced list.

But even those are “good enough, but not quite there.” Case in point: I can't create an advanced list to filter on candidates by school level, to allow me to create a distribution list to send pushpages to specific school levels. Basic enough, right? But it simply isn't possible. It does not exist. The only option I have to filter by school level is to send an official note via a SKY candidates list.

The information exists in the database! I could write a SQL query to pull it in thirty seconds! But I can't!

Another example: in a SKY student list, I can display New or Returning. I can't do that in an advanced list. I just confirmed that with support maybe two weeks ago, after posting a thread in this very forum about the issue. I have a workaround, and it's fine, but again, it requires a workaround where I could whip up a basic SQL call in the blink of an eye to solve the problem.

Now, am I the only one with these issues? Maybe not, but maybe I am. I don't expect Blackbaud to cater to me, specifically. But it would be nice to get the keys to the Ferrari and be able to allow the database to work for me, instead of needing to contort myself into all sorts of shapes to align what I need with what the database will give me.

Lindsey Bryant

@Brian LeBlanc
I completely agree with you. I've worked in other software solutions where there is a great UI experience that mimics writing an SQL query. Some that have been rolled out in the past 5-109 years.

Unfortunately, Blackbaud is being the curve here and has actually made things more difficult by trying to focus on multiple solutions (i.e. Sky List, Advance List, Analysis) verse investing in one great solution. Multiple reports, multiple solutions and excel skills seem to be the name of the game for reporting, data analysis, integrations, etc.

Coco Parham

@Brian LeBlanc
Brian, you said it! I am not very tech savvy when it comes to pulling data from Advanced lists but need them for certain tasks I do each week or month. I dont need thousands of lines of data, but why cant I pull a list of monthly visits, showing boarding or day, in either visits list or the Dashboard! I have a report saved in Advanced lists for that. Or lists of certain letters we send out. Personally, I would rather use sky lists for everything. I think sky lists are more intuitive for the way my brain works. I find Advanced lists hard to use.

Seth Battis

@Jessi Walters

Seems like the short answer, without getting derailed into discussions of specific use cases and frustrations, is simply this:

Advanced Lists allow users to work around places where Blackbaud's feature set doesn't meet their use case, and that limitations on the functionality of Advanced Lists negatively impinge users work in diverse and unpredictable ways.

Not sure how the 20k row limit was arrived out, but it does seem like it's in response to resource management issues. Perhaps identifying the users who are over-taxing resources and engaging with them directly would be more productive than a blanket policy change?

Jessi Walters

@Seth Battis
There are a couple things we are trying to do with my list of questions:
1) Identify common use cases our current toolset is unable to serve so we can address those directly in the product and reduce the use of “workarounds” for things needed by most schools.
2) Understand true spreadsheet, data extraction, and reporting needs so that we can deeply explore solutions for our next-gen reporting tool, whether it improves advanced list functions or, more likely, replaces it.

Unfortunately, the row limit was not related to resource management; that would be much simpler to solve. On a related note, a common misconception is that we could simply “give access to the database” - the system is far more complex and doesn't work that way. Instead we aim to modernize our architecture, improve performance, and simplify the user experience while expanding availability of fields and diligently addressing complexities introduced with more data connections.

We all wish this was quicker and easier to solve, but my team is deeply invested in getting it right.

Brian LeBlanc

@Jessi Walters Hey Jessi, thanks for being transparent with what is going on here. It's appreciated!

One thing I would request another look at in your goals is the “simplifying the user experience” target. In a SKY list? Absolutely; those are designed for everyday users and the UX is probably the top priority. But in an advanced list, where everyone using it is probably using it for a different reason and needs a different set of data output, I'd make the argument that the UX is actually less important than the performance and, to a lesser extent, modernized architecture.

If simplifying the UX of an advanced list without sacrificing the functionality that's already there (or even improving on it) is possible, then I'm all for it. I worry, though, that a simplified UX is going to necessarily mean a less fully featured product, and given that we are in some cases already working with one arm tied behind our backs, that's a big concern.

This isn't a request to complicate the UX, to be sure; it's simply a request to please make sure that whatever comes to eventually replace advanced lists, even as imperfect as they presently are, still at least gives us what we currently have.

John Ronan

@Jessi Walters,

It's good to know what you all are thinking and where you are considering going.

I'm glad the discussion has clearly described the fact that there are [at least]two classes of user:

• those who have simple needs and therefore need a simple UX and
• those who have to do everything else and [therefore] need to dig deep into the data.

I have always been in the latter camp and make almost no use of BEM reports and only slight use of non-advanced lists.

That said, I am often hindered by some holes in Advanced List access to objects in the database. Many of the new features have not been added in and even some old omissions have never been fixed.

As I see it, it's very important to have powerful, flexible access to [entire] the data model.
I must admit that having full SQL access might be a little dangerous, but it seems to me that some graphical representation of the [full] database in a query-building tool would be entirely feasible.

I hate to say it, but a good starting point would be the query system in Education Edge. This system, as old as it is, did a good and comprehensive job of presenting the db and facilitating access to it.