Many (all?) Notifications broken for at least 3 weeks(!) - Anyone else dealing with this?

Daniel Wallach

Noticed on Dec 19 that one of our notifications could not be switched to Active or Inactive. We originally thought this was just the one, but ended up seeing the same problem on many others. Reached out to support then and they've reported it internally, but no progress.

We then discovered that for notifications already set to Active (so in theory useable), although we could update the notification text, with the updated text showing after clicking Save, the old text was still going out! We only noticed that problem on auto-confirmation messages to parents for submitted enrollment contracts, but at this point I have to assume it affects others.

It's pretty clear there's a bug with notifications generally, but my main concern is the complete lack of communication about it from Blackbaud. If we hadn't caught it ourselves we would have never known about it. It's not something included on https://status.blackbaud.com/ or in monthly product updates. And the only follow-up we've had is essentially “we're working on it”.

Has anyone else been dealing with this? We basically can't use any notifications that aren't already active and using up-to-date text. #frustration #support #bug

Jessi Walters

@Daniel Wallach We sincerely apologize for the inconvenience and headache. Thank you for following our optimized path by reporting this to Support. Fixes for the 2 issues you've reported were released today. When a bug is widespread and affecting many users, then we prioritize a fix. Generally speaking, the greater the impact, the higher the priority. This was handled uniquely - despite wildly few reports from customers, this was resolved ASAP because we understand the value of your voice with your communications.

If you have further questions, please reach out to your account team so we can better assist you.

Brian LeBlanc

@Jessi Walters Hey Jessi, since I was one of those few reports, I want to give you a pat on the back and a little bit of feedback as to why, at least in our case, it took so long to report this issue. This was a very strange issue because it wasn't the typical “thing is not working, report to support that thing is not working, support will let someone know to fix thing” type of trouble ticket. Notifications were working; they just weren't working the way we thought they were!

And since our staff members weren't the end users receiving the notifications, there was necessarily no QA process in place to review what was being sent. It would have been a bigger deal if nothing was going out, but again, something was, and the end users - our parents - were none the wiser that it wasn't what we had intended to be sent. (In our case it was just some minor verbiage changes, not major wholesale overhauls, so not a big deal for us but it could have been in another scenario.)

In addition, we use official notes much more than actual notifications, and since those are largely just HTML-code placeholders in the notifications themselves, they weren't affected at all by this problem.

For me, the canary in the coal mine was when the change to making an inactive notification active didn't commit to the database. That's what prompted me to do more digging and realize that there was a bigger problem. Absent that, it would have probably dragged on even longer. This seems to dovetail with Daniel's experience as well.

This wasn't a case of schools sitting on our hands and assuming this wasn't an issue. It was an issue we had no idea was an actual issue, because it was an “invisible” issue that wasn't immediately apparent to us. Believe me, I am very sure that support has my picture posted all over the place with loud neon warnings for the number of times I contact them. I think this was just a weird coincidence more than anything else.

Susana Norcini

@Jessi Walters
Jessi, I do not believe this issue has been resolved. Yesterday I noticed the notifications that go out to parents after submitting a re-enrollment contract are not saving. (Enrollment Management > Communication > Notifications > Online Contract Submitted and Online Contract Processed). We have the current school year listed and are unable to change it to next school year. This is a big problem as it is a communication for a legal document - the re-enrollment contract for 2024-25.

I reported it to tech support and received a response: “Currently, there is no solution in place to get in and adjust the online contract submitted notification. This issue is currently being addressed by our product development team and we are actively working toward a resolution. At the moment there is no timeline for resolution.” I sure hope this can be resolved this week; we are releasing re-enrollment contracts Monday morning.

How are other schools dealing with this issue? Any suggestions? Thanks.

Nancy Kierstead

@Jessi Walters

Reminders for Contracts and Notification for Contract Processed and Contract Submitted are all not working. I cannot update them for this year, so the contracts going out next week are impacted. Any set time for a fix? I'll add a case tonight, but it sounds like a known issue, so hoping it's already on the timeline.

Jessi Walters

Hi all, another update was pushed into production on Friday evening. Please let Support know if you continue seeing any notification issues and we will escalate.

Lauren Marcus Eisenberg

@Jessi Walters I am having serious issues with enrollment contract reminder notifications NOT working. (See Support Case #019619850) Daniel Wallach - I feel your pain!

Jessi Walters

@Lauren Marcus Eisenberg thank you for submitting a ticket and for flagging me. I've got the team looking into it now.

Lauren Marcus Eisenberg

@Jessi Walters Waiting is very painful and quite disruptive. It is holding up our testing process and adding undue stress for me and my colleagues.

Lauren Marcus Eisenberg

@Daniel Wallach @Nancy Kierstead @Susana Norcini Brian LeBlanc - How are you managing around this grave issue? I feel as though our hands are tied.

@Jessi Walters Any progress?

Daniel Wallach

@Lauren Marcus Eisenberg

Support updated us recently that a fix was in place. On our end the issue seems to be resolved, but I'm keeping an eye on it just in case. Any issues still on your end? @Nancy Kierstead @Susana Norcini?

Susana Norcini

@Lauren Marcus Eisenberg

Same with us. Support updated us that the issue was fixed. On our end the issue was resolved.

Lauren Marcus Eisenberg

@Daniel Wallach @Susana Norcini I am testing the contract reminder email, but it is not being generated. What magic did you use to get it to work?

Lauren Marcus Eisenberg

@Daniel Wallach @Susana Norcini My notification hasn't worked since I started testing on 2/1.

Nancy Kierstead

@Lauren Marcus Eisenberg I am seeing we had contract reminders go out this month. Make sure both the reminder and contract are active.

Susana Norcini

@Lauren Marcus Eisenberg
We are done with re-enrollment; the last reminder emails were sent on Feb. 1 and they went out successfully. I recommend you work with tech support.

Daniel Wallach

@Lauren Marcus Eisenberg
Have you confirmed the following:

• Both the contract and the reminder are Active
• The contract is assigned+ generated for the appropriate people?
• If sending the reminder out manually, Reminder option is set to Send when activated before you click Save & Close.
• If scheduling it ahead of time based on due date, the appropriate option is checked off and the contract due date is correct

Lauren Marcus Eisenberg

@Susana Norcini Yes - on 2/1 the reminders worked for testing. I'm happy for you that you got through this process before they introduced the bug.

Lauren Marcus Eisenberg

@Daniel Wallach Yup: contract and reminder are Active. I assigned it to our sample students who are my children: LSlunch, MSLunch, and HSLunch. As their Parent, I see the bar indicating 3 contracts, so that points to the issue of the notifications.

Brian LeBlanc

@Lauren Marcus Eisenberg We're running fine on our end. I know that's not exactly music to your ears, but we've finished re-enrollment and got through the first round of new enrollment contracts with notifications going out as expected.

Jessi Walters

@Lauren Marcus Eisenberg and all here, I saw one new report of a notification issue today - though notifications are still sending in that case. If you experience any notification issues today, please reach out to Support so they can rule out any setup errors and attach your case to the issue if needed. They are truly best situated to dive into the intricacies of each scenario on the quickest timeline.

Lauren Marcus Eisenberg

@Jessi Walters Just got off the phone with Support who was able to replicate the issue and didn't see any setup glitches. Oddly, an old contract that had to be reissued pushed out the notification as expected. SOS!!!

Kelsey Huijgen

@Lauren Marcus Eisenberg my team is looking into the issue you reported now. We'll update you as soon as we get to the bottom of it!

Kim Hughes

@Kelsey Huijgen I had a Wednesday reminder (1 day before the due date) successfully sent. Then made a change on Wednesday to send another reminder on due date, Thursday. Those didn't send. I called support this morning, per this thread, and was informed that when the support person had a similar case, support was told by internal team it was working as expected and that changes needed to be there 24-48 hours before the reminder would go out. Using reminders on contracts for 6-7 years, I had no idea this was the case. Was support told correctly? If so, anyone who is making changes and thinking that they are immediately effective will be as surprised as I was.

Brian LeBlanc

@Kim Hughes:

@Kelsey Huijgen I had a Wednesday reminder (1 day before the due date) successfully sent. Then made a change on Wednesday to send another reminder on due date, Thursday. Those didn't send. I called support this morning, per this thread, and was informed that when the support person had a similar case, support was told by internal team it was working as expected and that changes needed to be there 24-48 hours before the reminder would go out. Using reminders on contracts for 6-7 years, I had no idea this was the case. Was support told correctly? If so, anyone who is making changes and thinking that they are immediately effective will be as surprised as I was.

Kelsey, not to pile on, but this is exactly the type of frustrating response that we hear from internal resources (via support) all the time that makes us want to break things. If this is the case, why is it not documented anywhere? There are so many of these little gotchas buried deep inside the database that it's inexcusable that we just stumble across them all the time, and it's just infuriating, because we're made to feel like we're the fools for not knowing that this is just the way it's always been and why are you bothering wasting our time for something that is obviously (eyeroll) working correctly.

I realize that this would be a heavier than heavy lift, but at some point we could use a top to bottom rewrite (or at least a review) of all the product documentation. If the last-updated date on a KB article is more than, say, a year ago, chances are it needs a lookover to make sure it's still accurate. Otherwise, why should we be the ones that are crawling to support all the time to ask seemingly benign questions like this when it really needs to be Blackbaud's responsibility to make sure their products are properly documented?

Susan Ott

@Jessi Walters I just found out that reminders for our faculty/staff reenrollment contract DID send on 2/9, 2/14, and 2/15. Reminders for the regular re-enrollment contract should have sent the same days but did not. Both contracts and notifications are set up exactly the same. Could that indicate a possible role connection? We have a case open and asked for it to be included on this issue.

Kelsey Huijgen

@Kim Hughes it looks like you were using the new “Days after generated” due date setting on your contracts. Reminder notifications were set up slightly differently when using this due date setting - any changes made to the reminder options on the contract are set to only apply to contracts generated after making that change and not for contracts that have already been generated. So in your case, I believe that would explain why making that change didn't result in reminders being sent. We made a change for the “Send when activated” option so that it does re-send notifications for existing contracts instead of just applying to new contracts. I'll have to check to see if our change applied to the other due date options as well, meaning changes made after generating contracts will apply to those existing contracts.

I will connect with our support team to go over your case to see if it could have been handled differently. New features always have nuances, so my apologies for the frustration and confusion this has caused!

Kim Hughes

@Kelsey Huijgen Thanks Kelsey for reaching out. Yes, this was our first go at using the new functionality. It gets very hard to tell what is causing issues. Is it new functionality and how it works or is it other issues. Now that you have said regenerating is the key, I'll know for the future.