Files & Forms Upload -- Flash end of life December 2020 -- any news from BB?

Lucien Piekut

Hello, not sure if I've missed the memo -- I've been checking the updates and going to the BB roadmaps. One item that hasn't shown up on my radar is for any schools using the K12 Files & Forms bulk upload, it uses Flash which is being discontinued December 2020.

https://www.adobe.com/ca/products/flashplayer/end-of-life.html


Wondering if any users (or Blackbaud) have any news where this is going? Workarounds?


Should Files & Forms upload not work come January that will be a fairly big impact on a number of schools.

Morgan Kunze

I hate to be the person dragging this conversation backward, but I have not used Files & Forms bulk upload, and can't even find where in the software I would do that!  I can't answer your Flash question, but I'd love if you can give me a quick path to this utility.... just in case it doesn't die with the Flash sunset. Thanks! -Morgan

Lucien Piekut

Morgan Kunze:

But I'd love if you can give me a quick path to this utility.... just in case it doesn't die with the Flash sunset. Thanks! -Morgan

No problem Morgan, it's a utility found under Core in K12/ON product. Under System Tools->Bulk Upload->Manage User Files


From there you set up folders, and subfolders to organize files which can then be attached to a User (students, parents, faculty, etc) based on what you upload.

To bulk upload files: Create a CSV which will contain the User ID and the Filename. Then in step 2, you will be prompted to choose the files that were listed under filename which corresponds to the User ID the file should attach to. This allows unique files to be attached to every user you uploaded to. Good for letters, or personalized reports.


The part that Flash controls is the upload specifically. I contacted my Ontario colleagues (thanks!) and one of them got this response:

BB Support

Our team is aware and working on making sure the functionality is still available once flash is discontinued. You will see no stop in its current functionality.

Morgan Kunze

Thank you, Lucien! 

Janet Wittenberg

Good news - we're down to the wire but we are working on a replacement for Bulk upload user files that will either be released Dec 22nd or the first opportunity in January.   Sorry we didn't call this out in our Q4 product update.

Lucien Piekut

No problem at all Janet saw it in patch notes today, thanks for you & your team's efforts.


For anyone interested:

Michael Panagos

Janet Wittenberg:

Good news - we're down to the wire but we are working on a replacement for Bulk upload user files that will either be released Dec 22nd or the first opportunity in January.   Sorry we didn't call this out in our Q4 product update.

I saw the Flash removal information on the December 15th release notes earlier this week (Monday), but it was removed. So this is definitely happening in the final hour?

Yitzchak Feiglin

Hi Janet, does that also relate to the PushPage bulk upload? I'm still seeing Flash trying to load there.

Janet Wittenberg

Hi there,

We are aware of the need to replace flash in Pushpage uploads as well, but we did not solve that at the same time as the Misc File uploads, so there may be a short delay to get that resolved.


Janet

Julian Daniel

The new non-Flash Bulk Upload (Core/System Tools/Bulk Upload) in K12/ON/Core is NOT a suitable replacement, since it doesn't replicate or improve on the functionality of the old solution.


It seems to be based on the same system used for uploading in other areas of K12/ON/Core, and what I've discovered thus far is that it supports a maximum of 122 uploads at a time before it starts erroring-out, there's no way to save your progress, and super-annoyingly they changed the fieldname from userid to user_id.


First, the feeble file upload maximum: I need to regularly upload a few hundred files, and upload 3500 statements monthly. The original Flash uploaded from a couple years ago could handle (albeit slowly) the whole upload in one shot. Around a year ago, they did something to screw it up, and then it maxed-out at 999 files without erroring-out. There was also a bug where if you didn't save your progress and back out that sometimes it would enter an infinite loop on uploading 999 files and you'd lose all your progress to-date. Now with the new non-Flash uploader, if you drag (or select from disk) more than 122 files, it does a 5-8 second "file upload error" with red flash at the top of the screen FOR EVERY FILE OVER 122! There's no way to cancel this helpful timewaster beyond canceling the upload thus far. So for my regular 3500 file upload, I will have to drag 122 files to the window 28 times and hope I don't miss the drop area and end up replacing my current browser window with an uploaded file and have to start from scratch.


Second, you can't save your progress: in the Flash uploader, you could save your job without processing it, and then go back in and carry on where you left off. With the new system, if you time out (ie answer a phone call while uploading) or drag your files to the wrong area of the window, or breathe funny, then you LOSE YOUR CURRENT UPLOAD JOB! The only way around this is to press the Update button at the bottom of the screen...which then tells you that you missed uploading a whole bunch of files and do you want to just upload what you have so far. Great! I've uploaded x files...but where do I continue from? There's no way to go back to the job where you left it. It's processed, and then you need to start a new job and work out where you left off.


And third: WHY CHANGE THE FIELD NAME?!?!?! It's been userid since the beginning of time...why is it now user-UNDERSCORE-id? All the other imports and uploads till now have been using userid with no underscore - why change this?


And finally, why on earth was this MAJOR feature not updated months before the December 31, 2020 deadline for the demise of Flash? Why wasn't there a testing period where all the horrible bugs could be worked out and at least there was a working system to fall back on? Now we're stuck with a crappy file upload that doesn't work, and I'm in a holding pattern with thousands of files to upload and no way to do it safely and reliably...and in anything close to the old amount of time it took with the Flash uploader.

Janet Wittenberg

Hi Julian,

I hope you have opened a support case as we recommended.  We would like to work with your specific scenario to make sure we can correctly diagnose what is limiting you to 122 files.  Your feedback on our releases is critical, and we will be working to fix this process to eliminate the relatively small limit on uploads.  That is clearly not a desirable behavior to have all the error messages.

We researched the user of "user_id" - this is the consistent naming in data imports, bulk upload photos and bulk upload temp passwords. So unfortunately it will require a change in your manifest files from the previous version of Bulk upload Misc Files.  In the long run this will be more consistent with the other import/upload tasks.


Using Bulk upload photos as our newer pattern, we do not intend to continue the feature of saving your work in progress.  However you can upload just a portion of the files that are listed in your manifest in each job.  I think this change in behavior will be less troublesome when we can allow you to upload all your files in one job, or at least a much larger portion of them.


Thank you for your patience, we will notify you via a support ticket when this fix is made.

Janet

Lucien Piekut

Thanks Julian, I was considering updating my post as I have been experiencing the same issues. I have a ticket with Blackbaud open right now, their developers are aware of the issue.


What I've seen:

• Breaking into chunks of 100 files at a time helps, but does NOT resolve the issue
• The error during upload will read "File not uploaded" or sometimes if you're lucky... "Firewall is blocking"
  • When I experienced the Firewall error I switched to a VPN just to continue trying to update, which sort of worked
• The files that fail are not at all consistent, so you can't really see when a file is missing/has not been uploaded without scrolling through the list

Realistically their team will eventually learn how to get this tool to NOT throttle uploads, but to have released it before was not a great move. This has added considerable troubleshooting time to get my weekly uploads running smoothly.


Perhaps a lucky Tuesday update tomorrow will see some results.

Julian Daniel

I submitted case #018155514 in reference to the non-Flash-upload-replacement-doesn't-work issue (and cases 018131693 and 018105479 begging for a Flash replacement prior to the December 31 deadline), and provided a 96K PDF file as an example of the type I was attempting to upload. I found that it happened after 122x ~100K files, so perhaps 122 is a magic number for throttling, or perhaps 12MB (122x 100K) is a maximum size limit.


The save-upload-job-so-I-can-continue-later feature in the Flash uploader was handy, since it's sometimes not possible to complete a large upload in one sitting. At the moment, I try to schedule my upload sessions for after-hours, because otherwise I inevitably get interrupted. With the old Flash uploader, at least I could save my progress and continue later. With the new non-Flash uploader, I HAVE to upload or I lose all my work thus far.


 

Janet Wittenberg

Sorry we didn't quite make today (Tuesday) but this is a top priority for the 26th or Feb 2nd depending on how quickly we can solve the issue.

Janet

Lucien Piekut

Thank you Janet for staying on top of this issue, and this thread. We're making due in the meantime.

Janet Wittenberg

As of today, we expect you have the bandwidth again to handle large bulk upload jobs. If you continue to have an issue, please reach out to Support again. Thanks for your patience.

Janet

Julian Daniel

Unfortunately, it now dies sometime after 2000 files...but before the 3500 I need to upload.


I can successfully upload around 2000 files (I did so in batches of 999 files, as before with the Flash uploader), but when I dragged over the third batch, I got the dreaded repeating red "File Upload Failed" banner for the next few hundred (can't tell how many, since there's no typical progress bar or counter).


Rather than watch the red banner every two seconds for the next hour, I pressed the Upload button, hoping to upload the first 2000 files, but got a message that I was about to process 3000 files - which wasn't accurate, since I was getting red banners saying most of the third batch of 999 had failed uploading. BUT pressing the button DID get rid of the annoying red banners.


So I tried re-uploading the "failed" 999, and this time everything worked flawlessly for some reason. Then I uploaded the final 503 files and again, no error messages.


I held my breath then pressed the "Upload" button to process the ~3500 100K PDF files I uploaded...and got a red banner stating "Error saving job"





So file upload has been fixed so it can handle more than 122 files - now it looks like the max is ~2000.


The interface is pretty but useless for large uploads - I need to see a list of filenames still to upload, not a total list of files with green (or no) checkmarks that I have to scroll through to try to find out which files are missing.


And some sort of upload progress bar or counter would be a blessing - the Upload button changes to Loading and there's a pulsing red line at the top of the screen, but no other indication that anything is happening.

Lucien Piekut

Thanks Janet, confirming on our end at around ~700 files it is working without issue, and much much faster upload than before. On our end I would say this process was an improvement.


For Julian however, sorry to hear it's having troubles past the 2000 range. In my case for the past couple weeks I was using a Python script to:


1) Separate the split files into their own folders

then

2) Make CSVs of the user+file from the various folders


Essentially doing all the breaking up of the files and automatically creating a CSV for me. This made the process much less of a headache. Feel free to PM if you have any interest in something like that.

Janet Wittenberg

As of last week we have expanded our bandwidth to support larger upload jobs. Thanks all for your patience.

Janet