Learn From My Import Mistakes, Not Yours 2530

Learn From My Import Mistakes, Not Yours

Published
83a30496482a42cb6c2042e3307280e0-huge-li
Learn from your mistakes. I wholeheartedly agree with that old adage—for the most part. But when it comes to the life of your database, I think that preventing errors is best for us all. Given that I’m involved in educational training, I have the luxury of playing and experimenting in a safe Raiser’s Edge sandbox, which allows for some trial-and-error and a whole lot of learning from my mistakes. This is particularly helpful when using tools that have system-wide effects, such as Import. In this post, I’d like to share some import tidbits—because trial-and-error in a live environment is definitely something I do not recommend.

You cannot ‘undo’ an import
Let’s get the most important stuff out of the way first. The repercussions of a bad import job are permanent—so always create a backup of your data before running an import. If needed, you can restore from backup, but extra preparation and caution before the import can save a lot of extra work.

Take the time to properly prepare your import file
Patience is not one of my virtues. When it comes to import, however, I have learned that time spent upfront on the data file can save much time and headache when actually importing the data. Once I tried to import someone else’s file without properly understanding the structure. The result? A messy import and messy data.

Pay attention to the location and name of your import files
I’ve made this mistake for the same reason I made the previous one. Let’s just say that rushing through an import is a disaster waiting to happen. I usually keep all my import files in the same folder, which can be helpful. But when all files are named accounts1, accounts2, and so on, it’s really easy to pick the wrong file. Depending on how many files you import, consider adding the date and import type to your file name. Raiser’s Edge provides many checkpoints along the way to ensure your import is a success, but be alert—data that you really don’t intend to import can still get imported, as long as it is formatted appropriately. I rejoice every time I see the “valid import file” message, but I no longer trust that I am automatically importing the correct data just because the file is valid.

.xls and .csv are not the same thing
This one gets me almost every time. Because I use Microsoft Excel to prepare my import files, and the default file type is not .csv, I usually end up having to re-save my .xls files as .csv files after trying to import the .xls file.

Take the time to test
Earlier, I spoke against trial-and-error activity that can potentially affect a large number of records. However, importing only one or two records can be a great way to test importing without any widespread negative effects.

Create a query of your imported data, and check it
This seems too obvious to even mention but trust me, I’ve made this mistake as well. I worked with a customer who didn’t realize until a couple of weeks after an import that they had transposed last names and first names. Unfortunately, because too much time had passed, they could not just restore from backup.  

If you are new to importing or would like to learn more about this powerful and timesaving tool, check out the Raiser’s Edge: Import 1 and Raiser’s Edge Import 2 instructor-led courses.
 
 
News Raiser's Edge® Blog 07/27/2017 12:01pm EDT

Leave a Comment

20 Comments
Great Post. Love the test idea. I had a bad import once, had I done a test I would have saved myself the time it took to remove thousands of duplicates. This article makes imports seem less intimidating. 
Thank you all for the comments, feedback, and ideas! Happy importing!
Melissa
I'm not comforable with Import at ALL. :-(
All sensible advice Melissa - thanks  

"You cannot ‘undo’ an import"… worth noting however that most import hiccups can be rectified by globally deleting everything you have just imported and starting again. I have been working with RE for 16 years now and have carried out thousands of imports - hundreds of these were mess-ups but were resolved through the global delete feature, then reimported correctly. You do need to tread very carefully with global delete feature in RE though as it is very powerful… The only time I have ever had to restore from backup was due to a particularly messy global delete (and yes, I am still burning from the irony of this!).  

“Take the time to test”… absolutely! Robust testing will prevent any need to restore from backup, or need to resort to the global delete feature. I ALWAYS advise you carry out test imports including at least 3 records prior to any full import. Select records holding info in as many fields as possible (alumni with a full address, emails, phones, degree, subject, establishment, start years / leaving year / graduation year, class of award, attributes entries, addressee / salutation entries, etc.) and check to ensure if the correct info has landed in correct fields.  

One last comment on the subject of import time slot: Import when the least amount of users are logged into RE, or ideally when no one is logged into RE. It’s a killer when you run an import to assign appeals to 50K+ records and to find it did not go on to half a dozen records as these records were open by staff. You are then left to figure out which records the appeal was not assigned to and add them manually; identifying these records can prove difficult dependent upon level of experience / tech knowhow, etc.  
My favorite import tip enables me to create a query of any number of constituent IDs. I used to copy and paste 500 IDs over and over into the "one of" RE ID sections of a query until I had them all into my query. Then, a friend advised me about importing a single column of RE IDs, creating an output query at the same time. The result is a static query containing as many RE IDs as you'd like. After I used it the first time, I was surprised I had not thought of creating the query this way.
Great tips Melissa - thanks!   There are also great templates on Knowledgebase for a variety of imports so you know column headers, etc.
Also, if you're having issues with matching headers on your file to fields in the import, it can be useful to first create an import file with just a few records using the import tool. This will give you the headers you need and make field matching easier!
<caveat> Creating an Import file doesn't always have all the fields you need, so watch out for that, but it can certainly be a time saver.
Testing is most important.  You'll know youre import at least works and it's way easier to clean up 3 records and start over than 3000.  Thanks for the tips.
Thank you for sharing!
 
I'd love it if RE NXT hosted databases had the ability to have a training version of the DB that could be created from our live database so that imports could be tested and then run live - and if there was an explicit 'create backup' command that we could run on the hosted DB so we had a snapshot at just the right time before running a major update.

This is one thing that I really miss from before we migrated to RE - we had a 'training database' which we could always recreate from live data, and as it was just on local SQL we could take snapshots whenever we liked. It's a lot more worrying having to do everything live, or arrange a convoluted local copy of the database for testing. Tips are great - wish I could take a backup myself though!
I am slowly becoming more comfortable with import.  These are good sugestions, thank you.
Jen Claudy Jen Claudy Jul '17
If importing new records, generate your own ImportIDs and import them.  Then, if you find you've forgotten a piece of the import file, or you split off some piece (like a relationship record) as a separate import, you can easily keep them matched up.  Also helpful to know when looking at Properties, that this record was created as part of the XYZ Import, since XYZ is built into the ImportID.  (I've done this...and gotten myself into trouble when I was grumbling about someone creating a duplicate record that was very obvious.  Went to Properties to find out who created it to find that it was me!  But as part of an Import that I didn't have time to fully vet for duplicates.)  And, if you test a couple of records and forget to remove them from the big import file or delete the records created by the import test, RE will treat them as exceptions because of the ImportID (which already exists in RE from the test).

I use MS Access to create my import files most of the time, and I export & save the file as txt.  As for which file to import, I find & check the file I'm ready to import, make a copy and place it on my local desktop with the filename [Import.txt].  That's what RE Import looks for.  Never have to update the import file path in RE, and never have to search from within RE for the right file.  (And bonus, when I just want to look at the available Import fields, I can use that file, which is always there, to get RE to allow me past the first tab and look at the fields!)
Sandra,

Great quote! Testing and doing a double-check of data is so very important. Speed in the short term can add a lot more work in the future.

Melissa
Testing! Absolutely right!! Can never be too careful when you are uploading information. And always be sure to check your info carefully, before you jump in and do the whole thing. A favorite quote: "Festina lente" -- make haste slowly.
Thank you Sherrie! I'm glad you found them helpful.
Melissa
Great tips, Melissa. I've imported using a different DataBase at my prior job and these tips are definitely important. Thank you!
Great idea Joan!
Thank you for sharing!
Mel
Joan Perry Joan Perry Aug '16
Great "tips" Melissa...the "xls" vs "csv" gets me all the time. Though my ritual is to now save as "xls" in my main folder system and then save as "csv" to my import folder!
Thank you Nora!
This is great, Melissa. Thank you!

Share: