Employment Data Imports

Shawn Schaeffer

We have several employment files from vendors where the data was either pulled from LinkedIn or from a survey, which makes for non-standard employer names, like 8 different ways to say Fifth Third Bank.  Going through 100,000+ lines in the files and trying to match what we have in the system will be extremely time consuming.  We would love to hear what you have done and what has worked for you with importing this kind of data?  Any help with this appreciated.

 

Matt Hall

Hi Shawn, a couple of things I've learned from having to tackle this before:

• It helps to create a lookup table (temporary or perhaps permanent) with normalized organization names.  By normalized I mean:
  • Remove punctuation and standardize spacing
  • Remove leading articles ("the", "a", "an")
  • Standardize or remove common abbreviations ("& = and", "llc. = , LLC", "corp. = corporation", ...)
• Do this for both the org name and all aliases
• Run the same normalization against the incoming data 

I'm usually doing this in C#.  Refine the above steps by repeatedly running against a test set and observing the matching results as new standardization/normalization steps are added--refine the steps based on patterns observed in matches/misses in the sample set.  That won't get you 100%, but ideally that gets you more than half.