What Does it Mean to Scrub Data?

March 26, 2019 Jacqui Wishart

data management tips for schools header image

Have you ever heard the term “scrubbing data” and wondered what it meant? Scrubbing data is a review process to remove duplicate records and inconsistent entries prior to importing a file. It's an important step in school database management because the cleaner the data, the less likely it is that inaccuracies will disrupt other workflows. Yes, it takes some time to review files prior to import, but it’s much easier to make changes in bulk in the import file than it is to edit the database online. 

Data scrubbing is best with Microsoft Excel. If you're not comfortable with Excel, there are links below to help with some of the scrubbing tasks. You can also use other spreadsheet tools like Google Sheets.  

Follow these tips for scrubbing user profile data prior to import: 

  1. Remove duplicate records from the file so that duplicate records are not created in the system.  
  2. Remove redundant ethnicities such as "Caucasian" and "White." Filters in Excel can help you identify redundant entries.  
  3. Remove redundant religions such as "Christian (Non-Denominational)" and "Non-Denominational Christian." Filters in Excel can you identify redundant entries.  
  4. Check to make sure phone numbers are formatted per the school's style guide: 555-555-5555, 555.555.5555, or 5555555555. Controlling how this data is returned in Excel can help standardize phone number values
  5. Ensure that the file's column headers match the column headers outlined in Blackbaud's file layout guide. If column headers don't match the correct layouts, then the data won't import. Do not supply additional columns like Role or Username in a General User import file.  
  6. Ensure that all users have a Host ID. Filters in Excel can help you to identify entries without a Host ID
  7. Ensure that all users have a unique Host ID. Filtering for duplicate values in the Host ID column is the best way to identify redundant entries. 
  8. Check to make sure that dates are in MM/DD/YYYY format for birthday, hire date, or enrollment date. Excel has a helpful tool to format dates
  9. Check to make sure Grad Year is in YYYY format. Filters in Excel can quickly identify if any aren't in the correct four-digit format.  
  10. Ensure that the salutations are for the address record and not the individual people within the household. 
  11. Ensure that the school levels and grade levels match what is in Blackbaud's education management system under Core > Settings > School information > School & Grade levelsFilters in Excel will enable you to identify entries that don't match what is in Core. 
  12. Look for data shifts in your file(s) for both the fields and records.  Examples may include:  
    • Data swapped in two fields.  
    • Data shifted left or right by one or more positions to the wrong fields; this may occur in all records or in a single record.  
    • Data shifted up or down in records (e.g. host IDs associated with the wrong users) 

Ensuring that data is accurate before import is the first step to having clean data. If you have any questions about data imports, please contact Blackbaud Support

About the Author

Jacqui Wishart

With a decade of experience working at Blackbaud K–12, we conservatively estimate that Jacqui has assisted over a 1,000 schools. Currently working in Managed and Retainer Services, she is passionate about helping schools streamline their processes with Blackbaud's education management portfolio. A New Hampshire native, Jacqui now resides in South Carolina with her two dachshunds, Fennel and Sage.

More Content by Jacqui Wishart
Previous Video
Blackbaud ID is here!
Blackbaud ID is here!

Learn how your school can use BBID to connect to your preferred identity provider for centralized managemen...

View More
Keeping Students Safe with Bark
Keeping Students Safe with Bark

Learn how Bark for Schools can help keep kids safe by monitoring Google Suite and Microsoft Office 365 acco...