I once read a posting for a VP of Sales that listed ensuring data hygiene as a top priority. It said so three times, as if repetition were rigor. And I remember thinking, with what? How exactly is a VP of sales supposed to ensure the hygiene of a phone number a rep typed with their thumbs? The honest answer is that nobody in that room could have told you what the phrase meant, because data hygiene is one of those terms everyone uses and no one defines. I have spent twenty-five years in and around data, and I am going to tell you something that sounds like heresy and is closer to plumbing: data hygiene, as it is usually imagined, does not exist. There is no clean state to return the data to. There is only fidelity to a source, and fitness for a use, and the word hygiene hides both.
The word is doing dishonest work
Hygiene is borrowed from washing. Dirt is foreign, you scrub it off, and underneath is the clean, true thing restored. Applied to data, the metaphor smuggles in three assumptions, and all three are false. That there is a correct version the data got dirtied away from. That the dirt, the typos and the duplicates and the gaps, is separable from the real data and can be lifted out without losing anything. And that lifting it out is always good. Hold those up against the actual work and they come apart one at a time.
Two different things share the name
There is a real thing hiding under the word, though, and I want to be precise, because I am not telling you to stop checking your work. Two completely different activities both get called data hygiene, and the entire confusion lives in the fact that they share a name. The first is production integrity: did the file we delivered contain the number of records we extracted, did it pass its checks, is that email address even shaped like an email address. That is real, it is verifiable, and I do it every single day. The most reliable tool I own is a record count. Nine times out of ten, when a client writes to say something is wrong with their file, it is a count mismatch, and more often than not the error is on their end, because they anticipated a number the county did not deliver. That kind of hygiene is just careful counting, and counting is honest.
The second activity wears the same word and means something entirely different: correcting the content of the data toward what someone believes is the truth. Overriding the county. Standardizing the spellings. Bolting on a better dataset. That is not hygiene. That is editing, and the trick the word plays is to borrow the unarguable legitimacy of the first activity to license the second. You cannot argue with counting records, so under that cover someone talks you into improving a square footage away from the assessor, and calls it the same word. It is not the same word. Editing a mirror is how you crack it.
You mirror the source
Because that is what this data actually is: a mirror of an authoritative but imperfect source. In my world the source is the county, and the county is the tax basis, which means for most purposes it is not merely the best answer, it is the only answer anyone can check. Years ago my company was offered a few million appraisal records from a large firm, and the pitch was seductive: this is appraisal data, appraisers are licensed, they measure carefully, it can fill the holes in our property characteristics. Then you look closer. A few million records against roughly a hundred and fifty million parcels in the country, and LA County alone is about nine million, so the impressive number turns out to be a rounding error. Many of the records were years old, with no permit data to say whether the house had been renovated since. And a good share of them flatly disagreed with the assessor.
So which do you believe? You believe the county, because the county is the standard everyone else can verify against. If you correct a bathroom count away from the assessor, you have not made your data more accurate. You have made it different, and different is precisely what a customer discovers when they check you against the county website and concludes you are the one who is wrong. You cannot prove you are right without walking into the house, which you cannot do. So you mirror the source, faithfully, and you refuse the seductive fix. Different is not more accurate. Different is just different.
You want the errors
The clearest proof that clean is incoherent is spelling, of all things. Run a spell-checker across a name file for people-search and you will destroy the very thing the user needs. Someone hunting a person, an investigator, a collections agent, wants the aliases, the common misspellings, the name someone wrote as Johnathen on a booking form when the birth certificate says Johnathan, the typo a clerk made twenty years ago that is now the only thread connecting two records. Standardize all of that into one correct spelling and you have quietly deleted the signal. The same goes for any free-form text, a legal description, anything transcribed off a document: you want it to match what is actually printed, misspellings and all, because the value is fidelity to the source, not conformance to a dictionary. In these datasets, correcting the errors is the error.
Even a good fix rots
And even when a fix is genuinely good, it can spoil with age. Say a county stops releasing property characteristics one year, because owners lobbied for privacy. Your automated process flags the gap, and you make a reasonable call: carry forward last year’s characteristics. In years one through three that is a fine solution. Run it for a decade, though, in an older metro where permits and additions and full rebuilds never stop, and your fix has quietly drifted into fiction, reporting square footage that no longer exists next to a tax bill for a house you are no longer describing. This is why experienced data people get conservative to the point of looking timid. They have watched enough fixes-over-here break something over-there to have learned the deepest rule of the trade: your job is to be an accurate mirror of the source, and almost every attempt to be better than the source is an attempt to be different from it, which is a slow way of becoming wrong.
The one honest kind of better
There is exactly one axis on which one vendor is honestly better than another, and it is not accuracy, it is currency: being a day or two ahead. And even that carries a catch, because when a record is late it is usually late for a reason, it failed a check, the parcel numbers changed, the match came back wrong, and a fresh record that did not pass QC is worse than a slightly stale one that did. So currency is a tradeoff too, not a virtue. And it only matters at all for particular uses: the service company that wants to be first to mail a new owner, the investor who wants to reach a distressed owner before the crowd, the analyst whose monthly report goes out on a fixed calendar day that has nothing to do with when the county actually files anything, a date chosen for the meeting it feeds and the headline it earns, not for the data. Fresh is not better. Fresh is fitter, for some purposes, at a cost.
The words that are actually real
So strike the word. There is no hygienic state of a dataset the way there is a clean state of a plate, because a dataset is not dirty. It is a faithful and partial record of a messy world, and what counts as an error depends entirely on what you are trying to do. The misspelling is garbage to a mailing list and gold to an investigator. The old appraisal is an upgrade to someone filling holes and a landmine to someone who has to match the tax roll. Quality was never a property the data had. It is a relationship between the data and your purpose, which is why the only honest questions are the boring ones. Is it faithful to its source? How much of it is actually there? How fresh is it, and what did that freshness cost? And above all, is it fit for what you are about to do with it? Ask a vendor to promise you clean data and you have asked for a unicorn. Ask for fidelity, completeness, currency, and fitness, and you have named the four things that are real, and you will finally know what you are buying.
Further reading on why there is no such thing as raw or clean data: Lisa Gitelman (ed.), "Raw Data" Is an Oxymoron (MIT Press, 2013); and the data-quality literature that defines quality as "fitness for use" (Richard Wang and Diane Strong, "Beyond Accuracy," 1996).