So, this blog is dedicated to stepping outside the comfort zone once again and into the world of chaos. Not only may you not want to persist in your data quality transformations, but you may not want to cleanse the data.
Current thinking: Purge poor data from your environment. Put the word “risk” in the same sentence as data quality and watch the hackles go up on data quality professionals. It is like using salt in your coffee instead of sugar. However, the biggest challenge I see many data quality professionals face is getting lost in all the data due to the fact that they need to remove risk to the business caused by bad data. In the world of big data, clearly you are not going to be able to cleanse all that data. A best practice is to identify critical data elements that have the most impact on the business and focus efforts there. Problem solved.
Not so fast. Even scoping the data quality effort may not be the right way to go. The time and effort it takes as well as the accessibility of the data may not meet business needs to get information quickly. The business has decided to take the risk, focusing on direction rather than precision.
Reboot: Don’t worry about bad data. Precision is not always the end game, and the business is balancing risk with reward. Understand the decision process. Decisions are based as much about what the data shows as experience and anecdotal evidence. This trifecta is a balance, and data may be a catalyst or validator, not the only guide. To determine if data cleansing if required, consider time available, deviation of analytic results to perceived or accepted hypothesis, and risk within the context of data use. It may be that data quality really doesn’t matter and the data is good enough.
However, don’t throw away your data quality best practices yet. Data quality measures and indexes created for data governance give you guide posts to build a trust continuum for data that helps determine when and when not to put data quality rules and efforts in place. Continuously profile data sources and the quality of data feeding analysis, not just to correct but to inform on when action is necessary.
Interested in more about the trust continuum Read Alan Weintraub’s recent report on information governance.
For part one of this blog series, click here. This blog originally appeared at Forrester Research.
Orientation is generally considered the most important mode. The Decision that emanates from it is wholly dependent on the quality of the data gathered during the Observe mode. If you buy this, then I guess it is possible to argue the reverse. The quality of data drives a quality decision.
In the end it is true that you can make a Decision...at any speed...but the quality of it depends on the quality of the data, which when combined with the pilot's knowledge and judgement about his/her surroundings during Orientation dictates his/her Action and the ultimate outcome. For a fighter pilot, that means precisely nailing the target or killing off a wedding party in some village near the target.
I can't help but consider what my supervisor said to me in my first "professional" job as a research assistant in a lab at Children's Hospital in Boston: "Don't worry. If you make a mistake, no one dies." Now across the street in the main hospital where the care is provided, that was very much not the case. I think your premise is true within spheres, such as most business settings where no one immediately can die and there is time to recover from a poor decision resulting from poor data.