Data quality is a critical issue affecting data warehouses, (DWs) yet there appears to be no sustainable solution with regard to quality input into operational systems that ultimately feed DWs. Is there any evidence to suggest tying bonuses or other forms of incentive to pre-determined data quality hurdle rates (say 95 percent for the quality of data under management control) improves data quality at the input end, thereby alleviating some of the dependency and additional expense of a DW data cleansing tools?
Les Barbusinski’s Answer: In my experience, data quality that is not enforced programmatically is not enforced at all. So offering incentive programs, running awareness campaigns or changing procedures won’t help. Paradoxically, it’s very difficult to get management to approve changes to operational systems that would tighten up edits and/or business rules. Such changes are perceived as "high pain, low gain" and are rarely budgeted for. Hence, the need to cleanse the data once it arrives in the data warehouse. Seems silly at times, but "that’s just the way it is."
Clay Rehm’s Answer: People are motivated by positive feedback that they are a worthwhile contribution to the company. Many times employees do not receive the attention they desire or need to perform up to expectations. I firmly believe that throwing money at this problem will not fix the situation.
You probably know some people who are putting their time, watching the clock, mumbling to themselves how much they hate their job, and you expect to have 100 percent accurate data? Look at a typical data entry job it gets boring very quickly, and the position is not respected by co-workers and even looked down upon. In underrated and undervalued positions such as data entry, we must tell the employee that we value their work and their loyalty.
Many data entry staff do not even understand what they are typing - I have heard comments such as "no one ever told me that!" There is a lack of training and education on the actual data that they are keying in.
You can have all the edits in the world, but that will not keep all the inaccurate data out. Happy, loyal, passionate people do. Find ways to keep the data entry staff motivated and excited about their work. Conduct regular performance reviews. Try providing company picnics, weekly barbecues, paying for education and flex time work hours.
Larissa Moss'Answer: Lack of data quality has a huge impact on the usefulness of data loaded into a data warehouse. Many company executives are not even aware of the extent of their dirty data problems and are surprised to learn that the information from their data warehouse is often not reliable. Misguidedly, these executives turn to IT to solve their dirty data problems as if these were technical problems instead of realizing them as business problems. IT does not invent data, IT does not set policy and business rules for data, IT does not create data or update data - business people are in charge of all that. IT merely provides the technology means to store and access data. It stands to reason that business people who invent, set policy and business rules, create and update data must be accountable to other business people who access and use that data. In other words, data originators must be accountable to data consumer! Therefore, data accountability and responsibility must be spelled out in business policies and in procedures that are enforced by the business people. Enforcement includes incentives and rewards for following these policies and procedures; it also includes punishment for breaking the policies and procedures. If the business people who are the data originators were incented (or punished) for the quality of data under their control, they would change their requirements for edit checking in operational programs/systems written by IT, they would also change their quality assurance process for data entry, and they most likely would change their data entry training to include these policies and procedures. An excellent book on this subject (which includes evidence of companies following and companies not following data policies and procedures) is Larry English's Improving Data Warehouse and Business Information Quality by John Wiley & Sons.
Register or login for access to this item and much more
All Information Management content is archived after seven days.
Community members receive:
- All recent and archived articles
- Conference offers and updates
- A full menu of enewsletter options
- Web seminars, white papers, ebooks
Already have an account? Log In
Don't have an account? Register for Free Unlimited Access