The role of a Shadow Data Team in maximizing business outcomes
Applications for data management, analytics and information delivery require higher collaboration between business and IT. Analytic applications deliver value to business only when the availability of right data, at the right time, in the right form to the right user is realized.
The teams that build and manage analytic applications need to exemplify higher business understanding and be agile to accommodate business demands. Hence, all business stakeholders have teams that interpret the information delivered by analytic applications and in many instances also have an extended team that handles data. This extended team is called the Shadow IT or Shadow Data team in the data and analytics landscape, they have the ability to quickly extract, transform and create datasets and build reports for business use.
There are various benefits that the business gets by having Shadow Data teams:
- Quicker response to the business
- Higher innovation in insights generation for decision support
- Embedded business knowledge in the team
- Closely tied together team, with the right understanding of the Analytics importance to Business
The reasons why Shadow Data teams exist are also due to certain aspects related to the way IT operates, such as:
- Rigidness due to extensive processes followed for application enhancements
- Longer time for application changes and deployment
- Lack of data and domain knowledge
- MIS being considered as a low critical system
There are also various challenges faced by an organization due to the existence of Shadow Data team, such as:
- Data security breaches, since much of data manipulation happens in an ungoverned desktop environment
- Regulatory compliance breaches, as personal data can potentially get stored in non-production environment
- Duplication of effort in application development
- Higher effort spent on data reconciliation between systems
- Data redundancy and silos of information
- Varied technology standards leading to higher maintenance cost (people and environment)
- Potential data insights variations at the department and enterprise level, leading to poor enterprise decisions
- Higher dependency on people , due to lack of process adherence
Is a Shadow Data Team required?
The part of the team that provides interpretation, insights to business needs to be retained as part of the business, they are the power users. The data preparation task done by team needs to be centralized, so that much of the business effort is on insights generation. But still the business should be able to wrangle and blend data for exploration in a governed environment.
Consolidating data work centrally in a governed environment can deliver many benefits such as:
- Avoid redundancy in terms of process , data and technology - save cost and effort
- Ensure consistency of data across the enterprise
- Business applies more time for gaining insight than data preparation
- Ensure safe access to data
Any change brought in should not affect the innovation and agility delivered by a Shadow Data team. Following are the three layers that needs to be brought in to ensure that the benefits delivered by a Shadow Data Team are still available for business:
- Data foundation – Data availability of all forms
- Data processes – Data governance and collaboration
- Data self service – Curated and exploratory data for reporting
Data Foundation: Building a data lake which adheres to the principles of data availability in terms of AS IS data, support all data forms, secured data, detail data, real time data, extended able as data vault model , enable curated data through cubes, marts and exploration through sandboxes.
Data Processes: This relates to ensuring all in the enterprise see the same information in terms of what data element is present, from where it origins, timestamp, who owns it, what is the quality score, how it’s used, who are the users and how is it classified in terms of governance.
We could classify a data element under below three levels:
- Full governance – Enterprise level data
- Light governance – Department level data, that are not of interest at the enterprise level
- No governance – Raw data forms, especially external data
Along with governance, Agile execution process has to be defined both for new projects (scrum method) and application enhancements/support (Kanban method), this will ensure that the business stakeholders are embedded within the IT processes enabling IT to respond quickly to business demands.
Data Self Service: Self Service should enable three ways by which data can be analysed viz., wrangling, reporting and mining (model building).
Wrangling should enable the user to perform manipulation and blending the enterprise data with external data.
Reporting should support ad hoc analysis with a defined semantic layer on the curated data.
Models should enable business get access to all forms of data for exploration in a sandbox environment.
The user should have the capability to search and locate the right report for a required metric, as well support interfacing through NLP, voice with data.
The objective is to enable the business focus on the insights and free up from searching for data, hence the need is to encourage higher business – IT collaboration in a win – win model, rather than getting into perfectionism of moving away from Shadow IT and becoming completely resistant to change.
Building a data management infrastructure with these three layers - data foundation, data processes and data self-service - will ensure that business and are IT working together.