How APIs can help prevent data warehouse hell

Register now

Modern business is all about information, and the potential insights that can come from analyzing large amounts of it. One problem though arises when companies have a lot of data stored in remote databases, isolated data silos that offer only limited access.

These silos can hold back a company from getting the most out of its data, and can become an obstacle in running a modern, data-driven business. The longer the silos persist, the more daunting the task of merging them can seem, but just a few easy tweaks can mean big improvements in how a company applies data strategically.

An eagerness to collect as much data as possible is laudable, but organizations often start collecting it without a plan on how to make the data sharable with their whole team. It’s not just technical architecture choices that prevent finding success with big data projects, company cultures that encourage keeping data in silos makes it worse.

It’s easy for even a relatively small business to end up with multiple data stores, all segregated from each other, each administered by a different executive or team. A company can fly half blind without all of the data being available to the right people as they need it. Making the effort to open up data to other departments is well worth it, and there are strategies that make surmounting any hurdles a much easier proposition for any company.

Technical vs Human Factor

On the technical side, some information in the different silos might be duplicated or updated at different intervals. Each silo may also have its own way of structuring the data. Putting all of that data into one warehouse all at once could be a recipe for chaos without delivering meaningful business results. There are architectural, lightweight tweaks that leave the data where it is, yet open it up for central cataloging.

The challenges of breaking down data silos is just as much a problem of company culture. Department heads and even individual technicians may become territorial about the data in their care, reluctant to share it with others and suspicious about any plans to end a data silo as is typical during your traditional central warehousing efforts. A mandate from the top of the company can start the process of openig a data silo, but data owners may want to be able to do it at their own pace and comfort level. Adapting to those preferences is all but impossible to accomplish in traditional big data projects.

Light up Data Silos Through an API

These problems shouldn’t cause despair. There’s a way to open data silos that addresses both the technical and human problems mentioned above. Data owners shouldn’t be forced to immediately dump all of the information together in a warehouse.

The process should happen at a deliberate pace, set in part by the owners of the data. The data doesn’t ever have to leave the silo to be shared. The right API inserted into the data silo by the data owner provides access to the information to everyone who might need it.

Allowing data owners to maintain control of the data in this way offers the best of both worlds, shared data for analytics and making sure no information is lost. It also helps reassure reluctant data owners that their hard work and data assets will not be compromised.

As departments start sharing data and the company analysing more diverse data sets, everyone will start experiencing incremental gains in insights. The results will be limited to whatever data comes in at first, but as more and more data silos connect, the analytics results become broader and more meaningful.

Those overseeing data silos will see the benefits and typically initiate more data sharing. And, the system they are using will grow over time until all of the data from the various silos is accessible from one efficient and transparent data catalogue for analytical purposes.

The combination of cultural and technological change is easy to execute and hard to beat. Instead of igniting a disruptive and abrupt shift in how a company operates, people will see the small changes as positive improvements. The people controlling data become active participants, even leaders of this new mindset, and the incremental additions of transparent data silos will lead to a company getting the best possible insights from its data -- all of it.

For reprint and licensing requests for this article, click here.