When I first started in this business, mainframe computers were all that existed, and they were the size of SUVs. Storage space was outrageously expensive, and programmers used dubious shortcuts (which bit us in the form of the Y2K bug) to save precious space. Computers were grinding, number-crunching behemoths that were almost more trouble than they were worth. By the mid-1980s, we had come a long way, but we still had information systems problems. We had incredible volumes of data about our operations, but we couldn't wring much information out of it. As one of my treasured former clients once said to me, "Jane, we're drowning in data, but dying for information."1 How right he was.
There were a number of business problems that drove the development of data warehousing as the preeminent business information-systems technology of the last 15 years. However, the problem of inadequate access to information in the face of mountains of data was the main driver. It wasn't that most organizations didn't have sufficient numbers of information systems; the problem was that the systems didn't talk to each other. Executive management couldn't view the entire enterprise.
The problem of information drought surfaced because of the way we designed information systems in the 1970s and 1980s. As the need for information grew, various departments within organizations would decide they needed information systems, so consultants would develop the required system in a vacuum for that department. As a result, organizational data became extremely stovepiped. The solution, which was the data warehouse, didn't go over well at first with most consultant- wary CIOs.
In the early days of data warehousing, CIOs were skeptical about the technology because of a well-founded protectionist attitude toward data and systems, as well as the fear of volatility. However, as more and more CIOs and their technical staffs began to understand the technology, they embraced it because while it gave them protection for their operational systems, it also gave them a platform for supporting more adventuresome types of data exploration.
Slowly, most CIOs began to see the tremendous advantages of building warehouses, and the race to build (and build and build) was on. However, the race was hardly without roadblocks. Most IT folks learned a lot along the way, mainly from mistakes and misconceptions. We've used the knowledge we gained and built a great future.
The first thing that most organizations learned when they started building their data warehouses was that big-bang data warehouse projects did not work. It quickly became apparent that data warehousing is a process, not a project. In most cases, data warehousing is a large, enterprise- wide effort. However, it has to be done step by step in manageable increments.
The solution became simple: begin with the end in mind. We found that the best first step is to define the desired future-state architecture and build that architecture piece by piece. The best tactic to achieve that strategy is to use the dependent data mart model and build the corporate data warehouse one slice at a time.
We also found that people issues are paramount when building the warehouse. Few data warehouses fail technically or architecturally; most fail because of people issues. Issues such as lack of (or spotty) executive sponsorship, dysfunctional teams and incorrect user expectations killed many projects in the early days and these days as well, unfortunately. Ill-defined expectations are, and have always been, a real problem. I recently had a conversation with a client who said, "I'm so afraid that if we actually interview these people, their expectations are going to be completely out of control."
This is a good point; expectations can get out of control. Some users expect the data warehouse to make their coffee for them. It's not intended to do that, and it certainly can't do it in the first phase of the project. We've learned it's important to set expectations correctly from the beginning and educate clearly about what you're doing.
Other failure points certainly included poor meta data and the lack of a viable technical architecture. Both of those problems increased costs and made maintenance more difficult, but not impossible. However, one infrastructure problem that did ensure the failure of many projects was the decision of many organizations to go the independent data mart route of systems development. I call this data mart anarchy. If organizations had problems reporting with a particular application, they built a data mart expressly for that application. That's like building another legacy system. It's still stove- piped. Those organizations still had the problem they started with. If they had built their infrastructure around a common data source and built dependent data marts, they would have eventually built their warehouse. As with any technology that grows over time into a mature set of technologies, data warehousing still has a way to go before its practitioners get it right every time.
One of the reasons that organizations don't always get it right, and perhaps the most important thing that I've learned in the last 15 years, is that things change rapidly and constantly. I am continually amazed at the speed of change in our world. In the last decade and a half, it seems as if there is always somebody that has a better and brighter idea; and those ideas come so quickly to the market that it's almost impossible to keep pace with the changes they bring. Those are amazing things to me IT folks are always reinventing, rethinking and improving.
The constant changes have also brought much more size and complexity to information systems than ever before. Now there are billion dollar startup companies that are going from a few hundred to thousands or millions of clients in very short order. To grow and stay competitive, they need to have proper, flexible infrastructures in place. They need to do the impossible and stay ahead of the change curve.
The speed of work is also amazing compared to ten years ago. It is no longer acceptable to say that the project costs $20 million and takes 24 months. Companies now say, "I don't care if it costs $20 million, I want it in six months." It is simply incredible that companies can complete these gargantuan projects in such extremely short time frames. It speaks of exciting things to come in the future.
The future of data warehousing, and information technology in general, is an exhilarating prospect for everyone in the marketplace. We're quickly moving away from the old competitive paradigms into a new era of total Market Integration.2 The market I'm talking about is the business community. Today, successful companies share information with their vendor and client partners and, in some cases, with their competitors. The challenge for companies in this environment is to get their own houses in order to integrate their information and thrive in this era of Market Integration. This is where the newer technologies in data warehousing are providing fantastic solutions.
With the new breed of data warehousing tools and technologies such as enterprise application integration (EAI) and Web warehouses, organizations are able to move to more real-time decision making. Many companies, especially those on the Web, have their customer analysis processes automated. They can spot specific patterns in customer behavior and tie them to what's happening with their inventory stocks.
EAI strategies, along with shrewd use of the Web, will enable savvy companies to set up an artificial intelligence layer to launch highly lucrative, customer-specific marketing campaigns. For example, a company will be able to offer different customers specific discounts if certain parameters are met within the clickstream data analysis while the customer is still online. Or, the company can send customers an e-mail coupon shortly after they've ordered or visited, based on their Web site navigation patterns.
I'm talking about mass customization. Companies can preselect shippers based on clickstream analysis in an almost real-time fashion. They can store customer preference information and present it to the customer as part of a smorgasbord of service offerings that will knock the customer's socks off. This ability to harness the power of the Web to delight the customer is an exponential leap in customer-service ability. Companies that take advantage of it and many already are will be leaders; the ones that don't simply will not be.
Speaking of leaders, I think there will be stiff competition for human resources in the industry, driven by these technological shifts. It takes bright people to do this kind of work. The war for talent these days is absolutely incredible. The dot-coms were making it very difficult for many organizations to find talented people; but now that a lot of dot-coms are hemorrhaging, many "refugees" are seeking jobs in the more traditional, or at least established, consulting and software companies. Even with the flood of survivors from the dot-coms, the best and brightest are hard to find.
It's amazing what information technology will drive. It can change the landscape of an entire industry. From now on, corporate culture as well as opportunities for continual learning and working on new and exciting projects is going to become the measuring stick by which industry leaders are measured against rest of the pack. Those that insist on sticking to the old paradigm of employer as benevolent dictator are going to fall behind in the marketplace because they can't attract the best people.
Today, based on what you offer your employees, you're known as a leader or as a laggard. I can't imagine business going back to the old paradigm of patriarchal employers and "lifer" employees. It's a seller's market now, and that's going to make for an exciting if uncertain future for all of us.
That's what I love most about this business the uncertainty and change. I can't predict very far into the future, and I am never completely sure of anything except change. The last 15 years have been an incredible period of technology growth and paradigm shifts for the business marketplace. What's more, the pace of technology growth seems to be increasing, if that's possible. I can't say for sure what the future will hold 15 years from now when I write my next retrospective article (if I'm not retired and on the beach by then), but I can say that I hope we will continue to learn from our experiences. It certainly should be a fantastic ride.
Register or login for access to this item and much more
All Information Management content is archived after seven days.
Community members receive:
- All recent and archived articles
- Conference offers and updates
- A full menu of enewsletter options
- Web seminars, white papers, ebooks
Already have an account? Log In
Don't have an account? Register for Free Unlimited Access