I lead a double life. My area of specialty is business intelligence (BI). In the early mornings, lunch hours and evenings I live in the BI world. However, for quite some time now, during normal work hours Im an application architect on a large CRM implementation effort. When I initially became involved in this CRM implementation, I was a little worried about leaving BI and losing my edge, but as CRM and BI are often tightly intertwined I thought this would be an excellent opportunity to experience the business intelligence life cycle from a different perspective.
This implementation is very large and integrates multiple applications and data sources. As the project progressed, it rapidly swept me into the realm of message-driven EAI and service-based architectures. I soon realized it was going to take me far, far away from BI. I was however, very surprised that it brought me back full circle to some of what I thought were BI-specific problems and solutions.
As we move towards pervasive computing, the complexity and diversity of information technology increases almost exponentially. This is primarily due to the immaturity of some of the solutions and technologies being developed. This will, over time, lessen considerably as winning solutions surface and standards take hold but, in the near term, the growth were experiencing with technology translates into increased complexity and increased diversity.
In order to effectively deal with this complexity, one tends to specialize in a specific discipline in order to develop their skills. This specialization makes the aisles between cubicles in a large IT organization seem like geographic borders separating entire societies. A J2EE Web services architect doesnt seem to be able to speak the same language as a DBA. While a DBA can, on the other hand, probably comprehend what the data warehouse developer is saying that conversation would more than likely be only one way. If the data warehouse developer strikes out in communicating with the DBA and turns instead to a PeopleSoft developer, their attempt at conversation may slow dramatically when one starts talking about hierarchical trees while the other is discussing tables.
While this specialization increases the inefficiencies of communication, it also tends to limit the potential for problem solving. Everyone has a "psychological inertia" or a tendency to rely on their own experiences and not think outside their cubicle (i.e., box). While your own experiences are a source of strength, limiting your analysis to just what you know can be a weakness.
My tenure on the CRM implementation effort showed me that my specialtys methods and problems are not exclusively its own. It also showed me that when considering solutions many people (including myself) tend to scope the solution set into their comfort zone and area of understanding.
On this CRM project, one of the first design problems we needed to tackle was integrating legacy data into the CRM system. Coming from the BI world, the solution seemed rather obvious to me. All we needed was an extract, transform and load (ETL) process to move the data from the source into target. However, we did have a requirement of near real-time integration. The only viable option I came up with was an ETL-driven process with a very small batch window (i.e., 15 minutes versus the typical 24 hours). I was initially surprised when another application architect with a specialty in Java services came up with a few additional options.
To my short list of one alternative, we then added:
- Presentation layer integration the data remains in two distinct data sources, but the CRM applications presentation layer ties them together.
- Stick to daily batch ETL loads but have the application in real-time query both its database along with the originating source (for just the days deltas).
- A messaged-based architecture using a MQ Series-based publish/subscribe approach.
In the end, we opted for a MQ Series messaging architecture. Instead of dealing with daily batch loads at a database table level, now as events occur in the source systems, the source then "publishes" these events. The CRM system acts as a subscriber "listening" for these changes and loads the changes into its database. This approach allowed the requirement of near real-time integration to be achieved. Although EAI messaging is not necessarily easy to implement, it does not appear any more difficult or time-consuming than implementing an equivalent ETL process.
This experience left me with an appreciation of the benefits of EAI with the potential to even impact the BI world. It gave me new alternatives to consider and a new way of looking at old problems.
In the engineering world, the issue of psychological inertia is addressed in what is called the Theory of Inventive Problem Solving or "TRIZ" for short. TRIZ was first described by Genrich Altshuller, an inventor turned patent expert in the Soviet Union in the 1940s. Genrich surveyed more than 200,000 patents and categorized them not by their industry or specialty but by the depth of their problem-solving approach. He categorized the 200,000 patents (solutions) into five levels and summarized the percent of patents that fell in each category.
- Level One (32 percent) Routine design problems solved by methods well known within the specialty. No invention needed.
- Level Two (45 percent) Minor improvements to an existing system by methods known within the industry. Usually with some compromise.
- Level Three (18 percent) Fundamental improvements to an existing system by methods known outside the industry. Contradictions resolved.
- Level Four (4 percent) A new generation that uses a new principal to perform the primary functions of the system. Solution found more in science than technology.
- Level Five (1 percent) A rare scientific discovery or pioneering invention of essentially a new system.
This theory appears highly applicable to information technology (IT)as well as engineering. Most problem statements are thinly veneered solutions. In the CRM example, the problem statement was described as needing to "integrate legacy data into the CRM system." As written, this problem statement already leads toward a preconceived solution. The real problem was that we needed a way to fulfill the requirement for displaying both customer data and product data that come from two separate sources in context on the same screen. The words "integrating" and "into" within the problem statement led me toward a data-level integration solution. Data integration was also my specialty, which further cemented this alternative in my mind. It took an application architect with a different specialty and frame of reference to come up with the additional alternatives.
As Genrich Altshullers survey illustrates more than 75 percent of solutions come from your own area of specialty; your own experience. While in a mature industry this may be a tremendous efficiency, in the rapidly changing and increasingly pervasive world of IT this is a double-edge sword. Regardless of what your area of specialty within IT is, it is undoubtedly evolving. As each area of IT evolves, it does not do so independently of other areas. Some areas collide, overlap or even merge. Even distinct areas can have overlapping problems, architectures and potential solutions. The issue of real-time integration may be recent for BI, but it has been around in many other IT disciplines for a few years and has already been addressed in a variety of ways.
Now when I tackle a new problem, I consciously review the problem statements for bias. I also frequently solicit input from other areas within IT. Instead of searching for that rare Level 5 innovation, dont forget to look just beyond your cubicle walls. Those .NET designers, WebSphere engineers, COBOL programmers or EAI architects may have just the solution you need.
1. Mazur, Glenn. Theory of Inventive Problem Solving (TRIZ). http://www.mazur.net/triz/. 1996.
Register or login for access to this item and much more
All Information Management content is archived after seven days.
Community members receive:
- All recent and archived articles
- Conference offers and updates
- A full menu of enewsletter options
- Web seminars, white papers, ebooks
Already have an account? Log In
Don't have an account? Register for Free Unlimited Access