Perhaps that prejudiced me, but at least Tainter's book got me thinking about the complexity of enterprise information systems, particularly in enterprises that automated in the 1960s and 1970s and are still around today.
Horror Stories
One of the first aspects of complexity in mature enterprises that comes to mind is what philosopher R. G. Collingwood called "the law of primitive survivals." This states that earlier, more primitive forms of things coexist with later, more sophisticated forms of things. We see this in biology, where primitive forms, like sharks, exist with modern forms, like bony fishes, and algae coexist with flowering plants. The primitive coexists with the modern to a surprising extent. This is not something that enterprises often talk about, but we all recognize it is a fact. For instance, I know of one large company that still uses a programming language that I thought had disappeared 30 years ago. They employ programmers who are now in their seventies, at great cost, to maintain certain key applications.
Another anecdote was recently told to me by a senior architect of a major bank. He noted that most of the people he knew in IT were his age. It was as if a cadre of staff had been recruited in the 1970s and risen through the organization together. Of course, the lower-level staff is much younger, but there is little age variation in the decision-making levels. My take was that the way this cadre learned to do things in the 1970s strongly influences the decisions they make today. Yet the world has changed significantly over this period (e.g., from process centricity to data centricity, and from in-house development to package acquisition). Maybe this partially explains why the primitive forms still exist, but I think there are additional factors.
Another piece of anecdotal information is the tendency of mature companies to rehire long-term employees that they had let go. This seems to be connected not merely with architecture, but with a lack of knowledge management. It appears that the staff who have been let go knows how things work, and this knowledge has never been documented in a usable form and/or transferred to other staff. It is like something out of an Isaac Asimov science fiction story – almost as if we have the emergence of a priestly caste that makes the machines run. The difference is that this caste does not train its successors.
What is Complexity?
The complexity of information, technology and application architectures in mature enterprises seems to take the form of the survival of previous generations of environments integrated with overlays of more modern generations. In Tainter's book, he describes the collapse of the Lowland Maya of Central America. The Maya had a curious habit of not rebuilding their pyramids, but making them bigger and better by building new layers on top of existing pyramids. This is not exactly analogous to what IT has done over various technical generations; IT has not simply added layers of new stuff on top of existing stuff, but has fused the whole lot together into one giant incredibly complex architecture. Loose coupling is a fine architectural principle, but invariably honored in the breach where it really matters.
Increase in scale has also played a role in the growth of complexity. It has always seemed odd to me that core applications that only took a couple of years to build back in the 1970s cannot be easily replaced. Why can’t a replacement be created in a similar time frame today? The fact that the enterprise has grown in size since the original application was developed seems to be one reason. Another reason is the undocumented integration of the original application with other applications and operational processes. IT cannot remember what it has built, but what it has built is large and connected in many different ways.
Evolution and Extinction
Large, mature enterprises have seen mainframes, PCs, distributed processing, the Internet, the cloud and are now in the early stages of another revolution: big data. How much more complexity can be added to their environments? I have tried looking for answers in evolutionary science, which I studied at university; the field teaches that specialization typically leads to the inability to adapt to changing circumstances. However, this is not necessarily the same as increased complexity. In fact, simplification is also an evolutionary theme; for instance, fishes have much more complex skulls than we do.
More menacingly, extinction is a very natural event. The vast majority of plants and animals that have ever lived are now extinct. This is very difficult for us to accept, but it is a fact. Is this where our large, mature enterprises are headed? Are their IT infrastructures so sclerotized by the accretion of unplanned changes that, one day, they will be unable to adapt to changed conditions or resist the advances of a younger competitor? I think this is a real possibility. We all know how vastly more difficult it is to change something than to build it from scratch.
When enterprises had purely manual operations, they couldn’t scale, but perhaps they were more flexible because the human beings who did the work could easily reorganize themselves. Perhaps automation has increased efficiency at the expense of flexibility. Perhaps organic growth in automation leads to the accumulation of mistakes, which cannot be shed, just as the aging process in an individual leads to his or her eventual demise.
Be the first to comment on this post using the section below.