The proliferation of data in this period of history, one that will be known as the beginning of the Information Age, has been both a blessing and a curse to information managers. (I use the term "beginning of the Information Age" because I believe the rapid development of information and associated technologies we are now experiencing will continue at exponentially increasing speeds for decades to come.)

The blessing of this expansion is found in the availability of all manner of business-applicable knowledge. Information is the key to making correct decisions at the correct time, and capitalizing on every piece of market intelligence in a timely fashion can make a significant difference in winning or losing an important contract or in achieving a better profit margin. Failure to act quickly, and in an informed manner, will result in critically reduced business effectiveness or even complete business failure.

The curse is having access to too much information and not finding the best information in time to make effective use of its worth. Fortunately, the very phenomenon that has created this overload of information has provided tremendously effective means of putting that information to good use.

Making Use of Valuable Information

Business intelligence comes in many different guises, from raw numerical data to form-based information (such as is gathered from questionnaires), to scholarly articles and essays, to news and wire reports. Companies spend a great deal of time researching markets, consumer habits, trends, forecasts and management techniques--anything they feel might give them a competitive edge. Others have augmented this research by investing large amounts of money and amassing huge amounts of information about their customers, potential customers and (in some cases) their competitors' customers in order to expand market share.

The challenge, as these already monstrous databases grow even bigger, is to be able to quickly and efficiently extract that information which can help decipher some vital piece of the market puzzle or link certain known trends with other as yet unknown influences. As an example, I once came across statistical data linking trends in skirt length to annual sunspot activity. While this sort of obscure correlation may be a bit extreme, it is akin to the kind of comparative research and extrapolation that some organizations are undertaking.

In the hands of an expert, information can be mined and analyzed in such a way as to yield rich ore. When information is looked upon with an untrained eye, however, it's as useless as rubble. The problem with requiring expert attention for the extraction of business information, however, is that expert attention costs, and the process of deriving useful information from very large databases (VLDBs) is usually time consuming.

Wouldn't it be nice if low-level (i.e., cheap) labor could be dedicated to the task of data mining? Wouldn't it be nice if the highly trained and highly paid expert could spend more of his or her time looking at information instead of looking for it?

Enter Neural Nets

Information technology has given us many different types of tools designed to help organizations locate the information they need. Typically, however, these tools rely on structured databases and strict query protocols to work at their greatest efficiency. The emergence of free-form databases, the Internet in particular, has placed a new set of obstacles in the path of data miners--how to locate information when rigid parameters, such as calling up information in a particular cross-referenced data cell, are not appropriate or cannot be employed. Networks, too, have created their own difficulties in efficient data mining as access to multiple databases--sometimes based on different formats--may render various methods of data mining insufficient or, at the very least, rather complicated and time consuming for searching across multiple databases.

Enter neural networks. Artificial intelligence has made tremendous progress in recent years, and no other form of electronic analysis has advanced as much as neural networks.

Designed to mimic the human brain in the techniques used to process information, neural networks have distinct advantages that, in combination, make them particularly adept at data mining: the ability to employ fuzzy logic to match query to result--even misspelled queries; the ability to "learn" as searches progress; the ability to accept colloquial queries; the ability to effectively search unstructured databases; and the ability to rank retrieved information based on the closeness of its query match. Neural networks even possess the ability to search graphical databases as the medium's powerful pattern-matching qualities extend to shapes and pictures.

There was a time when IT professionals spent their days cooped up in glass-enclosed rooms while huge mainframe computers whirled and buzzed. The power of the mainframe guru was immense, and department managers spent weeks plotting their strategy and currying favor with the individual whose job it was to extract precious information from the databanks of the behemoth. If all went well and the manager's gifts were sufficiently expensive and tasteful, a request for the previous quarter's performance figures might be delivered within two working weeks--printed unintelligibly in hard-to-read dot matrix text on two hundred sheets of green-bar paper. If the guru's gifts were deemed particularly nice, those two hundred sheets might be folded neatly.

While putting computers on the desktops of the managers who were previously at the mercy of the IT department certainly undermined the power of the technical illuminati, the subsequent informational explosion, coupled with relatively easy access to that information, has allowed the savvy infopro to regain some of that lost influence.

By chatting casually with department heads and throwing around terms like OLAP and three-dimensional data searches as if they were part of the national lexicon, IT types have once again created an aura of superiority. Knowing their non-techie counterparts would nod knowingly, then sweat out the possibility of being asked to procure some information that required the employment of one of these methods, managers have once again started to buddy-up to the IT folks. Information access is indeed a powerful commodity in the business world.

The data cartels that have formed are once again in for a nasty surprise, however. The considerable potential of neural network-based data mining could strike an even more critical blow to their dominion than did desktop access.

Neural network-based data mining is arising as a key ingredient in the delivery of mission-critical information because it allows searches to be conducted by anyone in an organization at any time, whether that person is a practiced data-search hand or an informational neophyte.

The key to this level of ubiquity is in the extreme flexibility afforded by a neural network-based search. Conducted through an intuitive, user-friendly Windows interface, natural language queries can be conducted that go beyond 3-D searches, and robust pattern matching allows the neural-networked data miner to request information without being precise. Furthermore, as queries progress, a neural network-based search engine can actually become an active partner in the search process.

By keeping track of information that is close, but not sufficiently related to information requested, yet which continues to appear on the fringes of search parameters, a neural network-based search will compile latent data clusters and, based on the information contained in those clusters, actively suggest to its user new areas of search. In investigative applications, this feature is particularly useful in assisting users who may have slipped into a rut.

That a neural network-based search engine can draw out information that may have been misspelled by the user is another feature that gives tremendous advantage to this type of ultra-intelligent search. Requests for names, in particular, can be made more effective through neural net-based search. For instance, an order such as: "Johnson, I need all the information I can get on an individual named Jan Wilem von Jaanderviersbergen," will probably yield little, if any, result if no other information is provided. However, neural networked searches based on variations such as John William von Janderwierbergen (or other aberrations) can quickly put the searcher hot on the trail.

Already Proving Its Worth

Neural network-based data mining systems are already paying off for a number of organizations, including such companies as Reuters, who use a neural net-driven engine for their news research and on-line information delivery service. The dynamic reasoning capabilities are proving a perfect fit for the global news giant's needs. The United Kingdom relies on a neural net-based system for its national crime investigation database, HOLMES II.

Another interesting application for neural networked data mining is found in South Africa, where that country's Truth and Reconciliation Commission is using an intelligent data mining system to aid its investigation into unsolved crimes dating back through the apartheid era, including the resolution in January, 1997, of the murder of anti-apartheid activist Steven Bantu Biko.

Because of the unique nature of that ongoing and exhaustive undertaking and the variety of formats in which over twenty years' worth of evidence is found, the flexibility and potency of their system has become a vital ingredient in sifting through decades of information including: court transcripts, police records and individual statements. With the help of neural nets, important progress has been made in the Commission's goal of building evidential links across South Africa's past, breathing new life into what had previously been considered "cold" leads.

By taking new information obtained through the Commission's investigations, such as in the case of the Biko murder, and combining it with existing records, statements made by individuals interviewed during the reconciliation process can be corroborated and connected to unsolved cases. As investigations continue, this combination of new and old information is used by the Commission to find key information, direct questioning and close previously unsolved cases. The system's ability to cluster data and provide both answers and additional leads increases the Commission's efficacy while decreasing the possibility that hidden clues will be overlooked.

Technologies become increasingly more sophisticated each day, it seems, and that can only mean that this method of search, already at the head of its class, will become even more effective. At a time when the rapidity with which data can be retrieved and put to good use has become tantamount to market leadership and business success, this must be seen as a good thing. But there's no reason to wait for that tomorrow: neural nets are ready now to provide a clear business edge.

Register or login for access to this item and much more

All Information Management content is archived after seven days.

Community members receive:
  • All recent and archived articles
  • Conference offers and updates
  • A full menu of enewsletter options
  • Web seminars, white papers, ebooks

Don't have an account? Register for Free Unlimited Access