Making the Case for Complex Event Processing Software

InfoManagement Direct, November 2006

John Morrell

Complex event processing (CEP) involves the continuous processing and analysis of high-volume, high-speed data streams from inside and outside an organization to detect business-critical issues as they happen. In comparison to traditional intelligence processes, which provide delayed analysis, CEP software processes data streams and detects business events in real-time. Some examples of CEP applications are:

• Real-time financial market data analysis and enrichment,
• Financial trade auditing and compliance,
• IT security event correlation,
• Asset management and tracking using RFID, and
• Manufacturing process, power grid or energy pipeline monitoring.

The vast majority of event processing applications today are custom-coded. Much of this custom coding effort, however, can be eliminated by using CEP software; the level of time and cost savings corresponds with the complexity of the event processing application. The remainder of this article will articulate a framework by which you can understand where and to what degree CEP software can offer cost savings over custom development.

What Does CEP Offer?

CEP software offers two major components: a high-level language for programmers to easily describe how to process the streams, and an infrastructure engine for processing and analyzing high-volume data streams. Although CEP software performs different functions, the component structure is mildly analogous to database software, where there is a language (SQL) and an engine (the database server).

Because some of the operations a programmer wants to perform on data streams are similar to a relational model, a select number of CEP vendors offer a language that is based on SQL. This provides a familiar programming environment, speeding the creation of event processing applications.

The engine provides the core components to execute the analysis at run-time. The engine takes on many complex tasks typical in data management infrastructure software as well as those unique to event processing:

• Stream management: Data streams are analogous to a database table of infinite size, with each new event appending a row onto the table. As streams often travel over a network, there can be issues such as dropped, delayed or out-of-order messages. A good CEP engine will automatically handle all these issues without requiring programmer intervention, ensure reliable message delivery and generate a valid, dependable stream for processing.

• Memory management: Data streams can become very large and have many queries running against them. A good CEP engine needs to optimize how memory is managed to ensure high throughput. Special care must be taken to avoid copying and ensure that every piece of data is only stored once.

• Parallel execution and synchronization: To maintain performance, a CEP engine will perform operations in parallel and synchronize data between the threads. Excess synchronization can hurt performance. Thus, a CEP engine not only has to automatically perform state synchronization for the programmer, but it must also balance the synchronization rates for efficient execution.

• Windows: Processing on data streams is performed in "windows," typically, units of time. An efficient CEP engine must be able to expire messages properly, both on new events and timer events.

• Indexing: Fast-moving data streams require indexes to be continually updated at a similar high rate for efficient processing. A good CEP engine will automatically manage these indexes so the programmer does not have to deal with such issues.

These and many more functions are abstracted from the programmer, making the development of CEP applications easier.

Types of Event Processing Applications

If a developer were to create a custom-coded event processing application, he or she would need to code some if not all of the CEP engine features mentioned above, depending on the complexity of the event processing application.

To simplify the framework for determining the applicability of CEP software, let's examine event processing applications in four tiers:

• Tier One: simple event processing applications,
• Tier Two: event processing applications involving multiple streams and/or stored data,
• Tier Three: complex analysis and pattern matching across event streams, and
• Tier Four: multiple, enterprise-class event processing applications.

As you move from Tier One to Tiers Two and Three, the analytical functions increase in complexity, raising the amount of code required to process the streams. Moving to Tier Four introduces infrastructure requirements, further increasing the programming task of the development team.

Tier One Applications

Basic event processing applications will work with a single message stream and typically perform processing one message at a time. The processing usually involves simple filtering, calculation and routing, looking for a specific event to notify a person or application.

An additional level of complexity to a Tier One application is aggregations over a window. This is where the application will continuously calculate a value based on fields in a single event, such as a moving average, over a specified period of time, called a window. A simple example query for such an application, is:

As you can see, coding such queries in a high-level CEP language is very straightforward. But a CEP engine also has its costs, mostly centered on the learning curve for the language, the learning curve for the CEP engine, and licensing the software.

The custom-developed code for Tier One event processing applications would also be relatively straightforward and can be created quickly and efficiently by most developers. Thus, the initial costs of using a CEP engine may not outweigh the efficient custom-coding effort.

An additional cost factor when comparing custom-coded event processing applications to using a CEP engine, however, is maintenance; custom code is often difficult to maintain. Programmers change jobs or firms. Code is often not well documented. The high-level language approach of a CEP engine will make it easier to maintain CEP applications. Maintenance costs should also be taken into consideration.

Page 1 of 3.