The telecommunications services industry has significantly influenced the growth and development of data warehousing over the past two decades. As telcos coped with huge data volumes in their operations, their regulatory and business environment forced them toward new and unique informational solutions.
The Numbers Game
According to META Group, telcos have led the data warehousing industry in average size of data warehouses and data marts; percent of IT budget spent on warehousing; and number of projects undertaken in both data warehousing and data marts.1
For 1999, the average telco warehouse was estimated at 3.1 terabytes, almost twice the 1.6 terabyte industry average. More than 50 percent of telco projects were projected to exceed one terabyte, and total telco spending on data warehousing projects was expected to exceed $3.5 billion for the year.
These numbers grow from the firms' basic operations requirements. A hypothetical local exchange carrier (LEC) may serve 20 million subscribers, each making five calls per day. That firm will record 100 million transactions per day a peak load of nearly 3,000 calls per second. Further, as a regulated common carrier, the LEC must store those records for 24 months and surrender them on demand to any legitimate law enforcement agency. Assuming the record (conservatively) contains 200 bytes, then more than 14 terabytes of source data will enter storage before database indexing or replication is accomplished for backup.
Modern telcos have evolved through a period of massive mergers and consolidations, and three to five times that volume can be expected from voice operations alone. In addition, the firm may purchase details from long distance and international partners so it can bill, collect and market those services as well. With telcos merging their mobile, fixed and ISP services onto a single convergent bill, and planning to add IP teleservices to its mix, the single telco can expect as much a 5 billion usage records per day from all services.
Yankee Group2 identifies six trends confronting telcos today:
- Unprecedented competition;
- Rapid consolidation;
- Infrastructure capacity explosion;
- Internet and inter-networking impact;
- Next-generation service opportunities; and
- Customer demand.
The firms must manage through these trends while simultaneously maintaining standards of quality, reliability and operational excellence that allow them to compete in a global commodity market.
The force that has mandated competition, more than any other, is the regulatory fabric that pervades the industry. In a sense, the public interest (as represented by governmental bodies) is another "customer dimension" in telcos' strategic decision making.
Since the AT&T breakup, telcos have faced a worldwide pattern of uneven regulation. U.S. antitrust policy has hobbled the "dominant" firms, while preferentially removing their competitors' barriers to market entry. States have acted at times to protect their local carriers, only later to declare themselves telco "free-trade zones." Most of Europe has transitioned to wide-open competition, where aggressive new entrants battle the privatized remnants of public telephone and telegraph companies (PTTs) and global consortia. All nations currently play out their own versions of the divestiture experience, but this time they do so in "Internet time."
These shifting sands of regulation now present telcos with a complex landscape dominated by three strategic challenges:
Scale: Gain capital strength to acquire and hold customers, and build new networks for broadband access to "innovative" services.
Competitive Niches: While achieving scale, do not bypass micro-markets capable of sustaining nimble start-up competitors un-hampered by regulation. As a group, they are the legacy telcos' number one source of competition.
IP Technology: Deploy packetized high-capacity networks to obtain the technology promise of 10:1 cost advantages over today's voice networks. Invest in fiber, network devices and operations support systems (OSSs) at unheard-of levels.
The Corporate Information Factory (CIF) has emerged as the primary technology for managing competition and customer relationships.3 This is especially true in telcos, where more than 75 percent of today's data warehouses and data marts support marketing missions. As telcos maneuver furiously to meet their strategic challenges, they will count on their data warehouses to map the way through the minefields of scarce capital, scarce talent and unrelenting competition.
Evolution to Data Warehousing
Because of the need to collect regulatory details, there is a tendency within telcos to manage databases as cost centers first and business opportunities second.
Following divestiture, firms struggled with their core operating functions, such as billing, customer service and end-to-end provisioning. The evolution of billing software, intermediaries and service processes dominated telcos' IT efforts into the early nineties. Bills require accurate data collection of calls placed, services performed and adjustments entered, followed by massive amounts of centralized processing. Random-access technology of the time made long-term storage of terabytes of data impractical on all but sequential media. Thus evolved the generally accepted practice for handling the most detailed, atomic information: one to three months in database format, then archiving history to tape.
However, bill processes were not sufficiently fast to accomplish tasks in single operational threads. Therefore, customers were assigned to regional billing centers, enabling parallel billing streams against regional partitions of the databases. As a result, the customers were separated for post-bill service operations as well. Until processing power evolved that could unify these regional centers under one roof, only the very largest of businesses attempted to unify card, dial and private line billing.
Little operational time was available to support informational demands. The response was to spawn end-user and workgroup technologies "outside" the IT budget. Billing data, the only source of details, was not generally available for ad hoc analysis. Marketing departments depended on regular batch feeds of such data into minicomputer or PC databases for subsequent analysis with desktop spreadsheets and graphics tools.
Two landmark competitive actions spotlighted IT's inadequacy to meet the challenges of the period, as well as the fragility of customer relationships in ultra- competitive long distance markets:
- 1992: MCI launched the "Friends and Family" campaign. This forced carriers to consider the whole circle of affinities with which a consumer might identify and increased demand for a single view of the customer relationship.
- 1994: The FCC mandated 800-number portability the number now belonged to the customer, not the carrier. The link between service provider and physical number was broken, so that other forms of customer retention had to be found to stem potential massive migrations.
Rapidly growing wireless companies faced the same problem: how to keep customers from migrating to competitive offers just as they had reached profitability. Because these carriers had to "seed" their markets with personal equipment, cost of acquisition was high. At the same time, competition from new personal communication system (PCS) technology was forcing revenues downward. From this situation came the classic term that dominates data warehousing discussions: churn management.
The net of these marketing challenges was to galvanize the movement toward issuing a single, unified bill and toward developing a customer-centric information base. The intent was to capture and integrate the whole customer experience across all offered services, rather than the service-by-service views given by the "smokestack" databases.
These "integrated customer view" projects were characterized by:
- In-house code development with some use of early ETL tools (COBOL generators),
- Enterprise-wide scale of project,
- IT driven, and
- Organized as normalized databases on DB2, Teradata or Oracle.
These were the first telco enterprise data warehouses (EDWs). Their scale demanded that all aspects of best practices for IT development requirements development, data modeling, operations and data quality be planned ahead of time. In this environment, telcos played the role of technology incubator for many of today's tool vendors.
In the environment at large, however, the nascent Internet revolution was proving that decision support system (DSS) projects could not be allowed a one to two year time frame before yielding results. The problems were of shorter duration than the structure to solve them. IT outsourcing and business consolidations meant that project teams seldom stayed together for whole projects. Yet, the technology infrastructure needed to build the Corporate Information Factory was exactly what these projects were attempting.
It's All in the Details
During the early 1990s, telcos realized that convergent billing also meant convergent handling of the detail records feeding those bills. A single database, across all services, was required both for handling regulated detail stored at lower cost and meeting marketing's demands for a single source of service consumption history.
The industry turned its attention to its long-forgotten call detail archives. What if archival processes already mandated by regulatory compliance could evolve into common, enterprise-wide data assets using open data interfaces? The resultant vision was the Communications Event Detail Repository (CEDR), an architected data store that straddles the areas of operational and information processing,
Figure 1: Event details reach into every aspect of a telco's business processes.
As shown in Figure 1, event details reach into every aspect of a telco's business processes including billing, finance, marketing, network, regulatory and operational support systems (OSSs). For example, they are the data support for high-payoff activities such as:
- Decreasing cost and time to access details for subpoena compliance from several months to 24 hours;
- Resolving inter-carrier settlement disputes, both international and national;
- Providing detailed historical demands to network planners for the development of new facilities plans and capital budgets.
For data warehousing architects, the intrinsic qualities of the CEDR data add new opportunities for deriving useful customer insights. CEDR data is original, the standard by which successive transformations should be compared. This largest single collection of atomic data need not be transformed, only accessed, significantly reducing ETL processing in data warehouse (DW) load operations. CEDR data is inherently cross-services in nature, containing all activity from all services. Quality and completeness of source data is not an issue when a CEDR is the source. CEDR data provides the finest grain of service consumption detail, from wireline and wireless switches, network mediation software and Signaling System 7 (SS7) transactions. These sources take the warehouse beyond a simple record of service consumption. They provide the additional dimensions of usage patterns and quality of experience.
For example, today's wireless carriers may include post-bill line items as the most granular detail in a warehouse, and that detail may show a disconnect incident. But using a CEDR, the carrier may also retain the actual disconnect event, the reason for the disconnect, location of the customer at the time of disconnect, the towers serving the call at time of disconnect, the number of failed tries to reconnect or the state of the customer's handset during the disconnected call.
From this knowledge, we can measure how likely people are to churn, or jump to a competitor, when exposed to these disruptions. We can also find others with the same experiences and see if they have churned more than those lured away with a competitive discount offer. We can build models to identify whole populations of customers whose similar patterns of service history might have a propensity to churn. Finally, we can evaluate pre-emptive actions to retain those customers we choose to retain, be it migration to other plans, handset replacement, build-out of additional capacity services or promotional discounts. None of these alternatives is truly quantifiable without consideration of those details showing the full depth of the customer experience.
As telcos begin to realize the information potential of these sources, new queries arise reflecting information demands for which the DSS structures were not designed. However, if the atomic details have been collected into a CEDR, detailed records are readily at hand to rapidly refresh the analytical data structures through SQL selection. The net effect is to "future-proof" a CIF, protecting it against obsolescence in the face of unanticipated demands.
Finally, the CEDR construct supports the data warehouse "explorer" through simplicity of organization and ease of access. Atomic data can be moved in large "chunks" into an exploration warehouse for data mining, knowledge discovery or data mart prototyping. Explorers' access to data remains unfettered and complete, but does not burden warehouse operations with ad hoc queries of unpredictable duration.
To construct a CEDR and obtain its benefits, telcos must employ the fundamental technologies:
- High-end multiprocessor servers with massive I/O capabilities;
- Large-scale deployment of 100MB or gigabyte LAN connectivity;
- High- speed tape drives to reduce both access and transfer times;
- Database software built specifically to manage SQL access to all storage classes, not just disk, and scalable to hundreds of terabytes or even petabytes.
All these items have become available in the past two to three years. However, to integrate these technologies into a working repository requires systems engineering, analysis and integration, as well as service-level agreements and operations plans to meet carrier-grade requirements. This vital expertise is the difference between a working CEDR and a room full of hardware.
The Future: IP Services
The last decade's price wars have resulted in a voice call market with razor- thin margins, minimal services differentiation and historic lows in customer brand loyalty. In short a commodity. Telcos view innovative services such as videoconferencing, video on demand, teleconferencing and selectable bandwidths on demand as a way to escape the commodity marketplace to increase ROI. To enable this vision, the industry is racing toward the vision of dense fiber backbone networks, accessed via single broadband pipes from the home or office, offering a mix of IP (Internet protocol) services.
In the future, the amount of detail to be managed will explode by several orders of magnitude. Unlike homogeneous switched voice networks, IP networks are assembled from multiple devices such as routers, gateways, security servers, etc. Hundreds of partial data elements must be correlated to assemble integrated records of service consumption, use and quality. While vendor-specific software exists and is installed today, no de facto standard exists for IP network usage details.
The IPDR Organization (www.ipdr.org), a broad-based industry group of telco operators and software suppliers, is working to formulate specifications for content and delivery of a standard IP billing detail records. In their first technology demonstration, the group showed how a video-on-demand service can capture information regarding bandwidth, quality of service, time viewed and content and then pass this in standard format to downstream applications such as billers or data warehouses.
The presence of a workable standard will create a common denominator between network capabilities and service offerings. The standard record will be richer in available content than its analog sibling in today's switched environments. A general acceleration of marketplace activity will follow an upward spiral as pent-up demand leads to differentiable billing, which leads to revenue, which leads to more carrier build-out of facilities and leads to even more demands for services.
The Corporate Information Factory must continue to evolve in directions that can cope with this flood of detailed information. The solution for telcos will be the CEDR, an architected repository built specially to handle the challenges posed by massive scope and scale.
- 1999 Data Warehousing Trends and Opportunities. The META Group.
- Rich, Rob. "The State of OSS: A Market in Transition." OSS2000. Orlando. January 31, 2000.
- Imhoff, Claudia. "The Corporate Information Factory." DM Review. December 1999.
Register or login for access to this item and much more
All Information Management content is archived after seven days.
Community members receive:
- All recent and archived articles
- Conference offers and updates
- A full menu of enewsletter options
- Web seminars, white papers, ebooks
Already have an account? Log In
Don't have an account? Register for Free Unlimited Access