Industry Data Challenges
In recent years, global energy and financial markets have grown significantly. As these markets continue to expand they require an increasing amount of transaction data to be captured, managed and delivered for analysis and actionable decision-making. Factors such as the California and Texas nodal initiatives, new financial instruments and the increasing popularity of derivatives have contributed to the dramatic increase in data creation. This increase of data is expected to continue exponentially in the future, placing higher demands on processes, technologies and resources.
Data quality is highly correlated to the accuracy of market research and thus profitability of clients. Missing and incorrect information can be extremely costly to decision-makers. The need and importance for accuracy, completeness and consistency of timely data cannot be overstated. Unfortunately, data from leading vendors is often inconsistent, incorrect or missing. A recent study researched four providers of energy futures data and concluded that each data vendor had inaccurate and incomplete data.1 This study brings to light that an organization must add intellectual capital to every sourced piece of data it relies on.
Enterprises are shifting toward real-time consumption and distribution of information and away from traditional batch approaches. Contributing to the shift are the increasing accessibility and reliability of real-time feeds and the expanding need for a competitive advantage based on the timing of trade execution. Information that arrives late is at best less valuable and often completely worthless.
The increasing importance of internal audits and regulatory compliance, such as the Sarbanes-Oxley Act and Financial Accounting Standards Board (FASB) 133/157 accounting standards, increase the workload of client IT departments.
The combination of these challenges and the need for them to be addressed concurrently multiply the degree of difficulty for energy and financial capital market companies. Specifically, organizations need the ability to acquire, standardize, correct, store, manage, audit and deliver more data than before - faster and with higher data quality.
This article introduces a framework for energy and financial capital markets (EFCMs) master data management (MDM). The framework is specifically developed for these industries and addresses the aforementioned challenges.
Energy and Financial Capital Markets MDM Framework
MDM provides the authoritative, reliable foundation for data used across the organization with the goal to provide a single version of the truth. The EFCMs introduce subtle differentiation to traditional operational MDM frameworks. Specifically:
-
Distributed time-series architecture is required to maximize storage and retrieval performance and efficiency,
-
Data quality focuses on quality assurance rather than cleansing and merging of information and
-
Reliability, throughput and real-time ability are pushed to extremes when millions of dollars are at stake every second.
A high-level architectural representation of a proposed EFCM-specific MDM framework with associated source and target systems is depicted on Figure 1 (located at the bottom of the article).
The EFCM MDM framework in Figure 1 includes two main components: the MDM process hub and the client data mart system. The MDM process hub handles centralization, quality and consistency of information. The ideal architecture for the MDM hub is an integrated MDM centralized repository that houses the “gold copy” of all master data and metadata information. This integrated approach provides the most complete, accurate and consistent view of master data.
The federated and hybrid architectures are not well suited for energy and financial systems. This is due to the volume of data, real-time needs and importance of consistency and quality. The MDM hub handles the lifecycle of data, master data and metadata. It includes faculties for process management (automation, notification, measurement, monitoring, logging and reporting), hierarchy management, model management, rule management, security and data governance. EFCM MDM systems need to be highly scalable and able to dynamically process millions of data points every hour.
