OneSource Integrates Extensive Business Information Repository with Data Junction

Pervasive Software - Data Junction Integration Studio

Information Management Magazine, July 2003

H. Peter Frese

REVIEWER: H. Peter Frese, senior systems engineer for OneSource Information Systems.

BACKGROUND: OneSource, a recognized leader in business information solutions, delivers company, executive and industry intelligence that supports vital business processes including serving customers, finding new opportunities and managing suppliers and partners. OneSource combines information, including executive and company profiles, financials, industry intelligence, analyst reports and trade press, from more than 2,500 information sources supplied by 30 world-class content providers. This enhanced information is provided to customers through a subscription-based Web portal and can be embedded directly into enterprise applications. OneSource uses its Global Business Taxonomy to organize and link information on more than 1.7 million companies worldwide, resulting in the most extensive company-linked repository of business information in the world.

PLATFORMS: Data Junction is deployed on Windows 2000 and Windows NT production server.

PROBLEM SOLVED: OneSource acquires information from data vendors all over the world. These large volumes must be qualified and integrated into an information repository to support our applications and services. For our problem example, one vendor alone provides more than 200,000 financial reports on more than 30,000 companies weekly. These balance, cash flow, income and ratio statements are submitted to OneSource in XML format. This data needs to be normalized and our OneSource Global Business Taxonomy applied before transformation into our modified XML format for product publication. Because our products are constantly refreshed with new information from many disparate sources, we need a data integration solution that enables us to accept data from virtually any source or format, manipulate and transform the data, and then routinely and swiftly integrate it into our delivery systems. Data Junction provides the extensive transformation process and high- volume processing capabilities that replace slower, custom-coded load programs.

PRODUCT FUNCTIONALITY: Our data integration process relies on the full functionality offered in the Map and Process Designers. Several features are key to simplifying and accelerating our processing of financial reports. We have realized significant performance gains using in-core memory lookup tables drawn from Oracle and SQL Server. We have also achieved faster processing with djMessage, a powerful feature that allows us to minimize disk activity by accessing large files with a single read into memory for internal line-by-line processing. We use the Data Junction FileList feature to scan directories for all report files, and we might consider the option to use djQueue for continuous monitoring for real-time processing. Data Junction provides maintenance ease as we change the transformation target from XML to SQL Server. With Data Junction's Integration Engine installed on a quad processor, we have been able to take advantage of parallel processing for a 3-to-1 performance gain, significantly reducing the time it takes to process a full set of files. Finally, the Process Designer incorporates all the conversion mapping, SQL, executable and condition steps into a single production process for easy operations management of complex processing.

STRENGTHS: Data Junction solutions support many different sources and applications, and the high-performance Integration Engines can scale to handle large data volumes. The Process Designer provides the ability to build complex transformation steps into a single integration process and to test transformation steps or make changes to transformations within the design environment without having to test the full cycle of the process. We consider Data Junction's Support Center to be one of the company's essential strengths.

WEAKNESSES: Data Junction's documentation could be more comprehensive, particularly in presenting the full capabilities of the products for higher-level processing.

SELECTION CRITERIA: It was imperative that our integration solution be able to connect to many different sources and provide high performance. OneSource evaluated solutions from a number of integration vendors and chose Data Junction because their solutions provided broad connectivity with the power and speed required to support our processes.

DELIVERABLES: Using Data Junction technology, OneSource prepares 200,000 financial reports for production use.

VENDOR SUPPORT: Data Junction's technical support is excellent. Their technicians have addressed some wide-ranging issues for us and have done so willingly and with great expertise. Our calls are always received with a friendly voice, and we always receive good technical information.

DOCUMENTATION: While the documentation is good at presenting the basics, we quickly progressed beyond that. We would not have gotten as far as we have without the assistance of Data Junction's support technicians.

H. Peter Frese, senior systems engineer for OneSource Information Systems.

For more information on related topics, visit the following channels:

• Data Integration