REVIEWER: Robert Anderson, senior specialist at Sempra Energy Trading Corp.

BACKGROUND: Sempra Energy Trading Corp. (SET) is a full- service energy trading company ­ one of the largest in North America. We market and trade physical and financial energy products: crude oil and refined products, natural gas and power. Sempra Energy Trading has more than 1,200 customers worldwide. Our customer base includes most of the major oil, gas and power companies in North America, Europe, Asia and South America. Our high- volume financial transactions can involve more than 100 billion cubic feet (bcf) of natural gas daily. That translates to approximately one half of all the natural gas consumed daily in the U.S. and Canada. SET is a wholly owned subsidiary of Sempra Energy, a Fortune 500 energy services holding company based in San Diego, California.

PLATFORMS: Microsoft Windows 2000 Server

PROBLEM SOLVED: The theory that a butterfly flapping its wings in Brazil can start a chain reaction causing a tornado in Oklahoma applies to energy trading. There are numerous environmental and man-made stimuli ­ weather, water flows, transmission capacities, etc. ­ that can affect the price of energy at any given time in any given location. Monitoring these closely and understanding how and when they impact supply and demand is critical to the fast-paced world of energy trading. A few seconds is a lifetime in this business. The information required to make these split- second decisions is available from a variety of sources ­ subscription services, the Web, internal data repositories, etc. ­ but gathering and analyzing this information in real time requires high-performance precision extraction and integration tools such as WebQL.

PRODUCT FUNCTIONALITY: With WebQL, we created SQL-like queries to extract unstructured and semi-structured data from hundreds of data sources; some queries run once per week, some run every few minutes. The scripts can be scheduled using the flexible integrated scheduler. The extracted data gets dumped to a SQL Server database where it is validated and scrubbed. The data is then pushed to a production database. Once there, it can be accessed by various front-end interfaces used by our traders.

STRENGTHS: With WebQL, there was no huge learning curve. If you know SQL, you can be querying unstructured data almost immediately. WebQL handles all the TCP/IP processing, error handling, data file formatting, etc. It just needs to be told where to go, what to capture and where to put the results. Another big benefit is the massively parallel deployment engine. WebQL can process a query or queries in parallel without any special coding. If you have a big data collection job, the horsepower is available to handle it. Also, you don't have to be concerned about overwhelming a Web site or being detected. WebQL provides sophisticated page request algorithms and identity protection.

WEAKNESSES: The only weakness is the price tag. It is certainly not a cheap solution. We questioned whether it was worth the money to have all of this data within our immediate reach. (As opposed to, "Is the software by itself worth the money?") The answer for us was a resounding yes.

SELECTION CRITERIA: We needed a product capable of handling all of our unstructured data extraction and integration requirements. This meant it not only had to be superb at making precise extractions, but it also had to navigate through tricky Web sites to get to the "deep Web." We have yet to discover any unstructured data WebQL could not extract or any Web site it could not navigate. I cannot say the same thing about other products we evaluated.

DELIVERABLES: As an example, we have a query that grabs hydroelectric data for the 17 primary units in the Pacific Northwest every hour. This information is published on a Web site by the U.S. Army Corps of Engineers. The query outputs the data into a temporary table in the SQL Server. Next, it executes a stored procedure on the database that will verify the data prior to moving it to a production table. When finished, the temporary data file is deleted.

VENDOR SUPPORT: The QL2 support team has been extremely responsive to any inquiry we have made. They even helped us write a couple of queries. One benefit of their pricing model is that operational support and all upgrades are included. We have not been nickel-and-dimed to death.

DOCUMENTATION: Initially the documentation was weak, but QL2 sought my advice and made substantial improvements. It is now comprehensive enough to get you going with basic functions. Some of the advanced functions could be better documented.

Register or login for access to this item and much more

All Information Management content is archived after seven days.

Community members receive:
  • All recent and archived articles
  • Conference offers and updates
  • A full menu of enewsletter options
  • Web seminars, white papers, ebooks

Don't have an account? Register for Free Unlimited Access