SAND Extensible Warehouse Suite
REVIEWER: Mark Spear, Lennon program director at Atos Origin.
BACKGROUND: Atos Origin is an international information technology services company with annual revenues of more than EUR 5 billion and 45,000 employees in 50 countries. Atos Origin manages Lennon, the hosted rail-ticketing analytics service for the Rail Settlement Plan (RSP), which is part of the United Kingdom's Association of Train Operating Companies (ATOC). The Lennon project requires that 401 days of data on transactional sales from all of the UK's train operating companies be kept accessible to users, at varying levels of aggregation. The data must be archived after 401 days, but it must still be available for quick restoration at any time for analytic purposes. Atos Origin was contracted to maintain this service for 15 years, and we estimated that more than 24 terabytes of storage would be required to hold the data for this period if a traditional relational warehouse design were adopted.
PLATFORM: Sun Solaris 8.
PROBLEM SOLVED: SAND Extensible Warehouse Suite includes SAND Searchable Archive and SAND Analytic Server. We used SAND Analytic Server for online analytics on 401 days' worth of current data and SAND Searchable Archive to store historical data for 15 years. The SAND Analytic Server provided excellent performance for online analytics. SAND Searchable Archive allowed us to implement an automated archiving process that resulted in cost-efficient storage of the aged ticketing data, while still keeping it available for analysis. Aged data (older than 401 days) can now be relocated in highly compressed "near-line" archive files and then restored to the online analytic environment as and when required - something that would have been extremely difficult using a tape-based archive. The SAND Extensible Warehouse architecture represents a solution to the challenge of keeping many years' worth of data available for analysis without negatively affecting the performance of the primary analytic systems.
PRODUCT FUNCTIONALITY: Current data for analysis is fed from the main Oracle database into a high-performance SAND Analytic Server. Once the data is more than 401 days old, SAND Searchable Archive's business rules engine automatically extracts the required selection of data and produces a highly compressed, read-only archive and then deletes the data from the online server. All transaction data for a 15-year period (representing nearly 13TB of uncompressed flat files) will be stored in just 690GB of SAND Searchable Archive, representing a 95% reduction in enterprise storage without compromising the data's availability to users. When a business user requests a particular archived data set for analysis, the system quickly locates the required data in the near-line archive, extracts it and then loads it into a SAND Analytic Server dedicated to serving restored data.
STRENGTHS: The main strengths of the SAND Extensible Warehouse architecture we have seen so far are: superior analytic performance against online data; high compression rates, giving very economical storage for near-line archives; speed of processing of queries or restore requests on archived data, comparable to a standard relational database; flexible business rules engine for automated archiving that is easily configured to meet our specific needs; and changes to the warehouse data model handled easily via a meta data repository.
WEAKNESSES: The early version of SAND Searchable Archive that we used lacked graphical utilities for system configuration and management. These will be supplied in upcoming releases.
SELECTION CRITERIA: We required fast query performance (for online analytics), high data compression rate (for archiving), low price, simplicity and ease of administration.
DELIVERABLES: The suite delivered a high-performance platform for complex analytics; automated process for producing compressed data archive files; fast on-demand restoration of selected data for analysis via standard business intelligence tools; automatic "housekeeping" of the database holding restored data.
VENDOR SUPPORT: SAND Technology provided experienced, capable consulting staff to assist in setting up the system and continues to provide excellent support.
DOCUMENTATION: The reference documentation is comprehensive and easy to understand.
SAND Extensible Warehouse Suite
SAND Technology, Inc.
215 Redfern, Suite 410
Canada H3Z 3L5
Register or login for access to this item and much more
All Information Management content is archived after seven days.
Community members receive:
- All recent and archived articles
- Conference offers and updates
- A full menu of enewsletter options
- Web seminars, white papers, ebooks
Already have an account? Log In
Don't have an account? Register for Free Unlimited Access