The Met. Office is the national weather bureau service of the United Kingdom. Its mission is to provide weather forecasting and meteorological research to a diverse group of civilian and government agencies including the Royal Air Force (RAF) and the British Broadcasting Corporation (BBC). Formed in 1854, the Met. Office was indispensable in providing accurate weather forecasts that aided allied military operations during both world wars. Today, the Met. Office has over 2,200 employees worldwide.

Years before global warming became a household term, the Met. Office established the Hadley Centre for Climate Prediction and Research. This division, formed in 1990, began investigating and building models of the earth's climate history for the past 2,000 years and forecasting it for the next two millenniums. Large amounts of data were collected from various land, space and ocean-based instruments throughout the world and used as input for those models. The data was then processed with the help of advanced Cray supercomputers, producing vast amounts of numerical data for each experiment. However, by 1996 we realized we had a dilemma.

As the data amounts grew, we were faced with a number of different challenges. First, we knew that if we continued collecting data at the same pace, we were soon going to run out of storage space. Second, the tape cartridges we were using had a data capacity which was too low for our needs, in addition to an unacceptable error rate. Third, we were using a lot of internal resources to write bespoke programs to assist with manual management of the data.

In 1997, we decided to update the system. The Met. Office consulted with the government's Central Computer and Telecommunications Agency (CCTA) to write an operational requirement, or request for proposal, to potential vendors. As we proceeded along the procurement process, we realized that the storage part of the solution was really secondary. The main issue was managing the data. This was crucial to the operation of the whole system.

After a thorough and competitive process (including benchmarking and proof-of-concept presentations), we chose FileTek as the prime contractor. Using their StorHouse data management software, we are able to manage, store and access single rows of detail data across all media types, including tape. This ability to directly access and manage high volumes of unsummarized data without retrieving complete files to disk was a major factor in our decision to choose FileTek. Our file sizes average 80MB. To move multiple files to disk would have created a severe bottleneck and would also have required us to install many more hard drives. Managing vast amounts of data and getting only the data we need, not the whole file, is key in reducing our network load.

In addition to the StorHouse solution (which runs on a Sun Microsystems Ultra Enterprise platform using Solaris 2.6), we also implemented StorageTek 9840 drives and media held in a PowderHorn silo. This, in turn, supports our two Cray T3Es, an IBM OS/390 system and also the scientists' HP-UX and Microsoft NT workstations.

We expect that by 2005, The Met. Office will be managing over one petabyte of data to support the Hadley Centre. By 2010, it should reach close to two and one-half petabytes.

The breadth of this type of project called for a focus on sensible requirements and timetables. We accomplished this by fully analyzing our needs and desires with the available technology. Thus, our expectations were more in sync with what could actually be delivered. It was also imperative not to rush the execution of selecting a lead vendor or implementing a chosen solution.

Throughout the course of this implementation, working closely with the vendor was also very important to reduce the time required to get the system up and running. Sometimes this required our department to work as a liaison between the vendor and other parts of the Met. Office. This turned out to be invaluable in the smooth installation and integration.

The major benefit of this implementation has been the ability to cost-effectively store and manage hundreds of terabytes of data in a logical and efficient fashion. This has enabled the Hadley scientists to quickly analyze huge amounts of information about global warming which in the past would have been painstakingly slow.

FileTek StorHouse permits access to row-level data stored across all storage media (e.g., tape, optical, magnetic disk). It transparently integrates with the most popular databases and supports industry- standard gateways and data access methods. StorHouse is the recipient of the High Technology Council of Maryland's Information Technology Product of the Year (2000).

Register or login for access to this item and much more

All Information Management content is archived after seven days.

Community members receive:
  • All recent and archived articles
  • Conference offers and updates
  • A full menu of enewsletter options
  • Web seminars, white papers, ebooks

Don't have an account? Register for Free Unlimited Access