October 3, 2012 – Big data has enormous potential in the government sector, though little in the way of uptake and strategy at this point, according to a new report from tech industry advocacy non-profit TechAmerica Foundation.

Leaders of TechAmerica’s Federal Big Data Commission on Wednesday unveiled “Demystifying Big Data: A Practical Guide to Transforming the Business of Government.” The 39-page report provides big data basics like definitions and IT options, as well as potentials for deeper data value and government policy talks. Rife in strategy and pointers more than hard numbers on the impact of existing government data initiatives, the report pointed to big data’s “potential to transform government and society itself” by way of cues from successful data-driven private sector enterprises.

“Unfortunately, in the federal government, daily practice frequently undermines official policies that encourage sharing of information both within and among agencies and with citizens. Furthermore, decision-making by leaders in Congress and the Administration often is accomplished without the benefit of key information and without using the power of Big Data to model possible futures, make predictions, and fundamentally connect the ever increasing myriad of dots and data available,” the report’s authors wrote.

In part the reason for the report that TechAmerica called a “first of its kind,” instances of federal big data programs were scant. This is particularly surprising at a time when data produced by the U.S. government is hitting annual highs, with the most recent estimates cited in the report as approximately 848 petabytes created by the federal government in 2009. The report also indicated that big data solutions were now more “affordable,” though there was no price gauge included in the report or in a request for how that term was defined.

The handful of existing examples of implementations in some form of production or use to handle large-scale and unstructured data sets include: a compliance data warehouse at the IRS; the electronic records bank of more than 142 terabytes of information and 7 billion objects at the National Archives and Records Administration; and medical records analytics used at the Centers for Medicare and Medicaid Services.

In quick-hit looks at individual federal agencies, the report pointed to big data potential such as: added health care quality with EHRs and early detection tools; harnessing information from distributed sensors to quickly react to traffic pattern changes and jams; analysis of student learning; proactive fraud detection for taxes; responses to performance issues and other triggers of cybersecurity breaches; better long-term weather pattern recognition models; and direct connections between an unemployed job seeker’s skill set and available work. Across departments, the report suggested annual reviews could add performance metrics, which pile on data from day-to-day activities; big data tools could help follow through on the federal “Digital Government Strategy” for efficiency and modernization; and agency IT could gauge new innovative roles, like the FCC’s Chief Data Officer.

Not unlike the private sector, the report stated that agencies should search for a business problem first, reliant on two main questions: “How will the business of government change to leverage big data?” and “How will legacy business models and systems be disrupted?”

To that second question, the report noted that data management infrastructure adjustments may be in order, such as more storage to take on unstructured elements, along with real-time analytics, MapReduce frameworks and data warehousing. The report also suggested reviews of application accelerators like text extraction tools, geospatial support and video mining, as well as integration support that comes with a governance plan that identifies and prioritizes information strategy and access. Additionally, federal agencies need to take into account network limitations for data loads, though some capacity constraints could be handled by cloud deployments, which the report states enables agencies to “succeed quickly or fail quickly and incorporate their lessons learned.”

On the policy front, the TechAmerica commission made a general call to “remove unnecessary obstacles” toward big data uptake, something that may evolve as more departments bring on data initiatives. It also lauded the $200 million award announced earlier this year for federal big data backing, which TechAmerica indicated is a sign of interest in big data from the top levels of government.

The report recommended a five-step path to moving ahead with big data initiatives:

  1. Define the big data business opportunity.
  2. Assess existing and needed technical capabilities.
  3. Select a deployment pattern based on the velocity, volume and variety of data involved.
  4. Deploy the program “with an eye toward flexibility and expansion.”
  5. Review program against ROI, government policy and user needs.

Big Data Commission leaders are co-chair Steve Mills, SVP and group executive at IBM, and co-chair Steve Lucas, global EVP and general manager of database and technology at SAP, and the board also includes members from AWS, Wyle, Western Governors University and North Carolina State University. The report was introduced Wednesday at the U.S. Capitol Visitor Center in Washington, D.C. TechAmerica is weighing its next steps with the report and its big data commission, according to a spokesperson.