Supporting cluster computing in the research community

Posted by Christophe Bisciglia, Senior Software Engineer, Google Research and James French, Program Director, National Science Foundation

Research and development is the foundation of innovation in the technology industry, and both Google and the National Science Foundation (NSF) are committed to making substantial investments in computer science research and education to ensure that our students, faculty and scientists remain on the leading edge of computing and have the tools necessary to make significant advances. As the technology industry moves into an exciting new phase of massively scaled, highly distributed computing, academic users have expressed a need for resources to engage and explore this emerging model, which is already responsible for many "internet-scale" applications that we now take for granted. That's why we're excited to announce that the NSF will use resources made available via Google and IBM's Academic Cluster Computing Initiative (ACCI) to reach the broader research community and explore new applications for massively scaled, highly distributed computing.

In October of 2007, Google and IBM announced the first pilot phase of the ACCI, which granted several prominent U.S. universities access to a cluster of thousands of processors running Apache's Hadoop, an open source distributed computing platform inspired by Google's file system and MapReduce programming model. (There's a YouTube video showing how students and faculty at the University of Washington have made use of the cluster.) Encouraged by these results, we sought out ways to extend the reach of this technology, and given its prominence in national research funding, the NSF emerged as a natural partner. For decades, the NSF has engaged the U.S. research community by setting research agendas and encouraging the development and adoption of disruptive technology, and this partnership will encourage a broader range of academic researchers to develop innovative new methods of data analysis using the unique advantages of massively parallel systems architecture - without the massive infrastructure costs usually associated with large-scale computer clusters.

Starting this year, the NSF will work to evaluate project proposals from academic researchers across many disciplines and select some of the most interesting and promising projects to receive ACCI computational grants. More details, including information on how to apply for access to these resources via the NSF's Cluster Exploratory (CluE) program, will be posted soon on the NSF site.