Caltech, SLAC and LANL Set New Network Performance Marks at the SC2003 Conference on High-performance Networking and Computing

PHOENIX, AZ -- Teams of physicists, computer scientists and network engineers from Caltech, SLAC, LANL, CERN and Manchester joined forces at the SuperComputing 2003 Bandwidth Challenge and captured the Sustained Bandwidth Award for their demonstration of "Distributed Particle Physics Analysis Using Ultra-High Speed TCP on the Grid," with a record bandwidth mark of 23.2 Gigabits/sec (or 23.2 billion bits per second).

The demonstration served to preview future Grid systems on a global scale, where communities of hundreds to thousands of scientists around the world would be able to access, process and analyze data samples of up to Terabytes, drawn from data stores thousands of times larger. A new generation of Grid systems are being developed in the US and Europe to meet these challenges and to support the next generation of high energy physics experiments that are now under construction at the CERN laboratory in Geneva. The currently operating high energy physics experiments at SLAC (Palo Alto, CA), Fermilab (Batavia, IL) and BNL (Upton, NY) are facing qualitatively similar challenges.

During the Bandwidth Challenge, the teams used all three of the 10 Gigabit/sec links provided by Level 3 Communications, Inc. (Nasdaq:LVLT) and Nortel, connecting the SC2003 site to Los Angeles, and from there to the Abilene backbone of Internet2, the TeraGrid, and to Palo Alto using a link provided by CENIC/National Lambda Rail. The bandwidth mark achieved was more than 500,000 times faster than an Internet user with a typical modem connection (43 kbits/sec). The amount of TCP data transferred during the 48 minute-long demonstration was over 6.6 TeraBytes (or 6.6 trillion bytes). Typical single stream host-to-host TCP data rates achieved were 3.5 to 5 Gbits/s, approaching the single stream bandwidth records set last month by Caltech and CERN.

The data, generated on the SC2003 showroom floor at Phoenix, a cluster at the StarLight facility in Chicago as well as the TeraGrid node at Caltech, was sent to sites in four countries (USA, Switzerland, Netherlands, and Japan) on three continents. Participating sites in the winning effort were the Caltech/DataTAG and Amsterdam/SURFnet PoPs at Chicago (hosted by StarLight), the Caltech PoP at Los Angeles (hosted by CENIC/NLR), the SLAC PoP at Palo Alto, the CERN and the DataTAG backbone in Geneva, the University of Amsterdam and SURFnet in Amsterdam, the AMPATH PoP at Florida International University in Miami, the University of Manchester, and the KEK Laboratory in Tokyo. Support was provided by DOE, NSF, PPARC, Cisco, Level 3, Nortel, Hewlett Packard, Intel and Foundry Networks.

The team showed the ability to use efficiently both dedicated and shared IP backbones. Peak traffic on the Los Angeles-Phoenix circuit, dedicated to this experiment, reached 10 Gbits/s utilizing more than 99% of the capacity. On the shared Abilene and TeraGrid circuits the experiment was able to share fairly over 85% of the available bandwidth. Snapshots of the maximum link utilizations during the demonstration showed 8.7 Gbits/s on the Abilene link and 9.6 Gbits/s on the TeraGrid link.

This performance would never have been achieved without the use of new TCP implementations because the widely deployed TCP RENO protocol performs poorly at Gbits/s speed. The primary TCP algorithm used was new FAST TCP stack developed at the Caltech Netlab. Additional streams were generated using HS-TCP implemented at Manchester and scalable TCP from Cambridge.

Dr. Harvey Newman, Professor of Physics at Caltech, said, "This was a milestone in our development of wide area networks, and of global data-intensive systems for science. Within the past year we have learned how to use shared networks up to the 10 Gbits/sec range effectively. In the next round we will combine these developments with the dynamic building of optical paths across countries and oceans. This paves the way for more flexible, efficient sharing of data by scientists in many countries, and could be a key factor enabling the next round of physics discoveries at the high energy frontier. There are also profound implications for integrating information sharing and on-demand audiovisual collaboration in our daily lives, with a scale and quality previously unimaginable."

Dr. Les Cottrell, assistant director of SLAC's computer services, said, "This demonstrates that commonly available standard commercial hardware and software, can effectively and fairly use and fill up today's high-speed Internet backbones, and sustain TCP flows of many Gbits/s on both dedicated and shared intra country and trans-continental networks. As 10Gbits/s Ethernet equipment follows the price reduction curve experienced by earlier lower speed standards this will enable the next generation of high speed networking and catalyze new data-intensive applications such as High Energy Physics, astronomy, global weather, bioinformatics, seismology, medicine, disaster recovery and media distribution."

Dr. Wu-chun (Wu) Feng, team leader of Research & Development in Advanced Network Technology in the Advanced Computing Laboratory at LANL, noted, "The SC2003 Bandwidth Challenge provided an ideal venue to demonstrate how a multi-institutional and multi-vendor team can quickly come together to achieve a feat that would otherwise be unimaginable today. Through the collaborative efforts of Caltech, SLAC, and LANL, we have once again pushed the envelope of high-performance networking. Moore's Law Move Over!"

"Level 3 was pleased to support the SC2003 conference again this year," said Paul Fernes, director of business development for Level 3. "We've provided network services for this event for the past three years because we view the conference as a leading indicator of the next-generation of scientific applications that distinguished researchers from all over the world are working diligently to unleash. Level 3 will continue to serve the advanced networking needs of the research and academic community, as we believe that we have a technologically superior broadband infrastructure that can help enable new scientific applications that are poised to significantly contribute to societies around the globe."

About Caltech
With an outstanding faculty, including four Nobel laureates, and such off-campus facilities as the Jet Propulsion Laboratory, Palomar Observatory, and the W.M. Keck Observatory, the California Institute of Technology is one of the world's major research centers. The Institute also conducts instruction in science and engineering for a student body of approximately 900 undergraduates and 1,000 graduate students who maintain a high level of scholarship and intellectual achievement. Caltech's 124-acre campus is situated in Pasadena, California, a city of 135,000 at the foot of the San Gabriel Mountains, approximately 30 miles inland from the Pacific Ocean and 10 miles northeast of the Los Angeles Civic Center. Caltech is an independent, privately supported university, and is not affiliated with either the University of California system or the California State Polytechnic universities.

About CERN
CERN, the European Organization for Nuclear Research, has its headquarters in Geneva. At present, its Member States are Austria, Belgium, Bulgaria, the Czech Republic, Denmark, Finland, France, Germany, Greece, Hungary, Italy, Netherlands, Norway, Poland, Portugal, Slovakia, Spain, Sweden, Switzerland and the United Kingdom. Israel, Japan, the Russian Federation, the United States of America, Turkey, the European Commission and UNESCO have observer status.

About DataTAG
The European DataTAG is a project co-funded by the European Union, the US Department of Energy and the National Science Foundation. It is led by CERN together with four other partners. The project brings together the following European leading research agencies: Italy's Istituto Nazionale di Fisica Nucleare (INFN), France's Institut National de Recherche en Informatique et en Automatique (INRIA), the UK's Particle Physics and Astronomy Research Council (PPARC), and the Dutch University of Amsterdam (UvA). The DataTAG project is very closely associated with the European Union DataGrid project, the largest Grid project in Europe also led by CERN.

About SLAC
The Stanford Linear Accelerator Center (SLAC) is one of the world 's leading research laboratories. Its mission is to design, construct and operate state-of-the-art electron accelerators and related experimental facilities for use in high-energy physics and synchrotron radiation research. In the course of doing so, it has established the largest known database in the world, which grows at 1 Terabyte per day. That, and its central role in the world of high energy physics collaboration, places SLAC at the forefront of the international drive to optimize the world-wide, high-speed transfer of bulk data.

About LANL
Los Alamos National Laboratory is operated by the University of California for the National Nuclear Security Administration of the U.S. Department of Energy and works in partnership with NNSA's Sandia and Lawrence Livermore national laboratories to support NNSA in its mission. Los Alamos enhances global security by ensuring the safety and reliability of the U.S. nuclear weapons stockpile, developing technical solutions to reduce the threat of weapons of mass destruction and solving problems related to energy, environment, infrastructure, health and national security concerns.

About StarLight
StarLight is an advanced optical infrastructure and proving ground for network services optimized for high-performance applications. Operational since summer 2001, StarLight is a 1GE and 10GE switch/router facility for high-performance access to participating networks and also offers true optical switching for wavelengths. StarLight is being developed by the Electronic Visualization Laboratory (EVL) at the University of Illinois at Chicago (UIC), the International Center for Advanced Internet Research (iCAIR) at Northwestern University, and the Mathematics and Computer Science Division at Argonne National Laboratory, in partnership with Canada's CANARIE and the Netherlands' SURFnet. STAR TAP and StarLight are made possible by major funding from the US National Science Foundation to UIC. StarLight is a service mark of the Board of Trustees of the University of Illinois.