Teraflow Testbed 2007: Sloan Digital Sky Survey

The Teraflow Testbed was extended to Russia in 2007. GLORIAD involved its Russian partners in the Teraflow project now that there are several gigabits of bandwidth between Chicago and Moscow (3x1Gb VLANs on TransLight/StarLight from Chicago to Amsterdam, and 10Gb from Amsterdam to Moscow funded by the Russians).

On May 14, 2007, the first experiment was to download and create a mirror site for Sloan Digital Sky Survey (SDSS) data on a data server at the Russian Space Research Institute (IKI). SDSS data was transferred from University of Illinois at Chicago (UIC) National Center for Data Mining (NCDM) servers to an IKI mirror site www.skyserver.ru over the TransLight/StarLight link. NCDM moved 1.4 TeraBytes (TB) of data in about 4.5 hours over a 1 Gbps VLAN between NCDM in Chicago and Moscow. Using NCDM’s transport protocol UDT (UDP-based Data Transfer), the SDSS 2.5 TB catalog was compressed to 1.4 TB, split into 60 files, distributed, and then decompressed in Moscow to its original size.

In December 2007, IKI installed NCDM’s UDT relay server in order to disseminate large scientific databases and visualization content. IKI has climate change and space weather databases ready for delivery to institutions that are part of the World Data Centers System under ICSU. These databases will be a major addition to the SDSS database. IKI plans to add open source port relaying services to the UDT protocol stack and will maintain the code at SourceForge http://sourceforge.net/projects/udt/. IKI also plans to use UDT to share satellite data with the National Geophysical Data Center at NOAA in Boulder.

URL:

www.ncdm.uic.edu
www.teraflowtestbed.net

Collaborators:

USA:
National Center for Data Mining, University of Illinois at Chicago; Johns Hopkins University University of California, San Diego NASA Goddard Space Flight Center StarLight TransPAC2 Switzerland:
CERN Australia:
University of Melbourne

China:
Chinese Academy of Sciences (CAS)/ Computer Network Information Center
National Astronomical Observatory

Germany:
Garching Computing Centre, Max-Planck-Institut für Plasmaphysik

Japan:
Institute for Cosmic Ray Research, University of Tokyo
JGN2

Korea:
Korea Astronomy and Space Science Institute
Korea Institute of Science and Technology Information
KREONet2

Netherlands:
SARA Computing and Networking Services
University of Amsterdam

Russia:
Russian Academy of Sciences/Space Research Institute (IKI)