Editor's Introduction: One evening in Moscow in late September Steve Goldstein told us the history of the NSF International Connections Program. We found it a fascinating story of how the National Science Foundation for less than a million dollars a year over a five year time frame has done some pump priming and helped to grow an infrastructure that has made the US a focal point for international Internet connections. In November we asked him if he would agree to a more formal interview ranging from its origins to plans for a 1995 solicitation that will replace the initial one of 1990.

COOK Report: What was the sequence of events that led to the initial solicitation in 1990?

Goldstein: When I came to NSF in 1989 I had been working for NASA and NASA funded astronomers were very interested in reaching the database known as SIMBAD: (Set of Identifications, Measurements and Bibliography for Astronomical Data) maintained on a Univac computer in Orsay near Paris. This is probably the world's finest database of its kind. In order to get to it, they were making an x.25 call and paying the French government a charge because the database was created and maintained by a French government agency. On this basis the cost to NASA astronomers varied between $100 and $200 a search.

Not surprisingly NASA astronomers were interested in being able to use an experimental line that NSF was funding to support research work between Larry Landweber at Wisconsin and Christian Huitema at INRIA in Sophia Antipolis in southern France. They were developing protocol translating gateways. Their objective was to develop a gateway that would translate between TCP of the Internet protocol stack and its equivalent layer in the ISO protocol stack. Initially they decided on a simpler goal: translation between the TCP and triple X protocols - X.3, X.28, X.29 and so on.

Before coming to NSF in the second half of 1989 I was anemployee of Mitre Corporation helping NASA on these issues including some of the initial setup of the NASA Sciences Internet. At this point you could reach the Simbad database by using Transpac the French packet switched network. NASA was cooperating with NSF, and in this particular situation we also received the help of the French government and in particular Christian Huitema who rigged a gateway between his Internet node at Sophia Antipolis and Transpac. Consequently by doing a telnet to Sophia Antipolis, an astronomer could gateway in Transpac and go up to Orsay to search the Simbad database. (By the way this same database is now on the World Wide Web - http://cdsweb.u-strasbg.fr/Simbad.html - one click and you are there.)

That was my introduction to the international stuff and I can remember being at the International Astronomical Union meeting in the Baltimore area where we did demos with dial up modems at 1200 baud. So later in 1989 when I went to NSF, the situation was that we had that line to France. There was a line to the UK funded through JVNC and another one to Nordunet. However the line for Larry Landweber's project to France went directly from the Princeton University Computer Center.

When Landweber's project was over he had little interest in keeping the link alive. But by then because of the Transpac gateway, people were coming to depend on the availability of that line.

COOK Report: To reach the Simbad database or do other things?

Goldstein: Hard to say. All of a sudden there was this international link to France that had taken on an operational flavor. Not only for Simbad, but maybe for other things too. When people found that the cost of Simbad searches for which they had been paying $100 to 200 for declined to the one to two dollar range because they no longer had the X.25 call to worry about, they became very interested in using the line.

So the idea at that point that hit me was let's see if we can consolidate this and get one organization to manage coordinate and engineer all of our international connections. Consequently in the spring of 1990 after all the necessary internal work and reviews were done, we went out with a competitive solicitation to do just that. We had a panel meeting in September and in record time we got the thing awarded by the end of the year. We went from final recommendations to the award in something like six weeks - normally that could have taken six months.

COOK Report: Any particular reason why it was able to go so quickly?

Goldstein: Well - for example every time the people in the Grants Office wanted to make changes in the wording of the cooperative agreement, I'd run downstairs and do it on my Macintosh and then run back upstairs and they wondered how we could get our secretaries to turn things around so fast. There was a lot of word noodling. But we kept a full court press on and got the job done. The award was made with an effective start date of January 1991 with a ramp up time of a few months.

COOK Report: We have heard someone at NSF say that you didn't expect a bid from an IXC and that when one came in you were delighted. Comments?

Goldstein: Traditionally the kinds of proposals that we would get would be from organizations like Merit. In other words an academically based organization that would involve an alliance with some kind of a telecommunications service provider in the package but usually as a sub awardee of the main awardee. For example in the NSFnet backbone MCI was a participant but the grant went to Merit. In this particular case Sprint had made a conscious decision that it wanted to do the value added stuff and it bid with Cornell University as a sub-awardee to do the Network Operations Center work. This was surprising because we just didn't normally get those kinds of bids. When after evaluation we made the award to Sprint we were delighted that yet another carrier was playing in the game. Because the NSFnet backbone was MCI we wanted to encourage as many telecommunications providers as possible to be in the game. We were equally pleased when AT&T became an InterNic awardee.

For Sprint it was a learning experience. They had picked a strong group at Cornell to do the operational management. Mark Oros was leader of the Cornell team. While the Cornell people did excellent work, Sprint after about a year migrated the NOC to the Washington DC area in full cooperation with Cornell.

COOK Report: How would you evaluate the reason for the move to Washington by Sprint?

Goldstein: Well, I think after doing this for a while Sprint realized that there was a real market there for a service that they could provide. If it was just going to be two circuits - one to Stockholm and the other to Paris, they probably would have left it that way in perpetuity. But they said: A hah! We see a real business opportunity here. And in so doing they credited us with helping them spawn a new line of business.

The next big thing to happen - roughly in 1992 was that NASA, NSF and DARPA were cooperating on a circuit we called the "fat pipe" to the UK.

COOK Report: Fat was T-1?

Goldstein: No I think it was 512. And at any rate by consent among the three of us we switched that over to the cooperative agreement with Sprint and Sprint began to manage that.

And then at some point later in 1992 some other countries that wanted to join the Internet got wind of things that Sprint was doing for us as International connections manager. Also for them the Internet meant the United States because so many of the resources that were on the Internet were located here.

Also there is another very stark reality that says every international circuit is paid for as though it were two half circuits. And each half circuit is paid to the telecommunications provider in each of the two respective countries. Moreover each of the two parties will have a different tariff rate for the half circuit. It is almost always the case that the other country's service provider charges a lot more than the US service provider - if for no other reason than since the 1984 break up we have a lot more competition here in this country.

COOK Report: So if you can get an American half circuit into the equation and you can get to the Internet by coming here you can essentially get to the world and lower your costs by connecting to the US?

Goldstein: Right. Cheaper than almost any other way of doing it anywhere. Now an example of one the greatest differentials that I can think of is a 64 kbs circuit to Indonesia. The US half of the circuit is something on the order of $3500 a month and the Indonesian half is roughly $8500 a month. If they were to connect anywhere but the US they would still pay $8500 a month for the Indonesian half and 4 to 6 thousand a month for that other country's half circuit.

So both because of the Internet activity here and because of our favorable international tariffs, the US became the best place to come. Also because of the infrastructure we were building with Sprint it was a logical place to connect. At this point everything was being done in the Washington area. MAE-East was formed around this time (late 92). One thing attracted another and you soon had a snowball effect in late 1992 or early 1993.

When at about this time Sprint did its SprintLink kick off it established a node in Stockton California. As a result they began attracting Pacific countries. I think Maylasia was one of the first ones. So a Pacific country could come into Stockton and then come over SprintLink and get to the East Coast where packets could be passed off to the other Federal networks at FIX East. Eventually they also connected to FIX west. But regardless, when you connected to SprintLink, you were in the Internet. European countries tended to come into Sprint's router at MAE-EAST and Pacific Rim countries into Stockton.

The Florida Connection
The key thing to understand here is that all these countries were connecting at their own expense - almost. And here's the almost. In December 1991 we had a meeting in Guadalajara Mexico of Latin American countries interested in connecting to the Internet. These countries were attracted to PanAm Sat because its rates which were cheaper than Intel Sat's. We said if enough of you want to come in at PanAm Sat, we will ask Sprint to extend the connection down there.

We got plenty of takers. Consequently Sprint put in a router right at the PanAm Sat teleport in Homestead Florida with a dedicated T-1 to the Washington POP. It was supposed to go in right before Hurricane Andrew hit. After the hurricane it was January of 1993 before the teleport was able to do the connect.

Sprint came in with an elaborate cost recovery formula just to pay for the T-1 link. When we examined the formula we felt that it was hopelessly complicated and that it would create problems with our internal people every time we wanted to bring in a new country. We asked Sprint to bundle the cost and come up with a single average figure. Sprint agreed and in doing so took a risk of losing money if fewer countries connected than they predicted. And in fact for the longest time only two countries connected. Ecuador was first and then Costa Rica. When the others saw the first connects they knew it was real and started working on their own governments. Soon there was another snowball.

COOK Report: Did NSF kick in some initial seed money?

Goldstein: No. All we did was once their signal landed in Homestead we paid Sprint what we came to call a port management fee. This was the bundled average fee for them to recover the cost of bringing the signal to Washington, paying for the router in Florida and managing the connection. This fee became traditional so that when some of the other countries connected elsewhere - for example when Maylasia connected on the West coast we agreed that we would pay that port fee.

Earlier this year after we had paid Sprint on the order of 10 or 15 port fees, I asked Sprint to take a look at that charge again because it seemed that we had quite a nice snowball effect going. Sprint agreed and voluntarily lowered their rate. I think it has been a very successful collaboration. They took risks on certain things and were very very good partners.

COOK Report: But what did the port fees finally cost? $500 a month? $900 a month?.

Goldstein: I'm not sure that the exact amount should be advertised. I think that may be Sprint's business.

COOK Report: In comparison with the total cost of the satellite link were they at least an order of magnitude less. What was the rational for spending Federal funds?

Goldstein: Yes. The reason for this program was to serve the needs of the research and education community. Scientific and educational collaboration now is global. Of course there are countries that can be considered as research or academic "hot spots." And there are some countries where collaboration opportunities for US scientists are somewhat more iffy. So in terms of some real hot spots where we would be willing and able to make major investments - given a very limited budget - we were sharing the actually costs of some links. But with very limited money there were other countries where we could see nice opportunities for collaborations. In these case there were no funds to share link costs but at least as a courtesy to these countries we could provide some help by paying the port fee.

COOK Report: But where did these port fees fall within the scope of the cooperative agreement?

Goldstein: Prior to our bringing the UK Fat Pipe under theICM Cooperative Agreement, the NSF Office of General Counsel reviewed the request and the solicitation and determined that it was global and not limited to France and the Nordic countries. The individual port fees were, and still are, approved by our Grants office (now the Division of Grants and Agreements) as an element of the ICM Annual Program Plan.

If a foreign country had connected directly to a US service provider the country would have paid rates (usually in the form of membership fees) greater than what Sprint was charging us. On the other hand I know of a few instances where US providers were allowing connections at no charge. This was merely a business decision that some service providers were making. In effect the Sprint port fee was like the annual ten to twenty thousand dollar membership fee that regional networks would charge. What we were paying Sprint was less than most of the annual membership fees we knew about.

COOK Report: At the end of 1994 about how many foreign countries are coming in through Sprint?

Goldstein: About 40. Others come directly into national providers or regional nets. Singapore and Taiwan into JVNC for example. Thailand and one Indian network into UUNET, Chile into Suranet, Israel into Nysernet. Also, many countries connect to NASA Science Internet or the Energy Sciences Network.

COOK Report: If there are a total of 80 or so nations with direct TCP/IP connectivity. Looking at what you have enumerated and subtracting one assumes that quite a few nations must be hooked directly to service providers in Europe or Asia?

Goldstein: Yes Egypt is one example. It connects to France. As a matter of fact a colleague in Egypt called me up about a connection to the US and I said, "Don't get me wrong. I don't want to discourage you. We'd be glad to have you connect, but you already have a connection to France and if you don't have enough bandwidth it would probably be more cost effective for you to increase the bandwidth of your French connection. You don't need to have a line here because we aren't the center of the universe." So they did increase their French connection. At the time we had a T-1 into Paris. Now we have three megabits into Paris.

COOK Report: So how did that increase in bandwidth to Paris evolve?

Goldstein: Its like roads onto a freeway. You add more lanes. Two weeks or two months later they're crowded. So you add more lanes and soon it gets crowded again. We had a T-1 that was getting congested. Then we got an E-1 in and were going to tear down the T-1. But the E-1 looked like it would become rapidly congested and the E-bone found a commercial customer and suggested that we share the T-1 between research and commercial customers.

The E-bone

COOK Report: But what is E-bone?

Goldstein: E-bone was at one point called the European backbone. It was an ad hoc consortium of European networks that created a backbone infrastructure. At one point the original organization was followed by a phoenix like rebirth of the "new" E-bone. The new Ebone is not to be confused with the old ebone which had about five nodes. Links between the nodes were paid for by a pool. Since then Europe also has this Europa Net (European Multi- protocol Backbone Service - EMPB) managed by an organization called DANTE. E-bone and EuropaNet are competitors in a sense. E-bone is a good deal smaller, but still it has nodes in Paris, Austria and elsewhere. EuNet may have been a part of the original E-bone operation but I don't think they are a part of the new operation.

The customer that French have I think is an E-bone customer. When we originally implemented 768 to Stockholm, 256 was subscribed to by commercial users one on the US side and one on the Nordic side. We did that on purpose. It was done in such a way that we were sure that we would get at least as much bandwidth as we were paying for. The way this was done was that the commercial customer entered the Trans Atlantic circuit with a serial line rated at 256 kilobits which meant that 256 KBS was they maximum they could pump. On the other hand we saw the entire 768. We could burst across that entire bandwidth. Since then the line grew to E-1 while we are still doing the sharing with commercial providers. This was done by conscious design, because the eventual direction of things seems to be that the international service will be provided by general commercial providers as is now the case in the US.

COOK Report: Well this gets into the question of the next solicitation. We'd appreciate hearing what you can say about it.

Goldstein: OK To recap. We are now in year 4 of a 5 year cooperative agreement with Sprint. We have two E-1s to Stockholm, two E-1s to London plus a T-1 to London - the T-1 is now shared by ARPA and NASA. We have an E-1 and one megabit of a T-1 to Paris. There is every indication that we will keep hitting congestion and have to add more bandwidth. In addition there are a number of countries that are connecting to this infrastructure that Sprint built with ICM and under its own SprintLink initiative. In many cases we are paying at our discretion a port management fee on behalf of those countries which are paying the total cost of their actual links to the US. A lot of these circuits are 64 kilobits but a lot are also congested. A very interesting indication of what could be a trend is Maylasia that started with 64 kilobits and plans now to upgrade in one step to T-1. I think we'll see more of this. South Africa is now at 128 and they are looking at the possibility of doing at least 256 and as soon as they can afford it T-1 or E-1.

There seems to be no end in sight. There is however a finite budget that we have to spend on this.

Money Spent and the New Solicitation

COOK Report: Are there any figures that you could give on the total amounts that you have spent in the program?

Goldstein: Well I haven't added it up. We started the first year at maybe $100,000. Ramped up to about a half million and then last year were roughly $700,000. And this year about 1.2 million I think. Almost all of this is for Trans Atlantic circuits. By the way we are getting nicely discounted prices from Sprint on them.

As we look at the future the problem is that the community we are serving is no longer going to be served by a well defined NSFnet backbone. Instead service will be spread out over several general purpose backbones. Thus the question becomes how we identify AUP (Acceptable Use Policy) traffic from this community to estimate in any rational way the cost of international service? We don't have packet counters that can distinguish between AUP acceptable packets. When traffic entered the ICM pipes from the NSFnet backbone, we could say that it was AUP clean. With no NSF backbone and traffic being aggregated at AUP free NAPs to be shipped overseas we can no longer distinguish between R&E traffic and that of Chase Manhattan Bank or other commercial users.

Remember that DANTE is the organization that runs EuropaNet. They have just announced that they are removing AUP restrictions from their backbone. My guess is that sooner or later in Europe there will be general service provision of backbones just as we have now in the US. What I am saying is that in the not to distant future - and I don't know if we are talking one three or five years - but certainly within that time frame a lot of the so called R&E community in Europe will be served by general purpose providers just as the R&E community will now be in the US.

In this kind of situation it doesn't make any sense for the US government to stay in there and maintain its own intercontinental links and with the exception of Scandinavia, England and France we really aren't doing that anyway. We are doing some cooperative funding with Mexico on their link, but they are paying by far the lions share. We have a little bit of assistance to South Africa, largely due to the fact that they are relaying mail from neighboring countries.

COOK Report: So how does this all focus on the solicitation that I believe you are working on?

Goldstein: Well I have ideas. But at this point I really have no way of know what is going to come out after all the necessary internal reviews. As to when? We hope it is going to come out very very early in calendar year 95.

COOK Report: It sounds like you are hinting that when it does come out one can expect a phased ramp down in direct support of what are now commodity level international connections? Is that a fair assessment?

Goldstein: Yes. One can expect that one element of the solicitation will be a transition for this kind of connection to be funded in some other way than by NSF. Now on the other hand we do expect to be involved in links support high performance kinds of activity. these would be of a pre-tariffed experimental nature.

COOK Report: International analogs of the vBNS here in the US?

Goldstein: Yes. There are some high speed networking experiments in various parts of Europe and maybe Australia. There would be an interest in being able to do high performance applications cooperatively with research interests in other countries. We would need intercontinental links to be able to support those.

Also new technology promises to make available the capability of being able to provide Internet services to people on remote or mobile platforms. The metaphor might be "World Wide Webb to the Rain Forest" and also to platforms such as oceanographic research vessels. I am not sure at this point whether this would fall into the same solicitation or whether it might go as a separate solicitation. Lets just say that we are looking for an appropriate means of serving remote and mobile applications that have here- to-fore not been feasible.

Randy Bush, John Klensin and Program Autonomy

COOK Report: Let's close with a couple of footnotes. How does Randy Bush fit into the international picture?

Goldstein: We made an award for Network Startup Support to co- PI's Randy Bush and John Klensin (John was then at the International Food Research Center, housed at MIT; he has since joined MCI). The grant paid for a small amount of John's time as a contributor to various startup activities in the IETF framework and for Randy's network (RAINnet) connection to the Internet (first via Alternet and later through SprintLink). Randy and John run a Gopher hole (gopher.psg.com) and Web site (www.psg.com) on which they have placed a lot of information that is very much in demand for network startup activities, both domestic and foreign.

Randy's work has been pro bono as regards anything but the network connection. And, indeed, Randy continues to accept assignments to help network startup activities by rolling up his sleeves and connecting the wires and configuring the routers and servers, often at much less remuneration than what he has to charge as an independent object-oriented programmer consultant in order to put bread on his table. He has been a key player in startup activities in Peru, Indonesia, Guinea and Sri Lanka, to name those known to me, and he also runs a UUCP relay site for other international networks not yet on the Internet (he did this for South Africa before we could accept their traffic--due to the international embargo on South Africa--and for Peru before they got their own Internet connection, for example). Our annual grant for this activity is in the neighborhood of $50,000 (for three years).

COOK Report: To sum up then: your record for connecting the world over the past years has been quite impressive. We wonder however to what extent you have been able to do this with complete independence of other arms of the Federal government such as, for example, the State Department?

Goldstein: There has been much informal consultation among the various agencies, and formal inquiries to the Commerce Department's Export Administration in the COCOM days. And, we certainly are guided by the positions of the State Department, as in not routing traffic from terrorist states. But, we have not applied to State directly for permission to route traffic from any country (though NSF has sought State's advice on occasion). Nor do we think it would be wise to establish such a precedent. In the case of Cuba, we applied for a license to route its traffic on the NSFNET backbone and in the Regionals from the Department of the Treasury, which administers the Trading with the Enemy Act. Treasury coordinated it with State and other government agencies, as did Commerce when we asked for advisory opinions about Russia, for example. (By the way, Treasury ruled that we did not require a license under the provisions of TWEA, so we are free to route any AUP-compliant Cuban traffic that may be presented to us.)

COOK Report: Perhaps the best known example was the listing of Russian IP sites in the MERIT policy routing data base?

Goldstein: This was done after consulting with the Export Administration which raised no objection.

COOK Report: Any other examples?

Goldstein: Well, there are some that go the other way. For example, though Iran is routed in Europe, we declined requests to route Iran on the NSFNET Backbone. We made the decision based on the State Department's having classified Iran as a terrorist state (and some well-remembered TV newsreels of the American Embassy and its inhabitants in Teheran.