Date: 2007-04-14 14:52 Subject: Mailbag: Routing in Orbit Security: Public Music: Eskimo - Does This Look Infected To You?

It's time once again to see what's in giantlaser's mailbag!

jwz asks: I eagerly await your post explaining whether this is cool or lame:

http://blog.wired.com/wiredscience/2007/04/space_symposium_5.html

It sounds like it's about routing packets satellite-to-satellite, instead of each satellite just being a point-to-point to a ground station.

Before I answer, some background information. The single biggest obstable for satellite Internet services is latency. In network terms, latency is the total round trip time for your packet to reach its destination and for the reply to return. When we speak of "ping time", what we mean is latency. Latency is usually measured in milliseconds (ms).

Here are some common latency measurements:

1 ms - within your LAN
20 ms - my home cable service in London to my co-located servers at UK5 (King's Cross, a few kilometers away)
90 ms - typical home DSL in the US to google.com
100-150 ms - the transatlantic cable between the UK and New York state
600-2000 ms - typical VSAT remote to hub link

Yes, that's big.

Geosynchronous satellites orbit at an altitude of 35,000 km. VSAT remotes connect to the Internet via a centralised hub design. When a remote pings google.com, its traffic must travel up to the satellite, then back down to our teleport, over the terrestrial Internet to google.com. The reply does the opposite, returning back over the terrestrial Internet, back up to the satellite and down again to the remote. That means every single ping you send is travelling 35,000 x 4 = 120,000 km. Divide by c, and we learn that minimum latency for any VSAT is 480 ms. Further time is added due to radio modulation, error correction, router delay, and various technical issues with shared bandwidth satellite networks. All that adds up to a minimum effective latency for all VSAT communications of about 600 ms. If that network is busy, it goes up.

For example, one of our competitors is known for latency as high as 2000 ms. We run a tighter ship than that, which is one reason we're one of the fastest growing VSAT providers in the world.

So when one of our customers makes a VoIP phone call, there is an audible (600 ms) delay between when he speaks and when he can expect a reply. We constantly work to keep that number as low as possible. People seem to tolerate up to 250 ms without noticing, but when you start edging up to one second, it quickly becomes intolerable. "Hi Tom, how are you? Over!" "Great, Jim, I'm fine. How is your new job? Over!"

What happens when two VSATs want to talk directly? They "double-hop"; traffic from VSAT1 goes to the hub, on to VSAT2, which replies back to the hub, and is then sent back to VSAT1. Which means any site-to-site communication has a latency of 1200 ms. There are some technologies for dealing with this (mesh mode, and mixed star/mesh mode), but they are too expensive for most commercial providers to deploy without a customer willing to pay for it up front.

So, what can this new IntelSat/Cisco experiment do for us? It can move the problem of routing to the satellite. This means latency between sites could be cut in half. Further, it could make it possible to route between sites on different networks of the same satellite. For example, if I run one network for my customers on EutelSat AM-22, and another provider is doing the same (say, on another transponder or frequency band), it may be possible for our packets to route directly on the satellite. Currently, if one of my customers wants to communicate with one of my competitors, his traffic must route to our hub, over the public Internet, out his hub, and off the satellite again.

You might say that's not good for me, because I want to lock in my customers. But that's deceptive; faster routing helps everyone. It might even mean Google suddenly wants to pay for a link just to the satellite in orbit, so customers can route directly to them without going down to my hub first.

The mesh mode I mentioned above is similar to what the router-in-space could do, except it is doing the job at layer 2 (equivalent to Ethernet, but really radio), and the router-in-space is doing it at layer 3 (IP). And it cannot route between two networks.

There are all kinds of technical and political limitations to overcome. For instance, some networks are encrypted at the radio level and couldn't be decoded at the satellite. All providers on a satellite would have to cooperate to provide routing information and agree to route packets directly. The satellite would require a higher power budget and failover routers, because now it would be doing more than simply retransmitting signals received in one transponder back out another.

Satellite businesses operate on a tier model. The satellite operator manages the bird itself and assigns space segment (radio spectrum) to wholesale providers. Providers take that space segment, create networks on them, and sell to resellers. Resellers then service individual customers.

It would be easy for IntelSat to provide this technology, but it would have to become a vertical monopoly. It would require them to cut out providers like my company altogether and go direct to the customer or reseller. Most satellite operators don't want to deal with "my Internet is broke" customers, and most dumb customers cannot understand radio. So there is a big need for providers like my company.

Secondly, most providers fall into one of two camps: radio guys who somehow configured a router, and network guys who somehow got radio gear working. IntelSat are radio guys, so I question how well they'll do at this, technically.

Conclusion: cool. Way cool. But with a lot of problems to solve.