Network Update: Multihomed, Increased Transit, Peering

November 2, 2016 10:00 am

Linode has completed initial Internet capacity upgrades in all facilities. All of our datacenters are now multihomed with several tier-1 transit providers and the largest peering exchanges in the world. This marks a milestone in Linode’s history: we now manage our own true service provider network, allowing us to deliver robust and reliable connectivity.

To accomplish this, our datacenters have been upgraded with Cisco ASR 9000 series edge routers, allowing us to scale to many terabits of capacity per datacenter. Currently, each datacenter has a few hundred gigabits of transit and peering, with more to come. We’re operating our network under ASN63949.

Why is this a good thing?

  • We are now completely independent to make routing decisions, allowing us to quickly route around trouble on the Internet in real time.
  • We’ve increased IP capacity to each facility by 5 to 10 times over our previous capacity amounts.
  • We have high capacity edge routers and self-lit fiber backhauls in carrier hotels providing many terabits of additional expansion capabilities. Since we are our own backhaul provider, bringing up additional capacity between locations is fast and easy.
  • We can do cool stuff in the future like inter-dc connectivity, per-customer VLANs, and Anycast announcements of some or all of our IP space.

Let’s take a look at the state of each datacenter:

Dallas

Retrofitted: July 25, 2016
Transit: Telia, NTT, Cogent, Zayo (soon: GTT, Tata)
Peering: Dallas Equinix Internet Exchange

We chose 1950 Stemmons and 2323 Bryan Street, two major regional carrier hotels, as our points of presence in Dallas. Also known as The Infomart, 1950 Stemmons is one of the largest buildings in Dallas, measuring at over 1.5 million square feet. Power and connectivity are plentiful at the Infomart; the building has four diverse power feeds and 100+ carriers redundantly built into the building.

Fremont

Retrofitted: September 23, 2016
Transit: Telia, Hurricane Electric, Cogent (soon: NTT, GTT, Tata)
Peering: San Jose Equinix Internet Exchange (soon: Any2 Los Angeles)

While there are several good options for points of presence in the Bay Area, we decided on 11 Great Oaks and 55 S Market, two large facilities with great carrier diversity. These are our longest fiber spans, requiring amplification in order to take a trip under the San Francisco Bay.

Atlanta

Retrofitted: September 21, 2016
Transit: Telia, NTT, Cogent, Zayo (soon: GTT, Tata)
Peering: Atlanta Telx Internet Exchange

In Atlanta we chose a single facility, 56 Marietta, to serve as our regional point of presence. This well-known carrier hotel was selected over all the other options in the Atlanta metro area simply because no other facilities could compete with Marietta’s connectivity. To ensure total physical redundancy, our diverse dark fiber enters 56 Marietta through different building entrances and lands in separate meet-me rooms using entirely distinct power systems.

Newark

Retrofitted: August 1, 2016
Transit: Telia, NTT, Cogent, Zayo (soon: GTT, Tata)
Peering: NYIIX, DE-CIX New York

In Newark, we chose to light points of presence in 165 Halsey and 111 8th Avenue, both famous carrier hotels in the NJ/NY metro area. We selected 165 Halsey in particular to give us physical diversity from the island of Manhattan. Having our physical footprint spread out over the entire New York metro area makes us much less susceptible to downtime during a catastrophic event like Superstorm Sandy.

London

Retrofitted: July 11, 2016
Transit: Telia, NTT, Cogent (soon: GTT, Tata, Vodafone, Sparkle)
Peering: LINX, AMS-IX, LONAP

In London, we chose 8/9 HEX and 14 Coriander Avenue to light points of presence. Because of the stature of these facilities and the more friendly peering climate in Europe, we are able to directly connect with hundreds of major networks over the largest Internet exchanges in Europe. This allows us to route up to 50% of our traffic over peering instead of transit, providing our customers with the shortest paths and lowest possible latencies to many of the largest European networks.

Singapore

Retrofitted: N/A
Transit: Telstra, PCCW, Tata (soon: NTT)
Peering: Singapore Equinix Internet Exchange

Tier 1 networks in the South Asian transit market are much more siloed than in other parts of the world, making it difficult to provide good connectivity using any single provider. This presented an unforeseen challenge when launching this facility in 2015: for the first time, we needed to be truly multihomed. With this realization, we decided to delay Singapore so that we could install our first pair of Cisco ASR 9000s as edge routers and launch on what would become our global public ASN. The facility we chose for Singapore is well connected, obviating the need to light dark fiber to remote carrier hotels.

Frankfurt

Retrofitted: N/A
Transit: Telia, Cogent (soon: GTT, Tata, Vodafone, Sparkle)
Peering: DE-CIX Frankfurt

Building on the experiences learned in Singapore, we now had a blueprint for all future Linode facilities. Choosing a colocation facility now considers how many quality networks are available to us, including in the colo facility itself. The Frankfurt datacenter is extremely well connected and we did not need to light dark fiber into remote carrier hotels.

Tokyo 2

Retrofitted: N/A
Transit: NTT, Tata, PCCW (more to come)
Peering: BBIX

We’re opening a brand new Tokyo facility next month. This will enable customers in the region to take advantage of several recent Linode announcements: KVM hypervisor, our latest plan specs including double the RAM and SSD servers, and mass availability. Stay tuned for the announcement within the next few weeks.

38 Responses

  1. Congrats everyone! This is an awesome accomplishment! Really exciting to see such a giant improvement in the network

  2. Linode rules!

  3. Congratulations and thank you Linode Team! You guys are awesome! 🙂

  4. Hey, we’re in Equinix, too! Hi neighbor 🙂

    Great job explaining all this.

  5. hello, congratulation on such great achievement.

    just had 1 question, having such massive capacity, is there any chance of increasing outbound limit in DO plans? thanks.

  6. Thank you, Stan! We’re doing our best and miss you tons.

  7. Hey, neighbor! We’re glad to hear that we may see you more often, I’ll bring pie over one day 🙂

  8. Thank you for the congrats!

    We don’t have any plans at the time, but maybe down the road some day.

  9. Thanks! You finally has some good news about the Tokyo 2 🙂

  10. Congrats guys! Do you guys plan on having a presence in Ashburn someday soon?

  11. Great job linode team!!

  12. Congrats guys!

  13. Congratulations! Good job!

  14. Linode are in Powergate, right?

  15. Congratulations! Good job!!!!
    It’s almost 2 years
    Tokyo datacenters love it waiting for it

  16. Wow. Up to 7 diverse providers in a single DC. Your BGP policies are going to be a nightmare. Good for you though.

  17. Linode continues to be exceptional. Thank you.

  18. I hope linode can provide more VPS of Japan region..always out of stock XD

  19. Would more providers be considered for the Tokyo 2 location? Specifically, NTT has consistently excellent latency to Mainland China, but NTT’s peering points with China Telecom and China Unicom tend to be extremely congested, causing ridiculous packet losses (>20%).

    Would it be possible to add KDDI (as in the current Tokyo location) and/or IIJ to the mix? They typically have much better throughput to Mainland China, and better connectivity to other “obscure” locations.

  20. I would like to reserve space in Tokyo 2.
    Is there enough space? Please consider migration time for existing users in Tokyo 1 before publicly open to new customer.
    How many times bigger than Tokyo 1?

  21. @Jack Linodes are located across each different datacenters around the world, soon to be nine. Did you mean any in particular?

  22. @Dex We plan to bring up a second Tokyo datacenter very soon, keep your eyes peeled!

  23. @ Lau Bun-sim

    Good question! We have a good blend of transit providers set for Tokyo 2, and we will explore adding more.

  24. @Buzz

    We aren’t taking reservations for Tokyo 2, but we will be accepting users for our Tokyo 2 beta very soon. Send a ticket over to the Support team to request access: linode.com/contact

    And in regards to your last question, there’s plenty of room in Tokyo 2 for our users.

  25. Take a transit from IIJ for Tokyo 2 would be really nice 🙂 Please consider if possible

  26. Soon for Tokyo 2 transit maybe: GTT

  27. Strongly wish Tokyo 2 have IIJ or other Chinese friendly transit with CN2 optimizations.
    Currently from my location (ChinaTelecom, Shanghai, China), NTT has 60% higher latency than IIJ, about 20% pocket loss and only about 1Mbps bandwidth.
    Please consider if possible.

  28. Finally, the Tokyo node will be upgraded….

  29. I notice that you’re not peering with Janet (AS786) – the UK’s academic internet provider. Please could you peer with them? (either over LINX or, better still, privately as I notice you’re both present in TeleCity Powergate)

    For bonus points, please also peer with AS12496 and AS42689. Thank you 🙂

  30. Awesome, I need a tokyo vps urgently

  31. NTT is not a good choice !!!

  32. @wdhwg001 , IIJ and KDDI are not available in Equinix Tokyo facilities, as far as I know.

    I guess available choices as additional upstream networks are, GTT, Vodafone (Asia/Global) and RETN.

    Linode alreadily announced NTT, Tata, and PCCW as upstream networks in the post above.

    Please see : https://www.peeringdb.com/fac/452

  33. @beautyleg , Please rest assured, NTT is constantly upgrading their backbone links to 100 Gbit/s Technology… Linode will have good blend of IP transit and peering links in Tokyo 2…

    Please keep in mind Linode always uses best-path routing except cases when a better latency or bandwidth availability can be observed through other circuits…

  34. @thanks you
    @mister anderson

    Okay. But IIJ has almost all aspects better than NTT when talking about accessing from China mainland. Maybe we can bear the high latency or trust the bandwidth upgrade promises, we still cannot take a 20%-25% pocket loss vps as a usable choice.

    I think I might feel sorry for the lack of IIJ and KDDI. Although Tokyo 1 has triple latency than IIJ’s 50ms latency, it’s still an option because it has only 0% – 2% pocket loss.

  35. I was a user for tokyo center(old data center),but as some reasons my machine now i haven’t .
    For recent days,some people say that maybe official will be opened Tokoy 1 again ,I am not sure ,so linode official staff can give us a reply?Very very expect for open Tokoy 1

  36. NTT is REALLY NOT FRIENDLY to China Telcom users!!! Hope you could add KDDI or IIJ as transit providers for Tokyo 2!

  37. Please join JPIX and JPNAP exchanges. Thanks!

  38. Do you have any kind of public roadmap for future features such as anycast?

Leave a Reply