13
Oct
14

The tyranny of distance, or why webscale in Australia is tough

While trying to understand some site performance observations with a guru from WordPress VIP Mr Barry Abrahamson I was reminded just how odd routes from Australia to the USA can be sometimes.

In my case from a Telstra ISP 4G service in Sydney:

  • wp.com resolved to Australia,Brisbane, via Sydney Kent St exchange about 2km from my location in inner Sydney, an odd little route given EdgeCast has a POP in Sydney !
  • wordpress.com routed to hong kong to bay area and the landings near LA
  • sat.wordpress.com routed to texas, san antonio via dallas peering which came from Hong Kong –> Taiwan then jumped the Pacific, seemingly not the most direct route 🙂

3 distinct locations and routes for one web page.

I see these odd routes a lot, so perhaps a little look at how to trace and decode the output is in order.

First thing you need is a machine that can run traceroute, ping and curl – I will use my trusty mac pro. Secondly you need a connection that is not excessively firewalled – many corporate environments block these useful tools, and some ISPs and home firewalls might too. If you have a firewall at home you might need to allow ICMP.

Ok so first we ping each host:

ping -c3 -n i0.wp.com
PING cs82.wac.edgecastcdn.net (68.232.44.111): 56 data bytes
64 bytes from 68.232.44.111: icmp_seq=0 ttl=57 time=78.895 ms
64 bytes from 68.232.44.111: icmp_seq=1 ttl=57 time=76.929 ms
64 bytes from 68.232.44.111: icmp_seq=2 ttl=57 time=137.781 ms

ping -c3 -n wordpress.com
PING wordpress.com (192.0.78.17): 56 data bytes
64 bytes from 192.0.78.17: icmp_seq=0 ttl=49 time=613.950 ms
64 bytes from 192.0.78.17: icmp_seq=1 ttl=49 time=556.325 ms
64 bytes from 192.0.78.17: icmp_seq=2 ttl=49 time=762.889 ms

ping -c3 -n sat.wordpress.com
PING sat.wordpress.com (76.74.254.120): 56 data bytes
64 bytes from 76.74.254.120: icmp_seq=0 ttl=45 time=963.449 ms
64 bytes from 76.74.254.120: icmp_seq=1 ttl=45 time=572.970 ms
64 bytes from 76.74.254.120: icmp_seq=2 ttl=45 time=805.222 ms

OK from this I can clearly see that i0.wp.com is somewhere fairly local, while wordpress.com is “over the pond”. Also looks like sat.wordpress.com might be father away than wordpress.com, interesting. Running ping a couple more times (I usually do it at least 3 times) will help see any other anomalies.

so now to the big guns – traceroute:

traceroute -I i0.wp.com
traceroute to cs82.wac.edgecastcdn.net (68.232.44.111), 64 hops max, 72 byte packets
1  172.20.10.1 (172.20.10.1)  28.801 ms  24.031 ms  1.678 ms
2  * * *
3  * * *
4  * * *
5  tengige0-5-0-15.chw-edge901.sydney.telstra.net (139.130.111.73)  170.490 ms  30.207 ms  28.705 ms
6  bundle-ether2.ken-edge901.sydney.telstra.net (203.50.11.102)  32.643 ms  29.309 ms  32.602 ms
7  ver1542775.lnk.telstra.net (139.130.197.82)  27.004 ms  24.701 ms  31.644 ms
8  68.232.44.111 (68.232.44.111)  28.148 ms  28.552 ms  30.501 ms

mdsl026351:~ macadmin$ traceroute -I wordpress.com
traceroute: Warning: wordpress.com has multiple addresses; using 192.0.78.9
traceroute to wordpress.com (192.0.78.9), 64 hops max, 72 byte packets
1  172.20.10.1 (172.20.10.1)  1.911 ms  1.460 ms  1.406 ms
2  * * *
3  * * *
4  * * *
5  tengige0-5-0-15.chw-edge901.sydney.telstra.net (139.130.111.73)  1253.110 ms  111.398 ms  24.262 ms
6  bundle-ether13.chw-core10.sydney.telstra.net (203.50.11.98)  32.387 ms  29.947 ms  40.252 ms
7  bundle-ether17.oxf-gw2.sydney.telstra.net (203.50.13.70)  31.405 ms  33.614 ms  38.195 ms
8  bundle-ether1.sydo-core01.sydney.reach.com (203.50.13.38)  42.390 ms  36.117 ms  31.180 ms
9  i-0-2-0-5.sydo-core02.bi.telstraglobal.net (202.84.223.42)  30.383 ms  31.534 ms  32.370 ms
10  i-0-2-0-0.1wlt-core01.bx.telstraglobal.net (202.84.141.146)  202.327 ms  201.500 ms  523.056 ms
11  i-0-5-0-3.tlot02.bi.telstraglobal.net (202.84.253.46)  603.329 ms  346.085 ms  577.464 ms
12  xe-11-2-0.edge1.losangeles6.level3.net (4.68.62.9)  613.410 ms  614.335 ms  613.184 ms
13  ae-4-90.edge2.losangeles9.level3.net (4.69.144.207)  307.141 ms  922.118 ms  920.883 ms
14  peer-1-netw.edge2.losangeles9.level3.net (4.53.230.6)  921.686 ms  607.118 ms  466.507 ms
15  * * *
16  192.0.78.9 (192.0.78.9)  812.169 ms  613.593 ms  614.946 ms

mdsl026351:~ macadmin$ traceroute -I sat.wordpress.com
traceroute: Warning: sat.wordpress.com has multiple addresses; using 76.74.254.123
traceroute to sat.wordpress.com (76.74.254.123), 64 hops max, 72 byte packets
1  172.20.10.1 (172.20.10.1)  24.285 ms  5.433 ms  5.907 ms
2  * * *
3  * * *
4  * * *
5  tengige0-5-0-15.chw-edge901.sydney.telstra.net (139.130.111.73)  169.809 ms  29.662 ms  29.179 ms
6  bundle-ether13.chw-core10.sydney.telstra.net (203.50.11.98)  30.460 ms  31.108 ms  29.384 ms
7  bundle-ether17.oxf-gw2.sydney.telstra.net (203.50.13.70)  32.356 ms  40.007 ms  40.267 ms
8  bundle-ether1.sydo-core01.sydney.reach.com (203.50.13.38)  36.161 ms  32.148 ms  41.099 ms
9  i-0-2-0-5.sydo-core02.bi.telstraglobal.net (202.84.223.42)  39.351 ms  39.832 ms  40.266 ms
10  i-0-4-0-5.1wlt-core01.bx.telstraglobal.net (202.84.140.102)  394.816 ms  258.567 ms  2206.390 ms
11  i-0-0-0-3.tlot02.bi.telstraglobal.net (202.84.251.189)  575.466 ms  614.253 ms  614.506 ms
12  gtt-peer.tlot02.pr.telstraglobal.net (134.159.61.98)  614.388 ms  615.384 ms  613.874 ms
13  xe-0-0-0.dal33.ip4.gtt.net (89.149.187.94)  306.713 ms  921.466 ms  642.280 ms
14  peer1-gw.ip4.gtt.net (77.67.71.30)  584.913 ms  700.331 ms  614.389 ms
15  * * *
16  * * *
17   (76.74.254.123)  718.998 ms  477.185 ms  442.695 ms

OK so let pick this apart:

the first thing to know is that all network engineers love order, and the naming of router nodes is easy to decode once you know how. Each line in a traceroute is a router, and each router is hit with an ICMP request that says (in effect) “name thy self”. Some say “go away!” (***). The time this “ping” takes is measured, and this helps judge the accumulated “cable distance” – you will note a general trend for time to go up for each step.

i0.wp.com:

  • clearly the closest
  • gotta love Telstra – very good node naming! from Sydney we go to …
  • some boundary node at 139.130.197.82 then to 68.232.44.111
  • a quick squiz in a tool like info sniper or maxmind will allow you to infer this seems to be in brisbane and is EdgeCast. NO I hear you cry! 68.232.44.111 is in the USA! Well, no. The address range is assigned in the USA but the ping surely tells us it it in nearer. How can this be! Easy – EdgeCast uses AnyCast BGP, which in layman’s terms means that it’s ip addresses are in many places at once. The brisbane former hop is the giveaway – no route from Australia gets to the USA in one hop (more on this in a sec)

wordress.com and sat.worpress.com:

  • clearly over the pond, both go to Hong Kong seeming with out any other hops first (more on this later)
  • sat then goes to Texas via Taiwan again with little in between (!)
  • the main domain routes to LA, straight from Hong Kong

So some odd things are:

  1. why Taiwan to Texas direct?
  2. how does one get from Australia to HongKong direct ?

The answer to both these question is the joys of optical networking, specifically ADM (Add Drop Mux). While there is no actual unbroken path between Australia and Hong Kong (Guam is in the middle) or Taiwan and Texas (the Pacific and Several States of the Union are in the way), there are high capacity “trails” (leased circuits) – digitally fractionated parts of massive optical under sea and land based trunks that effectively bridge the distance between those points at close to the speed of light.

The internet is stranger and more beautiful than most appreciate 🙂

Onwards and upwards 🙂

Advertisements

0 Responses to “The tyranny of distance, or why webscale in Australia is tough”



  1. Leave a Comment

What do you think?

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s


%d bloggers like this: