Sunday, March 05, 2006

Of Fibre Terminations & Inter-Switch Trunks

It's been a long, long weekend of work. It all started yesterday with our relocating about 48 pairs of fibre optics from one building to another building next door. Boring, but very time consuming stuff. After many hiccups we got it all up and running at about 10AM this morning. But that's not the interesting bit, and I'm well over it, so I'm not going to mention it again ;-)

What is cool is what we did this afternoon. We set up our first inter-switch trunk (IST). We have two Nortel ERS8600 routing switches situated about a kilometre appart (one in each of our data centres). As of about 4PM this afternoon, they're one logical switch.

The way this works is essentially VRRP, but with a few subtle differences. For each VLAN that we set up a VRRP gateway IP address, the two switches load balance by being the VRRP "master" for any traffic entering the IST through them. This is best explained with the following traceroutes:

kimbly# netstat -rn | grep default
default            146.231.160.1      UGS         0    15904    tx0
kimbly# traceroute hippo.ru.ac.za
traceroute to hippo.ru.ac.za (146.231.128.1), 64 hops max, 40 byte packets
 1  catering-amm.gw.ru.ac.za (146.231.160.3)  0.601 ms  0.424 ms  0.412 ms
 2  struben.core.ru.ac.za (146.231.0.17)  0.623 ms  0.524 ms  0.476 ms
 3  hippo.ru.ac.za (146.231.128.1)  0.381 ms  0.307 ms  0.324 ms
kimbly# traceroute nat.amm.ru.ac.za
traceroute to nat.amm.ru.ac.za (146.231.48.34), 64 hops max, 40 byte packets
 1  catering-amm.gw.ru.ac.za (146.231.160.3)  0.601 ms  0.424 ms  0.412 ms
 2  nat.amm.ru.ac.za (146.231.48.34)  0.404 ms  0.357 ms  0.417 ms

drostdy# netstat -rn | grep default
default            146.231.160.1      UGS         0    11895    rl0
drostdy# traceroute nat.amm.ru.ac.za
traceroute to nat.amm.ru.ac.za (146.231.48.34), 64 hops max, 40 byte packets
 1  catering-struben.gw.ru.ac.za (146.231.160.2)  0.779 ms  0.584 ms  0.556 ms
 2  amm.core.ru.ac.za (146.231.0.18)  0.716 ms  0.557 ms  0.528 ms
 3  nat.amm.ru.ac.za (146.231.48.34)  0.404 ms  0.357 ms  0.417 ms
drostdy# traceroute hippo.ru.ac.za
traceroute to hippo.ru.ac.za (146.231.128.1), 64 hops max, 40 byte packets
 1  catering-struben.gw.ru.ac.za (146.231.160.2)  0.779 ms  0.584 ms  0.556 ms
 2  hippo.ru.ac.za (146.231.128.1)  0.381 ms  0.307 ms  0.324 ms

For those who don't get it, both machines have a common default gateway (146.231.160.1). kimbly and nat.amm are connected to amm.core.ru.ac.za, drostdy and hippo are connected to struben.core. The first hop on each traceroute shows the non-VRRP IP address of the switch doing the routing — in other words, the two traceroutes are routed by two different switches, even though they have the same IP as next-hop.

That's just cool. And I'm tired. :)

posted by guy at: 20:17 SAST | path: /systems | permanent link