Read­ing Time: 4 min­utes

Back from China

After rough­ly one week in Shang­hai, Chi­na to set up a new site on our cor­po­rate net­work, it is painful­ly appar­ent that I need to get back into this writ­ing busi­ness.  The dates on my posts belie my weak attempts at cov­er­ing up my lazi­ness.  That said, there were at least a cou­ple of things of note worth using up a few words on.

Note One (where it real­ly is some­one else’s fault)

Due to an unfor­tu­nate series of poor deci­sions, poor project man­age­ment, and a quite sud­den and unrea­son­able expec­ta­tion of deliv­ery dates, we [IT] were forced to poach some band­width from a sis­ter com­pa­ny of ours who had a slight excess in the man­u­fac­tur­ing facil­i­ty where our new office was to be set up.  By slight, I mean exact­ly 256kb.  For those of you not accus­tomed to see­ing the abbre­vi­a­tion for kilo, well, you’re too young.  More on that in a moment.

All of that aside, we engi­neered the cir­cuit all the way from the provider’s edge in Shang­hai, back to our facil­i­ties on the West Coast and ver­i­fied traf­fic was flow­ing.  Once we got to Shang­hai and hooked up our router and start­ed build­ing out the net­work behind it, we noticed that we could­n’t move any traf­fic at all.  With a quick extend­ed ping using the inside net­work inter­face as the source, as well as some trace-routes from oth­er places, we ver­i­fied that the provider had neglect­ed to put a route in the BGP tables for the new net­work.  Thank God for 24-hour NOC sup­port, and with­in 30 min­utes that prob­lem was resolved.

Note Two (where the author tries to check the turn-sig­nal flu­id)

As we moved on to cre­at­ing and join­ing up a shiny new R2 Read-only Domain Con­troller (RODC) every­thing went off the rails.  Time­outs galore.  DNS would­n’t resolve quick­ly enough to allow the new DC to join the for­est.  Off I go on a jol­ly search for default time­out set­tings, reg­istry tweaks, offline meth­ods to install a DC (ugly at best) and gen­er­al­ly going fur­ther down the rab­bit hole of com­plex­i­ty, even going so far as to direct anoth­er engi­neer work­ing for me to prep for a call to Microsoft (nev­er fun.)

Hav­ing already racked a lot of gear, we decid­ed to call it a night and come back fresh in the morn­ing.  I always find it help­ful to con­tem­plate prob­lems like this over a good sin­gle-malt Scotch.  So I did, a few times, and that led to morn­ing and a face-palm moment.  To wit:

In the cab on the way to the office the next day it occurred to me that I should check the secu­ri­ty on the fire­walls back home.  I knew I did­n’t put any ACLs on that new link, know­ing I’d be test­ing and I pre­fer to test in the absence of arti­fi­cial prob­lems, then crank down the screws once I’m con­fi­dent of the design.  I thought, how­ev­er, that I had over­looked a NAT exemp­tion or some­thing else and decid­ed to spend some qual­i­ty time check­ing that por­tion of the infra­struc­ture.

So, I got my cof­fee at the office (thank­ful­ly the office man­ag­er is a Kiwi who favors Star­bucks) and start­ed to look over the con­fig­u­ra­tion of the fire­walls.  Right there was my face-palm moment: an ACL which, in some secu­ri­ty-con­scious delu­sion I had put on the link in ques­tion, allow­ing ICMP traf­fic and deny­ing every­thing else.  GAAAAAH  Long sto­ry short, I changed the ACL and every­thing “mag­i­cal­ly” start­ed work­ing.

All I can say here is that it does­n’t mat­ter who you are, or how much expe­ri­ence with trou­bleshoot­ing you have, always start with the sim­ple stuff first.  Some of the best advice I ever got was from an instruc­tor of mine who was fond of say­ing “be the pack­et.”  By that he just meant that you have to start from the pack­et’s point of view and slow­ly work through every­thing that hap­pens to said pack­et from begin­ning to end.  Wiring, arp­ing, rout­ing, etc. what­ev­er.  Be the pack­et.

Also, don’t be afraid to admit mis­takes.  They will hap­pen and hope­ful­ly oth­er peo­ple around you can learn from them.  At least after they’re done laugh­ing.  Which some­times takes a while.

Note Three (hey you kids, get off my @#$ lawn)

I just want­ed to take a quick moment to address the inevitable ques­tions about our link at 256kb, and the mus­ings that I obvi­ous­ly must have meant mb instead.  Band­width is always seen as one of those more is bet­ter kind of things, and ignor­ing the temp­ta­tion to toss out the stan­dard bandwidth/delay screed here, let me just tell you that you can get by with less than you think. By the way, those of us who can remem­ber when a 2400 baud modem was blow-your-hair-back fast (mul­ti­ple lines of text at once!) tend to be more prag­mat­ic about these things.

On that link we cur­rent­ly have a domain con­troller run­ning, sev­er­al work­sta­tions with email and our main cor­po­rate ERP soft­ware.  We also, and this is what I real­ly like, have two 7900-series IP phones run­ning.  These are both homed to our main CUCM, Uni­ty and IPCC servers back in the U.S. and have excel­lent call qual­i­ty.  In fact, we test­ed a phone call (no local call­ing for these phones in Chi­na) from those phones in Shang­hai, through the CUCM in the U.S., and back to a U.S. homed cell phone in Shang­hai and got no dis­cern­able jit­ter.

Moral of the sto­ry?  You can get by with less band­width than you think.  Would I choose to?  Hell no!  🙂