Here’s the lowdown from Cisco.

End of Sale – September 30, 2011
Last Ship Date –  October 30, 2011
End of Routine Failure Analysis – November 31, 2011
End of Service Contract Date – January 31, 2012
End of Support – April 1, 2012

That’s a very aggressive schedule and doesn’t really make very much sense.  They just announced an ASA Service Module for it, so I’m really not sure what this is all about.  I hope details will become more apparent in the press release to be published tomorrow.

Send any mischief questions to me.

Aaron Conaway

I like to lean my head to the left, hit it with the palm of my right hand, and document what knowledge falls out.

Website - More Posts

In the environment, we have the usual data and voice VLANs on every switch port, and PCs are connected through the built-in switch in Avaya 4610SW IP phones.  The DHCP server is a Windows something-something-something server (maybe 2008) serving both VLANs.  When the phones boot, they come up on the data VLAN, get DHCP option 176 (remember it's Avaya and not Cisco), reboot onto the voice VLAN, and wait for an address; after a few seconds, they get the DHCP Ack Error displayed on the phone.  Thanks for that very descriptive error, Avaya.  The data VLAN was working fine, and, when I put my switch port into the voice VLAN natively, I got an address on my laptop.  It's just the phones.

A quick search showed a solution that I've seen a thousand times but keep forgetting (another in a long list).  If the DHCP server is in the data VLAN and is also tagged in the voice VLAN, the phones won't get an address.  Huh?  We didn't change the DHCP server.  Well, it turns out that the server was in the original non-PoE switch, and we had moved it to a port configured for a workstation (a fact I left out in the original edit of this post).  As soon as I took the switchport voice vlan X off of the server port, the phones started getting addresses, and we could finally hear the sweet sound of dial tone.  Problem solved.

According to this Avaya users group post, this happens because the DHCP server receives a DHCP request in a tagged packet; the server doesn't like it and NAKs it.  The fix works, but I'm not really satisfied with the answer of "it doesn't like it".  I may take a day and test this myself in my home lab.

Aaron Conaway

I like to lean my head to the left, hit it with the palm of my right hand, and document what knowledge falls out.

Website - More Posts

First of all, I drove the 2 1/2 hours from my home office to the headquarters to get some work done.  We have a large migration going on this weekend, and I didn't want the guys there to have to do all the hands-on work.  I planned to get some switches installed and do some cabling so the systems guys could just plug in when the servers landed in the data center.  None of that got done thanks to the rest of the karmic retribution.

Moral of the story:  Give yourself some additional time to do projects just in case something happens.

Over the past weekend and into the week, we ran into an issue with a 48-port GE module on a 6513.  It kept reloading, and a TAC engineer figured out that we were seeing a bug that would cause the module to reload every 12 hours (don't ask me why it was showing up after years in the same configuration).  Each time it went down, we'd lose around half of our servers.  Bear in mind that we have a pair of 6513s and that each server is cabled to each switch for redundancy.  We shouldn't have seen any servers go down; they should have simply flipped over to the other NIC and kept trucking.  As you probably figured out, the bonding/teaming configurations on some of them was either wrong or missing, so the $100k+ switch for redundancy was just a big paperweight.

Moral of the story:  Having redundant network gear is worthless if the servers can't handle a lost link.

When we got back from lunch, the module had rebooted again, and we lost all those servers and services.  This time, however, the module didn't come back.  We reloaded the module via software but we couldn't make any headway.  It would come up with a status of "Other" and then reboot again.  We ran down to the data center to reseat it, but that didn't help.  It absolutely refused to come up, so we called our TAC engineer and had him generate an RMA for us.

Moral of the story:  Modules die all the time.  You have to have a plan to deal with it.

We tried one more time to reset the module in software but we lost our SSH connection to the 6500.  Nothing but solid red lights on the supervisor.  The console didn't respond at all, so we wound up pulling power to reboot the whole switch.  It came back up, allowed us to log into the console, and then crashed.  It did this over and over until we physically removed the bad module.  This wasn't exactly an easy feat.  The switch isn't fully populated, but all the 48-port cards were right on top of one another.  This made the cabling very, very dense and hard to maneuver through.  The cables were neat and tidy, mind you, but it's hard to keep one bundle above the card without snagging it, and there is always the one cable that has to be different and not follow the rest of the structured cabling.  Of course, that one winds up cutting right across the module you need to get out.

Moral of the story:  Your switch may be modular, but each modules needs the supervisor.  Break the sup, and nothing works.
Moral #2:  If the cabling weighs more than you do, you may want to spread them out a bit.

We finally got the 6500 more-or-less stable, but there's are 48 ports missing.  Of course, all the servers are still down  The servers aren't configured to use their other NICs, and the RMA is 4 hours out; what do we do now?  I said we have to wait, but upper management declared an emergency and said that it needed to be back up as soon as possible.  Remember the switches I was going to install for the migration?  Yeah…those are installed as replacements now.  There goes those plans, eh?

Moral of the story:  If you can't wait until the RMA gets there, you better have spare parts on hand.

The replacement switches are actually a pair of 3750s stacked together.  As you know from my previous rants on them, I absolutely hate the 3750s.  These were the only ones we had, though, so we configured them and went into the data center to install.  That's when we noticed that there was no appropriate power in that rack.  The 6513s had power, but it was an L21-30 (or something similar; I didn't look at the plugs) which doesn't exactly help when you have standard connectors.  We found a power outlet in the floor a few feet down the way and wound up putting in a strip to get power into the rack.

Here's our Senior Network Engineer doing his part to help.  The pic pretty much sums up the whole day.  Upside down.  Cold.  Parts and cables all over the place.  He looks unconscious.

There was another problem, though.  There was no room to mount the new switches.  There was only room for the patch panels and the 6513 with a single rack unit of space at the bottom.  Luckily, we were able to finagle the switches into the bottom of the rack using that single U and the extra space on the floor.  Once powered up, we go the cabled plugged in and checked the systems that we could while our boss walked around to get status from the different departments.  Everything was finally fine.

Not really.  While we were dong our checks, someone sent an email saying that one of the major systems was down.  We just had a major outage, and all hands were in the data center helping where they could.  Do you really think anyone checked their email?  After about an hour and a half, we finally saw the problem reports and went back at it to figure out what was wrong.

Moral of the story:  Use a proper trouble reporting and escalation system.  Email is not it.
 Moral #2:  Don't be so inflexible in your infrastructure planning that a rack can only serve one function.  That includes power, cabling, and rack arrangement.

I totally take credit for this one.  It turns out that VTP had slapped us in the face.  I configured the switches and didn't even check the VTP configuration in the rush to get all the servers back kup.  There were other contributing factors, though, that were beyond me.  First of all, the 6513 that we trunked up to had a null VTP domain in transparent mode, but the other 6513 had a domain in server mode.  When I plugged in my 3750 stack (also in server mode), down went the VLANs.  If you know how VTP works, you know that the domains have to match in order to give it the opportunity to take down the network.  It turns out that the 3750s and the other 6513 (the healthy one) were configured with same VTP domain.  It's not our standard practice to use the same VTP domains everywhere, but, for some reason the other 6513 was configured with the VTP domain of another location. 

Moral of the story:  Go through your configuration checklist no matter what.  If I would have taken 5 more minutes in configuration, we would have saved over an hour of downtime.
Moral #2:  Just because two devices are supposed to be configured the same doesn't mean that they are.  Always check.
 Moral #3:  Be consistent in your configurations.  Most configurations should be predictable.

Finally, after about 6 hours or so, everything was up and running.  The replacement module arrived, but management said that we couldn't take another outage to do the replacement.  I can understand that they don't want any more downtime and wasted time, so I have no problem with that.  There's an old adage, though, that says that nothing is temporary.  I'm not expecting to get those 3750s out of that rack for quite a while.

Oh, yeah.  The 2 1/2 hour drive back home sucked.

Send any stiff drinks questions my way.

Aaron Conaway

I like to lean my head to the left, hit it with the palm of my right hand, and document what knowledge falls out.

Website - More Posts

Switchport Configuration

• switchport mode access:  This config makes the port an access port that carries the primary and voice VLAN traffic
• switchport mode trunk:  This config akes the port a trunk unconditionally, but it will still send DTP messages
• switchport nonegotiate:  This config keeps the port from sending DTP messages.
• switchport mode dynamic auto:  If the port receives DTP messages, it will become a trunk.  If not, it will be an access port.
• switchport mode dynamic desirable:  The port actively sends DTP messages trying to become a trunk.  This is the default configuration on a Cisco switch.

Cisco IP Phone Boot Process

1. Phone connects to an Ethernet switch and gets power if needed
2. Switch tells the phone the correct voice VLAN through CDP
3. Phone sends DHCP request for its voice VLAN
4. DHCP offer includes the TFTP server from which to download the config
5. Phone downloads the config from the TFTP server
6. Phone contacts the call processing server as dictated in the config file

DHCP Settings on a Cisco Router or L3 Switch

R1(config)#ip dhcp pool MYPOOL
R1(dhcp-config)#network 192.168.0.0 255.255.255.0
R1(dhcp-config)#default-router 192.168.0.1
R1(dhcp-config)#dns-server 192.168.0.10
R1(dhcp-config)#option 150 ip 192.168.0.20  <– Tells the phone to download the config from this TFTP server
R1(dhcp-config)#exit
R1(config)#ip dhcp excluded-address 192.168.0.1 192.168.0.100  <– Don't use these IPs when handing out DHCP

NTP

Why should you use NTP for a CME setup?

• Phones display correct time
• Voicemails have the correct time
• CDRs are timestamped accurately
• Router logs are timestamped accurately
• Time-based access worked predictably

R1(config)#ntp server 1.1.1.1
R1(config)#clock timezone MYTZ -5  <– Sets the timezone to a zone called MYTZ that's 5 hours behind UTC

Aaron Conaway

I like to lean my head to the left, hit it with the palm of my right hand, and document what knowledge falls out.

Website - More Posts

That’s an interface descriptor block.  I admit that I’m not intimately familiar with them, bu they’re data structs in IOS used to keep track of the interfaces on that device.  They come in two flavors – hardware and software.  HWIDBs usually represent a physical interface but they also represent tunnels, SVIs, PortChannels, subinterfaces, and any other virtual interface that you can configure.  The SWIDBs represent the layer-2 encapsulation of each HWIDB, so you’ll see entries talking about Ethernet, HDLC, PPP, etc.  That means that every interface you have on a router consumes two IDBs (there are always exceptions).  That’s important because each platform and IOS version combination has a limit to the number IDBs that device supports.

If you check out one of Cisco’s pages on IDBs, you’ll see a pretty table showing the limits.  The 3640 running 12.4(25b) that I run in my GNS3 lab has a limit of 800 IDBs.  That means that I can have 400 interfaces configured at most.  That little 800 series router running 12.1T that you still have running at the VP’s house has an IDB limit of 300 or 150 interfaces.  The 7200 in the data center running 12.3 can handle 20,000 IDBS or 10,000 interfaces!

If you guessed that you can see your IDBs by typing show idb, then you guessed right.  That will show you the IDB limit, how many are being used, a summary table, and a list of all the IDBs with their details.  Remember that there may be more interfaces on your device that just physical.  You may have an SVI, loopback interface, or even a null or two.  These all count towards the limit.

Before you get freaked out and start checking the IDB limits on all your devices, take a breath.  I’ve never run into the IDB limit on any device and I’ve never heard of anyone who has.  I’m sure someone has, but I don’t remember hearing about any.  Think about it for a second.  If I took my 3640 and filled it with 4 NM-16ESWs, I’d only have 128 IDBs used (16 ports * 4 modules * 2 IDBs for each port).  Don’t forget the null interface and VLAN 1 SVI by default (VLANs take 1; VLAN SVIs take 2 each).  That brings the count to 133.  Let’s add 100 more VLANs and SVIs on this guy.  Now we’re up to 433.  How about we put each interface into a channel group of its own.  That adds another 128, which is 561.  Only 239 more to go.

Unless you’re doing something out of the ordinary, I don’t think the IDB limit will be a problem.  Of course, that depends on your definition of “ordinary”.

Send any sort indexes questions my way.

Aaron Conaway

I like to lean my head to the left, hit it with the palm of my right hand, and document what knowledge falls out.

Website - More Posts

I’m guessing here, but we have about 50 3750 stacks in the enterprise.  Most of them are pairs, you wind up with roughly 120 switches.  Since we’ve done about 20 replacements over the last 5 years, that means we have a 17% failure rate.  That’s pretty horrible, isn’t it?

For the most part and with few (if any) exception, we use the 3750s as aggregation points for our access switches.  We don’t do QoS on them.  We don’t do any access control on them.  We don’t even do routing on them.  They’re simply used to connect all the access switches in the closet to the core, so they’re not doing anything funky or burdensome.  The CPU and memory are always well within normal operating parameters.  They just fail and fail repeatedly.

The flies started dropping in closets at our corporate headquarters a few years ago.  It was the middle of summer, and the temperatures kept rising to over 90F (32C) until the we lost 3 switches in 3 weeks.  If you could stand to make it into the closet, you could feel that the sheet metal of the switches was hot enough to make you pull your hand back!  When the facilities team added more cooling, the temperatures dropped to around 82F there (28C), but we continued losing switches.  I figured the newly-failed switches were feeling the effects of the earlier heat wave and were just getting around to giving up the ghost.  Surely the heat was the culprit.

A few months after our headquarters meltdown, a tech for a satellite office called and asked if we could help with some latency issues.  He showed me the switch stacks throughout the building, and I noticed that only one of the 10 switches actually had a label.  The tech said that he never got around to relabeling them after they were replaced.  Some, he said, had been replaced multiple times.  The closets were running about 76F (24C), so heat didn’t seem to be the problem at this location.  The closets were clean as a whistle, and everything in the racks was on building UPS.  I couldn’t find a pattern at all.  For the record, all their latency issues were related to two unrelated 3750s.  Two RMAs later, and their problems were gone.

I’ve been trying to find patterns for the failures, but I can’t think of any.  If it’s heat, humidity, power, dust, etc., then why are we not replacing 2950s as well?  There are 4-10 of them for every 3750s stack we have.  We’re replacing them, but it’s a rate of less than 1%.  If it is environment, then the 2950s are English hooligans compared to the 3750s being French aristocracy.  Maybe it’s sabotage.  I still don’t know after years of watching RMA after RMA come in.

I have noticed one pattern, though.  The only deployments of 3750s that have never had a problem are in data centers.  They seem to love any room that has an ambient temperature of 62F (16C) with less than 40% humidity and large volumes of air flow.  If only we could install micro-data centers in all our closets, then I would be a happy network dude.

Send any wooden shoes questions my way.

Edit:  I went back and checked our TAC cases to see what switches we actually replaced.  It turns out that we’ve done 19 replacements, and they’ve all been 3750G-12S-S switches.

Aaron Conaway

I like to lean my head to the left, hit it with the palm of my right hand, and document what knowledge falls out.

Website - More Posts

I had a VTP server and client in the same VTP domain, and, when I cabled up the trunk, the client overwrote the VLAN database on the server.

The moral of the story is that the best revision number will win no matter what the operating mode of the switch.

Aaron Conaway

I like to lean my head to the left, hit it with the palm of my right hand, and document what knowledge falls out.

Website - More Posts

In my mind, the CCNP is a technical certification, so I expect to be tested on technical topics.  Are there topics beyond technology that P-levels should know?  Of course there are, but I really don’t think whole chunks of the test should be about a preparation plan and rollback procedures.  The BCMSN had a lot more technical questions at a much higher level of expertise; it seems much better suited to the CCNP track than the SWITCH test did.

I was really surprised at how many questions today were repeats from the SWITCH test last week.  Of the three lab exercises I worked, two of them were exactly the same as last week.  I would venture to guess that there were also 8 to 10 repeated multiple choice questions.  It seems that this is going against my argument of being more technical, though, doesn’t it?  If you mix in the remaining questions that were at a much higher technical level, you wind up with a pretty darn good test.

I’ve really got nothing more to say about the BCMSN.  It’s a good test with an appropriate level of technical (and paper-pushing) detail.  I’m very glad I was able to take it before the 31 July 2010 deadline, and I advise anyone who needs the SWITCH test to try and do the same.

The next stop is ROUTE (642-902) for me.  I’m taking a class on that one soon, so I’m confident I can pass it in the next 11 weeks we have left until the deadline.

Audio commentary

Aaron Conaway

I like to lean my head to the left, hit it with the palm of my right hand, and document what knowledge falls out.

Website - More Posts

The test was the absolute worst I’ve ever taken.  I know that I complain a lot, but this is totally justified in my eyes.  My 4th grade spelling tests were better than this.  I’ve seen kindergarten plays with better production value.

First of all, it was poorly written.  Whoever wrote those questions has a few pieces of information about English sentence structure missing from their skill set.  A sentence needs a verb, right?  Well, a lot of the sentences were missing those.  It’s kind of important to know what the whole point of the sentence is, or is that too much to ask?  The “drag this over here” exercise questions all started with the same 13-word phrase that left the question so long that it was unreadable.  A couple of commas would have been nice in some.  Others I just had to infer from the answers what they were trying to ask.

There were lots of spelling errors as well.  Most of them were just stupid stuff like switched letters or missing characters, but, at one point, I had to figure out that I needed to look at the “router” instead of the “route”.  That’s not really cool.  The misspellings were so bad that they were actually misspelling the hostnames on the diagrams provided.  Does anyone even try any more?

Let’s talk about the technical level of the test.  If I didn’t know any better, I would swear I was taking a CCNA test.  The technical material was so elementary that it bordered on comical.  If I recall correctly (which I never do), there were about 3 questions on trunking which were so easy that my wife could answer them.  There were about 4 FHRP questions that were out of the “Cisco for Dummies” book.  I could go on, but I have better things about which to complain.

“So,” you might ask, “why did you fail it if it was so easy?”  That’s a great question.  I failed it because the name of the test is misleading.  When Cisco says “Implementing Cisco IP Switched Networks”, they really mean “Collecting Documentation About VLANs.”  There were at least four questions on this test that asked what information you need to collect before implementing some unknown step of a project involving VLANs.  Sometimes, the reference was to rollback plans.  Sometimes it discussed IP assignments.  Sometimes it even talked about collecting user requirements.  It seemed that nearly half of the questions on the test discussed planning for making changes or preparing change documentation.  There was very little “implementing.”

To top it all off, too, one of my labs froze.  I entered a command into a router, and it didn’t come back.  I couldn’t change to the other lab windows, either (the “Scenario” or “Topology” windows included), but my timer kept ticking.  I could click around in the testing software, but the lab itself was toast.  I got the administrator who helped me out a bit after the machine was rebooted.  I didn’t run out of time or anything, but getting up to find help to troubleshoot a problem really throws you off.

How about some closing words?  First of all, I have given up on the Cisco Press books and other materials.  Each time I use them they have little to no coverage about topics on the test itself.  The ISCW was that way, and we all know about my problems with the ONT.  I figured that those were just aged text, but SWITCH is only a month or two old, isn’t it?  That means the test hasn’t had that much time to change, but the materials are totally different already.

I actually have an example of the books leading the reader directly away from the test materials.  I’m reading from the “CCNP SWITCH 642-813 Quick Reference” book by Donohue.  On page 8, it discusses the PPDIOO lifecycle approach.

Network engineers at the CCNP level will likely be involved at the implementation and following phases.  They can also participate in the design phase.

That doesn’t make any sense, does it?  Didn’t I just say that there were a good number of questions on preparation (the first P) and planning (the second P).  Both of those come before the design phase.

Somebody help me out here.  What am I missing?  Is there some magical book series that has the answers?

I should have bought testing vouchers in bulk when they were $150.

Audio commentary

UPDATE:  It seems that the idea of seeing topics on the exam that aren’t are the test go beyond just me.  I’m getting in touch with as many people related to the SWITCH book as I can to let them know that this is a serious problem.  I’m sure I’ll have a post or two on the outcome of that effort.

Aaron Conaway

I like to lean my head to the left, hit it with the palm of my right hand, and document what knowledge falls out.

Website - More Posts

SW3#sh span | incl 0/2[34]
Fa0/23           Altn BLK 3019      128.23   P2p
Fa0/24           Root FWD 3019      128.24   P2p

Now let’s unplug F0/24 and see what happens.

19:05:05: STP FAST: UPLINKFAST: make_forwarding on VLAN0001 FastEthernet0/23 roo
t port id new: 128.23 prev: 128.24

19:05:05: %SPANTREE_FAST-7-PORT_FWD_UPLINK: VLAN0001 FastEthernet0/23 moved to Forwarding (UplinkFast).
19:05:05: STP: UFAST: removing prev root port Fa0/24 VLAN0001 port-id 8018
SW3#
19:05:06: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/24, changed state to down
SW3#
19:05:07: %LINK-3-UPDOWN: Interface FastEthernet0/24, changed state to down

Before the switch even reports that F0/24 is down, F0/23 is brought into the forwarding state. Now let’s plug F0/24 back in.

19:07:16: %LINK-3-UPDOWN: Interface FastEthernet0/24, changed state to up
SW3#
19:07:17: STP FAST: make_forwarding: via UPLINKFAST: NOT: port FastEthernet0/23
VLAN0001 is: uplink enabled new root FastEthernet0/23 (me)prev root exists(8018/) cur state forwarding role uplink
19:07:17: STP FAST: make_forwarding: via UPLINKFAST: NOT: port FastEthernet0/24
VLAN0001 is: uplink enabled new root FastEthernet0/23 (not me)prev root exists(8018/) cur state blocking role looped
19:07:18: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/24, changed state to up
SW3#
19:07:18: STP FAST: make_forwarding: via UPLINKFAST: NOT: port FastEthernet0/23
VLAN0001 is: uplink enabled new root FastEthernet0/23 (me)prev root exists(8018/) cur state forwarding role uplink
SW3#sh span | incl 0/2[34]
Fa0/23           Root FWD 3019      128.23   P2p
Fa0/24           Altn BLK 3019      128.24   P2p

Notice that the port comes back up, but it isn’t returned as the root port immediately. It should be, though, right? The original STP convergence said that it was the closest to the root bridge, so it makes sense that it should be the root port again, right? Since the port just came up, STP still has to make sure there’s no loop, so it has to step through all the states like any good port does. If we wait a few more seconds, we see this.

19:07:53: STP FAST: UPLINKFAST: make_forwarding on VLAN0001 FastEthernet0/24 root port id new: 128.24 prev: 128.23

19:07:53: %SPANTREE_FAST-7-PORT_FWD_UPLINK: VLAN0001 FastEthernet0/24 moved to Forwarding (UplinkFast).

SW3#sh span | incl 0/2[34]
Fa0/23           Altn BLK 3019      128.23   P2p
Fa0/24           Root FWD 3019      128.24   P2p

Now we’re back to where we were originally. The moral of the story is that UplinkFast already knew the status of both ports, so it could quickly move the blocked port to fowarding when the port failed. Traditional STP would have to send a TCN message to the root bridge, which would then forward them out with the rest of the switches so they can reconverge. UplinkFast skips the whole reconverging thing.

Send any questions my way.

Aaron Conaway

I like to lean my head to the left, hit it with the palm of my right hand, and document what knowledge falls out.

Website - More Posts