Solving Security Challenges with Cisco IOS Embedded Events Manager: this session is cool for many reasons.  The presenter made videos demonstrating the creation and testing of several scripts for different purposes.  I have only ever used IOS EEM for one thing and it certainly wasn't my first choice, but it worked so well, I am very interested in expanding its use in my customer networks.  Some specific examples included bouncing an interface that was "stuck" after a special kind of attack or creating an email alert after interface utilization exceeds a pre-defined threshold.

FCoE - Design, Operations, and Managerment Best Practices: Since I am actively involved with designing a large datacenter, I thought this would be a good session to attend.  I attended many SAN classes last year, but just didn't really "get it".  The FC review at the beginning of this session really helped alot.  We covered alot of the architectural options available today, as well a those that will be available when FCoE multi-how is supported.

Advanced IEEE 802.1x Design and Troubleshooting: While I have deployed 802.1x several times, I thought it would be beneficial to see what is new out there.  They have several updated ideas.  802.1x Monitor Mode is a great way to evaluate how prepared an organization is for wired 802.1x.

Of course the customer appriciation event is a very popular destination.  We had several cover bands playing songs from a wide range of artists with Tina Turner, Garth Brooks and of course Elvis impersonators.  Smash Mouth was definately the star of the show.  Everything was high energy.  I got to spend some time with very intersting people talking, learning, and partying.  iPhone picture from the show with this years hat (orange Elvis hair with integrated glasses) will be posted as soon as I get around to syncing the phone.

Overlay Virtual Transport: This session covered the new L2 transport technologies for datacenters.  This technology is going to become very popular due to the L2 access requirements for VMotion across physical data center locations.

Keynote: John Chambers: John is a remarkable speaker and a visionary.  In some ways this keynote was quite similar to previous ones, however it helps highly the continuity of Cisco's long term vision.  John emphasized the speed at which Cisco is attempting to innovate and build/expand technologies.  Cisco also unvieled the new Cius - Cisco's tablet based answer to the iPad with mobile telepresence and two high definition (720p) cameras, 3G/4G wireless and a docking station that doubles as a Cisco IP phone. <- I want one :)

Introduction to Cisco Layer 2 Multipathing (L2MP): How many people out there love to hate Spanning-tree?  If you don't, you probably should.  What if you could change your blocked spanning tree ports into multiple active load balanced paths and never have to see spanning-tree again?  Well you can... Soon.  Trill (future standard), L2MP or FabricPath (new name) will become the way we handle redundant L2 paths in the future, at least on Nexus, once the new hardware that can support the appropriate tags has been released.

Cisco Trusted Security & Security Group Tagging: This session covered TrustSec and some of the features around making this happen.  Unfortunately I had to leave before the class was finished, so I hope to post more later after I finish reviewing the recorded content.

CCIE/CCDE NetVet Reception: John Chambers met with the CCIE/CCDE NetVets (those attended 3 or more of the previous 5 networkers events) and answered questions and accepted comments from the gathered audience.  Answers were very candid (some subject to NDA, but I forget which so I must be vague).  Questions included supply chain issues, Certification - related questions (Value of CCDE/CCA, Possible master level implementation certification), Network Management, UCX, HP/IBM partner status, etc. 

CCIE Party: Event at the VooDoo lounge on the 51st floor of the Rio Hotel.  The local signature drink, the Witch Doctor, was flowing quite readily.  Kicking party with a live band, and plenty of geeks, well lubricated, networking.  Always a good event.

Hangover - fortunately missing, but we will see how that goes after the Customer Appriciation event Wednesday night.  I am not sure that IOS EEM was the best class for the morning after events like these, but such is life!

The thing I miss the most is the USB flash drives Cisco gave out last year.  These had a wealth of information and participants didn't have to go and find it and waste precious bandwidth downloading it.  Only 13 minutes left to get the last 40% of this 100 MB file!

Wednesday, May 13th, 6:30 - 8:30 PM

LOCATION:
CALENCE
1130 Rutherford Lane

Suite 208

Austin, TX 78753

TOPIC:
Multi-Protocol Label Switching

This month we the user group will discuss Multi-Protocol Label Switching (MPLS) technology and configuration.  Ryan Hicks will be presenting MPLS fundamentals from both the carrier and customer perspective.  A MPLS core will be created to demonstrate the technology.  Virtual router labs will be provided, so please bring your laptop!  The following are some of the specific features/tasks that will be discussed:

•          Overview of carrier switching technologies (MPLS, ATM, Frame-Relay, etc)

•          Understanding of VRFs

•          MPLS PE and P router configuration (Provider)

•          MPLS CE router configuration (Customer)

There is no charge to attend and refreshments will be served. Meetings are open to anyone interested in discussing the network engineering industry in a vendor-neutral, education-focused environment.

We are a local community of network engineer professionals who come together to network and learn about engineer-related topics in a vendor-neutral, education-focused environment.

Meetings will be held the second Wednesday of every month.

If you know someone who would be interested in attending this event, please feel free to pass on this information.

We started a discussion for future topics.  It looks like we will be working through configuration and labs on Cisco's Call Manager Express product during the next two sessions.  Please make sure to bring your laptops for these, as we are going to have virtual routers set up in individual pods.  We will also have a limited number of physical FXO/FXS ports and phones available for testing.

After these two sessions we have options available for the next one.  There has been discussions on having a troubleshooting session which I am thinking might be a Layer-2 troubleshooting primer, involving spanning-tree and other switch technologies.  Other possibles are having Solar Winds demo/train on some of thier management tools or having Cisco back out to talk about WAAS or ACE.  Please send me your feedback on what you want for the next session.

We started noticing that the default route advertised by the headquarters through the MPLS cloud was flapping.  Turns out that the DS3 at the headquarters was the reason for the route flap.  The DS3 started bouncing up and down frequently.  We decided to back out the changes and remain on the original circuit until we cound determin the cause.  looking at the DS3 controller yeilded output like this:

   Framing is c-bit, Clock Source is Line
   Bandwidth limit is 44210, DSU mode 0, Cable length is 10
   rx FEBE since last clear counter 33743, since reset 67688179
   Data in current interval (297 seconds elapsed):
     0 Line Code Violations, 0 P-bit Coding Violation
     0 C-bit Coding Violation
     0 P-bit Err Secs, 0 P-bit Sev Err Secs
     0 Sev Err Framing Secs, 45 Unavailable Secs
     0 Line Errored Secs, 0 C-bit Errored Secs, 0 C-bit Sev Err Secs
  Data in Interval 1:
     0 Line Code Violations, 0 P-bit Coding Violation
     2 C-bit Coding Violation
     0 P-bit Err Secs, 0 P-bit Sev Err Secs
     0 Sev Err Framing Secs, 248 Unavailable Secs
     1 Line Errored Secs, 1 C-bit Errored Secs, 0 C-bit Sev Err Secs

The only problem is the telco can't seem to find a problem.  The most obvious thing to do here is validate linecode configurataions throughout the path of the circuit.  Unfortunately if this was the problem the circuit wouldn't be anywhere near as stable as it is.  We did the check anyway, but it was unhelpful.  The IXC decided to loop the circuit at the customer end and run patters with a T-Berd tester.  Typically testing circuits with a T-Berd yields usable results.  In this case everything came back clean. 

"It is an old maxim of mine that when you have excluded the impossible, whatever remains, however improbable, must be the truth." (Doyle, 1892)  It is impossible for the circuit to not work, yet have nothing wrong.  Therefore, either there is a bug in the router IOS, a hardwaare defect/failure, or the T-Berd's resultss are wrong.  So, to eliminate the first two possibilities, we disconnect the T3 at the DMARC and create a hard loop.  A neat trick you can do with virtually any circuit that can be looped is to issue the no keepalives command and ping the interfaces own address.  No keepalives turns off the Frame-relay LMI, HDLC or other frames, and forces the interfaces to believe they are up.  I don't have pattern testing abilities on this hardware, so a couple of pings with different patterns should do the trick!  Good news is, the pings all work, bad news is... one of the tests fails when the IXC has the circuit looped!

I suppose I should back up...  When doing ping tests, you should hit certain types of patterns.  Circuits have issues with bit syncronization, which is the reason for the different kinda of line coding in use on T1 and T3 circuits.  If changes in the bit pattern don't happen every so often the PLL circuit that establishes clock syncronization with the carrier.  Unfortuntely, user data can't be relied on to change 0s to 1s often enough.  To test for problems, certain patterns are quite common: 0x0000, 0xffff, 0xaaaa, and 0x5555.  The first two patterns are all ones and all zeros, The second two are the same, alternating ones and zeros.  Theses tests help you uncover a one's density problem on many circuits.

In this case, the 0x0000 pattern failed when connected to the telco.  Since this fails quite consistantly when I try it, why does the telco's tests all come back clean?  The answer is simple: The T-Berd doesn't do an all 0's pattern test at the DS3 level! 

Eventully with a near endless series of loops in both the IXC and LEC, we discover that 0x0001 (15 0s in a row) passes and 0x0000 (16 0s in a row) fails.  The IXC uses Alcatel equipment thaat by default has a setting that disallows excess 0s in DS3 circuits.  When we sent an all 0s pattern, the carrier recieved a different pattern, due to the Alcatel changing the bits in transit.  This introdiced linecoding errors on the DS3, and obviously caused problems with whatever traffic was modified.  After changing this setting in 3 of the Alcatel muxes, the all 0s pattern worked.

The CPU utilization issue became so unbearable that something had to be done, and unfortunately, modifying all Macintosh systems on the network to stop using mDNS wasn't an option.  A few of the systems were changed with no noticable impact, but that could be because the number of systems changed constituted less than 1% of the systems at that campus.  Instead something else would have to be tried... Somthing on the network...

Being confident that the previous troubleshooting was accurate, and with assurances that nothing production other than GhostCast was using multicast, multicast routing was turned off on the switch.  This was a bold and rather drastic move, but it should solve the problem, right?  Wrong.  The number of packets hitting the CPU with the router alert option set remained constant, and CPU usage continued to behave as it had before.  If I had thought thiss through a bit more, I would have realized that traffic to the 224.0.0.x/24 multicast address range are meant to stay local, therefore the traffic would continue. 

The next step was to create an access list that blocks the mDNS group address destination address.  After that is created, it is added as a Port ACL on each client facing switch port.  Suddenly, we now have a real change in status.  CPU Usage dropped from 100% to 21% as soon as the interface range command was completed.

The big question here is why did this work, and what caused the switches poor performance?  The 4500 is capable of handling much higher volumes of multicast traffic, and it has distributed hardware processing of multicast.  It turns out that the 224.0.0.0/24 range is reserved for L2 local multicast, such as routing protocols, All routers, All hosts, etc.  Because of this fact, the 4500 was designed to send all multicast traffic destined to any address in this range directly to the CPU weather it was needed/subscribed, or not.  I think an inbound 224.0.0.0/24 multicast filter should be considered a basic security requirement for every network in order to prevent inadvertant or intentional DoS against the switched infrastructure regardless of weather multicast is officially in use on the network!

To implement this, I changes the ISAKMP profile to use RSA-SIG authentication instead of pre-shared keys.  I set up the remote access VPN as normal, including X-AUTH.  When the VPN client connected, we discovered that everything we did actually worked!  Since making something work the first time is not typically my specialty, I knew something must have been wrong.  A bit of thinking and testing later, the other shoe finally dropped!  We connected to the VPN using USER1's certificate, but tried USER2's username and password.  Much to my dismay, it worked. 

While this does technically pass the definition of two factor authentication, at least as far as the auditors are concerned, I wouldn't call it a good long term plan.  TAC to the rescue!  My confidence was shattered when I was informed that this was a new feature according to the design engineers and wouldn't be available until spring-ish 2008.  Fast forward about a year and guess where I am?  In the past few months I happened to notice a command that I thought enabled the X-AUTH username to certificate CN check.  Apparently I was wrong, as it doesn't work.  Well to be exact, the command I am referencing, username-from-certificate, works when paired with another command, pre-fill-username.  Unfortunately, but not surprisingly, the latter command only applies to WebVPN/SSLVPN!

I am planning on opening a TAC case in the morning on this as well as calling my local SE, in the mean time I am going to assume this is another sign of the death of IPSec (the other sign being no 64-bit OS support).

UPDATE: It turns out that this feature was requested and supplied in version 8.0(3) 1.  Unfortunately only for AnyConnect.  There are no plans to support the IPSec client with certificate CN validation.  Ouch, another nail in the coffin for the good old IPSec VPN client.  I guess I better spend some time snuggling up next to the WebVPN/SSL VPN command reference guides!

Recently a problem came up on a single site of a multi-site network.  This one site had a problem with high CPU utilization on thier Cisco 4510R layer-3 switch.  This switch is the core of this particular site, and there are other similarly configured locations with similar hardware.

Troubleshooting begins with all the usual suspects: show ver, show mod, show proc cpu, show log.  Nothing stands out except the show proc cpu shows Cat4k Mgmt LoPri is taking up 70-80% of the CPU.  I am relatively new to the Catalyst 4500 Sup 6 card, and spend most of my time on the 6500 platform.  I quickly discover that this is a process with many threads.  The details can be seen by executing a differenet command, show platform health.  This command breaks down all the different threads that make up the management processes within a Catalyst 4500 switch.  It also details target and actual CPU utilization statistics. 

In this case, I find that K5CpuMan Review is taking up a lot more CPU than it is supposed to.  Now I must say the name of this process surely means something to someone, but not to me.  Apparently this is the name of the process that handles all CPU switched packets.  (That was my first guess obviously!)  Now packets can be CPU switched for many reasons.  They might be destined for the CPU, there could be a misconfiguration, or they could be marked in a way that forces intermediate systems (routers and switches) to look at them.

The next step is to determine what traffic is being directed at the CPU and why.  This step will probably invoice some debug commands.  The good news to those out there that shiver at the idea of using debugs in a live production network is that the commands we will use have to impact on the performance of the equipment.  Essentially, the platform is doing this stuff already; the commands just make it log the information so it is visible.  Oh, these commands do NOT survive a reboot and don't show up in when a show debug is executed.  What I would like to know is why not just leave these features on all the time?  If we want to see where packets are coming from that are destined for the CPU, we execute debug platform packet all count.  After the debug has been running for a bit, we can display the results with show platform cpu packet statistics.  Unfortunately in this case nothing stands out as an obvious source for the problem. 

For grins I decided to look at some other old standby commands: show interface stats and show ip traffic.  show interface stats will normally show what traffic is switched by which mechanism.  Unfortunately, process switched traffic is at 0 for all interfaces.  We did learn something though; the traffic affecting the processor is destined for the processor.  The next command, show ip traffic, is actually very telling.  Below is an example capture of what this symptom might look like.  Notice that there is a non-zero value for packets with options?  Out of the 3.4 million packets handled by the processor in this example, 1.8 million of them have an IP option defined.  Upon further examination, all of the 1.8 million packets have the router alert option defined!

LAB Router#show ip traffic
IP statistics:
  Rcvd:  3249411 total, 2367891 local destination
         0 format errors, 0 checksum errors, 200092 bad hop count
         0 unknown protocol, 0 not a gateway
         0 security failures, 0 bad options, 1857133 with options
  Opts:  0 end, 0 nop, 0 basic security, 0 loose source route
         0 timestamp, 0 extended security, 0 record route
         0 stream ID, 0 strict source route, 1857133 alert, 0 cipso, 0 ump
         0 other

I know what you are thinking, "What are IP options, and why is this important?"  Well, IP options were flags created to allow hosts and routers to be informed about the special nature of a packet.  RFC 2113 created the Router Alert, IP Option 20.  This option tells the router to inspect any packet with this option passing through it, even if it is not destined for it.  This inspection provides value when implementing RSVP and other similar protocols.

So lets see what's causing these packets to be marked with the Router Alert option.  Another platform specific and non-impacting debug command can create a 1024 packet deep circular sniffer trace buffer.  To enable this feature we execute debug platform packet all buffer.  To view the results, we show platform cpu packets buffered.  Now that's odd!  I know packets are going to the processor, but they are not getting logged by the debug command we just executed.  I am going to blame that on the difference between the Sup-6e and the other processors.  The Sup-6e doesn't always do what the others are capable of.  However other options exist!  A sniffer hooked to the switch might just do the trick, but what interface do we monitor?  Well, the CPU of course!  The change we would make from the normal SPAN setup is in the source: monitor session 1 source cpu queue all rx.  Now we just keep capturing until the options value of the show ip traffic command starts to increment...

Other things to consider here is the possibility of a Denial of Service (DoS) attack.  A few years ago Cisco released a field advisory that explains a vulnerability in IOS when a router is subjected to packets with certain IP options set.  However, if this issue was a DoS attack, one would expect to see a spike in traffic coming from one port, but as we have already seen, the show platform cpu packets statistics shown no specific spike in traffic, meaning the traffic is coming from everywhere, and is most likely normal traffic.

The sniffer is truly the tool to solve this particular issue.  Once the packet capture came back, IGMP packets correlated with the increase in the alert option in the show ip traffic counters were discovered.  Most of the IGM packets were multicast joins to 224.0.0.251: mDNS.  mDNS is very rare in production networks, typically isolated to HP printers that are set to thier defaults.  Apparently Mac OSX has and application called Bonjour that performs dynamic discovery on the network.  One of the protocols in use by this application is mDNS.  This application was discovered because the source MAC address in the sniffer trace ties to an Ethernet OUI registered to Apple.

The troubleshooting steps I have lined out here are a combination of personal experience and things TAC suggested.  Afterwards, TAC emailed me a link to a document that contains a lot of these steps, some of which may be more applicable to other situations.

http://www.cisco.com/en/US/products/hw/switches/ps663/products_tech_note09186a00804cef15.shtml