Unable to create San-Port-Channel Between Nexus 5548UP and UCS(-Mini)

The Issue

We implemented a new UCS-Mini for a customer with existing Nexus 5548UP (5.1(3)N1(1a)), on the SAN Part we faced some strange issues:

2017 Mar 25 12:11:30 NEX5548-2 %PORT-5-IF_DOWN_PORT_CHANNEL_MEMBERS_DOWN: %$VSAN 300%$ Interface san-port-channel 200 is down (No operational members)
2017 Mar 25 12:11:31 NEX5548-2 Mar 25 12:11:31 %KERN-3-SYSTEM_MSG: fc2_nsh_tx_frame: FC2 s_id/d_id/vsan error: sid=0xfffffe,did=0x0,vsan=300,rctl:0x23,type:0x1,oxid 0x4d,rxid:0xff25 - kernel
2017 Mar 25 12:12:10 NEX5548-2 %PORT-5-IF_PORT_QUIESCE_FAILED: Interface fc1/20 port quiesce failed due to failure reason: Force Abort Due to Link Failure (NOS/LOS) (0x119)
2017 Mar 25 12:12:10 NEX5548-2 %PORT-5-IF_DOWN_OLS_RCVD: %$VSAN 300%$ Interface fc1/20 is down (OLS received) san-port-channel 200
2017 Mar 25 12:12:10 NEX5548-2 Mar 25 12:12:10 %KERN-3-SYSTEM_MSG: fc2_nsh_tx_frame: FC2 s_id/d_id/vsan error: sid=0xfffffe,did=0x0,vsan=300,rctl:0x23,type:0x1,oxid 0x5a,rxid:0xff32 - kernel

The san-port-channel was really basic and added to just one VSAN

interface san-port-channel 200
  channel mode active
  switchport mode F
  switchport trunk mode off

vsan 220 interfaces:
    san-port-channel 100 san-port-channel 200

There was also an existing UCS where the san-port-channel worked without any issue

san-port-channel 100 is up
    Hardware is Fibre Channel


After some looking around i found a bug that matched pretty good on the cisco page.
I checked the MAC OUI on the UCS Mini

UCS-Mini-A# connect nxos
UCS-Mini-A(nxos)# show int fc1/1
fc1/1 is down
    Hardware is Fibre Channel, SFP is short wave laser w/o OFC (SN)
    Port WWN is XX:XX:00:de:fb:XX:XX:XX

These matches the OUIs described in the bug

Add MAC OUI “002a6a”, “8c604f”, “00defb” for 5k/UCS-FI

After upgrading the Nexus 5548UP to 5.2.1.N1.9b i was finally able to bring the san-port-channel up between the Nexus and the UCS-Mini.

  BIOS:      version 3.6.0
  loader:    version N/A
  kickstart: version 5.2(1)N1(9b)
  system:    version 5.2(1)N1(9b)

2017 Mar 26 07:52:12 NEX5548-2 %PORT-5-IF_UP: %$VSAN 300%$ Interface san-port-channel 200 is up in mode F

BFD and ip redirects

We faced some strange ICMP redirect messages today on one of our devices after we configured BFD for BGP.


ICMP: bogus redirect from - for use gw
      gateway address is one of our addresses
ICMP: bogus redirect from - for use gw
      gateway address is one of our addresses
ICMP: bogus redirect from - for use gw
      gateway address is one of our addresses

So we checked the device that was sending these redirects and did a short ethanalyzer capture

ethanalyzer local interface inband-in vdc vdc2 capture-filter "host" limit-captured-frames 0
Capturing on inband ->  UDP 60 Source port: 49152  Destination port: bfd-echo ->  UDP 60 Source port: 49152  Destination port: bfd-echo ->  UDP 60 Source port: 49152  Destination port: bfd-echo ->  UDP 60 Source port: 49152  Destination port: bfd-echo

So these redirect messages where triggered from the BFD Echo packets that Device2 received from Device1.
We simply forgot to disable `ip redirects` on the interface between Device2 and Device1, after we changed this the ICMP bogus redirect messages where gone.

interface port-channel1
  <strong>no ip redirects</strong>

This is documented on various points on the cisco page, for example here.

Before using BFD echo mode, you must disable the sending of Internet Control Message Protocol (ICMP) redirect messages by entering the no ip redirects command, in order to avoid high CPU utilization.

Cisco Champion 2017

I just got the mail that i was accepted to the Cisco Champions 2017 program, this is the first year for me!

What makes a Cisco Champion? Quote from Cisco.com:
Passion, plus a desire to share their perspectives with the community. There are Cisco Champions all over the world. They represent a variety of segments across the IT industry. And they offer their time to help others learn about Cisco and connect with Cisco in unique ways.

Thanks to Cisco for the opportunity to be member of this program!

Cisco ACI – Connect to the leaf/spine switches with the NX-OS

Some time ago i posted how you can connect to a spine or leaf switch -> Connect to a leaf/spine switch
With the introduction of NX-OS, the syntax changed a bit. You have now first to drop back to the bash shell and then you can attach the switches. Password is still the same as for the APIC.

apic1# bash
admin@apic1:~> attach leaf01
This command is being deprecated on APIC controller, please use NXOS-style equivalent command
# Executing command: ssh leaf01 -b

Cisco Nexus Operating System (NX-OS) Software
TAC support: http://www.cisco.com/tac
Copyright (c) 2002-2016, Cisco Systems, Inc. All rights reserved.
The copyrights to certain works contained in this software are
owned by other third parties and used and distributed under
license. Certain components of this software are licensed under
the GNU General Public License (GPL) version 2.0 or the GNU
Lesser General Public License (LGPL) Version 2.1. A copy of each
such license is available at
http://www.opensource.org/licenses/gpl-2.0.php and

There is also the possibility to directly run show commands from the APIC.
Run Commands on the Fabric Switches from your APIC

Nexus 1000v – Port-Profile Error ‘MSP-5-PP_UPDATE_FAILED’

I tried to create a new port-profile on a Nexus 1000V and got the error

2016 Oct 14 10:33:35 N1Kv %MSP-5-PP_UPDATE_FAILED: Update of port-profile 'New-Port-Group' on the vCenter Server failed. Please  verify port-profile config.

This error can appear if you configure more max-ports on the port-profiles than you specified in the ‘svs connection vcenter’. In my case i had overprovisioned the port-profiles with ‘max-group 512’, so i just reduced the max-port on some port-profiles and this solved the issue.

ACI/N9K – How to convert a Nexus 9000 from ACI Mode to NX-OS (Standalone)

Get the standalone software here

First you have to copy the new nexus 9000 standalone firmware to the APIC:

admin@apic1:~> scp richy@YOURIP:Downloads/nxos.7.0.3.I2.2a.bin .
nxos.7.0.3.I2.2a.bin                          100%  513MB   9.2MB/s   00:56

And now you can push it to the appropriate Nexus 9000. (The user/password matches the APIC)

admin@apic1:~> scp nxos.7.0.3.I2.2a.bin admin@leaf02:bootflash
nxos.7.0.3.I2.2a.bin                          100%  513MB   7.0MB/s   01:13

Now reboot the switch and break into the load prompt, this can be done with Control+C in Putty.

loader >
loader > boot nxos.7.0.3.I2.2a.bin

After the switch is booted up you got the default prompts (POAP, Secure Admin PW, etc..) When you finaliy reach the CLI you have to Set the Boot Path!

switch(config)# boot nxos bootflash:///nxos.7.0.3.I2.2a.bin
Performing image verification and compatibility check, please wait....

Also save the config!

switch# copy running-config startup-config
[########################################] 100%
Copy complete.

Now is time to verify that everything is fine with the new image (should, as it already booted ;-))

switch# show boot
Current Boot Variables:

NXOS variable = bootflash:/nxos.7.0.3.I2.2a.bin <---- Good
No module boot variable set

Boot Variables on next reload:

NXOS variable = bootflash:/nxos.7.0.3.I2.2a.bin <---- Good
No module boot variable set

switch# show install all impact


Compatibility check is done:
Module  bootable          Impact  Install-type  Reason
------  --------  --------------  ------------  ------
     1       yes  non-disruptive          none

Images will be upgraded according to following table:
Module       Image                  Running-Version(pri:alt)           New-Version  Upg-Required
------  ----------  ----------------------------------------  --------------------  ------------
     1        nxos                              7.0(3)I2(2a)          7.0(3)I2(2a)            no
     1        bios     v07.41(10/12/2015):v07.17(09/10/2014)    v07.34(08/11/2015)            no

That’s it, make a final boot and your switch is now ACI-Free!
Maybe there is the time you want to go back to ACI, check out my new post!

Cisco IOS XE – Install the New release 16.1.1 (Denali) on 3850

Cisco continues its strategy to merge the whole Campus Switch platform to a single Image, the newest step in this process is IOS XE 16.1.1. Currently only available for the Cat3k Platform (3650, 3850) but releases for the other Catalyst platforms should follow.

In this post i show you how to upgrade your Cisco 3650/3850 Switch from 3.x to 16.1.1

Frist, grab the software here:
IOS XE Denali 16.1.1 on Cisco.com

Just copy it to your switch

Switch#copy ftp://x:x@ flash:
Destination filename [cat3k_caa-universalk9.16.01.01.SPA.bin]?
Accessing ftp://*****:*****@!!!!!!!
*Dec 16 08:22:42.371: Loading cat3k_caa-universalk9.16.01.01.SPA.bin !!!!!!!!!!!!!...
[OK - 469677062/4096 bytes]

469677062 bytes copied in 432.250 secs (1086587 bytes/sec)

After that you can install the OS as always

Switch#software install file flash:cat3k_caa-universalk9.16.01.01.SPA.bin new
Preparing install operation ...
[1]: Starting install operation
[1]: Expanding bundle flash:cat3k_caa-universalk9.16.01.01.SPA.bin
[1]: Copying package files
[1]: Package files copied
[1]: Finished expanding bundle flash:cat3k_caa-universalk9.16.01.01.SPA.bin
[1]: Verifying and copying expanded package files to flash:
[1]: Verified and copied expanded package files to flash:
[1]: Starting compatibility checks
[1]: Finished compatibility checks
[1]: Starting application pre-installation processing
[1]: Finished application pre-installation processing
[1]: Old files list:
    Removed cat3k_caa-base.SPA.03.03.05SE.pkg
    Removed cat3k_caa-drivers.SPA.03.03.05SE.pkg
    Removed cat3k_caa-infra.SPA.03.03.05SE.pkg
    Removed cat3k_caa-iosd-universalk9.SPA.150-1.EZ5.pkg
    Removed cat3k_caa-platform.SPA.03.03.05SE.pkg
    Removed cat3k_caa-wcm.SPA.
[1]: New files list:
    Added cat3k_caa-rpbase.16.01.01E.SPA.pkg
    Added cat3k_caa-srdriver.16.01.01E.SPA.pkg
    Added cat3k_caa-wcm.16.01.01E.SPA.pkg
    Added cat3k_caa-webui.16.01.01E.SPA.pkg
[1]: Creating pending provisioning file
[1]: Finished installing software.  New software will load on reboot.
[1]: Committing provisioning file

[1]: Do you want to proceed with reload? [yes/no]: yes

System configuration has been modified. Save? [yes/no]: yes
Building configuration...
Compressed configuration from 2991 bytes to 1553 bytes[OK]
[1]: Reloading

It takes some time to boot up again, after that you can see the new Version is running

Switch Ports Model              SW Version        SW Image              Mode
------ ----- -----              ----------        ----------            ----
*    1 32    WS-C3850-24P       Denali 16.1.1     CAT3K_CAA-UNIVERSALK9 INSTALL

After that i created a user with priv 15

Switch(config)#username cisco privilege 15 secret cisco

Now you can head to the new WebGui, i attached two impressions of the new GUI. Compared to the old cisco switch GUIs it looks really nice. But if it ever is used?… 🙂

Cisco IOS XE 16.1.1 Interface Configuration Part 1

Cisco IOS XE 16.1.1 Interface Configuration Part 2

Recognise the Production Date of your Cisco Equipment based on the Serial

Today i learnt that the Production Year and Week is ‘hidden’ in the Cisco Serials 🙂

The format of the serial is always like ‘xxxYYWWxxxx’, ‘YY’ is Code for the Year, but not the Year itself! ‘WW’ is the week of manufacture.

Year Codes
Code Year
01 1997
02 1998
03 1999
04 2000
05 2001
06 2002
07 2003
08 2004
09 2005
10 2006
11 2007
12 2008
13 2009
14 2010
15 2011
16 2012
17 2013
18 2014
19 2015
20 2016


Week Codes
Code Week
1-5 January
6-9 February
10-14 March
15-18 April
19-22 May
23-27 June
28-31 July
32-35 August
36-40 September
41-44 October
45-48 November
49-52 December