cisco

Unable to create San-Port-Channel Between Nexus 5548UP and UCS(-Mini)

Posted on by richardstrnad Leave a comment

The Issue

We implemented a new UCS-Mini for a customer with existing Nexus 5548UP (5.1(3)N1(1a)), on the SAN Part we faced some strange issues:

2017 Mar 25 12:11:30 NEX5548-2 %PORT-5-IF_DOWN_PORT_CHANNEL_MEMBERS_DOWN: %$VSAN 300%$ Interface san-port-channel 200 is down (No operational members)
2017 Mar 25 12:11:31 NEX5548-2 Mar 25 12:11:31 %KERN-3-SYSTEM_MSG: fc2_nsh_tx_frame: FC2 s_id/d_id/vsan error: sid=0xfffffe,did=0x0,vsan=300,rctl:0x23,type:0x1,oxid 0x4d,rxid:0xff25 - kernel
2017 Mar 25 12:12:10 NEX5548-2 %PORT-5-IF_PORT_QUIESCE_FAILED: Interface fc1/20 port quiesce failed due to failure reason: Force Abort Due to Link Failure (NOS/LOS) (0x119)
2017 Mar 25 12:12:10 NEX5548-2 %PORT-5-IF_DOWN_OLS_RCVD: %$VSAN 300%$ Interface fc1/20 is down (OLS received) san-port-channel 200
2017 Mar 25 12:12:10 NEX5548-2 Mar 25 12:12:10 %KERN-3-SYSTEM_MSG: fc2_nsh_tx_frame: FC2 s_id/d_id/vsan error: sid=0xfffffe,did=0x0,vsan=300,rctl:0x23,type:0x1,oxid 0x5a,rxid:0xff32 - kernel

The san-port-channel was really basic and added to just one VSAN

interface san-port-channel 200
  channel mode active
  switchport mode F
  switchport trunk mode off

vsan 220 interfaces:
    san-port-channel 100 san-port-channel 200

There was also an existing UCS where the san-port-channel worked without any issue

san-port-channel 100 is up
    Hardware is Fibre Channel

Solution

After some looking around i found a bug that matched pretty good on the cisco page.
I checked the MAC OUI on the UCS Mini

UCS-Mini-A# connect nxos
.
.
UCS-Mini-A(nxos)# show int fc1/1
fc1/1 is down
    Hardware is Fibre Channel, SFP is short wave laser w/o OFC (SN)
    Port WWN is XX:XX:00:de:fb:XX:XX:XX

These matches the OUIs described in the bug

Add MAC OUI “002a6a”, “8c604f”, “00defb” for 5k/UCS-FI

After upgrading the Nexus 5548UP to 5.2.1.N1.9b i was finally able to bring the san-port-channel up between the Nexus and the UCS-Mini.

Software
  BIOS:      version 3.6.0
  loader:    version N/A
  kickstart: version 5.2(1)N1(9b)
  system:    version 5.2(1)N1(9b)

2017 Mar 26 07:52:12 NEX5548-2 %PORT-5-IF_UP: %$VSAN 300%$ Interface san-port-channel 200 is up in mode F

BFD and ip redirects

Posted on by richardstrnad Leave a comment

We faced some strange ICMP redirect messages today on one of our devices after we configured BFD for BGP.

Device1

ICMP: bogus redirect from 192.168.100.1 - for 192.168.100.2 use gw 192.168.100.2
      gateway address is one of our addresses
ICMP: bogus redirect from 192.168.100.1 - for 192.168.100.2 use gw 192.168.100.2
      gateway address is one of our addresses
ICMP: bogus redirect from 192.168.100.1 - for 192.168.100.2 use gw 192.168.100.2
      gateway address is one of our addresses

So we checked the device that was sending these redirects and did a short ethanalyzer capture
Device2

ethanalyzer local interface inband-in vdc vdc2 capture-filter "host 192.168.100.2" limit-captured-frames 0
Capturing on inband
192.168.200.2 -> 192.168.200.2  UDP 60 Source port: 49152  Destination port: bfd-echo
192.168.200.2 -> 192.168.200.2  UDP 60 Source port: 49152  Destination port: bfd-echo
192.168.200.2 -> 192.168.200.2  UDP 60 Source port: 49152  Destination port: bfd-echo
192.168.200.2 -> 192.168.200.2  UDP 60 Source port: 49152  Destination port: bfd-echo

So these redirect messages where triggered from the BFD Echo packets that Device2 received from Device1.
We simply forgot to disable `ip redirects` on the interface between Device2 and Device1, after we changed this the ICMP bogus redirect messages where gone.

interface port-channel1
  <strong>no ip redirects</strong>

This is documented on various points on the cisco page, for example here.

Before using BFD echo mode, you must disable the sending of Internet Control Message Protocol (ICMP) redirect messages by entering the no ip redirects command, in order to avoid high CPU utilization.

Cisco Champion 2017

Posted on by richardstrnad Leave a comment

I just got the mail that i was accepted to the Cisco Champions 2017 program, this is the first year for me!

What makes a Cisco Champion? Quote from Cisco.com:
Passion, plus a desire to share their perspectives with the community. There are Cisco Champions all over the world. They represent a variety of segments across the IT industry. And they offer their time to help others learn about Cisco and connect with Cisco in unique ways.

Thanks to Cisco for the opportunity to be member of this program!

Error: Error executing command on leaf02. Error Code: 255

Posted on by richardstrnad Leave a comment

I had a issue today with running remote commands on one of my fabric switches, always generated an error.

apic1# fabric leaf02 show switchname
----------------------------------------------------------------
 Node 102 (leaf02)
----------------------------------------------------------------
Error: Error executing command on leaf02. Error Code: 255

I tried to directly connect and found the error

admin@apic1:attach leaf02
This command is being deprecated on APIC controller, please use NXOS-style equivalent command
# Executing command: ssh leaf02 -b 10.127.240.1
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that the RSA host key has just been changed.
The fingerprint for the RSA key sent by the remote host is
d1:f1:c4:8a:3e:a7:df:4a:76:bf:ec:01:bb:0d:28:99.
Please contact your system administrator.
Add correct host key in /home/admin/.ssh/known_hosts to get rid of this message.
Offending key in /home/admin/.ssh/known_hosts:2
Password authentication is disabled to avoid man-in-the-middle attacks.
Keyboard-interactive authentication is disabled to avoid man-in-the-middle attacks.

Permission denied (publickey,password,keyboard-interactive).

I think this was the switch that i converted to NX-OS standalone mode, and thus it changed the host key.

The fix is easy, open the file and delete the offending key

vi /home/admin/.ssh/known_hosts

After that i was able to run commands from the apic and also connect to the fabric switch again.

Cisco ACI – Run Commands on the Switches from your APIC

Posted on by richardstrnad Leave a comment

With the introduction of the NX-OS like CLI Cisco also added the option to run commands on the switches directly from your apic. It’s now even easier to get some infos from one or multiple switches in your fabric.

Show switchname on a single fabric switch

apic1# fabric leaf01 show switchname
----------------------------------------------------------------
 Node 101 (leaf01)
----------------------------------------------------------------
leaf01

You can also run commands on multiple devices

apic1# fabric leaf01, leaf02 show switchname
----------------------------------------------------------------
 Node 101 (leaf01)
----------------------------------------------------------------
leaf01
----------------------------------------------------------------
 Node 102 (leaf02)
----------------------------------------------------------------
leaf02

The list of possible commands is pretty long, just to give an idea

 aaa               aaa
 bfd               BFD commands
 bgp               Display BGP status and configuration
 cdp               Show Cisco Discovery Protocol information
 clock             Display current Date
 coop              Show information about coop
 copp              Control Plane Policing (CoPP) information
 copyright         Copyright information
 cores             Show all core dumps for the current vdc
 dhcp              Show DHCP
 diagnostic        Diagnostic commands
 dpp               Data Plane Policing (DPP) information
 eigrp             Display EIGRP status and configuration
 endpoint          End point
 environment       Environment Information
 fc2               Show fc2 information
 fcoe              Show FCOE paramaters
 fex               Show FEX information
 forwarding        Display mfdm information
 hardware          Show hardware information
 hostname          Hostname
 hsrp              HSRP information
 interface         Show interface status and information
 inventory         system inventory information
 ip                Display IP information
 ipmgr             Show information about ipmgr
 ipv6              Show IPv6 information
 isis              Display IS-IS status and configuration
 istack            Show istack information
 lacp              LACP protocol
 lldp              Show information about lldp
 locator-led       Blink locator led on device
 logging           logging information
 mac               Mac addr information
 mcp               Show information about mcp
 mfdm              Show MFDM information
 module            Module
 monitor           Show SPAN information
 npv               Show Npv information
 ntp               Show NTP information
 oam               Show information about oam
 ospfv3            Display OSPFv3 status and configuration
 port-channel      Show port-channel information
 porttrack         Port Tracking
 processes         Show processes
 radius-server     Radius-server
 redundancy        Show system redundancy status
 resource          Show resource configuration for VDC
 route-map         Route-map information
 routing           Display routing information
 san-port-channel  Show san-port-channel information
 service           Display service information
 snmp              Display SNMP information
 sprom             show SPROM contents
 stats_manager     Show information about stats_manager
 switchname        Show the system's hostname
 system            System-related commands
 tacacs-server     Tacacs-server
 tunnel            Show information about Tunnel
 users             Show users logged onto the system and their sessions
 vdc               Show information about vdc_mgr
 version           Show running firmware version and basic system information
 vlan              VLAN status
 vpc               Virtual Port Channel configuration
 vrf               Display VRF information
 vsan              Show vsan information
 zoning-filter     Display Zoning-Filter information
 zoning-rule       Display Zoning-Rule information

Acitoolkit – Get all Nodes

Posted on by richardstrnad Leave a comment

Example script to print all your fabric nodes.

from acitoolkit.acitoolkit import Session
from acitoolkit.aciphysobject import Node

url = 'https://apic'
user = 'user'
pw = 'pw'

session = Session(url, user, pw)
session.login()

nodes = Node.get(session)
for node in nodes:
    print('=' * 50)
    print('Pod: {}'.format(node.pod))
    print('Node: {}'.format(node.node))
    print('Mode: {}'.format(node.mode))
    print('Model: {}'.format(node.model))
    print('Vendor: {}'.format(node.vendor))
    print('Serial: {}'.format(node.serial))

Python – Building a Cisco Type 7 Decrypt Tool

Posted on by richardstrnad Leave a comment

I wrote a small python script that can decrypt cisco type 7 passwords.
Cisco7Decrypt on Github

You can decrypt password directly through the CLI:

→ python cisco7decrypt.py 12090404011C03162E
password

If you want the command accessible from anywhere, just link it to a folder that’s included in your Path. In this example ‘~/bin’

ln -sn `pwd`/cisco7decrypt.py ~/bin/cisco7decrypt

Cisco IOS XE – Install the New release 16.1.1 (Denali) on 3850

Posted on by richardstrnad 2 Comments

Cisco continues its strategy to merge the whole Campus Switch platform to a single Image, the newest step in this process is IOS XE 16.1.1. Currently only available for the Cat3k Platform (3650, 3850) but releases for the other Catalyst platforms should follow.

In this post i show you how to upgrade your Cisco 3650/3850 Switch from 3.x to 16.1.1

Frist, grab the software here:
IOS XE Denali 16.1.1 on Cisco.com

Just copy it to your switch

Switch#copy ftp://x:x@10.32.31.15/cat3k_caa-universalk9.16.01.01.SPA.bin flash:
Destination filename [cat3k_caa-universalk9.16.01.01.SPA.bin]?
Accessing ftp://*****:*****@10.32.31.15/cat3k_caa-universalk9.16.01.01.SPA.bin...!!!!!!!
*Dec 16 08:22:42.371: Loading cat3k_caa-universalk9.16.01.01.SPA.bin !!!!!!!!!!!!!...
...
!!!
[OK - 469677062/4096 bytes]

469677062 bytes copied in 432.250 secs (1086587 bytes/sec)

After that you can install the OS as always

Switch#software install file flash:cat3k_caa-universalk9.16.01.01.SPA.bin new
Preparing install operation ...
[1]: Starting install operation
[1]: Expanding bundle flash:cat3k_caa-universalk9.16.01.01.SPA.bin
[1]: Copying package files
[1]: Package files copied
[1]: Finished expanding bundle flash:cat3k_caa-universalk9.16.01.01.SPA.bin
[1]: Verifying and copying expanded package files to flash:
[1]: Verified and copied expanded package files to flash:
[1]: Starting compatibility checks
[1]: Finished compatibility checks
[1]: Starting application pre-installation processing
[1]: Finished application pre-installation processing
[1]: Old files list:
    Removed cat3k_caa-base.SPA.03.03.05SE.pkg
    Removed cat3k_caa-drivers.SPA.03.03.05SE.pkg
    Removed cat3k_caa-infra.SPA.03.03.05SE.pkg
    Removed cat3k_caa-iosd-universalk9.SPA.150-1.EZ5.pkg
    Removed cat3k_caa-platform.SPA.03.03.05SE.pkg
    Removed cat3k_caa-wcm.SPA.10.1.150.0.pkg
[1]: New files list:
    Added cat3k_caa-rpbase.16.01.01E.SPA.pkg
    Added cat3k_caa-srdriver.16.01.01E.SPA.pkg
    Added cat3k_caa-wcm.16.01.01E.SPA.pkg
    Added cat3k_caa-webui.16.01.01E.SPA.pkg
[1]: Creating pending provisioning file
[1]: Finished installing software.  New software will load on reboot.
[1]: Committing provisioning file

[1]: Do you want to proceed with reload? [yes/no]: yes

System configuration has been modified. Save? [yes/no]: yes
Building configuration...
Compressed configuration from 2991 bytes to 1553 bytes[OK]
[1]: Reloading

It takes some time to boot up again, after that you can see the new Version is running

Switch Ports Model              SW Version        SW Image              Mode
------ ----- -----              ----------        ----------            ----
*    1 32    WS-C3850-24P       Denali 16.1.1     CAT3K_CAA-UNIVERSALK9 INSTALL

After that i created a user with priv 15

Switch(config)#username cisco privilege 15 secret cisco

Now you can head to the new WebGui, i attached two impressions of the new GUI. Compared to the old cisco switch GUIs it looks really nice. But if it ever is used?… 🙂

Cisco IOS XE 16.1.1 Interface Configuration Part 1

Cisco IOS XE 16.1.1 Interface Configuration Part 2