Cisco provides a custom ESXi Image/ISO to (re)install Hyperflex Servers. As i’m currently working on fully automating the deployment of a large quantity of there servers, i looked for a way to automate the reinstall of these Hyperflex Servers.
(more…)cisco
Cisco ACI – Convert Leaf Ports back (Downlink to Uplink)
As i described in another Post there is the possibility to change a ACI Leaf Uplink Port to Downlink mode. This can be useful if you require some 40/100G Ports but don’t want to add new Leafs.
If you now wan’t to convert some of these Downlinks back to Uplinks, you can follow this guide here.
(more…)Unable to create San-Port-Channel Between Nexus 5548UP and UCS(-Mini)
The Issue
We implemented a new UCS-Mini for a customer with existing Nexus 5548UP (5.1(3)N1(1a)), on the SAN Part we faced some strange issues:
2017 Mar 25 12:11:30 NEX5548-2 %PORT-5-IF_DOWN_PORT_CHANNEL_MEMBERS_DOWN: %$VSAN 300%$ Interface san-port-channel 200 is down (No operational members) 2017 Mar 25 12:11:31 NEX5548-2 Mar 25 12:11:31 %KERN-3-SYSTEM_MSG: fc2_nsh_tx_frame: FC2 s_id/d_id/vsan error: sid=0xfffffe,did=0x0,vsan=300,rctl:0x23,type:0x1,oxid 0x4d,rxid:0xff25 - kernel 2017 Mar 25 12:12:10 NEX5548-2 %PORT-5-IF_PORT_QUIESCE_FAILED: Interface fc1/20 port quiesce failed due to failure reason: Force Abort Due to Link Failure (NOS/LOS) (0x119) 2017 Mar 25 12:12:10 NEX5548-2 %PORT-5-IF_DOWN_OLS_RCVD: %$VSAN 300%$ Interface fc1/20 is down (OLS received) san-port-channel 200 2017 Mar 25 12:12:10 NEX5548-2 Mar 25 12:12:10 %KERN-3-SYSTEM_MSG: fc2_nsh_tx_frame: FC2 s_id/d_id/vsan error: sid=0xfffffe,did=0x0,vsan=300,rctl:0x23,type:0x1,oxid 0x5a,rxid:0xff32 - kernel
The san-port-channel was really basic and added to just one VSAN
interface san-port-channel 200
channel mode active
switchport mode F
switchport trunk mode off
vsan 220 interfaces:
san-port-channel 100 san-port-channel 200
There was also an existing UCS where the san-port-channel worked without any issue
san-port-channel 100 is up
Hardware is Fibre Channel
Solution
After some looking around i found a bug that matched pretty good on the cisco page.
I checked the MAC OUI on the UCS Mini
UCS-Mini-A# connect nxos
.
.
UCS-Mini-A(nxos)# show int fc1/1
fc1/1 is down
Hardware is Fibre Channel, SFP is short wave laser w/o OFC (SN)
Port WWN is XX:XX:00:de:fb:XX:XX:XX
These matches the OUIs described in the bug
Add MAC OUI “002a6a”, “8c604f”, “00defb” for 5k/UCS-FI
After upgrading the Nexus 5548UP to 5.2.1.N1.9b i was finally able to bring the san-port-channel up between the Nexus and the UCS-Mini.
Software BIOS: version 3.6.0 loader: version N/A kickstart: version 5.2(1)N1(9b) system: version 5.2(1)N1(9b) 2017 Mar 26 07:52:12 NEX5548-2 %PORT-5-IF_UP: %$VSAN 300%$ Interface san-port-channel 200 is up in mode F
BFD and ip redirects
We faced some strange ICMP redirect messages today on one of our devices after we configured BFD for BGP.
Device1
ICMP: bogus redirect from 192.168.100.1 - for 192.168.100.2 use gw 192.168.100.2
gateway address is one of our addresses
ICMP: bogus redirect from 192.168.100.1 - for 192.168.100.2 use gw 192.168.100.2
gateway address is one of our addresses
ICMP: bogus redirect from 192.168.100.1 - for 192.168.100.2 use gw 192.168.100.2
gateway address is one of our addresses
So we checked the device that was sending these redirects and did a short ethanalyzer capture
Device2
ethanalyzer local interface inband-in vdc vdc2 capture-filter "host 192.168.100.2" limit-captured-frames 0 Capturing on inband 192.168.200.2 -> 192.168.200.2 UDP 60 Source port: 49152 Destination port: bfd-echo 192.168.200.2 -> 192.168.200.2 UDP 60 Source port: 49152 Destination port: bfd-echo 192.168.200.2 -> 192.168.200.2 UDP 60 Source port: 49152 Destination port: bfd-echo 192.168.200.2 -> 192.168.200.2 UDP 60 Source port: 49152 Destination port: bfd-echo
So these redirect messages where triggered from the BFD Echo packets that Device2 received from Device1.
We simply forgot to disable `ip redirects` on the interface between Device2 and Device1, after we changed this the ICMP bogus redirect messages where gone.
interface port-channel1 <strong>no ip redirects</strong>
This is documented on various points on the cisco page, for example here.
Before using BFD echo mode, you must disable the sending of Internet Control Message Protocol (ICMP) redirect messages by entering the no ip redirects command, in order to avoid high CPU utilization.
Cisco Champion 2017
I just got the mail that i was accepted to the Cisco Champions 2017 program, this is the first year for me!
What makes a Cisco Champion? Quote from Cisco.com:
Passion, plus a desire to share their perspectives with the community. There are Cisco Champions all over the world. They represent a variety of segments across the IT industry. And they offer their time to help others learn about Cisco and connect with Cisco in unique ways.
Thanks to Cisco for the opportunity to be member of this program!
Error: Error executing command on leaf02. Error Code: 255
I had a issue today with running remote commands on one of my fabric switches, always generated an error.
apic1# fabric leaf02 show switchname ---------------------------------------------------------------- Node 102 (leaf02) ---------------------------------------------------------------- Error: Error executing command on leaf02. Error Code: 255
I tried to directly connect and found the error
admin@apic1:attach leaf02 This command is being deprecated on APIC controller, please use NXOS-style equivalent command # Executing command: ssh leaf02 -b 10.127.240.1 @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @ @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY! Someone could be eavesdropping on you right now (man-in-the-middle attack)! It is also possible that the RSA host key has just been changed. The fingerprint for the RSA key sent by the remote host is d1:f1:c4:8a:3e:a7:df:4a:76:bf:ec:01:bb:0d:28:99. Please contact your system administrator. Add correct host key in /home/admin/.ssh/known_hosts to get rid of this message. Offending key in /home/admin/.ssh/known_hosts:2 Password authentication is disabled to avoid man-in-the-middle attacks. Keyboard-interactive authentication is disabled to avoid man-in-the-middle attacks. Permission denied (publickey,password,keyboard-interactive).
I think this was the switch that i converted to NX-OS standalone mode, and thus it changed the host key.
The fix is easy, open the file and delete the offending key
vi /home/admin/.ssh/known_hosts
After that i was able to run commands from the apic and also connect to the fabric switch again.
Cisco ACI – Run Commands on the Switches from your APIC
With the introduction of the NX-OS like CLI Cisco also added the option to run commands on the switches directly from your apic. It’s now even easier to get some infos from one or multiple switches in your fabric.
Show switchname on a single fabric switch
apic1# fabric leaf01 show switchname ---------------------------------------------------------------- Node 101 (leaf01) ---------------------------------------------------------------- leaf01
You can also run commands on multiple devices
apic1# fabric leaf01, leaf02 show switchname ---------------------------------------------------------------- Node 101 (leaf01) ---------------------------------------------------------------- leaf01 ---------------------------------------------------------------- Node 102 (leaf02) ---------------------------------------------------------------- leaf02
The list of possible commands is pretty long, just to give an idea
aaa aaa bfd BFD commands bgp Display BGP status and configuration cdp Show Cisco Discovery Protocol information clock Display current Date coop Show information about coop copp Control Plane Policing (CoPP) information copyright Copyright information cores Show all core dumps for the current vdc dhcp Show DHCP diagnostic Diagnostic commands dpp Data Plane Policing (DPP) information eigrp Display EIGRP status and configuration endpoint End point environment Environment Information fc2 Show fc2 information fcoe Show FCOE paramaters fex Show FEX information forwarding Display mfdm information hardware Show hardware information hostname Hostname hsrp HSRP information interface Show interface status and information inventory system inventory information ip Display IP information ipmgr Show information about ipmgr ipv6 Show IPv6 information isis Display IS-IS status and configuration istack Show istack information lacp LACP protocol lldp Show information about lldp locator-led Blink locator led on device logging logging information mac Mac addr information mcp Show information about mcp mfdm Show MFDM information module Module monitor Show SPAN information npv Show Npv information ntp Show NTP information oam Show information about oam ospfv3 Display OSPFv3 status and configuration port-channel Show port-channel information porttrack Port Tracking processes Show processes radius-server Radius-server redundancy Show system redundancy status resource Show resource configuration for VDC route-map Route-map information routing Display routing information san-port-channel Show san-port-channel information service Display service information snmp Display SNMP information sprom show SPROM contents stats_manager Show information about stats_manager switchname Show the system's hostname system System-related commands tacacs-server Tacacs-server tunnel Show information about Tunnel users Show users logged onto the system and their sessions vdc Show information about vdc_mgr version Show running firmware version and basic system information vlan VLAN status vpc Virtual Port Channel configuration vrf Display VRF information vsan Show vsan information zoning-filter Display Zoning-Filter information zoning-rule Display Zoning-Rule information
Acitoolkit – Get all Nodes
Example script to print all your fabric nodes.
from acitoolkit.acitoolkit import Session
from acitoolkit.aciphysobject import Node
url = 'https://apic'
user = 'user'
pw = 'pw'
session = Session(url, user, pw)
session.login()
nodes = Node.get(session)
for node in nodes:
print('=' * 50)
print('Pod: {}'.format(node.pod))
print('Node: {}'.format(node.node))
print('Mode: {}'.format(node.mode))
print('Model: {}'.format(node.model))
print('Vendor: {}'.format(node.vendor))
print('Serial: {}'.format(node.serial))
APIC-EM – External Authentication ‘Invalid Login Credentials’
I struggled to configure external authentication on a APIC-EM Controller, i followed the official cisco guide. According to the guide the radius server should return ‘Scope: Scope:ALL, Role:ROLE_ADMIN’ but this is wrong!
Correct is ‘Scope=ALL: Role=ROLE_ADMIN’
Before that i always got a ‘Invalid Login Credentials’, not the best error message 🙂
Python – Building a Cisco Type 7 Decrypt Tool
I wrote a small python script that can decrypt cisco type 7 passwords.
Cisco7Decrypt on Github
You can decrypt password directly through the CLI:
→ python cisco7decrypt.py 12090404011C03162E password
If you want the command accessible from anywhere, just link it to a folder that’s included in your Path. In this example ‘~/bin’
ln -sn `pwd`/cisco7decrypt.py ~/bin/cisco7decrypt