ACI/N9K – How to convert a Nexus 9000 from NX-OS to ACI

Some time ago i posted how to convert a ACI switch to NX-OS, now the other way around.

Copy the file to the NX-OS Mode N9k

switch# copy scp: bootflash:
Enter source filename: Downloads/aci-n9000-dk9.12.2.1o.bin
Enter vrf (If no input, current vrf 'default' is considered): management
Enter hostname for the scp server: 192.168.0.5
Enter username: USER
Password:
aci-n9000-dk9.12.2.1o.bin                       1%   15MB   3.8MB/s   04:34 ETA

Change the boot mode to aci

switch(config)# boot aci bootflash:///aci-n9000-dk9.12.2.1o.bin
Warning: Please check list of all ACI supported hardware before doing this operation, not all hardware are supported.
Warning: Booting to an ACI image will remove all nxos configuration and format bootflash. Do you want to continue (y/n)?[n] y
Performing image verification and compatibility check, please wait....

Image verification successful.

That’s it, just boot the switch and you can join the fabric πŸ™‚
Be aware that the first reload can take quite some time! (15minutes+)

switch# reload
!!!WARNING! there is unsaved configuration!!!
This command will reboot the system. (y/n)?  [n] y

Error: Error executing command on leaf02. Error Code: 255

I had a issue today with running remote commands on one of my fabric switches, always generated an error.

apic1# fabric leaf02 show switchname
----------------------------------------------------------------
 Node 102 (leaf02)
----------------------------------------------------------------
Error: Error executing command on leaf02. Error Code: 255

I tried to directly connect and found the error

admin@apic1:attach leaf02
This command is being deprecated on APIC controller, please use NXOS-style equivalent command
# Executing command: ssh leaf02 -b 10.127.240.1
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that the RSA host key has just been changed.
The fingerprint for the RSA key sent by the remote host is
d1:f1:c4:8a:3e:a7:df:4a:76:bf:ec:01:bb:0d:28:99.
Please contact your system administrator.
Add correct host key in /home/admin/.ssh/known_hosts to get rid of this message.
Offending key in /home/admin/.ssh/known_hosts:2
Password authentication is disabled to avoid man-in-the-middle attacks.
Keyboard-interactive authentication is disabled to avoid man-in-the-middle attacks.

Permission denied (publickey,password,keyboard-interactive).

I think this was the switch that i converted to NX-OS standalone mode, and thus it changed the host key.

The fix is easy, open the file and delete the offending key

vi /home/admin/.ssh/known_hosts

After that i was able to run commands from the apic and also connect to the fabric switch again.

Cisco ACI – Run Commands on the Switches from your APIC

With the introduction of the NX-OS like CLI Cisco also added the option to run commands on the switches directly from your apic. It’s now even easier to get some infos from one or multiple switches in your fabric.

Show switchname on a single fabric switch

apic1# fabric leaf01 show switchname
----------------------------------------------------------------
 Node 101 (leaf01)
----------------------------------------------------------------
leaf01

You can also run commands on multiple devices

apic1# fabric leaf01, leaf02 show switchname
----------------------------------------------------------------
 Node 101 (leaf01)
----------------------------------------------------------------
leaf01
----------------------------------------------------------------
 Node 102 (leaf02)
----------------------------------------------------------------
leaf02

The list of possible commands is pretty long, just to give an idea

 aaa               aaa
 bfd               BFD commands
 bgp               Display BGP status and configuration
 cdp               Show Cisco Discovery Protocol information
 clock             Display current Date
 coop              Show information about coop
 copp              Control Plane Policing (CoPP) information
 copyright         Copyright information
 cores             Show all core dumps for the current vdc
 dhcp              Show DHCP
 diagnostic        Diagnostic commands
 dpp               Data Plane Policing (DPP) information
 eigrp             Display EIGRP status and configuration
 endpoint          End point
 environment       Environment Information
 fc2               Show fc2 information
 fcoe              Show FCOE paramaters
 fex               Show FEX information
 forwarding        Display mfdm information
 hardware          Show hardware information
 hostname          Hostname
 hsrp              HSRP information
 interface         Show interface status and information
 inventory         system inventory information
 ip                Display IP information
 ipmgr             Show information about ipmgr
 ipv6              Show IPv6 information
 isis              Display IS-IS status and configuration
 istack            Show istack information
 lacp              LACP protocol
 lldp              Show information about lldp
 locator-led       Blink locator led on device
 logging           logging information
 mac               Mac addr information
 mcp               Show information about mcp
 mfdm              Show MFDM information
 module            Module
 monitor           Show SPAN information
 npv               Show Npv information
 ntp               Show NTP information
 oam               Show information about oam
 ospfv3            Display OSPFv3 status and configuration
 port-channel      Show port-channel information
 porttrack         Port Tracking
 processes         Show processes
 radius-server     Radius-server
 redundancy        Show system redundancy status
 resource          Show resource configuration for VDC
 route-map         Route-map information
 routing           Display routing information
 san-port-channel  Show san-port-channel information
 service           Display service information
 snmp              Display SNMP information
 sprom             show SPROM contents
 stats_manager     Show information about stats_manager
 switchname        Show the system's hostname
 system            System-related commands
 tacacs-server     Tacacs-server
 tunnel            Show information about Tunnel
 users             Show users logged onto the system and their sessions
 vdc               Show information about vdc_mgr
 version           Show running firmware version and basic system information
 vlan              VLAN status
 vpc               Virtual Port Channel configuration
 vrf               Display VRF information
 vsan              Show vsan information
 zoning-filter     Display Zoning-Filter information
 zoning-rule       Display Zoning-Rule information

Cisco ACI – Connect to the leaf/spine switches with the NX-OS

Some time ago i posted how you can connect to a spine or leaf switch -> Connect to a leaf/spine switch
With the introduction of NX-OS, the syntax changed a bit. You have now first to drop back to the bash shell and then you can attach the switches. Password is still the same as for the APIC.

apic1# bash
admin@apic1:~> attach leaf01
This command is being deprecated on APIC controller, please use NXOS-style equivalent command
# Executing command: ssh leaf01 -b 10.127.240.1

Password:
Cisco Nexus Operating System (NX-OS) Software
TAC support: http://www.cisco.com/tac
Copyright (c) 2002-2016, Cisco Systems, Inc. All rights reserved.
The copyrights to certain works contained in this software are
owned by other third parties and used and distributed under
license. Certain components of this software are licensed under
the GNU General Public License (GPL) version 2.0 or the GNU
Lesser General Public License (LGPL) Version 2.1. A copy of each
such license is available at
http://www.opensource.org/licenses/gpl-2.0.php and
http://www.opensource.org/licenses/lgpl-2.1.php
leaf01#

There is also the possibility to directly run show commands from the APIC.
Run Commands on the Fabric Switches from your APIC

Acitoolkit – Get all Nodes

Example script to print all your fabric nodes.

from acitoolkit.acitoolkit import Session
from acitoolkit.aciphysobject import Node

url = 'https://apic'
user = 'user'
pw = 'pw'

session = Session(url, user, pw)
session.login()

nodes = Node.get(session)
for node in nodes:
    print('=' * 50)
    print('Pod: {}'.format(node.pod))
    print('Node: {}'.format(node.node))
    print('Mode: {}'.format(node.mode))
    print('Model: {}'.format(node.model))
    print('Vendor: {}'.format(node.vendor))
    print('Serial: {}'.format(node.serial))

ACI/N9K – How to convert a Nexus 9000 from ACI Mode to NX-OS (Standalone)

Get the standalone software here

First you have to copy the new nexus 9000 standalone firmware to the APIC:

admin@apic1:~> scp richy@YOURIP:Downloads/nxos.7.0.3.I2.2a.bin .
nxos.7.0.3.I2.2a.bin                          100%  513MB   9.2MB/s   00:56

And now you can push it to the appropriate Nexus 9000. (The user/password matches the APIC)

admin@apic1:~> scp nxos.7.0.3.I2.2a.bin admin@leaf02:bootflash
Password:
nxos.7.0.3.I2.2a.bin                          100%  513MB   7.0MB/s   01:13

Now reboot the switch and break into the load prompt, this can be done with Control+C in Putty.

loader >
loader > boot nxos.7.0.3.I2.2a.bin

After the switch is booted up you got the default prompts (POAP, Secure Admin PW, etc..) When you finaliy reach the CLI you have to Set the Boot Path!

switch(config)# boot nxos bootflash:///nxos.7.0.3.I2.2a.bin
Performing image verification and compatibility check, please wait....

Also save the config!

switch# copy running-config startup-config
[########################################] 100%
Copy complete.

Now is time to verify that everything is fine with the new image (should, as it already booted ;-))

switch# show boot
Current Boot Variables:

sup-1
NXOS variable = bootflash:/nxos.7.0.3.I2.2a.bin <---- Good
No module boot variable set

Boot Variables on next reload:

sup-1
NXOS variable = bootflash:/nxos.7.0.3.I2.2a.bin <---- Good
No module boot variable set


switch# show install all impact

.
.
.

Compatibility check is done:
Module  bootable          Impact  Install-type  Reason
------  --------  --------------  ------------  ------
     1       yes  non-disruptive          none



Images will be upgraded according to following table:
Module       Image                  Running-Version(pri:alt)           New-Version  Upg-Required
------  ----------  ----------------------------------------  --------------------  ------------
     1        nxos                              7.0(3)I2(2a)          7.0(3)I2(2a)            no
     1        bios     v07.41(10/12/2015):v07.17(09/10/2014)    v07.34(08/11/2015)            no

That’s it, make a final boot and your switch is now ACI-Free!
Maybe there is the time you want to go back to ACI, check out my new post!

Cisco ACI – Reset a APIC

If you want to reset one or all of your APIC Controllers to factory default, there is a easy command for that ‘eraseconfig setup’

β†’ ssh admin@10.127.129.50
Application Policy Infrastructure Controller
admin@10.127.129.50's password:
apic1# bash ---> Only required with Version 1.2+

admin@apic1:~> eraseconfig setup
Do you want to cleanup the initial setup data? The system will be REBOOTED. (Y/n):

When your intention is to reset a whole fabric, it’s recommended to reset the switches first:
Reset a ACI Spine/Leaf Switch
If you reset the APIC Controller first, you have to do this step through console afterwards.

Cisco ACI – NX-OS Style CLI

Cisco introduced a NX-OS like CLI for the Cisco ACI Solution with release 1.2(1i).
In this post i will demonstrate some of the things that can be achieved through the NX-OS CLI.

!Important!
There is no safety net, if you issue something like ‘no tenant XXX‘ the configuration is gone!
No commit, warning or similar!
!Important!

Basics

The NX-OS like CLI is the new default if you connect via SSH to the APIC

β†’ ssh admin@10.127.129.50
Application Policy Infrastructure Controller
admin@10.127.129.50's password:
apic1#

(more…)

Cisco ACI – Reset a ACI Spine/Leaf Switch to default

Login via SSH or Console to the switch

Make sure that there is an image on the leaf/spine

leaf01# dir /bootflash/aci-n9000*
/bootflash/aci-n9000-dk9.11.0.2j.bin

Make sure that this image is used as bootvar

leaf01# cat /mnt/cfg/0/boot/grub/menu.lst.local
boot aci-n9000-dk9.11.0.2j.bin
leaf01# cat /mnt/cfg/1/boot/grub/menu.lst.local
boot aci-n9000-dk9.11.0.2j.bin

If this Image is not set as bootvar, there is a script for that πŸ™‚

leaf01# setup-bootvars.sh aci-n9000-dk9.11.0.2j.bin

After the check you can reset the switch

leaf01# setup-clean-config.sh aci-n9000-dk9.11.0.2j.bin
In progress
In progress
In progress
In progress
In progress
In progress
In progress
In progress
In progress
In progress
In progress
In progress
In progress
In progress
In progress
Done

Just reload the switch

leaf01# reload
This command will reload the chassis, Proceed (y/n)? [n]: y

After this process you can rejoin this Switch to the fabric