Tag Archives: Cisco

Recover Cisco 9200 switch from firmware loss

recently I had an issue where the firmware on a 9200 switch was corrupted. This can help during an upgrade, or removal, etc.

After a reboot the switch came up to rom monitor mode, no biggie I thought- just copy the firmware from USB over to flash and install. This did not work, I continued to get error: “Unsupported destination device/filesystem.” see below-

switch: copy usbflash0:cat9k_lite_iosxe.17.03.04b.SPA.bin flash
Unsupported destination device/filesystem.

I then booted from the USB image and installed the system that way. My steps were:

  • Frist check the file on USB and make sure its there/get the name.

switch: dir usbflash0:

Attributes Size Name

D-HS- 0 System Volume Information
—-A 482856473 cat9k_lite_iosxe.17.03.04b.SPA.bin

Then set my switch to boot to this value :

switch: boot usbflash0:cat9k_lite_iosxe.17.03.04b.SPA.bin

This fully loaded up the switch, and came up at a default setup screen. Once there I copied over the file VIA TFTP to flash. The device would not recognize the USB drive now (Probably cause I booted from it). Once copied to flash, I just installed normally, and rebooted.

Switch# install add file flash:/cat9k_lite_iosxe.17.03.04b.SPA.bin activate commit
install_add_activate_commit: START Thu Feb 10 16:59:35 UTC 2022
Feb 10 16:59:36.334: %INSTALL-5-INSTALL_START_INFO: R0/0: install_engine: Started install one-shot flash:/cat9k_lite_iosxe.17.03.04b.SPA.bin
Feb 10 16:59:36.334 %INSTALL-5-INSTALL_START_INFO: R0/0: install_engine: Started install one-shot flash:/cat9k_lite_iosxe.17.03.04b.SPA.bin

*Feb 10 16:59:36.297: %INSTALL-5-INSTALL_START_INFO: Switch 1 R0/0: install_engine: Started install one-shot flash:/cat9k_lite_iosxe.17.03.04b.SPA.bininstall_add_activate_commit: Adding PACKAGE
install_add_activate_commit: Checking whether new add is allowed ….

This operation requires a reload of the system. Do you want to proceed?
Please confirm you have changed boot config to flash:packages.conf [y/n]y


Next you have to change the boot config – or else it will go back to rom monitor.

config t

boot system flash:packages.conf


wr mem

That’s it!

Cisco ASR “private key not found” Error

I was working on a ASR running code: asr1000rp1-ipbasek9. I wiped the config and started over from the last setup, and tried to SSH in. I was continually getting denied, and check the router – This error showed up.

1w2d: SSH2 0: RSA_sign: private key not found
1w2d: SSH2 0: signature creation failed, status -1

My first thought was to recreate the cert – so I did, still a no go.

I then recreated the cert with a different label or name, and then told the ASR to use this key pair instead – and everything worked. Its like I could not find the default key pair. The commands that I did to fix this are:

crypto key gen rsa general-keys label KEYPAIR-1 modulus 2048

ip ssh rsa keypair-name KEYPAIR-1

Then the ASR gave this back:

Jan 13 07:08:51: %IOSXE-7-PLATFORM: SIP0: sntp: resetting on error 0.273 > 0.1
Jan 13 07:08:52: %SSH-5-DISABLED: SSH 2.0 has been disabled
Jan 13 07:08:52: %SSH-5-ENABLED: SSH 2.0 has been enabled

Cisco ASR 1002-X bandwidth license increase

I installed a new license to a ASR 1002-X going from the default 5 gig to 20 gig throughput. Installing the license was no problem, but after the reboot nothing changed. I found that I had forgotten to change the hardware throughput settings – I thought the license would jus take care of this, but it didn’t.

Here are the commands/options to get the hardware throughput to match the license

First – lets check to make sure what the level is –

ASR#show platform hardware throughput level
The current throughput level is 5000000 kb/s

You can also do a show version and see this info.

Now, lets change to our installed license throughput level.

ASR(config)#platform hardware throughput level ?
10000000 throughput in kbps
20000000 throughput in kbps
36000000 throughput in kbps
40000000 throughput in kbps
5000000 throughput in kbps

ASR(config)#platform hardware throughput level 20000000

exit, and save config. The throughput level does not kick in until a reboot. After the reboot

ASR#show platform hardware throughput level
The current throughput level is 20000000 kb/s

Cisco – Event Management to enable backup interface

I work with a lot of sites that are multi-homed or have a backup connection that, even though its backup still needs to be used at the same time for load balancing.  Recently I was able to have a site that had a very slow MPLS link that was being moved to only backup – and a new SD-WAN link has been installed as primary.  The MPLS link will go away after were sure the SD-WAN option works well.

The only time we want this link to become active is if SD-WAN fails. My first thought is no problem, just  use a routing protocol and a static route – the routing protocol will be our preferred method, and we can just raise the admin distance of the default route to make it a backup. One problem in this scenario, the MPLS uses dynamic routing (EIGRP), and many other locations still use the MPLS for primary, or secondary connections, so we cannot even have this link up (admin state) or the MPLS will advertise the network.

So, to recap what I need is a preferred route to our SD-WAN, and if SD-WAN fails, bring up the backup connection admin state and move traffic to it. Then of course, auto fix everything if SD-WAN comes back online. No problem! Cisco’s Event Manager to the rescue. Cisco’s Embedded Event Manager (EEM) (Copied From Cisco) –is a distributed and customized approach to event detection and recovery offered directly in a Cisco IOS device. EEM offers the ability to monitor events and take informational, corrective, or any desired EEM action when the monitored events occur or when a threshold is reached. An EEM policy is an entity that defines an event and the actions to be taken when that event occurs. When creating EEM scripts, you have two options TCL or CLI – in this case I am using just CLI.

Using EEM on the 3850 core I was able to detect if the default route learned VIA OSPF was removed from the routing table – if that happened, then run the command “No shut” on my MPLS uplink and EIGRP would take over. If the default route was learned at some point through OSPF when things were corrected – then run the command “Shut” on my MPLS uplink. This worked extremely well in combination with Link detection on the Fortigate, and OSPF default route distribution.

The Fortigate is my SD-WAN device, and my default gateway for the network.  I am using link detection to test HTTP access to google. If my WAN interface cannot get a response from google.com in my set time (5 attempts, with 5 seconds between each attempt) then it will remove the default route from my routing table. When this happens the route will be removed from redistribution in OSPF, and removed from the Cisco core. EEM sees this event, and does my list of commands. Below shows the layout, and code to get this going. Interface 0/24 is my MPLS uplink.


First I made sure that the MPLS interface was  shutdown and OSPF was up, and I was receiving the redistributed default route from the Fortigate.

config t

event manager applet MPLS-UP
event routing network type remove protocol OSPF
action 1 cli command “enable”
action 2 cli command “config t”
action 3 cli command “int gig 1/0/24”
action 4 cli command “no shut”
action 5 cli command “exit”

event manager applet MPLS-DOWN
event routing network type add protocol OSPF
action 1 cli command “enable”
action 2 cli command “config t”
action 3 cli command “int gig 1/0/24”
action 4 cli command “shut”
action 5 cli command “exit”

You can check status and history of events by using the show event manager commands.


During a failure of my ISP, everything worked great. The default route was removed from OSPF, which caused an event that EEM matched – then it enabled the MPLS interface, and all routes/default was learned VIA EIGRP and the MPLS. When internet was restored, the MPLS interface was shutdown, and all traffic started flowing over SD-WAN.


Redundant Cisco ASA VPN scenario

Cisco ASA (Pre X series) are still extremely common.

This entry describes a redundant VPN setup of two ISPs on the Branch firewall (Cisco 5505), and one ISP on the Datacenter/hub side (Cisco ASA 5510).

The Branch office  has a cable connection as their primary ISP and a backup 4G Cradle Point. We will be using SLAs to track the internet status of the Cable connection, and a floating static route to control backup route priority.

The idea behind the branch office is that two different Crypto Maps exist, one mapped to each of the interfaces. If the SLA fails and brings down the primary internet the traffic starts going out of the backup connection which has a backup Crypto map applied.  When the primary interface comes back up, then traffic will start going over the crypto map applied to it. Therefore we do not have flip/flop VPNs and it solves the issue of having one crypto map applied to two different interface.




Branch ASA:

interface Vlan2
nameif PRIMARY
security-level 0
ip address
interface Vlan12
nameif BACKUP
security-level 0
ip address

object-group network CORE-SUBNETS        — Object group for Core subnets
object-group network BRANCH-SUBNETS    — Object group for Branch subnets

object network Any-Cable                                     — NAT For Primary
nat (inside,PRIMARY) dynamic interface
object network Any-Backup                                  — NAT For Backup Internet
nat (inside,BACKUP) dynamic interface


nat (inside,any) source static BRANCH-SUBNETS BRANCH-SUBNETS destination static CORE-SUBNETS CORE-SUBNETS

SLA config:

sla monitor 123
type echo protocol ipIcmpEcho interface PRIMARY
sla monitor schedule 123 life forever start-time now

route PRIMARY 1 track 2  – The Track statement maps that SLA to the route

route BACKUP 250     – Floating Static – Makes this a backup route. I set the distance to 250


access-list VPN-to-CORE permit ip object-group BRANCH-SUBNETS object-group CORE-SUBNETS

crypto ipsec ikev1 transform-set AES256SHA esp-aes-256 esp-sha-hmac

Primary Crypto

crypto map BRANCH_MAP 100 match address VPN-to-CORE
crypto map BRANCH_MAP 100 set peer
crypto map BRANCH_MAP 100 set ikev1 transform-set AES256SHA
crypto map BRANCH_MAP 100 set security-association lifetime seconds 28800

crypto ikev1 enable PRIMARY

crypto map BRANCH-MAP interface PRIMARY


crypto map BRANCH-MAP-BK 100 match address VPN-to-CORE
crypto map BRANCH-MAP-BK 100 set peer
crypto map BRANCH-MAP-BK 100 set ikev1 transform-set AES256SHA
crypto map BRANCH-MAP-BK 100 set security-association lifetime seconds 28800
crypto map BRANCH-MAP-BK interface BACKUP

crypto ikev1 enable BACKUP

crypto ikev1 policy 10
authentication pre-share
encryption aes-192
hash sha
group 2
lifetime 86400

Tunnel Group

tunnel-group type ipsec-l2l
tunnel-group ipsec-attributes
ikev1 pre-shared-key password


Core Config:

object-group network CORE-SUBNETS        — Object group for Core subnets
object-group network BRANCH-SUBNETS    — Object group for Branch subnets


nat (inside,any) source static BRANCH-SUBNETS BRANCH-SUBNETS destination static CORE-SUBNETS CORE-SUBNETS


access-list VPN-to-BRANCH permit ip object-group CORE-SUBNETS object-group BRANCH-SUBNETS

crypto ipsec ikev1 transform-set ESP-AES-256-SHA-TRANS esp-aes-256 esp-sha-hmac

crypto map outside_map 100 match address VPN-to-BRANCH
crypto map outside_map 100 set peer              —Notice both IPs
crypto map outside_map 100 set ikev1 transform-set ESP-AES-256-SHA
crypto map outside_map 100 set reverse-route

crypto ikev1 enable outside

crypto map outside_map interface outside

crypto ikev1 policy 10
authentication pre-share
encryption aes-192
hash sha
group 2
lifetime 86400

tunnel-group type ipsec-l2l
tunnel-group ipsec-attributes
ikev1 pre-shared-key password

tunnel-group type ipsec-l2l
tunnel-group ipsec-attributes
ikev1 pre-shared-key password

Cisco USB console setup for a 3750/3850/2960 – USB Mini

The other day I needed to use the the blue mini-USB console cable that Cisco will now send. Its been around a long time, but I always have my normal console laying around and just use that. When I attempted to use it I first installed the USB driver provided by Cisco, everything seemed to work, but I could not open the com port. Today I did some research and got it working – I was just missing a small part, but thought I would write up the steps to try and help someone else. My OS is Windows 10.

So first we have to install the USB driver this can be downloaded from Cisco.com , using your CCO account.  Then install according to the computer, and then reboot. The problem comes in after the reboot – Windows will use the Windows USB driver, and not the Cisco one. So you have to manually change it.

So to walk through, after the install/reboot I connected the cable – Went into device manager to see what com port it was associated to. Com3. Great, then I tried to console to that port – and it would not work.


So, after a lot of troubleshooting I found that you need to update the driver to a locally install one, and when you do that Cisco’s driver will pop up. Those steps are below.

So, lets first change the driver.


Select “update driver software”


Then Select “Let me pick from a list of device drivers on my computer”

driver pick

Bam! Now, select the Cisco driver.


Now, we see that Cisco serial driver is in use.

So, now we should be able to launch Putty and change it to COM3 and it should work.



Thats it!

cisco usb console driver not working

Cisco USB

Cisco BGP UnSuppress Maps

Unsupress maps in Cisco can really be a very helpful tool in situations where you might be summarizing a bunch of /24s to maybe a /20, but you need to leak out one of the /24s without summarization, and still advertise the larger summary route.

By default, once you use summarize all networks that fall under your summary route do not advertise any more.  In my situation I was testing ECMP and needed to advertise one /24 to each of my MPLS neighbors, so my hub router could get back on either path. I couldn’t test this with the full /24 due to outage concerns so we had to do this for a /24 that was not used that often. I am not going to show the layout of the Dual MPLS , but just one.

Below shows the topology


Great, now for config on the Cisco Routers.


  • Created Prefix list of subnets I need to be unsupressed.
  • Create a new route-map to match those subnets.
  • Add the BGP statement referencing my neighbor with the “unsuppress-map” keyword.
  • clear routes soft, to force a refresh.

My Prefix list name will be UMAP and my route-map will be named UMAP-MAP

So lets take a look at our advertised routes to my neighbor before making the changes.


Notice that just the /20 is being advertised. Now check out the config below, and lets apply.

config t

ip prefix-list UMAP seq 5 permit

route-map UMAP-MAP permit 10
 match ip address prefix-list UMAP

router bgp 64551
neighbor unsuppress-map UMAP-MAP

Then clear update BGP

clear ip bgp * soft

So that’s it for the config.  Lets look at the advertised routes now.


Great! we are advertising our /24 and everything is now working perfectly. Unsuppress maps to the rescue!

Cisco ASA VPN Spoke to Spoke communication in 8.3 and later

This configuration was in ASA 8.4

Spoke to spoke communication has always been super easy in ASA Site to Site VPNs. As long as your CRYPTO ACL has the remote subnets in it, and NO-NAT Statements are there  everything pretty much works.

The other day I had an issue getting it to work. After some research I was still struggling. All of my remote sites were in my Crypto ACL, my VPN was up and working to the hub, and any subnet behind the hub would work, but access to other IPSEC tunnels connected behind were not working. See rough sketch of the network below.


I checked Nat statements, looked great, but my traffic was not flowing. I decided to debug via ASDM this is the error I received.


Routing failed to locate next hop for ICMP then my outside (Louisville), and inside (Italy) address.

Other examples are:

Routing failed to locate next hop for TCP then my outside (Louisville), and inside (Italy) address.

Routing failed to locate next hop for UDP then my outside (Louisville), and inside (Italy) address.

Well, does not live inside my firewall – it should be connected to the outside (US-Signal) VIA the VPN. Boom, that’s when it clicked. My nat statement is wrong, well not wrong, just missing. Since these connections are connecting to my outside network, and then going to my outside network – I need to create the nat statement with the source interface and destination interface being US-Signal.

A few things to note about the below statement – I put it at the top of my manual nat entries, and notice the interface – both are US-Signal my outside interface.

object network Louisville-Subnet

object network Italy-Subnet

nat (US-Signal,US-Signal) source static Louisville-Subnet Louisville-Subnet destination static Italy-Subnet Italy-Subnet no-proxy-arp route-lookup

As soon as I added this statement everything worked great. All of my spoke to spoke communication flowed through the hub perfectly.





Cisco ASA 8.4+ manual nat – the only way to nat!

Before learning the more about Manual or “Twice Nat” I would use individual object NAT (Auto NAT) for my incoming services, and use Manual NAT for my No-NAT or if I had to NAT VPN traffic before encryption (Policy NAT).

Recently though I started using it for everything. Once you get the hang of it, it is much more applicable to everyday NAT needs.

Something to note about Manual NAT:

  • Processed before Auto NAT (Nat under the object command)
  • Considers source, or source and destination together (Policy)
    • For example – I need to  NAT traffic to this IP, only when it goes to this network
  • Configured directly from global config
  • Uses objects only, cannot specify direct IPs
  • Can specify to come after auto NAT.

Lets get started with a few examples. A list of all examples is below:

Static NAT – Public address/server to Private address/service

Group/Range of services forwarded into private server

Port redirection

Dynamic NAT

Policy Based Nat

Something to always note – 8.3 and above firmware’s require you to put in the private or real ip address of destination , not the public or Natted address.

Static NAT – Public address/server to Private address/service

Lets say my internal web servers  is at The external IP I am using is I want to NAT in only HTTP (80) traffic to my server. No problem.

Lets create the our address Objects

object service OBJ-TCP-80
 service tcp source eq 80

object network OBJ-
object network OBJ-

Great! Now lets create our Manual nat rule to allow that traffic in.

nat (inside,outside) source static OBJ- service OBJ-TCP-80 OBJ-TCP-80

Thats it, we would create our ACL to allow traffic to the Private host (Remember that, big time change in 8.3) and that’s it. Our traffic would be natted from Public, to Private port 80.

The next example will be a 1 to 1 NAT from our private object created above, to our public object also created above.

NAT (inside,outside) source static OBJ- OBJ-

Modify the ACL to allow traffic to and traffic should make it to the host.


Forward in a group or range of services

In this example we will forward in a group of service objects. Lets say HTTP, HTTPS, and SSH . Still using our local/public hosts. You have two options, 1 create a service group full of existing objects, or create a group of Service-objects. Int the below example I will create a group of predefined objects. This is due to having so many different configurations of groups.

object service OBJ-TCP-80
 service tcp source eq 80

object service OBJ-TCP-443
 service tcp source eq 443

object service OBJ-TCP-22
 service tcp source eq 22

object-group service Web-services
group-object OBJ-TCP-80
group-object OBJ-TCP-443
group-object OBJ-TCP-22

object network OBJ-
object network OBJ-

So now our NAT rule:

nat (inside,outside) source static OBJ- service Web-services Web-services


Port redirection

Sometimes we have the need to make a port such as 8080 on the outside go to our websever on the inside at port 80. The below example shows how to do that. In this example we will forward in port 8080 on our public IP to port 80 on our private webserver. First we need to create the objects for the service and networking addresses, and then apply the nat rule – an don’t forget our ACL. To help visualize whats happening here look at the format of the rule:

nat (source interface,destination interface) source static object ((private) IP) object ( Natted (Public IP))  service Private-Service Public-Service

object service OBJ-TCP-80
 service tcp source eq 80

object service OBJ-TCP-8080
 service tcp source eq 8080

object network OBJ-
object network OBJ-

So now our nat rule:
nat (inside,outside) source static OBJ- service OBJ-TCP-80 OBJ-TCP-8080


Dynamic NAT

I like to usually do this through Auto nat, but you can most definitely do this through Manual.

object network OBJ-

nat (inside,outside) source dynamic OBJ- interface

You could also specify “any” instead of the internal address object, or specify the public IP you want to be natted to instead of “interface”.


Policy Based manual NAT

Manual NAT is the only way I believe that Policy based natting is done. You would use this if you had to NAT traffic into some other IP when going to a certain destination address. In this example lets say we need to NAT traffic from int when going to destination This comes up a lot in healthcare when both sides need to nat into a Public address so there are no address conflicts.

Lets first create our objects, then our Nat rule.

object network OBJ-
object network OBJ-
 host 3.3.3
object network OBJ-

So now our nat rule:

nat (inside,outside) source static OBJ- OBJ- destination static OBJ- OBJ-

This reads that whenever is going to, nat into This might help:

nat (inside,outside) source static Private-IP Natted-IP destination static Real-destination Natted-Destination

So, if this was used for a VPN you would just create an Crypto-ACL and the source would be your Natted-IP, and destination would be your or whatever address lives across the tunnel that you set as your NAT destination.





Cisco Errdisable and recovery options

Errdisable is an extremely cool feature on Cisco switches that can place a port into a disabled state due to some reason/errors on the port. There are many reasons a port can be disabled:
Duplex mismatch
Port channel misconfiguration
BPDU guard violation
UniDirectional Link Detection (UDLD) condition
Link-flap detection
Security violation
Port Aggregation Protocol (PAgP) flap
DHCP snooping rate-limit
Incorrect GBIC / Small Form-Factor Pluggable (SFP) module or cable

And many more. Here is Cisco’s Documentation :http://www.cisco.com/c/en/us/support/docs/lan-switching/spanning-tree-protocol/69980-errdisable-recovery.html

The beauty of this feature is that if I screw something up, or if for example a I configure Port security and there’s and error it will shut down the port so that horrible things like loops or security violations are not allowed. By default Err-disable will shut down the port and it will take a manual shut/no shut of the port.

Finding out what ports and why they were put into ERRDisable

It is very frustrating to see ports come online, and then get shut off for some unknown reason. We can find out why they were shut off with a few simple commands

to find out what ports might be having errdisable problems we can do a :

show interfaces status errdisable

This command will show us all ports that are currently shutdown due to errdisable and the reason why. You can also get more specific with the :


show interfaces gig 1/0/12 status errdisable

to get more information just from that port.

You can of course also see what is happening through the logs or syslog showing something like this

   Received BPDU on port GigabitEthernet4/1 with BPDU Guard enabled. Disabling port.

Auto Recovery options

So how can we make this a temporary setting – what if I was putting a switch in a school, and I want to make sure that if someone plugs up another switch, and I see a BPDU, I shutdown the port and then want that port to come back online in x amount of time. There are two parts to that problem. 1, you have to set BPDU Guard on the port or whole switch. Once that is setup, it will automatically be put into Err-disable state. Now, to bring it out of that state automatically, we have to modify the err-disable recovery option, and the cause option (unless we want all causes to automatically come back up – which might not be good). There are a few commands to help us figure out what has been set already:

Show errdisable recovery

This command will report back to you any recovery options that have been set, and the default recovery value of 300 seconds.


Show errdisable detect

This command will show you if we are detecting this error. By default all should be detecting.


So, lets say I only want BPDUguard to recovery iteself every 60 seconds. This is what I would do:

Config t

errdisable recovery cause bpduguard

errdisable recovery interval 60

This will effectively enable recovery only for BPDUguard, and will change ALL recovery times to 60 seconds.

The following is the show recovery after the change:


Errdisable is a great feature that Cisco implements in almost all of their switches. It can really save some pain if you incorrectly configure a etherchannel, or have a bad cable that is really sending a ton of CRCs.