TravelingPacket – A blog of network musings

This is a design I need a few weeks ago to help with a redundancy issue. Currently we have a client that occupies two buildings separated by about 500 hundred feet. Soon they will start construction to add a structure right in the middle , connecting the two buildings. But guess what runs right in the middle of this area? The fiber connecting the two buildings. We are thinking that the construction will most certainly cut the fiber causing an outage, whether planned or not.

We decided to have a backup wireless bridge link to help with redundancy. Ruckus’s P300 AC bridges works great, and that is what we decided to do .

Currently the link between the buildings is a Layer2 Trunk, and we are routing over Vlan 254 which traverses the trunk. OSPF is used to advertise each building’s local subnets, and redistribute the default route.

The goal is that routing/layer 2 will only come active on the wireless bridge in case of a failure in the Fiber connections. So Spanning-tree will block all vlans other than the native 200 – going through the bridge. If there is a failure, those vlans will come online over the bridge, routing will come up, and all should work great.

The switching/routing that is used is a Nexus 9500 and 3850 stack.

To accomplish the above, we enable OSPF on vlan 254, and make sure all routing is correct – including redistribution. Vlan 254 our routing vlan is allowed along with a few other vlans – At some point this will be fixed and we will only route over this link, but for now we have to stretch (I know, not the best practice).  Building 1 is currently the STP root for all vlans stretching over layer 2 link.  The Spanning-tree path cost is increased on links connecting to the bridge, and special commands are enabled on the wireless bridge to disable the Ruckus loop detection mechanism. This is very important because it will stop STP from flowing by default. Follow this link to help with that:

https://usermanual.wiki/Ruckus/P30010010957ReleaseNotesRevA20170721.1820034031/help.

After the bridge was setup, path cost modified, and Cisco port configs set correctly – it is time to test. First we needed to make sure STP was indeed blocking the vlans that were needed. Yes! STP is blocking the redundant path.

Testing/Results

We tested fail over in two ways. 1 – just shutting down fiber links in CLI, and 2- physically unplugging the links. During fail over we saw that 2 pings were lost and then they were back up. I actually thought that OSPF drop, and then re-converge, but that did not happen. Instead, since the Hello-Dead timers were never reached, OSPF never dropped – fail over time was much better than I expected. The only way I could really tell we had failed over was a small increase in latency, and of course we were limited to around 300 Mbits.

Some notes on this – Make 100% sure that the Ruckus loop detection is disabled before even starting actual bridge configuration. Also create some kind of alerts VIA Prtg/Solarwinds/Cacti to send an alert if links go down, or their is a big increase in bandwidth on wireless bridges.

I wanted to create a backup link for a network using a P300 bridge. The current network has two 10 gig links going between two buildings, but construction is set to start soon, that could cut the fiber stretching between. One option was to use the P300 bridges to create a backup link between the two building, which would become active in case of a failure in the fiber links.

We are currently stretching maybe 12 vlans between the buildings. The goal was to have all data go over the 2×10 gig links, and Spanning-tree block the other vlans from using the Bridge. I increased the STP port cost on each side and brought up the bridges – and both fiber links and bridge were forwarding, causing a loop. It took a bit to understand, but according to Ruckus their Gateway bridge detection mechanism basically stops STP and LACP from forwarding. I found the below help doc from Ruckus which gave the command to disable this feature.

https://usermanual.wiki/Ruckus/P30010010957ReleaseNotesRevA20170721.1820034031/help

After disabling – Bam! Vlans are blocked VIA STP.

The command to disable this feature in CLI:

rkscli: set meshcfg loop_detect disable

Below is the topology

I modified the Path cost to something crazy high – 5000 with this command.

interface TenGigabitEthernet1/0/15
description “Connected to Ruckus Bridge”
switchport trunk native vlan 200
switchport trunk allowed vlan 5,10,12,14,15,20,22,40,150,200,254,1337
switchport mode trunk
auto qos trust
spanning-tree cost 5000

Then as soon as I disabled the loop Detect STP shutdown the secondary link for each vlan that the switch was not root for. .

For example, Building 1 is root for all vlans except 10, 254 and 12. So, Building 1 blocked those vlans from traversing port 15 which is the bridge.

Failover took just seconds to work.