TravelingPacket – A blog of network musings

VXLAN is a Layer2 overlay scheme over a Layer 3 network. VXLAN uses MAC Address-in-User Datagram Protocol (MAC-in-UDP) encapsulation to provide a means to extend Layer 2 segments across a layer3 segment. This basically means the layer2 packet gets a VXLAN header applied, then that frame gets encapsulated into a UDP IP packet and sent over to the layer3 network.

In later FortiOS 5.4 firmwares VXLAN (Virtual Extensible LAN) encapsulation was added. This is a great technology that can help connect to sites at layer2 over layer3. Something to take note of – as of FortiOS 5.6.2 – lots of improvements and enhancements to VXLAN encapsulation have been made. For example, vlan trunking works well now. Mutlicast also will traverse the VXLAN!

So far I have set this up for two different clients. Both were situations where we had to have layer 2 stretched for a certain purpose. in the last case it was to two different data centers. Below is the scenario and config of the Fortigates as well as show ARP/MAC from the Cisco switch. Fortinet has some great documentation as well on this feature (Links below).

http://kb.fortinet.com/kb/documentLink.do?externalID=FD38614

http://kb.fortinet.com/kb/documentLink.do?popup=true&externalID=FD40170&languageId=

Below shows our simple layout. The red line indicates the VXLAN encapsulation path. Encapsulation only happens at Fortigate firewalls.

Here is a check lists of things that are needed:

• Create VXLAN VPN
  • Local encap-local-gw4 is the public address on the local FW
  • encap-remote-gw4 is the peer address of the other side
  • remote-gw is the peer address of the other side
• Then create a new Switch interface
  • Add both the local network, and VXLAN-VPN interface to this switch
• Create firewall policies allow traffic

Thoughts and observations:

• Lowering the MTU of the VXLAN/internal interface might be a good idea. The VXLAN encapsulation adds around 50-bytes. Most Cisco documentation will mention increasing the MTU, but since we are going over the net with this, increasing MTU means lots of fragmentation.
• No IP address on the Switch interface is needed. Actually I have seen small issues when putting an IP address on the interface.
• In CLI use the commands below to help get broadcasts (be careful) and ARP to go across.
  • config sys int
    • edit VXLAN
      • set l2forward enable
      • set broadcast-foward enable
      • end
    • end
• In 5.6.2 VLANs tags will pass through the tunnel

CONFIG

SIDE 1 (60D)

config vpn ipsec phase1-interface
edit “VXLAN”
set interface “wan2”
set peertype any
set proposal aes256-sha1
set encapsulation vxlan
set encapsulation-address ipv4
set encap-local-gw4 1.1.1.1
set encap-remote-gw4 1.1.1.2
set remote-gw 1.1.1.2
set psksecret password
next
end
config vpn ipsec phase2-interface
edit “VXLAN_ph2”
set phase1name “VXLAN”
set proposal aes256-sha1
next
end

config system switch-interface
edit “VXLAN-SWITCH”
set vdom “root”
set member “internal1” “internal2” “VXLAN”
next
end

Lets look at the Switch in the gui

Then lets check out the Firewall Policies

SIDE 2 (60E)

config vpn ipsec phase1-interface
edit “VXLAN”
set interface “wan1”
set peertype any
set proposal aes256-sha1
set encapsulation vxlan
set encapsulation-address ipv4
set encap-local-gw4 1.1.1.2
set encap-remote-gw4 1.1.1.1
set remote-gw 1.1.1.1
set psksecret password
next
end
config vpn ipsec phase2-interface
edit “VXLAN_ph2”
set phase1name “VXLAN”
set proposal aes256-sha1
next
end

Lets look at the Switch in the Gui

Next lets check out the Firewall Policies

Testing

First make sure the VPN is up and working. Then a simple ping test between two devices on the same subnet will be enough to make sure things are working. TCP is always the best way to test . You can also check and make sure that the ARP/MAC address tables on each side show something on the remote side. For example the below shows the ARP/MAC of the Cisco 3650 switch at the Datacenter side (60D).

Datacenter-Stack#show arp

Protocol  Address          Age (min)  Hardware Addr   Type   Interface

Internet  192.168.19.21           0   000c.291c.b2a5  ARPA   Vlan1

Internet  192.168.19.51           0   000c.2918.b8be  ARPA   Vlan1  – 19.51 lives behind the 60E

Datacenter-Stack#show mac address-table
Mac Address Table
——————————————-

Vlan Mac Address Type Ports
—- ———– ——– —–
1 000c.2918.b8be DYNAMIC Gi1/0/1  — Fortinet 60D is connected to gig 1/0/1

Conclusion

Thats it! VXLAN is an open source protocol that is a great datacenter technology. Fortinet makes it very easy to get this up and going within a few minutes. EB