VXLAN is a Layer2 overlay scheme over a Layer 3 network. VXLAN uses MAC Address-in-User Datagram Protocol (MAC-in-UDP) encapsulation to provide a means to extend Layer 2 segments across a layer3 segment. This basically means the layer2 packet gets a VXLAN header applied, then that frame gets encapsulated into a UDP IP packet and sent over to the layer3 network.
In later FortiOS 5.4 firmwares VXLAN (Virtual Extensible LAN) encapsulation was added. This is a great technology that can help connect to sites at layer2 over layer3. Something to take note of – as of FortiOS 5.6.2 – lots of improvements and enhancements to VXLAN encapsulation have been made. For example, vlan trunking works well now. Mutlicast also will traverse the VXLAN!
So far I have set this up for two different clients. Both were situations where we had to have layer 2 stretched for a certain purpose. in the last case it was to two different data centers. Below is the scenario and config of the Fortigates as well as show ARP/MAC from the Cisco switch. Fortinet has some great documentation as well on this feature (Links below).
Below shows our simple layout. The red line indicates the VXLAN encapsulation path. Encapsulation only happens at Fortigate firewalls.
Here is a check lists of things that are needed:
Create VXLAN VPN
Local encap-local-gw4 is the public address on the local FW
encap-remote-gw4 is the peer address of the other side
remote-gw is the peer address of the other side
Then create a new Switch interface
Add both the local network, and VXLAN-VPN interface to this switch
Create firewall policies allow traffic
Thoughts and observations:
Lowering the MTU of the VXLAN/internal interface might be a good idea. The VXLAN encapsulation adds around 50-bytes. Most Cisco documentation will mention increasing the MTU, but since we are going over the net with this, increasing MTU means lots of fragmentation.
No IP address on the Switch interface is needed. Actually I have seen small issues when putting an IP address on the interface.
In CLI use the commands below to help get broadcasts (be careful) and ARP to go across.
config sys int
set l2forward enable
set broadcast-foward enable
In 5.6.2 VLANs tags will pass through the tunnel
SIDE 1 (60D)
config vpn ipsec phase1-interface edit “VXLAN” set interface “wan2” set peertype any set proposal aes256-sha1 set encapsulation vxlan set encapsulation-address ipv4 set encap-local-gw4 22.214.171.124 set encap-remote-gw4 126.96.36.199 set remote-gw 188.8.131.52 set psksecret password next end config vpn ipsec phase2-interface edit “VXLAN_ph2” set phase1name “VXLAN” set proposal aes256-sha1 next
config system switch-interface edit “VXLAN-SWITCH” set vdom “root” set member “internal1” “internal2” “VXLAN” next end
Lets look at the Switch in the gui
Then lets check out the Firewall Policies
SIDE 2 (60E)
config vpn ipsec phase1-interface edit “VXLAN” set interface “wan1” set peertype any set proposal aes256-sha1 set encapsulation vxlan set encapsulation-address ipv4 set encap-local-gw4 184.108.40.206 set encap-remote-gw4 220.127.116.11 set remote-gw 18.104.22.168 set psksecret password next end config vpn ipsec phase2-interface edit “VXLAN_ph2” set phase1name “VXLAN” set proposal aes256-sha1 next end
Lets look at the Switch in the Gui
Next lets check out the Firewall Policies
First make sure the VPN is up and working. Then a simple ping test between two devices on the same subnet will be enough to make sure things are working. TCP is always the best way to test . You can also check and make sure that the ARP/MAC address tables on each side show something on the remote side. For example the below shows the ARP/MAC of the Cisco 3650 switch at the Datacenter side (60D).
Protocol Address Age (min) Hardware Addr Type Interface
Internet 192.168.19.21 0 000c.291c.b2a5 ARPA Vlan1
Internet 192.168.19.51 0 000c.2918.b8be ARPA Vlan1 –19.51 lives behind the 60E
Datacenter-Stack#show mac address-table Mac Address Table ——————————————-
Vlan Mac Address Type Ports —- ———– ——– —– 1 000c.2918.b8be DYNAMIC Gi1/0/1 —Fortinet 60D is connected to gig 1/0/1
Thats it! VXLAN is an open source protocol that is a great datacenter technology. Fortinet makes it very easy to get this up and going within a few minutes. EB
Hi! Great Post!! You don´t have problems with the overhead of VXLAN, and need to adjust the MTU? , and the MSS of TCP?.
Yes, you would definitely have issues with MTU/MSS if you did not lower those. Even without changing the values, it works fine – but if you do a packet capture tons for fragments. So issues are there.
I’m running 5.6.3 and the setting below doesn’t exist anymore. I can’t get the dhcp request to go through the vxlan
config sys int
set broadcast-foward enable