VXLAN has been around for a while, so how do router vendors support it? Well, let’s use a dead simple topology to test them out.

Our setup today:

• All routers connected to the same dumb switch using IP range 169.254.0.0/24
• Multicast signaling on address 239.0.0.1, No PIM
• VXLAN UDP port 4789
• Network 10.0.0.0/24 on VNI 5000 (layer 3 termination / inter-VXLAN routing)

Generic Linux

Tested version: Debian 10, Linux kernel 4.19.0

Linux runs just fine with the most simple setup across all operating systems we tested. VXLAN VTEP is implemented as a layer 2 tunnel device which means you can either put it into a bridge or just assign IPs to it.

Notes:

• Support IPv6 underlay (yes!)
• UDP port is 8472 by default
• TTL is 1 by default

References:

• VXLAN & Linux
• VXLAN: BGP EVPN with Cumulus Quagga (or FRR)

VyOS

Tested version: 1.2.5, Linux kernel 4.19.106

VyOS is basically Linux with a better config interface, so there isn’t much difference.

Notes:

• UDP port is 8472 by default
• If you forget to set the remote-port on interface creation, you have to delete the interface, commit, create the interface again

References:

• VXLAN

Cisco IOS XE

Tested version: 16.12.3

IOS XE’s command line user experience and internal design is a mess, and the documentation is bad in many aspects. IOS XE does not support running layer 3 over a VXLAN VTEP (yes, it self identifies as a router) so here’s a config for bridging the VTEP with an ethernet port.

To bridge the VTEP with a tagged VLAN on an ethernet port, you can’t just create a VLAN subinterface and bridge it. Instead, you need a service instance like this:

Notes:

• All Cisco documentation write VXLAN as “VxLAN” which is not correct and triggers my OCD
• You can only use VNI 4096-16777215 (obviously a result of bad software engineering)
• IPv6 multicast group for underlay network is not supported
• You can’t configure IP/IPv6 address on NVE interface
• No inter-VXLAN routing, and adding BVI/BDI/Loopback to the bridge domain won’t work
• One device supports 8000 bridge domains at most
• IOS XE for catalyst switches does not support multicast VXLAN but support inter-VXLAN routing

Also I noticed that there will be a Tunnel interface automatically configured for every NVE interface, the tunnel interface will copy the IP address from NVE’s source interface.

References:

• Cisco CSR 1000V VxLAN Support

Cisco ASA

Tested version: 9.10.1

In my past experience, ASAs are little devices with some odd personalities that I had a hard time getting used to. However configuring VXLAN on the ASA turned out to be very simple.

Notes:

• Smart license means $$$
• IPv6 multicast group for underlay network is not supported
• No PBR support (and a lot of other protocols/features, but routing on ASA has always been weird so I’d not bother mentioning them)
• Only 1 NVE instance per context for now
• You can have only 10000 VNI interfaces per context
• To make a physical interface only able to accept VXLAN traffic, you can use the nve-only command

References:

• CLI Book 1: Cisco ASA Series General Operations CLI Configuration Guide, 9.4

Junos OS for Juniper EX/QFX/MX

Juniper like to overdesign everything. Their design is sometimes very useful, but make network designing complicated.

Configuration for vQFX:

Notes:

• Junos OS must use an IP configured on the loopback interface as the VTEP source IP address
• Only one interface on a device can receive and parse VXLAN packets
• IPv6 multicast group for underlay network is not supported
• If you need to run inter-VXLAN routing, you need to configure irb-interface on the vlan interface, and you must configure a vlan-id for every irb-interface enabled vlan interface, which limits the available vlan interface count to 4092
• Why 4092? because don’t use vlan-id 1 and 2 on the VXLAN vlan interface, these are cursed numbers
• The ability to run inter-VXLAN routing differs from device to device
• SRX devices do not support VXLAN
• If you are using vQFX for lab purposes, please use version 15.1X53-D60.4; other versions (no matter higher or lower) have problems on VXLAN, RSVP, EoMPLS (l2circuit), etc.

References:

• VXLAN MULTICAST
• Examples: Manually Configuring VXLANs on QFX Series and EX4600 Switches
• VXLAN Constraints on QFX Series and EX Series Switches

MikroTik RouterOS

Tested version: 7.1 beta 1

RouterOS is the latest one to support VXLAN and the support is limited to their cutting-edge development version. But let’s try it out, shall we?

The config is as simple as Linux. (Well, I guess they are using the stock Linux implementation of VXLAN tunnel.) I’m using the new slash/seperated/config/grammar here.

Notes:

• UDP port is 8472 by default
• You must set a MTU for VXLAN interface because the default value 1500 is not going to work (for an underlay network with MTU 1500, the VXLAN MTU is 1450)

Conclusion

Although BGP EVPN is the future, unicast/multicast EVPN is still handy for small-scale, autonomous deployments without a dedicated controller or something. Enterprise solutions tend to overlook the simple needs and require a lot just to make everything work. Also enterprise solutions tend to have very bad software quality. On the contrary, Linux implements VXLAN in a simple and elegant way, but its inability to atomically save and restore network config added some management overhead to the operation guys.