{"id":15,"date":"2020-03-18T10:11:09","date_gmt":"2020-03-18T02:11:09","guid":{"rendered":"https:\/\/blog.swineson.me\/en\/?p=15"},"modified":"2020-10-08T19:57:57","modified_gmt":"2020-10-08T11:57:57","slug":"use-linux-as-an-mpls-router","status":"publish","type":"post","link":"https:\/\/blog.swineson.me\/en\/use-linux-as-an-mpls-router\/","title":{"rendered":"Use Linux as an MPLS Router"},"content":{"rendered":"<p>Two things happened in 2017:<\/p>\n<ul>\n<li>FRRouting came into existence, which mainlined the Quagga ldpd patch<\/li>\n<li>Cumulus <a href=\"https:\/\/cumulusnetworks.com\/blog\/vrf-for-linux\/\" target=\"_blank\" rel=\"noopener noreferrer\">contributed its VRF implementation to mainline Linux<\/a><\/li>\n<\/ul>\n<p>Linux finally got native, working MPLS (L3VPN) and VRF support. 3 years later, a thorough documentation of MPLS configuration on Linux is still largely missing. Recently, after digging into all kinds of codes and documentation, I had a standard MPLS core network up and running in my lab. This article is a write-up for my lab setup.<\/p>\n<p><!--more--><\/p>\n<h1>MPLS?<\/h1>\n<p>Those IP routing guys without the experience of large-scale carrier network may be unfamiliar with the MPLS technology. MPLS (L3VPN) is like VLAN trunking but for layer 3: you put an interface of a router into a VRF instance (like a VLAN); every router can have multiple routing tables, one for every VRF (like ARP tables for VLANs); when core routers routing packets, these packets will be prefixed by MPLS labels (like VLAN tags) so different packets from different routing tables do not interfere. MPLS brings a bunch of good things:<\/p>\n<ul>\n<li>Use one set of routers to provide different kinds of service to different customers<\/li>\n<li>Allow different customers to use overlapped IP address ranges<\/li>\n<li>Relieve core routers from IP route lookup; they only need to do MPLS label swapping, so better performance can be achieved<\/li>\n<li>Eliminate the need for core routers to store and process all the routes from customers, greatly lowering memory requirements for the core routers<\/li>\n<li>Allow transparently connecting multiple sites of a customer<\/li>\n<li>Simplify layer 3 configurations for complex, multihomed setups<\/li>\n<li>Allow manual route selection of certain flows, so you can easily balance different links<\/li>\n<\/ul>\n<p>All good protocols come with some requirements. MPLS is a &#8220;layer 2.5&#8221; protocol, which means it runs on a layer 2 link. For MPLS to work, all your core routers need to be connected via either a layer 2 link (e.g. Ethernet), or a non-layer-2 tunnel that explicitly supports MPLS (via <a href=\"https:\/\/tools.ietf.org\/html\/draft-ietf-l3vpn-rfc2547bis\" target=\"_blank\" rel=\"noopener noreferrer\">RFC2547bis<\/a>, for example, <a href=\"https:\/\/archive.nanog.org\/meetings\/nanog30\/presentations\/townsley.pdf\" target=\"_blank\" rel=\"noopener noreferrer\">GRE or L2TP<\/a>). MPLS typically have a header of 8 octets, so you might want to set a slightly larger MTU.<\/p>\n<p>All routers in a MPLS core network are loosely divided into 2 classes: P (Provider) routers are those who doesn&#8217;t connect to a customer device; PE (Provider Edge) routers are those have at least one connection to a customer device. A customer router connecting to a PE is called a CE (Customer Edge) router. I&#8217;m not going to mention more theories here but to focus on get simple things running, so these are the steps to start an MPLS core network:<\/p>\n<ol>\n<li>Add a fixed IP address for every P and PE router<\/li>\n<li>Set up IGP so every P and PE routers can ping each other&#8217;s loopback IP address<\/li>\n<li>Enable MPLS processing and LDP service on every P and PE routers<\/li>\n<li>Configure iBGP\/MP-BGP (or anything else that support MPLS L3VPN SAFI) on every PE<\/li>\n<li>Configure VRF instances on PE routers, and add customer-facing ports into corresponding VRF instances<\/li>\n<li>Redistribute customer routes into VRF routing tables<\/li>\n<\/ol>\n<p>Here I&#8217;m going to demonstrate how to configure all the steps under Linux using a minimal lab environment.<\/p>\n<h1>MPLS!<\/h1>\n<h2>The Problem<\/h2>\n<p>Here&#8217;s a classic topology to demonstrate MPLS&#8217;s capability to provide transit to customers with overlapping IP addresses. Say we are a regional small ISP and provide a service to connect customer sites located in 2 cities. One day 2 customers (the &#8220;customer1&#8221; and the &#8220;customer2&#8221;) purchased our transit service but their LAN IP address ranges are the same. If we use the traditional IP core network, we will be at least losing one customer. But with an MPLS core network, we can easily satisfy both customers.<\/p>\n<p><a href=\"https:\/\/blog.swineson.me\/use-linux-as-mpls-router\/snipaste_2020-02-22_19-45-09\/\" rel=\"attachment wp-att-2605\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-large wp-image-2605\" src=\"https:\/\/blog.swineson.me\/wp-content\/uploads\/2020\/02\/Snipaste_2020-02-22_19-45-09-1024x504.png\" alt=\"\" width=\"625\" height=\"308\" \/><\/a><\/p>\n<p>So here is our topology. Notes:<\/p>\n<ul>\n<li>All solid lines are Ethernet connections<\/li>\n<li>Connections between core routers use 192.168.1.0\/24 in a point-to-point configuration<\/li>\n<li>All core routers have a 10.0.0.x\/32 IP address configured on its loopback interface<\/li>\n<li>The e0\/e1\/e2 interfaces on this graph matches ens3\/ens4\/ens5 below in the commands respectively<\/li>\n<\/ul>\n<h2>The Problems We are not Solving for Now<\/h2>\n<p>To limit the length of this article, we are not discussing the following advanced topics:<\/p>\n<ul>\n<li>MTU mismatch problem (for simplicity, I recommend clamping MSS to 1410)<\/li>\n<li>Dynamic routing protocols between CE and PE<\/li>\n<li>Auto configuring of Linux network stack (all the configuration we use in this article will be lost after a reboot; how to use your favorite network manager to configure them on boot is an exercise left for readers)<\/li>\n<li>VPLS (a.k.a. MPLS L2VPN. Pseudowire interface is not supported by Linux by the time of writing)<\/li>\n<\/ul>\n<p>My friend LittleWolf wrote an excellent article covering some topics I&#8217;ve left out: how to use IS-IS as the iBGP protocol in core, and how to configure BGP between CE and PE routers, all in a similar topology. Read it here: <em><a href=\"https:\/\/littlewolf.moe\/linux-platform\/193\/\" target=\"_blank\" rel=\"noopener noreferrer\">[ Linux ] \u4f7f\u7528 Debian Linux \u6784\u67b6 MPLS L3 \u7f51\u7edc<\/a><\/em><\/p>\n<h2>Reference Software Setup<\/h2>\n<p>All devices we use run the following software:<\/p>\n<ul>\n<li>Debian 10 (Linux 4.19.0)<\/li>\n<li>iproute2 4.20.0<\/li>\n<li>FRRouting 7.2.1 (deb provided by FRRouting)<\/li>\n<\/ul>\n<h2>The Configuration<\/h2>\n<p>The first line of each code block describes which environment the following code should be typed in:<\/p>\n<ul>\n<li><code class=\"EnlighterJSRAW\" data-enlighter-language=\"raw\"># linux shell<\/code> means run it under a Linux native shell (bash, etc.)<\/li>\n<li><code class=\"EnlighterJSRAW\" data-enlighter-language=\"raw\"># vtysh<\/code> means run it under vtysh<\/li>\n<li><code class=\"EnlighterJSRAW\" data-enlighter-language=\"raw\"># vtysh config<\/code> means run it under vtysh config mode (<code class=\"EnlighterJSRAW\" data-enlighter-language=\"raw\">conf t<\/code>)<\/li>\n<\/ul>\n<h3>Setup Linux<\/h3>\n<p>Install FRRouting:<\/p>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"shell\"># linux shell\r\ncurl -s https:\/\/deb.frrouting.org\/frr\/keys.asc | sudo apt-key add -\r\necho deb https:\/\/deb.frrouting.org\/frr buster frr-stable | sudo tee -a \/etc\/apt\/sources.list.d\/frr.list\r\napt update\r\napt install frr<\/pre>\n<p>Launch FRRouting via systemd:<\/p>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"null\">#\u00a0linux\u00a0shell\r\nsed -i \"s\/=no\/=yes\/g\" \/etc\/frr\/daemons\r\nsystemctl enable frr\r\nsystemctl restart frr<\/pre>\n<p>Load MPLS kernel module:<\/p>\n<pre class=\"EnlighterJSRAW \" data-enlighter-language=\"shell\"># linux shell\r\nmodprobe mpls_router\r\nmodprobe mpls_iptunnel\r\nmodprobe mpls_gso\r\nmodprobe dummy\r\ncat &gt;\/etc\/modules-load.d\/mpls.conf &lt;&lt;EOF\r\nmpls_router\r\nmpls_iptunnel\r\nmpls_gso\r\ndummy\r\nEOF<\/pre>\n<p>Initial config for MPLS kernel module:<\/p>\n<pre class=\"EnlighterJSRAW \" data-enlighter-language=\"shell\"># linux shell\r\ncat &gt;\/etc\/sysctl.d\/90-mpls-router.conf &lt;&lt;EOF\r\nnet.ipv4.ip_forward=1\r\nnet.ipv6.conf.all.forwarding=1\r\nnet.ipv4.conf.all.rp_filter=0\r\nnet.mpls.platform_labels=1048575\r\nnet.ipv4.tcp_l3mdev_accept=1\r\nnet.ipv4.udp_l3mdev_accept=1\r\nnet.mpls.conf.lo.input=1\r\nEOF\r\nsysctl -p \/etc\/sysctl.d\/90-mpls-router.conf<\/pre>\n<h3>Core Network Layer 3 &amp; Loopback IP Setup<\/h3>\n<p>Loopback IP address is not strictly necessary, but if you don&#8217;t configure it, you will need to set up source IP addresses for every protocol we need. Configuring a loopback IP address will make your life easier and prevents a lot of dark magic problems.<\/p>\n<h4>P<\/h4>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"shell\"># linux shell\r\nip link set ens3 up\r\nip link set ens4 up\r\nip link set ens5 up\r\nip addr add 192.168.1.5 peer 192.168.1.6 dev ens3\r\nip addr add 192.168.1.2 peer 192.168.1.1 dev ens4\r\nip addr add 192.168.1.3 peer 192.168.1.4 dev ens5\r\nip link add dummy0 type dummy\r\nip link set dummy0 up\r\nip addr add 10.0.0.2\/32 dev dummy0<\/pre>\n<h4>PE1<\/h4>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"shell\"># linux shell\r\nip link set ens3 up\r\nip link set ens4 up\r\nip link set ens5 up\r\nip addr add 192.168.1.1 peer 192.168.1.2 dev ens3\r\nip link add dummy0 type dummy\r\nip link set dummy0 up\r\nip addr add 10.0.0.1\/32 dev dummy0<\/pre>\n<h4>PE2<\/h4>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"shell\"># linux shell\r\nip link set ens3 up\r\nip link set ens4 up\r\nip link set ens5 up\r\nip addr add 192.168.1.4 peer 192.168.1.3 dev ens3\r\nip link add dummy0 type dummy\r\nip link set dummy0 up\r\nip addr add 10.0.0.3\/32 dev dummy0<\/pre>\n<h4>RR<\/h4>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"shell\"># linux shell\r\nip link set ens3 up\r\nip addr add 192.168.1.6 peer 192.168.1.5 dev ens3\r\nip link add dummy0 type dummy\r\nip link set dummy0 up\r\nip addr add 10.0.0.4\/32 dev dummy0<\/pre>\n<h4>Checking<\/h4>\n<p>You should be able to ping a router&#8217;s interface IP from its direct neighbor router.<\/p>\n<h3>Set Up IGP in the Core Network<\/h3>\n<p>The object here is to make all loopback IP addresses reachable from any point of your core network. <del>Because I don&#8217;t know how to configure other protocols<\/del> To simplify things, we use OSPF as an example and put all the routers in Area 0. Note that we configured all the Ethernet connections between routers as point-to-point links, so we need to manually set interface type here.<\/p>\n<h4>P<\/h4>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"null\">#\u00a0vtysh\u00a0config\r\ninterface ens3\r\n ip ospf network point-to-point\r\ninterface ens4\r\n ip ospf network point-to-point\r\ninterface ens5\r\n ip ospf network point-to-point\r\nrouter ospf\r\n ospf router-id 10.0.0.2\r\n redistribute connected\r\n redistribute static\r\n network 10.0.0.0\/24 area 0\r\n network 192.168.1.0\/24 area 0<\/pre>\n<h4>PE1<\/h4>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"null\">#\u00a0vtysh\u00a0config\r\ninterface ens3\r\n ip ospf network point-to-point\r\nrouter ospf\r\n ospf router-id 10.0.0.1\r\n redistribute connected\r\n redistribute static\r\n network 10.0.0.0\/24 area 0\r\n network 192.168.1.0\/24 area 0<\/pre>\n<h4>PE2<\/h4>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"null\">#\u00a0vtysh\u00a0config\r\ninterface ens3\r\n ip ospf network point-to-point\r\nrouter ospf\r\n ospf router-id 10.0.0.3\r\n redistribute connected\r\n redistribute static\r\n network 10.0.0.0\/24 area 0\r\n network 192.168.1.0\/24 area 0<\/pre>\n<h4>RR<\/h4>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"null\">#\u00a0vtysh\u00a0config\r\ninterface ens3\r\n ip ospf network point-to-point\r\nrouter ospf\r\n ospf router-id 10.0.0.4\r\n redistribute connected\r\n redistribute static\r\n network 10.0.0.0\/24 area 0\r\n network 192.168.1.0\/24 area 0<\/pre>\n<h4>Checking<\/h4>\n<p>Any router&#8217;s loopback IP address should be pingable from any other router in the core network. For example we ping RR from PE1:<\/p>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"null\"># linux shell\r\nping -I 10.0.0.1 10.0.0.4<\/pre>\n<h3>Enable MPLS processing and LDP service<\/h3>\n<p>Every core-facing interface should enable MPLS processing and enable LDP protocol:<\/p>\n<h4>P<\/h4>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"shell\"># linux shell\r\nsysctl net.mpls.conf.ens3.input=1\r\nsysctl net.mpls.conf.ens4.input=1\r\nsysctl net.mpls.conf.ens5.input=1<\/pre>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"null\">#\u00a0vtysh\u00a0config\r\nmpls ldp\r\n router-id 10.0.0.2\r\n address-family ipv4\r\n  discovery transport-address 10.0.0.2\r\n  interface ens3\r\n  interface ens4\r\n  interface ens5<\/pre>\n<h4>PE1<\/h4>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"shell\"># linux shell\r\nsysctl net.mpls.conf.ens3.input=1<\/pre>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"null\">#\u00a0vtysh\u00a0config\r\nmpls ldp\r\n router-id 10.0.0.1\r\n address-family ipv4\r\n  discovery transport-address 10.0.0.1\r\n  interface ens3<\/pre>\n<h4>PE2<\/h4>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"shell\"># linux shell\r\nsysctl net.mpls.conf.ens3.input=1<\/pre>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"null\">#\u00a0vtysh\u00a0config\r\nmpls ldp\r\n router-id 10.0.0.3\r\n address-family ipv4\r\n  discovery transport-address 10.0.0.3\r\n  interface ens3\r\n<\/pre>\n<h4>RR<\/h4>\n<p>RR doesn&#8217;t really need to be practicing this MPLS network here; it only needs to be able to reach all the PEs. But <a href=\"https:\/\/github.com\/FRRouting\/frr\/issues\/4433#issuecomment-510938439\" target=\"_blank\" rel=\"noopener noreferrer\">current LDP daemon implementation of FRRouting does not support Downstream Unsolicited mode<\/a> so we are left with the only option of Downstream-on-Demand mode. In this mode, if you enable IGP on the RR but do not enable MPLS on it, packets going from PE to RR will be discarded at P because P doesn&#8217;t have an entry for RR in its MPLS switching table. Again, to make things simple, we enable MPLS on the RR too:<\/p>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"shell\"># linux shell\r\nsysctl net.mpls.conf.ens3.input=1<\/pre>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"null\"># vtysh config\r\nmpls ldp\r\n router-id 10.0.0.4\r\n address-family ipv4\r\n  discovery transport-address 10.0.0.4\r\n  interface ens3<\/pre>\n<p>If you really doesn&#8217;t want to run MPLS on the RR, there is another option: do not enable IGP on the RR, write a static route to RR on its neighbor P or PE router and redistribute that static route. The third option is to use <span class=\"lang:default highlight:0 decode:true crayon-inline \">label local allocate for<\/span> to force LDP allocate a label for the prefix to RR.<\/p>\n<h4>Checking<\/h4>\n<p>All adjacent routers should have LDP session established. For example, on the P:<\/p>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"null\"># vtysh\r\n\r\nshow mpls ldp neighbor<\/pre>\n<p>You should see 3 LDP sessions:<\/p>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"null\">AF   ID              State       Remote Address    Uptime\r\nipv4 10.0.0.1        OPERATIONAL 10.0.0.1        09:40:33\r\nipv4 10.0.0.3        OPERATIONAL 10.0.0.3        09:40:01\r\nipv4 10.0.0.4        OPERATIONAL 10.0.0.4        08:18:28<\/pre>\n<p>Plus all core routers should still be able to ping each other&#8217;s loopback IP. (These pings are now going through MPLS network rather than IP network. That&#8217;s a sign that your MPLS network is working.)<\/p>\n<h3>Configuring iBGP on PEs<\/h3>\n<p>iBGP only needs to be configured on PEs but not Ps. This is one of MPLS&#8217;s benefits: core routers only need to process label switching but do not need to store and update routing tables. You can use either a full mesh iBGP setup or a Route Reflector; we have a RR on the topology, so we are sticking to the RR method. To simplify things, I&#8217;m only demonstrating IPv4 config (disable IPv4 unicast SAFI, enable IPv4 VPNv4 SAFI); if you need IPv6 inside MPLS then repeat the config for IPv6.<\/p>\n<p>In a production environment, a single RR instance is a single point of failure. Packet Pusher authored a good article about <a href=\"https:\/\/packetpushers.net\/bgp-rr-design-part-1\/\" target=\"_blank\" rel=\"noopener noreferrer\">RR redundancy architecture designing<\/a> for reference.<\/p>\n<h4>RR<\/h4>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"null\"># vtysh config\r\nrouter bgp 65000\r\n no bgp default ipv4-unicast\r\n neighbor 10.0.0.1 remote-as 65000\r\n neighbor 10.0.0.1 update-source dummy0\r\n neighbor 10.0.0.3 remote-as 65000\r\n neighbor 10.0.0.3 update-source dummy0\r\n address-family ipv4 vpn\r\n  neighbor 10.0.0.1 activate\r\n  neighbor 10.0.0.1 route-reflector-client\r\n  neighbor 10.0.0.3 activate\r\n  neighbor 10.0.0.3 route-reflector-client<\/pre>\n<h4>PE1 and PE2<\/h4>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"null\"># vtysh config\r\nrouter bgp 65000\r\n no bgp default ipv4-unicast\r\n neighbor 10.0.0.4 remote-as 65000\r\n neighbor 10.0.0.4 update-source dummy0\r\n address-family ipv4 vpn\r\n  neighbor 10.0.0.4 activate<\/pre>\n<h4>Checking<\/h4>\n<p>All BGP sessions should come up.<\/p>\n<h3>Configure VRF instances on PE routers<\/h3>\n<p>There are 2 ways to implement VRF on Linux, net namespace and VRF interface. Net namespace is a network stack abstraction unique to the Linux kernel, it provides complete isolation which is useful for paravirtualization and containers, but it also prevents route leaking and data exchange so we are not going to use it. VRF interface is more like the VRF you would see on a commercial router system, since we are only using Linux as a router, we choose this method.<\/p>\n<p>The config is quite simple. Every VRF interface will be associated to a routing table identified by its table ID during creation. Then you can add routed interfaces to a VRF just like how you add a switched port to a bridge. To make the new routing table available to an application, here we will add an unreachable default route with its metric set to the max value.<\/p>\n<p>To prevent human errors, we use the same VRF name and table ID for the same customer on every PE router.<\/p>\n<h4>PE1<\/h4>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"shell\"># linux shell\r\n\r\n# customer 1\r\nip link add customer1 type vrf table 100\r\nip link set customer1 up\r\nip route add vrf customer1 unreachable default metric 4278198272\r\nip -6 route add vrf customer1 unreachable default metric 4278198272\r\nip link set ens4 vrf customer1 \r\nip link set ens4 up\r\nip addr add 172.19.1.1\/24 dev ens4\r\n\r\n# customer 2\r\nip link add customer2 type vrf table 200\r\nip link set customer2 up\r\nip route add vrf customer2 unreachable default metric 4278198272\r\nip -6 route add vrf customer2 unreachable default metric 4278198272\r\nip link set ens5 vrf customer2 \r\nip link set ens5 up\r\nip addr add 172.19.1.1\/24 dev ens5<\/pre>\n<h4>PE2<\/h4>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"shell\"># linux shell\r\n\r\n# customer 1\r\nip link add customer1 type vrf table 100\r\nip link set customer1 up\r\nip route add vrf customer1 unreachable default metric 4278198272\r\nip -6 route add vrf customer1 unreachable default metric 4278198272\r\nip link set ens4 vrf customer1 \r\nip link set ens4 up\r\nip addr add 172.19.2.1\/24 dev ens4\r\n\r\n# customer 2\r\nip link add customer2 type vrf table 200\r\nip link set customer2 up\r\nip route add vrf customer2 unreachable default metric 4278198272\r\nip -6 route add vrf customer2 unreachable default metric 4278198272\r\nip link set ens5 vrf customer2 \r\nip link set ens5 up\r\nip addr add 172.19.2.1\/24 dev ens5<\/pre>\n<h3>Redistribute Customer Routers to VRF Routing Tables<\/h3>\n<p>This article is way too long as what I expected, so we are going to redistribute connected route only just to keep things simple. Dynamic routing protocols between PE and CE routers can be configured in the same way. In this step, you need to create an empty BGP configuration on every VRF, import all the routes you need to redistribute to that BGP instance&#8217;s RIB, add a route distinguisher community to the route, then import it to the main BGP instance&#8217;s RIB.<\/p>\n<p>To prevent human error, we use the same RT and RD as the table ID for every customer, but you need to be aware that they can all be different if required. Another thing you need to remind yourself is that although FRRouting vtysh tried hard to mimic a grammar like the Cisco IOS, they still differ in some way.<\/p>\n<h4>PE1 and PE2<\/h4>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"null\"># vtysh config\r\nrouter bgp 65000 vrf customer1\r\n address-family ipv4 unicast\r\n  redistribute connected\r\n  redistribute static\r\n  label vpn export auto\r\n  rd vpn export 65000:100\r\n  rt vpn both 65000:100\r\n  export vpn\r\n  import vpn\r\nrouter bgp 65000 vrf customer2\r\n address-family ipv4 unicast\r\n  redistribute connected\r\n  redistribute static\r\n  label vpn export auto\r\n  rd vpn export 65000:200\r\n  rt vpn both 65000:200\r\n  export vpn\r\n  import vpn<\/pre>\n<p>When you are typing the config, FRRouting will alert you about some error; don&#8217;t panic and continue.<\/p>\n<h4>Checking<\/h4>\n<p>Now we should see connected routes redistributed by BGP in every VRF of every PE. For example, on PE1:<\/p>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"null\"># vtysh\r\nshow ip bgp vrf customer1<\/pre>\n<p>You should see something like this:<\/p>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"null\">BGP table version is 2, local router ID is 172.19.1.1, vrf id 9\r\nDefault local pref 100, local AS 65000\r\nStatus codes:  s suppressed, d damped, h history, * valid, &gt; best, = multipath,\r\n               i internal, r RIB-failure, S Stale, R Removed\r\nNexthop codes: @NNN nexthop's vrf id, &lt; announce-nh-self\r\nOrigin codes:  i - IGP, e - EGP, ? - incomplete\r\n\r\n   Network          Next Hop            Metric LocPrf Weight Path\r\n*&gt; 172.19.1.0\/24    0.0.0.0                  0         32768 ?\r\n*&gt; 172.19.2.0\/24    10.0.0.3@0&lt;              0    100      0 ?\r\n\r\nDisplayed  2 routes and 2 total paths<\/pre>\n<h3>Configure Customer Site<\/h3>\n<p>To simulate a case where different customers&#8217; IP ranges overlap, we configure the same IP address on the two customer&#8217;s router on the same side.<\/p>\n<h4>customer1-site1 and customer2-site1<\/h4>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"shell\"># linux shell\r\nip link set ens3 up\r\nip addr add 172.19.1.2\/24 dev ens3\r\nip route add default via 172.19.1.1<\/pre>\n<h4>customer1-site2 and customer2-site2<\/h4>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"shell\"># linux shell\r\nip link set ens3 up\r\nip addr add 172.19.2.2\/24 dev ens3\r\nip route add default via 172.19.2.1<\/pre>\n<h4>Checking<\/h4>\n<p>The first thing you need to check is the routers of the same customer pings fine. As we have <a href=\"https:\/\/en.wikipedia.org\/wiki\/Penultimate_hop_popping\" target=\"_blank\" rel=\"noopener noreferrer\">PHP<\/a> and TTL propagation enabled, 3 MPLS hops in the core will result in 2 hops of non-IP routing. So, if you use traceroute or mtr, you&#8217;ll see something like this:<\/p>\n<p><a href=\"https:\/\/blog.swineson.me\/use-linux-as-mpls-router\/snipaste_2020-02-22_21-05-36\/\" rel=\"attachment wp-att-2606\"><img loading=\"lazy\" decoding=\"async\" class=\"alignnone size-large wp-image-2606\" src=\"https:\/\/blog.swineson.me\/wp-content\/uploads\/2020\/02\/Snipaste_2020-02-22_21-05-36-1024x252.png\" alt=\"\" width=\"625\" height=\"154\" \/><\/a><\/p>\n<p>Sadly, Linux only support <a href=\"https:\/\/tools.ietf.org\/html\/rfc4884\" target=\"_blank\" rel=\"noopener noreferrer\">RFC4884<\/a> partially and doesn&#8217;t support <a href=\"https:\/\/tools.ietf.org\/html\/rfc4950\" target=\"_blank\" rel=\"noopener noreferrer\">RFC4950<\/a> at the time of writing, so <code class=\"EnlighterJSRAW\" data-enlighter-language=\"shell\">traceroute -e<\/code> and <code class=\"EnlighterJSRAW\" data-enlighter-language=\"shell\">mtr --mpls<\/code> is not going to help you at all.<\/p>\n<p>Next, we need to verify that we do correctly connected the sites of the same customer. To do this we use netcat to simulate a simple TCP chat service. Start the server on one of the customer sites:<\/p>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"shell\"># linux shell\r\nnc -l -p 8888<\/pre>\n<p>And connect to the server on another site of the same customer:<\/p>\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"shell\"># linux shell\r\nnc 172.19.2.2 8888<\/pre>\n<p>After a successful connection, you can type anything into both the server and the client, press enter, and the message should be sent to the other side.<\/p>\n<h1>MPLS\u2026<\/h1>\n<p>The whole story of Linux development can be summed up in one story: someday you want to make a model plane, so you built one using sticks and paper in the backyard. The next day when you woke up, you are surprised to find that the model plane is already in the sky flown by your neighbor carrying tens of passengers, and there are two people standing on the wings trying to fix the engine and maintain balance as well.<\/p>\n<p>&#8220;It is provided free of charge, and if it fails, I won&#8217;t be hurt, so why to bother?&#8221; And you went on your daily work.<\/p>\n<hr \/>\n<p>Thanks:<\/p>\n<p>Two CCIE certified friends helped a lot during the writing of this article.<\/p>\n<p>References:<\/p>\n<ul>\n<li><a href=\"https:\/\/docs.cumulusnetworks.com\/cumulus-linux\/Layer-3\/Virtual-Routing-and-Forwarding-VRF\/\" target=\"_blank\" rel=\"noopener noreferrer\">Virtual Routing and Forwarding: Cumulus Linux<\/a><\/li>\n<li><a href=\"https:\/\/cumulusnetworks.com\/blog\/vrf-for-linux\/\" target=\"_blank\" rel=\"noopener noreferrer\">VRF for Linux \u2014 a contribution to the Linux Kernel <\/a><\/li>\n<li><a href=\"https:\/\/schd.ws\/hosted_files\/ossna2017\/fe\/vrf-tutorial-oss.pdf\" target=\"_blank\" rel=\"noopener noreferrer\">Using the Linux VRF Solution<\/a><\/li>\n<li><a href=\"https:\/\/github.com\/FRRouting\/frr\/wiki\/Configuring-a-VRF-to-work-properly-for-FRR\" target=\"_blank\" rel=\"noopener noreferrer\">Configuring a VRF to work properly for FRR<\/a><\/li>\n<li><a href=\"https:\/\/stackoverflow.com\/questions\/31926342\/iproute2-commands-for-mpls-configuration\" target=\"_blank\" rel=\"noopener noreferrer\">iproute2 commands for MPLS configuration<\/a><\/li>\n<li><a href=\"http:\/\/docs.frrouting.org\/en\/latest\/ldpd.html\" target=\"_blank\" rel=\"noopener noreferrer\">LDP: FRRouting<\/a><\/li>\n<li><a href=\"https:\/\/github.com\/FRRouting\/frr\/blob\/master\/doc\/developer\/ldpd-basic-test-setup.md\" target=\"_blank\" rel=\"noopener noreferrer\">ldpd basic test setup<\/a><\/li>\n<li><a href=\"https:\/\/gist.github.com\/hkwi\/0bed2a315e419609ef0b58497082311b\" target=\"_blank\" rel=\"noopener noreferrer\">frrouting\/linux bgp mpls example<\/a><\/li>\n<li><a href=\"https:\/\/github.com\/FRRouting\/frr\/issues\/4433\" target=\"_blank\" rel=\"noopener noreferrer\">LDP: Label allocation for &#8220;not connected&#8221; IP Prefix.<\/a><\/li>\n<li><a href=\"http:\/\/docs.frrouting.org\/en\/latest\/bgp.html\" target=\"_blank\" rel=\"noopener noreferrer\">BGP: FRRouting<\/a><\/li>\n<li><a href=\"https:\/\/github.com\/FRRouting\/frr\/issues\/4872\" target=\"_blank\" rel=\"noopener noreferrer\">End-to-End MPLS Not Working On Docker Container<\/a><\/li>\n<li><a href=\"https:\/\/github.com\/FRRouting\/frr\/issues\/5605\" target=\"_blank\" rel=\"noopener noreferrer\">Inconsistency in auto-derived RT value of EVPN show commands.<\/a><\/li>\n<li><a href=\"http:\/\/www.h3c.com\/cn\/d_201212\/922113_30005_0.htm\" target=\"_blank\" rel=\"noopener noreferrer\">MPLS\u53caLDP\u534f\u8bae\u57fa\u7840<\/a><\/li>\n<li><a href=\"https:\/\/www.cisco.com\/c\/en\/us\/support\/docs\/ip\/border-gateway-protocol-bgp\/113555-mp-ebgp-config-00.html\" target=\"_blank\" rel=\"noopener noreferrer\">MP-EBGP Configuration Example<\/a><\/li>\n<li><a href=\"https:\/\/www.cisco.com\/c\/en\/us\/td\/docs\/ios-xml\/ios\/mp_l3_vpns\/configuration\/15-mt\/mp-l3-vpns-15-mt-book\/mp-bgp-mpls-vpn.html\" target=\"_blank\" rel=\"noopener noreferrer\">MPLS: Layer 3 VPNs Configuration Guide, Cisco IOS Release 15M&amp;T<\/a><\/li>\n<li><a href=\"https:\/\/www.cisco.com\/c\/en\/us\/support\/docs\/multiprotocol-label-switching-mpls\/mpls\/13733-mpls-vpn-basic.html\" target=\"_blank\" rel=\"noopener noreferrer\">Configuring a Basic MPLS VPN<\/a><\/li>\n<li><a href=\"http:\/\/www.ciscopress.com\/articles\/article.asp?p=391649&amp;seqNum=2\" target=\"_blank\" rel=\"noopener noreferrer\">Troubleshooting Any Transport over MPLS Based VPNs<\/a><\/li>\n<li><a href=\"https:\/\/linoxide.com\/tools\/simple-chat-netcat-linux\/\" target=\"_blank\" rel=\"noopener noreferrer\">How to Create a Simple Chat with netcat in Linux<\/a><\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"<p>Two things happened in 2017: FRRouting came into existence, which mainlined the Quagga ldpd patch Cumulus contributed its VRF implementation to mainline Linux Linux finally got native, working MPLS (L3VPN) and VRF support. 3 years later, a thorough documentation of MPLS configuration on Linux is still largely missing. Recently, after digging into all kinds of [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[2,3],"tags":[],"class_list":["post-15","post","type-post","status-publish","format-standard","hentry","category-linux","category-router"],"acf":[],"_links":{"self":[{"href":"https:\/\/blog.swineson.me\/en\/wp-json\/wp\/v2\/posts\/15"}],"collection":[{"href":"https:\/\/blog.swineson.me\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.swineson.me\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.swineson.me\/en\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.swineson.me\/en\/wp-json\/wp\/v2\/comments?post=15"}],"version-history":[{"count":16,"href":"https:\/\/blog.swineson.me\/en\/wp-json\/wp\/v2\/posts\/15\/revisions"}],"predecessor-version":[{"id":156,"href":"https:\/\/blog.swineson.me\/en\/wp-json\/wp\/v2\/posts\/15\/revisions\/156"}],"wp:attachment":[{"href":"https:\/\/blog.swineson.me\/en\/wp-json\/wp\/v2\/media?parent=15"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.swineson.me\/en\/wp-json\/wp\/v2\/categories?post=15"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.swineson.me\/en\/wp-json\/wp\/v2\/tags?post=15"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}