NOTE: Unlike most of the articles in this series, this article isn’t intended to exist as a standalone article. This is a continuation of the lab built in part 1, so you should have completed that series, or at least scan that series to build the base lab for this post.

In part 1 of this series, we configured VxLAN on VyOS over NetBird. This provided a secure solution to enable layer 3 traffic between sites. VxLAN is so much more than that. In this post, we’re going to add the ability to stretch Layer 2 traffic between sites.

Why use an L2VPN

You may wonder why you would need to send layer 2 traffic between sites over a layer 3 transport. There are some services that will only operate at layer 2 (vMotion is notorious for this).

Another possibility is if you’re functioning as a service provider. You may want to offer L2VPN services to provide a pseudowire between the customer sites. This allows the user to just plug in a cable on each of their sites, and build something as simple as OSPF between them. No additional configuration would be required on your end like you’d have with L3VPNs.

Topology

We’re going to use the same topology we used in part 1 of this series.

VLAN/VNI Info:

• VLAN 20:
  • Subnet: 172.17.0.0/24
  • VNI: 1002
• VLAN 30:
  • Subnet: 172.18.0.0/24
  • VNI: 1003

NOTE: There is currently an issue where if an interface isn’t present at boot when VyOS loads its configuration, it will remove all config where that interface is applied. You can track the progress of this bug here: https://vyos.dev/T5991

If you need to reboot this lab, and config is removed, just do this to recover the config. You may need to commit a couple of times since some of the config items have dependencies on other applied config:

vyos@Spoke1:~$ configure
vyos@Spoke1# rollback-soft 0
vyos@Spoke1# commit

Configuring VLANs on Bridge Interface

We need to add our VLANs to the bridge interface before they are able to be used for L2VPN bridge traffic. Notice that this config doesn’t get put into a VRF. This needs to be in the global table. You also don’t neccessarily need IP addresses on these interfaces, but we’re going to use them for initial testing.

Hub:
set interfaces bridge br0 vif 20 address '172.17.0.1/24'
set interfaces bridge br0 vif 30 address '172.18.0.1/24'

Spoke1:
set interfaces bridge br0 vif 20 address '172.17.0.11/24'
set interfaces bridge br0 vif 30 address '172.18.0.11/24'

Spoke2:
set interfaces bridge br0 vif 20 address '172.17.0.12/24'
set interfaces bridge br0 vif 30 address '172.18.0.12/24'

Adding LAN interface to Bridge

We need to add a LAN facing physical interface to the bridge, and allow VLANs 20 and 30 through it.

set interfaces bridge br0 member interface eth1 allowed-vlan '20'
set interfaces bridge br0 member interface eth1 allowed-vlan '30'

Mapping VLANs to VNI

If you remember in part 1, to be able to use a single bridge domain like we’re doing, we need to be able to map VLANs to VNI. We do that in the VxLAN interface config.

set interfaces vxlan vxlan0 vlan-to-vni 20 vni '1002'
set interfaces vxlan vxlan0 vlan-to-vni 30 vni '1003'

Advertising MAC Addresses into EVPN

We need to ensure that MACs for our bridge interface are advertised into EVPN.

set protocols bgp address-family l2vpn-evpn advertise-svi-ip

Verification

Before we start testing between sites from LAN hosts, let’s check how we’re looking right now.

Let’s look at BGP. We’re going to look at type-2 EVPN routes, which are used to advertise MAC/IP that have been learned. Right now, we should only have the MACs and IPs that are configured directly on the nodes.

vyos@Spoke1:~$ show bgp l2vpn evpn route type 2
Route Distinguisher: 172.18.0.1:3
 *>i[2]:[0]:[48]:[a2:9b:78:2b:11:cb]:[32]:[172.17.0.1]
                    100.90.53.188                 100      0 i
                    RT:65000:1002 ET:8
 *>i[2]:[0]:[48]:[a2:9b:78:2b:11:cb]:[128]:[fe80::a09b:78ff:fe2b:11cb]
                    100.90.53.188                 100      0 i
                    RT:65000:1002 ET:8
Route Distinguisher: 172.18.0.1:4
 *>i[2]:[0]:[48]:[a2:9b:78:2b:11:cb]:[32]:[172.18.0.1]
                    100.90.53.188                 100      0 i
                    RT:65000:1003 ET:8
 *>i[2]:[0]:[48]:[a2:9b:78:2b:11:cb]:[128]:[fe80::a09b:78ff:fe2b:11cb]
                    100.90.53.188                 100      0 i
                    RT:65000:1003 ET:8
Route Distinguisher: 172.18.0.11:3
 *> [2]:[0]:[48]:[02:ec:2d:6b:2c:78]:[32]:[172.17.0.11]
                    172.18.0.11                        32768 i
                    ET:8 RT:65000:1002
 *> [2]:[0]:[48]:[02:ec:2d:6b:2c:78]:[128]:[fe80::ec:2dff:fe6b:2c78]
                    172.18.0.11                        32768 i
                    ET:8 RT:65000:1002
Route Distinguisher: 172.18.0.11:4
 *> [2]:[0]:[48]:[02:ec:2d:6b:2c:78]:[32]:[172.18.0.11]
                    172.18.0.11                        32768 i
                    ET:8 RT:65000:1003
 *> [2]:[0]:[48]:[02:ec:2d:6b:2c:78]:[128]:[fe80::ec:2dff:fe6b:2c78]
                    172.18.0.11                        32768 i
                    ET:8 RT:65000:1003
Route Distinguisher: 172.18.0.12:3
 *>i[2]:[0]:[48]:[7e:5b:5a:51:2a:36]:[32]:[172.17.0.12]
                    100.90.71.45             0    100      0 i
                    RT:65000:1002 ET:8 Default Gateway
 *>i[2]:[0]:[48]:[7e:5b:5a:51:2a:36]:[128]:[fe80::7c5b:5aff:fe51:2a36]
                    100.90.71.45             0    100      0 i
                    RT:65000:1002 ET:8 Default Gateway ND:Router Flag
Route Distinguisher: 172.18.0.12:4
 *>i[2]:[0]:[48]:[7e:5b:5a:51:2a:36]:[32]:[172.18.0.12]
                    100.90.71.45             0    100      0 i
                    RT:65000:1003 ET:8 Default Gateway
 *>i[2]:[0]:[48]:[7e:5b:5a:51:2a:36]:[128]:[fe80::7c5b:5aff:fe51:2a36]
                    100.90.71.45             0    100      0 i
                    RT:65000:1003 ET:8 Default Gateway ND:Router Flag

Let’s try some pings.

vyos@Spoke1:~$ ping 172.17.0.1 count 1
64 bytes from 172.17.0.1: icmp_seq=1 ttl=64 time=3.28 ms
1 packets transmitted, 1 received, 0% packet loss, time 0ms

vyos@Spoke1:~$ ping 172.17.0.12 count 1
64 bytes from 172.17.0.12: icmp_seq=1 ttl=64 time=3.07 ms
1 packets transmitted, 1 received, 0% packet loss, time 0ms

Pings are looking fine, let’s configure a LAN host so we can fully test stretching layer 2 between the site.

Configuring a LAN host

We already have ‘eth1’ as part of our bridge, so we can connect a host directly to that. For my testing, I’m going to use a Cisco router image, but feel free to use whatever you would like.

site1-host:
interface Ethernet0/0.20
 encapsulation dot1Q 20
 ip address 172.17.0.111 255.255.255.0
!
interface Ethernet0/0.30
 encapsulation dot1Q 30
 ip address 172.18.0.111 255.255.255.0

site2-host:
interface Ethernet0/0.20
 encapsulation dot1Q 20
 ip address 172.17.0.122 255.255.255.0
!
interface Ethernet0/0.30
 encapsulation dot1Q 30
 ip address 172.18.0.122 255.255.255.0

Let’s test some pings from our site1-host.

site1-host#ping 172.17.0.1
.!!!!
Success rate is 80 percent (4/5), round-trip min/avg/max = 1/1/3 ms

site1-host#ping 172.17.0.12
.!!!!
Success rate is 80 percent (4/5), round-trip min/avg/max = 1/1/2 ms

site1-host#ping 172.18.0.1 
.!!!!
Success rate is 80 percent (4/5), round-trip min/avg/max = 1/1/2 ms

site1-host#ping 172.18.0.12
.!!!!
Success rate is 80 percent (4/5), round-trip min/avg/max = 1/2/3 ms

Pings work fine, let’s see if our 2 sites can build a peering over multicast, this will confirm that it is a true L2VPN.

site1-host:
interface Ethernet0/0.20
 ip ospf network point-to-point
 ip ospf 100 area 0

site2-host:
interface Ethernet0/0.20
 ip ospf network point-to-point
 ip ospf 100 area 0

*Mar 11 18:33:12.777: %OSPF-5-ADJCHG: Process 100, Nbr 172.18.0.111 on Ethernet0/0.20 from LOADING to FULL, Loading Done

Everything is working great, but there’s a few things we want to add to optimize our design.

Optimizations

We’re deploying VxLAN in concert with EVPN, but you could configure VxLAN to operate alone with traditional flood and learn behavior. Since we are using EVPN for our control plane, there’s a couple of optimizaitons we can apply.

nolearning

Since we’re using BGP to advertise MAC addresses between sites, we don’t need to have Dynamic MAC Learning enabled. We can disable it with this command:

set interfaces vxlan vxlan0 parameters nolearning

neighbor-suppress

Since EVPN will learn not just a remote MAC address, but also the IP that maps to that MAC, you don’t need to forward ARP (or ND for IPv6) messages to attempt to resolve MACs to IP addresses. You can just use the local EVPN table to have the local node respond to LAN side clients. You can configure it using this command:

set interfaces vxlan vxlan0 parameters neighbor-suppress

A quick note on L2VPNs over the internet

Whenever you stretch Layer2 traffic across a routed infrastructure, you need to ensure you follow all of the proper design hygiene you would typically follow in a flat layer 2 network. Ensure you don’t allow traffic to loop, and make sure you limit the impact of any broadcast storms.

Conclusion

That concludes this one. One of the greatest things about VxLAN is that you can use the same solution for both L3VPNs an L2VPNs.

In the next post, I’m going to be going over multitenancy in our VxLAN deployment.