In this post, we’re going to create a Dynamic Multipoint Solution using ZeroTier and VyOS. When you hear Dynamic Multipoint, you probably immediately think of Cisco’s DMVPN. DMVPN is a widely deployed and proven solution, but it ultimately is a Cisco solution meant for Cisco environments.

A open variety of DMVPN was created called OpenNHRP, which does work and is supported by VyOS, but any extensibility of it like is offered in Cisco’s DMVPN (MPLS, IWAN) isn’t present in the open variant.

Why ZeroTier

I guess before the why, we should identify what ZeroTier is. ZeroTier’s website says this:

ZeroTier lets you build modern, secure multi-point virtualized networks of almost any type. From robust peer-to-peer networking to multi-cloud mesh infrastructure, we enable global connectivity with the simplicity of a local network.

What that ultimately means is that it creates a “potentially” full mesh of devices in an emulated layer 2 network.

Back to the why? By using ZeroTier to create our VPNs between sites, we can leverage any normal network function that can operate in an ethernet network. This means there’s almost limitless extensibility. You can even bridge the network to physical gear from Cisco, Juniper, etc…

Why VyOS

VyOS uses FRR for it’s advanced network functions, so realistically you can use almost any distro that you can install FRR on. I’m a fan of VyOS, so it’s what I went with here. We’ll be using VyOS 1.4.0-rc1 in this lab.

The 1.4.0-rc1 VyOS ISO can be downloaded from this blog post: https://blog.vyos.io/vyos-1.4.0-rc1-release-candidate

Our Topology

All devices will be connected to the internet to build our ZeroTier tunnels. ZeroTier does offer some options like self-hosting a controller and root server(s) to move a lot of this function away from ZeroTier Central, but all of your data is encrypted end-to-end.

Installing VyOS

After downloading the VyOS ISO from the VyOS website, we need to boot it. VyOS is very lightweight and can be run on almost any x86 system that you may have. You can fully run it as a live-CD if desired, though any configuration won’t be persistent.

To make it persistent, we’ll install the image to a disk using the “install image” command. It will run you through a guided install process:

vyos@vyos:~$ install image
Welcome to VyOS installation!
This command will install VyOS to your permanent storage.
Would you like to continue? [y/N] y
What would you like to name this image? (Default: 1.4.0-rc1) 
Please enter a password for the "vyos" user (Default: vyos) 
What console should be used by default? (K: KVM, S: Serial, U: USB-Serial)? (Default: K) 
Probing disks
1 disk(s) found
The following disks were found:
Drive: /dev/sda (8.0 GB)
Which one should be used for installation? (Default: /dev/sda) 
Installation will delete all data on the drive. Continue? [y/N] y
Searching for data from previous installations
No previous installation found
Would you like to use all the free space on the drive? [Y/n] Y
Creating partition table...
Creating temporary directories
Mounting new partitions
Creating a configuration file
Copying system image files
Installing GRUB configuration files
Installing GRUB to the drive
Cleaning up
Unmounting target filesystems
Removing temporary files
The image installed successfully; please reboot now.
vyos@vyos:~$ reboot now

Broadcast message from root@vyos on pts/0 (Mon 2024-01-08 15:39:07 UTC):

The system will reboot now!

After the reboot is finished, we need to get the nodes access to the internet. We’ll configure the WAN interface to get an IP and default via DHCP. Our WAN interface will be eth0. We’ll also configure a name-server so that our node can resolve hostnames.

vyos@vyos:~$ configure 
vyos@vyos# set interfaces ethernet eth0 address dhcp
vyos@vyos# set system name-server 4.2.2.2
vyos@vyos# commit
vyos@vyos# 

Once we have an IP and default (and are able to resolve hostnames), then we can move on to installing ZeroTier:

vyos@vyos# run show ip route
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, F - PBR,
       f - OpenFabric,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup
       t - trapped, o - offload failure

S>* 0.0.0.0/0 [210/0] via 10.0.95.1, eth0, weight 1, 00:00:01
C>* 10.0.95.0/24 is directly connected, eth0, 00:00:01

vyos@vyos# run ping www.google.com
PING www.google.com (172.253.62.106) 56(84) bytes of data.
64 bytes from bc-in-f106.1e100.net (172.253.62.106): icmp_seq=1 ttl=110 time=9.35 ms

vyos@vyos# sudo su
root@vyos:/home/vyos# curl -s https://install.zerotier.com | sudo bash

ZeroTier will be installed, and if everything worked correctly, you should be presented with something like this:

*** Success! You are ZeroTier address [ xxxxxxxxxx ].

You’ll need to sign up for a ZeroTier Central account in order to create networks to join.

Once you have an account, you’ll need to create a network.

The network will autopopulate a name and subnet, but we can edit that by clicking on the network that was created.

I made the subnet for this lab 10.13.0.0/16, with the entire range eligible for auto assignment of IPs:

Before we join a network, we may want to map our ZeroTier interface to an ethernet interface. It isn’t strictly necessary, but by doing so, we can do additional configuration on the interface like applying a firewall filter to it. We do this by making a devicemap, and then restarting the ZeroTier service.

The devicemap file should be created in /var/lib/zerotier-one. I created eth10 here, but it could have been anything that doesn’t already exist on the node.

vyos@vyos# sudo su
root@vyos:/home/vyos# cd /var/lib/zerotier-one
root@vyos:/var/lib/zerotier-one# cat >devicemap
# Example: <networkID>=<interface>
xxxxxxxxxxxxxxxx=eth10
^C

root@vyos:/var/lib/zerotier-one# cat devicemap 
# Example: <networkID>=<interface>
xxxxxxxxxxxxxxxx=eth10

root@vyos:/var/lib/zerotier-one# sudo /etc/init.d/zerotier-one restart
Restarting zerotier-one (via systemctl): zerotier-one.service.

We can now join our ZeroTier network.

root@vyos:/var/lib/zerotier-one# zerotier-cli join xxxxxxxxxxxxxxxx
200 join OK

Before we can get an IP address from ZeroTier, we’ll need to authorize the ZeroTier node in ZeroTier Central

Once Authorized, the nodes will get an IP assigned to them. It may also be wise to give them names to keep track of what they are.

It auto assigns IPs from the large block we defined earlier, but we can statically assign IPs to it if we want (we can also have more than one IP on the interface if we wanted).

If we did everything correctly, we should have an IP assigned by ZeroTier on the interface we defined in the devicemap (eth10 in this case)

vyos@Hub:~$ show interfaces 
Codes: S - State, L - Link, u - Up, D - Down, A - Admin Down
Interface    IP Address     MAC                VRF        MTU  S/L    Description
-----------  -------------  -----------------  -------  -----  -----  -------------
eth0         10.0.95.70/24  0c:bd:51:e2:00:00  default   1500  u/u
eth1         -              0c:bd:51:e2:00:01  default   1500  u/u
eth2         -              0c:bd:51:e2:00:02  default   1500  u/D
eth10        10.13.0.1/16   aa:bb:cc:dd:ee:f0  default   2800  u/u
lo           127.0.0.1/8    00:00:00:00:00:00  default  65536  u/u

You’ll need to repeat this process for as many nodes as you want to add to your topology. Let’s get to configuring VyOS

Configuring VyOS

When deciding on how we want to do the routing, we have a few options we need to weigh.

• ZeroTier has a managed routes section where we can add routes which could make this very easy.
  • We can just add routes here and be done with everything, but there is a potential drawback in doing this. These routes will not have any kind of health check like would be present in traditional routing protocols. So the route would be present even if it is not reachable via that 10.13.0.1 IP.
  • Additionally, this won’t lead well to a solution where we want to have multiple nodes supporting a site for redundancy, or equal cost multipathing.
• EIGRP/IS-IS/OSPF/RIP
  • All of these will send out multicast hellos that will be received by all nodes. While this will solve the multipoint part of our solution, it won’t solve for the dynamic portion, since it will build a full mesh between all nodes. In a small environment, this won’t be an issue, but in a large deployment, it won’t scale to lower powered devices.
  • You can still acheive dynamic VPNs using this if you filter the hellos from the spokes to only send to the Hub, but that isn’t necessarily a great solution.
• BGP
  • In this lab, we’re going to use BGP. Since BGP builds all of its neighbors using unicast TCP messages, it is well suited for building our routing table. It is also incredibly granular, extensible, and scalable, which can be very useful in Hub/Spoke topologies like this.
  • We’re going to use iBGP with route-reflection since it will already have next-hop-unchanged as the default behavior, which is what will allow for dynamic spoke-to-spoke communications.

Building the Hub BGP Configuration

The Hub can potentially have hundreds (even thousands) of BGP peerings, so we need a dynamic way to create BGP peerings. We can use a BGP Listener that attaches to a peer-group. We’ll also configure the LAN interface now:

vyos@Hub:~$ configure 
vyos@Hub# set protocols bgp system-as 65000
vyos@Hub# set protocols bgp listen range 10.13.0.0/16 peer-group PG
vyos@Hub# set protocols bgp peer-group PG remote-as 65000
vyos@Hub# set protocols bgp peer-group PG address-family ipv4-unicast route-reflector-client 
vyos@Hub# set protocols bgp address-family ipv4-unicast network 10.0.1.0/24
vyos@Hub# set interfaces ethernet eth1 address 10.0.1.1/24

Now we can configure the spokes. Just like with DMVPN, this is very scalable because the configuration is almost identical for all spokes. If you redistributed connected routes instead of using a network statement, you could make them identical. If you did that, you’d want to apply a route-map on the redistribute command to make sure the Hub doesn’t get multiple copies of the 10.13.0.0/16 route (it would get one from each spoke).

vyos@Spoke1:~$ configure 
vyos@Spoke1# set protocols bgp system-as 65000
vyos@Spoke1# set protocols bgp neighbor 10.13.0.1 remote-as 65000
vyos@Spoke1# set protocols bgp neighbor 10.13.0.1 address-family ipv4-unicast nexthop-self 
vyos@Spoke1# set protocols bgp address-family ipv4-unicast network 10.0.11.0/24
vyos@Spoke1# set interfaces ethernet eth1 address 10.0.11.1/24
vyos@Spoke1# commit

vyos@Spoke2:~$ configure 
vyos@Spoke2# set protocols bgp system-as 65000
vyos@Spoke2# set protocols bgp neighbor 10.13.0.1 remote-as 65000
vyos@Spoke2# set protocols bgp neighbor 10.13.0.1 address-family ipv4-unicast nexthop-self 
vyos@Spoke2# set protocols bgp address-family ipv4-unicast network 10.0.12.0/24
vyos@Spoke2# set interfaces ethernet eth1 address 10.0.12.1/24
vyos@Spoke2# commit

If you want to do ‘redistribute connected to make them identical, here is an example:

vyos@Spoke1# set policy prefix-list ZT_LAN_PFX rule 10 action 'permit'
vyos@Spoke1# set policy prefix-list ZT_LAN_PFX rule 10 prefix '10.13.0.0/16'
vyos@Spoke1# set policy route-map DENY_ZT_LAN rule 10 action 'deny'
vyos@Spoke1# set policy route-map DENY_ZT_LAN rule 10 match ip address prefix-list 'ZT_LAN_PFX'
vyos@Spoke1# set policy route-map DENY_ZT_LAN rule 20 action 'permit'

vyos@Spoke1# set protocols bgp address-family ipv4-unicast redistribute connected route-map 'DENY_ZT_LAN'
vyos@Spoke1# set protocols bgp neighbor 10.13.0.1 address-family ipv4-unicast nexthop-self
vyos@Spoke1# set protocols bgp neighbor 10.13.0.1 remote-as '65000'
vyos@Spoke1# set protocols bgp system-as '65000'

vyos@Spoke1# commit

We should now have peerings on the Hub. Notice the ‘*’ in front of the neighbors indicating that they’re dynamic neighbors.

vyos@Hub:~$ show ip bgp summary 

IPv4 Unicast Summary (VRF default):
BGP router identifier 10.13.0.1, local AS number 65000 vrf-id 0
BGP table version 15
RIB entries 5, using 960 bytes of memory
Peers 2, using 41 KiB of memory
Peer groups 1, using 64 bytes of memory

Neighbor        V         AS   MsgRcvd   MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd   PfxSnt Desc
*10.13.0.11     4      65000         4         6       15    0    0 00:00:02            1        3 N/A
*10.13.0.12     4      65000         4         6       15    0    0 00:00:02            1        3 N/A

Total number of neighbors 2
* - dynamic neighbor
2 dynamic neighbor(s), limit 100

And on the spokes, we should have routes to the other spokes, with the next-hop-unchanged. Since the next-hop is unchanged, the traffic will try to go directly to the owner of the destination IP (the other Spoke in this example). Like any other ethernet traffic, the node will attempt to ARP for the destination IP.

vyos@Spoke1:~$ show ip bgp
BGP table version is 13, local router ID is 10.13.0.11, vrf id 0
Default local pref 100, local AS 65000
Status codes:  s suppressed, d damped, h history, * valid, > best, = multipath,
               i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes:  i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

    Network          Next Hop            Metric LocPrf Weight Path
 *>i10.0.1.0/24      10.13.0.1                0    100      0 i
 *> 10.0.11.0/24     0.0.0.0                  0         32768 i
 *>i10.0.12.0/24     10.13.0.12               0    100      0 i

Displayed  3 routes and 3 total paths

On Spoke1, you can see that there is not a resolved ARP in the arp table for Spoke2 (10.13.0.12):

vyos@Spoke1:~$ show arp | match 10.13
10.13.0.1       eth10        aa:bb:cc:dd:ee:f0     REACHABLE

If we look at ZeroTier’s CLI, we can also see that we don’t have a direct path to Spoke2 built, only a direct path to the Hub (created when BGP was initiated). Initial traffic will go through ZeroTier’s root servers (remember, it’s end-to-end encrypted).

vyos@Spoke1:~$ sudo zerotier-cli peers
Hub        1.12.2 LEAF       2 DIRECT   3216     3215     10.0.95.70/9993

Let’s try to ping Spoke2’s LAN (10.0.12.1)

vyos@Spoke1:~$ ping 10.0.12.1 source-address 10.0.11.1
PING 10.0.12.1 (10.0.12.1) from 10.0.11.1 : 56(84) bytes of data.
64 bytes from 10.0.12.1: icmp_seq=1 ttl=64 time=621 ms
64 bytes from 10.0.12.1: icmp_seq=2 ttl=64 time=1.65 ms
64 bytes from 10.0.12.1: icmp_seq=3 ttl=64 time=1.77 ms

We can see that the first ping is very latent. This is because ARP is being resolved, and the initial ping will traverse ZeroTier’s Root Servers. The subsequent pings go directly to Spoke2. You can see we now have an ARP resolution.

vyos@Spoke1:~$ show arp | match 10.13
10.13.0.1       eth10        aa:bb:cc:dd:ee:f0     REACHABLE
10.13.0.12      eth10        aa:bb:cc:dd:ee:f1     REACHABLE

We can see our connection is direct in the ZeroTier CLI as well:

vyos@Spoke1:~$ sudo zerotier-cli peers
Hub        1.12.2 LEAF       2 DIRECT   2538     2535     10.0.95.70/9993
Spoke2     1.12.2 LEAF      10 DIRECT   12531    17555    10.0.95.57/9993

You can repeat everything to add as many spokes as you want to. You can also add one or more additional Hubs for redundancy.

Conclusion

That’s it for this post. This will ultimately be a multi-part post. In part 2, we will add multi-tenancy to this solution using MPLS.

Video

My friend is turning this series into videos on YouTube: