Using ZeroTier for Site-to-Site Routing on VyOS

Previously, I’ve written a number of articles on how to use ZeroTier to create an enterprise ready SD-WAN solution for VyOS. This included using BGP to ensure that ZeroTier could scale to a large number of sites, as well as achieve multitenancy through MPLS.

If you have a smaller deployment, and don’t need advanced features like multitenancy, you can actually have ZeroTier handle the site-to-site routing for you, which simplifies the deployment for a large number of users.

Our Topology

We will have 5 sites with a single router. All routing between sites will be handled by ZeroTier directly. We won’t need any additional protocols like OSPF or BGP. I named all nodes “RTR<n>”.

Creating a ZeroTier Account

Before we can deploy ZeroTier, you’ll need to create a ZeroTier account if you don’t already have one.

NOTE: You can self-host ZeroTier if you want, but that is beyond the scope of this article.

First we need to go to the ZeroTier Central website at :
https://my.zerotier.com

You’ll need to sign up for a ZeroTier Central account in order to create networks to join.

Once you have an account, you’ll need to create a network.

The network will autopopulate a name and subnet, but we can edit that by clicking on the network that was created.

I made the subnet for this lab 10.13.0.0/16, with the entire range eligible for auto assignment of IPs:

Configuring VyOS

For this lab, we’re going to be using 1.4.0-rc3, which you can grab from this blog post:
https://blog.vyos.io/vyos-1.4.0-rc3-release-candidate

VyOS will need to have a path to the internet as well as the ability to resolve DNS to get everything setup.

configure 
set interfaces ethernet eth0 address dhcp
set system name-server 4.2.2.2
commit

If you’re able to ping something like google, you should be fine.

vyos@vyos:~$ ping www.google.com
PING www.google.com (142.251.167.106) 56(84) bytes of data.
64 bytes from ww-in-f106.1e100.net (142.251.167.106): icmp_seq=1

ZeroTier Container

We’re going to install ZeroTier in a container. This makes for the easiest management of additional services added to VyOS. First we need to pull down the image.

vyos@vyos:~$ add container image zerotier/zerotier:latest
vyos@vyos:~$

In 1.4, the download is entirely silent, so it may not seem like it’s doing anything, but it should be pulling the image. You can verify that time image was pulled currently with “show container image”.
vyos@RTR5# run show container image 
REPOSITORY                   TAG         IMAGE ID      CREATED      SIZE
docker.io/zerotier/zerotier  latest      5cad1d053743  6 weeks ago  193 MB

Next, we want to make a directory so that our configuration will be persistent if we need to restart our container (or restart VyOS). I make my folder in /config/containers/, but you can do anything as long as it is within /config/.

sudo mkdir -p /config/containers/zt1

Now it’s time to apply our container config.

configure
set container name zt1 allow-host-networks
set container name zt1 cap-add 'net-admin'
set container name zt1 device tun destination '/dev/net/tun'
set container name zt1 device tun source '/dev/net/tun'
set container name zt1 image 'zerotier/zerotier:latest'
set container name zt1 volume ZT_Path destination '/var/lib/zerotier-one'
set container name zt1 volume ZT_Path source '/config/containers/zt1'
commit

Let’s go over some of these commands:

allow-host-networks – This tells the container to use VyOS as it’s network. ZeroTier will create an interface that VyOS can use for routing, so it needs to be on the same network as the host, and not its own container network.
cap-add ‘net-admin’ – ZeroTier will be creating an interface on the host; this capability allows the container to do this.
device tun – This will allow the container to connect to the TUN/TAP driver in linux
image ‘zerotier/zerotier:latest’ – This tells the container which image to use. This is the image we added with the “add container image” command earlier.
volume ZT_Path – To ensure that the container’s configuration is persistent, it needs to map to a volume on the host. Destination is the volume on the container. Source is the volume on the host, which we created previously with the “mkdir” command. Anything that the container puts in its “/var/lib/zerotier-one” directory will actually be VyOS’ “/config/containers/zt1” directory.

You can verify the status of the container with the below command:

vyos@vyos:~$ show container 
CONTAINER ID  IMAGE                               COMMAND     CREATED        STATUS            PORTS       NAMES
b82dd2bf5689  docker.io/zerotier/zerotier:latest              6 minutes ago  Up 6 minutes ago              zt1

Joining the ZeroTier Network

When you created the network earlier, a Network ID was created, which can be found at the top of the Network’s settings inside ZeroTier Central. You will need that value for the next steps.

Before we join a network, we may want to map our ZeroTier interface to an ethernet interface. It isn’t strictly necessary, but by doing so, we can do additional configuration on the interface like applying a firewall filter to it. We do this by making a devicemap, and then restarting the ZeroTier service.

The devicemap file should be created in “/config/containers/zt1” (or whatever you named your directory). The format of the string that needs to be created is <network-id>=<interface name>. I created eth10 here, but it could have been anything that doesn’t already exist on the node.

sudo su
cd /config/containers/zt1
cat >devicemap
0123456789abcdef=eth10
^C

You can verify what was created with “cat /config/containers/zt1/devicemap”.

We’re now ready to join the network. There’s a few ways that we can do this, but we’re going to do it from within the container using the “zerotier-cli join” command.

First, connect to the container using “connect container zt1”

vyos@RTR1:~$ connect container zt1

Next, from within the container type “zerotier-cli join <network-id>”.

# zerotier-cli join 0123456789abcdef
200 join OK

While you’re still connected to the container, go ahead and also type “zerotier-cli info”, since it’ll be useful to have the Node ID for the next steps.

Go ahead and repeat this process for the rest of your nodes.

Authenticating the nodes

Back in your network on ZeroTier Central, you should now see all of the nodes listed.

Before authenticating, let’s give a name to each node so we can track which nodes is which. Though I blanked out the information under address, what you’ll see is the same values you got when running the “zerotier-cli info” command from within the container. Match those up to what is in ZeroTier Central.

You can now authorize each node by selecting the box under the auth column:

Once authorized, each node should have an IP under the “Managed IPs” column.

Back on VyOS, we should see that our “eth10” interface has an IP assigned to it.

vyos@RTR1:~$ show interfaces ethernet 
Codes: S - State, L - Link, u - Up, D - Down, A - Admin Down
Interface        IP Address                        S/L  Description
---------        ----------                        ---  -----------
eth0             10.0.95.246/24                    u/u  
eth1             -                                 u/D  
eth2             -                                 u/D  
eth10            10.13.214.145/16                  u/u

Configuring Site-to-Site Networking

Before we can configure ZeroTier to handle our routing, we need to have routes for it to use.

For this lab, I’m going to use Dummy interfaces to simulate prefixes that each site will own, but this would work the same if they were functional connections at the site.

RTR1:
set interfaces dummy dum0 address 10.1.0.1/24

RTR2:
set interfaces dummy dum0 address 10.2.0.1/24

RTR3:
set interfaces dummy dum0 address 10.3.0.1/24

RTR4:
set interfaces dummy dum0 address 10.4.0.1/24

RTR5:
set interfaces dummy dum0 address 10.5.0.1/24

If we were to look at the current routing table for one of our nodes, it should be pretty simple. We should have our default route, and a couple of connected routes.

vyos@RTR1# run show ip route
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, F - PBR,
       f - OpenFabric,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup
       t - trapped, o - offload failure

S>* 0.0.0.0/0 [210/0] via 10.0.95.1, eth0, weight 1, 04:16:47
C>* 10.0.95.0/24 is directly connected, eth0, 04:16:47
C>* 10.1.0.0/24 is directly connected, dum0, 00:00:04
C>* 10.13.0.0/16 is directly connected, eth10, 00:05:23

We don’t yet have our routes to the other sites, but doing so can easily be accomplished from within the ZeroTier Central console. From within the Network’s settings page, find the “Managed Routes section”.

You will need to add the routes that each node owns, with the IP under the “Via” section being the IP that node was assigned.

Hit submit to add the routes to the managed routes. Do this for all 5 sites.

NOTE: ZeroTier Central limits the number of managed routes to 128.

If we look back on our routing table for one of our VyOS nodes, you can now see we have routes between all of the other sites.

vyos@RTR1# run show ip route
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, F - PBR,
       f - OpenFabric,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup
       t - trapped, o - offload failure

S>* 0.0.0.0/0 [210/0] via 10.0.95.1, eth0, weight 1, 04:23:18
C>* 10.0.95.0/24 is directly connected, eth0, 04:23:18
C>* 10.1.0.0/24 is directly connected, dum0, 00:06:35
K>* 10.2.0.0/24 [0/5000] via 10.13.205.20, eth10, 00:01:30
K>* 10.3.0.0/24 [0/5000] via 10.13.125.173, eth10, 00:01:21
K>* 10.4.0.0/24 [0/5000] via 10.13.153.91, eth10, 00:00:03
K>* 10.5.0.0/24 [0/5000] via 10.13.107.40, eth10, 00:00:03
C>* 10.13.0.0/16 is directly connected, eth10, 00:11:54

NOTE: ZeroTier sets the metric for all routes it pushes at 5000 to avoid conflicting with any routes local to the site.

Testing

Let’s test connectivity between our sites. From within RTR1, I’m going to ping each remote site, sourcing from that “dum0” interface.

RTR2:
vyos@RTR1# run ping 10.2.0.1 source-address 10.1.0.1 count 2
64 bytes from 10.2.0.1: icmp_seq=1 ttl=64 time=26.3 ms

RTR3:
vyos@RTR1# run ping 10.3.0.1 source-address 10.1.0.1 count 2
64 bytes from 10.3.0.1: icmp_seq=1 ttl=64 time=14.7 ms

RTR4:
vyos@RTR1# run ping 10.4.0.1 source-address 10.1.0.1 count 2
64 bytes from 10.4.0.1: icmp_seq=1 ttl=64 time=25.2 ms

RTR5:
vyos@RTR1# run ping 10.5.0.1 source-address 10.1.0.1 count 2
64 bytes from 10.5.0.1: icmp_seq=1 ttl=64 time=23.6 ms

Everything works! Adding additional routes is as simple as repeating that process. You can also do an aggregate route for the entire site rather than per prefix, allowing you to have up to 127 sites that ZeroTier can manage the routing for.

Conclusion

That is it for this post. Managing site-to-site networks securely using ZeroTier as a VPN solution can make the process simple for a lot of users. It removes a lot of the complexity of the networking, and allows you to quickly deploy new sites.