In this series, we opted for ZeroTier due to its straightforward configuration process for multipoint VPNs, which perfectly suited our need for dynamic spoke-to-spoke configurations. But there’s more to ZeroTier. It can function as a versatile bolt-on SD-WAN solution, particularly for systems like VyOS. This integration provides advanced features like intelligent failover and sophisticated load-balancing, enhancing the efficiency and reliability of our network traffic management.

Path Selection

ZeroTier, with its out-of-the-box functionality, excels in path selection without requiring any user intervention. When multiple paths exist between endpoints, ZeroTier actively works to identify and select the most efficient route. For example, consider a scenario with two nodes connected through different channels – like an internet connection and a P2P or Private IP service. ZeroTier will intelligently analyze these options and determine the optimal path for communication, ensuring seamless and efficient connectivity between the devices.

Let’s build a quick lab to demo this:

Topology

In our setup, each device is configured with four internet paths. On one of these devices, I implemented packet filters to introduce varying degrees of latency on each link. The latency increments by 10ms for each subsequent link, starting from 10ms on the first link and increasing to 20ms, 30ms, and finally 40ms on the others. This latency is bidirectional, meaning a 10ms delay on one link results in a total of 20ms Round-Trip Time (RTT) latency, accounting for both the outbound and inbound journey.

Given that all default paths are set to equal cost, conducting a ping to random IP addresses on the internet reveals interesting behavior. Due to this equal cost configuration, the network’s hashing mechanism will randomly select any of the four paths for the internet traffic.

root@vyos:/home/vyos# show ip route
S>* 0.0.0.0/0 [210/0] via 10.0.101.1, eth0, weight 1, 00:20:03
  *                   via 10.0.102.1, eth1, weight 1, 00:20:03
  *                   via 10.0.103.1, eth2, weight 1, 00:20:03
  *                   via 10.0.104.1, eth3, weight 1, 00:20:03

root@vyos:/home/vyos# ping 4.2.2.3
64 bytes from 4.2.2.3: icmp_seq=1 ttl=56 time=26.4 ms --> Path with 10ms of added latency

root@vyos:/home/vyos# ping 1.1.1.1
64 bytes from 1.1.1.1: icmp_seq=1 ttl=56 time=48.1 ms --> Path with 20ms of added latency

root@vyos:/home/vyos# ping 4.2.2.5
64 bytes from 4.2.2.5: icmp_seq=1 ttl=56 time=65.8 ms --> Path with 30ms of added latency

root@vyos:/home/vyos# ping 4.2.2.2
64 bytes from 4.2.2.2: icmp_seq=1 ttl=56 time=86.9 ms --> Path with 40ms of added latency

Redundancy is great, but we really should be taking that path with only 20ms of additional latency. Let’s install some ZeroTier and try ping the ZeroTier IP of the other node:

vyos@vyos# run ping 10.13.195.149
PING 10.13.195.149 (10.13.195.149) 56(84) bytes of data.
64 bytes from 10.13.195.149: icmp_seq=1 ttl=64 time=22.7 ms

You can see it just picks the lowest latency path without us needing to do anything. Let’s test failover by suspending the lowest latency path between the 2 nodes:

vyos@vyos# run ping 10.13.195.149
PING 10.13.195.149 (10.13.195.149) 56(84) bytes of data.
64 bytes from 10.13.195.149: icmp_seq=66 ttl=64 time=22.7 ms
64 bytes from 10.13.195.149: icmp_seq=85 ttl=64 time=48.0 ms

You can see from the sequence numbers, about 19 pings were dropped before the failover occurred. It’s important to know that the default route out of the primary path is still available, so the node not taking that path is a ZeroTier action. Normally, traffic could be blackholed in this scenario. Let’s keep peeling back the onion and suspend our next best path.

vyos@vyos# run ping 10.13.195.149
PING 10.13.195.149 (10.13.195.149) 56(84) bytes of data.
64 bytes from 10.13.195.149: icmp_seq=5 ttl=64 time=47.3 ms
64 bytes from 10.13.195.149: icmp_seq=19 ttl=64 time=62.3 ms

We dropped 14 pings while waiting for the path to failover. You can see that ZeroTier can help us make our traffic more predictable when we have redundant circuits.

Private Root Servers

Throughout this series, we’ve touched on the fact that initial traffic in a ZeroTier network typically passes through ZeroTier’s root servers. While this might initially sound concerning, it’s important to note that the traffic remains encrypted end-to-end between your nodes and isn’t decrypted within ZeroTier’s infrastructure. However, there might be scenarios where you desire more direct control over where your traffic goes. Perhaps you’re managing a site without internet access, or you might prefer having root servers geographically closer to you if ZeroTier’s servers are not in a convenient location. Whatever the motivation, these situations could lead you to consider deploying your own root servers for a more tailored network routing solution.

Deploying a Private Root

Deploying Private Roots for ZeroTier is pretty easy, it uses the same zerotier-one installations we’ve already been using.

We can even run them on a VyOS instance to keep things consistent. Let’s build a lab for this:

Topology

We’re going to have our basic Hub and 2 Spoke toplogy. To that, we’re going to add 2 additional VyOS instances to run our Roots.

NOTE: In some ZeroTier documentation, Private Roots are referred to as Moons. The latest documentation for Private Roots can be found here: https://docs.zerotier.com/roots/

One thing we’re going to do for this lab, is block our Hub and Spokes from accessing ZeroTier’s public roots. A list of their public roots can be found here: https://zerotier.atlassian.net/wiki/spaces/SD/pages/7241732/Root+Server+IP+Addresses

set firewall group address-group ZT_ADDR address '104.194.8.134'
set firewall group address-group ZT_ADDR address '103.195.103.66'
set firewall group address-group ZT_ADDR address '50.7.252.138'
set firewall group address-group ZT_ADDR address '84.17.53.155'
set firewall group address-group ZT_ADDR address '107.170.197.14'
set firewall ipv4 output filter default-action 'accept'
set firewall ipv4 output filter rule 10 action 'drop'
set firewall ipv4 output filter rule 10 destination group address-group 'ZT_ADDR'

We can see that our nodes can’t reach any of the ZeroTier roots. This will prevent any of our traffic from working:

vyos@vyos# sudo podman container exec zt1 zerotier-cli peers
200 peers
<ztaddr>   <ver>  <role> <lat> <link>   <lastTX> <lastRX> <path>
62f865ae71 -      PLANET    -1 RELAY
778cde7190 -      PLANET    -1 RELAY
cafe04eba9 -      PLANET    -1 RELAY
cafe9efeb9 -      PLANET    -1 RELAY

Configuring Our Roots

We’re going to configure 2 roots in a cluster. The initial step in this process involves creating an identity for our Root server. This is accomplished using the ‘zerotier-idtool’ utility, a tool used to create ZeroTier identities. This utility can be executed on either of our root servers.

First we need to know the identity of our roots. To determine the identity, we can use the “zerotier-cli info -j” command. This command, when executed, will reveal the unique identity of our node.

zerotier-cli info -j | grep publicIdentity
 "publicIdentity": "abcdefghij:0:0123456789abcdefghijklmnopqrstuvwxyz0123456789abcdefghijklmnopqrstuvwxyz0123456789abcdefghijklmnopqrstuvwxyz0123456789abcdefghij",

Now we can run the following command:

zerotier-idtool initmoon abcdefghij:0:0123456789abcdefghijklmnopqrstuvwxyz0123456789abcdefghijklmnopqrstuvwxyz0123456789abcdefghijklmnopqrstuvwxyz0123456789abcdefghij >>moon.json

You’ll have a file called moon.json that should look like this:

{
 "id": "abcdefghij",
 "objtype": "world",
 "roots": [
  {
   "identity": "abcdefghij:0:0123456789abcdefghijklmnopqrstuvwxyz0123456789abcdefghijklmnopqrstuvwxyz0123456789abcdefghijklmnopqrstuvwxyz0123456789abcdefghij",
   "stableEndpoints": []
  }
 ],
 "signingKey": "<there will be secrets here; make sure you don't share these>",
 "signingKey_SECRET": "<there will be secrets here; make sure you don't share these>",
 "updatesMustBeSignedBy": "<there will be secrets here; make sure you don't share these>",
 "worldType": "moon"
}	

You will have some secrets in here that you’ll want to make sure you maintain positive control of. We’ll need to add any other nodes we want to add to this cluster. Get the publicIdentity of Root2 and add it to the moon.json file. We can also configure the stableEndpoints section at this time:

{
 "id": "abcdefghij",
 "objtype": "world",
 "roots": [
  {
   "identity": "abcdefghij:0:0123456789abcdefghijklmnopqrstuvwxyz0123456789abcdefghijklmnopqrstuvwxyz0123456789abcdefghijklmnopqrstuvwxyz0123456789abcdefghij",
   "stableEndpoints": ["10.0.95.80/9993"]
  },
  {
   "identity": "klmnopqrst:0:abcdefghijklmnopqrstuvwxyz0123456789abcdefghijklmnopqrstuvwxyz0123456789abcdefghijklmnopqrstuvwxyz0123456789abcdefghijklmnopqrst",
   "stableEndpoints": ["10.0.95.71/9993"]
  }
 ],
 "signingKey": "<there will be secrets here; make sure you don't share these>",
 "signingKey_SECRET": "<there will be secrets here; make sure you don't share these>",
 "updatesMustBeSignedBy": "<there will be secrets here; make sure you don't share these>",
 "worldType": "moon"
}	

Now we need to sign our moon.json file. We can do this with the following command:

zerotier-idtool genmoon moon.json

It should create a signed .moon file:

-rw-r--r-- 1 root         zerotier-one   338 Jan 16 19:59 000000abcdefghij.moon

We need to create a folder for our moons, and place our signed .moon file in there:

mkdir -p /var/lib/zerotier-one/moons.d

cp 000000abcdefghij.moon /var/lib/zerotier-one/moons.d/

Restart ZeroTier on Root1, and check the peers to see if you can see our new moon in there.

zerotier-cli peers
200 peers
<ztaddr>   <ver>  <role> <lat> <link>   <lastTX> <lastRX> <path>
62f865ae71 -      PLANET   234 DIRECT   -1       1578     50.7.252.138/9993
778cde7190 -      PLANET    32 DIRECT   -1       1780     103.195.103.66/9993
abcdefghij 1.12.2 MOON      68 DIRECT   1743     1743     10.0.95.71/9993
cafe04eba9 -      PLANET   107 DIRECT   -1       1705     84.17.53.155/9993
cafe9efeb9 -      PLANET    68 DIRECT   -1       1744     104.194.8.134/9993

Now let’s get the second root up and running. For that, all we need to do is transfer the signed .moon file to the /var/lib/zerotier-one/moons.d folder like we just did. We can spin up a quick http server to get the file to our other devices. Run this from the moons.d folder in Root1:

python3 -m http.server 8123

From within VyOS, you can copy from our http server to the moons.d directory. Make sure you’ve created the moons.d directory first.

vyos@vyos# run copy file http://10.0.95.80:8123/0000008e5d56cff6.moon to /config/containers/zt1/moons.d/
Downloading...
The file is 0.330 KiB.
[##########################################################################################################################################################################################%
Download complete.

Restart Root2’s ZeroTier, and you should see that it also now shows a moon in its peers.

vyos@vyos:~$ sudo podman container exec zt1 zerotier-cli peers
200 peers
<ztaddr>   <ver>  <role> <lat> <link>   <lastTX> <lastRX> <path>
62f865ae71 -      PLANET   236 DIRECT   18140    153576   50.7.252.138/9993
778cde7190 -      PLANET    28 DIRECT   18140    153784   103.195.103.66/9993
8e5d56cff6 1.12.2 MOON       1 DIRECT   3644     3640     10.0.95.80/9993
cafe04eba9 -      PLANET   107 DIRECT   18140    153704   84.17.53.155/9993
cafe9efeb9 -      PLANET    72 DIRECT   18140    153738   104.194.8.134/9993

Now you can copy the file to our Hub and Spokes following that same process, and restart the ZeroTier nodes. We should now see the moons listed in our ZeroTier peers

# zerotier-cli peers
200 peers
<ztaddr>   <ver>  <role> <lat> <link>   <lastTX> <lastRX> <path>
62f865ae71 -      PLANET    -1 RELAY
778cde7190 -      PLANET    -1 RELAY
abcdefghij 1.12.2 MOON       4 DIRECT   22630    7607     10.0.95.80/22775
klmnopqrst 1.12.2 MOON       1 DIRECT   2598     2598     10.0.95.71/50393
cafe04eba9 -      PLANET    -1 RELAY
cafe9efeb9 -      PLANET    -1 RELAY

ZeroTier states the below message in their guide, but I have found that you need to run the orbit command either way. What they state may work fine if you are just trying to add your own Roots in addition to theirs, but it doesn’t appear to hold true when your nodes can only reach your private roots.

“You can add these roots to regular nodes in one of two ways: by placing the same world definition file in their moons.d directories or by using the zerotier-cli orbit command: zerotier-cli orbit deadbeef00 deadbeef00. The first argument is the world ID (which we can shorten by removing the two leading zeroes) and the second is the address of any of its roots. This will contact the root and obtain the full world definition from it if it’s online and reachable.”

https://docs.zerotier.com/roots/

I run this command below to orbit our new moon.

zerotier-cli orbit abcdefghij

Notice that I didn’t put the second address when orbiting. This should mean we orbit the full moon cluster, and not just a single node in the cluster. After performing these actions, we should see that our traffic is working. We can see this in BGP being up now:

vyos@vyos:~$ show ip bgp summary 
Neighbor        V         AS   MsgRcvd   MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd   PfxSnt Desc
*10.13.0.12     4      65000         5         6        2    0    0 00:01:56            1        2 N/A
*10.13.153.97   4      65000         5         6        2    0    0 00:01:54            1        2 N/A

You can deploy additional root clusters if desired. Private Roots take very little resources, so you really just need to make sure they’re somewhere with stable connections.

Conclusion

There’s still loads more to ZeroTier, but you can see how powerful ZeroTier can be in this solution.