In previous articles, I created a multi-part series on Dynamic Multipoint VPNs using ZeroTier and VyOS. After postings these, I was asked if I could port the design to MikroTik. In this article, we’ll cover how to create a multipoint VPN solution using ZeroTier on MikroTik devices.

The VyOS series can be found here:

Dynamic Multipoint VPN with ZeroTier and VyOS

NOTE: Though the solution is a Dynamic Multipoint VPN, it should not be confused with DMVPN, which is a known solution using Multipoint GRE (mGRE), Next Hop Resolution Protocol (NHRP), and IPsec. I call the solution Zero Path VPN (ZVPN), but feel free to call it whatever you like.

What problem is this meant to solve

In traditional site-to-site VPNs, the implementation typically favors point-to-point solutions. Even modern VPNs like WireGuard are still primarily meant to be point-to-point solutions. This creates a configuration nightmare at scale. If you have 500 sites that need to communicate, you need to build a massive full-mesh configuration to connect each site.

You can of course build VPNs from remote sites back to a hub and route all of your traffic hub and spoke, but not only is that not an efficient use of bandwidth, but even though it is less config than a full mesh, it would still require a massive configuration effort as the Hub would need to know about all of the remote sites.

Some traditional networking solutions were developed to help solve this problem, some examples being DMVPN, ADVPN, and GETVPN, but these solutions are often either vendor only, or will often receive a vendor specific implementation that can struggle to play nice with other implementations.

This solution uses ZeroTier and iBGP. Provided your device can use both of those technologies, then you’re able to run this solution. Because BGP is arguably the most compatible routing protocol, your deployment can have devices running multiple network operating systems and interoperate just fine. We can even bridge ZeroTier down to enterprise gear and run BGP on them if needed.

Topology

Our topology will be fairly simple and be comprised of a single Hub and 2 Spokes.

Creating a ZeroTier Central account

The first step is to create an account and network in ZeroTier Central. If you already have an account and network, you can skip these steps.

We’ll first need to sign up for an account. Go to https://my.zerotier.com/ and register.

Once you’ve created an account and logged in, you’ll be presented with an option to Create A Network, select that option.

The network will autopopulate a name and subnet, but we can edit that by clicking on the network that was created. I’m going to call mine “Lab Network”, but feel free to name yours whatever you’d like.

I made the subnet for this lab 10.13.0.0/16, with the entire range eligible for auto assignment of IPs: You’re free to use whatever scheme you’d like to use.

Installing ZeroTier on MikroTik

Installing ZeroTier on MikroTik is pretty simple. We first need to download the ZeroTier package for the RouterOS version that we’re running. Let’s see what version we’re running:

WebUI

In the top right of the WebUI, select the dropdown and select “about”.

In the about screen, look for “Version”

CLI:

Type “/system resources print” to see similar output to the about screen in the WebUI:

[admin@Hub] > /system resource print
                   uptime: 1h27m36s
                  version: 7.13.3 (stable)
                      cpu: ARM

In both outputs, we can see that we’re on 7.13.3 with ARM architecture, so we need to download additional packages for ARM architecture running 7.13.3. We can download the “Extra Packages” here: https://mikrotik.com/download.

Download the Extra Packages for your software version, and extract the files to a folder on your computer.

We’re specifically going to need a file named something like “zerotier-7.13.3-arm.npk”.

In the WebUI, go the “Files” section located on the far left of the screen

Once in the Files menu, select the upload option, and upload the ZeroTier package that you previously extracted.

You should see the “.npk” file in your list of files.

Once it’s there, reboot your router from the menu.

After the reboot, we should see a new menu in the WebUI:

We should also see new options in the CLI.

[admin@Hub] > zerotier/
controller     interface     peer     add     comment     disable     edit     enable     export     find     print     remove     reset     set

Configuring ZeroTier

You can configure most of these in the WebUI, but I’m going to be moving to the CLI for the remainder of this guide. If you’re not comfortable with the CLI, feel free to configure everything in the WebUI, but it’ll be much quicker to configure all of our devices using the CLI since we can copy and paste for as many spokes as we want.

Let’s configure our ZeroTier Interface.

[admin@Hub] /zerotier/interface> add allow-managed=yes instance=zt1 name="Zero Path VPN" network=0123456789abcdef disabled=no

The network should be the value found inside of ZeroTier Central for the network you created earlier.

Now we need to enable the ZeroTier instance. We do that with this command:

[admin@Hub] > zerotier/enable zt1

We can verify that our ZeroTier is functioning with this:

[admin@Hub] > /zerotier/interface/print
# NAME           MAC-ADDRESS        NETWORK           STATUS
0 Zero Path VPN  00:11:22:33:44:55  0123456789abcdef ACCESS_DENIED

We can see that our status is “ACCESS_DENIED”. This is actually a good indicator. It means that our node has tried to communicate with ZeroTier Central, and is not authorized. Let’s go ahead and authorize it.

Once Authorized, the nodes will get an IP assigned to them. It may also be wise to give them names to keep track of what they are.

It auto assigns IPs from the large block we defined earlier, but we can statically assign IPs to it if we want (we can also have more than one IP on the interface if we wanted). I’m going to give the Hub a static IP of 10.13.0.1.

We should now have an IP in MikroTik. Let’s check:

[admin@Hub] > /ip/address/print
2 D 10.13.0.1/16     10.13.0.0     Zero Path VPN

Everything looks good, now we just need to add our other devices. We’re going to make Spoke1 10.13.0.11, and Spoke2 10.13.0.12. Once that’s complete, you should have something like this in ZeroTier Central.

Let’s make sure we can ping our other nodes. We’ll ping 10.13.0.11 (Spoke1) and 10.13.0.12 (Spoke2) from the Hub (10.13.0.1).

[admin@Hub] > ping 10.13.0.11 count=1
  0  10.13.0.11    56   64  3ms78us

[admin@Hub] > ping 10.13.0.12 count=1
  0  10.13.0.12    56   64  2ms997us

Yep, pings are working fine. Now we can finally get to configuring BGP.

Configuring BGP

We’ll start with the Hub. We’re going to do iBGP between the Hub and Spokes, with the Hub being a route-reflector.

[admin@Hub] /routing/bgp/connection> add address-families=ip as=65000 disabled=no local.address=10.13.0.1 listen=yes local.address=10.13.0.1 .role=ibgp-rr name=ZT_HUB remote.address=10.13.0.0/16 .as=65000 routing-table=main

Most of this is fairly standard. Some things to look at is that we’re setting the remote address to a subnet, and enabling “listen”. This means that any attempt by a peer coming from that range will be accepted, and replied to using 10.13.0.1. This solves the problem we discussed initially, where you would need to manually create a number of peers on a Hub.

NOTE: If you’re familiar with Cisco IOS, this may look a little foreign to you, but this is actually quite similar to how it is done on Cisco NX-OS.

Let’s configure the Spokes! These will just be pointed to that 10.13.0.1 subnet as route-reflector clients.

Spoke1:
[admin@Spoke1] /routing/bgp/connection> add address-families=ip as=65000 disabled=no local.address=10.13.0.11 local.role=ibgp-rr-client name=ZT_HUB remote.address=10.13.0.1 remote.as=65000 routing-table=main

Spoke2:
[admin@Spoke2] /routing/bgp/connection> add address-families=ip as=65000 disabled=no local.address=10.13.0.12 local.role=ibgp-rr-client name=ZT_HUB remote.address=10.13.0.1 remote.as=65000 routing-table=main

One of the greatest thing about designs like this, is that the configuration is almost identical for the Spokes. The only thing that changes is the local-address. We can easily deploy this to another 5 (or 500) spokes quickly if needed.

Let’s see if we have any peers on the Hub.

[admin@Hub] /routing/bgp/connection> /routing/bgp/session/print
     remote.address=10.13.0.11 .as=65000 .id=10.13.0.11 .capabilities=mp,rr,em,gr,as4,ap,err,llgr,fqdn .messages=4 .bytes=80 .gr-time=120 .eor=ip
     remote.address=10.13.0.12 .as=65000 .id=10.13.0.12 .capabilities=mp,rr,em,gr,as4,ap,err,llgr,fqdn .messages=2 .bytes=42 .gr-time=120 .eor=ip

Yep, we we have our 2 dynamic peerings up. Let’s go ahead and advertise some routes between our spokes.

Spoke1:
[admin@Spoke1] > /interface/bridge/add name=loopback0
[admin@Spoke1] > /ip/address/add address=10.0.0.11 interface=loopback0 network=10.0.0.11
[admin@Spoke1] > /ip/firewall/address-list/add address=10.0.0.11 list=BGP_REDISTRIBUTE
[admin@Spoke1] > /routing/bgp/connection/set output.network=BGP_REDISTRIBUTE numbers=0

Spoke2:
[admin@Spoke2] > /interface/bridge/add name=loopback0
[admin@Spoke2] > /ip/address/add address=10.0.0.12 interface=loopback0 network=10.0.0.12
[admin@Spoke2] > /ip/firewall/address-list/add address=10.0.0.12 list=BGP_REDISTRIBUTE
[admin@Spoke2] > /routing/bgp/connection/set output.network=BGP_REDISTRIBUTE numbers=0

We should be able to see these routes on our spokes now.

[admin@Spoke1] > /ip/route/print
    DST-ADDRESS      GATEWAY        DISTANCE
DAc 10.0.0.11/32     loopback0             0
DAb 10.0.0.12/32     10.13.0.12          200

Yep, they’re there. We’re almost ready to be able to test, but let’s look at our baseline right now to make sure we’re not connected to the other Spoke. Remember, we want this to by dynamic. Nodes shouldn’t be connected unless they want to actively communicate. Let’s check ARP.

[admin@Spoke1] > /ip/arp/print where interface="Zero Path VPN"
 #    ADDRESS    MAC-ADDRESS        INTERFACE
21 DC 10.13.0.1  4E:24:26:54:DB:32  Zero Path VPN

We only have an ARP resolution for the Hub. Let’s check our peers.

[admin@Spoke1] > /zerotier/peer/print where instance=zt1
Columns: INSTANCE, ZT-ADDRESS, LATENCY, ROLE, PATH
 # INSTANCE  ZT-ADDRESS  LATENCY  ROLE    PATH
 0 zt1       62f865ae71  237ms    PLANET  active,preferred,50.7.252.138/9993,recvd:2m38s291ms,sent:13s349ms
 1 zt1       778cde7190  35ms     PLANET  active,preferred,103.195.103.66/9993,recvd:2m38s498ms,sent:3s337ms
 2 zt1       cafe04eba9  105ms    PLANET  active,preferred,84.17.53.155/9993,recvd:2m38s424ms,sent:13s349ms
 3 zt1       cafe9efeb9  66ms     PLANET  active,preferred,104.194.8.134/9993,recvd:2m38s463ms,sent:13s349ms
 4 zt1       0123456789  31ms     LEAF    active,preferred,35.209.81.208/21003,recvd:5s174ms,sent:5s174ms
 9 zt1       Hub         3ms      LEAF    active,preferred,192.168.88.254/51488,recvd:13s345ms,sent:5s863ms

We only have one peer that isn’t owned by ZeroTier (the Hub). Finally, let’s try to ping between those loopbacks and see what happens.

[admin@Spoke1] > ping 10.0.0.12 src-address=10.0.0.11
  SEQ HOST                                     SIZE TTL TIME      STATUS
    0 10.0.0.12                                  56  64 624ms108us
    1 10.0.0.12                                  56  64 5ms238us

The pings worked. Our first ping has a bit of latency, but that is because ARP was resolving, and the initial packet traverses the ZeroTier Root Servers. The second ping has a lower latency, showing we’re connected directly between the 2 nodes. Let’s check ARP and our peers again.

[admin@Spoke1] > /ip/arp/print where interface="Zero Path VPN"
 #    ADDRESS     MAC-ADDRESS        INTERFACE
21 DC 10.13.0.1   4E:24:26:54:DB:32  Zero Path VPN
24 DC 10.13.0.12  4E:01:FB:55:4E:A8  Zero Path VPN

We can see we now have ARP resolved for our other Spoke. What about our peers?

[admin@Spoke1] > /zerotier/peer/print where instance=zt1
Columns: INSTANCE, ZT-ADDRESS, LATENCY, ROLE, PATH
 # INSTANCE  ZT-ADDRESS  LATENCY  ROLE    PATH
 0 zt1       62f865ae71  237ms    PLANET  active,preferred,50.7.252.138/9993,recvd:3m13s524ms,sent:33s575ms
 1 zt1       778cde7190  31ms     PLANET  active,preferred,103.195.103.66/9993,recvd:2m12s63ms,sent:3s543ms
 2 zt1       cafe04eba9  104ms    PLANET  active,preferred,84.17.53.155/9993,recvd:3m13s657ms,sent:33s575ms
 3 zt1       cafe9efeb9  66ms     PLANET  active,preferred,104.194.8.134/9993,recvd:3m13s694ms,sent:33s575ms
 4 zt1       0123456789  31ms     LEAF    active,preferred,35.209.81.208/21003,recvd:15s659ms,sent:15s659ms
 9 zt1       Hub         3ms      LEAF    active,192.168.88.254/26243,recvd:3s540ms,sent:3s543ms
10 zt1       Spoke2      6ms      LEAF    active,preferred,10.0.101.144/63218,recvd:10s457ms,sent:3s540ms

And predictably, we now have Spoke2 as a directly connected peer.

Conclusion

And that’s it! You can easily scale this out to a large number of spokes. Once you have the Spoke config, you can apply nearly the same config to all of the spokes, making configuration management simple.

This should scale to as many connections and routes as the Hub can handle. In Part 2 of this series, we’ll explore scaling the design even further by creating a regionalized design.