Using NetBird for Site-to-Site Routing on VyOS

I previously wrote an article where I used NetBird to build DMVPN between sites using WireGuard. This solution would allow for massive scale of DMVPN using WireGuard, but when you have a smaller number of sites, you can actually use NetBird to fully handle your site-to-site routing. That’s what we’ll explore in this post.

Our Toplogy

We’re going to have 5 sites connected via NetBird. All routing between sites will be handled by NetBird directly. We won’t need any additional protocols like OSPF or BGP. I named all nodes “RTR<n>”. Each site will have a single host for testing.

Create a NetBird account

If you don’t already have one, you’ll need to create a NetBird account.

NOTE: You can self-host NetBird if you want, but that is beyond the scope of this article.

First we need to go to the NetBird website at :
https://netbird.io/

Once there, click the “Try for Free” button in the top right:

There, we will be presented the signup/login page.

Once your account is created, and you’re logged in, you’ll be presented with the NetBird console.

We need to do a few things before we move to VyOS. Go to the settings page and turn on “Peer approval” under Authentication. This is just an added step to ensure you control who can access your devices.

Next, we need to create a setup key. Go to the “Setup Keys” section, and select “Create Setup Key”.

We’re going to create a setup key called “NetBird on VyOS”, select it to be reusable, and set the usage limit to 5 (since we have 5 sites; change this value if you have more/less).

It’ll present you with your newly created key (code removed for security). You’ll need this in a few steps, so copy this and paste it into a notepad for right now.

Now on to VyOS.

Configuring VyOS

For this lab, we’re going to be using 1.4.0-rc3, which you can grab from this blog post.

https://blog.vyos.io/vyos-1.4.0-rc3-release-candidate

VyOS will need to have a path to the internet as well as the ability to resolve DNS to get everything setup.

configure 
set interfaces ethernet eth0 address dhcp
set system name-server 4.2.2.2
commit

If you’re able to ping something like google, you should be fine.

vyos@vyos:~$ ping www.google.com
PING www.google.com (142.251.167.106) 56(84) bytes of data.
64 bytes from ww-in-f106.1e100.net (142.251.167.106): icmp_seq=1 ttl=59 time=12.4 ms

NetBird Container

We’re going to install NetBird in a container. This makes for the easiest management of additional services added to VyOS. First we need to pull down the image.

vyos@vyos:~$ add container image netbirdio/netbird:latest
vyos@vyos:~$

In 1.4, the download is entirely silent, so it may not seem like it’s doing anything, but it should be pulling the image. You can verify that time image was pulled currently with “show container image”.
vyos@RTR1# run show container image 
REPOSITORY                   TAG         IMAGE ID      CREATED      SIZE
docker.io/netbirdio/netbird  latest      3ef8ba0b6ef0  3 weeks ago  44.7 MB

Next, we want to make a directory so that our configuration will be persistent if we need to restart our container (or restart VyOS). I make my folder in /config/containers/, but you can do anything as long as it is within /config/.

sudo mkdir -p /config/containers/nb1

Now it’s time to apply our container config. Notice the NB_SETUP_KEY environment variable. The string after “value” should be the setup key you created in the NetBird console.

NOTE: NetBird will name your devices in the console based on their hostname. It is useful to ensure that your device has a unique hostname configured before registering the node. This prevents you from needing to rename it afterwards.

configure
set container name nb1 allow-host-networks
set container name nb1 cap-add 'net-admin'
set container name nb1 cap-add 'net-raw'
set container name nb1 image 'netbirdio/netbird:latest'
set container name nb1 volume NB_PATH destination '/etc/netbird'
set container name nb1 volume NB_PATH source '/config/containers/nb1'
set container name nb1 environment NB_SETUP_KEY value '01234567-89AB-CDEF-0123-456789ABCDEF'
commit

Let’s go over some of these commands:

allow-host-networks – This tells the container to use VyOS as it’s network. NetBird will create an interface that VyOS can use for routing, so it needs to be on the same network as the host, and not its own container network.
cap-add ‘net-admin’ – NetBird will be creating an interface on the host; this capability allows the container to do this.
cap-add ‘net-raw’ – NetBird will be creating sockets from the host; this capability allows the container to do this.
image ‘netbirdio/netbird:latest’ – This tells the container which image to use. This is the image we added with the “add container image” command earlier.
volume NB_PATH – To ensure that the container’s configuration is persistent, it needs to map to a volume on the host. Destination is the volume on the container. Source is the volume on the host, which we created previously with the “mkdir” command. Anything that the container puts in its “/etc/netbird” directory will actually be VyOS’ “/config/containers/nb1” directory.

After committing the container config in VyOS, you should see our new device in NetBird’s console under Peers.

You can see our node needs approval. If we didn’t turn the previous setting on, the node would already be online. Let’s approve it by clicking “Approve” next to “Approval required”. Do this for all nodes.

NetBird will auto assign IP addresses to our nodes.

Let’s see if RTR1 can ping all of the other nodes.

Ping RTR2:
vyos@RTR1# run ping 100.90.73.127
PING 100.90.73.127 (100.90.73.127) 56(84) bytes of data.
64 bytes from 100.90.73.127: icmp_seq=1 ttl=64 time=7.59 ms

Ping RTR3:
vyos@RTR1# run ping 100.90.201.60
PING 100.90.201.60 (100.90.201.60) 56(84) bytes of data.
64 bytes from 100.90.201.60: icmp_seq=1 ttl=64 time=5.01 ms

Ping RTR4:
vyos@RTR1# run ping 100.90.95.99
PING 100.90.95.99 (100.90.95.99) 56(84) bytes of data.
64 bytes from 100.90.95.99: icmp_seq=1 ttl=64 time=2.93 ms

Ping RTR5:
vyos@RTR1# run ping 100.90.72.252
PING 100.90.72.252 (100.90.72.252) 56(84) bytes of data.
64 bytes from 100.90.72.252: icmp_seq=1 ttl=64 time=4.30 ms

Everything is looking good.

Configuring Site-to-Site Networking

Before we can configure NetBird to handle our routing, we need to have routes for it to use.

For this lab, I’m going to have a single host connected to VyOS for testing. A single LAN interface will be configured.

RTR1:
set interfaces ethernet eth1 address 10.1.0.1/24

RTR2:
set interfaces ethernet eth1 address 10.2.0.1/24

RTR3:
set interfaces ethernet eth1 address 10.3.0.1/24

RTR4:
set interfaces ethernet eth1 address 10.4.0.1/24

RTR5:
set interfaces ethernet eth1 address 10.5.0.1/24

NOTE: It is typical to use dummy interfaces when testing in a lab, but I’m unable to do that for this. More on that later.

If we were to look at the current routing table for one of our nodes, it should be pretty simple. We should have our default route, and a couple of connected routes.

vyos@RTR1# run show ip route
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, F - PBR,
       f - OpenFabric,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup
       t - trapped, o - offload failure

S>* 0.0.0.0/0 [210/0] via 10.0.95.1, eth0, weight 1, 01:45:06
C>* 10.0.95.0/24 is directly connected, eth0, 01:45:06
C>* 10.1.0.0/24 is directly connected, eth1, 00:13:33
C>* 100.90.0.0/16 is directly connected, wt0, 01:10:57

We don’t yet have our routes to the other sites, but doing that is as simple as saying what routes a peer owns. We do that in the NetBird console.

Click on RTR1 (or whatever you named your node) in the Peers section, and find the Network Routes section.

Select Add Route, and then select “New Network Route”.

You will need to populate the Network Range, and select the Distribution group. The “All” group is created automatically when you create your NetBird account. You can make a specific group for these nodes, but that’s outside the scope of this article.

After populating the Network Range and Distribution Groups, select Continue and you’ll see the “Name & Description” tab. Give the network a unique name (this are globally unique, so it can’t be named the same as other routes).

After providing a “Network Identifier” (and Description if desired), hit Continue and you’ll see the “Additional Settings” command. Here you’ll want to ensure the “Enable Route” is selected, and Masquerade is deselected (since we’re doing site-to-site routing).

Click “Add Route” and you should now see the route listed under your “Network Routes”.

Do that for the rest of your sites. You should then see all of these routes in the “Network Routes” section on the left side menu.

Now back to VyOS, let’s check out that routing table again.

vyos@RTR1# run show ip route
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, F - PBR,
       f - OpenFabric,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup
       t - trapped, o - offload failure

S>* 0.0.0.0/0 [210/0] via 10.0.95.1, eth0, weight 1, 01:55:12
C>* 10.0.95.0/24 is directly connected, eth0, 01:55:12
C>* 10.1.0.0/24 is directly connected, eth1, 00:23:39
C>* 100.90.0.0/16 is directly connected, wt0, 01:21:03

Nothing changed, that’s odd! Maybe not all that odd, as this can by typical for a lot of solutions like NetBird and Tailscale. They frequently put routes that they push into other routing tables. I’m really not a fan of this, but keep in mind that the main focus of these solutions is typically not site-to-site routing, and is more typically host-to-host.

We can see the routes in table “7120”.

vyos@RTR1# run show ip route table 7120
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, F - PBR,
       f - OpenFabric,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup
       t - trapped, o - offload failure

VRF default table 7120:
K>* 10.2.0.0/24 [0/0] is directly connected, wt0, 00:21:41
K>* 10.3.0.0/24 [0/0] is directly connected, wt0, 00:24:35
K>* 10.4.0.0/24 [0/0] is directly connected, wt0, 00:20:06
K>* 10.5.0.0/24 [0/0] is directly connected, wt0, 00:19:43

This nuance is also why we are not able to test with the dummy interfaces. Pings to the subnets at the other sites will not work directly from VyOS, but they will work through VyOS (like from our test hosts).

NOTE: This is only partly true. You can still ping from VyOS provided the source is the NetBird interface. If you tried to source the ping off of another IP on VyOS, it would fail.

Let’s test our pings from a test host at Site1.

Host IPs:

Site1_PC: 10.1.0.10
Site2_PC: 10.2.0.10
Site3_PC: 10.3.0.10
Site4_PC: 10.4.0.10
Site5_PC: 10.5.0.10

Site2:
PC1> ping 10.2.0.10 -c 1
84 bytes from 10.2.0.10 icmp_seq=1 ttl=62 time=4.939 ms

Site3:
PC1> ping 10.3.0.10 -c 1
84 bytes from 10.3.0.10 icmp_seq=1 ttl=62 time=4.654 ms

Site4:
PC1> ping 10.4.0.10 -c 1
84 bytes from 10.4.0.10 icmp_seq=1 ttl=62 time=5.131 ms

Site5:
PC1> ping 10.5.0.10 -c 1
84 bytes from 10.5.0.10 icmp_seq=1 ttl=62 time=5.095 ms

Everything’s working!

Conclusion

That is it for this post. I know it may seem odd with how NetBird places routes within a separate table. Functionally, it doesn’t impact how the solution works. It’s simply a nuance you need to be aware of when managing the network.