After posting part 1 of “Using NetBird for Site-to-Site Routing”, a user on the VyOS forums asked about using NetBird for site-to-site networking, with the addition of an exit node.
You can find the original blog post here: Using NetBird for Site-to-Site Routing on VyOS
To understand what the ask is, let’s first go over what an Exit Node is.
What is an Exit Node
The term Exit Node may be unfamiliar to people coming from Enterprise networking, but it’s pretty common if you’re familiar with services like ZeroTier, Tailscale, or NetBird. In simple terms, it is a node that allows VPN clients access to the internet via a trusted internet connection.
Typically, exit nodes are used for Remote Access VPNs (RA-VPNs). This is like a user on their Cell Phone that is using insecure internet at a hotel or convention. That doesn’t mean that it’s the only use case though. You can do the same thing with site-to-site networking as well.
You may ask why would you want to do that, since you’re probably communicating between your nodes using the internet, but that actually won’t be true for everyone. Some people may only have a few sites with internet access, and all other sites are interconnected via a private MPLS service, or P2P circuits.
It’s also normal for users to want all internet bound traffic to go through a security stack, and deploying that same stack at every site may be overly complex. You can put secure internet points of presence (POPs) at strategic sites to ensure secure internet access, without the burden of deploying it at every site with internet.
Our Toplogy
Our topology will consist of 2 NetBird nodes. One node will be a user site, and the other node will be our exit node. One benefit to using VyOS for the exit node is we get all of the advanced firewall features to ensure internet access behaves as we want it to.
NOTE: Traffic will actually hairpin at NB-EXIT. Traffic will come into NB-Exit via NetBird (using WireGuard), and will exit out of it on the same internet, but without using WireGuard. The data path will be:
Host -> NB-LAN -> NB-EXIT (NetBird) -> NB-Ext (Internet)
Creating our Nodes
We’re going to follow the same steps we did in the original post. You can find that post here if you need more detailed instructions: Using NetBird for Site-to-Site Routing on VyOS
I’m not going to go over deploying WireGuard on VyOS instances again, but here’s a summary of the steps needed.
- Create a NetBird Account (or deploy a self hosted instance)
- Enable “Peer Approval” under “Settings”.
- Create a reusable setup key with a limit of 2. Make sure you copy the setup key once it is shown, you won’t be able to view it later.
- Configure VyOS with internet access and DNS resolution.
- Add the netbird container with “add container image netbirdio/netbird:latest“.
- Create a persistent directory for NetBird using “sudo mkdir -p /config/containers/nb1“.
- Configure a container in VyOS under conf-mode.
- Approve the Nodes in the NetBird Console.
Once we have our nodes configured, we’ll need to create a LAN interface on NB-LAN. I’m going to use “eth1” and use an IP of “10.0.1.1/24”.
set interfaces ethernet eth1 address '10.0.1.1/24'
Lastly, we need to connect a host to that interface. I’m using GNS3 for this lab, which has a lightweight virtual PC for testing. That’s what I’m going to use, but feel free to use whatever makes sense for you.
PC1> ip 10.0.1.10/24 10.0.1.1
Checking for duplicate address...
PC1 : 10.0.1.10 255.255.255.0 gateway 10.0.1.1
PC1> ping 10.0.1.1 -c 3
84 bytes from 10.0.1.1 icmp_seq=1 ttl=64 time=0.928 ms
84 bytes from 10.0.1.1 icmp_seq=2 ttl=64 time=0.892 ms
84 bytes from 10.0.1.1 icmp_seq=3 ttl=64 time=0.982 ms
PC1>
Adding routes in The NetBird Console
Now we’re all ready to add our routes to our nodes. Let’s start with NB-LAN. We’re going to say that NB-LAN owns 10.0.1.0/24. This will let our Exit Node know where to send return traffic.
Select Add Route, and then select “New Network Route”.
You will need to populate the Network Range, and select the Distribution group. The “All” group is created automatically when you create your NetBird account. You can make a specific group for these nodes, but that’s outside the scope of this article.
After populating the Network Range and Distribution Groups, select Continue and you’ll see the “Name & Description” tab. Give the network a unique name (these are globally unique, so it can’t be named the same as other routes).
After providing a “Network Identifier” (and description if desired), hit Continue and you’ll see the “Additional Settings” command. Here you’ll want to ensure the “Enable Route” is selected, and Masquerade is deselected (since we’re doing site-to-site routing).
Creating The Exit Node
We’re going to setup NB-EXIT as the exit node. In the NB-EXIT peer settings, there’s a “Set Up Exit Node” button next to “Add Route”; select that.
We’ll select a Distribution Group. This will be every node that we want to know about this exit node. I’m going to use that default “All” group. Once done, select “Continue”
A Network Identifier will be auto-populated. I’m perfectly fine with what it generated, so I’m going to hit continue. Feel free to modify what you see if desired.
Ensure the route is enabled, and select “Add Exit Node”.
Under “Network Routes”, you should see our Exit Node and our Route for NB-LAN.
Testing
Now that we have everything configured in our console, let’s go back to our test host and see if we can ping the internet.
PC1> ping 4.2.2.2 -c 3
4.2.2.2 icmp_seq=1 timeout
4.2.2.2 icmp_seq=2 timeout
4.2.2.2 icmp_seq=3 timeout
Well, that’s not working. Let’s do a little bit of troubleshooting. NetBird just pushes allowed-ips to each node, so let’s check if those look like we’d expect.
vyos@NB-LAN:~$ sudo wg
peer:
endpoint: 10.0.95.212:51820
allowed ips: 100.90.34.56/32, 0.0.0.0/0
“100.90.34.56/32” is the WG IP of NB-EXIT, and we have a default pointed to NB-EXIT. So that looks good. Let’s look at NB-EXIT:
vyos@NB-EXIT:~$ sudo wg
peer:
endpoint: 10.0.95.213:51820
allowed ips: 100.90.216.72/32, 10.0.1.0/24
“100.90.216.72/32” is the WG IP of NB-LAN, and we have the LAN route of our test host. Everything seems like it should work, let’s look for our traffic leaving the exit node.
vyos@NB-EXIT:~$ sudo tcpdump -ni eth0 icmp
03:34:15.671830 IP 10.0.1.10 > 4.2.2.2: ICMP echo request, id 46953, seq 1, length 64
We have our traffic leaving, but you might see that we have a problem. The internet won’t know how to get back to an internal IP. For reference, my WAN IP will be in the “10.0.95.0/24” range (yours will likely be a public IP). We’ll need to configure NAT to ensure that the internet will know how to get back to our hosts.
confgure
set nat source rule 10 outbound-interface name 'eth0'
set nat source rule 10 translation address 'masquerade'
commit
Let’s try that test again.
PC1> ping 4.2.2.2 -c 3
84 bytes from 4.2.2.2 icmp_seq=1 ttl=56 time=11.506 ms
84 bytes from 4.2.2.2 icmp_seq=2 ttl=56 time=10.427 ms
84 bytes from 4.2.2.2 icmp_seq=3 ttl=56 time=10.837 ms
Our pings are working now. Let’s make sure that our traffic is actually using the Exit Node, and not it’s local internet.
PC1> trace 4.2.2.2 -P 1 -m 15
trace to 4.2.2.2, 15 hops max (ICMP), press Ctrl+C to stop
1 10.0.1.1 1.139 ms 0.539 ms 0.562 ms
2 100.90.34.56 2.676 ms 2.156 ms 2.395 ms
3 10.0.95.1 5.189 ms 5.016 ms 5.078 ms
.... hops omitted
10 4.2.2.2 9.765 ms 9.555 ms 8.009 ms
NOTE: “-P 1” says to use ICMP as the protocol, and “-m 15” sets the TTL to 15.
We can see that we hit 10.0.1.1 (NB-LAN), then 100.90.34.56 (NB-EXIT NetBird Interface), and then leaves out towards the internet.
Conclusion
That’s all there is to adding an Exit Node to our site-to-site setup. A couple quick notes about using Exit Nodes.
- You can’t use IPv6 over an Exit Node, it’s currently unsupported.
- You can add more than 1 exit node using High Availability, which you can read about here: High Availability
You can find more information on NetBird Exit Nodes in their documentation. https://docs.netbird.io/how-to/configuring-default-routes-for-internet-traffic
Leave a Reply