In part 2 of this blog, I showed you how to use this design in a multi-tenant environment using MPLS. You could easily talk between 100 different customers without having to do a tunnel per VRF. While incredibly useful, a lot of people may not have enough tenants to warrant using MPLS. A lot of people won’t even have multiple tenants.
This doesn’t immediately mean that a use case of the MPLS design can’t still provide value. There’s something else we can do with this design…..enter microsegmentation. Microsegmentation can isolate customer traffic at the routing level. While not as granular as explicit firewall rules, it is far more scalable. Even if using firewall rules, this can be a great addition to your security posture. Remember; “Defense in Depth”!
Our Topology
We’re going to use a basic topology with a single hub, and 2 spokes. The Hub won’t serve any traffic at a hub site, it’ll will strictly be serving as a route-server. All customer traffic will be on the Spokes. Site 2 will also host a secure internet connection.
You may ask if every node already has to build ZeroTier over their local Internet Connection, why would we want to route inside traffic to the internet via another site’s internet? When you have a large number of sites, you may wish to limit the amount of paths to the internet you’d like to secure. By routing Internet out of a smaller number of sites, you can more easily maintain a sound security posture.
Image
We’re going to use 1.4.0-rc3 since it fixes the issue with the IPv4-Labeled Unicast Address-Family not being available in VyOS. You can find rc3 here: https://blog.vyos.io/vyos-1.4.0-rc3-release-candidate
VRFs
Instead of using VRFs to separate customers, we’re going to use them to separate services for a single customer. Here is a list of VRFs we’re going to configure:
- Users
- RD: 65200:1
- Route-Target: 65200:1
- Admins
- RD: 65200:2
- Route-Target: 65200:2
- Servers
- RD: 65200:3
- Route-Target: 65200:3
- Management
- RD: 65200:4
- Route-Target: 65200:4
- Internet
- RD: 65200:100
- Route-Target: 65200:100
Configuraton
I’m not going to walk through the configuration here. Reference Part 2 of this series for the configuration. Part 2 can be found here: https://lev-0.com/2024/01/09/dynamic-multipoint-vpn-with-zerotier-and-vyos-part-2-mpls/
VRF Config (both spokes)
set vrf name INET protocols bgp address-family ipv4-unicast export vpn
set vrf name INET protocols bgp address-family ipv4-unicast import vpn
set vrf name INET protocols bgp address-family ipv4-unicast label vpn export 'auto'
set vrf name INET protocols bgp address-family ipv4-unicast rd vpn export '65200:100'
set vrf name INET protocols bgp address-family ipv4-unicast route-target vpn import '65200:100'
set vrf name INET protocols bgp address-family ipv4-unicast route-target vpn export '65200:100'
set vrf name INET protocols bgp system-as '65000'
set vrf name INET table '100'
set vrf name USERS protocols bgp address-family ipv4-unicast export vpn
set vrf name USERS protocols bgp address-family ipv4-unicast import vpn
set vrf name USERS protocols bgp address-family ipv4-unicast label vpn export 'auto'
set vrf name USERS protocols bgp address-family ipv4-unicast rd vpn export '65200:1'
set vrf name USERS protocols bgp address-family ipv4-unicast redistribute connected
set vrf name USERS protocols bgp address-family ipv4-unicast route-target vpn import '65200:1'
set vrf name USERS protocols bgp address-family ipv4-unicast route-target vpn export '65200:1'
set vrf name USERS protocols bgp system-as '65000'
set vrf name USERS table '101'
set vrf name ADMINS protocols bgp address-family ipv4-unicast export vpn
set vrf name ADMINS protocols bgp address-family ipv4-unicast import vpn
set vrf name ADMINS protocols bgp address-family ipv4-unicast label vpn export 'auto'
set vrf name ADMINS protocols bgp address-family ipv4-unicast rd vpn export '65200:2'
set vrf name ADMINS protocols bgp address-family ipv4-unicast redistribute connected
set vrf name ADMINS protocols bgp address-family ipv4-unicast route-target vpn import '65200:2'
set vrf name ADMINS protocols bgp address-family ipv4-unicast route-target vpn export '65200:2'
set vrf name ADMINS protocols bgp system-as '65000'
set vrf name ADMINS table '102'
set vrf name SERVERS protocols bgp address-family ipv4-unicast export vpn
set vrf name SERVERS protocols bgp address-family ipv4-unicast import vpn
set vrf name SERVERS protocols bgp address-family ipv4-unicast label vpn export 'auto'
set vrf name SERVERS protocols bgp address-family ipv4-unicast rd vpn export '65200:3'
set vrf name SERVERS protocols bgp address-family ipv4-unicast redistribute connected
set vrf name SERVERS protocols bgp address-family ipv4-unicast route-target vpn import '65200:3'
set vrf name SERVERS protocols bgp address-family ipv4-unicast route-target vpn export '65200:3'
set vrf name SERVERS protocols bgp system-as '65000'
set vrf name SERVERS table '103'
set vrf name MGMT protocols bgp address-family ipv4-unicast export vpn
set vrf name MGMT protocols bgp address-family ipv4-unicast import vpn
set vrf name MGMT protocols bgp address-family ipv4-unicast label vpn export 'auto'
set vrf name MGMT protocols bgp address-family ipv4-unicast rd vpn export '65200:4'
set vrf name MGMT protocols bgp address-family ipv4-unicast redistribute connected
set vrf name MGMT protocols bgp address-family ipv4-unicast route-target vpn import '65200:4'
set vrf name MGMT protocols bgp address-family ipv4-unicast route-target vpn export '65200:4'
set vrf name MGMT protocols bgp system-as '65000'
set vrf name MGMT table '104'
Interface Config
Spoke1:
set interfaces ethernet eth1 vif 10 address '10.11.1.1/24'
set interfaces ethernet eth1 vif 10 vrf 'USERS'
set interfaces ethernet eth1 vif 20 address '10.11.2.1/24'
set interfaces ethernet eth1 vif 20 vrf 'ADMINS'
set interfaces ethernet eth1 vif 30 address '10.11.3.1/24'
set interfaces ethernet eth1 vif 30 vrf 'SERVERS'
set interfaces ethernet eth1 vif 40 address '10.11.4.1/24'
set interfaces ethernet eth1 vif 40 vrf 'MGMT'
Spoke2:
set interfaces ethernet eth1 vif 10 address '10.12.1.1/24'
set interfaces ethernet eth1 vif 10 vrf 'USERS'
set interfaces ethernet eth1 vif 20 address '10.12.2.1/24'
set interfaces ethernet eth1 vif 20 vrf 'ADMINS'
set interfaces ethernet eth1 vif 30 address '10.12.3.1/24'
set interfaces ethernet eth1 vif 30 vrf 'SERVERS'
set interfaces ethernet eth1 vif 40 address '10.12.4.1/24'
set interfaces ethernet eth1 vif 40 vrf 'MGMT'
set interfaces ethernet eth2 address '10.12.100.1/24'
set interfaces ethernet eth2 vrf 'INET'
MP-BGP Config
Hub:
set protocols bgp address-family ipv4-labeled-unicast
set protocols bgp address-family ipv4-unicast network 10.0.0.1/32
set protocols bgp address-family ipv4-unicast redistribute connected route-map 'DENY_ZT_LAN'
set protocols bgp interface eth10 mpls forwarding
set protocols bgp listen range 10.0.0.0/24 peer-group 'VPNv4_PEERS'
set protocols bgp listen range 10.13.0.0/16 peer-group 'ZT_PEERS'
set protocols bgp peer-group VPNv4_PEERS address-family ipv4-vpn route-reflector-client
set protocols bgp peer-group VPNv4_PEERS remote-as '65000'
set protocols bgp peer-group VPNv4_PEERS update-source 'dum0'
set protocols bgp peer-group ZT_PEERS remote-as '65000'
set protocols bgp peer-group ZT_PEERS address-family ipv4-labeled-unicast route-reflector-client
set protocols bgp system-as '65000'
Both Spokes:
set protocols bgp address-family ipv4-labeled-unicast
set protocols bgp address-family ipv4-unicast redistribute connected route-map 'DENY_ZT_LAN'
set protocols bgp interface eth10 mpls forwarding
set protocols bgp neighbor 10.0.0.1 address-family ipv4-vpn
set protocols bgp neighbor 10.0.0.1 remote-as '65000'
set protocols bgp neighbor 10.0.0.1 update-source 'dum0'
set protocols bgp neighbor 10.13.0.1 address-family ipv4-labeled-unicast nexthop-self
set protocols bgp neighbor 10.13.0.1 remote-as '65000'
set protocols bgp system-as '65000'
Internet Firewall Related Config
Spoke2:
set vrf name INET protocols bgp neighbor 10.12.100.2 address-family ipv4-unicast
set vrf name INET protocols bgp neighbor 10.12.100.2 remote-as '65100'
Internet firewall:
set interfaces ethernet eth0 address 'dhcp'
set interfaces ethernet eth2 address '10.12.100.2/24'
set nat source rule 10 outbound-interface name 'eth0'
set nat source rule 10 translation address 'masquerade'
set protocols bgp address-family ipv4-unicast network 0.0.0.0/0
set protocols bgp neighbor 10.12.100.1 address-family ipv4-unicast
set protocols bgp neighbor 10.12.100.1 remote-as '65000'
set protocols bgp system-as '65100'
Verification
Let’s make sure our base lab is functioning correctly.
vyos@Spoke1# run show ip bgp vrf all
Instance ADMINS:
Network Next Hop Metric LocPrf Weight Path
*> 10.11.2.0/24 0.0.0.0 0 32768 ?
*> 10.12.2.0/24 10.0.0.12@0< 0 100 0 ?
Instance SERVERS:
Network Next Hop Metric LocPrf Weight Path
*> 10.11.3.0/24 0.0.0.0 0 32768 ?
*> 10.12.3.0/24 10.0.0.12@0< 0 100 0 ?
Instance USERS:
Network Next Hop Metric LocPrf Weight Path
*> 10.11.1.0/24 0.0.0.0 0 32768 ?
*> 10.12.1.0/24 10.0.0.12@0< 0 100 0 ?
Instance MGMT:
Network Next Hop Metric LocPrf Weight Path
*> 10.11.4.0/24 0.0.0.0 0 32768 ?
*> 10.12.4.0/24 10.0.0.12@0< 0 100 0 ?
Instance INET:
Network Next Hop Metric LocPrf Weight Path
*> 0.0.0.0/0 10.0.0.12@0< 0 100 0 65100 i
*> 10.12.100.0/24 10.0.0.12@0< 0 100 0 ?
BGP Looks good! Let’s do some quick pings to make sure our label switched paths are good.
vyos@Spoke1# run ping 10.12.1.1 source-address 10.11.1.1 vrf USERS
64 bytes from 10.12.1.1: icmp_seq=2 ttl=63 time=2.04 ms
vyos@Spoke1# run ping 10.12.2.1 source-address 10.11.2.1 vrf ADMINS
64 bytes from 10.12.2.1: icmp_seq=1 ttl=63 time=2.66 ms
vyos@Spoke1# run ping 10.12.3.1 source-address 10.11.3.1 vrf SERVERS
64 bytes from 10.12.3.1: icmp_seq=1 ttl=63 time=6.19 ms
vyos@Spoke1# run ping 10.12.4.1 source-address 10.11.4.1 vrf MGMT
64 bytes from 10.12.4.1: icmp_seq=2 ttl=63 time=1.95 ms
vyos@Spoke2# run ping 4.2.2.3 vrf INET
64 bytes from 4.2.2.3: icmp_seq=1 ttl=57 time=9.62 ms
Let’s also make sure we can’t ping between VRFs.
Ping Site2 Users from Site1 Admins:
vyos@Spoke1# run ping 10.12.1.1 source-address 10.11.2.1 vrf ADMINS
3 packets transmitted, 0 received, 100% packet loss, time 2058ms
Everything looks as it should!
Planning for Microsegmentation
NOTE: In this lab, we are using very broad VRFs (all users, all admins, all servers, etc…). If you’re looking to do microsegmentation in an actual deployment, you’ll want smaller blocks within the VRFs. Take the SERVERS VRF as an example, you may actually want these VRFs:
- Active Directory
- SNMP Tools
- NetFlow Tools
- DNS
- DHCP
- ….or any other service that would exist as its own system
Here is our plan for segmentation:
- Users can talk to:
- Servers
- Internet
- Admins
- Admins can talk to:
- Users
- Admins
- Servers
- Internet
- Management
- Servers can talk to:
- Users
- Admins
- Management can talk to:
- Admins
- Internet can talk to:
- Users
- Admins
Enabling the route-leaking for microsegmentation is very easy. It’ll likely take you longer to plan out the segmentation than it will to create the config. All we need to do is import additional route-targets into the VRFs.
NOTE: There is currently a bug in 1.4.0-rc1 which prevents additional imports from working if you use “route-target both xxxxx:xxxxx”. For this lab, make sure you explicitly call out import and export. A bug has been created and can be tracked here:
https://vyos.dev/T5930
NOTE: When typing “route-target import”, it will overwrite any existing import statement. Make sure you account for that when adding additional route-targets.
Both Spokes:
set vrf name INET protocols bgp address-family ipv4-unicast route-target vpn import '65200:1 65200:2 65200:100'
set vrf name USERS protocols bgp address-family ipv4-unicast route-target vpn import '65200:2 65200:3 65200:100'
set vrf name ADMINS protocols bgp address-family ipv4-unicast route-target vpn import '65200:1 65200:2 65200:3 65200:4 65200:100'
set vrf name SERVERS protocols bgp address-family ipv4-unicast route-target vpn import '65200:1 65200:2'
set vrf name MGMT protocols bgp address-family ipv4-unicast route-target vpn import '65200:2 65200:4'
Let’s look at the updated routing tables.
vyos@Spoke1# run show ip bgp vrf all
Instance ADMINS:
Network Next Hop Metric LocPrf Weight Path
*> 0.0.0.0/0 10.0.0.12@0< 0 100 0 65100 i
*> 10.11.1.0/24 0.0.0.0@37< 0 32768 ?
*> 10.11.2.0/24 0.0.0.0 0 32768 ?
*> 10.11.3.0/24 0.0.0.0@36< 0 32768 ?
*> 10.11.4.0/24 0.0.0.0@35< 0 32768 ?
*> 10.12.1.0/24 10.0.0.12@0< 0 100 0 ?
*> 10.12.2.0/24 10.0.0.12@0< 0 100 0 ?
*> 10.12.3.0/24 10.0.0.12@0< 0 100 0 ?
*> 10.12.4.0/24 10.0.0.12@0< 0 100 0 ?
*> 10.12.100.0/24 10.0.0.12@0< 0 100 0 ?
Instance SERVERS:
Network Next Hop Metric LocPrf Weight Path
*> 10.11.1.0/24 0.0.0.0@37< 0 32768 ?
*> 10.11.2.0/24 0.0.0.0@33< 0 32768 ?
*> 10.11.3.0/24 0.0.0.0 0 32768 ?
*> 10.12.1.0/24 10.0.0.12@0< 0 100 0 ?
*> 10.12.2.0/24 10.0.0.12@0< 0 100 0 ?
Instance USERS:
Network Next Hop Metric LocPrf Weight Path
*> 0.0.0.0/0 10.0.0.12@0< 0 100 0 65100 i
*> 10.11.1.0/24 0.0.0.0 0 32768 ?
*> 10.11.2.0/24 0.0.0.0@33< 0 32768 ?
*> 10.11.3.0/24 0.0.0.0@36< 0 32768 ?
*> 10.12.2.0/24 10.0.0.12@0< 0 100 0 ?
*> 10.12.3.0/24 10.0.0.12@0< 0 100 0 ?
*> 10.12.100.0/24 10.0.0.12@0< 0 100 0 ?
Instance MGMT:
Network Next Hop Metric LocPrf Weight Path
*> 10.11.2.0/24 0.0.0.0@33< 0 32768 ?
*> 10.11.4.0/24 0.0.0.0 0 32768 ?
*> 10.12.2.0/24 10.0.0.12@0< 0 100 0 ?
*> 10.12.4.0/24 10.0.0.12@0< 0 100 0 ?
Instance INET:
Network Next Hop Metric LocPrf Weight Path
*> 0.0.0.0/0 10.0.0.12@0< 0 100 0 65100 i
*> 10.11.1.0/24 0.0.0.0@37< 0 32768 ?
*> 10.11.2.0/24 0.0.0.0@33< 0 32768 ?
*> 10.12.1.0/24 10.0.0.12@0< 0 100 0 ?
*> 10.12.2.0/24 10.0.0.12@0< 0 100 0 ?
*> 10.12.100.0/24 10.0.0.12@0< 0 100 0 ?
Finally, let’s do some more pings:
User to Internet:
vyos@Spoke1# run ping 4.2.2.2 source-address 10.11.1.1 vrf USERS
64 bytes from 4.2.2.2: icmp_seq=1 ttl=55 time=11.7 ms
User to Users:
vyos@Spoke1# run ping 10.12.1.1 source-address 10.11.1.1 vrf USERS
2 packets transmitted, 0 received, 100% packet loss, time 1056ms
User to Admins:
vyos@Spoke1# run ping 10.12.2.1 source-address 10.11.1.1 vrf USERS
64 bytes from 10.12.2.1: icmp_seq=1 ttl=63 time=2.48 ms
User to Servers:
vyos@Spoke1# run ping 10.12.3.1 source-address 10.11.1.1 vrf USERS
64 bytes from 10.12.3.1: icmp_seq=1 ttl=63 time=3.41 ms
User to Management:
vyos@Spoke1# run ping 10.12.4.1 source-address 10.11.1.1 vrf USERS
2 packets transmitted, 0 received, 100% packet loss, time 1047ms
Predictably, the user can access Internet, Admins, and Servers. They cannot reach other Users or Management.
Conclusion
That’s all that is involved with allowing the VRFs to communicate with each other. Just add some additional imports to let VRFs intercommunicate, or remove them to limit communications.
A friend of mine is turning my blog posts into videos. Check out the video for this post:
Leave a Reply