In a previous article, I covered how to make VyOS an Application Aware Firewall (this article builds off of that article, so I suggest reading that article before this one). There’s more that we can do by leveraging nprobe, like shaping traffic based on a service or category of services. That’s what we’ll explore in this article.
Why would you shape based on application traffic
Most organizations will experience the struggle of trying to balance the happiness and productivity of their employees. For this reason, a lot of organizations allow their users to access sites like Social Networking (Facebook, Twitter) and Media consumption (YouTube, Spotify, NetFlix).
Whether you decide to allow users to access entertainment services during work or not, the decision is not as simple as “Should I?” or “Should I not?”. Allowing users to use bandwidth intensive applications like streaming services can have adverse effects on the functioning of your network. Remember, ever bit transferred that is non-business related can conflict with your business related traffic.
But there’s another option other than just outright denying your users access to these services. You can simply limit traffic to these services. While doing this with traditional QoS is nearly impossible, since you can’t just say “I’m going to limit NetFlix traffic!”. You’d need to collect every possible IP that NetFlix could use and attempt to match on that. And what if you wanted to match NetFlix, Disney+, Paramount+, and a dozen other services? It could take you months to curate a list of IPs, and you’d likely still not catch all of them (or they could change over time).
By using Deep Packet Inspection (DPI), you’re able to match on a single application. More than that, you can even match on a category like streaming services. You could easily limit all streaming services to 20% of your overall bandwidth, just with a single rule like this.
Our Toplogy
The topology will be simple, it’ll be a single VyOS instance that is inline for our traffic. Here are the interfaces on my router:
- WAN: Eth0
- LAN: Eth1
Installing nprobe
We covered installing and setting up nprobe in depth in our previous article. Please check out that article if you don’t have nprobe setup.
Configuring the IPS Rules
nprobe uses a JSON based format to filter traffic (again, we go over this in more detail in the previous article). For this lab, we’re going be define the 192.168.2.0/24 and 10.0.95.0/24 networks as our user networks. We can have multiple pools, and map those pools to individual policies. This can be useful if you want to ensure you don’t limit traffic for executives, or if you wanted to limit traffic even further for guests.
Here’s what our rules look like:
# Pool definition
{"pool":{"id":1,"name":"User Networks","ip": [ "192.168.2.0/24","10.0.95.0/24" ], "mac": []},"policy": {"id": 1} }
# Policy definition
{"policy":{"id":0, "name":"Default Rule", "default_marker": "pass" } }
{"policy":{"id":1, "name":"Limit Streaming", "default_marker": "pass", "markers": { "categories": { "Streaming": 50, "SocialNetwork": 50 }, "protocols": { "YouTube": 50 } } } }
Let’s look at those rules a little closer:
- We first create a pool definition. The pool ID is arbitrary, but should be unique from other pool IDs. The policy ID a the end of it will state which of the below policies will use that pool, in this case policy 1.
- The first policy (policy 0) is our default policy. Since we don’t have a configured pool for this policy, all traffic outside of the configured pools will use this policy. This policy does not take any action on traffic, and simply accepts it.
- The second policy (policy 1) will match:
- Streaming: AppleTV+, Disney+, Hulu, Showtime, etc…
- SocialNetwork: Facebook, Instagram, TikTok, Twitter, etc…
- YouTube: YouTube is part of the media category, so we match only on the application so as not to limit the other traffic in that category.
- Each match will set a mark of ’50’, which is what we will match when shaping our traffic. All other traffic will be permitted.
Initiating our rules
We need to place our rules file (or update it if it already exists) in the location designated in our container config. I defined “/data/nprobe/ips-config/ips-rules.conf” as my location for the rules file. This maps to “/config/containers/nprobe/ips-config/ips-rules.conf” on my VyOS host.
l0crian@NPB7:~$ sudo su
root@NPB7:/home/l0crian# cd /config/containers/nprobe/ips-config/
root@NPB7:/config/containers/nprobe/ips-config# cat >ips-rules.conf
# Pool definition
{"pool":{"id":1,"name":"User Networks","ip": [ "192.168.2.0/24","10.0.95.0/24" ], "mac": []},"policy": {"id": 1} }
# Policy definition
{"policy":{"id":0, "name":"Default Rule", "default_marker": "pass" } }
{"policy":{"id":1, "name":"Limit Streaming", "default_marker": "pass", "markers": { "categories": { "Streaming": 50, "SocialNetwork": 50 }, "protocols": { "YouTube": 50 } } } }
^C
Configurng our Firewall Rules
If you’re doing this after reading the previous article, this is already set up for you. But in case you’re starting with this article, let’s go over configuring the firewall rule again. Check out the previous article if you need more information.
set firewall ipv4 forward filter rule 10 action 'queue'
set firewall ipv4 forward filter rule 10 description 'Allow return traffic from inside network - send to nprobe'
set firewall ipv4 forward filter rule 10 queue '25'
set firewall ipv4 forward filter rule 10 queue-options 'bypass'
set firewall ipv4 forward filter rule 10 state 'established'
set firewall ipv4 forward filter rule 10 state 'related'
- We set the action to ‘queue’, which tells nftables to use nfqueue.
- We set the queue to ’25’, which is what we defined when we started the nprobe container
- We set the queue-options to ‘bypass’. What this means is if the container is down or unreachable, traffic will still be forwarded. This prevents traffic from blackholing, but you can omit it if you actually want traffic to be blackholed if it is down.
- We are also only sending traffic to nprobe if it is return traffic from the internet. We are matching on all return traffic, but you can further scope the traffic by src/dst ip/port if needed.
We can check to see if we have traffic hitting our rule.
l0crian@NPB7# run show firewall ipv4 forward filter rule 10
Rule Information
---------------------------------
ipv4 Firewall "forward filter"
Rule Action Protocol Packets Bytes Conditions
------- -------- ---------- --------- ------- -----------------------------------------------------------
10 queue all 548 88481 ct state { established, related } queue flags bypass to 25
We can also see if we have any traffic being matched by nprobe. This let’s us know that nfqueue is sending traffic to nprobe.
l0crian@NPB7# run show container log nprobe | egrep "4117|4125|4133|4147"
24/Jun/2024 17:23:36 [nprobe.c:4117] Average traffic: [151.00 pps][All Traffic 309.29 Kb/sec][IP Traffic 277.44 Kb/sec][ratio 0.91]
24/Jun/2024 17:23:36 [nprobe.c:4125] Current traffic: [151.00 pps][309.29 Kb/sec]
24/Jun/2024 17:23:36 [nprobe.c:4133] L7 Proto Diff Total
24/Jun/2024 17:23:36 [nprobe.c:4147] Unknown/0 8.82 KB 8.82 KB
24/Jun/2024 17:23:36 [nprobe.c:4147] IMAPS/51 3.38 KB 3.38 KB
24/Jun/2024 17:23:36 [nprobe.c:4147] WireGuard/206 17.00 KB 17.00 KB
24/Jun/2024 17:23:36 [nprobe.c:4147] AmazonAWS/265 4.64 KB 4.64 KB
24/Jun/2024 17:23:36 [nprobe.c:4147] Azure/276 866 B 866 B
Configuring Shaping
Shaping is an outbound traffic function, so we need to apply the filter to the interface that will send our traffic outbound. So I am going to apply a shaper to my LAN interface (outbound towards the user).
We start by configuring a default policy. In this policy, I will say that it it can use 80% of bandwidth during times of congestion, and 100% if there is no congestion.
set qos policy shaper nprobe_shape default bandwidth '80%'
set qos policy shaper nprobe_shape default ceiling '100%'
Now we need to configure a class that will match our traffic. We have nprobe setting a mark of 50 for us, so we can match on that. That means any traffic that has a firewall mark of 50 will hit the policy and be shaped. We’re going to shape it down to 20% of available traffic, but allow it to use up to 100% of traffic if congestion isn’t present.
set qos policy shaper nprobe_shape class 10 bandwidth '20%'
set qos policy shaper nprobe_shape class 10 ceiling '100%'
set qos policy shaper nprobe_shape class 10 description 'Limit non-business traffic to 200Mbps'
set qos policy shaper nprobe_shape class 10 match Non_Business description 'Non Business traffic'
set qos policy shaper nprobe_shape class 10 match Non_Business mark '50'
And finally, we need to apply the policy to an interface. I apply mine in the egress direction on eth1, which is my LAN interface.
set qos interface eth1 egress 'nprobe_shape'
Let’s start up a YouTube video and see if we have any traffic matching on our QoS rule.
l0crian@NPB7# sudo tc -s filter show dev eth1
filter parent 1: protocol all pref 1 fw chain 0
filter parent 1: protocol all pref 1 fw chain 0 handle 0x32 classid 1:a
action order 1: police 0x1 rate 200Mbit burst 15325b mtu 2Kb action reclassify overhead 0b
ref 1 bind 1 installed 241 sec used 2 sec firstused 237 sec
Action statistics:
Sent 902264 bytes 1231 pkt (dropped 0, overlimits 354 requeues 0)
backlog 0b 0p requeues 0
NOTE: There’s not currently a VyOS op command to view QoS, but in a later release, you may be able to type “show qos shaper” to view shaping policies. You’d see output like this:
l0crian@NPB7# run show qos shaper
--------------------------------------------------------------------------------
Interface: eth1
Policy Name: test
Class Type BW Max. BW Bytes Pkts. Drops Queued
------- -------- ---------- --------- ---------- ------- ------- --------
root htb 1.000 Gb 1.000 Gb 230.895 MB 189085 0 0
10 fq_codel 200.000 Mb 1.000 Gb 34.939 MB 29348 0 0
default fq_codel 800.000 Mb 1.000 Gb 195.956 MB 159737 0 0Because this output is easier to read, I’m going to use it, but just remember that you may have to use the “tc” command to see the usage until that command is merged into VyOS.
Our policy is working. Let’s make the values arbitrarily low so that we can see this in practice. First, let’s remove the ceiling. If there is no ceiling configured, the bandwidth and ceiling will be the same. This means that the traffic cannot use any more than whatever is configured for the bandwidth.
delete qos policy shaper nprobe_shape class 10 ceiling '100%'
We can use the “stats for nerds” on YouTube to see the shaping in action. Let’s look at that before I drop the shaper down to a low amount. We’re looking at the Connection Speed specifically.
Now let’s drop the class down to 1Mbps.
set qos policy shaper test class 10 bandwidth 1mbit
We should see traffic getting queued in our class.
l0crian@NPB7# run show qos shaper
--------------------------------------------------------------------------------
Interface: eth1
Policy Name: test
Class Type BW Max. BW Bytes Pkts. Drops Queued
------- -------- ---------- --------- ---------- ------- ------- --------
root htb 1.000 Gb 1.000 Gb 129.275 MB 104015 0 0
10 fq_codel 1.000 Mb 1.000 Mb 1.165 MB 1311 0 1292
default fq_codel 800.000 Mb 1.000 Gb 128.110 MB 102704 0 0
The video should auto adjust the quality, and we’ll see the speed has been greatly reduced in the “stats for nerds” connection speed.
NOTE: Depending on how you have your firewall configured, you may need to restart the page if you have a stateful connection that is being matched.
That is a pretty drastic drop off, but that is in line with the 1Mbps that we set the traffic to.
Custom Protocols
It’s possible to create a custom protocol to be matched and filtered/shaped on. The syntax for the custom protocols is pretty easy.
host:"vyos.io"@VyOS
Here we create a custom protocol. Let’s break down the string.
- host:
- This says the custom protocol can be matched using the host “vyos.io”. This would include sub-domains like “support.vyos.io” and “forum.vyos.io”. It will match both DNS, as well as fields like a Common Name (CN) in a certificate.
- You can also match on a number of other fields. You can see an example here.
- “vyos.io”:
- This is the hostname that we want to match on.
- @VyOS:
- This is the name we give to our custom protocol.
Now we need to tell nprobe to use our custom files. I’m going to save the file as proto.txt in the “/config/containers/nprobe” folder. We need to modify the “ips-rules.conf” file. We place this at the top of the file and save it.
### Custom protocols definition ###
# Note: keep protocols and categories definitions before the policy definitions
{ "custom_protocols": "/data/nprobe/proto.txt" }
NOTE: Remember that we mapped “/data/nprobe” to “/config/containers/nprobe”
Now restart nprobe and check the nprobe container log. You should see this.
l0crian@NPB7# run restart container nprobe
Container "nprobe" restarted!
l0crian@NPB7# run show log container nprobe | match "custom protocols"
24/Jun/2024 18:11:40 [nprobe.c:10300] Loading nDPI custom protocols from /data/nprobe/proto.txt
Finally, we can go to https://vyos.io/ and see if nprobe matches our new protocol after.
l0crian@NPB7# run show container log nprobe | match VyOS
24/Jun/2024 18:13:41 [nprobe.c:4147] VyOS/305 47.60 KB 47.60 KB
Conclusion
That’s it for this post, we were able to configure a shaper to limit traffic, as well as create a custom protocol that nprobe can use.
There’s one final note on this. You may find that shaping traffic may be inconsistent for some services. Content is often times delivered from cache by Internet Service Providers. This is to save on their bandwidth when leaving their local environment.
An example of this is Disney+. In all of my testing, any video played on Disney+ was not matched by nprobe, because it was being cached by my ISP. This makes perfect sense when you think about it. Imagine if 1 million people wanted to play the newest Marvel movie, and your ISP needed to pull it from Disney’s servers for every user. That would be impractical for them. Delivering from a local cache to all of their customers makes far more sense.
You will have to make the decision to outright deny services that you’re unable to limit, or just assume that users won’t use too much bandwidth when using that service.
Leave a Reply