Why do you need an Application Aware Firewall (L7 Firewall)?

Blocking traffic at scale has always been a pain for network engineers. Compound that with internet traffic to specific services that may come from dozens of IPs and hostnames, and the whole thing becomes a nightmare. A few approaches to solve this are: using a proxy with content filtering, whitelist/blacklist, DNS filtering, and L7 Firewalls. L7 Firewalls are the topic of this article, but let’s talk about DNS filtering a little.

DNS filtering is a common method to scalably filter traffic to websites and services. Some common and freely available tools for this are PiHole and AdGuard. Using them couldn’t be simpler. Install them, then point all of your hosts to them for DNS. When the user tries to go somewhere you don’t want them to go, the DNS requests will get denied; preventing them from ever resolving an IP address to route to the site/service.

Story Time

I remember being back in middle school in the Photoshop lab. We take for granted how easy it is for modern computers to effectively run Photoshop, but back then, you needed a very capable machine to run it. This meant that this lab had some really nice and powerful computers in it; the best in the school!

So naturally, we all tried to play video games on it. The problem was, these being school computers, they were locked down. But “locked down” back then is not the same as sophisticated Group Policy heavy solutions we’re familiar with today. They simply whitelisted the names of the applications we were allowed to run. We quickly learned we could rename anything we wanted to run to word.exe, and we were good to play Unreal Tournament to our hearts content…..or at least as long as the teacher never walked to the back row of computers.

Getting back on topic

What was the point of that story? Well, back to DNS filtering, all the user has to do is change their DNS to some public DNS, and they’ve completely bypassed your DNS filtering. Additionally, this makes the resiliency and redundancy of DNS servers your concern, rather than allowing providers like Google or Cloudflare to use their massive anycast DNS deployments to serve you records. You could of course only allow destination traffic to UDP/53 from PiHole or AdGuard, but that would completely defeat the purpose of me telling you a story about how I was playing video games instead of learning how to edit photos in Photoshop.

Additionally, that won’t solve the requirement to host your own DNS servers, as well as in a world of CDNs and mirrors, blocking DNS may not be a consistent solution on its own. You may also wish to actually be able to reach a website, but only for some ports, or if their certificate matches “safe” values.

How do L7 Firewalls work?

L7 Firewalls use a combination of Deep Packet Inspection (DPI), matching traffic to well known ports or IP lists, and even inspecting DNS requests to help identify sites and services for traffic. Most major network venders have some form of DPI for their products, like NBAR for Cisco, or AppSecure for Juniper.

In the world of Linux and FreeBSD routing, the primary DPI Engine you’ll see in most NOS software is nDPI, which you’ll commonly find in tools like nprobe and ntopng. Their primary focus is on identification of flows, but nprobe has additional functionality to serve in IPS mode, which allows you to filter traffic based on applications and services that are identified. nprobe will be the utility we’re going to use in this article.

Our Topology

Our topology will be simple. We will have VyOS connected to the internet at the edge of our network. That will be where we filter our traffic.

Installing nprobe as a container

One of the best things VyOS did was add the ability to run containers with host networking. This makes VyOS incredibly extensible. Need SD-WAN capabilities, install something like ZeroTier. Need advanced IDP/IDS functionality, install Suricata. We need to make VyOS a L7 Firewall, so we’re going to install nprobe to add that functionality.

Adding the container image

We first need to add the container image to VyOS from “op” mode.

add container image ntop/nprobe:latest

Once the command line returns, we should have the image installed. We can verify with this.

l0crian@NPB7:~$ show container image 
REPOSITORY                   TAG         IMAGE ID      CREATED        SIZE
docker.io/ntop/nprobe        latest      be20b6bd6b78  15 months ago  388 MB

Configuring the container in VyOS

Below are the commands to be run in “conf mode”

sudo mkdir -p /config/containers/nprobe/

set container name nprobe allow-host-networks
set container name nprobe arguments '-i nf:25 --ips-mode /data/nprobe/ips-config/ips-rules.conf -n none -b 1'
set container name nprobe cap-add 'net-admin'
set container name nprobe cap-add 'sys-admin'
set container name nprobe image 'ntop/nprobe:latest'
set container name nprobe volume IPS_CONF destination '/data/nprobe'
set container name nprobe volume IPS_CONF source '/config/containers/nprobe/'

If you’re familiar with containers in VyOS, most of that will be pretty familiar to you, but the arguments warrant more explanation.

• -i nf:25:
  • nprobe uses NF_QUEUE to to send traffic to the nprobe container and return a verdict (or mark the traffic for further processing, like QoS). We create a nfqueue of 25 that we can call in a firewall rule later.
• –ips-mode /data/nprobe/ips-config/ips-rules.conf:
  • nprobe has a few different modes, but we specifically want to use it to allow/or block traffic based on an identified application or group of applications. You need to create a rules file for filtering, so you point to that rules file here. This will be the directory within the container, not on your VyOS host.
• -n none:
  • nprobe commonly serves as a method to send flows to a NetFlow collector. We don’t need that functionality, so we set the NetFlow collector to none.
• -b 1:
  • This switch gives us traffic statistics in the nprobe log, which can be useful to validate you are in fact identifying traffic.

Configuring the IPS Rules

nprobe uses a JSON based format to filter traffic. For this lab, we’re going be blocking user’s in the 192.168.2.0/24 and 10.0.95.0/24 networks from accessing Facebook and it’s many services. We’re also going to block everyone not in those subnets from accessing Twitch.

Here’s what our rules look like:

# Pool definition
{"pool":{"id":1,"name":"User Networks","ip": [ "192.168.2.0/24","10.0.95.0/24" ], "mac": []},"policy": {"id": 1} }

# Policy definition
{"policy":{"id":0,"name":"Default Rule", "default_marker": "pass", "markers": { "protocols": { "Twitch": "drop" } } } }
{"policy":{"id":1, "name":"Drop Facebook", "default_marker": "pass", "markers": { "protocols": { "Facebook": "drop" } } } }

Let’s look at those rules a little closer:

• We first create a pool definition. The pool ID is arbitrary, but should be unique from other pool IDs. The policy ID a the end of it will state which of the below policies will use that pool, in this case policy 1.
• The first policy (policy 0) is our default policy. Since we don’t have a configured pool for this policy, all traffic outside of the configured pools will use this policy. This policy will block access to Twitch and allow everything else.
• The second policy (policy 1) will block access to traffic identified as Facebook traffic. Putting all of the parts together, this means that users in the 192.168.2.0/24 and 10.0.95.0/24 networks cannot access Facebook, but will be allowed to access everything else.

NOTE: This is only a fraction of the match criteria for rules. You can also match on categories like Social Networks or Streaming sites, as well as GeoIP data. You can even match on information inside of the site certificate to make sure it is secure. You can read more about nprobe IPS rules here: https://www.ntop.org/guides/nprobe/ips_mode.html#policies-configuration

We then need to restart the container to load the rules.

restart container nprobe

NOTE: We only need to restart the container right now because it couldn’t find our rules config when it started. Any subsequent change to your rules config will be picked up by nprobe without restarting.

We can check the container logs to see if our rules were imported correctly:

l0crian@NPB7:~$ show log container nprobe | match "RuleManager|ips"
03/May/2024 16:21:26 [RuleManager.cpp:54] [line 2] Loading {"pool":{"id":1, "name":"User Networks","ip": [ "192.168.2.0/24","10.0.95.0/24" ], "mac": []},"policy": {"id": 1} }
03/May/2024 16:21:26 [RuleManager.cpp:54] [line 5] Loading {"policy":{"id":0, "name":"root policy rule", "default_marker": "pass", "markers": { "protocols": { "Twitch": "drop" } } } }
03/May/2024 16:21:26 [RuleManager.cpp:54] [line 6] Loading {"policy":{"id":1, "name":"Drop Facebook", "default_marker": "pass", "markers": { "protocols": { "Facebook": "drop" } } } }
03/May/2024 16:21:26 [ips.c:116] Loaded IPS rules from /data/nprobe//ips-config/ips-rules.conf

Let’s check our access to Facebook and Twitch from some users:

192.168.2.25:
C:\Users>curl -m 5 -I www.facebook.com
HTTP/1.1 301 Moved Permanently
Location: https://www.facebook.com/
Content-Type: text/plain
Server: proxygen-bolt
Date: Fri, 03 May 2024 15:17:13 GMT
Connection: keep-alive
Content-Length: 0

10.0.103.25
C:\Users>curl -m 5 -I www.twitch.tv
HTTP/1.1 301 Moved Permanently
Connection: close
Content-Length: 0
Retry-After: 0
Location: https://www.twitch.tv/
Date: Fri, 03 May 2024 15:17:43 GMT
Set-Cookie: twitch.lohp.countryCode=US; domain=.twitch.tv; path=/; expires=Wed, 28 May 2025 15:17:43 GMT;

Our traffic is working, but that’s to be expected since we haven’t pointed any firewall rules to nprobe.

Configuring the Firewall

We’re going to configure the forward chain to use nprobe. Let’s get a good base firewall config:

set firewall flowtable OT1 interface 'eth0'
set firewall flowtable OT1 interface 'eth1'
set firewall flowtable OT1 offload 'software'

set firewall ipv4 forward filter default-action 'drop'
set firewall ipv4 forward filter description 'Traffic through router'

set firewall ipv4 forward filter rule 5 action 'offload'
set firewall ipv4 forward filter rule 5 description 'Allow return traffic to fast path - Post nprobe'
set firewall ipv4 forward filter rule 5 offload-target 'OT1'
set firewall ipv4 forward filter rule 5 state 'established'
set firewall ipv4 forward filter rule 5 state 'related'

set firewall ipv4 forward filter rule 1000 action 'accept'
set firewall ipv4 forward filter rule 1000 inbound-interface name 'eth1'

• We configure the default-action as ‘drop’, which will drop all traffic not allowed by a rule.
• Rule 5 is leveraging the flow table feature in nftables to create a fast path for packet filtering. Once a flow enters the flow table, it no longer needs to have each packet filtered, which speeds up filtering greatly. Traffic is allowed by this rule if there is already connection state for it.
• Rule 1000 explicitly allows all traffic from ‘eth1’, which is my LAN side interface. This is what will create the return state that is matched by rule 5.

Now let’s create the rule for sending traffic to nprobe:

set firewall ipv4 forward filter rule 10 action 'queue'
set firewall ipv4 forward filter rule 10 description 'Allow return traffic from inside network - send to nprobe'
set firewall ipv4 forward filter rule 10 queue '25'
set firewall ipv4 forward filter rule 10 queue-options 'bypass'
set firewall ipv4 forward filter rule 10 state 'established'
set firewall ipv4 forward filter rule 10 state 'related'

• We set the action to ‘queue’, which tells nftables to use nfqueue.
• We set the queue to ’25’, which is what we defined when we started the nprobe container
• We set the queue-options to ‘bypass’. What this means is if the container is down or unreachable, traffic will still be forwarded. This prevents traffic from blackholing, but you can omit it if you actually want traffic to be blackholed if it is down.
• We are also only sending traffic to nprobe if it is return traffic from the internet. We are matching on all return traffic, but you can further scope the traffic by src/dst ip/port if needed.
  
  Our packet flow would be like this:
• A user sends a packet and it is received on eth1
• The packet is sent to the internet, and sent back where it will hit the forward chain.
• Rule 5 will only offload traffic with an allowed state, which isn’t present yet, so it skips Rule 5.
• Rule 10 is hit which will send the traffic to the nprobe container. The traffic will either be accepted or dropped. If the packet is accepted, then it will be added to the flow table, which will make subsequent filtering faster.

Let’s apply the firewall filters and check our traffic again.

192.168.2.25:
C:\Users>curl -m 5 -I www.facebook.com
curl: (28) Resolving timed out after 5001 milliseconds

10.0.103.25:
C:\Users>curl -m 5 -I www.twitch.tv
curl: (28) Operation timed out after 5003 milliseconds with 0 bytes received

Traffic is now being blocked to those sites. While I won’t show it here, those sites are indeed unreachable from the users in a browser or app.

We can see matches in the nprobe container log. The traffic statistics are updated about every minute. The values for the amount of traffic matched will not be accurate to actual traffic as nprobe moves traffic to the kernel after it is matched.

l0crian@NPB7:~$ show log container nprobe | egrep "L7 Proto|Twitch|Facebook"
03/May/2024 15:31:52 [nprobe.c:4133] L7 Proto                   Diff      Total
03/May/2024 15:31:52 [nprobe.c:4147]    Facebook/119          116.82 KB  702.24 KB
03/May/2024 15:31:52 [nprobe.c:4147]    Twitch/195             120.54 KB  659.13 KB

You can view just the traffic stats in the log with this command:

show container log nprobe | egrep  "4117|4125|4133|4147"

nprobe limitations

When you install the nprobe container, it will install as nprobe Pro, with a demo license. The demo license has a number of limitations, mainly total flows exported and number of flows exported. It also expires after 300 seconds. What is interesting is, the demo license doesn’t appear to have any limitations to using it in IPS mode. I can easily max out my 1Gbps internet. If there is a limit, it seems to be beyond that of a simple household’s internet requirements. Let me know if you find any limitations related to IPS mode.

There is a general limit that is present for nprobe Pro whether you have a license or not. That is there is a limit of 4 pools/rules. While that might seem like very little, you can chain a lot of different “pass”, “drop”, and marks in each rule. Additionally, you can drop traffic to categories like Streaming sites, rather than needing to match Disney Plus, NetFlix, or Hulu. So with a little planning, you can lump users/networks into groups to be filtered.

nprobe performance

I mentioned that there doesn’t appear to be any throttling by the container in IPS mode, but I should also speak to the general performance of this solution. I am only using one instance of nprobe, so it only uses one queue, and my system can easily handle 1Gbps of throughput without much stress. You can add additional nprobe containers with different queues and load balance traffic. My computer is a mini-PC which has a 13700H, which is a mobile CPU. Here is my utilization when maxing 1Gbps of traffic:

Use case ideas

Here’s some ideas that you can use for this:

• If you have kids that do school from home, you can add time based restrictions to the firewall rules, that will block access to sites that could be distractions (social media, streaming sites, etc…).
• You can shape all traffic for a specific site/service/category (like Netflix or all Streaming Sites) to something like 20Mpbs.

Conclusion

That wraps this up. This was actually very easy to setup and get running. There’s a little bit of research you may need to do if you’re not comfortable with NF_QUEUE, but it’s not too deep of a concept.

A friend of mine is turning my blog posts into videos. Check out the video for this post: