Bouncer metrics collector fails when using custom nftables rules

[Icosa-Consulting]

Hello,

I have a configuration that only sets the decisions in element lists using more customized rules for processing. when I enable the prometheus collector on the bouncer I get no stats and see errors in the logs (listed below)

Config: (Im using the .local extension to override the .yml file in /etc/crowdsec/bouncers)

NFT Chain Rule

    set crowdsec-blacklists4 {
            type ipv4_addr;
            flags timeout;
    }

    chain block_ingress4 {
            # IPv4 Ingress
            ip saddr @bad-actors4 log prefix "DROP-IN-actor" group 0 counter drop
            ip saddr @blackhole4 log prefix "DROP-IN-blackhole" group 0 counter drop
            ip saddr @crowdsec-blacklists4 log prefix "DROP-IN4" group 0 counter drop
            return
    }

Override Config

blacklists_ipv4: crowdsec-blacklists4
blacklists_ipv6: crowdsec-blacklists6

nftables:
ipv4:
enabled: true
set-only: true
table: crowdsec4
chain: block_ingress4
ipv6:
enabled: true
set-only: true
table: crowdsec6
chain: block_ingress6

nftables_hooks:

• input
• forward

prometheus:
enabled: true
listen_addr: 172.20.250.1
listen_port: 60601

LOGS

The crowdsec-firewall-bouncer.log shows

time="29-12-2023 16:25:35" level=info msg="1 decision added"
time="29-12-2023 16:25:42" level=error msg="can't collect dropped packets for ipv4 from nft: exit status 1"
time="29-12-2023 16:25:42" level=error msg="can't collect total banned IPs for ipv4 from nft:exit status 1"
time="29-12-2023 16:25:42" level=error msg="can't collect dropped packets for ipv6 from nft: exit status 1"
time="29-12-2023 16:25:42" level=error msg="can't collect total banned IPs for ipv6 from nft:exit status 1"
time="29-12-2023 16:25:52" level=error msg="can't collect dropped packets for ipv4 from nft: exit status 1"
time="29-12-2023 16:25:52" level=error msg="can't collect total banned IPs for ipv4 from nft:exit status 1"
time="29-12-2023 16:25:52" level=error msg="can't collect dropped packets for ipv6 from nft: exit status 1"
time="29-12-2023 16:25:52" level=error msg="can't collect total banned IPs for ipv6 from nft:exit status 1"
time="29-12-2023 16:26:02" level=error msg="can't collect dropped packets for ipv4 from nft: exit status 1"
time="29-12-2023 16:26:02" level=error msg="can't collect total banned IPs for ipv4 from nft:exit status 1"
time="29-12-2023 16:26:02" level=error msg="can't collect dropped packets for ipv6 from nft: exit status 1"
time="29-12-2023 16:26:02" level=error msg="can't collect total banned IPs for ipv6 from nft:exit status 1"
time="29-12-2023 16:26:12" level=error msg="can't collect dropped packets for ipv4 from nft: exit status 1"
time="29-12-2023 16:26:12" level=error msg="can't collect total banned IPs for ipv4 from nft:exit status 1"
time="29-12-2023 16:26:12" level=error msg="can't collect dropped packets for ipv6 from nft: exit status 1"
time="29-12-2023 16:26:12" level=error msg="can't collect total banned IPs for ipv6 from nft:exit status 1"
time="29-12-2023 16:26:22" level=error msg="can't collect dropped packets for ipv4 from nft: exit status 1"
time="29-12-2023 16:26:22" level=error msg="can't collect total banned IPs for ipv4 from nft:exit status 1"
time="29-12-2023 16:26:22" level=error msg="can't collect dropped packets for ipv6 from nft: exit status 1"
time="29-12-2023 16:26:22" level=error msg="can't collect total banned IPs for ipv6 from nft:exit status 1"

Sourced from /pkg/nftables/metrics.go

metrics.go

It appears in the code the collector is looking for a specific chain name, which I have specified in the config.
Which translates into:

nft -j list chain inet crowdsec4 block_ingress4

The error "exit status 1" must be coming from the NFT binary itself. If I run that command I get the following:

{ "nftables": [ { "metainfo": { "version": "1.0.6", "release_name": "Lester Gooch #5", "json_schema_version": 1 } }, { "chain": { "family": "inet", "table": "crowdsec4", "name": "block_ingress4", "handle": 1 } }, { "rule": { "family": "inet", "table": "crowdsec4", "chain": "block_ingress4", "handle": 19, "expr": [ { "match": { "op": "==", "left": { "payload": { "protocol": "ip", "field": "saddr" } }, "right": "@bad-actors4" } }, { "log": { "prefix": "DROP-IN-actor", "group": 0 } }, { "counter": { "packets": 47, "bytes": 2162 } }, { "drop": null } ] } }, { "rule": { "family": "inet", "table": "crowdsec4", "chain": "block_ingress4", "handle": 20, "expr": [ { "match": { "op": "==", "left": { "payload": { "protocol": "ip", "field": "saddr" } }, "right": "@blackhole4" } }, { "log": { "prefix": "DROP-IN-blackhole", "group": 0 } }, { "counter": { "packets": 0, "bytes": 0 } }, { "drop": null } ] } }, { "rule": { "family": "inet", "table": "crowdsec4", "chain": "block_ingress4", "handle": 21, "expr": [ { "match": { "op": "==", "left": { "payload": { "protocol": "ip", "field": "saddr" } }, "right": "@crowdsec-blacklists4" } }, { "log": { "prefix": "DROP-IN4", "group": 0 } }, { "counter": { "packets": 249, "bytes": 12882 } }, { "drop": null } ] } }, { "rule": { "family": "inet", "table": "crowdsec4", "chain": "block_ingress4", "handle": 22, "expr": [ { "return": null } ] } } ] }

Mitigation

I've currently disabled the prometheus collector in the config, which stops the error in the logs, however I'd still like to collect some metrics with it. I know you guys are busy over there so I'm not stressed about it.

[Icosa-Consulting]

Also,

My bad for not using the template. I just wanted to post this more for awareness.

[ne20002]

Hi @mmetc
I can confirm the problem with the current 0.0.29-rc3 on OpenWrt.

If I use the set-only mode where I create the nftables rules with the scripts of OpenWrt package the log file states:
time="2024-07-16T11:54:47Z" level=error msg="can't collect dropped packets for ipv4 from nft: while running /usr/sbin/nft -j list chain ip crowdsec crowdsec-chain: exit status 1"
The names of the tables (crowdsec, crowdsec6) and chains are identical to the ones created by the bouncer itself if not running in set-only mode.
The names of the chains are indeed crowdsec-chain-input, crowdsec-chain-forward, crowdsec6-chain-input and crowdsec6-chain-forward whereas the configured chain names in the config are crowdsec-chain and crowdsec6-chain.

I'm not sure what the problem is but it seems as if the -input and -forward postfixes are not used when trying to build the chain names for collecting the data via using ntf in set-only mode.

For OpenWrt I need to create the rules myself. Even though the current version is near to what the OpenWrt package already used, in OpenWrt the filtering also needs to act on defined interfaces and conntrack state (to reduce usage of queries to the set).

[ne20002]

This is my setup on OpenWrt with bouncer 0.0.29-rc3 for comparison:

With the OpenWrt scripts creating the rules and the bouncer running set-only:

table ip crowdsec {
	set crowdsec-blacklists {
		type ipv4_addr
		flags timeout
		elements = { ... }
	}

	chain crowdsec-chain-input {
		type filter hook input priority filter + 4; policy accept;
		iifname { "wg1", "eth1" } ct state new ip saddr @crowdsec-blacklists counter packets 0 bytes 0 drop
	}

	chain crowdsec-chain-forward {
		type filter hook forward priority filter + 4; policy accept;
		iifname { "wg1", "eth1" } ct state new ip daddr != 224.0.0.0/4 ip saddr @crowdsec-blacklists counter packets 17 bytes 928 drop
	}
}

table ip6 crowdsec6 {
	set crowdsec6-blacklists {
		type ipv6_addr
		flags timeout
		elements = { ... }
	}

	chain crowdsec6-chain-input {
		type filter hook input priority filter + 4; policy accept;
		iifname { "wg1", "eth1" } ct state new ip6 saddr @crowdsec6-blacklists counter packets 0 bytes 0 drop
	}

	chain crowdsec6-chain-forward {
		type filter hook forward priority filter + 4; policy accept;
		iifname { "wg1", "eth1" } ct state new ip6 saddr @crowdsec6-blacklists counter packets 84 bytes 6720 drop
	}
}

This is how the bouncer creates the rules if I don't use set-only:

table ip crowdsec {
	set crowdsec-blacklists {
		type ipv4_addr
		flags timeout
		elements = { ... }
	}

	chain crowdsec-chain-input {
		type filter hook input priority filter + 4; policy accept;
		ip saddr @crowdsec-blacklists counter packets 0 bytes 0 drop
	}

	chain crowdsec-chain-forward {
		type filter hook forward priority filter + 4; policy accept;
		ip saddr @crowdsec-blacklists counter packets 0 bytes 0 drop
	}
}

table ip6 crowdsec6 {
	set crowdsec6-blacklists {
		type ipv6_addr
		flags timeout
		elements = { ... }
	}

	chain crowdsec6-chain-input {
		type filter hook input priority filter + 4; policy accept;
		ip6 saddr @crowdsec6-blacklists counter packets 0 bytes 0 drop
	}

	chain crowdsec6-chain-forward {
		type filter hook forward priority filter + 4; policy accept;
		ip6 saddr @crowdsec6-blacklists counter packets 0 bytes 0 drop
	}
}

In the second version (rules created by the bouncer) metrics on blocked ips are working. In the first version (rules created by the OpenWrt script) the metrics on blocked ips are not working and the logfile is flooded with the error messages as shown above.

[blotus]

Hello,

I'm going to close this issue as the way metrics are computed was changed in 0.0.30 (the nft binary is no longer used).

Do not hesitate to re-open the issue if it's still not fixed.