Excessive Memory Usage with Multiple WAF Instances in Coraza Using CRS

[jptosso]

Issue Summary

It is a well-documented issue that Coraza consumes significant memory when multiple coraza.WAF instances are created with the CRS. Previously, we mitigated this issue by employing memoization. However, a new challenge has emerged: we need to share Coraza's memory across Apache/Nginx workers efficiently.

Problem Statement

In my opinion, it is unreasonable for 50 CRS implementations to occupy 2GB of memory. This raises a critical need to investigate the source of this overhead and implement strategies to reduce the memory footprint of CRS.

Expected Outcome

Identify the root cause(s) of the excessive memory usage.
Explore and implement solutions to minimise memory consumption without compromising functionality.
Additional Context:

The use of memoization previously provided a workaround, but the current scenario demands a more scalable solution, especially for environments with Apache/Nginx workers.
Your insights and suggestions on addressing this issue would be greatly appreciated.

[fzipi]

Was there any progress here? Is this repeatable? Can we provide a script, or something, to reproduce the claim that 50 crs implementations use 2GB of memory?

[hseljenes]

I'm also looking at high memory usage for a coraza-caddy deployment.

Flat	Flat%	Sum%	Cum	Cum%	Name	Inlined?
2498.34MB	80.33%	80.33%	2510.34MB	80.71%	github.com/corazawaf/coraza/v3/internal/corazawaf.(*Rule).transformArg	
84.47MB	2.72%	83.04%	84.47MB	2.72%	github.com/corazawaf/coraza/v3/internal/collections.(*Map).FindAll	
80.76MB	2.60%	85.64%	80.76MB	2.60%	github.com/corazawaf/coraza/v3/internal/collections.(*NamedCollectionNames).FindAll	
40.54MB	1.30%	86.94%	58.05MB	1.87%	crypto/internal/bigmod.(*Nat).montgomeryMul	
36.52MB	1.17%	88.12%	36.52MB	1.17%	crypto/internal/bigmod.(*Nat).reset	(inline)
28.42MB	0.91%	89.03%	28.42MB	0.91%	github.com/corazawaf/coraza/v3/internal/collections.(*Map).SetIndex	
26.88MB	0.86%	89.89%	27.88MB	0.90%	github.com/corazawaf/coraza/v3/internal/collections.(*Map).Add	
21.25MB	0.68%	90.58%	212.63MB	6.84%	github.com/corazawaf/coraza/v3/internal/corazawaf.(*Transaction).GetField	
20MB	0.64%	91.22%	20MB	0.64%	github.com/petar-dambovaliev/aho-corasick.AhoCorasick.IterByte	
16.65MB	0.54%	91.75%	2868.21MB	92.22%	github.com/corazawaf/coraza/v3/internal/corazawaf.(*Rule).doEvaluate	
16.50MB	0.53%	92.29%	16.50MB	0.53%	strings.(*Builder).WriteString	(inline)
13.93MB	0.45%	92.73%	23.86MB	0.77%	io.copyBuffer	
13MB	0.42%	93.15%	16.50MB	0.53%	github.com/corazawaf/libinjection-go.isXSS	
10.01MB	0.32%	93.47%	20.80MB	0.67%	github.com/corazawaf/coraza/v3/internal/corazawaf.(*Transaction).MatchRule	
9.77MB	0.31%	93.79%	88.53MB	2.85%	github.com/corazawaf/coraza/v3/internal/collections.(*ConcatCollection).FindAll	
7.38MB	0.24%	94.02%	86.35MB	2.78%	github.com/corazawaf/coraza/v3/internal/collections.(*ConcatKeyed).FindAll	
4.50MB	0.14%	94.17%	24.50MB	0.79%	github.com/petar-dambovaliev/aho-corasick.AhoCorasick.Iter	(inline)
2MB	0.06%	94.23%	29.38MB	0.94%	github.com/corazawaf/coraza/v3/internal/corazawaf.(*Transaction).matchVariable	

Source view:

github.com/corazawaf/coraza/v3/internal/corazawaf.(*Rule).transformArg
/go/path/pkg/mod/github.com/corazawaf/coraza/[email protected]/internal/corazawaf/rule.go

  Total:      2.44GB     2.45GB (flat, cum) 80.71%
    390            .        5MB           ???
    394          8MB        8MB           ???
    398            .          .           ???
    412            .        7MB           ???
    413         94MB       94MB           ???
    415       2.34GB     2.34GB           ???
github.com/corazawaf/coraza/v3/internal/collections.(*Map).FindAll
/go/path/pkg/mod/github.com/corazawaf/coraza/[email protected]/internal/collections/map.go

  Total:     84.47MB    84.47MB (flat, cum)  2.72%
    104      84.47MB    84.47MB           ???
github.com/corazawaf/coraza/v3/internal/collections.(*NamedCollectionNames).FindAll
/go/path/pkg/mod/github.com/corazawaf/coraza/[email protected]/internal/collections/named.go

  Total:     80.76MB    80.76MB (flat, cum)  2.60%
    117      80.76MB    80.76MB           ???
crypto/internal/bigmod.(*Nat).reset
/usr/local/go/src/crypto/internal/bigmod/nat.go

  Total:     36.52MB    36.52MB (flat, cum)  1.17%
     97      36.52MB    36.52MB           ???
crypto/internal/bigmod.(*Nat).resetFor
/usr/local/go/src/crypto/internal/bigmod/nat.go

  Total:    512.25kB   512.25kB (flat, cum) 0.016%
    510     512.25kB   512.25kB           ???
crypto/internal/bigmod.(*Nat).montgomeryMul
/usr/local/go/src/crypto/internal/bigmod/nat.go

  Total:     40.54MB    40.54MB (flat, cum)  1.30%
    596      40.54MB    40.54MB           ???
crypto/internal/bigmod.(*Nat).montgomeryMul
/usr/local/go/src/crypto/internal/bigmod/nat.go

  Total:           0    17.51MB (flat, cum)  0.56%
    651            .    17.51MB           ???

I built with this branch https://github.com/corazawaf/coraza-caddy/tree/dependabot/go_modules/main/github.com/corazawaf/coraza/v3-3.2.0 since it was on the latest coraza.

The top of my go.mod looks like this

require (
	github.com/caddyserver/caddy/v2 v2.7.6
	github.com/caddyserver/replace-response v0.0.0-20231221003037-a85d4ddc11d6
	github.com/corazawaf/coraza-caddy/v2 v2.0.0-rc.3.0.20240620173425-4d5437f2dcb3
	github.com/gr33nbl00d/caddy-revocation-validator v1.0.4
)

I'm fairly new to golang but it seems like it's being used by this transformation cache map?

If it matters this is an API app that uses RSA client certificates for authentication.

---

Edit: Since the transformation cache is only used when you have transformations I dropped the t:lowercase from the requestBodyProcessor rules in the default config and memory is much more stable at around 300mb.