-
Notifications
You must be signed in to change notification settings - Fork 1.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
hyperscan: add caching mechanism for hyperscan contexts v5 #11811
Conversation
This variant of hashlittle2() ensures that it avoids accesses beyond the last byte of the string, which will cause warnings from tools like Valgrind or Address Sanitizer.
Cache Hyperscan serialized databases to disk to prevent compilation of the same databases when Suricata is run again with the same ruleset. The current work operates in /tmp/ folder and caches individual signature group heads - potentially the ruleset might be even slightly changed and it still can reuse part of the unchanged signature groups. Loading *fresh* ET Open ruleset: 19 seconds Loading *cached* ET Open ruleset: 07 seconds
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #11811 +/- ##
==========================================
- Coverage 82.58% 82.49% -0.10%
==========================================
Files 914 914
Lines 249500 249821 +321
==========================================
+ Hits 206045 206078 +33
- Misses 43455 43743 +288
Flags with carried forward coverage won't be shown. Click here to find out more. |
@@ -1685,6 +1685,10 @@ detect: | |||
toclient-groups: 3 | |||
toserver-groups: 25 | |||
sgh-mpm-context: auto | |||
# Cache MPM contexts to the disk for avoid rule compilation at the startup. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
interestingly, this version had better preposition usage :P
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Commit explanations look good to me, considering past reviews.
I did notice however that we failed to notice that they're not referencing the redmine ticket >_<'
|
||
**Note**: | ||
You might need create and adjust permissions to the default caching folder path, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit:
You might need create and adjust permissions to the default caching folder path, | |
You might need to create and adjust permissions to the default caching folder path, |
Hitting an issue when running with privilege dropping. For example:
It appears the caching is done before privileges are dropped, and there is no access after privileges are dropped. And then during a rule reload, where privileges are dropped, we don't have access to this directory. |
Ok, so the directory creation is done before privilege dropping. If that can be delayed til after privilege dropping we should be good. For example, the file store directory layout is made after dropping privileges. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Comments in-line with respect to some required changes when running with privilege dropping.
WARNING:
Pipeline 22886 |
goto #11887 |
Followup of #11774
Cache Hyperscan serialized databases to disk to prevent compilation of the same databases when Suricata is run again with the same ruleset.
The current work operates in the logging folder and caches individual Hyperscan databases - potentially the ruleset might be even slightly changed and it still can reuse part of the unchanged signature groups.
Loading fresh ET Open ruleset: 19 seconds
Loading cached ET Open ruleset: 07 seconds
Link to ticket: https://redmine.openinfosecfoundation.org/issues/7170
Describe changes:
v5:
v4:
v3
v2
v1