Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

hyperscan: add caching mechanism for hyperscan contexts v7 #11887

Closed

Conversation

lukashino
Copy link
Contributor

Followup of #11811

Cache Hyperscan serialized databases to disk to prevent compilation of the same databases when Suricata is run again with the same ruleset.
The current work operates in the logging folder and caches individual Hyperscan databases - potentially the ruleset might be even slightly changed and it still can reuse part of the unchanged signature groups.
Loading fresh ET Open ruleset: 19 seconds
Loading cached ET Open ruleset: 07 seconds

Link to ticket: https://redmine.openinfosecfoundation.org/issues/7170

Describe changes:
v7: (v6 was private)

  • fix docs and add ticket number to the commit
  • fix privilege drop issue, files are created after privilege drop, tested also rule reload - it worked fine
  • refactor the util-mpm-hs code, primarily prepare function
  • rebase

v5:

  • rebased
  • commit message update
  • docs update

v4:

  • rebased
  • changed the default caching directory to somewhere /var/lib/suricata/cache/hs
  • custom cache directory path option added
  • docs added
  • the default settings changed - enabled on the config generation, disabled when the option is not present in the config

v3

  • rebased
  • MPM caching is still left on by default.

v2

  • improved styling to follow Suricata code styleguide
  • increased cache file name length from 10 to 20 characters
  • cache file name is a hash of the patterns - now only HS relevant fields are hashed - as long as the group of patterns itself is not changed then it is reused
  • minor refactors
  • added a safe variant of littlehash2 function
  • added suricata.yaml option to enable/disable caching
  • changed the storage location to the configured logging directory

v1

  • initial work to cache and load Hyperscan databases from the disk

Lukas Sismis added 6 commits October 5, 2024 20:31
This variant of hashlittle2() ensures that it avoids
accesses beyond the last byte of the string, which will
cause warnings from tools like Valgrind or Address
Sanitizer.
Cache Hyperscan serialized databases to disk to prevent compilation
of the same databases when Suricata is run again with the same
ruleset.
The current work operates in /tmp/ folder and caches individual
signature group heads - potentially the ruleset might be even
slightly changed and it still can reuse part of the unchanged
signature groups.
Loading *fresh* ET Open ruleset:  19 seconds
Loading *cached* ET Open ruleset: 07 seconds

Ticket: 7170
Copy link

codecov bot commented Oct 6, 2024

Codecov Report

Attention: Patch coverage is 16.62921% with 371 lines in your changes missing coverage. Please review.

Project coverage is 82.47%. Comparing base (3f0512e) to head (493446e).

Additional details and impacted files
@@            Coverage Diff             @@
##           master   #11887      +/-   ##
==========================================
- Coverage   82.61%   82.47%   -0.14%     
==========================================
  Files         912      912              
  Lines      249354   249718     +364     
==========================================
- Hits       205992   205949      -43     
- Misses      43362    43769     +407     
Flag Coverage Δ
fuzzcorpus 60.49% <3.37%> (-0.14%) ⬇️
livemode 18.71% <4.49%> (-0.14%) ⬇️
pcap 44.07% <4.49%> (-0.02%) ⬇️
suricata-verify 61.96% <5.05%> (-0.04%) ⬇️
unittests 58.85% <15.73%> (-0.08%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

@lukashino
Copy link
Contributor Author

Continues in #11888

@lukashino lukashino closed this Oct 6, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

1 participant