Update lightgrep scanner for bulk_extractor 2.0 #421
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…rmance regression. The timings below are from the following command: ./src/bulk_extractor -F ../lightgrep/pytest/keys/shuf10.txt -Z -o ~/be_timed_output_without_thread_local_`printf %04d $i` -E scan_lightgrep ~/ev/terry-2009-12-11-002.E01 Thread_local? Clocktime (Min.) Clocktime (Max.) Clocktime (Average) Scan Lightgrep Time (Min.) Scan Lightgrep Time (Max.) Scan Lightgrep Time (Average) FALSE 162.965479 168.628229 164.2545712 494.810946 528.368114 504.1799554 TRUE 163.681386 173.587754 167.233617 499.815450 532.324335 516.4901762 This reverts commit 0ca43ec.
|
I'm going to close this and re-open it as a draft PR. |
|
Apparently that's not how you did it. I found instructions here. It's a draft now. |
Codecov Report
@@ Coverage Diff @@
## main #421 +/- ##
=======================================
Coverage 47.94% 47.94%
=======================================
Files 112 112
Lines 13224 13224
=======================================
Hits 6339 6339
Misses 6885 6885 📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more |
|
I didn't know about draft PRs, TIL. |
|
Is this PR ready to go? |
|
[jeez, terrible formatting for reply-by-email]
Good question: yes, and no.
We think this PR works, but it depends on the current main branch of lightgrep. To make for a good user experience, we need to release a new version of lightgrep and then update this PR with updated build scripts that can pull that release.
The current plan is to get the new release of lightgrep out before the end of the year. It has been under continual development for the past few months, as a ~25% time project. It has several minor improvements and bug fixes (per the spirit of the ACM paper). If you’ve got a specific date in mind for a new bulk_extractor release, that would be good to know and we may be able to adjust.
We are _not_ entirely confident in our usage of the new sbuf/scanner API. We would _love_ a code review of this PR from you. We could also push up the requisite lightgrep code for you to test, if you’d prefer.
|
|
Hi. What's the status on this? |
|
We're getting ready to make a new lightgrep release for this to target. Can you review |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR has the following functionality changes:
-fand-Foptions, by default searching for both UTF-8 and UTF-16LE versions, with case-sensitivityscan_accts_lgscan_base16_lgscan_email_lgscan_gps_lgWith the deletion of other lightgrep-based scanners, we were able to delete a lot of scaffolding code.
This PR is not yet ready, but we're opening it for comment. The following remains to be done:
Please let us know if you have any questions or comments.