v2.1.0: Merge pull request #450 from simsong/rel-2.10

[simsong]

Announcing bulk_extractor 2.1.0
January 22, 2024
RELEASE NOTES

The digital forensics tool bulk_extractor version 2.1.0 is now available for general use.

Release download point:
https://github.com/simsong/bulk_extractor/releases

GIT repository:
https://github.com/simsong/bulk_extractor

I am pleased to announce the general availability of bulk_extractor
version 2.1. This is the first release of bulk_extractor version 2
that is recommended for general use.

Bulk_extractor 2 is a significant rewrite of bulk_extractor. Verison 2
significantly improves the performance and portability of version 1.
The rewrite started in 2016 and was largely completed by January
2021.

Details of the rewrite, including a detailed report of the performance
improvements and lessons learned, can be found in Sharpening Your
Tools: Updating bulk_extractor for the 2020s, Simson Garfinkel and
Jon Stewart. Communications of the ACM, August 2023,

Bulk_extractor version 2.1 is the first stable version of bulk_extractor version 2 that is recommended for general use. It corrects a problem with the string search scanner that caused bulk_extractor to hang on open-ended regular expressions such as [a-z]*@company.com specified with the -F flag. With version 2.1, we have replaced the C++17 regex compiler with Google's RE2 regex compiler that avoids backtracking. As a result, these open-ended regular expressions no longer hang.

2.0/2.1 Improvements over Version 1:

• BE2 is significantly faster on multi-core systems than BE1.

Release 2.1 Limitations

• BEViewer is not included in this release. Although it works with Version 2, it is not yet officially supported.

• scan_outlook and scan_hiberfile are now disabled by default because they did not have unit tests. These scanners can be re-enabled by specifying -eoutlook and -ehiberfile on the command line.

• scan_aes no longer scans for 192-bit AES keys by default, although this behavior can be re-enabled.

Known bugs:

• The RAR decompressor does not reliably decompress all RAR files and only supports RAR v1, v2, and v3.

• The RAR scanner will not reliably name carved RAR file components that contain UTF-8 characters in their name.

You can help

We are looking for help to implement the following algorithms:

• WkdmDecompress - http://www.opensource.apple.com/source/xnu/xnu-1456.1.26/iokit/Kernel/WKdmDecompress.c

• xz, 7zip, and LZMA/LZMA2 decompression

• lzo decompression

• BZIP2 decompression

• CAB decompression

• Scanning for the start of BitLocker protected volumes.

• NTFS decompression

• Better handling of MIME encoding

• Process more data with -e xor and look for CCN hits. Most will be false positives

• Demonstration of bulk_extractor running on a grid (how fast can it run?)

• Python Bridge - run multiple copies of python to let scanners be written in python

• scan_pipe - runs every sbuf through an external program.

---

This discussion was created from the release v2.1.0: Merge pull request #450 from simsong/rel-2.10.

[presianbg]

Hi, thanks for the keeping this project alive for such a long time.

With the recent release and changes:

Bulk_extractor version 2.1 is the first stable version of bulk_extractor version 2 that is recommended for general use. It corrects a problem with the string search scanner that caused bulk_extractor to hang on open-ended regular expressions such as [a-z]*@company.com specified with the -F flag. With version 2.1, we have replaced the C++17 regex compiler with Google's RE2 regex compiler that avoids backtracking. As a result, these open-ended regular expressions no longer hang.

I was not able to compile the binary no matter what. I've tried both, for Linux and cross-compile (Windows) configuration scripts to no avail.

What is the recommended distro for this release ?

Cheers,
PY

[simsong]

Can you post your error? You need to install re2. Which version of Linux are you using? The prep scripts prep for Ubuntu and fedora and Mac. Did you run them? How should I document dependencies so that people will see them? Thanks.

…

_____________

On Tue, Feb 13, 2024 at 11:29 AM Presian Yankulov ***@***.***> wrote: Hi, thanks for the keeping this project alive for such a long time. With the recent release and changes: Bulk_extractor version 2.1 is the first stable version of bulk_extractor version 2 that is recommended for general use. It corrects a problem with the string search scanner that caused bulk_extractor to hang on open-ended regular expressions such as ***@***.*** specified with the -F flag. With version 2.1, we have replaced the C++17 regex compiler with Google's RE2 regex compiler that avoids backtracking. As a result, these open-ended regular expressions no longer hang. I was not able to compile the binary no matter what. I've tried both, for Linux and cross-compile (Windows) configuration scripts to no avail. What is the recommended distro for this release ? Cheers, PY — Reply to this email directly, view it on GitHub <#451 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAMFHLF2B56WHJMEV4DSO5LYTOIHLAVCNFSM6AAAAABCJVB5JOVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4DINJWGE2DO> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[presianbg]

I understand your frustration, sorry for omitting important details.
I've tried both Ubuntu22/win and Fedora 39/win (bypassed the OS check from Fedora36 script).
The preparation scripts for the linux binary are not producing any errors, rather not starting the compilation.
Now I can see that I need to execute this:
sh bootstrap.sh && ./configure && make
which I haven't.

I will install a clean VM and try again the compilation process.
The cross compile version for Windows looks like a bit more tricky, do you have any recommendation which exact distro should be used ?

[simsong]

cross-compile isn't working for the new release right now.
Yes, if you do a git clone, you natural need to create the configure script. THat's true of any automake system. Putting a configure script into the git repo doesn't make sense.

re2 is a problem, and I'm thinking of including all of it and ansible as sub modules.

[presianbg]

Meanwhile, looks like someone at the Kali linux project is keeping the bulk-extractor package up-to-date with your releases.
https://pkg.kali.org/pkg/bulk-extractor

Consider putting this info somewhere in the wiki, might save some time to you and anyone looking to use it without going trough the hustle of manual compilation.

[MikeOCa]

Hi @simsong,

Thank you for continuing to develop and support this fantastic tool.

I would like to ask a question, can lightgrep be integrated into version 2.1.0 of BE?

Kr

[simsong]

We're waiting for the new version of lightgrep. Speak to @jonstewart

[MikeOCa]

Thank you for the quick reply, I will ask @jonstewart.

[jonstewart]

It's coming. There's a draft PR but it needs to target a lightgrep release, and I'm working to resolve a long-standing issue with lightgrep as part of that (a potential stack overflow under extreme conditions, not otherwise a correctness bug).