Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

reduce clone in search.rs #1518

Merged
merged 1 commit into from
Nov 30, 2024
Merged

reduce clone in search.rs #1518

merged 1 commit into from
Nov 30, 2024

Conversation

YamatoSecurity
Copy link
Collaborator

I've reduced some usage of clone in search.rs

Before:

./hayabusa-2.18.0 search -d ../hayabusa-sample-evtx -r ".*" -o before.csv --debug -C

┏┓ ┏┳━━━┳┓  ┏┳━━━┳━━┓┏┓ ┏┳━━━┳━━━┓
┃┃ ┃┃┏━┓┃┗┓┏┛┃┏━┓┃┏┓┃┃┃ ┃┃┏━┓┃┏━┓┃
┃┗━┛┃┃ ┃┣┓┗┛┏┫┃ ┃┃┗┛┗┫┃ ┃┃┗━━┫┃ ┃┃
┃┏━┓┃┗━┛┃┗┓┏┛┃┗━┛┃┏━┓┃┃ ┃┣━━┓┃┗━┛┃
┃┃ ┃┃┏━┓┃ ┃┃ ┃┏━┓┃┗━┛┃┗━┛┃┗━┛┃┏━┓┃
┗┛ ┗┻┛ ┗┛ ┗┛ ┗┛ ┗┻━━━┻━━━┻━━━┻┛ ┗┛
   by Yamato Security

Searching...

Start time: 2024/11/30 09:07

Total event log files: 598
Total file size: 139.2 MB

Currently searching. Please wait.

[00:00:09] 598 / 598 ⠉ [========================================] 100%

Saved results: before.csv (34.9 MB)
Elapsed time: 00:00:08.1258
Analysis Processing Time: 00:00:08.965
Output Processing Time: 00:00:00.287

Memory usage stats:
heap stats:     peak       total       freed     current        unit       count
  reserved:     2.0 GiB     2.0 GiB     0           2.0 GiB
 committed:     2.0 GiB     2.0 GiB    10.7 GiB    -8.7 GiB                          ok
     reset:     0
    purged:   872.8 MiB
   touched:   128.5 KiB     9.5 MiB     8.6 GiB    -8.6 GiB                          ok
  segments:     5         153         150           3                                not all freed
-abandoned:     1           1           1           0                                ok
   -cached:     0           0           0           0                                ok
     pages:     0           0          36.9 Ki    -36.9 Ki                           ok
-abandoned:     2           2           2           0                                ok
 -extended:     0
 -noretire:     0
    arenas:     2
-crossover:     0
 -rollback:     0
     mmaps:     0
   commits:     0
    resets:     0
    purges:   260
   threads:    33          33           1          32                                not all freed
  searches:     0.0 avg
numa nodes:     1
   elapsed:     9.295 s
   process: user: 10.998 s, system: 1.010 s, faults: 921, rss: 186.4 MiB, commit: 2.0 GiB

After:

./target/release/hayabusa search -d ../hayabusa-sample-evtx -r ".*" -o after.csv --debug -C

┏┓ ┏┳━━━┳┓  ┏┳━━━┳━━┓┏┓ ┏┳━━━┳━━━┓
┃┃ ┃┃┏━┓┃┗┓┏┛┃┏━┓┃┏┓┃┃┃ ┃┃┏━┓┃┏━┓┃
┃┗━┛┃┃ ┃┣┓┗┛┏┫┃ ┃┃┗┛┗┫┃ ┃┃┗━━┫┃ ┃┃
┃┏━┓┃┗━┛┃┗┓┏┛┃┗━┛┃┏━┓┃┃ ┃┣━━┓┃┗━┛┃
┃┃ ┃┃┏━┓┃ ┃┃ ┃┏━┓┃┗━┛┃┗━┛┃┗━┛┃┏━┓┃
┗┛ ┗┻┛ ┗┛ ┗┛ ┗┛ ┗┻━━━┻━━━┻━━━┻┛ ┗┛
   by Yamato Security

Searching...

Taking threat detection to the next level~

Start time: 2024/11/30 09:07
Total event log files: 598
Total file size: 139.2 MB

Currently searching. Please wait.

[00:00:09] 598 / 598   [========================================] 100%

Scanning finished.
                                                                                                                                                                                                 Total findings: 47,623
Saved results: after.csv (34.9 MB)

Elapsed time: 00:00:09.852

不動心 - Fudoushin - Immovable mind.

Analysis Processing Time: 00:00:09.551
Output Processing Time: 00:00:00.299

Memory usage stats:
heap stats:     peak       total       freed     current        unit       count
  reserved:     2.0 GiB     2.0 GiB     0           2.0 GiB
 committed:     2.0 GiB     2.0 GiB    12.1 GiB   -10.1 GiB                          ok
     reset:     0
    purged:   955.7 MiB
   touched:   128.5 KiB     9.5 MiB     8.6 GiB    -8.6 GiB                          ok
  segments:     5         154         151           3                                not all freed
-abandoned:     1           1           1           0                                ok
   -cached:     0           0           0           0                                ok
     pages:     0           0          36.0 Ki    -36.0 Ki                           ok
-abandoned:     4           4           4           0                                ok
 -extended:     0
 -noretire:     0
    arenas:     2
-crossover:     0
 -rollback:     0
     mmaps:     0
   commits:     0
    resets:     0
    purges:   300
   threads:    33          33           1          32                                not all freed
  searches:     0.0 avg
numa nodes:     1
   elapsed:     9.864 s
   process: user: 11.552 s, system: 1.192 s, faults: 0, rss: 186.2 MiB, commit: 2.0 GiB

Unfortunately, memory usage only goes down .2 MiB and the results get corrupted in a weird way... 😦

Before:
"2016-09-20 22:07:36.609 +09:00","-","IE10Win7","Sys",26,8286,"ComplexData: 01 ¦ Group: 0 ¦ IdleStateCount: 1 ¦ Name: PerfState ¦ Number: 0 ¦ PerfStateCount: 0 ¦ ThrottleStateCount: 8","../hayabusa-sample-evtx/DeepBlueCLI/many-events-system.evtx"

After:
"2016-09-20 22:07:36.609 +09:00","-","IE10Win7","Sys",26,8286,"ComplexData: ¦ Group: 0 ¦ IdleStateCount: 1 ¦ Name: IdleState ¦ Name: PerfState ¦ Number: 0 ¦ PerfStateCount: 0 ¦ ThrottleStateCount: 8","../hayabusa-sample-evtx/DeepBlueCLI/many-events-system.evtx"

@fukusuket Do you know why the results differ?

@YamatoSecurity YamatoSecurity self-assigned this Nov 30, 2024
@YamatoSecurity YamatoSecurity added the enhancement New feature or request label Nov 30, 2024
Copy link
Collaborator

@fukusuket fukusuket left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@YamatoSecurity
The cause of the difference had nothing to do with this PR, so I created another issue! LGTM!!🚀

@YamatoSecurity
Copy link
Collaborator Author

@fukusuket Oh great, thanks for finding the bug and creating the other issue! I will merge this then.

@YamatoSecurity YamatoSecurity merged commit 9c3927d into main Nov 30, 2024
5 checks passed
@YamatoSecurity YamatoSecurity deleted the 787-reduce-clone-search branch November 30, 2024 02:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants