[BUG] ScanYara - Traceback Error

[derfel1989]

Describe the bug
A clear and concise description of what the bug is.

Environment details

• Operating System: Ubuntu
• Architecture: Docker (Standalone + Portainer)

Steps to reproduce
Steps to reproduce the behavior:

1. Submit any file
2. Click on Scan Yara within the result
3. Error

Traceback (most recent call last): File "/usr/local/lib/python3.10/dist-packages/strelka-0.0.0-py3.10.egg/strelka/strelka.py", line 779, in scan_wrapper self.scan(data, file, options, expire_at) File "/usr/local/lib/python3.10/dist-packages/strelka-0.0.0-py3.10.egg/strelka/scanners/scan_yara.py", line 58, in scan self.load_yara_rules(options) File "/usr/local/lib/python3.10/dist-packages/strelka-0.0.0-py3.10.egg/strelka/scanners/scan_yara.py", line 162, in load_yara_rules self.rules_loaded = len(list(self.compiled_yara)) TypeError: 'NoneType' object is not iterable

compiling_error_general_/etc/strelka/yara/rules/malware/MALW_Torte_ELF.yar(31): undefined identifier "is__elf"

Expected behavior
See no error on ScanYara session as a fresh install and a correct Yara rule that matches the file, if applicable.

Screenshots

Release

• Release: [latest - installed on 2023-11-09]

Additional context
Backend Docker Logs: _strelka-backend-1_logs.txt

[phutelmyer]

Thank you @derfel1989. I'm able to reproduce this and I'll get this fixed up today. I'll @ you when its fixed.

[phutelmyer]

Seems its an issue with the YARA Rule itself:

E       +  'flags': ['compiling_error_general_/strelka/src/python/strelka/tests/fixtures/test_elk_linux_torte.yara(31): '
E       +            'undefined identifier "is__elf"',
E       +            'timed_out']}

[phutelmyer]

Should be fixed in the latest commit. I'll push a release later today as well.

As noted in PR #411, Strelka supports pre-compiled rulesets, so we recommend using those to prevent rule compilation failures.

[derfel1989]

Hi @phutelmyer, thank you for your support.

The issue concerning the YARA rule has been resolved. However, the problem below persists for all submitted files. Am I missing something?

Traceback (most recent call last): File "/usr/local/lib/python3.10/dist-packages/strelka-0.0.0-py3.10.egg/strelka/strelka.py", line 779, in scan_wrapper self.scan(data, file, options, expire_at) File "/usr/local/lib/python3.10/dist-packages/strelka-0.0.0-py3.10.egg/strelka/scanners/scan_yara.py", line 58, in scan self.load_yara_rules(options) File "/usr/local/lib/python3.10/dist-packages/strelka-0.0.0-py3.10.egg/strelka/scanners/scan_yara.py", line 130, in load_yara_rules if compiled.get("enabled", False): AttributeError: 'NoneType' object has no attribute 'get'

[phutelmyer]

@derfel1989 Reopening...

I rebuilt Strelka and am not getting this issue with the defaults.

Can you provide more details on the YARA rules you're importing or any modifications / deviations from the original repo?

Does you backend.yaml look like this for ScanYara?

  'ScanYara':
    - positive:
        flavors:
          - '*'
      priority: 5
      options:
        location: '/etc/strelka/yara/'
        compiled:
          enabled: False
          filename: "rules.compiled"
        store_offset: True
        offset_meta_key: "StrelkaHexDump"
        offset_padding: 32

[derfel1989]

My backend.yaml at 'strelka/configs/python/backend/backend.yaml' is:

'ScanYara':
    - positive:
flavors:
 - '*'
priority: 5
options:
location: '/etc/strelka/yara/rules.yara'
store_offset: True
offset_meta_key: "StrelkaHexDump"
offset_padding: 32

At 'strelka/src/python/strelka/config/backend.yaml' is:

'ScanYara':
    - positive:
 flavors:
 - '*'
 priority: 5
 options:
 location: '/etc/strelka/yara/rules.yara'
 store_offset: True
 offset_meta_key: "StrelkaHexDump"
 offset_padding: 32

[phutelmyer]

The backend config has been updated about a month or two ago with compiled support, please take a look at the latest config. I recommend modifying your current config file to match the latest.

In the meantime, a PR has been implemented to fix the unhandled exception you've ran into. Sorry about that.

[derfel1989]

That's odd. I'm using a fresh docker pre-build setup. It should come updated, isn't it?

I just removed 2 Yara rules, which were present in ScanYara issue flags:

• deprecated/Android/
  • Android_VirusPolicia.yar
  • Android_VikingOrder.yar

I decided to delete the whole folder, as it related to deprecated rules.

Same warning about this file.

• malware/MALW_Torte_ELF.yar

Update: Result for all scans:

I will update the backend.yml.

Thank you for your support. 🙂

[derfel1989]

@phutelmyer, I've reviewed all the backend.yml files. They look like yours.

strelka/configs/python/backend/backend.yaml
strelka/src/python/strelka/config/backend.yaml
/var/lib/docker/overlay2/[redacted]/diff/etc/strelka/backend.yaml
/var/lib/docker/overlay2/[redacted]/diff/strelka/strelka/config/backend.yaml

'ScanYara':
- positive:
   flavors:
 - '*'
 priority: 5
options:
 location: '/etc/strelka/yara/'
   compiled:
          enabled: False
   filename: "rules.compiled"
 store_offset: True
  offset_meta_key: "StrelkaHexDump"
 offset_padding: 32

[phutelmyer]

@derfel1989

Based on the error showing in this image:

The expectation is that perhaps the YARA folder is not being handled properly.
I tried to mimic how I think you have yours set up, but I'm curious where you're storing your YARA rules prior to bootup?

The above seems to scan fine, although it fails because the specific YARA is not supported. As Strelka compiles YARA rules prior to scanning, I'm unable to verify all of the YARA rules without significant cost.

That doesn't seem like it's the issue though as you have a different error (a file read error?).
If you'd like, I can provide guidance on how you can compile your YARA rules prior to launch.
That should resolve any issues you have.

However, I would like to try to fix that compile error 🤔

[derfel1989]

Many thanks for your support.

Well, my strelka is a fresh install.

I'm curious where you're storing your YARA rules prior to bootup?

Everything is under "/opt/strelka" and I didn't change anything in the configurations.

[derfel1989]

@phutelmyer , please let me know if you need anything from me to replicate the issue I'm facing here.

Thank you.

[derfel1989]

@phutelmyer - Could you guide me on how to turn off the Yara evaluation before compilation?

I changed the Python code below, but it didn't work.

File: scan_yara.py

From:
try: # Load compiled YARA rules from a file. if compiled.get("enabled", False):

To:
try: # Load compiled YARA rules from a file. if compiled.get("enabled", True):

Issue

'yara': {
'flags': ['compiling_error_general_/etc/strelka/yara/rules/malware/TOOLKIT_Mandibule.yar(65): undefined identifier "is__elf"', 'no_rules_loaded'],
'elapsed': 3.804577,
'rules_loaded': 0

Thank you.

[phutelmyer]

@derfel1989
I'm putting together a PR that warns users in their terminal if a YARA failure exists (so you do not need to look at YARA flags).

As a solution: I recommend precompiling your YARA rules and providing the compiled file to Strelka to avoid the issues you're seeing.

Here's how you can do it:

Python Script to Remove Non-Compilable YARA Rules

If you encounter rules that cannot be compiled, you can use the following Python script to iterate through your YARA rules folder, attempt to compile each rule, and remove any files that cause a syntax error:

import os
import yara
import glob

def compile_yara_rules(rules_dir):
    compiled_rules_path = os.path.join(rules_dir, "rules.compiled")
    rule_files = glob.glob(os.path.join(rules_dir, '**/*.yar'), recursive=True)
    
    for rule_file in rule_files:
        try:
            yara.compile(filepath=rule_file)
            print(f"Successfully compiled: {rule_file}")
        except yara.SyntaxError as e:
            print(f"Syntax error in {rule_file}: {e}")
            os.remove(rule_file)
            print(f"Removed: {rule_file}")

    # Compile all remaining rules into a single file
    if rule_files:
        yara.compile(filepaths={os.path.basename(f): f for f in rule_files}, outfile=compiled_rules_path)
        print(f"All remaining rules compiled into {compiled_rules_path}")

compile_yara_rules("/path/to/yara/rules")

Remember to replace /path/to/yara/rules with your actual directory path.
This script will leave you with a single .compiled file containing all the successfully compiled YARA rules.

You can then reference the path to that file in the Backend config.

This will resolve your issue at the expense of removing the bad YARA rules. Apologies for the headache here. Strelka does not do this as it is an issue with YARA and its inability to ignore any bad rules. Strelka could iterate every rule and strip out bad rules, but that would decrease performance significantly.

Let me know how else I can help, I hope I can assist you quicker in the future.

[derfel1989]

@phutelmyer Many thanks for your help, as always.

I will test it and get back with the results.

[derfel1989]

@phutelmyer Last question: where should I add the reference of the Python file to the backend.yml?
I'm asking because I didn't see any similar mention in the backend.yml file.

[phutelmyer]

Apologies for missing this @derfel1989...

The Python file above would be your own pipeline / script just ahead of Strelka.
You would run that file to check all of your YARA rules, and then the output of that script would be a directory of YARA rules that can be compiled by YARA and processed by Strelka.

The script itself does not produce a compiled .yar file that you can use. I can add to it if you'd like.

[phutelmyer]

Feel free to reopen this issue if issues persist.