Added RST IoC Lookup connector. Fixes for Report Hub and Threat Feed

external-import/rst-report-hub/Dockerfile

@@ -1,4 +1,4 @@
-FROM python:3.12-alpine
+FROM python:3.11-alpine
 ENV CONNECTOR_TYPE=EXTERNAL_IMPORT
 
 # Copy the connector

external-import/rst-report-hub/README.md

@@ -5,7 +5,7 @@ The **RST Report Hub Connector** integrates various APT reports from security co
 ## Key Features
 
 - **Brilliant Time Saver**: Manual import of threat reports is a time consuming activity that does not need to happen anymore.
-- **Threat Report Library**: Keep all APT reports and their metadata, extracted objects in one place.
+- **Threat Report Library**: Keep all APT reports and their metadata, extracted objects in one place. 
 - **OpenCTI Integration**: Seamlessly integrates the fetched data into OpenCTI's database.
 
 This connector provides users with an enhanced and comprehensive understanding of the cybersecurity threat landscape by leveraging the detailed threat intelligence provided by RST Cloud.
@@ -22,7 +22,7 @@ This connector is aligned with data populated by common OpenCTI connectors. We r
  - CISA Known Exploited Vulnerabilities (https://github.com/OpenCTI-Platform/connectors/tree/master/external-import/cisa-known-exploited-vulnerabilities)
 
 
-## Configuration:
+## Configuration
 
 Configuration of the connector is straightforward. The minimal configuration requires you just enter the RST Cloud API key to be provided and OpenCTI connection settings specified. Below is the full list of parameters you can set:
 
@@ -33,17 +33,15 @@ Configuration of the connector is straightforward. The minimal configuration req
 | Connector ID | `CONNECTOR_ID` | Yes | A unique `UUIDv4` identifier for this connector instance. |
 | Connector Name | `CONNECTOR_NAME` | Yes | Name of the connector. For example: `RST Report Hub`. |
 | Connector Scope | `CONNECTOR_SCOPE` | Yes | The scope or type of data the connector is importing, either a MIME type or Stix Object. E.g. application/json |
-| Confidence Level | `CONNECTOR_CONFIDENCE_LEVEL` | Yes | The default confidence level for created sightings. It's a number between 1 and 100, with 100 being the most confident. |
 | Log Level | `CONNECTOR_LOG_LEVEL` | Yes | Determines the verbosity of the logs. Options are `debug`, `info`, `warn`, or `error`. |
-| Run and Terminate | `CONNECTOR_RUN_AND_TERMINATE` | Yes | If set to true, the connector will terminate after a successful run. Useful for debugging or one-time runs. |
-| Update Existing Data | `CONFIG_UPDATE_EXISTING_DATA` | Yes | Decide whether the connector should update already existing data in the database. |
-| Interval | `CONFIG_INTERVAL` | Yes | Determines how often the connector will run, set in hours. |
-| RST Report Hub API Key | `RST_REPORT_HUB_API_KEY` | Yes | Your API Key for accessing RST Cloud. |
 | RST Report Hub Base URL | `RST_REPORT_HUB_BASE_URL` | No | By default, use https://api.rstcloud.net/v1/. In some cases, you may want to use a local API endpoint |
+| RST Report Hub API Key | `RST_REPORT_HUB_API_KEY` | Yes | Your API Key for accessing RST Cloud. |
 | RST Report Hub Connection Timeout | `RST_REPORT_HUB_CONNECTION_TIMEOUT` | No | Connection timeout to the API. Default (sec): `30` |
 | RST Report Hub Read Timeout | `RST_REPORT_HUB_READ_TIMEOUT` | No | Read timeout for each feed. If the connector is unable to fetch a report in time, increase the read timeout. Default (sec): `60` |
-| RST Report Hub Read Timeout | `RST_REPORT_HUB_RETRY_DELAY` | No | Specifies how long to wait in seconds before next attempt to connect to the API. Default (sec): `30` |
+| RST Report Hub Retry Delay | `RST_REPORT_HUB_RETRY_DELAY` | No | Specifies how long to wait in seconds before next attempt to connect to the API. Default (sec): `30` |
 | RST Report Hub Download Retry Count | `RST_REPORT_HUB_RETRY_ATTEMPTS` | No | Default (attempts): `5` |
 | RST Report Hub Fetch Interval | `RST_REPORT_HUB_FETCH_INTERVAL` | No | Default (sec): `300` |
-| RST Report Hub Minimal Score to Import | `RST_REPORT_HUB_IMPORT_START_DAY` | No | Specify the date from which you want to retrieve the reports. Data import for each day will occur with a delay equal to the RST_REPORT_HUB_FETCH_INTERVAL. By default, this start date is calculated as 7 days ago. |
-| RST Report Hub Minimal Score for IP to be marked for Detection | `RST_REPORT_HUB_LANGUAGE` | No | Reach out to [email protected] if you want to update thids parameter. Default: `eng` |
+| RST Report Hub Date to start pulling data from | `RST_REPORT_HUB_IMPORT_START_DATE` | No | Specify the date from which you want to retrieve the reports in the format "%Y%m%d" (for example, 20240527). Data import for each day will occur with a delay equal to the RST_REPORT_HUB_FETCH_INTERVAL. By default, this start date is calculated as 7 days ago. |
+| RST Report Hub Language | `RST_REPORT_HUB_LANGUAGE` | No | Reach out to [email protected] if you want to update this parameter. Default: `eng` |
+| RST Report Hub Connector is to create observables | `RST_REPORT_HUB_CREATE_OBSERVABLES` | No | A user can select if observables are to be created in addition to indicators. Options are `true`, `false`. Default: `false` |
+| RST Report Hub Connector is to create related-to relationships | `RST_REPORT_HUB_CREATE_RELATED_TO` | No | A user can select if `related-to` relationships are to be created or not. Options are `true`, `false`. Default: `true` |

external-import/rst-report-hub/docker-compose.yml

@@ -18,6 +18,8 @@ services:
       - RST_REPORT_HUB_IMPORT_START_DATE=${RST_REPORT_HUB_STARTDATE}
       - RST_REPORT_HUB_FETCH_INTERVAL=300
       - RST_REPORT_HUB_LANGUAGE=eng
+      - RST_REPORT_HUB_CREATE_OBSERVABLES=false
+      - RST_REPORT_HUB_CREATE_RELATED_TO=true
     restart: always
     depends_on:
       - opencti

external-import/rst-report-hub/src/config.yml.sample

@@ -4,11 +4,8 @@ opencti:
 
 connector:
   id: 'ChangeMe' # Valid UUIDv4
-  type: 'EXTERNAL_IMPORT'
   name: 'RST Report Hub'
   scope: 'application/json' # MIME type or SCO
-  confidence_level: 80 # From 0 (Unknown) to 100 (Fully trusted)
-  update_existing_data: true
   run_and_terminate: true
   log_level: 'info'
 
@@ -19,6 +16,8 @@ rst-report-hub:
   read_timeout: 30
   retry_delay: 30
   retry_attempts: 5
-  import_start_date: '20230904'
+  import_start_date: '20240101'
   fetch_interval: 300
-  language: 'eng'
+  language: 'eng'
+  create_observables: false
+  create_related_to: true

external-import/rst-report-hub/src/main.py

@@ -1,6 +1,7 @@
 import base64
 import json
 import os
+import re
 import sys
 import time
 import traceback
@@ -17,7 +18,7 @@ class ReportHub:
     def __init__(self):
         config_file_path = os.path.dirname(os.path.abspath(__file__)) + "/config.yml"
         config = (
-            yaml.safe_load(open(config_file_path))
+            yaml.safe_load(open(config_file_path, encoding="UTF-8"))
             if os.path.isfile(config_file_path)
             else {}
         )
@@ -43,33 +44,137 @@ def __init__(self):
             ),
             "fetch_interval": int(self.get_config("fetch_interval", config, 300)),
             "language": str(self.get_config("language", config, "eng")),
+            "create_observables": bool(
+                self.get_config("create_observables", config, False)
+            ),
+            "create_related_to": bool(
+                self.get_config("create_related_to", config, True)
+            ),
         }
         self.update_existing_data = get_config_variable(
             "CONNECTOR_UPDATE_EXISTING_DATA",
             ["connector", "update_existing_data"],
             config,
+            True,
         )
 
     @staticmethod
     def get_config(name: str, config, default=None):
-        env_name = "RST_REPORT_HUB_{}".format(name.upper())
-        # usually this connector gets its config from variables
-        # but if these are not defined, then it
-        # reads 'rst-report-hub' property in the file config.yml
+        env_name = f"RST_REPORT_HUB_{name.upper()}"
         result = get_config_variable(env_name, ["rst-report-hub", name], config)
-        return result or default
+        if result is not None:
+            return result
+        else:
+            return default
+
+    def extract_file_hashes(self, pattern: str):
+        hashes = {}
+        # Regular expression to match hash types and values
+        hash_pattern = re.compile(
+            r"file:hashes\.'?(MD5|SHA-1|SHA-256)'? ?= ?'([a-fA-F0-9]{32,64})'"
+        )
+        # Find all hash occurrences in the pattern
+        matches = hash_pattern.findall(pattern)
+        for hash_type, hash_value in matches:
+            hashes[hash_type] = hash_value
+        return hashes
+
+    def create_observable(self, stix_indicator):
+        ioc_type = stix_indicator.get("x_opencti_main_observable_type", "")
+        shared = {
+            "object_marking_refs": stix_indicator.get("object_marking_refs", []),
+            "custom_properties": {
+                "x_opencti_score": stix_indicator.get("x_opencti_score", []),
+                "x_opencti_labels": stix_indicator.get("labels", []),
+                "x_opencti_created_by_ref": stix_indicator.get("created_by_ref", ""),
+                "x_opencti_external_references": stix_indicator.get(
+                    "external_references", []
+                ),
+            },
+        }
+        if ioc_type == "IPv4-Addr":
+            stix_observ = stix2.v21.IPv4Address(
+                value=stix_indicator["pattern"].split("'")[1], **shared
+            )
+        elif ioc_type == "Domain-Name":
+            stix_observ = stix2.v21.DomainName(
+                value=stix_indicator["pattern"].split("'")[1], **shared
+            )
+        elif ioc_type == "Url":
+            stix_observ = stix2.v21.URL(
+                value=stix_indicator["pattern"].split("'")[1], **shared
+            )
+        elif ioc_type == "StixFile":
+            stix_observ = stix2.v21.File(
+                hashes=self.extract_file_hashes(stix_indicator["pattern"]), **shared
+            )
+        else:
+            stix_observ = None
+        if stix_observ:
+            based_on = stix2.v21.Relationship(
+                source_ref=stix_indicator["id"],
+                relationship_type="based-on",
+                target_ref=stix_observ["id"],
+                **shared,
+            )
+        else:
+            based_on = None
+        return stix_observ, based_on
 
     def _combine_report_and_send(self, stix_bundle, x_opencti_file, report_id):
         # Parse the STIX bundle
         parsed_bundle = json.loads(stix_bundle)
         stix_bundle_main = []
+        message = f"Importing {report_id}"
+        observ_ids = []
+        observ_rel_ids = []
+        rel_to_ids = []
         for entry in parsed_bundle.get("objects", []):
+            # create an observable for every indicator
+            # if user selects that option
+            if (
+                entry.get("type", "") == "indicator"
+                and self._downloader_config["create_observables"]
+            ):
+                observ_obj, based_on = self.create_observable(entry)
+                if observ_obj and based_on:
+                    stix_bundle_main.append(observ_obj)
+                    observ_ids.append(observ_obj.id)
+                    stix_bundle_main.append(based_on)
+                    observ_rel_ids.append(based_on.id)
+            # remove related-to relationships from the bundle
+            # if user selects that option
+            elif (
+                entry.get("type", "") == "relationship"
+                and entry.get("relationship_type", "") == "related-to"
+                and not self._downloader_config["create_related_to"]
+            ):
+                rel_to_ids.append(entry["id"])
+                continue
+            # attach a pdf
+            elif entry.get("type", "") == "report":
+                if x_opencti_file:
+                    entry["x_opencti_files"] = [x_opencti_file]
+                else:
+                    message = f"{message}. No PDF found."
             stix_bundle_main.append(entry)
-            # attach PDFs only to the Report object
-            if x_opencti_file and entry.get("type", "") == "report":
-                entry["x_opencti_files"] = [x_opencti_file]
+        # add observables and based_on rels to the report
+        if (
+            self._downloader_config["create_observables"]
+            or not self._downloader_config["create_related_to"]
+        ):
+            new_object_refs = []
+            for entry in stix_bundle_main:
+                if entry.get("type", "") == "report":
+                    for obj_id in observ_ids:
+                        new_object_refs.append(obj_id)
+                    for obj_id in observ_rel_ids:
+                        new_object_refs.append(obj_id)
+                    for obj_id in entry["object_refs"]:
+                        if obj_id not in rel_to_ids:
+                            new_object_refs.append(obj_id)
+                    entry["object_refs"] = new_object_refs
 
-        message = "Importing " + report_id.replace("_", " ")
         work_id = self.helper.api.work.initiate_work(self.helper.connect_id, message)
         self._send_stix_data(work_id, stix_bundle_main)
         message = f"Processed {len(stix_bundle_main)} objects from RST Report Hub for {report_id}"
@@ -126,7 +231,7 @@ def _convert_and_attach_pdfs(self, headers, reports, lang):
                     self.helper.log_error(
                         f"Failed to download and save entry {report_id} as PDF. {ex}"
                     )
-                    self._combine_report_and_send(stix_report, {}, "")
+                    self._combine_report_and_send(stix_report, {}, report_id)
         return True
 
     def _fetch_stix_reports(self, current_state):
@@ -169,7 +274,7 @@ def _fetch_stix_reports(self, current_state):
                         "report_count"
                     ] >= len(reports):
                         self.helper.log_info(
-                            f"Skipping as all reports for the current day {today} have been downloaded"
+                            f"Skipping as all reports for the current day {today} are downloaded"
                         )
                         return True
                     else:

external-import/rst-threat-feed/Dockerfile

@@ -1,4 +1,4 @@
-FROM python:3.12-alpine
+FROM python:3.11-alpine
 ENV CONNECTOR_TYPE=EXTERNAL_IMPORT
 
 # Copy the connector

external-import/rst-threat-feed/README.md

@@ -36,7 +36,6 @@ Configuration of the connector is straightforward. The minimal configuration req
 | Connector Scope | `CONNECTOR_SCOPE` | Yes | The scope or type of data the connector is importing, either a MIME type or Stix Object. E.g. application/json |
 | Log Level | `CONNECTOR_LOG_LEVEL` | Yes | Determines the verbosity of the logs. Options are `debug`, `info`, `warn`, or `error`. |
 | Run and Terminate | `CONNECTOR_RUN_AND_TERMINATE` | Yes | If set to true, the connector will terminate after a successful run. Useful for debugging or one-time runs. |
-| Update Existing Data | `CONFIG_UPDATE_EXISTING_DATA` | Yes | Decide whether the connector should update already existing data in the database. |
 | Interval | `CONFIG_INTERVAL` | Yes | Determines how often the connector will run, set in hours. |
 | RST Threat Feed API Key | `RST_THREAT_FEED_API_KEY` | Yes | Your API Key for accessing RST Cloud. |
 | RST Threat Feed Base URL | `RST_THREAT_FEED_BASEURL` | No | By default, use https://api.rstcloud.net/v1/. In some cases, you may want to use a local API endpoint |
@@ -45,10 +44,8 @@ Configuration of the connector is straightforward. The minimal configuration req
 | RST Threat Feed Download Retry Count | `RST_THREAT_FEED_RETRY` | No | Default (attempts): `5` |
 | RST Threat Feed Fetch Interval | `RST_THREAT_FEED_INTERVAL` | No | Default (sec): `86400` |
 | RST Threat Feed Minimal Score to Import | `RST_THREAT_FEED_MIN_SCORE_IMPORT` | No | Import only indicators with risk score more than X. The objects that are related to these indicators will also be imported with corresponding relations. Default (score): `20` |
-| RST Threat Feed Minimal Score for IP to be marked for Detection | `RST_THREAT_FEED_MIN_SCORE_DETECTION_IP` | No | Indicators with risk score more than X are marked with x_opencti_detection=true. Default (score): `50` |
+| RST Threat Feed Minimal Score for IP to be marked for Detection | `RST_THREAT_FEED_MIN_SCORE_DETECTION_IP` | No | Indicators with risk score more than X are marked with x_opencti_detection=true. Default (score): `45` |
 | RST Threat Feed Minimal Score for Domain to be marked for Detection | `RST_THREAT_FEED_MIN_SCORE_DETECTION_DOMAIN` | No | Indicators with risk score more than X are marked with x_opencti_detection=true. Default (score): `45` |
-| RST Threat Feed Minimal Score for URL to be marked for Detection | `RST_THREAT_FEED_MIN_SCORE_DETECTION_URL` | No | Indicators with risk score more than X are marked with x_opencti_detection=true. Default (score): `30` |
-| RST Threat Feed Minimal Score for Hash to be marked for Detection | `RST_THREAT_FEED_MIN_SCORE_DETECTION_HASH` | No | Indicators with risk score more than X are marked with x_opencti_detection=true. Default (score): `25` |
-| RST Threat Feed Import only New Indicators | `RST_THREAT_FEED_ONLY_NEW` | No | Defines if you only want to import indicators with recent "First Seen" or also want to re-import changes to the indicators with "First Seen" < yesterday. If set to `False`, there will be a big queue as we provide a lot of information. It is recommended to import with `False` once to get more data and observe performance. Then switch to `True` if you system is not ready to process all data we provide. Default: `True` |
-| RST Threat Feed Temp Dir Path inside the container | `RST_THREAT_FEED_DIRS_TMP` | No | Maybe used for troubleshooting. Default: `/tmp` |
-| RST Threat Feed State Dir Path inside the container | `RST_THREAT_FEED_DIRS_STATE` | No | Maybe used for troubleshooting. Default: `/tmp` |
+| RST Threat Feed Minimal Score for URL to be marked for Detection | `RST_THREAT_FEED_MIN_SCORE_DETECTION_URL` | No | Indicators with risk score more than X are marked with x_opencti_detection=true. Default (score): `45` |
+| RST Threat Feed Minimal Score for Hash to be marked for Detection | `RST_THREAT_FEED_MIN_SCORE_DETECTION_HASH` | No | Indicators with risk score more than X are marked with x_opencti_detection=true. Default (score): `45` |
+| RST Threat Feed Import only New Indicators | `RST_THREAT_FEED_ONLY_NEW` | No | Defines if you only want to import indicators with recent "First Seen" or also want to re-import changes to the indicators with "Last Seen" >= yesterday. Default: `true` |

external-import/rst-threat-feed/docker-compose.yml

@@ -16,13 +16,11 @@ services:
       - RST_THREAT_FEED_RETRY=5
       - RST_THREAT_FEED_INTERVAL=86400
       - RST_THREAT_FEED_MIN_SCORE_IMPORT=20
-      - RST_THREAT_FEED_MIN_SCORE_DETECTION_IP=50
+      - RST_THREAT_FEED_MIN_SCORE_DETECTION_IP=45
       - RST_THREAT_FEED_MIN_SCORE_DETECTION_DOMAIN=45
-      - RST_THREAT_FEED_MIN_SCORE_DETECTION_URL=30
-      - RST_THREAT_FEED_MIN_SCORE_DETECTION_HASH=25
-      - RST_THREAT_FEED_ONLY_NEW=True
-      - RST_THREAT_FEED_DIRS_TMP=/tmp
-      - RST_THREAT_FEED_DIRS_STATE=/tmp
+      - RST_THREAT_FEED_MIN_SCORE_DETECTION_URL=45
+      - RST_THREAT_FEED_MIN_SCORE_DETECTION_HASH=45
+      - RST_THREAT_FEED_ONLY_NEW=true
     restart: always
     depends_on:
       - opencti

external-import/rst-threat-feed/src/config.yml.sample

@@ -7,7 +7,6 @@ connector:
   type: "EXTERNAL_IMPORT"
   name: "RST Threat Feed"
   scope: "application/json" # MIME type or SCO
-  update_existing_data: true
   log_level: "info"
 
 rst-threat-feed:
@@ -17,13 +16,13 @@ rst-threat-feed:
   readtimeout: 30
   retry: 5
   interval: 300
-  min_score_import: 45
-  # sets x_opencti_detection for each indicator type to True or False if more than the specified threshold
-  min_score_detection_ip: 50
+  min_score_import: 20
+  # sets x_opencti_detection for each indicator type to true or false if more than the specified threshold
+  min_score_detection_ip: 45
   min_score_detection_domain: 45
-  min_score_detection_url: 30
-  min_score_detection_hash: 25
+  min_score_detection_url: 45
+  min_score_detection_hash: 45
   only_new: true
-  only_attributed: true
+  only_attributed: false
   dirs_tmp: "./"
   dirs_state: "./"