Stix transmission (#62)

README.md

@@ -6,47 +6,71 @@ Requires Python 3.6
 
 ## It is a who that does what now? What is STIX Patterning? What are STIX Observations?
 
-[Structured Threat Information Expression (STIX™)](https://oasis-open.github.io/cti-documentation/) is a language and serialization format used to exchange cyber threat intelligence (CTI). STIX 2 Patterning is a part of STIX that deals with the "matching things" part of STIX, which is an integral component of STIX Indicators. 
+[Structured Threat Information Expression (STIX™)](https://oasis-open.github.io/cti-documentation/) is a language and serialization format used to exchange cyber threat intelligence (CTI). STIX 2 Patterning is a part of STIX that deals with the "matching things" part of STIX, which is an integral component of STIX Indicators.
 
 This library takes in STIX 2 Patterns as input, and "finds" data that matches the patterns inside various products that house repositories of cybersecurity data. Examples of such products include SIEM systems, endpoint management systems, threat intelligence platforms, orchestration platforms, network control points, data lakes, and more.
 
-In addition to "finding" the data using these patterns, STIX-Shifter uniquely also *transforms the output* into STIX 2 Observations. Why would we do that you ask? To put it simply - so that all of the security data, regardless of the source, mostly looks and behaves the same. As anyone with experience in data science will tell you, the cleansing and normalizing of the data accross domains, is one of the largest hurdles to overcome with attempting to build cross-platform security analytics. This is one of the barriers we are attempting to break down with STIX Shifter.
+In addition to "finding" the data using these patterns, STIX-Shifter uniquely also _transforms the output_ into STIX 2 Observations. Why would we do that you ask? To put it simply - so that all of the security data, regardless of the source, mostly looks and behaves the same. As anyone with experience in data science will tell you, the cleansing and normalizing of the data accross domains, is one of the largest hurdles to overcome with attempting to build cross-platform security analytics. This is one of the barriers we are attempting to break down with STIX Shifter.
 
 ## This sounds like Sigma, I already have that
 
-[Sigma](https://github.com/Neo23x0/sigma) and STIX Patterning have goals that are related, but at the end of the day have slightly different scopes. While Sigma seeks to be "for log files what Snort is for network traffic and YARA is for files", STIX Patterning's goal is to encompass *all three* fundamental security data source types - network, file, and log - and do so simultaneously, allowing you to create complex queries and analytics that span domains. As such, so does STIX Shifter. We feel it is critical to be able to create search patterns that span SIEM, Endpoint, Network, and File levels, in order to detect the complex patterns used in modern campaigns.
+[Sigma](https://github.com/Neo23x0/sigma) and STIX Patterning have goals that are related, but at the end of the day have slightly different scopes. While Sigma seeks to be "for log files what Snort is for network traffic and YARA is for files", STIX Patterning's goal is to encompass _all three_ fundamental security data source types - network, file, and log - and do so simultaneously, allowing you to create complex queries and analytics that span domains. As such, so does STIX Shifter. We feel it is critical to be able to create search patterns that span SIEM, Endpoint, Network, and File levels, in order to detect the complex patterns used in modern campaigns.
 
 ## Why would I want to use this?
 
 You may want to use this library and/or contribute to development, if any of the follwing are true:
 
-* You are a vendor or project owner who wants to add some form of query or enrichment functionality to your product capabilities
-* You are an end user and want to have a way to script searches and/or queries as part of your orchestrsation flow
-* You are a vendor or project owner who has data that could be made available, and you want to contribute an adapter
-* You just want to help make the world a safer place!
+- You are a vendor or project owner who wants to add some form of query or enrichment functionality to your product capabilities
+- You are an end user and want to have a way to script searches and/or queries as part of your orchestrsation flow
+- You are a vendor or project owner who has data that could be made available, and you want to contribute an adapter
+- You just want to help make the world a safer place!
 
 # How to use
 
-## Converting from STIX Patterns to data source queries
+## Converting from STIX Patterns to data source queries (query) or from data source results to STIX cyber observables (results)
 
 ### Call the stix_shifter in the format of
 
 ```
 usage: stix_shifter.py translate [-h]
-                                 {qradar,dummy}
-                                 {results,query} data
+                                 {qradar, dummy, splunk}
+                                 {results, query} data
 
 positional arguments:
-  {qradar,dummy}         What translation module to use
-  {results,query}                        What translation action to perform
-  data source                            A STIX identity object
-  data                                   The data to be translated
+{qradar, dummy}                        What translation module to use
+{results, query}                       What translation action to perform
+data source                            A STIX identity object
+data                                   STIX pattern or data to be translated
 
 optional arguments:
   -h, --help            show this help message and exit
   -x                    run STIX validation on each observable as it's written to the output JSON
 ```
 
+## Connecting to a data source
+
+### Call the stix_shifter in the format of
+
+```
+usage: stix_shifter.py transmit [-h]
+                      {async_dummy, synchronous_dummy, qradar, splunk, bigfix}
+
+positional arguments:
+{<async_dummy, synchronous_dummy, qradar, splunk, bigfix>}         Transmission module to use
+  {"host": <host IP>, "port": <port>, "cert": <certificate>} Data source connection
+  {"auth": <authentication>} Data source authentication
+  {
+    "type": <ping, query, results, is_async, status>,   Translation method to be used
+    "search_id": <uuid> (for results and status),
+    "query": <native datasource query string> (for query),
+    "offset": <offset> (for results),
+    "length": <length> (for results)
+  }
+
+optional arguments:
+  -h, --help            show this help message and exit
+```
+
 ### Example of converting a STIX pattern to an IBM QRadar AQL query:
 
 [See the QRadar module documentation](stix_shifter/src/modules/qradar/README.md)

main.py

@@ -4,58 +4,113 @@
 import json
 
 
+TRANSLATE = 'translate'
+TRANSMIT = 'transmit'
+
+
 def __main__():
     """
-    In the case of converting a stix pattern to datasource query, arguments will take the form of...
-    <module> <translate_type> <data> <options>
-    The module and translate_type will determine what module and method gets called
-    Options argument comes in as:
+    Stix-shifter can either be called to either translate or transmit.
+    In the case of translation, stix-shifter either translates a stix pattern to a datasource query, 
+    or converts data source query results into JSON of STIX observations. 
+    Arguments will take the form of...
+    "translate" <module> <translate_type (query or results)> <data (STIX pattern or query results)> <options>
+    The module and translate_type will determine what module and method gets executed.
+    Option arguments comes in as:
       "{
-          "select_fields": <string array of fields in the datasource select statement>},
-          "mapping": <mapping hash for either stix pattern to datasource or data results to stix observation objects>,
+          "select_fields": <string array of fields in the datasource select statement> (In the case of QRadar),
+          "mapping": <mapping hash for either stix pattern to datasource or mapping hash for data results to stix observation objects>,
           "result_limit": <integer limit number for max results in the data source query>,
-          "timerange": <integer time window in minutes used in the data source query when START STOP qualifiers are absent>
+          "timerange": <integer time range for LAST x MINUTES used in the data source query when START STOP qualifiers are absent>
        }"
+    In the case of transmission, stix-shifter connects to a datasource to execute queries, status updates, and result retrieval.
+    Arguments will take the form of...
+    "transmit" <module> '{"host": <host IP>, "port": <port>, "cert": <certificate>}', '{"auth": <authentication>}',
+        <
+            query <query string>,
+            status <search id>,
+            results <search id> <offset> <length>,
+            ping,
+            is_async
+        >
     """
 
     # process arguments
-    parser = argparse.ArgumentParser(description='stix_shifter')
-    subparsers = parser.add_subparsers(dest='command')
+    parent_parser = argparse.ArgumentParser(description='stix_shifter')
+    parent_subparsers = parent_parser.add_subparsers(dest='command')
 
     # translate parser
-    translate_parser = subparsers.add_parser(
-        'translate', help='Translate a query or result set using a specific translation module')
+    translate_parser = parent_subparsers.add_parser(
+        TRANSLATE, help='Translate a query or result set using a specific translation module')
+
     # positional arguments
     translate_parser.add_argument(
-        'module', choices=stix_shifter.MODULES, help='what translation module to use')
+        'module', choices=stix_shifter.TRANSLATION_MODULES, help='what translation module to use')
     translate_parser.add_argument('translate_type', choices=[
         stix_shifter.RESULTS, stix_shifter.QUERY], help='what translation action to perform')
     translate_parser.add_argument(
         'data_source', help='STIX identity object representing a datasource')
     translate_parser.add_argument(
         'data', type=str, help='the data to be translated')
     translate_parser.add_argument('options', nargs='?', help='options that can be passed in')
+
     # optional arguments
     translate_parser.add_argument('-x', '--stix-validator', action='store_true',
                                   help='run stix2 validator against the converted results')
     translate_parser.add_argument('-m', '--data-mapper',
                                   help='module to use for the data mapper')
 
-    args = parser.parse_args()
+    # transmit parser
+    transmit_parser = parent_subparsers.add_parser(
+        TRANSMIT, help='Connect to a datasource and exectue a query...')
+
+    # positional arguments
+    transmit_parser.add_argument(
+        'module', choices=stix_shifter.TRANSMISSION_MODULES,
+        help='choose which connection module to use'
+    )
+    transmit_parser.add_argument(
+        'connection',
+        type=str,
+        help='Data source connection with host, port, and certificate'
+    )
+    transmit_parser.add_argument(
+        'configuration',
+        type=str,
+        help='Data source authentication'
+    )
+
+    # operation subparser
+    operation_subparser = transmit_parser.add_subparsers(title="operation", dest="operation_command")
+    operation_subparser.add_parser(stix_shifter.PING, help="Pings the data source")
+    query_operation_parser = operation_subparser.add_parser(stix_shifter.QUERY, help="Executes a query on the data source")
+    query_operation_parser.add_argument('query_string', help='native datasource query string')
+    results_operation_parser = operation_subparser.add_parser(stix_shifter.RESULTS, help="Fetches the results of the data source query")
+    results_operation_parser.add_argument('search_id', help='uuid of executed query')
+    results_operation_parser.add_argument('offset', help='offset of results')
+    results_operation_parser.add_argument('length', help='length of results')
+    status_operation_parser = operation_subparser.add_parser(stix_shifter.STATUS, help="Gets the current status of the query")
+    status_operation_parser.add_argument('search_id', help='uuid of executed query')
+    operation_subparser.add_parser(stix_shifter.IS_ASYNC, help='Checks if the query operation is asynchronous')
+
+    args = parent_parser.parse_args()
 
     if args.command is None:
-        parser.print_help(sys.stderr)
+        parent_parser.print_help(sys.stderr)
         sys.exit(1)
 
-    options = json.loads(args.options) if bool(args.options) else {}
-    if args.stix_validator:
-        options['stix_validator'] = args.stix_validator
-    if args.data_mapper:
-        options['data_mapper'] = args.data_mapper
-
     shifter = stix_shifter.StixShifter()
-    result = shifter.translate(
-        args.module, args.translate_type, args.data_source, args.data, options=options)
+
+    if args.command == TRANSLATE:
+        options = json.loads(args.options) if bool(args.options) else {}
+        if args.stix_validator:
+            options['stix_validator'] = args.stix_validator
+        if args.data_mapper:
+            options['data_mapper'] = args.data_mapper
+        result = shifter.translate(
+            args.module, args.translate_type, args.data_source, args.data, options=options)
+    elif args.command == TRANSMIT:
+        result = shifter.transmit(args)
 
     print(result)
     exit(0)

setup.py

@@ -45,7 +45,7 @@
     # This is a one-line description or tagline of what your project does. This
     # corresponds to the "Summary" metadata field:
     # https://packaging.python.org/specifications/core-metadata/#summary
-    description='Tools and interface to translate STIX formatted results and queries to different data source formats',  # Required
+    description='Tools and interface to translate STIX formatted results and queries to different data source formats and to set up appropriate connection strings for invoking and triggering actions in openwhisk',  # Required
 
     # This is an optional longer description of your project that represents
     # the body of text which users will see when they visit PyPI.
@@ -179,6 +179,11 @@
     #         'sample=sample:main',
     #     ],
     # },
+    entry_points={
+        'console_scripts': [
+            'stix-transmission=stix_transmission.stix_transmission:main',
+        ],
+    },
 
     # List additional URLs that are relevant to your project as a dict.
     #

stix_shifter/src/modules/qradar/stix_to_aql.py

@@ -7,6 +7,9 @@
 
 logger = logging.getLogger(__name__)
 
+DEFAULT_LIMIT = 10000
+DEFAULT_TIMERANGE = 5
+
 
 class StixToAQL(BaseQueryTranslator):
 
@@ -25,8 +28,8 @@ def transform_query(self, data, options, mapping=None):
 
         query_object = generate_query(data)
         data_model_mapper = qradar_data_mapping.QRadarDataMapper(options)
-        result_limit = options['result_limit'] if 'result_limit' in options else 10000
-        timerange = options['timerange'] if 'timerange' in options else 5
+        result_limit = options['result_limit'] if 'result_limit' in options else DEFAULT_LIMIT
+        timerange = options['timerange'] if 'timerange' in options else DEFAULT_TIMERANGE
         query_string = aql_query_constructor.translate_pattern(
             query_object, data_model_mapper, result_limit, timerange)
         return query_string

stix_shifter/stix_shifter.py

@@ -1,14 +1,20 @@
-import sys
 import importlib
 from stix_shifter.src.patterns.parser import generate_query
 from stix2patterns.validator import run_validator
 from stix_shifter.src.stix_pattern_parser import stix_pattern_parser
 import re
+from stix_transmission import stix_transmission
+import json
 
 
-MODULES = ['qradar', 'dummy', 'car', 'cim', 'splunk', 'elastic']
+TRANSLATION_MODULES = ['qradar', 'dummy', 'car', 'cim', 'splunk', 'elastic']
+TRANSMISSION_MODULES = ['async_dummy', 'synchronous_dummy', 'qradar', 'splunk', 'bigfix']
 RESULTS = 'results'
 QUERY = 'query'
+DELETE = 'delete'
+STATUS = 'status'
+PING = 'ping'
+IS_ASYNC = 'is_async'
 
 
 class StixValidationException(Exception):
@@ -27,7 +33,7 @@ def translate(self, module, translate_type, data_source, data, options={}):
         """
         Translated queries to a specified format
         :param module: What module to use
-        :type module: one of MODULES 'qradar', 'dummy'
+        :type module: one of TRANSLATION_MODULES 'qradar', 'dummy'
         :param translate_type: translation of a query or result set must be either 'results' or 'query'
         :type translate_type: str
         :param data: the data to translate
@@ -38,7 +44,7 @@ def translate(self, module, translate_type, data_source, data, options={}):
         :rtype: str
         """
 
-        if module not in MODULES:
+        if module not in TRANSLATION_MODULES:
             raise NotImplementedError
 
         translator_module = importlib.import_module(
@@ -70,3 +76,44 @@ def translate(self, module, translate_type, data_source, data, options={}):
             return interface.translate_results(data_source, data, options)
         else:
             raise NotImplementedError
+
+    def transmit(self, args):
+        """
+        Connects to datasource and executes a query, grabs status update or query results
+        :param args:
+        args: <module> '{"host": <host IP>, "port": <port>, "cert": <certificate>}', '{"auth": <authentication>}',
+        <
+            query <query string>,
+            status <search id>,
+            results <search id> <offset> <length>,
+            ping,
+            is_async
+        >
+        """
+        connection_dict = json.loads(args.connection)
+        configuration_dict = json.loads(args.configuration)
+        operation_command = args.operation_command
+
+        connector = stix_transmission.StixTransmission(args.module, connection_dict, configuration_dict)
+
+        if operation_command == QUERY:
+            query = args.query_string
+            result = connector.query(query)
+        elif operation_command == STATUS:
+            search_id = args.search_id
+            result = connector.status(search_id)
+        elif operation_command == RESULTS:
+            search_id = args.search_id
+            offset = args.offset
+            length = args.length
+            result = connector.results(search_id, offset, length)
+        elif operation_command == DELETE:
+            search_id = args.search_id
+            result = connector.delete(search_id)
+        elif operation_command == PING:
+            result = connector.ping()
+        elif operation_command == IS_ASYNC:
+            result = connector.is_async()
+        else:
+            raise NotImplementedError
+        return result

stix_transmission/src/modules/async_dummy/async_dummy_connector.py

@@ -0,0 +1,17 @@
+from ..base.base_connector import BaseConnector
+from .async_dummy_ping import AsyncDummyPing
+from .async_dummy_query_connector import AsyncDummyQueryConnector
+from .async_dummy_status_connector import AsyncDummyStatusConnector
+from .async_dummy_results_connector import AsyncDummyResultsConnector
+
+
+class Connector(BaseConnector):
+    def __init__(self, connection, configuration):
+        host = connection.get('host')
+        port = connection.get('port')
+        path = connection.get('path')
+        self.query_connector = AsyncDummyQueryConnector(host, port, path)
+        self.status_connector = AsyncDummyStatusConnector(host, port, path)
+        self.results_connector = AsyncDummyResultsConnector(host, port, path)
+        self.is_async = True
+        self.ping_connector = AsyncDummyPing(host, port, path)

stix_transmission/src/modules/async_dummy/async_dummy_ping.py

@@ -0,0 +1,11 @@
+from ..base.base_ping import BasePing
+
+
+class AsyncDummyPing(BasePing):
+    def __init__(self, host, port, path):
+        self.host = host
+        self.port = port
+        self.path = path
+
+    def ping(self):
+        return 'async ping'