Merge pull request #1206 from opencybersecurityalliance/develop

Merge develop into master.

.github/workflows/main.yml

@@ -11,7 +11,7 @@ jobs:
     runs-on: ubuntu-latest
     strategy:
       matrix:
-        python-version: [3.6, 3.7, 3.8, 3.9]
+        python-version: [3.8, 3.9]
 
     steps:
     - uses: actions/checkout@v2

.gitignore

@@ -62,6 +62,8 @@ coverage.xml
 venv/
 ENV/
 virtualenv*/
+labenv/
+labenv*/
 
 # mkdocs documentation
 /site

CHANGELOG.md

@@ -16,6 +16,110 @@ We have started this changelogs from version 4.0.0. So, changes on previously re
 
 -------------------------------------
 
+## 4.5.2 (2022-11-21)
+
+### Breaking changes:
+
+### Deprecations:
+
+### Changes:
+
+* AWS Athena, added external id support [#1187](https://github.com/opencybersecurityalliance/stix-shifter/pull/1187)
+* Update aws athena supported attribute [#1184](https://github.com/opencybersecurityalliance/stix-shifter/pull/1184)
+* Update AWS Athena for OCSF schema support [#1178](https://github.com/opencybersecurityalliance/stix-shifter/pull/1178)
+* Upgrade pytests version for dev environment [#1170](https://github.com/opencybersecurityalliance/stix-shifter/pull/1170)
+* ocsf schema support in aws Athena [#1134](https://github.com/opencybersecurityalliance/stix-shifter/pull/1134)
+* Add RHACS and Google Chronicle group params [#1150](https://github.com/opencybersecurityalliance/stix-shifter/pull/1150)
+* return proxy translation error [#1130](https://github.com/opencybersecurityalliance/stix-shifter/pull/1130)
+* Updated the readme mappings for GCP Chronicle [#1146](https://github.com/opencybersecurityalliance/stix-shifter/pull/1146)
+
+### Fixes:
+
+* Updated to support query without milliseconds in darktrace connector [#1199](https://github.com/opencybersecurityalliance/stix-shifter/pull/1199)
+* fix formatting of commit list generated by changelog script [#1200](https://github.com/opencybersecurityalliance/stix-shifter/pull/1200)
+* fixed timestamp issue for start and end filter and mapping correction [#1142](https://github.com/opencybersecurityalliance/stix-shifter/pull/1142)
+* Fixed pagination and meta files delete for aws athena [#1176](https://github.com/opencybersecurityalliance/stix-shifter/pull/1176)
+* gcp chronicle: removed an invalid unittest [#1166](https://github.com/opencybersecurityalliance/stix-shifter/pull/1166)
+* Remove optional word from indices label [#1157](https://github.com/opencybersecurityalliance/stix-shifter/pull/1157)
+* Fixed deployment script with --platform linux/amd64 [#1154](https://github.com/opencybersecurityalliance/stix-shifter/pull/1154)
+* Updated connector.py file for the bug fix #1103 [#1104](https://github.com/opencybersecurityalliance/stix-shifter/pull/1104)
+
+### Dependency update:
+
+* Bump flask from 2.0.3 to 2.2.2 in /stix_shifter [#1072](https://github.com/opencybersecurityalliance/stix-shifter/pull/1072)
+* Bump requests-toolbelt from 0.9.1 to 0.10.1 in /stix_shifter [#1180](https://github.com/opencybersecurityalliance/stix-shifter/pull/1180)
+* Bump jsonmerge from 1.8.0 to 1.9.0 in /stix_shifter [#1194](https://github.com/opencybersecurityalliance/stix-shifter/pull/1194)
+* Bump boto3 from 1.26.5 to 1.26.10 in /stix_shifter [#1193](https://github.com/opencybersecurityalliance/stix-shifter/pull/1193)
+* Bump boto3 from 1.21.21 to 1.26.1 in /stix_shifter [#1175](https://github.com/opencybersecurityalliance/stix-shifter/pull/1175)
+* Bump pyopenssl from 21.0.0 to 22.1.0 in /stix_shifter [#1144](https://github.com/opencybersecurityalliance/stix-shifter/pull/1144)
+
+--------------------------------------
+
+## 4.4.0 (2022-10-06)
+
+### Breaking changes:
+
+### Deprecations:
+
+### Changes:
+
+* Add optional group parameter to connector configs [#1094](https://github.com/opencybersecurityalliance/stix-shifter/pull/1094)
+* Adding GCP Chronicle UDI Connector [#1075](https://github.com/opencybersecurityalliance/stix-shifter/pull/1075)
+* Update Secretserver mappings [#1092](https://github.com/opencybersecurityalliance/stix-shifter/pull/1092)
+* Connector template for lab [#1117](https://github.com/opencybersecurityalliance/stix-shifter/pull/1117)
+
+### Fixes:
+
+* Get rid of StixObjectIdEncoder [#1124](https://github.com/opencybersecurityalliance/stix-shifter/pull/1124)
+* Fixed IBM Security Verify config file [#1125](https://github.com/opencybersecurityalliance/stix-shifter/pull/1125)
+* edits to coding lab [#1120](https://github.com/opencybersecurityalliance/stix-shifter/pull/1120)
+* Update epoch time to 10 digits for demo data [#1119](https://github.com/opencybersecurityalliance/stix-shifter/pull/1119)
+* update coding lab [#1114](https://github.com/opencybersecurityalliance/stix-shifter/pull/1114)
+* Lab fixes [#1116](https://github.com/opencybersecurityalliance/stix-shifter/pull/1116)
+
+### Dependency update:
+
+* Bump colorlog from 6.6.0 to 6.7.0 in /stix_shifter [#1095](https://github.com/opencybersecurityalliance/stix-shifter/pull/1095)
+
+--------------------------------------
+
+## 4.3.0 (2022-09-09)
+
+### Breaking changes:
+
+### Deprecations:
+
+### Changes:
+
+* CLI and coding tutorials [#1105](https://github.com/opencybersecurityalliance/stix-shifter/pull/1105)
+* Adding RHACS(StackRox) UDI connector [#1055](https://github.com/opencybersecurityalliance/stix-shifter/pull/1055)
+* Added Utility for normalization of connectors [#1078](https://github.com/opencybersecurityalliance/stix-shifter/pull/1078)
+* CrowdStrike: Added User-Agent string to API Client for tracking [#1064](https://github.com/opencybersecurityalliance/stix-shifter/pull/1064)
+* Process unique ID [#1051](https://github.com/opencybersecurityalliance/stix-shifter/pull/1051)
+* Added matcher lib support for 2.1 [#960](https://github.com/opencybersecurityalliance/stix-shifter/pull/960)
+* In query Enhancement [#1022](https://github.com/opencybersecurityalliance/stix-shifter/pull/1022)
+* Infoblox add docstrings for module [#719](https://github.com/opencybersecurityalliance/stix-shifter/pull/719)
+* Release/3.3.x json to stix [#598](https://github.com/opencybersecurityalliance/stix-shifter/pull/598)
+
+### Fixes:
+
+* Id contributing properties from json to py [#1093](https://github.com/opencybersecurityalliance/stix-shifter/pull/1093)
+* splunk: fix STIX timestamp processing [#1084](https://github.com/opencybersecurityalliance/stix-shifter/pull/1084)
+* Fixing absolute path for id_contributing_properties.json [#1079](https://github.com/opencybersecurityalliance/stix-shifter/pull/1079)
+* Fix mapping and added hex to int transformer [#1068](https://github.com/opencybersecurityalliance/stix-shifter/pull/1068)
+* Downgrade boto3 version to 1.21.21 [#1036](https://github.com/opencybersecurityalliance/stix-shifter/pull/1036)
+* Fix the length of the results of Qradar connector [#1034](https://github.com/opencybersecurityalliance/stix-shifter/pull/1034)
+* Revert "Change certificate parameter type for consistency" [#1031](https://github.com/opencybersecurityalliance/stix-shifter/pull/1031)
+* reaqta: enable certification authentication [#1028](https://github.com/opencybersecurityalliance/stix-shifter/pull/1028)
+* fix configuration in proofpoint and sumologic [#745](https://github.com/opencybersecurityalliance/stix-shifter/pull/745)
+* Validator review code change for Proofpoint [#739](https://github.com/opencybersecurityalliance/stix-shifter/pull/739)
+
+### Dependency update:
+
+
+--------------------------------------
+
+
 ## 4.2.0 (2022-06-29)
 ### Breaking changes:
 

OVERVIEW.md

@@ -145,7 +145,7 @@ List updated: October 29, 2021
 |       [Micro Focus ArcSight](adapter-guide/connectors/arcsight_supported_stix.md)       |    arcsight    |  Default   | IBM Security |     Yes     |     Yes      |   Released    |
 |       [CrowdStrike Falcon](adapter-guide/connectors/crowdstrike_supported_stix.md)       |    crowdstrike    |  Default   | IBM Security |     Yes     |     Yes      |   Released    |
 |       [Trend Micro Vision One](adapter-guide/connectors/trendmicro_vision_one_supported_stix.md)       |    trendmicro_vision_one    |  Default   | Trend Micro |     Yes     |     Yes      |   Released    |
-|       [Secret Server](adapter-guide/connectors/secretserver_supported_stix.md)       |    secretserver    |  Default   | IBM |     Yes     |     Yes      |   Released    |
+|       [IBM Security Verify Privilege Vault](adapter-guide/connectors/secretserver_supported_stix.md)       |    secretserver    |  Default   | IBM |     Yes     |     Yes      |   Released    |
 |       [One Login](adapter-guide/connectors/onelogin_supported_stix.md)       |    onelogin    |  Default   | GS Lab |     Yes     |     Yes      |   Released    |
 |       MySQL                                                                  |    mysql    |  Default   | IBM |     Yes     |     Yes      |   Released    |
 |       [Sumo Logic](adapter-guide/connectors/sumologic_supported_stix.md)       |    sumologic    |  Default   | GS Lab |     Yes     |     Yes      |   Released    |
@@ -157,13 +157,16 @@ List updated: October 29, 2021
 |       [SentinelOne](https://github.com/opencybersecurityalliance/stix-shifter/blob/develop/adapter-guide/connectors/sentinelone_supported_stix.md)                        | sentinelone              | Default    | IBM Security | Yes         | Yes          | Released     |
 |       [Darktrace](https://github.com/opencybersecurityalliance/stix-shifter/blob/develop/adapter-guide/connectors/darktrace_supported_stix.md)                           | darktrace              | Default    | IBM Security | Yes         | Yes          | Released     |
 |       [IBM Security ReaQta](https://github.com/opencybersecurityalliance/stix-shifter/blob/develop/adapter-guide/connectors/reaqta_supported_stix.md)                           | reaqta             | Default    | IBM Security | Yes         | Yes          | Released     |
+|       [IBM Security Verify](https://github.com/opencybersecurityalliance/stix-shifter/blob/develop/adapter-guide/connectors/ibm_security_verify_supported_stix.md)                           | ibm_security_verify             | Default    | IBM Security | Yes         | Yes          | Released     |
+|       [Red Hat Advanced Cluster Security for Kubernetes (StackRox)](https://github.com/opencybersecurityalliance/stix-shifter/blob/develop/adapter-guide/connectors/rhacs_supported_stix.md)                           | rhacs             | Default    | IBM Security | Yes         | Yes          | Released     |
+|      [GCP Chronicle](https://github.com/opencybersecurityalliance/stix-shifter/blob/develop/adapter-guide/connectors/gcp_chronicle_supported_stix.md)                   | gcp_chronicle              | Default    | IBM Security | Yes         | Yes          | Released     |
 
 
 ## How to use
 
 ### Prerequisites
 
-Python 3.6 is required to use stix-shifter.
+Python 3.8 or greater is required to use stix-shifter.
 
 Stix-shifter provides several functions: `translate` and `transmit` are the primary functions, `execute` offers a way to test the complete stix-shifter flow.
 

README.md

@@ -24,7 +24,7 @@ This stix-shifter has the following dependencies:
 - [antlr4-python3-runtime==4.8](https://pypi.org/project/antlr4-python3-runtime/)
 - [python-dateutil==2.8.1](https://pypi.org/project/python-dateutil/)
 
-Your development environment must use Python version: 3.6, 3.7, 3.8 or 3.9
+Your development environment must use Python version: 3.8 greater
 
 ## Installation
 
@@ -92,16 +92,19 @@ response = translation.translate('<MODULE NAME>', 'query', '{}', '<STIX PATTERN>
 print(response)
 ```
 
+
 ## Contributing
 
 We are thrilled you are considering contributing! We welcome all contributors.
 
 Please read our [guidelines for contributing](https://github.com/opencybersecurityalliance/stix-shifter/blob/develop/CONTRIBUTING.md).
 
-## Guide for creating new connectors
+## Developer Guides
 
 If you want to create a new connector for STIX-shifter, see the [developer guide](https://github.com/opencybersecurityalliance/stix-shifter/blob/develop/adapter-guide/develop-stix-adapter.md)
 
+There are also a few [Jupyter Notebook labs](https://github.com/opencybersecurityalliance/stix-shifter/blob/develop/lab) that cover the CLI commands and dev process.
+
 ## Licensing
 
 Licensed under the Apache License, Version 2.0 (the "License");

adapter-guide/connectors/alertflex_supported_stix.md

@@ -1,4 +1,4 @@
-##### Updated on 06/01/22
+##### Updated on 11/04/22
 ## Alertflex
 ### Supported STIX Operators
 | STIX Operator | Data Source Operator |
@@ -15,7 +15,29 @@
 | IN | IN |
 | MATCHES | LIKE |
 | <br> | |
-### Supported STIX Objects and Properties
+### Searchable STIX objects and properties
+| STIX Object and Property | Mapped Data Source Fields |
+|--|--|
+| **ipv4-addr**:value | a.dstIp, a.srcIp |
+| **network-traffic**:src_port | a.srcPort |
+| **network-traffic**:dst_port | a.dstPort |
+| **network-traffic**:src_ref | a.srcIp |
+| **network-traffic**:dst_ref | a.dstIp |
+| **file**:name | a.fileName |
+| **file**:hashes.'SHA-256' | a.hashSha256 |
+| **file**:hashes.'SHA-1' | a.hashSha1 |
+| **file**:hashes.MD5 | a.hashMd5 |
+| **process**:name | a.processName |
+| **process**:pid | a.processId |
+| **user-account**:user_id | a.userName |
+| **x_org_alertflex**:agent | a.agentName |
+| **x_org_alertflex**:node | a.nodeId |
+| **x_org_alertflex**:source | a.alertSource |
+| **x_org_alertflex**:type | a.alertType |
+| **x_org_alertflex**:id | a.eventId |
+| **x_org_alertflex**:severity | a.alertSeverity |
+| <br> | |
+### Supported STIX Objects and Properties for Query Results
 | STIX Object | STIX Property | Data Source Field |
 |--|--|--|
 | file | name | file |

adapter-guide/connectors/arcsight_supported_stix.md

@@ -1,4 +1,4 @@
-##### Updated on 06/01/22
+##### Updated on 11/04/22
 ## Micro Focus ArcSight
 ### Supported STIX Operators
 | STIX Operator | Data Source Operator |
@@ -16,7 +16,94 @@
 | MATCHES | CONTAINS |
 | ISSUBSET | insubnet |
 | <br> | |
-### Supported STIX Objects and Properties
+### Searchable STIX objects and properties
+| STIX Object and Property | Mapped Data Source Fields |
+|--|--|
+| **ipv4-addr**:value | sourceAddress, destinationAddress |
+| **ipv6-addr**:value | fulltextSearch |
+| **mac-addr**:value | sourceMacAddress, destinationMacAddress |
+| **network-traffic**:src_port | sourcePort |
+| **network-traffic**:dst_port | destinationPort |
+| **network-traffic**:protocols[*] | transportProtocol, applicationProtocol |
+| **network-traffic**:src_ref.value | sourceAddress, sourceMacAddress |
+| **network-traffic**:dst_ref.value | destinationAddress, destinationMacAddress |
+| **directory**:path | filePath |
+| **file**:parent_directory_ref.path | filePath |
+| **file**:name | fileName |
+| **file**:hashes.'SHA-256' | fulltextSearch |
+| **file**:hashes.'SHA-1' | fulltextSearch |
+| **file**:hashes.MD5 | fulltextSearch |
+| **process**:name | destinationProcessName, sourceProcessName |
+| **process**:parent_ref.name | sourceProcessName |
+| **process**:command_line | destinationServiceName, sourceServiceName |
+| **domain-name**:value | sourceHostName, destinationHostName |
+| **user-account**:user_id | destinationUserId, sourceUserId |
+| **user-account**:account_login | destinationUserName, sourceUserName |
+| **windows-registry-key**:key | filePath |
+| **windows-registry-key**:values[*] | deviceCustomString4 |
+| **x-arcsight-event**:priority | priority |
+| **x-arcsight-event**:base_event_count | baseEventCount |
+| **x-arcsight-event**:event_id | eventId |
+| **x-arcsight-event**:external_id | externalId |
+| **x-arcsight-event**:name | name |
+| **x-arcsight-event**:type | type |
+| **x-arcsight-event**:start_time | startTime |
+| **x-arcsight-event**:end_time | endTime |
+| **x-arcsight-event**:request_url | requestUrl |
+| **x-arcsight-event**:request_method | requestMethod |
+| **x-arcsight-event-category**:category_behavior | categoryBehavior |
+| **x-arcsight-event-category**:category_device_group | categoryDeviceGroup |
+| **x-arcsight-event-category**:category_object | categoryObject |
+| **x-arcsight-event-category**:category_outcome | categoryOutcome |
+| **x-arcsight-event-category**:category_significance | categorySignificance |
+| **x-arcsight-event-category**:category_technique | categoryTechnique |
+| **x-arcsight-event-device**:product | deviceProduct |
+| **x-arcsight-event-device**:vendor | deviceVendor |
+| **x-arcsight-event-device**:device_action | deviceAction |
+| **x-arcsight-event-device**:device_receipt_time | deviceReceiptTime |
+| **x-arcsight-event-device**:device_event_category | deviceEventCategory |
+| **x-arcsight-event-device**:device_severity | deviceSeverity |
+| **x-arcsight-event-device**:device_version | deviceVersion |
+| **x-arcsight-event-device**:device_address | deviceAddress |
+| **x-arcsight-event-device**:device_external_id | deviceExternalId |
+| **x-arcsight-event-device**:device_asset_id | fulltextSearch |
+| **x-arcsight-event-device**:device_asset_name | fulltextSearch |
+| **x-arcsight-event-device**:device_dns_domain | fulltextSearch |
+| **x-arcsight-event-device**:device_domain | fulltextSearch |
+| **x-arcsight-event-device**:device_nt_domain | fulltextSearch |
+| **x-arcsight-event-destination**:destination_asset_id | fulltextSearch |
+| **x-arcsight-event-destination**:destination_asset_name | fulltextSearch |
+| **x-arcsight-event-destination**:destination_dns_domain | destinationDnsDomain |
+| **x-arcsight-event-destination**:destination_fqdn | fulltextSearch |
+| **x-arcsight-event-destination**:destination_nt_domain | destinationNtDomain |
+| **x-arcsight-event-destination**:destination_geo_location_info | fulltextSearch |
+| **x-arcsight-event-destination**:destination_geo_postal_code | fulltextSearch |
+| **x-arcsight-event-source**:source_asset_id | fulltextSearch |
+| **x-arcsight-event-source**:source_asset_name | fulltextSearch |
+| **x-arcsight-event-source**:source_dns_domain | fulltextSearch |
+| **x-arcsight-event-source**:source_fqdn | fulltextSearch |
+| **x-arcsight-event-source**:source_nt_domain | sourceNtDomain |
+| **x-arcsight-event-source**:source_geo_location_info | fulltextSearch |
+| **x-arcsight-event-source**:source_geo_postal_code | fulltextSearch |
+| **x-arcsight-event-vulnerability**:vulnerability | fulltextSearch |
+| **x-arcsight-event-vulnerability**:vulnerability_external_id | vulnerabilityExternalID |
+| **x-arcsight-event-vulnerability**:vulnerability_id | fulltextSearch |
+| **x-arcsight-event-vulnerability**:vulnerability_name | fulltextSearch |
+| **x-arcsight-event-vulnerability**:vulnerability_reference_id | fulltextSearch |
+| **x-arcsight-event-vulnerability**:vulnerability_resource | fulltextSearch |
+| **x-arcsight-event-vulnerability**:vulnerability_uri | vulnerabilityURI |
+| **x-ibm-finding**:name | name |
+| **x-ibm-finding**:finding_type | categorySignificance |
+| **x-ibm-finding**:src_device | fulltextSearch |
+| **x-ibm-finding**:dst_device | fulltextSearch |
+| **x-ibm-finding**:src_geolocation | fulltextSearch |
+| **x-ibm-finding**:dst_geolocation | fulltextSearch |
+| **x-ibm-finding**:src_ip_ref.value | sourceAddress |
+| **x-ibm-finding**:dst_ip_ref.value | destinationAddress |
+| **x-oca-asset**:hostname | deviceHostName, deviceAssetName |
+| **x-oca-asset**:host_id | deviceAssetId |
+| <br> | |
+### Supported STIX Objects and Properties for Query Results
 | STIX Object | STIX Property | Data Source Field |
 |--|--|--|
 | directory | path | filePath |