Merge pull request #259 from splunk/ci_for_links

adding liche CI test

.circleci/config.yml

@@ -15,7 +15,12 @@ apt-run: &apt-install
   name: install system packages
   command: |
     sudo apt update -qq
-    sudo apt install -y python-dev -qq
+    sudo apt install -y python-dev snapd -qq
+    # install go for other testing tools
+    wget https://raw.githubusercontent.com/canha/golang-tools-install-script/master/goinstall.sh
+    sudo chown circleci goinstall.sh
+    chmod +x goinstall.sh
+    ./goinstall.sh
 
 executors:
   content-executor:
@@ -61,7 +66,25 @@ jobs:
             cd security-content
             source venv/bin/activate
             python bin/validate.py --path . --verbose
-
+      - run:
+          name: run doc-gen
+          command: |
+            cd security-content
+            source venv/bin/activate
+            python bin/doc-gen.py --path . --output docs -v
+      - run:
+          name: check for broken links using liche 
+          command: |
+            echo 'export GOROOT=~/.go' >> $BASH_ENV
+            echo 'export PATH=$GOROOT/bin:$PATH' >> $BASH_ENV
+            echo 'export GOPATH=~/go' >> $BASH_ENV
+            echo 'export PATH=$GOPATH/bin:$PATH' >> $BASH_ENV
+            echo 'export GO111MODULE="on"' >> $BASH_ENV 
+            source $BASH_ENV
+            go get -u github.com/raviqqe/liche
+            cd security-content
+            liche -r docs/
+            liche README.md
   build-sources:
     executor: content-executor
     steps:

README.md

@@ -38,7 +38,7 @@ curl -s https://content.splunkresearch.com | jq
 ```
 
 # What's in an Analytic Story?
-[Analytic Stories](https://github.com/splunk/security-content/blob/develop/docs/stories_categories.md) and their corresponding searches are composed of **.yml** files (manifests) and associated .conf files. The stories reside in [/stories](/stories) and the searches live in [/detections](/detections). 
+[Analytic Stories](https://github.com/splunk/security-content/blob/develop/docs/stories_categories.md) and their corresponding searches are composed of **.yml** files (manifests) and associated .conf files. The stories reside in [/stories](https://github.com/splunk/security-content/tree/develop/stories) and the searches live in [/detections](https://github.com/splunk/security-content/tree/develop/detections). 
 
 Manifests contain a number of mandatory and optional fields. You can see the full field list for each piece of content [here](https://github.com/splunk/security-content/tree/develop/docs#spec-documentation).
 

docs/CONTRIBUTING.md

@@ -34,7 +34,7 @@ If you wish to be a contributing member of our community, please see the agreeme
 Please make sure to read and observe our [Code of Conduct](contributing/code-of-conduct.md). Please follow it in all of your interactions involving the project.
 
 ##### Setup Development Environment
-see [Developing section](README.MD/#Developing)
+see [Developing section](https://github.com/splunk/security-content#developing)
 
 ## Contribution Workflow
 Help is always welcome! For example, documentation can always use improvement. There's always code that can be clarified, functionality that can be extended, and tests to be added to guarantee behavior. If you see something you think should be fixed, don't be afraid to own it.

docs/contributing/code-of-conduct.md

@@ -0,0 +1,73 @@
+## Code of Conduct
+
+### Our Pledge
+
+In the interest of fostering an open and welcoming environment, we as
+contributors and maintainers pledge to making participation in our project and
+our community a harassment-free experience for everyone, regardless of age, body
+size, disability, ethnicity, gender identity and expression, level of experience,
+nationality, personal appearance, race, religion, or sexual identity and
+orientation.
+
+### Our Standards
+
+Examples of behavior that contributes to creating a positive environment
+include:
+
+* Using welcoming and inclusive language
+* Being respectful of differing viewpoints and experiences
+* Gracefully accepting constructive criticism
+* Focusing on what is best for the community
+* Showing empathy towards other community members
+
+Examples of unacceptable behavior by participants include:
+
+* The use of sexualized language or imagery and unwelcome sexual attention or
+advances
+* Trolling, insulting/derogatory comments, and personal or political attacks
+* Public or private harassment
+* Publishing others' private information, such as a physical or electronic address, without explicit permission
+* Other conduct which could reasonably be considered inappropriate in a professional setting
+
+### Our Responsibilities
+
+Project maintainers are responsible for clarifying the standards of acceptable
+behavior and are expected to take appropriate and fair corrective action in
+response to any instances of unacceptable behavior.
+
+Project maintainers have the right and responsibility to remove, edit, or
+reject comments, commits, code, wiki edits, issues, and other contributions
+that are not aligned to this Code of Conduct, or to ban temporarily or
+permanently any contributor for other behaviors that they deem inappropriate,
+threatening, offensive, or harmful.
+
+### Scope
+
+This Code of Conduct applies both within project spaces and in public spaces
+when an individual is representing the project or its community. Examples of
+representing a project or community include using an official project e-mail
+address, posting via an official social media account, or acting as an appointed
+representative at an online or offline event. Representation of a project may be
+further defined and clarified by project maintainers.
+
+### Enforcement
+
+Instances of abusive, harassing, or otherwise unacceptable behavior may be
+reported by contacting the project team at [email protected]. All
+complaints will be reviewed and investigated and will result in a response that
+is deemed necessary and appropriate to the circumstances. The project team is
+obligated to maintain confidentiality with regard to the reporter of an incident.
+Further details of specific enforcement policies may be posted separately.
+
+Project maintainers who do not follow or enforce the Code of Conduct in good
+faith may face temporary or permanent repercussions as determined by other
+members of the project's leadership.
+
+### Attribution
+
+This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4,
+available at [http://contributor-covenant.org/version/1/4][version]
+
+[homepage]: http://contributor-covenant.org
+[version]: http://contributor-covenant.org/version/1/4/
+

docs/splunk_docs_categories.wiki

@@ -1317,7 +1317,7 @@ Notable events will include IP addresses, URLs, and user data. Drilling down can
 * DE.CM
 
 ====References====
-* https://blog.domaintools.com/tag/brand-monitor/
+* https://www.zerofox.com/blog/what-is-digital-risk-monitoring/
 * https://securingtomorrow.mcafee.com/consumer/family-safety/what-is-typosquatting/
 * https://blog.malwarebytes.com/cybercrime/2016/06/explained-typosquatting/
 
@@ -1371,7 +1371,7 @@ The search in this story can help you to detect if attackers are abusing your co
 
 ====References====
 * https://www.us-cert.gov/ncas/alerts/TA13-088A
-* https://deepthought.isc.org/article/AA-00897/0/What-is-a-DNS-Amplification-Attack.html
+* https://www.cloudflare.com/learning/ddos/dns-amplification-ddos-attack/
 
 creation_date = 2016-08-24
 
@@ -3041,6 +3041,8 @@ Once a phishing message has been detected, the next steps are to answer the foll
 
 ====Providing Technologies====
 * Microsoft Exchange
+* Bro
+* Splunk Stream
 
 ====Data Models====
 * Email

docs/stories_categories.md

@@ -1491,7 +1491,7 @@ Web
 * company = [email protected]
 
 ##### References
-* https://blog.domaintools.com/tag/brand-monitor/
+* https://www.zerofox.com/blog/what-is-digital-risk-monitoring/
 * https://securingtomorrow.mcafee.com/consumer/family-safety/what-is-typosquatting/
 * https://blog.malwarebytes.com/cybercrime/2016/06/explained-typosquatting/
 
@@ -1547,7 +1547,7 @@ Network_Resolution
 
 ##### References
 * https://www.us-cert.gov/ncas/alerts/TA13-088A
-* https://deepthought.isc.org/article/AA-00897/0/What-is-a-DNS-Amplification-Attack.html
+* https://www.cloudflare.com/learning/ddos/dns-amplification-ddos-attack/
 
 ### Data Protection
 * id = `91c676cf-0b23-438d-abee-f6335e1fce33`
@@ -3461,12 +3461,15 @@ Once a phishing message has been detected, the next steps are to answer the foll
 1. Have any users interacted with the content of the messages (by downloading an attachment or clicking on a malicious URL)?This Analytic Story provides detection searches to identify suspicious emails, as well as contextual and investigative searches to help answer some of these questions.
 
 ##### Detections
+* Monitor Email For Brand Abuse
 * Suspicious Email Attachment Extensions
 * Email Attachments With Lots Of Spaces
 * Suspicious Email - UBA Anomaly
 
 ##### Providing Technologies
 * Microsoft Exchange
+* Bro
+* Splunk Stream
 
 ##### Data Models
 Email

package/default/analytic_stories.conf

@@ -1,6 +1,6 @@
 #############
 # Automatically generated by generator.py in splunk/security-content
-# On Date: 2019-11-05T20:22:48 UTC
+# On Date: 2019-11-18T18:30:07 UTC
 # Author: Splunk Security Research
 # Contact: [email protected]
 #############
@@ -163,7 +163,7 @@ creation_date = 2017-06-01
 modification_date = 2017-12-19
 id = 91c676cf-0b23-438d-abee-f6335e1fce78
 version = 1.0
-reference = ["https://blog.domaintools.com/tag/brand-monitor/", "https://securingtomorrow.mcafee.com/consumer/family-safety/what-is-typosquatting/", "https://blog.malwarebytes.com/cybercrime/2016/06/explained-typosquatting/"]
+reference = ["https://www.zerofox.com/blog/what-is-digital-risk-monitoring/", "https://securingtomorrow.mcafee.com/consumer/family-safety/what-is-typosquatting/", "https://blog.malwarebytes.com/cybercrime/2016/06/explained-typosquatting/"]
 detection_searches = ["ESCU - Monitor DNS For Brand Abuse - Rule", "ESCU - Monitor Email For Brand Abuse - Rule", "ESCU - Monitor Web Traffic For Brand Abuse - Rule"]
 mappings = {"cis20": ["CIS 7"], "kill_chain_phases": ["Actions on Objectives", "Delivery"], "mitre_attack": [], "nist": ["PR.IP"]}
 investigative_searches = ["ESCU - Get Authentication Logs For Endpoint", "ESCU - Get Email Info", "ESCU - Get Emails From Specific Sender", "ESCU - Get Notable History", "ESCU - Get Notable Info", "ESCU - Get Process Responsible For The DNS Traffic", "ESCU - Get Risk Modifiers For Endpoint", "ESCU - Get Risk Modifiers For User", "ESCU - Get User Information from Identity Table", "ESCU - Investigate Web Activity From Host"]
@@ -291,7 +291,7 @@ version = 2.0
 reference = ["https://www.us-cert.gov/ncas/alerts/TA18-074A"]
 detection_searches = ["ESCU - Create local admin accounts using net.exe - Rule", "ESCU - Detect New Local Admin account - Rule", "ESCU - Detect Outbound SMB Traffic - Rule", "ESCU - Detect PsExec With accepteula Flag - Rule", "ESCU - First time seen command line argument - Rule", "ESCU - Malicious PowerShell Process - Execution Policy Bypass - Rule", "ESCU - Processes launching netsh - Rule", "ESCU - Registry Keys Used For Persistence - Rule", "ESCU - SMB Traffic Spike - MLTK - Rule", "ESCU - SMB Traffic Spike - Rule", "ESCU - Sc.exe Manipulating Windows Services - Rule", "ESCU - Scheduled Task Name Used by Dragonfly Threat Actors - Rule", "ESCU - Single Letter Process On Endpoint - Rule", "ESCU - Suspicious Reg.exe Process - Rule"]
 mappings = {"cis20": ["CIS 12", "CIS 16", "CIS 2", "CIS 3", "CIS 5", "CIS 7", "CIS 8"], "kill_chain_phases": ["Actions on Objectives", "Command and Control", "Installation"], "mitre_attack": ["AppInit DLLs", "Authentication Package", "Command and Control", "Command-Line Interface", "Commonly Used Port", "Credential Access", "Defense Evasion", "Disabling Security Tools", "Execution", "Lateral Movement", "Modify Existing Service", "Modify Registry", "New Service", "Persistence", "PowerShell", "Privilege Escalation", "Registry Run Keys / Start Folder", "Scheduled Task", "Scripting", "Valid Accounts"], "nist": ["DE.AE", "DE.CM", "ID.AM", "PR.AC", "PR.AT", "PR.DS", "PR.IP", "PR.PT"]}
-investigative_searches = ["ESCU - Get Authentication Logs For Endpoint", "ESCU - Get Notable History", "ESCU - Get Notable Info", "ESCU - Get Parent Process Info", "ESCU - Get Process Info", "ESCU - Get Process Information For Port Activity", "ESCU - Get Registry Activities", "ESCU - Get Risk Modifiers For Endpoint", "ESCU - Get Risk Modifiers For User", "ESCU - Get User Information from Identity Table", "ESCU - Get Vulnerability Logs For Endpoint", "ESCU - Investigate Web Activity From Host"]
+investigative_searches = ["ESCU - Get Authentication Logs For Endpoint", "ESCU - Get Notable History", "ESCU - Get Notable Info", "ESCU - Get Parent Process Info", "ESCU - Get Process File Activity", "ESCU - Get Process Info", "ESCU - Get Process Information For Port Activity", "ESCU - Get Process Registry Activity", "ESCU - Get Registry Activities", "ESCU - Get Risk Modifiers For Endpoint", "ESCU - Get Risk Modifiers For User", "ESCU - Get User Information from Identity Table", "ESCU - Get Vulnerability Logs For Endpoint", "ESCU - Investigate Web Activity From Host"]
 support_searches = ["ESCU - Baseline of SMB Traffic - MLTK", "ESCU - Previously seen command line arguments"]
 data_models = ["Authentication", "Endpoint", "Network_Traffic", "Risk", "Vulnerabilities", "Web"]
 providing_technologies = ["Bluecoat", "Bro", "Carbon Black Response", "CrowdStrike Falcon", "Linux", "Microsoft Windows", "Nessus", "Palo Alto Firewall", "Splunk Enterprise Security", "Splunk Stream", "Sysmon", "Tanium", "Ziften", "macOS"]
@@ -307,7 +307,7 @@ creation_date = 2016-08-24
 modification_date = 2016-09-13
 id = e8afd39e-3294-11e6-b39d-a45e60c6700
 version = 1.0
-reference = ["https://www.us-cert.gov/ncas/alerts/TA13-088A", "https://deepthought.isc.org/article/AA-00897/0/What-is-a-DNS-Amplification-Attack.html"]
+reference = ["https://www.us-cert.gov/ncas/alerts/TA13-088A", "https://www.cloudflare.com/learning/ddos/dns-amplification-ddos-attack/"]
 detection_searches = ["ESCU - Large Volume of DNS ANY Queries - Rule"]
 mappings = {"cis20": ["CIS 11", "CIS 12"], "kill_chain_phases": ["Actions on Objectives"], "mitre_attack": [], "nist": ["DE.AE", "PR.IP", "PR.PT"]}
 investigative_searches = ["ESCU - Get Notable History", "ESCU - Get Notable Info", "ESCU - Get Risk Modifiers For Endpoint", "ESCU - Get Risk Modifiers For User"]
@@ -927,12 +927,12 @@ modification_date = 2017-09-19
 id = 2b1800dd-92f9-47ec-a981-fdf1351e5d55
 version = 1.0
 reference = ["https://www.splunk.com/blog/2015/06/26/phishing-hits-a-new-level-of-quality/"]
-detection_searches = ["ESCU - Email Attachments With Lots Of Spaces - Rule", "ESCU - Suspicious Email Attachment Extensions - Rule"]
+detection_searches = ["ESCU - Email Attachments With Lots Of Spaces - Rule", "ESCU - Monitor Email For Brand Abuse - Rule", "ESCU - Suspicious Email Attachment Extensions - Rule"]
 mappings = {"cis20": ["CIS 12", "CIS 3", "CIS 7"], "kill_chain_phases": ["Delivery"], "mitre_attack": ["Defense Evasion", "Execution"], "nist": ["DE.AE", "PR.IP"]}
-investigative_searches = []
-support_searches = []
-data_models = ["Email", "UEBA"]
-providing_technologies = ["Cuckoo", "DeepSight", "Microsoft Exchange", "SMTP", "Splunk Enterprise Security", "VirusTotal"]
+investigative_searches = ["ESCU - Get Authentication Logs For Endpoint", "ESCU - Get Email Info", "ESCU - Get Emails From Specific Sender", "ESCU - Get Notable History", "ESCU - Get Notable Info", "ESCU - Get Risk Modifiers For Endpoint", "ESCU - Get Risk Modifiers For User", "ESCU - Get User Information from Identity Table", "ESCU - Investigate Web Activity From Host"]
+support_searches = ["ESCU - DNSTwist Domain Names"]
+data_models = ["Authentication", "Email", "Risk", "UEBA", "Web"]
+providing_technologies = ["Bluecoat", "Bro", "Cuckoo", "DeepSight", "Linux", "Microsoft Exchange", "Microsoft Windows", "Palo Alto Firewall", "SMTP", "Splunk Enterprise", "Splunk Enterprise Security", "Splunk Stream", "VirusTotal", "macOS"]
 description = Email remains one of the primary means for attackers to gain an initial foothold within the modern enterprise. Detect and investigate suspicious emails in your environment with the help of the searches in this Analytic Story.
 narrative = It is a common practice for attackers of all types to leverage targeted spearphishing campaigns and mass mailers to deliver weaponized email messages and attachments. Fortunately, there are a number of ways to monitor email data in Splunk to detect suspicious content.\
 Once a phishing message has been detected, the next steps are to answer the following questions: \