Feature: Adding version enforcement

[cmcginley-splunk]

Context

• Detections now have versions
• Prior to publishing content, we want to enforce that if a detection changes, its version gets bumped appropriately

Code changes

• SplunkApp extended to be able to download an app directly from Splunkbase
• inspect action extended to perform checks against a current build's detections against a prior build's detections
  • new previous_build config option; if unspecified, latest release from Splunkbase is downloaded to use for detection comparison as previous build
• DetectionStanza class added to model an individual stanza representing a detection in savedsearches.conf
• SavedsearchesConf class added to model and parse a savedsearches.conf file
  • Note that currently this class does not model the full file, but only the ESCU detections as the rest is not relevant to us at this time
• Validations performed:
  • No detections were deleted between builds
  • No detection IDs changed
  • No versions were decremented
  • If a detection changed between releases (based on stanza hash), its version was bumped appropriately
• Other changes include some formatting and typing cleanup

Testing

• Tested internally against the v4.39.1 release, with a few local manual edits to force the generation of errors

AppInspect found [error,failure,manual_check,warning] that MAY cause a failure during AppInspect API:
 - 9 warnings
Details:
 - warning [check_app_configuration_file: check_custom_conf_replication]
 - warning [check_app_configuration_file: check_for_updates_disabled]
 - warning [check_app_configuration_file: check_for_valid_package_id]
 - warning [check_bias_language: check_for_bias_language]
 - warning [check_malware: check_hostnames_and_ips]
 - warning [check_meta_files: check_kos_are_accessible]
 - warning [check_packaging_standards: check_valid_version_number]
 - warning [check_packaging_standards: check_version_is_valid_semver]
 - warning [check_slim_ssai: check_custom_confs]
Downloading latest ESCU build from Splunkbase to serve as previous build during validation...
Latest release downloaded from Splunkbase to: downloads/escu_latest.tgz
Detection Metadata Enforcement:
	❌ ESCU - Email Attachments With Lots Of Spaces - Rule
		🔸 Detection version in current build should be bumped to at least 4.
	❌ ESCU - O365 Compliance Content Search Started - Rule
		🔸 Detection from previous build not found in current build.
	❌ ESCU - Detect API activity from users without MFA - Rule
		🔸 Detection version (1) in current build is less than version (2) in previous build.
	❌ ESCU - Zscaler Virus Download threat blocked - Rule
		🔸 Detection ID aa19e627-d448-4a31-85cd-82068dec5692 in current build does not match ID aa19e627-d448-4a31-85cd-82068dec5691 in previous build.
		🔸 Detection version (1) in current build is less than version (2) in previous build.
		🔸 Detection version in current build should be bumped to at least 3.
Verbose error logging is DISABLED.
Please use the --verbose command line argument if you need more context for your error or file a bug report.
Validation errors when comparing detection stanzas in current and previous build: (6 sub-exceptions)

TODO

• Create PR to external security_content adding downloads/ to .gitignore and adding --enable-metadata-validation to CLI invocation OR contentctl.yml in CI/CD
• Create MR to internal security_content adding downloads/ to .gitignore and adding --enable-metadata-validation to CLI invocation OR contentctl.yml in CI/CD

[pyth0n1c]

Feel free to close out any comments as resolved if you feel that's appropriate.
If you agree that suggestions should be future enhancements, then let me know and I will create the issue(s).

[pyth0n1c]

Most of the comments are resolved, but still one or two more open.
I think these will be a very small change.

[pyth0n1c]

We had some extended discussions in the comments, but I wanted to add the functional testing I performed on this build. I have tested the following which worked as anticipated:

1. Checking for changes with the newest version of security_content against the latest version of ESCU in Splunkbase. All anticipated changes were found.
2. Check for changes against two local builds of security_content. The first with security_content exactly as it appears in main and the second with the following changes:
   a. No changes.
   b. One detection updated
   c. several detections updated
   d. The name/filename of a detection changes (this gives a different error, as it should)
   e. Detection has the same name, but a different UUID. This also gives a different error, as it should.
   f. Detection removed from newer version of a package. This also throws an error as it should.
3. Created a NEW app using contentctl init with some content. Built it, then made some changes (as you see above) and re-inspected. It picked up all changes between this new local app and the older local app correctly as well.

Great job by @cmcginley-splunk , I approve. Thank you!