Feature: validation of detections against cms_main

[cmcginley-splunk]

Context

• We need to validate that our detections not only work in ES, but are also being parsed correctly by the content versioning service

Code change

• Added ContentVersioningService
  • Enables content versioning
  • Activates CMS parser
  • Validates detections against CMS events, reports any missing detections or other validation issues. Validations include:
    • Counts between cms_main and detections based on YAMLs match
    • Each detection can be matched to a cms_event
    • Each cms_event can be matched to a detection
    • No cms_event is repeated
    • UUIDs match between cms/detection pairs
    • Versions match between cms/detection pairs
    • Full search name matches between cms/detection pairs
    • Correlationsearch label matches between cms/detection pairs
    • Sourcetype matches between cms/detection pairs
  • Returns a list of setup functions which are optionally inserted into the main setup of DetectionTestingInfrastructure
• Aggregating errors across all threads (setup, testing, view, view shutdown) and reporting en masse
• Broke up setup functions list to allow for optional inclusion of the content versioning setup functions
• NOTE: Now we raise any exceptions that crop up during setup, instead of just logging them
• Added logic to DetectionTestingInfrastructure to query installed ES version, so we can determine when it is appropriate to run content versioning validation
  • Content versioning validation is only run when 1) integration testing is enabled, 2) ES 8.0.0+ is installed, and 3) when the testing mode is All
• Moved logging utility in CorrelationSearch (useful for debugging) to helper.utils.Utils so it could be re-used in ContentVersioningService

Testing

• I've tested locally w/ some quick and dirty edits to the savedsearches.conf file and all seems to work well
• Want to test in CI, but currently the changes to dashboards in contentctl preclude me from testing this on the security_content repo as it exists today

Caveats

• Note that we now raise exceptions from setup, instead of logging and dropping them; I believe this is desirable
• I went back and forth as to whether this validation should happen as part of setup, OR if I should create some new type of test case. I opted for the former as this type of testing doesn't fit our current testing paradigm; validation against the cms_main index should happen in aggregate, a test of the whole package, not individual detections. We do not have an easy way to insert testing at that level presently.
• Additionally, this validation probably only needs to be run against a single instance and not necessarily all instances in a single run, but I don't see an easy or clean way to do that as well, and there's no harm or impact on execution time as far as I can tell

TODOs

• disable logging
• remove TODOs or create follow up issues
• Test in CI when able