Make ScubaResults.json the default JSON output (#1316)

* Invert MergeJSON to KeepIndividualJSON * remove debuging output message * rm trailing whitespace * rm other trailing whitespace * add todo comment * fix files * update smoketests * address missing mergejson documentation * Update KeepIndividualJSON description Co-authored-by: Alden Hilton <[email protected]> * add missing keep json call * Update docs/execution/reports.md from review feedback Co-authored-by: mitchelbaker-cisa <[email protected]> * Update PowerShell/ScubaGear/Modules/Orchestrator.psm1 from review feedback Co-authored-by: mitchelbaker-cisa <[email protected]> * add in nobom fix to ScubaCached * fix comment language * innoculous change * retrigger the action --------- Co-authored-by: Alden Hilton <[email protected]> Co-authored-by: mitchelbaker-cisa <[email protected]>
cisagov · Sep 20, 2024 · b8f5480 · b8f5480
1 parent 4d9c7a4
commit b8f5480
Show file tree

Hide file tree

Showing 8 changed files with 59 additions and 34 deletions.
diff --git a/PowerShell/ScubaGear/Modules/Orchestrator.psm1 b/PowerShell/ScubaGear/Modules/Orchestrator.psm1
@@ -66,11 +66,11 @@ function Invoke-SCuBA {
     .Parameter OutReportName
     The name of the main html file page created in the folder created in OutPath.
     Defaults to "BaselineReports".
-    .Parameter MergeJson
-    Set switch to merge all json output into a single file and delete the individual files
-    after merging.
+    .Parameter KeepIndividualJSON
+    Keeps ScubaGear legacy output where files are not merged into an all in one JSON.
+    This parameter is for backwards compatibility for those working with the older ScubaGear output files.
     .Parameter OutJsonFileName
-    If MergeJson is set, the name of the consolidated json created in the folder
+    If KeepIndividualJSON is not set, the name of the consolidated json created in the folder
     created in OutPath. Defaults to "ScubaResults".
     .Parameter OutCsvFileName
     The CSV created in the folder created in OutPath that contains the CSV version of the test results.
@@ -208,7 +208,7 @@ function Invoke-SCuBA {
         [Parameter(Mandatory = $false, ParameterSetName = 'Report')]
         [ValidateNotNullOrEmpty()]
         [switch]
-        $MergeJson,
+        $KeepIndividualJSON,
 
         [Parameter(Mandatory = $false, ParameterSetName = 'Configuration')]
         [Parameter(Mandatory = $false, ParameterSetName = 'Report')]
@@ -277,7 +277,7 @@ function Invoke-SCuBA {
                 'OutProviderFileName' = $OutProviderFileName
                 'OutRegoFileName' = $OutRegoFileName
                 'OutReportName' = $OutReportName
-                'MergeJson' = $MergeJson
+                'KeepIndividualJSON' = $KeepIndividualJSON
                 'OutJsonFileName' = $OutJsonFileName
                 'OutCsvFileName' = $OutCsvFileName
             }
@@ -408,7 +408,7 @@ function Invoke-SCuBA {
             }
             Invoke-ReportCreation @ReportParams
 
-            if ($MergeJson) {
+            if (-not $KeepIndividualJSON) {
                 # Craft the complete json version of the output
                 $JsonParams = @{
                     'ProductNames' = $ScubaConfig.ProductNames;
@@ -1544,11 +1544,11 @@ function Invoke-SCuBACached {
     .Parameter OutReportName
     The name of the main html file page created in the folder created in OutPath.
     Defaults to "BaselineReports".
-    .Parameter MergeJson
-    Set switch to merge all json output into a single file and delete the individual files
-    after merging.
+    .Parameter KeepIndividualJSON
+    Keeps ScubaGear legacy output where files are not merged into an all in one JSON.
+    This parameter is for backwards compatibility for those working with the older ScubaGear output files.
     .Parameter OutJsonFileName
-    If MergeJson is set, the name of the consolidated json created in the folder
+    If KeepIndividualJSON is set, the name of the consolidated json created in the folder
     created in OutPath. Defaults to "ScubaResults".
     .Parameter OutCsvFileName
     The CSV created in the folder created in OutPath that contains the CSV version of the test results.
@@ -1651,7 +1651,7 @@ function Invoke-SCuBACached {
         [Parameter(Mandatory = $false, ParameterSetName = 'Report')]
         [ValidateNotNullOrEmpty()]
         [switch]
-        $MergeJson,
+        $KeepIndividualJSON,
 
         [Parameter(Mandatory = $false, ParameterSetName = 'Report')]
         [ValidateNotNullOrEmpty()]
@@ -1732,8 +1732,23 @@ function Invoke-SCuBACached {
                 }
                 Invoke-ProviderList @ProviderParams
             }
-            $FileName = Join-Path -Path $OutPath -ChildPath "$($OutProviderFileName).json"
-            $SettingsExport = Get-Content $FileName | ConvertFrom-Json
+
+            $ProviderJSONFilePath = Join-Path -Path $OutPath -ChildPath "$($OutProviderFileName).json"
+            if (-not (Test-Path $ProviderJSONFilePath)) {
+                # When running Invoke-ScubaCached, the provider output might not exist as a stand-alone
+                # file depending on what version of ScubaGear created the output. If the provider output
+                # does not exist as a stand-alone file, create it from the ScubaResults file so the other functions
+                # can execute as normal.
+                $ScubaResultsFileName = Join-Path -Path $OutPath -ChildPath "$($OutJsonFileName).json"
+                $SettingsExport = $(Get-Content $ScubaResultsFileName | ConvertFrom-Json).Raw
+
+                # Uses the custom UTF8 NoBOM function to reoutput the Provider JSON file
+                $ProviderContent = $SettingsExport | ConvertTo-Json -Depth 20
+                $ActualSavedLocation = Set-Utf8NoBom -Content $ProviderContent `
+                -Location $ProviderJSONFilePath -FileName "$OutProviderFileName.json"
+                Write-Debug $ActualSavedLocation
+            }
+            $SettingsExport = Get-Content $ProviderJSONFilePath | ConvertFrom-Json
             $TenantDetails = $SettingsExport.tenant_details
             $RegoParams = @{
                 'ProductNames' = $ProductNames;
@@ -1757,7 +1772,7 @@ function Invoke-SCuBACached {
             Invoke-RunRego @RegoParams
             Invoke-ReportCreation @ReportParams
 
-            if ($MergeJson) {
+            if (-not $KeepIndividualJSON) {
                 # Craft the complete json version of the output
                 $JsonParams = @{
                     'ProductNames' = $ProductNames;

diff --git a/PowerShell/ScubaGear/Testing/Unit/PowerShell/Orchestrator/Invoke-Scuba.Tests.ps1 b/PowerShell/ScubaGear/Testing/Unit/PowerShell/Orchestrator/Invoke-Scuba.Tests.ps1
@@ -22,6 +22,12 @@ InModuleScope Orchestrator {
             function Get-ScubaDefault {throw 'this will be mocked'}
             Mock -ModuleName Orchestrator Get-ScubaDefault {"."}
 
+            function Merge-JsonOutput {throw 'this will be mocked'}
+            Mock -ModuleName Orchestrator Merge-JsonOutput {}
+
+            function ConvertTo-ResultsCsv {throw 'this will be mocked'}
+            Mock -ModuleName Orchestrator ConvertTo-ResultsCsv {}
+
             Mock -CommandName New-Item {}
             Mock -CommandName Copy-Item {}
         }
@@ -30,6 +36,7 @@ InModuleScope Orchestrator {
                 [Diagnostics.CodeAnalysis.SuppressMessageAttribute('PSUseDeclaredVarsMoreThanAssignments', 'SplatParams')]
                 $SplatParams = @{
                     M365Environment = 'commercial';
+                    KeepIndividualJSON = $true;
                 }
             }
             It 'Do it quietly (Do not automatically show report)' {

diff --git a/PowerShell/ScubaGear/Testing/Unit/PowerShell/Orchestrator/Invoke-ScubaCached.Tests.ps1 b/PowerShell/ScubaGear/Testing/Unit/PowerShell/Orchestrator/Invoke-ScubaCached.Tests.ps1
@@ -21,6 +21,10 @@ InModuleScope Orchestrator {
             function Disconnect-SCuBATenant {}
             Mock -ModuleName Orchestrator Disconnect-SCuBATenant
 
+            function Set-Utf8NoBom {}
+            Mock -ModuleName Orchestrator Set-Utf8NoBom
+
+            Mock -CommandName Write-Debug {}
             Mock -CommandName New-Item {}
             Mock -CommandName Get-Content {}
         }

diff --git a/PowerShell/ScubaGear/Testing/Unit/PowerShell/Orchestrator/Invoke-ScubaConfig.Tests.ps1 b/PowerShell/ScubaGear/Testing/Unit/PowerShell/Orchestrator/Invoke-ScubaConfig.Tests.ps1
@@ -35,6 +35,8 @@ InModuleScope Orchestrator {
                 Mock -ModuleName Orchestrator Merge-JsonOutput {
                     $script:TestSplat.Add('OutJsonFileName', $ScubaConfig.OutJsonFileName)
                 }
+                function ConvertTo-ResultsCsv {throw 'this will be mocked'}
+                Mock -ModuleName Orchestrator ConvertTo-ResultsCsv {}
                 function Disconnect-SCuBATenant {
                     $script:TestSplat.Add('DisconnectOnExit', $DisconnectOnExit)
                 }
@@ -65,8 +67,7 @@ InModuleScope Orchestrator {
                     }
                 }
                 [ScubaConfig]::ResetInstance()
-                Invoke-SCuBA -ConfigFilePath (Join-Path -Path $PSScriptRoot -ChildPath "orchestrator_config_test.yaml")`
-                    -MergeJson
+                Invoke-SCuBA -ConfigFilePath (Join-Path -Path $PSScriptRoot -ChildPath "orchestrator_config_test.yaml")
             }
 
             It "Verify parameter ""<parameter>"" with value ""<value>""" -ForEach @(
@@ -100,7 +101,6 @@ InModuleScope Orchestrator {
                   -OutFolderName "MyReports" `
                   -OutProviderFileName "MySettingsExport" `
                   -OutRegoFileName "RegoResults" `
-                  -MergeJson:$true `
                   -OutReportName "MyReport" `
                   -OutJsonFileName "JsonResults" `
                   -Organization "good.four.us" `

diff --git a/Testing/Functional/Products/Products.Tests.ps1 b/Testing/Functional/Products/Products.Tests.ps1
@@ -205,10 +205,10 @@ BeforeAll {
     function RunScuba() {
         if (-not [string]::IsNullOrEmpty($Thumbprint))
         {
-            Invoke-SCuBA -CertificateThumbPrint $Thumbprint -AppId $AppId -Organization $TenantDomain -Productnames $ProductName -OutPath . -M365Environment $M365Environment -Quiet
+            Invoke-SCuBA -CertificateThumbPrint $Thumbprint -AppId $AppId -Organization $TenantDomain -Productnames $ProductName -OutPath . -M365Environment $M365Environment -Quiet -KeepIndividualJSON
         }
         else {
-            Invoke-SCuBA -Login $false -Productnames $ProductName -OutPath . -M365Environment $M365Environment -Quiet
+            Invoke-SCuBA -Login $false -Productnames $ProductName -OutPath . -M365Environment $M365Environment -Quiet -KeepIndividualJSON
         }
     }
 }
@@ -241,7 +241,7 @@ Describe "Policy Checks for <ProductName>"{
 
                 Set-Content -Path $TestConfigFilePath -Value ($ScubaConfig | ConvertTo-Yaml)
                 SetConditions -Conditions $Preconditions.ToArray()
-                Invoke-SCuBA -ConfigFilePath $TestConfigFilePath -Quiet
+                Invoke-SCuBA -ConfigFilePath $TestConfigFilePath -Quiet -KeepIndividualJSON
             }
             # Ensure case matches driver in test plan
             elseif ('RunScuba' -eq $TestDriver){
@@ -256,7 +256,7 @@ Describe "Policy Checks for <ProductName>"{
                 $ReportFolders = Get-ChildItem . -directory -Filter "M365BaselineConformance*" | Sort-Object -Property LastWriteTime -Descending
                 $OutputFolder = $ReportFolders[0].Name
                 SetConditions -Conditions $Preconditions.ToArray() -OutputFolder $OutputFolder
-                Invoke-SCuBACached -Productnames $ProductName -ExportProvider $false -OutPath $OutputFolder -OutProviderFileName 'ModifiedProviderSettingsExport' -Quiet
+                Invoke-SCuBACached -Productnames $ProductName -ExportProvider $false -OutPath $OutputFolder -OutProviderFileName 'ModifiedProviderSettingsExport' -Quiet -KeepIndividualJSON
             }
             else {
                 Write-Debug "Driver: $TestDriver"

diff --git a/Testing/Functional/SmokeTest/SmokeTest001.Tests.ps1 b/Testing/Functional/SmokeTest/SmokeTest001.Tests.ps1
@@ -51,11 +51,11 @@ Describe "Smoke Test: Generate Output" {
     Context "Invoke Scuba for $Organization" {
         BeforeAll {
             if ($PSCmdlet.ParameterSetName -eq 'Manual'){
-                { Invoke-SCuBA -ProductNames "*" -M365Environment $M365Environment -Quiet} |
+                { Invoke-SCuBA -ProductNames "*" -M365Environment $M365Environment -Quiet -KeepIndividualJSON} |
                 Should -Not -Throw
             }
             else {
-                { Invoke-SCuBA -CertificateThumbprint $Thumbprint -AppID $AppId -Organization $Organization -ProductNames "*" -M365Environment $M365Environment -Quiet} |
+                { Invoke-SCuBA -CertificateThumbprint $Thumbprint -AppID $AppId -Organization $Organization -ProductNames "*" -M365Environment $M365Environment -Quiet -KeepIndividualJSON} |
                 Should -Not -Throw
             }
             $ReportFolders = Get-ChildItem . -directory -Filter "M365BaselineConformance*" | Sort-Object -Property LastWriteTime -Descending

diff --git a/docs/configuration/parameters.md b/docs/configuration/parameters.md
@@ -160,9 +160,9 @@ The list of acceptable values are:
 | Government cloud tenants (high) | gcchigh    |
 | Department of Defense tenants   | dod        |
 
-## MergeJson
+## KeepIndividualJSON
 
-**MergeJson** combines the individual JSON files (named `TeamsReport.json`) in the `IndividualReports` folder together with the `ProviderSettingsExport.json` into an uber JSON file named `ScubaResults.json`. The individual JSON files are deleted.
+**KeepIndividualJSON** Keeps the individual JSON files (e.g., `TeamsReport.json`) in the `IndividualReports` folder along with `ProviderSettingsExport.json` without combining the results in to one uber JSON file named the `ScubaResults.json`. The parameter is for backwards compatibility with older versions of ScubaGear.
 
 | Parameter   | Value  |
 |-------------|--------|
@@ -172,9 +172,9 @@ The list of acceptable values are:
 | Config File | No     |  
 
 ```powershell
-# Create a merged JSON file
+# Outputs legacy ScubaGear individual JSON output 
 Invoke-SCuBA -ProductNames teams `
-  -MergeJson
+  -KeepIndividualJSON
 ```
 
 ## OPAPath
@@ -238,7 +238,7 @@ Invoke-SCuBA -ProductNames teams `
 
 ## OutJsonFileName
 
-**OutJsonFileName** renames the uber JSON file that is created if the [MergeJson](#mergejson) parameter is used. This should only be the base file name, as the extension `.json` will automatically be added. 
+**OutJsonFileName** renames the uber output JSON file that is created after a ScubaGear run. This should only be the base file name, as the extension `.json` will automatically be added. 
 
 | Parameter   | Value              |
 |-------------|--------------------|
@@ -247,13 +247,12 @@ Invoke-SCuBA -ProductNames teams `
 | Default     | `ScubaResults.json` |
 | Config File | No                 |  
 
-> **Note**: This parameter does not work if the `-MergeJson` parameter is not present.
+> **Note**: This parameter does not work if the `-KeepIndividualJSON` parameter is present.
 
 ```powershell
 # Change the output JSON file
 Invoke-SCuBA -ProductNames teams `
-  -OutJsonFileName myresults `
-  -MergeJson
+  -OutJsonFileName myresults
 ```
 
 ## OutCsvFileName

diff --git a/docs/execution/reports.md b/docs/execution/reports.md
@@ -6,8 +6,8 @@ When ScubaGear runs, it creates a new time-stamped subdirectory wherein it will
 |------------------------------|-----------|
 | `IndividualReports`          | This directory contains the detailed reports for each product tested. |
 | `BaselineReports.html`       | This HTML file is a summary of the detailed reports. By default, this file is automatically opened in a web browser after running ScubaGear. |
-| `ProvideSettingsExport.json` | This JSON file contains all of the information that ScubaGear extracted from the products.  A highly-motivated admin might find this useful for understanding how ScubaGear arrived at its results. Only present if ScubaGear is run without the `MergeJson` flag; if run with the `MergeJson` parameter, the contents of this file will be merged into the ScubaResults.json file. |
-| `ScubaResults.json`           | This JSON file encapsulates all ScubaGear output in a format can could be automatically parsed by a downstream system. It contains metadata about the run and the tenant, summary counts of the test results, the test results, and the raw provider output. Only present if ScubaGear is run with the `MergeJson` flag. |
+| `ProvideSettingsExport.json` | This JSON file contains all of the information that ScubaGear extracted from the products.  A highly-motivated admin might find this useful for understanding how ScubaGear arrived at its results. Only present if ScubaGear is run with the `KeepIndividualJson` flag; if run without the `KeepIndividualJSON` parameter, the contents of this file will be merged into the ScubaResults.json file. |
+| `ScubaResults.json`           | This JSON file encapsulates all ScubaGear output in a format that is automatically parsed by a downstream system. It contains metadata about the run and the tenant, summary counts of the test results, the test results, and the raw provider output. Not present if ScubaGear is run with the `KeepIndividualJSON` flag. |
 | `ScubaResults.csv`            | This CSV file contains the test results in a format can could be automatically parsed by a downstream system. Note that this CSV file only contains the results (i.e., the control ID, requirement string, etc.). It does not contain all data contained in the HTML or JSON versions of the output (i.e., the metadata, summary counts, or raw provider output) due to the limitations of CSV files. Additionally, it includes fields where users can document reasons for failures and timelines for remediation, if they so choose. |
 
 You can learn more about setting parameters on the [parameters](../configuration/parameters.md) page.