Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

OSOE-351: Offer security checks in Lombiq.UITestingToolbox #614

Merged
merged 74 commits into from
Dec 5, 2023
Merged
Show file tree
Hide file tree
Changes from 62 commits
Commits
Show all changes
74 commits
Select commit Hold shift + click to select a range
70f8c22
Prototype ZAP security scanning
Piedone Nov 14, 2023
0d15a1d
Minimal CI build
Piedone Nov 14, 2023
a5279b9
Test run of security scans
Piedone Nov 14, 2023
43cd4a5
Even leaner CI build
Piedone Nov 14, 2023
180c561
Fixing ZAP network access under Linux
Piedone Nov 14, 2023
28dffac
ZAP-using test now passes
Piedone Nov 14, 2023
5d4e437
Lean Windows build
Piedone Nov 14, 2023
040d25b
Even leaner Windows build
Piedone Nov 14, 2023
4415157
Windows builds now exclude security tests since those won't run on Gi…
Piedone Nov 14, 2023
99fcb1e
Temporarily disabling code analysis for Windows builds too
Piedone Nov 14, 2023
b7db6ca
Revert "Temporarily disabling code analysis for Windows builds too"
Piedone Nov 15, 2023
5735311
Adding support for ZAP YAML config files
Piedone Nov 15, 2023
ad0a4a5
Ability to assert on SARIF security reports
Piedone Nov 15, 2023
fe10039
Making SecurityScanShouldPass pass
Piedone Nov 15, 2023
30deed4
Refactoring and docs
Piedone Nov 16, 2023
76ea893
Simpler sample assertion in SecurityScanningTests
Piedone Nov 16, 2023
1ee97a2
Removing spiderAjax from the default plans, more configurability
Piedone Nov 16, 2023
33d3a9d
Second security scanning sample test with more configurability
Piedone Nov 16, 2023
aa815f9
Small improvements, docs
Piedone Nov 17, 2023
b6ed30b
Configurability for disabling a rule just for a single URL
Piedone Nov 19, 2023
0ffbfe9
Adding simplified fluent configuration
Piedone Nov 20, 2023
f418fbf
ZAP scans can now log in
Piedone Nov 21, 2023
e77c0ba
Low-level config sample, docs, refactoring
Piedone Nov 21, 2023
a65033f
Docs
Piedone Nov 21, 2023
d0b4995
Merge remote-tracking branch 'origin/dev' into issue/OSOE-351
Piedone Nov 21, 2023
53a387d
Code styling
Piedone Nov 21, 2023
d7f7968
Code styling
Piedone Nov 21, 2023
ac7b9c7
Merge remote-tracking branch 'origin/dev' into issue/OSOE-351
Piedone Nov 22, 2023
24e24b9
Excluding irrelevant technologies from ZAP scans, making them faster
Piedone Nov 22, 2023
d1f26a1
Merge remote-tracking branch 'origin/issue/OSOE-733' into issue/OSOE-351
Piedone Nov 22, 2023
3c5d227
Removing now unnecessary browser configs
Piedone Nov 22, 2023
63b6d11
Resetting debug code
Piedone Nov 23, 2023
1650ef2
Docs, spelling
Piedone Nov 23, 2023
3b571ab
More spelling
Piedone Nov 23, 2023
3ed2d38
Updating the NuGetTest solution too
Piedone Nov 23, 2023
8152baf
Running the NuGetTest also on BuildJet to test Docker
Piedone Nov 23, 2023
44c105b
Testing SecurityScanningTests on GitHub-hosted runner
Piedone Nov 23, 2023
a0d2d2c
Leaner build
Piedone Nov 23, 2023
8971049
Trying to fix report creation issue under GitHub-hosted GHA runners
Piedone Nov 23, 2023
679ff2c
Another attempt to fix ZAP report creation issue under GitHub-hosted …
Piedone Nov 23, 2023
26da24a
Another attempt to fix report creation issue under GitHub-hosted GHA …
Piedone Nov 23, 2023
d456935
Attempting to fix test temp directory cleanup failing due to somethin…
Piedone Nov 23, 2023
f4117a7
How about chmod a+x?
Piedone Nov 23, 2023
eef8241
Trying to restore the original permissions of the reports folder
Piedone Nov 23, 2023
4ac0fbb
Workarounds for making the clean-up succeed under GHA
Piedone Nov 23, 2023
b0fb2e0
Fixing restoring the original folder permission
Piedone Nov 23, 2023
fe41747
Removing leftover directory deletion
Piedone Nov 23, 2023
075e5dd
Adding debug code to see if the chmod alone breaks clean-up
Piedone Nov 23, 2023
8825edc
Removing chmod to see if anything else breaks clean-up
Piedone Nov 23, 2023
18cb1fe
Intentionally failing security scan to test artifacts
Piedone Nov 23, 2023
2619958
Removing MaxRetryCount = 0
Piedone Nov 23, 2023
9d9e208
Removing intentional test fail
Piedone Nov 23, 2023
a4ffd0e
Restoring original CI workflow
Piedone Nov 23, 2023
c165996
Fixing and simplifying SQL Server Docker CLI calls
Piedone Nov 23, 2023
242f2ee
Adding script to display display the runtime of scan rules
Piedone Nov 24, 2023
604b13b
Spelling
Piedone Nov 24, 2023
f12827b
Merge remote-tracking branch 'origin/issue/OSOE-733' into issue/OSOE-351
Piedone Nov 24, 2023
c30d836
Merge remote-tracking branch 'origin/dev' into issue/OSOE-351
Piedone Nov 26, 2023
56019ba
Merge remote-tracking branch 'origin/dev' into issue/OSOE-351
Piedone Nov 27, 2023
fe048bd
Updating Lombiq.Tests.UI* NuGet references
Piedone Nov 27, 2023
ae5ca84
Excluding SecurityScanningTests from Windows builds for the NuGet sol…
Piedone Nov 27, 2023
26be931
Merge remote-tracking branch 'origin/dev' into issue/OSOE-351
Piedone Nov 27, 2023
6b7872e
Merge remote-tracking branch 'origin/dev' into issue/OSOE-351
Piedone Nov 30, 2023
a79f97b
Comment on why SecurityScanningTests is excluded under Windows CI runs
Piedone Nov 30, 2023
dc692d3
NuGetTest docs
Piedone Nov 30, 2023
f165802
Docs, cleaning up the ZAP container after completion, simplification
Piedone Nov 30, 2023
01b28ab
Updating NuGetTest too
Piedone Nov 30, 2023
4c87e8d
Merge remote-tracking branch 'origin/dev' into issue/OSOE-351
Piedone Dec 1, 2023
c42f76c
Ability to configure active scan rules to optimize "Cross Site Script…
Piedone Dec 2, 2023
7ffba26
Fixing scan config
Piedone Dec 2, 2023
a5f6bc9
MD, C# linting fixes
Piedone Dec 2, 2023
c23a336
Fixing MD indentation again
Piedone Dec 2, 2023
4fbd5b3
Code styling and docs
Piedone Dec 4, 2023
c45b826
Releasing the UI Testing Toolbox
Piedone Dec 5, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/actions/spelling/allow.txt
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@ Overconstrained
parallelizable
qrcode
retriable
Runtimes
sanitizers
scrollbars
shortcutting
Expand Down
3 changes: 3 additions & 0 deletions .github/actions/spelling/excludes.txt
Original file line number Diff line number Diff line change
@@ -1,6 +1,9 @@
Assets/Vendors/
\QLombiq.VueJs.Tests.UI/Assets/Media/\E.*\.mjpeg$
^\Qtest/Lombiq.UITestingToolbox/Lombiq.Tests.UI/Constants/CommonDisplayResolutions.cs\E$
^\Qtest/Lombiq.UITestingToolbox/Lombiq.Tests.UI/SecurityScanning/AutomationFrameworkPlanFragments/\E.*.yml$
^\Qtest/Lombiq.UITestingToolbox/Lombiq.Tests.UI/SecurityScanning/AutomationFrameworkPlans/\E.*.yml$
^\Qtest/Lombiq.UITestingToolbox/Lombiq.Tests.UI.Samples/Tests/CustomZapAutomationFrameworkPlan.yml\E$
^\Qtools/Lombiq.GitHub.Actions/\E
\QUnmanagedNodeModules/\E
\QUploadingTestFileDOCX.docx\E$
Expand Down
3 changes: 3 additions & 0 deletions .github/workflows/build-and-test-windows.yml
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,7 @@ jobs:
ui-test-parallelism: 0
build-create-binary-log: "true"
blame-hang-timeout: "5m"
test-filter: "FullyQualifiedName!~SecurityScanningTests"

build-and-test-standard-runners:
# Since dev builds are not awaited by anyone, they can run on the slower free runners.
Expand All @@ -55,6 +56,7 @@ jobs:
set-up-azurite: "true"
build-create-binary-log: "true"
blame-hang-timeout: "5m"
test-filter: "FullyQualifiedName!~SecurityScanningTests"

build-and-test-nuget-test:
if: github.ref_name == github.event.repository.default_branch ||
Expand All @@ -67,6 +69,7 @@ jobs:
build-directory: NuGetTest
timeout-minutes: 25
blame-hang-timeout: "5m"
test-filter: "FullyQualifiedName!~SecurityScanningTests"

powershell-static-code-analysis:
if: github.ref_name == github.event.repository.default_branch ||
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -47,8 +47,8 @@
<PackageReference Include="Lombiq.Privacy" Version="7.0.2-alpha.2.osoe-638" />
<PackageReference Include="Lombiq.Privacy.Samples" Version="7.0.2-alpha.2.osoe-638" />
<PackageReference Include="Lombiq.SetupExtensions" Version="5.0.0" />
<PackageReference Include="Lombiq.Tests.UI.AppExtensions" Version="8.2.1-alpha.1.osoe-733" />
<PackageReference Include="Lombiq.Tests.UI.Shortcuts" Version="8.2.1-alpha.1.osoe-733" />
<PackageReference Include="Lombiq.Tests.UI.AppExtensions" Version="8.2.1-alpha.4.osoe-351" />
<PackageReference Include="Lombiq.Tests.UI.Shortcuts" Version="8.2.1-alpha.4.osoe-351" />
<PackageReference Include="Lombiq.UIKit" Version="6.0.1-alpha.0.osoe-638" />
<PackageReference Include="Lombiq.VueJs" Version="3.0.1-alpha.0.osoe-638" />
<PackageReference Include="Lombiq.VueJs.Samples" Version="3.0.1-alpha.0.osoe-638" />
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@
</ItemGroup>

<ItemGroup>
<PackageReference Include="Lombiq.HelpfulLibraries" Version="8.0.2" />
<PackageReference Include="Lombiq.HelpfulLibraries" Version="8.0.3-alpha.2.osoe-351" />
<PackageReference Include="OrchardCore.Module.Targets" Version="1.7.0" />
<PackageReference Include="OrchardCore.ContentManagement" Version="1.7.0" />
<PackageReference Include="OrchardCore.ContentTypes.Abstractions" Version="1.7.0" />
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@
<PackageReference Include="Lombiq.OrchardCoreApiClient.Tests.UI" Version="4.0.1-alpha.0.osoe-638" />
<PackageReference Include="Lombiq.Privacy.Tests.UI" Version="7.0.2-alpha.2.osoe-638" />
<PackageReference Include="Lombiq.HelpfulExtensions.Tests.UI" Version="7.0.2-alpha.2.osoe-683" />
<PackageReference Include="Lombiq.Tests.UI" Version="8.2.1-alpha.1.osoe-733" />
<PackageReference Include="Lombiq.Tests.UI" Version="8.2.1-alpha.4.osoe-351" />
<PackageReference Include="Lombiq.VueJs.Tests.UI" Version="3.0.1-alpha.0.osoe-638" />
<PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.7.2" />
<PackageReference Include="xunit.runner.visualstudio" Version="2.5.1">
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
using Lombiq.Tests.UI.SecurityScanning;
using Shouldly;
using System.Threading.Tasks;
using Xunit;
using Xunit.Abstractions;

namespace Lombiq.OSOCE.NuGet.Tests.UI.Tests;

public class SecurityScanningTests : UITestBase
{
public SecurityScanningTests(ITestOutputHelper testOutputHelper)
: base(testOutputHelper)
{
}

// Only scanning the homepage, since this is just to make sure that ZAP still works from NuGet.
[Fact]
public Task BasicSecurityScanShouldPass() =>
ExecuteTestAfterSetupAsync(
context => context.RunAndAssertBaselineSecurityScanAsync(
configuration => configuration.ExcludeUrlWithRegex(".*:[0-9]+\\/.+"),
BenedekFarkas marked this conversation as resolved.
Show resolved Hide resolved
sarifLog => sarifLog.Runs[0].Results.Count.ShouldBeLessThan(6)));
BenedekFarkas marked this conversation as resolved.
Show resolved Hide resolved
}
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
</PropertyGroup>

<ItemGroup>
<PackageReference Include="Lombiq.HelpfulLibraries" Version="8.0.2" />
<PackageReference Include="Lombiq.HelpfulLibraries" Version="8.0.3-alpha.2.osoe-351" />
<PackageReference Include="Lombiq.Tests" Version="2.2.5" />
<PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.7.2" />
<PackageReference Include="xunit.runner.visualstudio" Version="2.5.1">
Expand Down
2 changes: 1 addition & 1 deletion test/Lombiq.UITestingToolbox
Submodule Lombiq.UITestingToolbox updated 43 files
+4 −0 Lombiq.Tests.UI.Samples/Lombiq.Tests.UI.Samples.csproj
+1 −0 Lombiq.Tests.UI.Samples/Readme.md
+114 −0 Lombiq.Tests.UI.Samples/Tests/CustomZapAutomationFrameworkPlan.yml
+1 −0 Lombiq.Tests.UI.Samples/Tests/InteractiveModeTests.cs
+158 −0 Lombiq.Tests.UI.Samples/Tests/SecurityScanningTests.cs
+1 −1 Lombiq.Tests.UI.Shortcuts/Controllers/CurrentUserController.cs
+1 −1 Lombiq.Tests.UI.Shortcuts/Lombiq.Tests.UI.Shortcuts.csproj
+ Lombiq.Tests.UI/Docs/Attachments/ZapReportScreenshot.png
+1 −1 Lombiq.Tests.UI/Docs/Configuration.md
+1 −0 Lombiq.Tests.UI/Docs/CreatingTests.md
+28 −0 Lombiq.Tests.UI/Docs/SecurityScanning.md
+1 −0 Lombiq.Tests.UI/Docs/Tools.md
+4 −0 Lombiq.Tests.UI/Docs/Troubleshooting.md
+49 −4 Lombiq.Tests.UI/Extensions/FailureDumpUITestContextExtensions.cs
+17 −11 Lombiq.Tests.UI/Extensions/ShortcutsUITestContextExtensions.cs
+30 −5 Lombiq.Tests.UI/Extensions/TypedRouteUITestContextExtensions.cs
+1 −1 Lombiq.Tests.UI/Helpers/DirectoryHelper.cs
+23 −4 Lombiq.Tests.UI/Lombiq.Tests.UI.csproj
+25 −0 Lombiq.Tests.UI/SecurityScanning/AutomationFrameworkPlanFragments/DisplayActiveScanRuleRuntimesScript.yml
+12 −0 Lombiq.Tests.UI/SecurityScanning/AutomationFrameworkPlanFragments/RequestorJob.yml
+8 −0 Lombiq.Tests.UI/SecurityScanning/AutomationFrameworkPlanFragments/SpiderAjaxJob.yml
+14 −0 Lombiq.Tests.UI/SecurityScanning/AutomationFrameworkPlanFragmentsPaths.cs
+13 −0 Lombiq.Tests.UI/SecurityScanning/AutomationFrameworkPlanPaths.cs
+102 −0 Lombiq.Tests.UI/SecurityScanning/AutomationFrameworkPlans/Baseline.yml
+107 −0 Lombiq.Tests.UI/SecurityScanning/AutomationFrameworkPlans/FullScan.yml
+116 −0 Lombiq.Tests.UI/SecurityScanning/AutomationFrameworkPlans/GraphQL.yml
+107 −0 Lombiq.Tests.UI/SecurityScanning/AutomationFrameworkPlans/OpenAPI.yml
+216 −0 Lombiq.Tests.UI/SecurityScanning/SecurityScanConfiguration.cs
+15 −0 Lombiq.Tests.UI/SecurityScanning/SecurityScanResult.cs
+27 −0 Lombiq.Tests.UI/SecurityScanning/SecurityScanningAssertionException.cs
+36 −0 Lombiq.Tests.UI/SecurityScanning/SecurityScanningConfiguration.cs
+20 −0 Lombiq.Tests.UI/SecurityScanning/SecurityScanningException.cs
+163 −0 Lombiq.Tests.UI/SecurityScanning/SecurityScanningUITestContextExtensions.cs
+405 −0 Lombiq.Tests.UI/SecurityScanning/YamlDocumentExtensions.cs
+15 −0 Lombiq.Tests.UI/SecurityScanning/YamlHelper.cs
+25 −0 Lombiq.Tests.UI/SecurityScanning/YamlNodeExtensions.cs
+244 −0 Lombiq.Tests.UI/SecurityScanning/ZapManager.cs
+3 −0 Lombiq.Tests.UI/Services/OrchardCoreUITestExecutorConfiguration.cs
+1 −1 Lombiq.Tests.UI/Services/SmtpService.cs
+4 −10 Lombiq.Tests.UI/Services/SqlServerManager.cs
+12 −2 Lombiq.Tests.UI/Services/UITestContext.cs
+28 −3 Lombiq.Tests.UI/Services/UITestExecutionSession.cs
+2 −0 Readme.md