Azure load tests #1665

somesylvie · 2024-12-18T23:46:36Z

Description

Describe what changed in this PR at a high level.

Issue

#1122

Checklist

I have added tests to cover my changes
I have added logging where useful (with appropriate log level)
I have added JavaDocs where required
I have updated the documentation accordingly

Note: You may remove items that are not applicable

… running in Azure Co-authored-by: somesylvie <[email protected]>

github-actions · 2024-12-18T23:47:31Z

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶 1122 - Partially compliant Fully compliant requirements: Sign Flexion and CDC up for the Locust preview program. Implement load test in internal. Not compliant requirements: Set up a GH Action to run tests on a schedule. Update existing ADR and/or create new one. Update readme with how to run and modify load tests in deployed envs. Configure test criteria (failure thresholds) in Azure. Set up mock RS endpoints for load testing. Rename internal load testing. Set-up load test for staging.
⏱️ Estimated effort to review: 3 🔵🔵🔵⚪⚪
🧪 No relevant tests
🔒 No security concerns identified
⚡ Recommended focus areas for review Thread Safety The use of ApplicationContext.clearThreadRegistrations() needs careful review to ensure thread safety and proper resource cleanup. Conditional Logic The conditional logic for registering MockRSEndpointClient based on the 'Load-Test' header and REPORT_STREAM_URL_PREFIX property should be validated for correctness and security implications.

github-actions · 2024-12-18T23:47:32Z

shared/src/main/java/gov/hhs/cdc/trustedintermediary/context/ApplicationContext.java

+        THREAD_OBJECT_MAP.set(threadObjectMap);
+    }
+
+    public static void clearThreadRegistrations() {


Consider using a more robust thread-local cleanup strategy. The current use of THREAD_OBJECT_MAP.remove() might lead to memory leaks if not all paths that add to the thread-local store also ensure to clear it after use. [important]

github-actions · 2024-12-18T23:47:32Z

etor/src/main/java/gov/hhs/cdc/trustedintermediary/etor/EtorDomainRegistration.java

@@ -226,6 +227,12 @@ protected DomainResponse handleMessageRequest(
        boolean markMetadataAsFailed = false;
        String errorMessage = "";

+        if ("True".equals(request.getHeaders().get("Load-Test"))


Refactor the conditional logic for registering the MockRSEndpointClient to ensure it's only applied under the correct conditions and does not interfere with production settings. [important]

github-actions · 2024-12-18T23:47:32Z

operations/locustfile.py

@@ -118,6 +121,10 @@ def test_start(environment):

 @events.quitting.add_listener
 def assert_stats(environment):
+    if in_azure:


Ensure that the environment-specific paths and configurations, such as file paths and environment variables, are documented and validated to prevent runtime errors in different environments. [medium]

github-actions · 2024-12-18T23:49:18Z

PR Code Suggestions ✨

Explore these optional code suggestions:

Category	Suggestion	Score
Possible issue	Ensure thread-local registrations are always cleared after use to prevent memory leaks or context pollution Ensure that the `ApplicationContext.registerForThread` method is called within a try-finally block to guarantee that `ApplicationContext.clearThreadRegistrations` is called even if an exception occurs, preventing potential memory leaks or incorrect context data during subsequent operations. etor/src/main/java/gov/hhs/cdc/trustedintermediary/etor/EtorDomainRegistration.java [230-234] if ("True".equals(request.getHeaders().get("Load-Test")) && ApplicationContext.isPropertyPresent("REPORT_STREAM_URL_PREFIX")) { - ApplicationContext.registerForThread( - RSEndpointClient.class, MockRSEndpointClient.getInstance()); + try { + ApplicationContext.registerForThread( + RSEndpointClient.class, MockRSEndpointClient.getInstance()); + // processing code here + } finally { + ApplicationContext.clearThreadRegistrations(); + } } Suggestion importance[1-10]: 8 Why: The suggestion to use a try-finally block for thread-local registrations is crucial for ensuring that resources are cleaned up properly, especially in a multi-threaded environment. This prevents memory leaks and ensures that no stale or incorrect context data persists across different operations, which could lead to hard-to-track bugs.	8

basiliskus · 2025-01-03T18:06:56Z

etor/src/main/java/gov/hhs/cdc/trustedintermediary/etor/EtorDomainRegistration.java

@@ -226,6 +227,12 @@ protected DomainResponse handleMessageRequest(
        boolean markMetadataAsFailed = false;
        String errorMessage = "";

+        if ("True".equals(request.getHeaders().get("load-test"))


Would it be better to do something like Boolean.parseBoolean(request.getHeaders().get("load-test"))? This handles casing and null safety

I like that.

saquino0827 · 2025-01-03T20:44:05Z

etor/src/main/java/gov/hhs/cdc/trustedintermediary/etor/EtorDomainRegistration.java

@@ -173,6 +174,12 @@ DomainResponse handleResults(DomainRequest request) {
    }

    DomainResponse handleMetadata(DomainRequest request) {
+        if (Boolean.parseBoolean(request.getHeaders().get("load-test"))
+                && ApplicationContext.isPropertyPresent("REPORT_STREAM_URL_PREFIX")) {


Is Application Context is set when we build the app?

I'm having a hard time understanding your question here. Can you rephrase it?

saquino0827 · 2025-01-03T20:48:30Z

operations/locustfile.py

-        auth_token = f.read()
+
+    # TODO - notes/clarification on 2 different creds, plus expiration date of jwt
+    # TODO - do we want to TF the tests? If yes which envs? In CDC envs, may need to adjust IP allow list on app. Also set as private endpoints in test config?


This is a good question. My assumption is that we don't want load test bombarding CDC environments unless they're okay with that. If they aren't, this could just be an internal running load test. However, I feel if we only run it in internal, it loses value since that would mean we're only running these until we pass the application off to the CDC.

somesylvie · 2025-01-03T22:07:29Z

etor/src/main/java/gov/hhs/cdc/trustedintermediary/etor/EtorDomainRegistration.java

@@ -226,6 +233,12 @@ protected DomainResponse handleMessageRequest(
        boolean markMetadataAsFailed = false;
        String errorMessage = "";

+        if (Boolean.parseBoolean(request.getHeaders().get("load-test"))


Since any new places that call RS will also need to copy this code, we should document that (probably both in a comment here and the other location we evaluate this, plus in docs)

Co-Authored-By: Jeremy Rosenfeld <[email protected]> Co-Authored-By: halprin <[email protected]>

…intermediary into azure-load-tests

sonarqubecloud · 2025-01-04T00:05:03Z

Quality Gate passed

Issues
1 New issue
0 Accepted issues

Measures
0 Security Hotspots
37.5% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

halprin and others added 4 commits December 16, 2024 16:36

Locust load test works in Azure

9c64c15

Use an env var to tell whether running in Azure

6ef84a2

Add some TODOs and update comment on why we avoid failing locust when…

e90565c

… running in Azure Co-authored-by: somesylvie <[email protected]>

Play with doing thread local registrations in the ApplicationContext

790abd7

github-actions bot reviewed Dec 18, 2024

View reviewed changes

halprin added 3 commits December 18, 2024 20:58

Make Inject work on the per thread injects

0f55439

Merge branch 'main' into azure-load-tests

34a783a

Add some comments and delete some commented out code

54cfc1b

halprin temporarily deployed to internal January 3, 2025 17:51 — with GitHub Actions Inactive

halprin temporarily deployed to internal January 3, 2025 17:53 — with GitHub Actions Inactive

halprin temporarily deployed to internal January 3, 2025 17:55 — with GitHub Actions Inactive

Pass in the Load-Test header in the load tests

7f8b16b

basiliskus reviewed Jan 3, 2025

View reviewed changes

Check for load tests on the metadata endpoint too

7896fd0

halprin temporarily deployed to internal January 3, 2025 19:01 — with GitHub Actions Inactive

halprin temporarily deployed to internal January 3, 2025 19:03 — with GitHub Actions Inactive

halprin deployed to internal January 3, 2025 19:05 — with GitHub Actions View deployment

pass load-test header to metadata endpoint for load tests

022e9b5

saquino0827 reviewed Jan 3, 2025

View reviewed changes

somesylvie commented Jan 3, 2025

View reviewed changes

halprin and others added 5 commits January 3, 2025 15:58

Add additional code comments about the thread specific implementations

8be024f

Remove old debug logging

fcd9c16

Started Azure Load Test Action creation

be3a898

Co-Authored-By: Jeremy Rosenfeld <[email protected]> Co-Authored-By: halprin <[email protected]>

Merge branch 'azure-load-tests' of https://github.com/CDCgov/trusted-…

9a013b0

…intermediary into azure-load-tests

add permissions block to workflow

565d2b6

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Azure load tests #1665

Azure load tests #1665

somesylvie commented Dec 18, 2024

github-actions bot commented Dec 18, 2024

github-actions bot Dec 18, 2024

github-actions bot Dec 18, 2024

github-actions bot Dec 18, 2024

github-actions bot commented Dec 18, 2024

basiliskus Jan 3, 2025

halprin Jan 3, 2025

saquino0827 Jan 3, 2025

halprin Jan 3, 2025

saquino0827 Jan 3, 2025

somesylvie Jan 3, 2025

sonarqubecloud bot commented Jan 4, 2025

Azure load tests #1665

Are you sure you want to change the base?

Azure load tests #1665

Conversation

somesylvie commented Dec 18, 2024

Description

Issue

Checklist

github-actions bot commented Dec 18, 2024

PR Reviewer Guide 🔍

github-actions bot Dec 18, 2024

Choose a reason for hiding this comment

github-actions bot Dec 18, 2024

Choose a reason for hiding this comment

github-actions bot Dec 18, 2024

Choose a reason for hiding this comment

github-actions bot commented Dec 18, 2024

PR Code Suggestions ✨

basiliskus Jan 3, 2025

Choose a reason for hiding this comment

halprin Jan 3, 2025

Choose a reason for hiding this comment

saquino0827 Jan 3, 2025

Choose a reason for hiding this comment

halprin Jan 3, 2025

Choose a reason for hiding this comment

saquino0827 Jan 3, 2025

Choose a reason for hiding this comment

somesylvie Jan 3, 2025

Choose a reason for hiding this comment

sonarqubecloud bot commented Jan 4, 2025

Quality Gate passed