4.x: Concurrency limits module, and support in Helidon WebServer #9295

tomas-langer · 2024-09-27T13:39:30Z

Description

Resolves #8897
Resolves #9229

Documentation

This PR introduces

a new module common/concurrency/limits that provides API and SPI for concurrency limit implementations, and a couple of default implementations (AIMD, Fixed).
a new module webserver/concurrency-limits that provides feature with a filter to impose limits within a filter in routing
update of webserver/webserver to use a Limit instead of a Semaphore in connection handlers (backward compatible)

Configuration reference will be in the generated documentation.

Configure limits on WebServer

This will configure limit for a server listener (configurable per listener), enforced on the connection level (i.e. outside of routing and filters).

The default behavior is the same (unlimited).
If server.max-concurrent-requests is configured (value is not -1), it will be used and concurrency-limit configuration for the listener will be ignored.

Configuration:

server:
  port: 8080
  concurrency-limit:
    aimd: # `limit type`
      # AIMD limit configuration

Configure limits for routing

This will configure limit for as a server feature, enforced in an HTTP filter.
Configuration:

server:
  features:
    limits: # the feature is called `limits`
      enabled: true
      concurrency-limit: # `limit` configuration of the `limits` server feature
        fixed: # `limit type`
          permits: 1
          queue-length: 10

Signed-off-by: Tomas Langer <[email protected]> Co-authored-by: André Rouél <[email protected]> Signed-off-by: Tomas Langer <[email protected]>

Signed-off-by: Tomas Langer <[email protected]>

arouel

LGTM, great rework and integration

vasanth-bhat · 2024-09-30T04:14:09Z

Have few clarifications

Going through Readme , there is mention about 'Semaphore' concurrency limit type in addition to aimd & BulkHead. Is this same as. "basic" (BasicLimit)? Also how is this different from specifying. "server.max-concurrent-requests"?
For the BulkHead, it would be good to have metrics support , around queueing , for example % queue full. This is important in production to monitor the queueing.
For AMID , it would be good to have a metrics around the dynamic concurrency limit in use. A Histogram may be useful here.

tomas-langer · 2024-09-30T15:57:18Z

After discussing the needs in Helidon, we must do this using the permit approach, so this can be reused in webclient.
I will refactor the PR.

arouel · 2024-09-30T18:39:47Z

I wanted to bring up that metrics would be super important for production as @vasanth-bhat brought it up as well. Within my draft I figured quickly that the limiting works as expected but metrics would give me the insights I need to build convidence. Maybe this is something for a follow-up Pull Request, idk.

Signed-off-by: Tomas Langer <[email protected]>

tomas-langer · 2024-09-30T19:45:12Z

Metrics must be a follow up, as it would require much more work. I cannot just add a dependency on helidon-metrics-api, as it would introduce it everywhere (maybe that is the solution, but it requires a bit more thought).
I will finish this PR as the introduction of concurrency limits. We can look into the metrics as a follow up (this would make sense for fault tolerance as well, not just concurrency limits).

Signed-off-by: Tomas Langer <[email protected]>

tomas-langer · 2024-09-30T19:49:20Z

Pushed a new version, updated original description to reflect new implementation.
I have implemented the fixed limit as a more general feature - it supports queing now, so we no longer need fault tolerance for this.
How to add this to metrics needs to be a follow up.

tomas-langer · 2024-09-30T19:57:42Z

Created a follow up issue: #9304

Signed-off-by: Tomas Langer <[email protected]>

vasanth-bhat · 2024-10-01T03:22:07Z

Went through the new changes.

I think it would help to clarify between the below 2 , with example to understand the difference between the two. Do both them provide same functionality but via different approaches ?

Fixed Limit with queueing support at the connection level (or a server listener )
Fixed limit with queueing as server feature enforced via filter.

Verdent · 2024-10-07T12:32:07Z

I think we should have also some method, which ignores waiting to obtain the token, if wait period is configured. Such as tryAcquire(boolean wait).

Justification
If we try reuse objects which are counted via Limit. Lets say all of these reusable objects are in use by other threads and we know this Limit will not have any tokens unblocked, since they are reusable. Currently we have to wait the whole timeout. It would probably be more beneficial to just ignore timeout in this case, since we can move and wait on the cache directly

…onfiguration. Added support to bypass queuing and return immediately. Signed-off-by: Tomas Langer <[email protected]>

arouel · 2024-10-08T16:37:31Z

webserver/http2/src/main/java/io/helidon/webserver/http2/Http2ServerStream.java

+                    try {
+                        routing.route(ctx, request, response);
+                    } finally {
+                        if (response.status() == Status.NOT_FOUND_404) {


Maybe something for a future PR: Should we allow a user to specify which 4xx codes can be ignored?

arouel · 2024-10-08T16:39:47Z

webserver/webserver/src/main/java/io/helidon/webserver/http1/Http1Connection.java

+                        permit.success();
+                        this.lastRequestTimestamp = DateTime.timestamp();
+                    } catch (Throwable e) {
+                        permit.dropped();


Maybe something for a future PR: Should we allow a user to specify which exception can be recorded as ignored instead of being counted as dropped?

common/concurrency/limits/src/main/java/io/helidon/common/concurrency/limits/AimdLimitImpl.java

common/concurrency/limits/src/main/java/io/helidon/common/concurrency/limits/FixedLimit.java

Signed-off-by: Tomas Langer <[email protected]>

…idon-io#9295) * Concurrency limits module, and support in Helidon WebServer * Align configuration key for server feature and server. * Refactored to use tokens. * Added tests for configuration based limits. * Added copy to a limit, so we can get another instance with the same configuration. * Added support to bypass queuing and return immediately. Signed-off-by: Tomas Langer <[email protected]> Co-authored-by: André Rouél <[email protected]> Signed-off-by: Tomas Langer <[email protected]>

…9384) * 4.x: Concurrency limits module, and support in Helidon WebServer (#9295) * Concurrency limits module, and support in Helidon WebServer * Align configuration key for server feature and server. * Refactored to use tokens. * Added tests for configuration based limits. * Added copy to a limit, so we can get another instance with the same configuration. * Added support to bypass queuing and return immediately. Signed-off-by: Tomas Langer <[email protected]> Co-authored-by: André Rouél <[email protected]> Signed-off-by: Tomas Langer <[email protected]> --------- Signed-off-by: Tomas Langer <[email protected]> Co-authored-by: Tomas Langer <[email protected]> Co-authored-by: André Rouél <[email protected]>

tomas-langer self-assigned this Sep 27, 2024

oracle-contributor-agreement bot added the OCA Verified All contributors have signed the Oracle Contributor Agreement. label Sep 27, 2024

tomas-langer and others added 4 commits September 27, 2024 16:29

Concurrency limits module, and support in Helidon WebServer

f7e0dad

Signed-off-by: Tomas Langer <[email protected]> Co-authored-by: André Rouél <[email protected]> Signed-off-by: Tomas Langer <[email protected]>

Align configuration key for server feature and server.

5065977

Signed-off-by: Tomas Langer <[email protected]>

Fix configuration metadata docs

64e7c08

Signed-off-by: Tomas Langer <[email protected]>

Update new modules to latest version

cd3282b

Signed-off-by: Tomas Langer <[email protected]>

tomas-langer force-pushed the 8913-concurrency-limits branch from d85a7d0 to cd3282b Compare September 27, 2024 14:32

This was referenced Sep 27, 2024

Provide adaptive concurrency limits #8897

Closed

Provide configurable option to queue requests when concurrency is limited with "max-concurrent-requests" #9229

Closed

tomas-langer requested review from danielkec and romain-grecourt September 27, 2024 14:40

arouel approved these changes Sep 27, 2024

View reviewed changes

Refactored to use tokens.

d8ed947

Signed-off-by: Tomas Langer <[email protected]>

Added tests for configuration based limits.

5a0e844

Signed-off-by: Tomas Langer <[email protected]>

tomas-langer requested a review from arouel September 30, 2024 19:51

tomas-langer added 2 commits September 30, 2024 22:00

Fixed dependency.

8e992ea

Signed-off-by: Tomas Langer <[email protected]>

Test fix (intermittent failures)

2135b38

Signed-off-by: Tomas Langer <[email protected]>

tomas-langer added this to the 4.1.3 milestone Oct 7, 2024

Added copy to a limit, so we can get another instance with the same c…

8ab46dc

…onfiguration. Added support to bypass queuing and return immediately. Signed-off-by: Tomas Langer <[email protected]>

arouel approved these changes Oct 8, 2024

View reviewed changes

danielkec reviewed Oct 8, 2024

View reviewed changes

common/concurrency/limits/src/main/java/io/helidon/common/concurrency/limits/AimdLimitImpl.java Outdated Show resolved Hide resolved

common/concurrency/limits/src/main/java/io/helidon/common/concurrency/limits/FixedLimit.java Outdated Show resolved Hide resolved

Review fixes.

210fd71

Signed-off-by: Tomas Langer <[email protected]>

tomas-langer requested a review from danielkec October 9, 2024 09:15

danielkec approved these changes Oct 9, 2024

View reviewed changes

barchetta mentioned this pull request Oct 9, 2024

4.1.3 backports #9341

Closed

16 tasks

Verdent approved these changes Oct 10, 2024

View reviewed changes

tomas-langer merged commit f39decf into helidon-io:main Oct 14, 2024
44 checks passed

tomas-langer deleted the 8913-concurrency-limits branch October 14, 2024 08:57

barchetta mentioned this pull request Oct 14, 2024

4.1.3: Concurrency limits module, and support in Helidon WebServer #9384

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

4.x: Concurrency limits module, and support in Helidon WebServer #9295

4.x: Concurrency limits module, and support in Helidon WebServer #9295

tomas-langer commented Sep 27, 2024 •

edited

Loading

arouel left a comment

vasanth-bhat commented Sep 30, 2024 •

edited

Loading

tomas-langer commented Sep 30, 2024

arouel commented Sep 30, 2024

tomas-langer commented Sep 30, 2024

tomas-langer commented Sep 30, 2024

tomas-langer commented Sep 30, 2024

vasanth-bhat commented Oct 1, 2024

Verdent commented Oct 7, 2024

arouel Oct 8, 2024 •

edited

Loading

arouel Oct 8, 2024

4.x: Concurrency limits module, and support in Helidon WebServer #9295

4.x: Concurrency limits module, and support in Helidon WebServer #9295

Conversation

tomas-langer commented Sep 27, 2024 • edited Loading

Description

Documentation

Configure limits on WebServer

Configure limits for routing

arouel left a comment

Choose a reason for hiding this comment

vasanth-bhat commented Sep 30, 2024 • edited Loading

tomas-langer commented Sep 30, 2024

arouel commented Sep 30, 2024

tomas-langer commented Sep 30, 2024

tomas-langer commented Sep 30, 2024

tomas-langer commented Sep 30, 2024

vasanth-bhat commented Oct 1, 2024

Verdent commented Oct 7, 2024

arouel Oct 8, 2024 • edited Loading

Choose a reason for hiding this comment

arouel Oct 8, 2024

Choose a reason for hiding this comment

tomas-langer commented Sep 27, 2024 •

edited

Loading

vasanth-bhat commented Sep 30, 2024 •

edited

Loading

arouel Oct 8, 2024 •

edited

Loading