[MLOB] add LLM obs configs #8076
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
gary-huang
force-pushed
the
gary/add-llmobs-configs
branch
from
3863387 to
19d2dc3
Compare
BenchmarksStartupParameters
See matching parameters
SummaryFound 1 performance improvements and 0 performance regressions! Performance is the same for 59 metrics, 3 unstable metrics.
Startup time reports for petclinicgantt
title petclinic - global startup overhead: candidate=1.45.0-SNAPSHOT~36b653bf75, baseline=1.45.0-SNAPSHOT~1951e6f72e
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.05 s) : 0, 1049679
Total [baseline] (10.437 s) : 0, 10436757
Agent [candidate] (1.059 s) : 0, 1058669
Total [candidate] (10.492 s) : 0, 10491736
section appsec
Agent [baseline] (1.19 s) : 0, 1189860
Total [baseline] (10.759 s) : 0, 10758581
Agent [candidate] (1.187 s) : 0, 1187093
Total [candidate] (10.706 s) : 0, 10706181
section iast
Agent [baseline] (1.182 s) : 0, 1182248
Total [baseline] (10.982 s) : 0, 10981548
Agent [candidate] (1.18 s) : 0, 1179970
Total [candidate] (11.005 s) : 0, 11005097
section profiling
Agent [baseline] (1.272 s) : 0, 1272405
Total [baseline] (10.806 s) : 0, 10806199
Agent [candidate] (1.279 s) : 0, 1278884
Total [candidate] (10.872 s) : 0, 10871956
gantt
title petclinic - break down per module: candidate=1.45.0-SNAPSHOT~36b653bf75, baseline=1.45.0-SNAPSHOT~1951e6f72e
dateFormat X
axisFormat %s
section tracing
BytebuddyAgent [baseline] (711.574 ms) : 0, 711574
BytebuddyAgent [candidate] (716.686 ms) : 0, 716686
GlobalTracer [baseline] (256.238 ms) : 0, 256238
GlobalTracer [candidate] (256.084 ms) : 0, 256084
AppSec [baseline] (54.935 ms) : 0, 54935
AppSec [candidate] (57.647 ms) : 0, 57647
Remote Config [baseline] (716.676 µs) : 0, 717
Remote Config [candidate] (690.713 µs) : 0, 691
Telemetry [baseline] (11.195 ms) : 0, 11195
Telemetry [candidate] (12.533 ms) : 0, 12533
section appsec
BytebuddyAgent [baseline] (731.755 ms) : 0, 731755
BytebuddyAgent [candidate] (728.969 ms) : 0, 728969
GlobalTracer [baseline] (254.058 ms) : 0, 254058
GlobalTracer [candidate] (252.992 ms) : 0, 252992
AppSec [baseline] (170.665 ms) : 0, 170665
AppSec [candidate] (171.762 ms) : 0, 171762
Remote Config [baseline] (661.998 µs) : 0, 662
Remote Config [candidate] (660.919 µs) : 0, 661
Telemetry [baseline] (8.051 ms) : 0, 8051
Telemetry [candidate] (7.991 ms) : 0, 7991
IAST [baseline] (19.317 ms) : 0, 19317
IAST [candidate] (19.628 ms) : 0, 19628
section iast
BytebuddyAgent [baseline] (831.416 ms) : 0, 831416
BytebuddyAgent [candidate] (831.425 ms) : 0, 831425
GlobalTracer [baseline] (247.215 ms) : 0, 247215
GlobalTracer [candidate] (245.954 ms) : 0, 245954
AppSec [baseline] (58.06 ms) : 0, 58060
AppSec [candidate] (57.568 ms) : 0, 57568
Remote Config [baseline] (648.762 µs) : 0, 649
Remote Config [candidate] (650.09 µs) : 0, 650
Telemetry [baseline] (8.544 ms) : 0, 8544
Telemetry [candidate] (8.422 ms) : 0, 8422
IAST [baseline] (21.331 ms) : 0, 21331
IAST [candidate] (20.969 ms) : 0, 20969
section profiling
ProfilingAgent [baseline] (95.793 ms) : 0, 95793
ProfilingAgent [candidate] (96.427 ms) : 0, 96427
BytebuddyAgent [baseline] (701.831 ms) : 0, 701831
BytebuddyAgent [candidate] (704.768 ms) : 0, 704768
GlobalTracer [baseline] (370.656 ms) : 0, 370656
GlobalTracer [candidate] (373.056 ms) : 0, 373056
AppSec [baseline] (54.021 ms) : 0, 54021
AppSec [candidate] (54.065 ms) : 0, 54065
Remote Config [baseline] (651.572 µs) : 0, 652
Remote Config [candidate] (657.587 µs) : 0, 658
Telemetry [baseline] (7.779 ms) : 0, 7779
Telemetry [candidate] (7.963 ms) : 0, 7963
Profiling [baseline] (95.821 ms) : 0, 95821
Profiling [candidate] (96.455 ms) : 0, 96455
Startup time reports for insecure-bankgantt
title insecure-bank - global startup overhead: candidate=1.45.0-SNAPSHOT~36b653bf75, baseline=1.45.0-SNAPSHOT~1951e6f72e
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.056 s) : 0, 1055597
Total [baseline] (8.599 s) : 0, 8599217
Agent [candidate] (1.051 s) : 0, 1051236
Total [candidate] (8.593 s) : 0, 8592595
section iast
Agent [baseline] (1.18 s) : 0, 1180451
Total [baseline] (9.198 s) : 0, 9197859
Agent [candidate] (1.179 s) : 0, 1179407
Total [candidate] (9.253 s) : 0, 9253319
section iast_HARDCODED_SECRET_DISABLED
Agent [baseline] (1.18 s) : 0, 1179981
Total [baseline] (9.144 s) : 0, 9143985
Agent [candidate] (1.175 s) : 0, 1175057
Total [candidate] (9.15 s) : 0, 9149626
section iast_TELEMETRY_OFF
Agent [baseline] (1.182 s) : 0, 1182012
Total [baseline] (9.17 s) : 0, 9169865
Agent [candidate] (1.174 s) : 0, 1173637
Total [candidate] (9.178 s) : 0, 9177920
gantt
title insecure-bank - break down per module: candidate=1.45.0-SNAPSHOT~36b653bf75, baseline=1.45.0-SNAPSHOT~1951e6f72e
dateFormat X
axisFormat %s
section tracing
BytebuddyAgent [baseline] (713.92 ms) : 0, 713920
BytebuddyAgent [candidate] (709.806 ms) : 0, 709806
GlobalTracer [baseline] (257.119 ms) : 0, 257119
GlobalTracer [candidate] (255.865 ms) : 0, 255865
AppSec [baseline] (55.221 ms) : 0, 55221
AppSec [candidate] (56.572 ms) : 0, 56572
Remote Config [baseline] (719.245 µs) : 0, 719
Remote Config [candidate] (684.183 µs) : 0, 684
Telemetry [baseline] (13.595 ms) : 0, 13595
Telemetry [candidate] (13.384 ms) : 0, 13384
section iast
BytebuddyAgent [baseline] (830.082 ms) : 0, 830082
BytebuddyAgent [candidate] (829.321 ms) : 0, 829321
GlobalTracer [baseline] (246.624 ms) : 0, 246624
GlobalTracer [candidate] (246.626 ms) : 0, 246626
AppSec [baseline] (58.183 ms) : 0, 58183
AppSec [candidate] (57.916 ms) : 0, 57916
IAST [baseline] (21.335 ms) : 0, 21335
IAST [candidate] (21.407 ms) : 0, 21407
Remote Config [baseline] (640.559 µs) : 0, 641
Remote Config [candidate] (657.698 µs) : 0, 658
Telemetry [baseline] (8.597 ms) : 0, 8597
Telemetry [candidate] (8.498 ms) : 0, 8498
section iast_HARDCODED_SECRET_DISABLED
BytebuddyAgent [baseline] (829.687 ms) : 0, 829687
BytebuddyAgent [candidate] (826.311 ms) : 0, 826311
GlobalTracer [baseline] (246.772 ms) : 0, 246772
GlobalTracer [candidate] (246.134 ms) : 0, 246134
AppSec [baseline] (58.145 ms) : 0, 58145
AppSec [candidate] (57.496 ms) : 0, 57496
IAST [baseline] (21.221 ms) : 0, 21221
IAST [candidate] (21.019 ms) : 0, 21019
Remote Config [baseline] (666.876 µs) : 0, 667
Remote Config [candidate] (645.178 µs) : 0, 645
Telemetry [baseline] (8.505 ms) : 0, 8505
Telemetry [candidate] (8.578 ms) : 0, 8578
section iast_TELEMETRY_OFF
BytebuddyAgent [baseline] (831.14 ms) : 0, 831140
BytebuddyAgent [candidate] (824.913 ms) : 0, 824913
GlobalTracer [baseline] (247.874 ms) : 0, 247874
GlobalTracer [candidate] (246.383 ms) : 0, 246383
AppSec [baseline] (57.984 ms) : 0, 57984
AppSec [candidate] (57.629 ms) : 0, 57629
IAST [baseline] (20.917 ms) : 0, 20917
IAST [candidate] (20.742 ms) : 0, 20742
Remote Config [baseline] (640.189 µs) : 0, 640
Remote Config [candidate] (654.513 µs) : 0, 655
Telemetry [baseline] (8.468 ms) : 0, 8468
Telemetry [candidate] (8.398 ms) : 0, 8398
LoadParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 11 metrics, 17 unstable metrics. Request duration reports for insecure-bankgantt
title insecure-bank - request duration [CI 0.99] : candidate=1.45.0-SNAPSHOT~36b653bf75, baseline=1.45.0-SNAPSHOT~1951e6f72e
dateFormat X
axisFormat %s
section baseline
no_agent (374.373 µs) : 354, 394
. : milestone, 374,
iast (496.536 µs) : 475, 518
. : milestone, 497,
iast_FULL (657.313 µs) : 636, 679
. : milestone, 657,
iast_GLOBAL (523.809 µs) : 502, 546
. : milestone, 524,
iast_HARDCODED_SECRET_DISABLED (492.827 µs) : 472, 514
. : milestone, 493,
iast_INACTIVE (456.102 µs) : 435, 477
. : milestone, 456,
iast_TELEMETRY_OFF (480.631 µs) : 459, 502
. : milestone, 481,
tracing (452.635 µs) : 431, 474
. : milestone, 453,
section candidate
no_agent (376.052 µs) : 356, 396
. : milestone, 376,
iast (490.129 µs) : 469, 512
. : milestone, 490,
iast_FULL (652.589 µs) : 631, 674
. : milestone, 653,
iast_GLOBAL (534.871 µs) : 511, 558
. : milestone, 535,
iast_HARDCODED_SECRET_DISABLED (496.304 µs) : 474, 518
. : milestone, 496,
iast_INACTIVE (454.122 µs) : 433, 475
. : milestone, 454,
iast_TELEMETRY_OFF (486.157 µs) : 464, 508
. : milestone, 486,
tracing (450.614 µs) : 429, 472
. : milestone, 451,
Request duration reports for petclinicgantt
title petclinic - request duration [CI 0.99] : candidate=1.45.0-SNAPSHOT~36b653bf75, baseline=1.45.0-SNAPSHOT~1951e6f72e
dateFormat X
axisFormat %s
section baseline
no_agent (1.347 ms) : 1327, 1366
. : milestone, 1347,
appsec (1.744 ms) : 1720, 1768
. : milestone, 1744,
appsec_no_iast (1.753 ms) : 1729, 1777
. : milestone, 1753,
iast (1.523 ms) : 1500, 1545
. : milestone, 1523,
profiling (1.519 ms) : 1496, 1542
. : milestone, 1519,
tracing (1.497 ms) : 1472, 1522
. : milestone, 1497,
section candidate
no_agent (1.365 ms) : 1344, 1386
. : milestone, 1365,
appsec (1.754 ms) : 1729, 1778
. : milestone, 1754,
appsec_no_iast (1.757 ms) : 1732, 1783
. : milestone, 1757,
iast (1.485 ms) : 1461, 1508
. : milestone, 1485,
profiling (1.502 ms) : 1478, 1525
. : milestone, 1502,
tracing (1.471 ms) : 1446, 1496
. : milestone, 1471,
DacapoParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 12 metrics, 0 unstable metrics. Execution time for tomcatgantt
title tomcat - execution time [CI 0.99] : candidate=1.45.0-SNAPSHOT~36b653bf75, baseline=1.45.0-SNAPSHOT~1951e6f72e
dateFormat X
axisFormat %s
section baseline
no_agent (1.46 ms) : 1449, 1472
. : milestone, 1460,
appsec (2.34 ms) : 2298, 2382
. : milestone, 2340,
iast (2.08 ms) : 2027, 2133
. : milestone, 2080,
iast_GLOBAL (2.124 ms) : 2071, 2178
. : milestone, 2124,
profiling (1.947 ms) : 1905, 1990
. : milestone, 1947,
tracing (1.92 ms) : 1880, 1961
. : milestone, 1920,
section candidate
no_agent (1.465 ms) : 1453, 1476
. : milestone, 1465,
appsec (2.33 ms) : 2289, 2372
. : milestone, 2330,
iast (2.072 ms) : 2020, 2125
. : milestone, 2072,
iast_GLOBAL (2.126 ms) : 2073, 2179
. : milestone, 2126,
profiling (1.968 ms) : 1925, 2011
. : milestone, 1968,
tracing (1.928 ms) : 1888, 1969
. : milestone, 1928,
Execution time for biojavagantt
title biojava - execution time [CI 0.99] : candidate=1.45.0-SNAPSHOT~36b653bf75, baseline=1.45.0-SNAPSHOT~1951e6f72e
dateFormat X
axisFormat %s
section baseline
no_agent (15.0 s) : 15000000, 15000000
. : milestone, 15000000,
appsec (15.04 s) : 15040000, 15040000
. : milestone, 15040000,
iast (18.834 s) : 18834000, 18834000
. : milestone, 18834000,
iast_GLOBAL (17.801 s) : 17801000, 17801000
. : milestone, 17801000,
profiling (15.781 s) : 15781000, 15781000
. : milestone, 15781000,
tracing (15.171 s) : 15171000, 15171000
. : milestone, 15171000,
section candidate
no_agent (15.569 s) : 15569000, 15569000
. : milestone, 15569000,
appsec (15.009 s) : 15009000, 15009000
. : milestone, 15009000,
iast (18.789 s) : 18789000, 18789000
. : milestone, 18789000,
iast_GLOBAL (17.985 s) : 17985000, 17985000
. : milestone, 17985000,
profiling (14.852 s) : 14852000, 14852000
. : milestone, 14852000,
tracing (14.769 s) : 14769000, 14769000
. : milestone, 14769000,
|
gary-huang
force-pushed
the
gary/add-llmobs-configs
branch
2 times, most recently
from
fd136e0 to
ba03bdd
Compare
gary-huang
force-pushed
the
gary/add-llmobs-configs
branch
from
ba03bdd to
a8108c6
Compare
|
Hi! 👋 Looks like you updated a Git Submodule.
|
|
Hi! 👋 Thanks for your pull request! 🎉 To help us review it, please make sure to:
If you need help, please check our contributing guidelines. |
Yun-Kim
approved these changes
Dec 30, 2024
Comment on lines
+2673
to
+2678
| public boolean isLlmObsAgentlessEnabled() { | ||
| return llmObsAgentlessEnabled; | ||
| } | ||
|
|
||
| public String getLlmObsMlApp() { | ||
| return llmObsMlApp; |
There was a problem hiding this comment.
Are these two options not set on the instrumenterConfig object? We're fine to just store this directly on the Config instance?
There was a problem hiding this comment.
i think it is fine to store these in the config instance for now, i am not sure but it is possible we may need them, instrumenterConfig seems to be about starting actual implementations classes if a product is enabled, i think we can add them later as needed
There was a problem hiding this comment.
Yup that's fine with me to add them later when needed. Thanks for clarifying
| */ | ||
| public final class LlmObsConfig { | ||
|
|
||
| public static final String LLM_OBS_ENABLED = "llmobs.enabled"; |
There was a problem hiding this comment.
Tiny nit, I think the other two language SDKs use the naming pattern LLMOBS_ instead of LLM_OBS_, would be nice to stay consistent but not that serious so up to you
There was a problem hiding this comment.
yes for sure, good catch!
|
Hi! 👋 Looks like you updated a Git Submodule.
|
|
Hi! 👋 Looks like you updated a Git Submodule.
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What Does This Do
adds llm observability related configs and some start up behavior that are related to the enablement of llm observability
dd.llmobs.enabledis set to true and there is no ml app defined - throw errordd.llmobs.agentless.enabledis set to true and there is no API key supplied - throw errorjava system property success cases
agent mode configuration
agentless mode configuration with API key env var
env variable success cases
agent mode configuration
agentless mode configuration
Motivation
Additional Notes
tested with a sample spring app
the following stack satisfies 1
the follow stack satisfies 2
Contributor Checklist
type:and (comp:orinst:) labels in addition to any usefull labelsclose,fixor any linking keywords when referencing an issue.Use
solvesinstead, and assign the PR milestone to the issueJira ticket: [PROJ-IDENT]