Performance testing

Performance

Microsoft.Identity.Test.Performance project uses BenchmarkDotNet library for performance testing of MSAL methods. Classes AcquireTokenForClientLargeCacheTests, AcquireTokenForOboCacheTests, TokenCacheTests contain benchmark test methods which test the cache for different scenarios.

This performance test project is a console app. Behind the scenes when the project is run, BenchmarkDotNet builds and outputs this test project into a temporary working directory. Then it creates a separate process where all the benchmarking measurements are done.

BenchmarkDotNet is customizable. BenchmarkDotNet tests are set up similarly to unit tests, using attributes. Benchmarks can be parametrized. There are global and iteration setup and cleanup methods that can be used to setup the environment before running actual tests. The number of times that a benchmark should run can be customized, although it is recommended to use the defaults, as the BenchmarkDotNet does it's own pre-processing to find the optimal number of runs. How it works guide describes the steps that BenchmarkDotNet takes to run the benchmarks. BenchmarkDotNet supports running tests on multiple frameworks.

Running tests

There are multiple ways to run the tests. First:

• Build Microsoft.Client.Test.Performance in Release mode.
• Go to the {project directory}/bin/Release/{framework directory}/ and run the project executable.

Another one:

• Go to the project directory.
• Run dotnet run -c Release in the console window.

BenchmarkDotNet.Artifacts folder with the exported results will be created in the directory from which the executable was run from.

The test project can be ran multiple times using the methods above and then the results aggregated manually. Second way is to call WithLaunchCount(this Job job, int count) method in Program.cs when setting up the BenchmarkDotNet job. This will specify how many times the BenchmarkDotNet will launch the benchmark process.

Testing code changes

• Build and run the perf project to establish baseline numbers.
• Make desired MSAL code changes.
• Again build and run the perf project.
• Compare the results between the runs.
• Include the before and after results in the pull request that includes these changes. Also mention the PR and the improvements in the Improvements and test results section below.

Viewing results

Sample table with summary results:

Method	TokenCacheSize	Mean	Error	StdDev
AcquireTokenForClientTestAsync	100	62.14 μs	0.934 μs	0.873 μs
AcquireTokenForClientTestAsync	1000	383.90 μs	7.596 μs	9.876 μs
AcquireTokenForClientTestAsync	10000	5,111.33 μs	97.121 μs	103.918 μs
AcquireTokenForClientTestAsync	100000	98,313.18 μs	783.933 μs	733.292 μs

Results are consolidated across all the iterations and launches.. They are written to the console at the end of the run and also exported into .md, .csv, and .html files in BenchmarkDotNet.Artifacts folder by default. The results are grouped by the benchmark method and any parameters. The main data point is the mean value. Compare this value across runs, before and after code changes. Some other potentially interesting data points exported include median, min, max, skewness, kurtosis, confidence interval. The run log, which contains how many times benchmarks were executed and general debug information, is also exported into the same folder.

Improvements and test results

We regularly work on improving MSAL.NET performance. Listed are some of the recent major improvements. See pull requests for performance data.

PR #2261 includes improvements for AcquireTokenForClient method, especially when an internal token cache is large (100k+ items). Testing showed 10% - 30% speed improvement. Released in MSAL 4.24.0.

PR #2309 includes a way to disable legacy caching with WithLegacyCacheCompatibility builder method. Disabling legacy cache, if not used, speeds up MSAL cache operations, especially for large caches. Released in MSAL 4.25.0.

PR #2834 improved performance by removing unnecessary serialization and adding partitioning in default app token cache used in client credentials flow. Released in MSAL 4.36.0. Diagram shows a performance improvement for P99 latency in milliseconds for client credentials call.

PR #2881 significantly improved caching performance by adding partitioning to the default in-memory user cache used in user flows (like acquire token on-behalf-of, by authorization code). Released in MSAL 4.37.0-preview.

Metrics

MSAL gathers metrics specific to each request. These are located in AuthenticationResult.AuthenticationResultMetadata.

• TokenSource - the source of the token: identity provider, MSAL cache, or broker. It is set when the AuthenticationResult is created after the token response is received.
• DurationTotalInMs - time spent servicing the request from when AcquireTokenX is invoked to when the authentication result is returned (for all flows, client credential, silent, interactive, etc. and includes external work, like time spent in HTTP calls and in user interaction).
• DurationInCacheInMs - time MSAL spent during a request reading and writing to the token cache, i.e. cache serialization in the OnBeforeAccess, OnAfterAccess, etc. callbacks. This time does not include internal MSAL logic for searching through the cache once it's loaded.
• DurationInHttpInMs - time MSAL spent for HTTP communication during this request. It is set around the send HTTP request call on the HTTP client.

These MSAL-wide aggregate metrics are located in the Metrics class.

• TotalAccessTokensFromIdP - Aggregate count of tokens obtained by MSAL from the identity provider. This value is incremented after a valid token response is retrieved from a token endpoint.
• TotalAccessTokensFromCache - Aggregate count of tokens obtained by MSAL from cache. This value is incremented after a valid token is retrieved from the MSAL cache and the response is returned (for example, no proactive refresh needed, token wasn't expired, etc.).
• TotalAccessTokensFromBroker - Aggregate count of tokens obtained by MSAL from broker. This value is incremented after a valid token is retrieved from the broker (Android, WAM, etc.) and the response is returned.
• TotalDurationInMs - Total time spent in MSAL for all requests. Aggregate of AuthenticationResultMetadata.DurationInCacheInMs, added at the end of each request.