Performance testing

Performance

Microsoft.Identity.Test.Performance project uses BenchmarkDotNet library for performance testing of MSAL methods. Program.cs lists the classes which have the benchmark methods which test the cache for different scenarios.

This performance test project is a console app. Behind the scenes when the project is run, BenchmarkDotNet builds and outputs this test project into a temporary working directory. Then it creates a separate process where all the benchmarking measurements are done.

BenchmarkDotNet is customizable. BenchmarkDotNet tests are set up similarly to unit tests, using attributes. Benchmarks can be parametrized. There are global and iteration setup and cleanup methods that can be used to setup the environment before running actual tests. The number of times that a benchmark should run can be customized, although it is recommended to use the defaults, as the BenchmarkDotNet does it's own pre-processing to find the optimal number of runs. How it works guide describes the steps that BenchmarkDotNet takes to run the benchmarks. BenchmarkDotNet supports running tests on multiple frameworks.

Running tests

There are multiple ways to run the tests locally.

One way:

• Build Microsoft.Client.Test.Performance in Release mode.
• Go to the {project directory}/bin/Release/{framework directory}/ and run the project executable.
• The results will be printed to the console window.

Another way:

• Go to the project directory.
• Run dotnet run -c Release in the console window.

BenchmarkDotNet.Artifacts folder with the exported results will be created in the directory from which the executable was run from.

The test project can be ran multiple times using the methods above and then the results aggregated manually. Another way to run the project multiple times is to add WithLaunchCount(this Job job, int count) method in Program.cs when setting up the BenchmarkDotNet job. This will specify how many times the BenchmarkDotNet will launch the benchmark process.

Automated tests

The process of running the tests described above is automated. The tests run as part of the build pipeline when changes are merged into the main branch and can also be run manually on a feature branch. Currently the results have to be compared manually.

Testing code changes

When making code changes to a performance critical code, make sure to run the tests to check for regressions.

To test locally:

• Build and run the perf project with the 'before' code state to establish baseline numbers.
• Make desired MSAL code changes.
• Again build and run the perf project to get the results for the 'after' state.
• Compare the results between the runs.
• Include the before and after results in the pull request that includes these changes. Also mention the PR and the improvements in the Improvements and test results section below.

The comparison can also be done using the build pipeline. Simply run the automated tests on the feature branch that has the new changes. Compare the results with any previous results from the runs on the main branch.

Viewing results

Sample table with summary results:

Method	TokenCacheSize	Mean	Error	StdDev
AcquireTokenForClientTestAsync	100	62.14 μs	0.934 μs	0.873 μs
AcquireTokenForClientTestAsync	1000	383.90 μs	7.596 μs	9.876 μs
AcquireTokenForClientTestAsync	10000	5,111.33 μs	97.121 μs	103.918 μs
AcquireTokenForClientTestAsync	100000	98,313.18 μs	783.933 μs	733.292 μs

Results are consolidated across all the iterations and launches.. They are written to the console at the end of the run and also exported into .md, .csv, and .html files in BenchmarkDotNet.Artifacts folder by default. The results are grouped by the benchmark method and any parameters. The main data point is the median value. Compare this value across runs, before and after code changes. Some other potentially interesting data points exported include mean, min, max, skewness, kurtosis, confidence interval. The run log, which contains how many times benchmarks were executed and general debug information, is also exported into the same folder.

About test cases

TBD

Improvements and test results

We regularly work on improving MSAL.NET performance. Performance related GitHub issues are tagged with performance label. Listed are some of the recent major improvements. See pull requests for performance data.

PR #2261 includes improvements for AcquireTokenForClient method, especially when an internal token cache is large (100k+ items). Testing showed 10% - 30% speed improvement. Released in MSAL 4.24.0.

PR #2309 includes a way to disable legacy caching with WithLegacyCacheCompatibility builder method. Disabling legacy cache, if not used, speeds up MSAL cache operations, especially for large caches. Released in MSAL 4.25.0.

PR #2834 improved performance by removing unnecessary serialization and adding partitioning in default app token cache used in client credentials flow. Released in MSAL 4.36.0. Diagram shows a performance improvement for P99 latency in milliseconds for client credentials call.

PR #2881 significantly improved caching performance by adding partitioning to the default in-memory user cache used in user flows (like acquire token on-behalf-of, by authorization code). Released in MSAL 4.37.0.

PR #3233 improves token filtering and reduces allocated memory. Released in MSAL 4.43.0.

PR #3250 shows the performance of creating a new app builder and difference between acquire token calls that use serialization and don't.

PR #3605 adds new .NET 6 target which uses System.Text.Json (STJ) library for JSON operations (instead of Newtonsoft Json.NET). The test results show about TBD% improvement of using .NET 6 MSAL binary with STJ over .NET Core 2.1 binary with Json.NET. Released in MSAL 4.47.0.

Metrics

See information about MSAL-provided metrics in token cache documentation.