Writing Performance Tests

Performance Test Setup Instructions

Follow the steps below to set up a performance test harness for your library:

1. Create a maven module under sdk/<service>/azure-<service>-perf/ (if one doesn't exist)
   1. Tests will live under sdk/<service>/azure-<service>-perf/src
   2. Pom file will live under sdk/<service>/azure-<service>-perf/pom.xml
2. Pom file structure:

<?xml version="1.0" encoding="UTF-8"?>

<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">


<parent>
  <groupId>com.azure</groupId>
  <artifactId>azure-client-sdk-parent</artifactId>
  <version>1.7.0</version> <!-- {x-version-update;com.azure:azure-client-sdk-parent;current} -->
  <relativePath>../../parents/azure-client-sdk-parent</relativePath>
</parent>

<modelVersion>4.0.0</modelVersion>

<groupId>com.azure</groupId>
<artifactId>azure-<service-name>-perf</artifactId>
<version>1.0.0-beta.1</version> <!-- {x-version-update;com.azure:azure-storage-perf;current} -->
<packaging>jar</packaging>

<dependencies>
  <dependency>
    <groupId>com.azure</groupId>
    <artifactId><sdk-artifact-id></artifactId>
    <version><sdk-version></version> <!-- {x-version-update;com.azure:azure-storage-blob;current} -->
  </dependency>
  <dependency>
    <groupId>com.azure</groupId>
    <artifactId>perf-test-core</artifactId>
    <version>1.0.0-beta.1</version> <!-- {x-version-update;com.azure:perf-test-core;current} -->
  </dependency>
</dependencies>

<build>
  <plugins>
    <plugin>
      <groupId>org.apache.maven.plugins</groupId>
      <artifactId>maven-assembly-plugin</artifactId>
      <version>3.2.0</version> <!-- {x-version-update;org.apache.maven.plugins:maven-assembly-plugin;external_dependency} -->
      <executions>
        <execution>
          <phase>package</phase>
          <goals>
            <goal>single</goal>
          </goals>
          <configuration>
            <archive>
              <manifest>
                <mainClass>
                  com.azure.<service>.<sub-service>.<your-main-class-from-step-1-below>
                </mainClass>
              </manifest>
            </archive>
            <descriptorRefs>
              <descriptorRef>jar-with-dependencies</descriptorRef>
            </descriptorRefs>
          </configuration>
        </execution>
      </executions>
    </plugin>
  </plugins>
</build>
</project>

3. Main Class: The project's main class should follow the following structure:

/**
 * Runs the <Service-Name> performance test.
 *
 * <p>To run from command line. Package the project into a jar with dependencies via mvn clean package.
 * Then run the program via java -jar 'compiled-jar-with-dependencies-path' </p>
 *
 * <p> To run from IDE, set all the required environment variables in IntelliJ via Run -&gt; EditConfigurations section.
 * Then run the App's main method via IDE.</p>
 */
public class App {
    public static void main(String[] args) {
        Class<?>[] testClasses;

        try {
            testClasses = new Class<?>[] {
                Class.forName("com.azure.<service>.<sub-package>.perf.<APIName>Test"),
                Class.forName("com.azure.<service>.<sub-package>.perf.<APIName>Test")
            };
        } catch (ClassNotFoundException e) {
            throw new RuntimeException(e);
        }

        PerfStressProgram.run(testClasses, args);
    }
}

4. Performance Test Class Structure: Create your performance Test classes under the com.azure.<service>.<sub-package>.perf package.

Create abstract test classes for doing client setup and clean up that is shared across all performance test classes.

The leaf nodes will be the performance test classes. Here is the sample structure of abstract Test class:

public abstract class ServiceTest<TOptions extends PerfStressOptions> extends PerfStressTest<TOptions> {
    protected final <ServiceClientName> serviceClientName;
    protected final <ServiceClientAsyncName> serviceClientAsyncName;
    public ServiceTest(TOptions options) {
        super(options);

        // Setup the service client
    }
}

Here is the sample structure of Performance Test class:

// The Options class specifies the options bundle which can be passed via command-line arguments. 
// The default options class PerfStressOptions is available in the framework. 
// To provide any custom options bundle. Create your options class extending from PerfStressOptions and specify it // below.
public class <API-NAME>Test extends ServiceTest<{OptionsClass}> {
    public <API-NAME>Test({OptionsClass} options) {
        super(options);
    // Optional Client setup is any child client needs to be setup specifically for this test.
    }

    // Required resource setup goes here.
    public Mono<Void> globalSetupAsync() {
        return super.globalSetupAsync()
                    .then(<service-call-to-perform-setup>)
                    .then();
    }


    // Perform the API call to be tested here
    @Override
    public void run() {
        serviceClient.apiCall();
    }

    // Perform the Async API call to be tested here
    @Override
    public Mono<Void> runAsync() {
        return serviceAsyncClient.apiCall()
            .map(<process-output>
            }).then();
    }

}

For Reference, look at Storage Performance Tests setup structure here.

Specifying Custom Options

The PerfStressOptions class in the performance framework comes with default options bundle applicable to any generic performance test. If there is a need to provide custom options to the performance test then a custom options class needs to be created extending PerfStressOptions and then it should be referenced in the performance test class setup above.

Executing the performance test.

1. Compile the performance project into a standalone jar. mvn clean package -f <path-to-perf-project-pom>

2. Execute the perf test: java -jar <path-to-packaged-jar-with-dependencies-from-step-1> <options-for-the-test>