Frequently Asked Questions

Table of Contents

• My app uses the async client libraries, but it never seems to get any results?
• I'm getting a NoSuchMethodError or NoClassDefFoundError
• Creating shaded Jars to avoid dependency conflicts
• How to manage dependencies of my application running on Azure Data Bricks?
• My app uses Java's Security Manager and I have granted it all Permissions, yet it tells me it doesn't have the permission to do something

My app uses the async client libraries, but it never seems to get any results?

The Java client library asynchronous methods returns reactive types - Flux<T> and Mono<T>, or types extending from these.

These reactive types are referred to as "cold". That is, all underlying work will not be activated without a subscription. When an application invokes an async method, it will return without receiving any result. Only once a subscription is added will work be initiated to receive results.

The simplest way to create a subscription is to call subscribe() on the reactive type, but sometimes even after subscription you might not receive result. Consider the following example:

Mono<Response<Key>> result = asyncClient.getKey();
result.subscribe(response -> {
    System.out.println(response.value());
});

While the subscribe() call causes execution of the underlying work that makes an API call to retrieve the key, sometimes the lambda provided to print out the response value does not get called.

The reason for this is obvious in hindsight: reactive types are asynchronous. The rest of the application continues to operate as per usual, and this includes having the application exit before the results are returned!

Knowing this, the easiest means to block execution and be sure you get the results before your application continues execution (and / or exits) is to use the various block calls available, for example:

Mono<Response<Key>> result = asyncClient.getKey();
result.subscribe(response -> {
    System.out.println(response.value());
}).block();

---

I'm getting a NoSuchMethodError or NoClassDefFoundError

Please check out Troubleshoot dependency version conflicts for more details.

Explanation of the Cause

This exception is commonly the result of dependency conflicts. When two libraries depend on different versions of the same package, Maven will simply pick the first version that it sees and ignore the others ("First" is determined by which version is closer to the root project, and if they are the same distance, then the one declared first in the pom will be chosen) . If the two versions are sufficiently incompatible, it is possible that they offer a different set of types and methods. The library depending on the version that wasn't picked may then be unable to find the types or methods it depends on at runtime, resulting in these errors.

For example, suppose package A depends on version 1.0 of package C and package B depends on version 2.0 of version C. Version 1 contains type Foo, but in version 2 this type was removed and replaced with type Bar. If my application specifies a dependency on package A and then package B, only version 1 of package C will be included in my build. When my application's code requires package B to do something using type Bar, the JVM will throw this error.

Resolution

• Step 1: Validate the Cause of the Error In the same directory as the pom for your project, run the command mvn dependency:tree. This will output a recursive list of all your dependencies and their dependencies. In this list, search for the package that contains the type or method that is throwing the error. There will likely be multiple entries of different versions. Be aware that not all packages follow semantic versioning strictly. Just because both entries specify the same major version of a given package, that is not sufficient to assume they are compatible.

• Step 2: Resolve the dependency conflict In your project's pom, include a dependency entry that specifies the version of the shared dependency you want both packages to use. This will force Maven to use the specified version that should work for both. Because only one version can be included at a time, you must pick a single version that works for both packages. In the event that this is not possible, it may be necessary to upgrade one of your direct dependencies. If this still does not resolve the problem, your best course of action is to post an issue on the repo on each of the projects and request that they upgrade their dependencies.

---

Creating shaded Jars to avoid dependency conflicts

We have documented how you can create a shaded JAR file that includes all dependencies. This should allow for dependency conflicts to be mitigated.

---

How to manage dependencies of my application running on Azure Data Bricks?

Azure Data Bricks provides an Apache Spark environment for your application. For example: Azure Data Bricks Runtime version 6.1 comes with Apache Spark 2.4.4 pre-installed. When running applications in ADB, ADB injects Apache Spark and it’s dependencies into application's class-path. Given runtime is providing Spark, normally application uses <provided> scope for Spark dependency.

Side effect of dependency injection

A side effect of ADB injecting Spark and its dependencies into the app class-path is, they take precedence over the direct and transitive dependencies that the application want to use. This is true for many hosting service that offer environment similar to ADB. When the application dependencies conflict with the ADB injected dependencies, errors such as NoSuchMethodError/NoClassDefFoundError can occur.

Example: Azure Storage SDK v12.0.0 has a dependency on Jackson version 2.10.0. If an application uses this version of storage SDK in ADB, it conflicts with the Jackson version 2.6.7 that DBRv6.1 place in the class-path. Jackson v2.10.0 is not back compact with v2.6.7, hence in this setup the application is likely to fail with NoSuchMethodError/NoClassDefFoundError linkage error.

How do I use these libraries in Azure Data Bricks?

Follow the steps below to upload these libraries into Azure Data Bricks:

1. Follow the guidance above to create a shaded Jar.

2. From azure portal navigate to your Azure Data bricks portal (e.g. West US ADB portal)

3. Create a cluster based on Runtime: 6.1 (Scala 2.11, Spark 2.4.4)

4. Once cluster is ready, create a Job:

   a. Provide name for job

   b. Choose task as "Set JAR", this will bring up option to upload the JAR, point it to the shaded JAR

   c. Enter Main class name that you provided for mainClass entry of shaded plugin

What if application still fail with conflict?

Jackson is the known conflict at the time of writing this. If you run into more conflicts (NoSuchMethodError/NoClassDefFoundError exceptions) then follow the sample relocation approach outlined above for other conflicting dependencies.

---

My app uses Java's Security Manager and I have granted it all Permissions, yet it tells me it doesn't have the permission to do something

This is probably related to how Java’s common thread pool (ForkJoinPool) and Security Manager interact. In short, the library the Azure SDKs use for authentication (MSAL4J) requests for an access token asynchronously using the CompletableFuture class, which in turn uses a thread provided by the common pool. Threads from this pool by default have no Permissions enabled and errors will be raised when using them to run tasks that require any Permission. You can find more on this here and here.

There are a few workarounds for this problem:

1. Probably the simplest one is to use a different thread pool and providing it to the CredentialBuilder. This is currently only possible on version 1.1.0 of azure-identity and onwards. Here is some sample code demonstrating this approach:

   ExecutorService executorService = Executors.newCachedThreadPool(); // Get a non-default thread pool.  
   
   try {  
       final ClientSecretCredential clientSecretCredential = new ClientSecretCredentialBuilder()  
           .clientId("<client-ID>")  
           .clientSecret("<client-secret>")  
           .tenantId("<tenant-ID>")  
           .executorService(executorService) // ExecutorService explicitly given to CredentialBuilder.  
           .build();  
   
       // Build a client(s) that uses the Credential object created above.  
       final SecretClient secretClient = new SecretClientBuilder()  
           .vaultUrl("<your-kay-vault-URI>")  
           .credential(clientSecretCredential)  
           .buildClient();  
   
       // Execute all desired operations.  
       final KeyVaultSecret secret = secretClient.getSecret("<secret-name>");  
   
       System.out.printf("Retrieved secret with name \"%s\" and value \"%s\"\n", secret.getName(),  
           secret.getValue());  
   } finally {  
       // This block will be reached after the above code is executed or if an uncaught exception is raised.  
   
       // It is important to shut down the ExecutorService after it is no longer required (i.e. once the client(s)  
       // using the Credential object with the ExecutorService will not be used any longer).  
   
       // Use the following if it is possible there are still operations that require authentication running  
       // asynchronously. Otherwise, it is safe to use the alternative "executorService.shutdownNow();"  
       executorService.shutdown();  
   }

2. A different approach is to use a custom Fork/Join thread factory:

   public static class MyForkJoinWorkerThreadFactory implements ForkJoinPool.ForkJoinWorkerThreadFactory {
       public final ForkJoinWorkerThread newThread(ForkJoinPool pool) {
           return new ForkJoinWorkerThread(pool) {};
       }
   }

   and have the Fork/Join framework use it for the common pool by adding the following program argument:

   -Djava.util.concurrent.ForkJoinPool.common.threadFactory=MyForkJoinWorkerThreadFactory

   The behavior of MyForkJoinWorkerThreadFactory is equivalent to that of ForkJoinPool.defaultForkJoinWorkerThreadFactory and it can be further customized.