DT-937 - Remove nested threadpool

[snf2ye]

Jira ticket: https://broadworkbench.atlassian.net/browse/DT-937

Addresses

We were continuously running into instances where Azure snapshot create flights would get stuck. After adding some logging in previous PRs, we were able to pinpoint the issue.

For large requests, we are running out of threads in a nested, shared threadpool. The parent call would hand out all of the threads and then the nested call wouldn't have any more threads to hand out. It was stuck forever waiting for a new thread.

    List<Future<Void>> futures = new ArrayList<>();
    for (List<String> fileIdsBatch :
        ListUtils.partition(List.copyOf(fileIds), MAX_FILTER_CLAUSES)) {
      futures.add(
          azureTableThreadpool.submit(
              () -> {
                // Do stuff
                /// ... 
                // Then, in batchStoreDirectoryEntry we try to pull from the azureTableTreadpool again, 
                // but all of the threads have already been handed out 
                // So, we just wait forever for more threads to become available 
                batchStoreDirectoryEntry(snapshotTableServiceClient, snapshotId, snapshotEntries);
               
                return null;
              }));
    }
    FutureUtils.waitFor(futures);

Summary of changes

Remove nested thread pool. Note: I expect that this means we'll see Azure Snapshot Create Jobs take longer.

Testing Strategy

• Manually decrease the azure.threading.numTableThreads setting locally and reproduce issue with the bio.terra.service.filedata.azure.tables.TableDaoConnectedTest#testComputeSnapshot connected test.
• With changes, this connected test passes with limited threads and with the normal, higher setting.

[rushtong]

Looks reasonable 👍🏽

[fboulnois]

Looks good! Should we unit test the storeDirectoryEntries method?

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
83.3% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud