Unable to use DefaultAzureCredential with .Net Framework 4.8 WebJobs - AzureWebJobsStorage problem

[GMacAussie]

Unable to use DefaultAzureCredential with .Net Framework 4.8 WebJobs. There seems to be an issue with AzureWebJobsStorage setting being parsed as a connection string, not Uri/token.

Removal of AzureWebJobsStorage setting and replaced by AzureWebJobsStorage__credential/AzureWebJobsStorage__accountName settings has no effect, nor does blobServiceUri/BlobConnection__blobServiceUri setting - startup complains that AzureWebJobsStorage is not supplied.

From the stack trace, it appears that even though a call is made for Uri/token creation of the BlobServiceClient for WebJobs storage, the connection string constructor is called instead.

Note that the only real change from our existing production system, which is running Microsoft.Azure.WebJobs 3.0.37 along with Microsoft.Azure.WebJobs.Host.Storage 5.0.0, is the upgrade of identity and storage packages to make sure DefaultAzureCredential
works.

Any insights appreciated.

Repro steps

Provide the steps required to reproduce the problem

1. Setup .Net Frameworks 4.8 with latest libraries for WebJobs and DefaultAzureCredential. Includes latest Azure.Identity, Azure.Storage.*, Microsoft.Azure.WebJobs.*

2. Attempt to startup the project locally using a DefaultAzureCredential and the storage connection via Uri/credential.

Expected behavior

The await host.RunAsync() successfully starts the host.

Actual behavior

The await host.RunAsync() takes an exception attempting to parse the AzureWebJobsStorage as an connection string (vs Uri/token).

Known workarounds

None.

Related information

Provide any related information

Packages:

Azure.Core 1.39.0
Azure.Identity 1.11.3
Azure.Storage.Blobs 12.19.1
Azure.Storage.Common 12.18.1
Azure.Storage.Queues 12.17.1
Microsoft.Azure.WebJobs 3.0.39
Microsoft.Azure.WebJobs.Core 3.0.39
Microsoft.Azure.WebJobs.Extensions 5.0.0
Microsoft.Azure.WebJobs.Extensions.Storage 5.3.0
Microsoft.Azure.WebJobs.Extensions.Storage.Blobs 5.3.0
Microsoft.Azure.WebJobs.Extensions.Storage.Queues 5.3.0
Microsoft.Azure.WebJobs.Host.Storage 5.0.0
Microsoft.Azure.WebJobs.Sources 3.0.39
Microsoft.Identity.Client 4.60.3
Microsoft.Web.WebJobs.Publish 17.1.361

Configuration in Main:

DefaultAzureCredential credential = new DefaultAzureCredential(new DefaultAzureCredentialOptions() {
    ExcludeAzureCliCredential = true,
    ExcludeVisualStudioCredential = false,
    ExcludeEnvironmentCredential = true,
    ExcludeAzurePowerShellCredential = true,
    ExcludeManagedIdentityCredential = false,
    ExcludeSharedTokenCacheCredential = true,
    ExcludeVisualStudioCodeCredential = true,
    ExcludeWorkloadIdentityCredential = true,
    ExcludeAzureDeveloperCliCredential = true,
    ExcludeInteractiveBrowserCredential = true,

    TenantId = "<tenantId>",
});

IHostBuilder hostBuilder = new HostBuilder()
   .UseEnvironment(Environments.Development)
   .ConfigureAppConfiguration((hostContext, config) => {
       config.AddEnvironmentVariables();
       config.Add(new MemoryConfigurationSource() {
           InitialData = new Dictionary<string, string> {
               // Usual local development connection string works
               //{ "AzureWebJobsStorage", "UseDevelopmentStorage=true" },
               //{ "AzureWebJobsDashboard", "UseDevelopmentStorage=true" },
               
               // Does not work with credential
               { "AzureWebJobsStorage", "https://<account>.blob.core.windows.net/" },
               { "AzureWebJobsDashboard", "https://<account>.blob.core.windows.net/" },

               // Does not work, with managedIdentity or managedidentity
               { "AzureWebJobsStorage__credential", "managedIdentity" },
               { "AzureWebJobsStorage__accountName", "<account>" },

               // Doesn't seem to get used for AzureWebJobsStorage purposes
               { "blobServiceUri", "https://<account>.blob.core.windows.net/" },
               { "BlobConnection__blobServiceUri", "https://<account>.blob.core.windows.net/" },

               // Don't know if this works - cannot get that far
               { "queueServiceUri", "https://<account>.queue.core.windows.net/" },
               { "QueueConnection__queueServiceUri", "https://<account>.queue.core.windows.net/" },

           }
       });
   })
   .ConfigureWebJobs((builder) => {
       builder.AddAzureStorageCoreServices();
       builder.AddExecutionContextBinding();
       builder.AddAzureStorageBlobs();
       builder.AddAzureStorageQueues();
       builder.Services.AddAzureClients(configure => {
           configure.UseCredential(credential);

           // credential above seems to get passed correctly, but WebJobs storage still tries to use connection string format BlobServiceClient 
           // constructor for host storage vs. URI/token constructor
           configure.AddBlobServiceClient(new Uri("https://<account>.blob.core.windows.net/"));
           configure.AddQueueServiceClient(new Uri("https://<account>.queue.core.windows.net/"));
       });
       builder.AddTimers();
   })
   .ConfigureLogging((context, builder) => {
       // add logging for debug
       builder.AddNLog();
       builder.AddFilter("Azure.Core", Microsoft.Extensions.Logging.LogLevel.Error);
   });

Console.WriteLine("WebJobs starting...");

using (IHost host = hostBuilder.Build()) {
    // issue information on version
    try {
        QueueServiceClient queueServiceClient = null;

        //string storageConnection = UseDevelopmentStorage=true;
        string storageConnection = "https://<account>.queue.core.windows.net/";

        if (storageConnection == "UseDevelopmentStorage=true") {
            queueServiceClient = new QueueServiceClient(storageConnection, new QueueClientOptions { MessageEncoding = QueueMessageEncoding.Base64 });
        } else {
            queueServiceClient = new QueueServiceClient(new Uri(storageConnection), credential, new QueueClientOptions { MessageEncoding = QueueMessageEncoding.Base64 });
        }

        // retreive the queue client
        QueueClient queueClient = queueServiceClient.GetQueueClient("queuerequest");

        // make sure it exists
        await queueClient.CreateIfNotExistsAsync().ConfigureAwait(false);

        await queueServiceClient.GetQueueClient("queuerequest-poison").CreateIfNotExistsAsync().ConfigureAwait(false);

        // and queue
        SendReceipt sendReceipt = await queueClient.SendMessageAsync("").ConfigureAwait(false);
    } catch (Exception ex) {
        Console.WriteLine($"Unable to queue version request job: {ex}");
    }

    // run WebJobs
    await host.RunAsync();
}

Console.WriteLine("Queue terminating...");

Trigger for running state:

public class VersionQueue {
    /// <summary>
    /// Handle version requests.
    /// </summary>
    /// <param name="content">The content string, being empty</param>
    /// <param name="id">The message id of the message</param>
    /// <param name="dequeueCount">The number of times this message has ben dequeued</param>
    /// <param name="insertionTime">The insertion date/time the message was queued</param>
    public static void GetVersion([QueueTrigger("queuerequest")] string content, string id, long dequeueCount, DateTimeOffset insertionTime) {
        // if this is not our first attempt we log this as a warning
        if (dequeueCount > 1) {
            Console.WriteLine($"MessageId={id}: Dequeue count raised: {dequeueCount}.");
        }

        // completed
        Console.WriteLine($"Type: Identity, Version: Testing");
    }

    /// <summary>
    /// Handle get version failed messages.
    /// </summary>
    /// <param name="message">The bad version request</param>
    /// <param name="id">The message id of the message</param>
    public static void GetVersionPoison([QueueTrigger("queuerequest-poison")] string message, string id) {
        Console.WriteLine($"MessageId={id}: Unable to process Version Request Message='{message}'.");
    }
}

Stack trace where connection string parse occurs: