Getting "Name or service not known (login.microsoftonline.com:443)" regularly, but occasionally it succeeds?

[ericmqt]

Summary: My container consistently cannot resolve login.microsoftonline.com, although rarely it will succeed, while I am unable to replicate the issue running the app outside of the container.

Base: mcr.microsoft.com/dotnet/aspnet:5.0
Build: mcr.microsoft.com/dotnet/sdk:5.0

Yesterday, I containerized one of our web apps, and I immediately ran into a head-scratcher.

This web app uses client certificates to authenticate to a back-end service. The web app obtains the client cert from KeyVault at application start-up using the Azure.Security.KeyVault.Certificates and Azure.Security.KeyVault.Secrets packages (both version 4.2.0). This worked perfectly fine up until I containerized the app, where it consistently (but not always!) fails with the following:

Azure.Identity.AuthenticationFailedException: ClientSecretCredential authentication failed: Retry failed after 4 tries. Retry settings can be adjusted in ClientOptions.Retry. (Name or service not known (login.microsoftonline.com:443)) (Name or service not known (login.microsoftonline.com:443)) (Name or service not known (login.microsoftonline.com:443)) (Name or service not known (login.microsoftonline.com:443))
       ---> System.AggregateException: Retry failed after 4 tries. Retry settings can be adjusted in ClientOptions.Retry. (Name or service not known (login.microsoftonline.com:443)) (Name or service not known (login.microsoftonline.com:443)) (Name or service not known (login.microsoftonline.com:443)) (Name or service not known (login.microsoftonline.com:443))
       ---> Azure.RequestFailedException: Name or service not known (login.microsoftonline.com:443)
       ---> System.Net.Http.HttpRequestException: Name or service not known (login.microsoftonline.com:443)
       ---> System.Net.Sockets.SocketException (0xFFFDFFFF): Name or service not known
         at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
         at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
         at System.Net.Sockets.Socket.<ConnectAsync>g__WaitForConnectWithCancellation|283_0(AwaitableSocketAsyncEventArgs saea, ValueTask connectTask, CancellationToken cancellationToken)
         at System.Net.Http.HttpConnectionPool.DefaultConnectAsync(SocketsHttpConnectionContext context, CancellationToken cancellationToken)
         at System.Net.Http.ConnectHelper.ConnectAsync(Func`3 callback, DnsEndPoint endPoint, HttpRequestMessage requestMessage, CancellationToken cancellationToken)
         --- End of inner exception stack trace ---
         at System.Net.Http.ConnectHelper.ConnectAsync(Func`3 callback, DnsEndPoint endPoint, HttpRequestMessage requestMessage, CancellationToken cancellationToken)
         at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
         at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
         at System.Net.Http.HttpConnectionPool.GetHttpConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
         at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
         at System.Net.Http.HttpClient.SendAsyncCore(HttpRequestMessage request, HttpCompletionOption completionOption, Boolean async, Boolean emitTelemetryStartStop, CancellationToken cancellationToken)
         at Azure.Core.Pipeline.HttpClientTransport.ProcessAsync(HttpMessage message, Boolean async)
         --- End of inner exception stack trace ---
         at Azure.Core.Pipeline.HttpClientTransport.ProcessAsync(HttpMessage message, Boolean async)
         at Azure.Core.Pipeline.ResponseBodyPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
         at Azure.Core.Pipeline.RedirectPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
         at Azure.Core.Pipeline.RetryPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
         --- End of inner exception stack trace ---
         at Azure.Core.Pipeline.RetryPolicy.ProcessAsync(HttpMessage message, ReadOnlyMemory`1 pipeline, Boolean async)
         at Azure.Core.Pipeline.HttpPipeline.SendRequestAsync(Request request, CancellationToken cancellationToken)
         at Azure.Core.HttpPipelineMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
         at System.Net.Http.HttpClient.SendAsyncCore(HttpRequestMessage request, HttpCompletionOption completionOption, Boolean async, Boolean emitTelemetryStartStop, CancellationToken cancellationToken)
         at Microsoft.Identity.Client.Http.HttpManager.ExecuteAsync(Uri endpoint, IDictionary`2 headers, HttpContent body, HttpMethod method, ICoreLogger logger, CancellationToken cancellationToken)
         at Microsoft.Identity.Client.Http.HttpManager.ExecuteWithRetryAsync(Uri endpoint, IDictionary`2 headers, HttpContent body, HttpMethod method, ICoreLogger logger, Boolean doNotThrow, Boolean retry, CancellationToken cancellationToken)
         at Microsoft.Identity.Client.Http.HttpManager.SendGetAsync(Uri endpoint, IDictionary`2 headers, ICoreLogger logger, Boolean retry, CancellationToken cancellationToken)
         at Microsoft.Identity.Client.OAuth2.OAuth2Client.ExecuteRequestAsync[T](Uri endPoint, HttpMethod method, RequestContext requestContext, Boolean expectErrorsOn200OK, Boolean addCommonHeaders)
         at Microsoft.Identity.Client.OAuth2.OAuth2Client.DiscoverAadInstanceAsync(Uri endPoint, RequestContext requestContext)
         at Microsoft.Identity.Client.Instance.Discovery.NetworkMetadataProvider.SendInstanceDiscoveryRequestAsync(Uri authority, RequestContext requestContext)
         at Microsoft.Identity.Client.Instance.Discovery.NetworkMetadataProvider.FetchAllDiscoveryMetadataAsync(Uri authority, RequestContext requestContext)
         at Microsoft.Identity.Client.Instance.Discovery.NetworkMetadataProvider.GetMetadataAsync(Uri authority, RequestContext requestContext)
         at Microsoft.Identity.Client.Instance.Discovery.InstanceDiscoveryManager.FetchNetworkMetadataOrFallbackAsync(RequestContext requestContext, Uri authorityUri)
         at Microsoft.Identity.Client.Instance.Discovery.InstanceDiscoveryManager.GetMetadataEntryAsync(AuthorityInfo authorityInfo, RequestContext requestContext, Boolean forceValidation)
         at Microsoft.Identity.Client.Instance.AuthorityManager.RunInstanceDiscoveryAndValidationAsync()
         at Microsoft.Identity.Client.Internal.Requests.ClientCredentialRequest.FetchNewAccessTokenAsync(CancellationToken cancellationToken)
         at Microsoft.Identity.Client.Internal.Requests.ClientCredentialRequest.ExecuteAsync(CancellationToken cancellationToken)
         at Microsoft.Identity.Client.Internal.Requests.RequestBase.RunAsync(CancellationToken cancellationToken)
         at Microsoft.Identity.Client.ApiConfig.Executors.ConfidentialClientExecutor.ExecuteAsync(AcquireTokenCommonParameters commonParameters, AcquireTokenForClientParameters clientParameters, CancellationToken cancellationToken)
         at Azure.Identity.AbstractAcquireTokenParameterBuilderExtensions.ExecuteAsync[T](AbstractAcquireTokenParameterBuilder`1 builder, Boolean async, CancellationToken cancellationToken)
         at Azure.Identity.MsalConfidentialClient.AcquireTokenForClientAsync(String[] scopes, Boolean async, CancellationToken cancellationToken)
         at Azure.Identity.ClientSecretCredential.GetTokenAsync(TokenRequestContext requestContext, CancellationToken cancellationToken)

My initial thought was (being inexperienced with Docker) that I had a network configuration problem, and that the internet was not accessible to my containers or that they needed DNS configuration.

However, I can ping google.com just fine, I can use a System.Net.Http.HttpClient to pull the HTML from the google home page too. When the call to KeyVault fails, so too will an HttpClient attempt to pull HTML from login.microsoftonline.com. Container DNS and internet connectivity seems to be okay outside of this one hostname.

After much frustration yesterday, it finally started working before I was done for the day, and I thought I had fixed the problem (I changed the network defined in my docker-compose to a user-defined bridge, and magically it started working. Today I realized I defined the network incorrectly and that my change had not in fact connected the containers to my bridge). Worse, it was broken again this morning, having changed nothing since it had started working last night.

Then around 12PM ET today, it worked again. Briefly. My smoke-testing HttpClient pulled the HTML from login.microsoftonline.com and the KeyVault call succeeded! Then it failed again. Then it worked again! And now it has done nothing but fail.

Meanwhile I cannot replicate the error by running the app outside of the container at all, seemingly ruling out a problem with my internet connection or a widespread issue with login.microsoftonline.com.

I'm at my wits' end here and I'm out of ideas for further diagnosis. The issue seems to be limited exclusively to this one hostname, exclusively to my containers, and not the host.

I'm not even sure this is the right place to be asking, I sort of doubt the images I'm using have anything special in them that would cause this problem, but it doesn't appear to be a bug in the Azure SDK and it doesn't appear to be an issue with my internet, DNS, or container networking configuration. I don't see any service alerts on Microsoft's end either.

Any ideas?

UPDATE:

I changed from WSL2 back-end to Hyper-V and the issue persists. Changed back to WSL2 and it worked once, then back to failing endlessly.

Running host from the container CLI resolves login.microsoftonline.com just fine. So does dig.

Quite infuriatingly, the below code with a breakpoint in each exception handler only breaks on the login.microsoftonline.com block:

System.Net.IPHostEntry googleEntry;
System.Net.IPHostEntry msLoginEntry;

try { googleEntry = System.Net.Dns.GetHostEntry("google.com"); }
catch (Exception ex)
{

}

try { msLoginEntry = System.Net.Dns.GetHostEntry("login.microsoftonline.com"); }
catch (Exception ex)
{

}

googleEntry is returned as expected. "Name or service not known" thrown for msLoginEntry.

[scotmeiste]

@ericmqt I'm running into this exact issue... Did you ever resolve it?

[ericmqt]

Nope, I gave up shortly after posting this. However, I had also posted a discussion on reddit about the issue, and it looks like I missed a couple of replies that came a month later claiming to have a workaround: 1) setting DNS in docker-compose.yml and 2) running Restart-NetAdapter -Name "vEthernet (WSL)" on the host.

I haven't tested those yet myself, but they're worth giving a shot!

[scotmeiste]

FWIW - I got around it with this solution at the WSL level. Had to restart WSL and Docker before it took effect though.

microsoft/WSL#5991 (comment)

[ericmqt]

I finally had some time to revisit Docker support in our solution and, after making some changes to my original attempt at configuring Docker, I didn't run into this issue at all.

In re-reading the issue I posted, I failed to mention my container was set up behind an nginx reverse proxy, also in a container, both of which were to run under docker-compose. I think, at the time, this didn't seem relevant, because I figured it was a more fundamental networking problem.

The biggest difference between my initial attempt and the solution that finally worked, besides moving to Windows 11 and not having Hyper-V installed, was removing the upstream section from my nginx reverse proxy and setting proxy_pass to use the docker-compose service's hostname.

Are you running behind a reverse proxy, by any chance? My new hypothesis is in my misconfiguration of nginx I somehow wound up with login.microsoftonline.com:443 requests being black-holed by nginx, with requests only succeeding when my app container started before nginx had a chance to bind to the network interface. Clearly I did not understand docker networking to the extent I needed to, so this would not be surprising to me and would seem to fit the issue originally encountered.

If you're not behind a reverse proxy then I'm at a loss as to how this happened or why it's working for me now.

[scotmeiste]

Unfortunately, no we are not running a reverse proxy. its actually a pretty basic setup. The only other factor I can think of is that I installed a Docker Desktop update in the middle of troubleshooting. It could be possible that the update fixed it as well.

I was going to try and undo my work around to see if the issue came back, but for some reason, i can no longer start a WSL session. another thing to troubleshoot for another day... :)