Periodic socket exceptions

[dandenton]

Hi, we're occationally getting errors while publishing health checks. Are we missing something with configuration or should the default ElmahioAPI.Create's HttpContext be using Polly or something similar to provide automatic retries for transient failures?

Exception: Health check "Elmah.Io.AspNetCore.HealthChecks.ElmahIoPublisher" threw an unhandled exception after 1.8186ms
Exception stack trace:

System.Net.Http.HttpRequestException: An error occurred while sending the request.
 ---> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host..
 ---> System.Net.Sockets.SocketException (10054): An existing connection was forcibly closed by the remote host.
   --- End of inner exception stack trace ---
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource<System.Int32>.GetResult(Int16 token)
   at System.Net.Security.SslStream.ReadAsyncInternal[TIOAdapter](Memory`1 buffer, CancellationToken cancellationToken)
   at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1.StateMachineBox`1.System.Threading.Tasks.Sources.IValueTaskSource<TResult>.GetResult(Int16 token)
   at System.Net.Http.HttpConnection.<CheckUsabilityOnScavenge>g__ReadAheadWithZeroByteReadAsync|44_0()
   at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.DiagnosticsHandler.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
   at Elmah.Io.Client.HeartbeatsClient.CreateAsync(String id, String logId, CreateHeartbeat body, CancellationToken cancellationToken) in /_/src/Elmah.Io.Client/ElmahioClient.cs:line 802
   at Elmah.Io.AspNetCore.HealthChecks.ElmahIoPublisher.PublishAsync(HealthReport report, CancellationToken cancellationToken) in /_/src/Elmah.Io.AspNetCore.HealthChecks/ElmahIoPublisher.cs:line 79
   at Microsoft.Extensions.Diagnostics.HealthChecks.HealthCheckPublisherHostedService.RunPublisherAsync(IHealthCheckPublisher publisher, HealthReport report, CancellationToken cancellationToken)
   at Microsoft.Extensions.Diagnostics.HealthChecks.HealthCheckPublisherHostedService.RunAsyncCore(CancellationToken cancellationToken)
   at Microsoft.Extensions.Diagnostics.HealthChecks.HealthCheckPublisherHostedService.RunAsync()

[ThomasArdal]

You shouldn't experience those kind of errors very often. It looks like either the elmah.io API isn't available or there's some kind of connection problem.

There are a couple of things you can do. First, I would increase the default timeout in the client used to create heartbeats:

var api = ElmahioAPI.Create("API_KEY", new ElmahIoOptions
{
    Timeout = new TimeSpan(0, 0, 30)
});

The default timeout is 5 seconds which should be fine in most cases. But increasing it for scenarios like this where no user is (typically) waiting for a reply is a good approach.

The elmah.io client does not implement retries itself. But setting up a retry with Polly is a good idea. You can provide your own HttpClient when creating the client:

var api = ElmahioAPI.Create("API_KEY", options, httpClient);

Setting up a retry can be done in multiple ways. I copied and adjusted some code from here:

builder.Services
    .AddHttpClient("elmahio")
    .AddPolicyHandler(HttpPolicyExtensions
        .HandleTransientHttpError()
        .WaitAndRetryAsync(3, i => TimeSpan.FromSeconds(1)));

I hope that helps.