gRPC Sender Help

[js8080]

I'm trying to get a .NET Core application to use the gRPC Sender in this library to bypass the Jaeger Agent and go straight to the Collector via gRPC.

I already have an Agent working and a Collector where the Agent is communicating to the Collector via gRPC and using Mutual TLS (mTLS) but now I am trying to bypass the Agent and use mTLS straight to the Collector via gRPC.

I have my own configuration mechanism and do not use environment variables so I am manually setting up my gRPC Sender in my Startup.ConfigureServices() as follows:

ChannelCredentials grpcCredentials = new MutualTlsChannelCredentials(tracingOptions);
sender = new Jaeger.Senders.Grpc.GrpcSender(tracingOptions.CollectorEndpoint, grpcCredentials, (int)tracingOptions.MaxPayloadSize);

var reporter = new RemoteReporter.Builder()
    .WithMetrics(metrics)
    .WithLoggerFactory(loggerFactory)
    .WithSender(sender)
    .Build();

var initialSampler = new ConstSampler(true);
var sampler = new RemoteControlledSampler.Builder(serviceName)
    .WithMetrics(metrics)
    .WithLoggerFactory(loggerFactory)      // optional, defaults to no logging
    .WithInitialSampler(initialSampler)    // optional, defaults to ProbabilisticSampler(0.001D)
    .WithPollingInterval(pollingInterval)  // optional, defaults to TimeSpan.FromMinutes(1)
    .WithSamplingManager(samplingManager)  // optional, defaults to HttpSamplingManager("localhost:5778")
    .Build();

OpenTracing.ITracer tracer = new Jaeger.Tracer.Builder(serviceName)
    .WithMetrics(metrics)
    .WithLoggerFactory(loggerFactory)
    .WithReporter(reporter)
    .WithSampler(sampler).Build();

The MutualTlsChannelCredentials class is an implementation of ChannelCredentials:

    public class MutualTlsChannelCredentials : ChannelCredentials
    {
        private readonly TracingOptions _tracingOptions;

        public MutualTlsChannelCredentials(TracingOptions tracingOptions)
        {
            _tracingOptions = tracingOptions;
        }

        public override void InternalPopulateConfiguration(ChannelCredentialsConfiguratorBase configurator, object state)
        {
            KeyCertificatePair clientCertKeyPair = new KeyCertificatePair(_tracingOptions.GrpcSenderClientCertChain, _tracingOptions.GrpcSenderClientPrivateKey);
            configurator.SetSslCredentials(state, _tracingOptions.GrpcSenderServerRootCertifcates, clientCertKeyPair, VerifyPeerCallback);
        }

        private bool VerifyPeerCallback(VerifyPeerContext context)
        {
            // TDB -- implement this.
            return true;
        }
    }

As you can see from this code, I read all my settings from this TracingOptions class which is just a POCO with various properties that I read from my own configuration source.

Now the issue is, this all works great for when I am using the UDP Agent and Thrift UDP Sender but as soon as I use the gRPC Sender like I am attempting in this code above, it doesn't actually report any Spans to the Collector and I have no idea why. There doesn't appear to be any logging as I have the "Jaeger" logging level turned up to "Trace" and still I see nothing. Looking at the source, there appears to be no logging happening.

I also noticed that my VerifyPeerCallback is never being called.

How can I figure out what is going wrong without logging?

[Falco20019]

Hi, thanks for your report. The GrpcSender has no real logging since it's a very slim wrapper. The real logging happens in the RemoteReporter or in gRPC logging.

The default endpoint is localhost:14250. How does your target look like? If has to be a valid gRPC target in any resolvable form, pointing to the collector. Are you sure that the port is accessible?

You can enable tracing on the gRPC level by setting up the ConsoleLogger or TextWriterLogger using the GrpcEnvironment. That's part of gRPC and not our library. This also needs to be done by setting environment variables (possible from code) as documented here:

• https://grpc.github.io/grpc/csharp/api/Grpc.Core.GrpcEnvironment.html#Grpc_Core_GrpcEnvironment_SetLogger_Grpc_Core_Logging_ILogger_
• https://grpc.github.io/grpc/csharp/api/Grpc.Core.Logging.ConsoleLogger.html#Grpc_Core_Logging_ConsoleLogger__ctor
• https://github.com/grpc/grpc/blob/master/TROUBLESHOOTING.md

Another common issue is that the application gets shutdown too quick when testing. Make sure to call Dispose on the tracer before stopping the application to gracefully try sending the pending spans. Otherwise, they might only be enqueued for reporting. You will definitely see that when checking the logs written by the RemoteReporter. gRPC usually only established the connection on the first real send data (not when creating the client stub). So the handshake would also only appear when the sender is flushed and not only appended.

[js8080]

I am also adding OpenTracing dependencies in my ConfigureServices:

services.AddOpenTracingCoreServices(otBuilder =>
{
	otBuilder.AddAspNetCore()
		.ConfigureAspNetCore(options =>
		{
			// This sets the main-level Span Operation Name which normally has a less than useful value in my experience.
			options.Hosting.OperationNameResolver = (httpContext) =>
			{
				return httpContext.Request.Path.Value;
			};
		})
		.AddHttpHandler()
		.AddEntityFrameworkCore()
		.AddLoggerProvider();
});

Also, I tried creating a composite reporter like this:

var remoteReporter = new RemoteReporter.Builder()
	.WithMetrics(metrics)
	.WithLoggerFactory(loggerFactory)
	.WithSender(sender)
	.Build();

var loggingReporter = new LoggingReporter(loggerFactory);

var reporter = new CompositeReporter(remoteReporter, loggingReporter);

And setting up the GrpcEnvironment logger:

GrpcEnvironment.SetLogger(new Grpc.Core.Logging.ConsoleLogger());

But I still get nothing in my console output...

I also tried setting GRPC_VERBOSITY to DEBUG in my launch.json and still nothing :(

[js8080]

I had my env vars set in the wrong launch.json -- now I am getting some output...

[js8080]

OK, one error I found is:

E0706 08:31:01.856123 0 T:\src\github\grpc\workspace_csharp_ext_windows_x64\src\core\tsi\ssl_transport_security.cc:777: Invalid cert chain file.
E0706 08:31:01.856174 0 T:\src\github\grpc\workspace_csharp_ext_windows_x64\src\core\lib\security\security_connector\ssl\ssl_security_connector.cc:113: Handshaker factory creation failed with TSI_INVALID_ARGUMENT.
E0706 08:31:01.856187 0 T:\src\github\grpc\workspace_csharp_ext_windows_x64\src\core\ext\transport\chttp2\client\secure\secure_channel_create.cc:132: Failed to create secure subchannel for secure name 'jaeger-collector.container-dev:443'
E0706 08:31:01.856194 0 T:\src\github\grpc\workspace_csharp_ext_windows_x64\src\core\ext\transport\chttp2\client\secure\secure_channel_create.cc:50: Failed to create channel args during subchannel creation.

I had my certificate chain specified via configuration as a PEM certificate without the header and trailer:

"MIIDP...(ommitted)...OuuOTA=="

When it should have included the header and trailer:

"-----BEGIN CERTIFICATE-----\nMIIDP...(ommitted)...OuuOTA==\n-----END CERTIFICATE-----"

Now I am seeing spans in my collector! I'm a little confused because I had the cert chain set this way to begin with and it wasn't working but I don't know if I changed something else without realizing.

[Falco20019]

Glad that you found the cause! Since replies on threads can't be marked as answer, I will mark the post it belongs to with the instructions in case someone else has similar problems.

[js8080]

Thank you for all the help. Another piece of useful info... I also set this env var in my launch.json:

"GRPC_TRACE": "api,connectivity_state,client_channel_call,client_channel_routing,channel"