Add tracing for VC calls (OpenTelemetry support)

[fstrudel]

Is your feature request related to a problem? Please describe.
We're calling govmomi in our code with tracing and it would be good to continue the trace when govmomi calls are made and they call back into VC.

Describe the solution you'd like
We'd like to have govmomi support OpenTelemetry and add spans when a call is made to VC.
Using go.opentelemetry.io/otel would be ideal.

Describe alternatives you've considered
OpenTelemetry being hopefully the standard, we have not considered other alternatives.

cc @mayankbh

[github-actions]

Howdy 🖐   fstrudel ! Thank you for your interest in this project. We value your feedback and will respond soon.

If you want to contribute to this project, please make yourself familiar with the CONTRIBUTION guidelines.

[dougm]

govmomi has an option to set the opID for each API call, which VC will include in its log messages. This was added to support tracing. See for example: https://github.com/vmware/vic/blob/master/pkg/trace/operation.go#L77

[github-actions]

This issue is stale because it has been open for 90 days with no
activity. It will automatically close after 30 more days of
inactivity. Mark as fresh by adding the comment /remove-lifecycle stale.

[changemyminds]

Can we reopen this issue, as I also have this question? @dougm

I think with just a little modification, we can add extra value to the context by integrating the go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp library. This would allow us to track the VC calls with OpenTelemetry but I'm not sure this is a good ideal.

For example:

package main

import (
	"context"
	"fmt"
	"log"
	"net/http"
	"net/url"
 
	"github.com/vmware/govmomi/session"
	"github.com/vmware/govmomi/vim25"
	"github.com/vmware/govmomi/vim25/soap"
	"go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"
	"go.opentelemetry.io/otel/attribute"
	"go.opentelemetry.io/otel/trace"
)

func main() {
    // init OpenTelemetry Trace
    
    u, err := url.Parse("YOUR_VCENTER_URL")
    if err != nil {
	   log.Fatal(err)
    }
	
    ctx, cancel := context.WithCancel(context.Background())
    defer cancel()

    // create SOAP client
    soapClient := soap.NewClient(u, true)
    vc, err := vim25.NewClient(context.Background(), soapClient)
    if err != nil {
        log.Fatal(err)
    }
    
    // wrapper the original vc.Client.Client.Transport
    vc.Client.Client.Transport = otelhttp.NewTransport(
        &OtelRoundTripper{Base: vc.Client.Client.Transport},
        otelhttp.WithSpanNameFormatter(func(operation string, r *http.Request) string {
            // get the VC method name
            value := r.Context().Value("operation")
            if value != nil {
                if operation, ok := value.(string); ok {
                    return operation
		}
	    }
            return "HTTP " + r.Method
        }))

    // run Authenticated
    sm := session.NewManager(vc)
    err = sm.Login(ctx, u.User)
    if err != nil {
        log.Fatal(err)
    }
}

type OtelRoundTripper struct {
	Base http.RoundTripper
}

func (rt *OtelRoundTripper) RoundTrip(req *http.Request) (*http.Response, error) {
	ctx := req.Context()
	tracer := trace.SpanFromContext(ctx)

	// add extra values
	tracer.SetAttributes(
		attribute.Int64("http.request.content_length", req.ContentLength),
	)
	return rt.Base.RoundTrip(req)
}

Modifed sm.Login(ctx, u.User) internal method

func Login(ctx context.Context, r soap.RoundTripper, req *types.Login) (*types.LoginResponse, error) {
	var reqBody, resBody LoginBody

	reqBody.Req = req
       
        // Added additional values ​​to identify VC call API methods
	if err := r.RoundTrip(context.WithValue(ctx, "operation", "govmomi Login"), &reqBody, &resBody); err != nil {
		return nil, err
	}

	return resBody.Res, nil
}

We can view the results on the Jaeger Dashboard and they look good.

[dougm]

Thanks @changemyminds , re-opened and will ping some folks to discuss