Define the problem set for distros/instrumentation/SDK-configuration

[ocelotl]

As discussed in the last SIG, here is a list of the problems that we have right now that are directly or indirectly related to the distro/instrumentation/SDK-configuration topic. The problems listed below are in no particular order, nevertheless, some problem solutions may be independent from other solutions, some other solutions for certain problems may require that other problems be solved first:

1. Distros are coupled with instrumentation
2. We need to define a mechanism to select a distro among many
3. We need to define a mechanism to configure the SDK
4. We have Celery-specific code in our instrumentation package

The intention of this discussion is only to make a complete list of the problems of this kind. Please add here in the comments any other problem that is missing from this list that is also related directly or indirectly to the distro/instrumentation/SDK-configuration topic, I'll update this list with the mentioned problems. Please add any comments related to the specific possible solutions for these problems in their respective PR, issue or discussion.

Distros are coupled with instrumentation

Right now, distros are coupled with instrumentation in several ways:

1. We have a distro base class in our instrumentation package.
2. We use opentelemetry-distro entry points in our auto instrumentation process.
3. We require a distro to be installed in order to be able to use auto instrumentation.

I consider this to be a problem because distros are a way of making configuration happen before auto instrumentation is run, but they are only one of many ways that this can be achieved. Users may want to perform their configuration with configuration files, or environment variables or any other mechanism they prefer.

These are the PRs that attempt to solve this problem:

1. Add pre and post instrument entrypoints #1983

We need to define a mechanism to select a distro among many

While there will be many distros, we want to select only one to be applied. Right now, the first distro loaded from the opentelemetry_distro entry point will be selected. This is a problem because it is not deterministic which distro will be the first one in this case, so if the user has more than one distro installed, any one can end up being selected.

These are the PRs that attempt to solve this problem

1. PR 571 in contrib
2. Refactor distros #1966
3. Not a PR, but @owais has explained another solution for this problem

We need to define a mechanism to configure the SDK

The spec has a document for the configuration of the SDK. Basically, we need to implement a feature for this to be spec compliant. What we have right now is the code that was added by #1937 which moved into the SDK several functions that can be used to configure the SDK. These functions were added in a private _configuration module to move forward with this PR but we have a pending task to find a final solution because we are going against Python standard practice by importing symbols from this private module.

Right now there are no open PRs that attempt to solve this problem. There is an important disclaimer, though: we may decide that distros are to be used to configure the SDK. I am not considering #1983 as a solution to this problem even when it introduces entry points that could be used to configure the SDK because I consider that PR to be focused on decoupling distros from instrumentation instead.

We have Celery-specific code in our instrumentation package

Right now we have Celery-specific code in our auto instrumentation process. This should not be, because the auto instrumentation process should be completely decoupled from any third-party library.

Right now there are no open PRs that attempt to solve this problem. Again, there is a similar disclaimer: #1983 introduces general purpose entry points that can be used to load a callable that handles this Celery-specific code and solves this problem. For the same reason as above, though, I am not considering #1983 as a solution for this problem.

[owais]

Thanks for the detailed explanation Diego. I think problem 2. should be restated as

We need to allow distro authors to use other distros as dependencies to allow code-reuse

as this was the original problem that came up. IMO the title for problem 2. as you've stated is one of the viable solutions to the problem but not the problem itself.

[owais]

In addition to Celery, we might also want to attempt to solve gunicorn, uwsgi which do not work today unless users add custom framework specific SDK initialization code. I'm not saying solving gunicorn and uwsgi auto-instrumentation should be a requirement for this but thinking about it might help make the solution more generic instead of something that only works for celery.

[ocelotl]

Thanks for the detailed explanation Diego. I think problem 2. should be restated as

We need to allow distro authors to use other distros as dependencies to allow code-reuse

as this was the original problem that came up. IMO the title for problem 2. as you've stated is one of the viable solutions to the problem but not the problem itself.

I think that is not a problem with the code as it is right now. In fact, we currently use other distros to reuse code. Right now someone may create a package that implements a new distro, UserDistro which inherits from OpenTelemetryDistro, calls super in _configure and then adds some more configuration, reusing the code from OpenTelemetryDistro._configure.

The problem right now is not not being able to reuse distro code, but the fact that if there are multiple distros installed, it is not known which one will be selected by the auto instrumentation process.

[ocelotl]

In addition to Celery, we might also want to attempt to solve gunicorn, uwsgi which do not work today unless users add custom framework specific SDK initialization code. I'm not saying solving gunicorn and uwsgi auto-instrumentation should be a requirement for this but thinking about it might help make the solution more generic instead of something that only works for celery.

Actually, the solution that I had in mind is not one that works only for Celery but for anything else. The solution for Celery would be to create a callable that implements opentelemetry-pre-instrument and does its Celery business when called by the auto instrumentation process. This solution uses a general purpose entry point which can be the same approach we use for any other thing, including gunicorn.

[owais]

Sounds great. Just wanted update the description for people participating think broader and not just about celery. Thanks

[owais]

I think we should approach these problems in a different order. Before anything else, we should try to figure out the answer to problem 3.

We need to define a mechanism to configure the SDK

Once we know the answer to this, we'll know whether de-coupling instrumentation and distro packages makes sense (most likely still does) and how the decoupling should exactly work. Can distro take dependency on SDK? Can may be the SDK include the base distro class? A lot of these questions will be answered if we solve 3. first. Once we have these answers, it'll be easy to solve 1. (decoupling) and possibly we might not even need a solution for 3 (code re-use between distros). 4 (celery special case) IMO is an instrumentation related issue. Distro design can definitely affect it but I think it can be retrofitted for almost any design so I'd keep it for the last.

[ocelotl]

I think we should approach these problems in a different order. Before anything else, we should try to figure out the answer to problem 3.

We need to define a mechanism to configure the SDK

Sorry, I did not intend to imply any particular order to solve these problems. I have updated the issue description to clarify that

Once we know the answer to this, we'll know whether de-coupling instrumentation and distro packages makes sense (most likely still does) and how the decoupling should exactly work.

I don't think it is necessary to know how to configure the SDK to decide if decoupling instrumentation from distro makes sense. I think decoupling instrumentation from distro makes sense independently of anything else and it makes sense because distro is just another configuration mechanism that our users may want to use (or not) when running auto instrumentation.

We could decouple distro from instrumentation right now and it won't affect the decision on how to configure the SDK. We may choose to use a distro to configure the SDK or we may choose to do something else, the decision we take regarding decoupling distros from instrumentation won't affect this.

Can distro take dependency on SDK?

A specific implementation of a distro could. A base distro package should not:

1. It does not look like it would need to, a distro base package would just provide a base distro class with a configuration method (very similar to what we have now).
2. We should aim to make the base distro available to any implementation of an SDK

Can may be the SDK include the base distro class?

I disagree for the same reason as stated above, distros should be decoupled from instrumentation and anything else. Also, distros are something that could perfectly be used by an user who uses any implementation of the SDK.

[owais]

Sorry, I did not intend to imply any particular order to solve these problems. I have updated the issue description to clarify that

Yes of course. Did not mean that you did. Just that it could be clearer :)

[aabmass]

Do the pre and post instrument entrypoints added in #1983 address problem #3 then? It gives a generic hook where users/distros/whatever can override SDK configuration before auto instrumentation. What else is needed?

[ocelotl]

Do the pre and post instrument entrypoints added in #1983 address problem #3 then? It gives a generic hook where users/distros/whatever can override SDK configuration before auto instrumentation. What else is needed?

I don't consider #1983 as a solution for the third problem because the specific mechanism to configure the SDK has not yet been decided (making this decision is the third problem). Using the entry points provided by #1983 could be the solution but other solutions could be decided as well.