[feature] Middleware for customization of all recipes

[gmeeker]

Allow users to inject code to modify all recipes, allowing users to run additional commands in all build() and package() methods. One use case would be code signing all executables. The ultimate goal would be support for multiple architectures; see below. This feature request should cover only the core functionality of middleware support.

Because of the complexity of this idea, I've been experimenting with coding prototypes rather than abstract designs, to see how they play out in practice. I think the implementation in #9637 is finally ready for feedback. This conceptually easier to understand than previous implementations (which created wrappers around classes. It's more isolated and doesn't touch the ConanFile model or the graph code. It handles the altered package ids and host/build profiles without extra work.

It should be noted that some of this can implemented with hooks, but the result is pretty awful, and highly dependent on conan's internals. One such attempt:
https://github.com/gmeeker/conan-multi-build-hook

Extending conanfiles

A middleware is an adapter that can subclass any or all conanfiles, and can be chained with other middleware.

A complete example is here:
https://github.com/gmeeker/conan-codesign-middleware

class codesign(Middleware):
    settings = "os", "codesign_identity", "codesign_flags"
    options = {"identity": [None, "ANY"], "flags": [None, "ANY"]}
    default_options = {"identity": None, "flags": None}

    @property
    def identity(self):
        return self.settings.codesign_identity or self.options.identity

    @property
    def flags(self):
        return self.settings.codesign_flags or self.options.flags

    def should_apply(self, base):
        if not self.identity:
            return False
        return True

    def __call__(self, base):
        class CodeSignConan(base):
            # Some settings might only be used in 'should_apply' but we use all here
            settings = Middleware.extend(base.settings, codesign.settings)
            options = Middleware.extend(base.options, codesign.options)
            default_options = Middleware.extend(base.default_options, codesign.default_options)

            @property
            def identity(self):
                return self.settings.codesign_identity or self.options.identity

            @property
            def flags(self):
                return self.settings.codesign_flags or self.options.flags

            def package(self):
                super(CodeSignConan, self).package()
                self.run(...)

        return CodeSignConan

Loading middleware

Because middleware alters conanfile classes, it must be handled before any other loading. I propose a new profile field
[middleware] which references middleware by name (like generators). [middleware_requires] is necessary to load packages before [build_requires] processing.

[settings]
codesign_identity=My Company
[middleware_requires]
codesign/0.1@
[middleware]
codesign
[env]

Alternatively, middleware could be loaded from the local cache, much like generators. Copy to ~/.conan/middleware/codesign.py and remove the ConanFile part. You still need to enable the middleware in the profile.

[settings]
codesign_identity=My Company
[middleware]
codesign
[env]

It's also possible to constraint this to specific packages:

[middleware_requires]
 my_pkg*: codesign/0.1@
 &: codesign/0.1@
 &!: codesign/0.1@
[middleware]
 my_pkg*: codesign
 &: codesign
 &!: codesign

Settings

As shown in the example above, settings and options work like any conanfile. They can be applied to a specific middleware by name too. When a middleware creates a new subclass, it must mix in any settings or options that will be required.

Multiple architectures
While multi-arch is beyond the scope of this initial feature request, I think it's important to show why this is desirable, and possibly to influence the discussion of settings and configuration. This idea was born out of the desire to support multi-arch and previous discussion around an all-in-one feature seems to have stalled.

conan-io/conan-extensions#59
#6384

By keeping all tool specific code outside Conan, this middleware proposal allows user customization without the maintenance or compatibility concerns of trying to implement multi-arch directly. I believe it can encourage development of experimental features without requiring familiarity with Conan's source code.

An earlier middleware and multi-arch attempt is here in the middleware branch, which is now substantially different. I will be adapting this to use this new middleware approach.
[https://github.com/gmeeker/conan]

macOS/iOS multi-arch middleware is here, using pyreq to provide settings:
[https://github.com/gmeeker/conan-lipo-middleware]

(I'm not familiar with Android, but it looks like there are similar issues there as well.)

Previous implementation in #9637 was very different. Middleware behaved like a conanfile and the result looked like this. This lead to confusion between conanfile settings and middleware settings.

class codesign(Middleware):
    settings_middleware = "os", "codesign_identity"

    def valid(self):
        # Is this middleware relevant for this conanfile?
        return True

    def package(self):
        self.conanfile.package()
        self.run(...)

class CodesignPackage(ConanFile):
    name = "codesign"
    version = "0.1"

• [ X] I've read the CONTRIBUTING guide.

[memsharded]

Closing because of #9859 (comment)