[DISCUSS] Finalize configuration behaviors within remote kernel providers

[kevin-bates]

Since providers are now "detached" from the hosting application (i.e., dynamically loaded at runtime) the "flow" of configurables is prevented. In addition, one of the "goals" of kernel providers was to remove their dependence on traitlets. However, the fact remains that some providers require configuration. For example, the YarnKernelProvider needs the YARN RM endpoint (if not on an edge node), KubernetesKernelProvider needs to know if it should create namespaces for each kernel or use the hosting application's namespace, etc.

One approach to resolve these is to map each configurable item to an environment variable so long as reasonable defaults are in place. (Enterprise Gateway uses this approach to manage its default values.) . However, this can become unwieldy.

Another approach is to actually use the traitlets but in an "unbound" manner. Since Application is a singleton, a kernel provider can access that singleton instance, then look for its "section" based on either its class name or, more likely, a fixed string like "RemoteKernelProvider" that 'acts' as the "section" or "container" for the configurable items. The following shows how that "section" would be accessed...

    def _get_app_config(self):
        """Pulls application configuration 'section' relative to current class."""

        app_config = {}
        parent_app = Application.instance()
        if parent_app:
            # Collect config relative to our class instance.
            app_config = parent_app.config.get('RemoteKernelProvider', {}).copy()
        return app_config

and then used. In this case, the "global" port-range configurable is accessed...

        port_range = self.kernel_manager.app_config.get('port_range', '0..0') 

We could also replace the default value (0..0 in the example) with something like the following...

        default_port_range = os.getenv('RKP_PORT_RANGE', '[0..0]')
...
        port_range = self.kernel_manager.app_config.get('port_range', default_port_range) 

With this approach, admins could still define the configurable item in the application's configuration file...

    c.RemoteKernelProvider.port_range = '[31000..32000]'

but no type validation would be performed, nor would commands like jupyter <application> --generate-config include the available configurable items. As a result, users would need to rely on documentation or let each provider's "application" support its own --generate-config defaults - which could be included in the application's configuration file.