Feature/lean kernel exposure

[DominicDirkx]

Copy of #152, which was undone due to failures on Azure

This PR simplifies the tudat kernel exposure in tudatpy:

Exposure
Explicit kernel exposure with Python wrappers deprecated in favor of exposure by modifying sys.path after importing the compiled tudatpy.kernel module

Autocompletions
Achieved by the use of stubs, generated at compile-time using mypy.stubgen

[alopezrivera]

Kernel exposure now done with import hooks

Functionality:

• Importing kernel modules from tudatpy
• In the case of hybrid modules -modules with both kernel (C++) and Python source code-, allow Python to search for attributes in both the kernel and Python sources

Characteristics:

• No need for ahead-of-time import of entire kernel: if at some point the kernel is split into submodules this will work, so long as kernel modules are still named tudatpy.kernel.<>
  • Extending the implementation to multiple kernels (kernel to be split into kernel_astro, kernel_numerical_simulation etc. to reduce kernel load time) is trivial
• Python import system modified to intercept Tudatpy imports on-the-fly and 1. expose kernel modules 2. create hybrid Python-C++ modules

Background

Exposure is implemented through an import hook. An import hook is composed of:

• A module finder: the purpose of which is to intercept tudatpy import statements
• One or many module loaders: the purpose of which is to explicitly 1. create a module object for the imported module and 2. populate the module by executing its source

Tale of an import

1. The Python import system scans sys.modules for the imported module. If it exists, it is returned. Else:
2. The Python import system loops through sys.meta_path, a list of importlib.abc.MetaPathFinder objects. These objects have a method called find_spec: this function is responsible for creating a package specification for the object; if it cannot, it will return None, and the next finder in sys.meta_path will be tried out until:
   • Any of the finders in sys.meta_path return a package specification (an object that is not None)
   • A ModuleNotFound error is raised, in the case in which all known finders return None

Once a package specification has been created, the following takes place:

1. The module is created
2. The partly initialized module is added to sys.modules
3. The source of the module is executed

module = importlib.util.module_from_spec(spec)
sys.modules[module_name] = module
spec.loader.exec_module(module)

Complete pseudocode

import sys
import importlib

module_name = 'example_module'

# Step 1: Check built-in modules
if module_name in sys.builtin_module_names:
    module = importlib.import_module(module_name)
    return module

# Step 2: Check sys.modules
if module_name in sys.modules:
    return sys.modules[module_name]

# Step 3: Find the module
for finder in sys.meta_path:
    spec = finder.find_spec(module_name)
    if spec is not None:
        break
else:
    raise ModuleNotFoundError(f"No module named '{module_name}'")

# Step 4: Load the module
module = importlib.util.module_from_spec(spec)
sys.modules[module_name] = module
spec.loader.exec_module(module)

Implementation

• Finder: TudatpyModuleFinder
• Loaders:
  • KernelModuleWrapper
  • HybridModuleLoader

Mechanics

The entire process takes place through TudatpyModuleFinder (used by the import system by calling: TudatpyModuleFinder.find_spec):

1. Intercepts tudatpy imports
2. Determines whether the imported module is (a) a Tudatpy Python module (b) a wrapped Tudat kernel module (ie: tudatpy.math, which corresponds to tudatpy.kernel.math) (c) both
   1. (a) Defer loading to the default import system mechanism
   2. (b) Use the KernelModuleWrapper loader to create a module called tudatpy.math that wraps tudatpy.kernel.math
   3. (c) Use the HybridModuleLoader loader to
      1. Create and load the Tudatpy Python module
      2. Assign the Python module's __getattr__ method to a function called getattr_from_hybrid_module, which will attempt to find attributes missing from the Python module in the associated kernel module

[DominicDirkx]

@alopezrivera, I think the same modification will need to be made for other 'hybrid' modules as well (right not, it's only in gravity_field, right?). Is there a way to automate this to prevent issues with other (and future) hybrid modules?

[alopezrivera]

Assuming you refer to 48fe01c, the change is cosmetic and everything would work even if we kept the explicit import of kernel objects. I removed it because the exposure makes that import unnecessary.

More generally, the only requirement for tudatpy Python code is that tudatpy subpackages (aka folders under tudatpy that contain .py files, such as tudatpy/numerical_simulation) must have an __init__ (doesn't matter if it's empty or not).

[DominicDirkx]

Closing, since all this is now part of #159