Transformations: Re-organise inline and extract sub-packages

[mlange05]

This change creates two new separate transformation sub-packages loki.transformations.inline and loki.transformations.extract that are conceptual complements. For the inline sub-package, this allows us to separate the different types of inlining (constants, small functions and full procedures), and the associated tests. For "extract", this change brings together the two disparate current implementations (formerly extract_contained and region_to_call), and provides the counter-passes to the respective inlining operations as extract_internals and extract_marked. I've also added a wrapper ExtractTransformation class that provides both functionalities for batch processing.

Please note that, other than a minor bug-fix, I've left the implementations untouched. Further consolidation and additional features are planned, but I wanted to get the diff-heavy stuff out of the way first.

In some more detail:

• Created separate loki.transformations.inline sub-package with constants/functions/mapper/procedures modules. Importantly, I also split the test accordingly and localised it!
• Renamed extract_contained to extract_internals for consistency and created new loki.transformations.extract sub-package.
• Renamed region_to_call to extract_marked and moved to loki.transformations.extract.
• Added a new ExtractTransformation as batch-processing entry point. Importantly, this works on Module and Sourcefile objects only, as that defines where the newly created procedures go. Also added tests for this
• A small bug-fix for extract_internals and a few linter-compliance fixes for the transformation standalone tests. (Turns out pylint does not recurse into test directories, as there is no automatic package import 😉 )

[github-actions]

Documentation for this branch can be viewed at https://sites.ecmwf.int/docs/loki/376/index.html

[codecov]

Codecov Report

Attention: Patch coverage is 97.75199% with 31 lines in your changes missing coverage. Please review.

Project coverage is 95.55%. Comparing base (1d95896) to head (06774ae).
Report is 20 commits behind head on main.

Files with missing lines	Patch %	Lines
loki/transformations/inline/mapper.py	53.33%	21 Missing ⚠️
loki/transformations/inline/functions.py	97.43%	4 Missing ⚠️
loki/transformations/inline/procedures.py	97.26%	4 Missing ⚠️
loki/transformations/extract/outline.py	98.30%	1 Missing ⚠️
loki/transformations/inline/constants.py	96.87%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #376      +/-   ##
==========================================
+ Coverage   95.54%   95.55%   +0.01%     
==========================================
  Files         188      199      +11     
  Lines       39468    39584     +116     
==========================================
+ Hits        37709    37825     +116     
  Misses       1759     1759              

Flag	Coverage Δ
lint_rules	96.39% <ø> (ø)
loki	95.54% <97.75%> (+0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[reuterbal]

Thanks, this is great! Makes for a much cleaner separation of functionality and the clean-up on the side in tests is much appreciated.

I left two minor comments: very minor nitpick and a question about naming.

[mlange05]

@reuterbal Ok, I've rebased this over recent master and renamed the outline method, flags and pragma names. The rebase was quite tricky (esp. splitting the new inline sub-package), so I'd appreciate a quick double-check that everything is still as it's supposed to be.

[reuterbal]

Thanks, I can't spot anything that may have come from the rebasing.

While looking through the changes again, I spotted an outdated docstring and a potentially unnecessary way of instantiating a visitor. Both don't come from this PR but maybe it's easy to fix them here?

Also, I've noticed that the InlineSubstitutionMapper is very much undertested (https://app.codecov.io/gh/ecmwf-ifs/loki/pull/376/blob/loki/transformations/inline/mapper.py), probably because it is only being used when inlining statement functions (and therefore the other code parts are never hit). I'm wondering if it would be easy to expand the test coverage, e.g., by adding a test with a statement function that uses a scalar and an inline call? I've left a suggestion that uses the existing test and modifies it accordingly, but have not actually tested that this works as expected.

[reuterbal]

I must have missed this before: This shouldn't be necessary, since the FindInlineCallsSkipInlineCallParameters is declared inside the function, we should be able to directly instantiate the retriever with the provided values for inline_elementals_only and functions.

[mlange05]

Yes, now I understand. Pushed a fix to this extend.

[reuterbal]

Many thanks!