Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improved support for flags #3

Closed
5 tasks
hiker opened this issue Jul 12, 2024 · 1 comment
Closed
5 tasks

Improved support for flags #3

hiker opened this issue Jul 12, 2024 · 1 comment

Comments

@hiker
Copy link
Owner

hiker commented Jul 12, 2024

The new compiler class introduces a separate Flag class, in addition to the existing AddFlags and FlagsConfig.

The Flags and FlagsConfig classes should be combined into one, allowing conditional (AddFlags) and unconditional flags to be used in one object. Additionally to path-specific flags, we need to:

  • Support compiler-specific flags in general (likely move flags from the steps into the compiler class, so we can have different flags for intel and gnu etc)
  • Support compilation modes: fast_debug, release etc - different sets of flags.
  • have conditionals depending on compiler version
  • combination of compiler version and path
  • Need to be able to remove flags (esp. taking OpenMP into account, which might need to work tightly with the compiler object, since the name of the openmp flag is part of the compiler tool)
jasonjunweilyu pushed a commit that referenced this issue Jul 15, 2024
jasonjunweilyu pushed a commit that referenced this issue Jul 15, 2024
jasonjunweilyu pushed a commit that referenced this issue Jul 15, 2024
jasonjunweilyu pushed a commit that referenced this issue Jul 15, 2024
jasonjunweilyu pushed a commit that referenced this issue Jul 15, 2024
jasonjunweilyu pushed a commit that referenced this issue Jul 15, 2024
jasonjunweilyu pushed a commit that referenced this issue Jul 15, 2024
jasonjunweilyu pushed a commit that referenced this issue Jul 15, 2024
jasonjunweilyu pushed a commit that referenced this issue Jul 15, 2024
jasonjunweilyu pushed a commit that referenced this issue Jul 15, 2024
jasonjunweilyu pushed a commit that referenced this issue Jul 15, 2024
jasonjunweilyu pushed a commit that referenced this issue Jul 15, 2024
jasonjunweilyu pushed a commit that referenced this issue Jul 15, 2024
jasonjunweilyu pushed a commit that referenced this issue Jul 15, 2024
jasonjunweilyu pushed a commit that referenced this issue Jul 15, 2024
jasonjunweilyu pushed a commit that referenced this issue Jul 15, 2024
jasonjunweilyu pushed a commit that referenced this issue Jul 15, 2024
jasonjunweilyu pushed a commit that referenced this issue Jul 15, 2024
jasonjunweilyu pushed a commit that referenced this issue Jul 15, 2024
jasonjunweilyu pushed a commit that referenced this issue Jul 15, 2024
jasonjunweilyu pushed a commit that referenced this issue Jul 15, 2024
jasonjunweilyu pushed a commit that referenced this issue Jul 15, 2024
jasonjunweilyu pushed a commit that referenced this issue Jul 15, 2024
jasonjunweilyu pushed a commit that referenced this issue Jul 15, 2024
jasonjunweilyu pushed a commit that referenced this issue Jul 15, 2024
jasonjunweilyu pushed a commit that referenced this issue Jul 15, 2024
jasonjunweilyu pushed a commit that referenced this issue Jul 15, 2024
jasonjunweilyu pushed a commit that referenced this issue Jul 15, 2024
jasonjunweilyu pushed a commit that referenced this issue Jul 15, 2024
@hiker
Copy link
Owner Author

hiker commented Jul 26, 2024

I'm closing this since the PR/issue number from an unrelated issue turned up in this thread, making it confusing. I have created the same issue as #11.

@hiker hiker closed this as completed Jul 26, 2024
hiker added a commit that referenced this issue Aug 12, 2024
* Removed absolute path patch, since this breaks a test.

* Fixed failing tests.

* Fix styling problems reported by Flake8

* Fix "_metric_send_conn not set, cannot send metrics" warning not issued, as reported by CI pipeline on Github

* Remove cosmetic change done to follow other coding standards.

* Remove more cosmetic changes (which pylint required).

* More removal of cosmetic changes, and some useless code.

* Add --links option to rsync.

* Extend Fortran analyser to detect use statements with an OpenMP sentinel.

* Use existing variable.

* Remove unused glob import.

* Removed unused glob import.

* Fixed typo picked up in review.

* Fixed failing tests.

* Replaced _artefact_store with getter as much as possible.

* #3 Introduce Tool, Compiler, and ToolRepository.

* #3 Made ToolRepository a singleton.

* #3 Added ToolBox.

* #3 Added dedicated enum for categories.

* #3 Add the category to a tool.

* #3 Use tool category when adding a tool to the tool box.

* #3 Added preprocessor, and dictionary-like accesses to ToolBox.

* #3 Added tool_box as mandatory parameter for a BuildConfig.

* #3 Start to use new compiler objects for Fortran compilation.

* #3 Added tests for Compiler class.

* #3 Add specific C- and Fortran-compiler classes. Updated tests.

* #3 Fixed typing.

* #3 Moved function to remove into Flags object, added test file for Flags.

* # Fixed tests to work with new Fortran compiler handling.

* #3 Use  for compiler variables if defined.

* #3 Add is_available flag to tools, added types.

* #3 Added cpp as Fortran preprocessor.

* #3 Removed unnecessary code.

* #3 Properly ignore mypy warnings.

* Support ToolBox for preprocessing.

* #3 Add a specific 'preprocess' method to Preprocessor.

* #3 Changed order in which compiler flags are used.

* #3 Use toolbox in compile_c step.

* #3 Added get_hash function to compiler.

* #3 Added conftest.py file (with a fixture to create a C-compiler).

* #3 Introduce fixtures for Fortran compiler and tool_box.

* #3 Remove explicit compiler information from MpCommonArgs (since it's already part of the config toolbox).

* #3 Added linker as tool.

* #3 Added test for linking shared libraries.

* #3 Pass compiler flags to the linker if a compiler was specified.

* #3 Remove unused function.

* #3 Removed more unused code.

* #3 Automatically add a linker for each compiler.

* #3 Fixed typo.

* #3 Support vendor for compiler and linker.

* #3 Make linker having a vendor, too.

* #3 Add set_default_vendor method to tool repository.

* Ignore build directory for git.

* Updated test.

* # Fix some mypy errors and warnings.

* Avoid using get() for singleton, instead use __new__ which makes mypy happier.

* Changed the transformation_script parameter of function psyclone to accept a function that can return file-specific transformation scripts

* Make mypy happy by using patch.object.

* Remove more comments and confusion about mypy :)

* Try to make mypy happy on older python versions.

* Make flake8 happy.

* Sort imported name alphabetically.

* Try to fix failing hash test (and add some additional improvements in the test).

* Removed fpath= for input transformation_script function to pass mypy test for Python 3.7; Moved transformation_script_hash test to unit test from system test

* Fix mypy typing check errors for psyclone unit test

* Fix config typing issue with mypy in psyclone unit test

* Fix flake8 issues; Revert Config mypy typing fix

* Add comment to ignore typing check for fpath parameter of input transformation_script function

* Fix assert check after transformation_script function is changed from being called twice to once

* Filter out 'no transformation script' warning for psyclone system test

* Replace 'ignore' typing of fpath of transformation_script with removing keyword argument

* #3 Support proper tests to check if tools are available.

* 1. Updated transformation_script description; 2. Modified mock_transformation_script; 3.Removed redundant _analysis_for_prebuilds

* Updated lfric/atm.py and lfric/gungho.py examples to pass in transformation_script functions

* Added description for the psyclone step to instructions on writing a config

* #3 Added git as a tool.

* #3 Fix incorrect | usage in typing.

* #3 Added unit tests for git.

* #3 Renamed git.py to versioning.py, to avoid name clash with the corresponding test_git.py tests.

* #3 Converted svn and fcm to tools.

* #3 Fixed missing whitespace.

* Modified the documentation for writing a config with PSyclone

* Add config as a parameter for run_psyclone for the transformation_script to use;Updated the related functions and tests;
Changed the logic of the transformation_script examples

* #3 Replaced ar with tool object.

* #3 Added tests for ar.py.

* #3 Removed debug output.

* #3 Converted PSyclone to be a tool.

* #3 Removed debug print, fixed python 3.7 typing information.

* #3 Updated comments.

* Modified the get_optimisation_script function examples and updated the doc formatting

* #3 Add Rsync tool.

* #3 Removed now unused function.

* #3 Added test for rsync.

* #3 Fixed all mypy warnings about functions not checked.

* #3 Replace all mock-tests to use subprocess so the name of the executable is tested as well.

* #3 Remove duplicated flags.

* #3 Fixed changed order of linking.

* #3 Removed run_command function.

* #3 Fixed 3.8 typing error.

* #3 Fixed unused imports.

* #3 Move flags checksum into Flags, and remove now unused tools.py file.

* #3 Renamed newtools to tools.

* #3 Made custom function for all git functions called (instead of just calling run).

* #3 Updated and fixed comments.

* #3 Fixed errors in comments.

* Fixed minor errors in documentation.

* #3 Make it easier to create wrapper around standard compiler.

* #3 Added documentation for all tool related classes and their usage.

* #3 Added MISC category.

* Addressed reviewer's comments.

* Updated cli to properly use ToolBox etc, removing hard-coded gnu command linker option.

* Fixed mypy failures, including changes to import statement to avoid cyclic imports :(.

* #3 Fix circular import.

* Added #TODO so that this can be removed once fparser supports sentinels.

* Fix typing problems by ignoring fparser.

* Replaced more string names for artefacts with enums.

* Removed EXECUTABLES from constants.

* Moved Artefact class out of ArtefactStore and renamed it to ArtefactSet.

* Moved OBJECT_FILES from constants into ArtefactSet.

* Moved OBJECT_ARCHIVES from constants to ArtefactSet.

* Moved PRAGMAD_C from constants to ArtefactSet.

* Turned 'all_source' into an enum.

* Allow integer as revision.

* Fixed flake8 error.

* Removed specific functions to add/get fortran source files etc.

* Removed non-existing and unneccessary collections.

* Try to fix all run_configs.

* Fixed rebase issues.

* Added replace functionality to ArtefactStore, updated test_artefacts to cover all lines in that file.

* Started to replace artefacts when files are pre-processed.

* Removed linker argument from linking step in all examples.

* Try to get jules to link.

* Fixed build_jules.

* Fixed other issues raised in reviews.

* Try to get jules to link.

* Fixed other issues raised in reviews.

* Simplify handling of X90 files by replacing the X90 with x90, meaning only one artefact set is involved when running PSyclone.

* Make OBJECT_ARCHIVES also a dict, migrate more code to replace/add files to the default build artefact collections.

* Fixed some examples.

* Fix flake8 error.

* Fixed failing tests.

* Support empty comments.

* Fix preprocessor to not unnecessary remove and add files that are already in the output directory.

* Allow find_soure_files to be called more than once by adding files (not replacing artefact).

* Updated lfric_common so that files created by configurator are written in build (not source).

* Use c_build_files instead of pragmad_c.

* Removed unnecessary str.

* Documented the new artefact set handling.

* Fixed typo.

* Make the PSyclone API configurable.

* Fixed formatting of documentation, properly used ArtefactSet names.

* Support .f and .F Fortran files.

* Removed setter for tool.is_available, which was only used for testing.

* #3 Fix documentation and coding style issues from review.

* Renamed Categories into Category.

* Minor coding style cleanup.

* Removed more unnecessary ().

* Re-added (invalid) grab_pre_build call.

* Fixed typo.

* Renamed set_default_vendor to set_default_compiler_suite.

* Renamed VendorTool to CompilerSuiteTool.

* Also accept a Path as exec_name specification for a tool.

* Move the check_available function into the base class.

* Fixed some types and documentation.

* Fix typing error.

* Added explanation for meta-compiler.

* Improved error handling and documentation.

* Replace mpiifort with mpifort to be a tiny bit more portable.

* Use classes to group tests for git/svn/fcm together.

* Fixed issue in get_transformation script, and moved script into lfric_common to remove code duplication.

* Code improvement as suggested by review.

* Fixed run config

* Added reference to ticket.

* Updated type information.

* More typing fixes.

* Fixed typing warnings.

* As requested by reviewer removed is_working_copy functionality.

* Issue a warning (which can be silenced) when a tool in a toolbox is replaced.

* Fixed flake8.

* Fixed flake8.

* Fixed failing test.

* Addressed issues raised in review.

* Removed now unnecessary operations.

* Updated some type information.

* Fixed all references to APIs to be consistent with PSyclone 2.5.

* Added api to the checksum computation.

* Fixed type information.

* Added test to verify that changing the api changes the checksum.

* Provide the name of the PSyclone API for LFRic as a conftest (to make it easier to update when the name changes with the next PSyclone release.

* Define API constant in lfric_common to have the correct PSyclone API.

* Fixed flake8 error.

---------

Co-authored-by: Junwei Lyu <[email protected]>
Co-authored-by: Junwei Lyu <[email protected]>
Co-authored-by: jasonjunweilyu <[email protected]>
Co-authored-by: Junwei Lyu <[email protected]>
Co-authored-by: Matthew Hambley <[email protected]>
Co-authored-by: Junwei Lyu <[email protected]>
Co-authored-by: Junwei Lyu <[email protected]>
Co-authored-by: Junwei Lyu <[email protected]>
lukehoffmann added a commit that referenced this issue Aug 21, 2024
* Removed unused glob import.

* Fixed typo picked up in review.

* Fixed failing tests.

* Replaced _artefact_store with getter as much as possible.

* #3 Introduce Tool, Compiler, and ToolRepository.

* #3 Made ToolRepository a singleton.

* #3 Added ToolBox.

* #3 Added dedicated enum for categories.

* #3 Add the category to a tool.

* #3 Use tool category when adding a tool to the tool box.

* #3 Added preprocessor, and dictionary-like accesses to ToolBox.

* #3 Added tool_box as mandatory parameter for a BuildConfig.

* #3 Start to use new compiler objects for Fortran compilation.

* #3 Added tests for Compiler class.

* #3 Add specific C- and Fortran-compiler classes. Updated tests.

* #3 Fixed typing.

* #3 Moved function to remove into Flags object, added test file for Flags.

* # Fixed tests to work with new Fortran compiler handling.

* #3 Use  for compiler variables if defined.

* #3 Add is_available flag to tools, added types.

* #3 Added cpp as Fortran preprocessor.

* #3 Removed unnecessary code.

* #3 Properly ignore mypy warnings.

* Support ToolBox for preprocessing.

* #3 Add a specific 'preprocess' method to Preprocessor.

* #3 Changed order in which compiler flags are used.

* #3 Use toolbox in compile_c step.

* #3 Added get_hash function to compiler.

* #3 Added conftest.py file (with a fixture to create a C-compiler).

* #3 Introduce fixtures for Fortran compiler and tool_box.

* #3 Remove explicit compiler information from MpCommonArgs (since it's already part of the config toolbox).

* #3 Added linker as tool.

* #3 Added test for linking shared libraries.

* #3 Pass compiler flags to the linker if a compiler was specified.

* #3 Remove unused function.

* #3 Removed more unused code.

* #3 Automatically add a linker for each compiler.

* #3 Fixed typo.

* #3 Support vendor for compiler and linker.

* #3 Make linker having a vendor, too.

* #3 Add set_default_vendor method to tool repository.

* Ignore build directory for git.

* Updated test.

* # Fix some mypy errors and warnings.

* Avoid using get() for singleton, instead use __new__ which makes mypy happier.

* Changed the transformation_script parameter of function psyclone to accept a function that can return file-specific transformation scripts

* Make mypy happy by using patch.object.

* Remove more comments and confusion about mypy :)

* Try to make mypy happy on older python versions.

* Make flake8 happy.

* Sort imported name alphabetically.

* Try to fix failing hash test (and add some additional improvements in the test).

* Removed fpath= for input transformation_script function to pass mypy test for Python 3.7; Moved transformation_script_hash test to unit test from system test

* Fix mypy typing check errors for psyclone unit test

* Fix config typing issue with mypy in psyclone unit test

* Fix flake8 issues; Revert Config mypy typing fix

* Add comment to ignore typing check for fpath parameter of input transformation_script function

* Fix assert check after transformation_script function is changed from being called twice to once

* Filter out 'no transformation script' warning for psyclone system test

* Replace 'ignore' typing of fpath of transformation_script with removing keyword argument

* #3 Support proper tests to check if tools are available.

* 1. Updated transformation_script description; 2. Modified mock_transformation_script; 3.Removed redundant _analysis_for_prebuilds

* Updated lfric/atm.py and lfric/gungho.py examples to pass in transformation_script functions

* Added description for the psyclone step to instructions on writing a config

* #3 Added git as a tool.

* #3 Fix incorrect | usage in typing.

* #3 Added unit tests for git.

* #3 Renamed git.py to versioning.py, to avoid name clash with the corresponding test_git.py tests.

* #3 Converted svn and fcm to tools.

* #3 Fixed missing whitespace.

* Modified the documentation for writing a config with PSyclone

* Add config as a parameter for run_psyclone for the transformation_script to use;Updated the related functions and tests;
Changed the logic of the transformation_script examples

* #3 Replaced ar with tool object.

* #3 Added tests for ar.py.

* #3 Removed debug output.

* #3 Converted PSyclone to be a tool.

* #3 Removed debug print, fixed python 3.7 typing information.

* #3 Updated comments.

* Modified the get_optimisation_script function examples and updated the doc formatting

* #3 Add Rsync tool.

* #3 Removed now unused function.

* #3 Added test for rsync.

* #3 Fixed all mypy warnings about functions not checked.

* #3 Replace all mock-tests to use subprocess so the name of the executable is tested as well.

* #3 Remove duplicated flags.

* #3 Fixed changed order of linking.

* #3 Removed run_command function.

* #3 Fixed 3.8 typing error.

* #3 Fixed unused imports.

* #3 Move flags checksum into Flags, and remove now unused tools.py file.

* #3 Renamed newtools to tools.

* #3 Made custom function for all git functions called (instead of just calling run).

* #3 Updated and fixed comments.

* #3 Fixed errors in comments.

* Fixed minor errors in documentation.

* #3 Make it easier to create wrapper around standard compiler.

* #3 Added documentation for all tool related classes and their usage.

* #3 Added MISC category.

* Addressed reviewer's comments.

* Updated cli to properly use ToolBox etc, removing hard-coded gnu command linker option.

* Fixed mypy failures, including changes to import statement to avoid cyclic imports :(.

* #3 Fix circular import.

* Added #TODO so that this can be removed once fparser supports sentinels.

* Fix typing problems by ignoring fparser.

* Replaced more string names for artefacts with enums.

* Removed EXECUTABLES from constants.

* Moved Artefact class out of ArtefactStore and renamed it to ArtefactSet.

* Moved OBJECT_FILES from constants into ArtefactSet.

* Moved OBJECT_ARCHIVES from constants to ArtefactSet.

* Moved PRAGMAD_C from constants to ArtefactSet.

* Turned 'all_source' into an enum.

* Allow integer as revision.

* Fixed flake8 error.

* Removed specific functions to add/get fortran source files etc.

* Removed non-existing and unneccessary collections.

* Try to fix all run_configs.

* Fixed rebase issues.

* Added replace functionality to ArtefactStore, updated test_artefacts to cover all lines in that file.

* Started to replace artefacts when files are pre-processed.

* Removed linker argument from linking step in all examples.

* Try to get jules to link.

* Fixed build_jules.

* Fixed other issues raised in reviews.

* Try to get jules to link.

* Fixed other issues raised in reviews.

* Simplify handling of X90 files by replacing the X90 with x90, meaning only one artefact set is involved when running PSyclone.

* Make OBJECT_ARCHIVES also a dict, migrate more code to replace/add files to the default build artefact collections.

* Fixed some examples.

* Fix flake8 error.

* Fixed failing tests.

* Support empty comments.

* Fix preprocessor to not unnecessary remove and add files that are already in the output directory.

* Allow find_soure_files to be called more than once by adding files (not replacing artefact).

* Updated lfric_common so that files created by configurator are written in build (not source).

* Use c_build_files instead of pragmad_c.

* Removed unnecessary str.

* Documented the new artefact set handling.

* Fixed typo.

* Make the PSyclone API configurable.

* Fixed formatting of documentation, properly used ArtefactSet names.

* Support .f and .F Fortran files.

* Removed setter for tool.is_available, which was only used for testing.

* #3 Fix documentation and coding style issues from review.

* Renamed Categories into Category.

* Minor coding style cleanup.

* Removed more unnecessary ().

* Re-added (invalid) grab_pre_build call.

* Fixed typo.

* Renamed set_default_vendor to set_default_compiler_suite.

* Renamed VendorTool to CompilerSuiteTool.

* Also accept a Path as exec_name specification for a tool.

* Move the check_available function into the base class.

* Fixed some types and documentation.

* Fix typing error.

* Added explanation for meta-compiler.

* Improved error handling and documentation.

* Replace mpiifort with mpifort to be a tiny bit more portable.

* Use classes to group tests for git/svn/fcm together.

* Fixed issue in get_transformation script, and moved script into lfric_common to remove code duplication.

* Code improvement as suggested by review.

* Fixed run config

* Added reference to ticket.

* Updated type information.

* More typing fixes.

* Fixed typing warnings.

* As requested by reviewer removed is_working_copy functionality.

* Issue a warning (which can be silenced) when a tool in a toolbox is replaced.

* Fixed flake8.

* Fixed flake8.

* Fixed failing test.

* Addressed issues raised in review.

* Removed now unnecessary operations.

* Updated some type information.

* Fixed all references to APIs to be consistent with PSyclone 2.5.

* Added api to the checksum computation.

* Fixed type information.

* Added test to verify that changing the api changes the checksum.

* Make compiler version a tuple of integers

* Update some tests to use tuple versions

* Explicitly test handling of bad version format

* Fix formatting

* Tidying up

* Make compiler raise an error for any invalid version string

Assume these compilers don't need to be hashed.
Saves dealing with empty tuples.

* Check compiler version string for compiler name

* Fix formatting

* Add compiler.get_version_string() method

Includes other cleanup from PR comments

* Use different version checkout for each compiler vendor with mixins

* Refactoring, remove unittest compiler class

* Fix some mypy errors

* Use 'Union' type hint to fix build checks

* Return run_version_command to base Compiler class

Provides default version command that can be overridden for other compilers.
Also fix some incorrect tests
Other tidying

* Add a missing type hint

* Remove inheritance from mixins and use protocol

* Simplify compiler inheritance

Mixins have static methods with unique names,
overrides only happen in concrete classes

* Simplify usage of compiler-specific parsing mixins.

* Test for missing mixin.

* Fixed test.

* Added more tests for invalid version numbers.

* Added more test cases for invalid version number, improved regex to work as expected.

* Fixed typo in test.

* Fixed test.

---------

Co-authored-by: Joerg Henrichs <[email protected]>
Co-authored-by: Junwei Lyu <[email protected]>
Co-authored-by: jasonjunweilyu <[email protected]>
Co-authored-by: Junwei Lyu <[email protected]>
Co-authored-by: Matthew Hambley <[email protected]>
Co-authored-by: Junwei Lyu <[email protected]>
Co-authored-by: Junwei Lyu <[email protected]>
Co-authored-by: Junwei Lyu <[email protected]>
Co-authored-by: Joerg Henrichs <[email protected]>
Co-authored-by: Luke Hoffmann <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant