Revised e3sm_to_cmip --info handling. Details within. #199

TonyB9000 · 2023-07-21T23:06:11Z

Modifications to main.py (_run_info_mode) include:

Testing for presence of "info_out_path" as first option for output (otherwise defaults to "output_path" or stdout.)
Reject mode-3 data test for MPAS data with error message
Revise mode-3 data test output to provide table=, variable=, and DataSupport= (True,False) for each supporting variable.
Created function "get_handler_info_msg(handler)" in util.py to replace 3 repetitions of the same 10 lines.
Added "info" mode to util.py print_message for colorless output.
Revised help text to clarify --info functionality.

Closes --info option fails on some inputs with TypeError: expected str, bytes or os.PathLike object, not NoneType #181

(unrelated) In lib.py, added "if do_pbar" switch before instances of progress-bar codes to reduce batch noise.

chengzhuzhang

Codes look fine by visual check. Tony, thanks for summarizing the code change, that makes it easier to review. Regarding the progress bar hack, I recall that I saw the discussion somewhere, but can't seem to find it. Would you add here why you implemented this.

TonyB9000 · 2023-07-24T15:43:50Z

@chengzhuzhang Regarding the progress bar (hack): When examining detailed output logs, I often uncovered multiple megabytes of output (vi for instance does not compress the progress-bar output) and this output has no value when running jobs in batch mode (no human to stare at the progress bar). I wanted to implement a "--quiet" or "--no-progress-bar" command-line option but didn't want to puzzle through it at the time. Probably a "self.progressbar" (True/False) attribute would work.

TonyB9000 · 2023-07-24T16:21:53Z

@tomvothecoder (I hope I did the issue-link correctly). Are you interested in reviewing the code mods, or should I merge?

tomvothecoder · 2023-07-24T16:52:26Z

@tomvothecoder (I hope I did the issue-link correctly). Are you interested in reviewing the code mods, or should I merge?

Hey @TonyB9000, I usually do code review for e3sm_to_cmip. Were you hoping to have this merged soon to integrate into datasm?

I don't expect us to have a new release candidate by 8/1 (E3SM Unified deadline) unless there is an urgent bug fix.
We can take our time working on this PR and reviewing. I'll aim to review either on Wed (7/26) before I go on vacation or sometime next week when I get back

TonyB9000 · 2023-07-24T18:17:11Z

@tomvothecoder No Hurry. For datasm operation I can (I hope) just pip install my local e2c.

TonyB9000 · 2024-02-15T22:00:40Z

@tomvothecoder Sorry I abandoned tracking this. When I create the datasm conda env with "prod.yml", the value for e2c is specified as e3sm_to_cmip >=1.9.1, and in fact, what gets included is e3sm_to_cmip 1.11.1. It is currently failing for v2_1 amip, (77 datasets) with

ERROR:root:UTIL:ERROR checking variables: No data returned from e3sm_to_cmip --info:

I had inadvertently been running an environment that "pip installed" local e2c changes, and does not reflect your latest work.

I am going to attempt the "Use the command line to resolve conflicts" before anything else, and see how far I get.

TonyB9000 · 2024-02-15T23:23:07Z

@tomvothecoder "All checks passed" (FWIW). Please review. I know some changes in the "conflict-resolved" merge to "main" are yours and some are mine. I left out my addition, around line 710, that read

                    if handler['table'] in [ "CMIP6_Omon.json", "CMIP6_SImon.json" ]:
                        stat_msg = f"Cannot presently test for variable support in MPAS files"
                        print_message(stat_msg, status="info")

because I could not tell if subsequent changes made this obsolete.

tomvothecoder

Hi Tony, I had some minor comments and suggestions. After reviewing and addressing, feel free to merge if your testing works out.

tomvothecoder · 2024-02-16T17:23:21Z

e3sm_to_cmip/__main__.py

+    def get_handler_info_msg(self, handler):
+        msg = {
+            "CMIP6 Name": handler["name"],
+            "CMIP6 Table": handler["table"],
+            "CMIP6 Units": handler["units"],
+            "E3SM Variables": ", ".join(handler["raw_variables"]),
+        }
+        if handler.get("unit_conversion"):
+            msg["Unit conversion"] = handler["unit_conversion"]
+        if handler.get("levels"):
+            msg["Levels"] = handler["levels"]
+        return msg
+


Suggested change

def get_handler_info_msg(self, handler):

msg = {

"CMIP6 Name": handler["name"],

"CMIP6 Table": handler["table"],

"CMIP6 Units": handler["units"],

"E3SM Variables": ", ".join(handler["raw_variables"]),

}

if handler.get("unit_conversion"):

msg["Unit conversion"] = handler["unit_conversion"]

if handler.get("levels"):

msg["Levels"] = handler["levels"]

return msg

This looks like a duplicate of util.get_handler_info_msg and is not being used in this module.

It can be deleted.

tomvothecoder · 2024-02-16T17:23:52Z

e3sm_to_cmip/__main__.py

@@ -648,44 +665,30 @@ def _run_info_mode(self):  # noqa: C901

        # if the user just asked for the handler info
        if self.freq == "mon" and not self.input_path and not self.tables_path:
+            messages = []


messages = [] can be defined once outside the conditional.

tomvothecoder · 2024-02-16T17:24:05Z

e3sm_to_cmip/__main__.py


        # if the user asked if the variable is included in the table
        # but didnt ask about the files in the inpath
        elif self.freq and self.tables_path and not self.input_path:
+            messages = []


Suggested change

messages = []

tomvothecoder · 2024-02-16T17:24:11Z

e3sm_to_cmip/__main__.py

            file_path = next(Path(self.input_path).glob("*.nc"))

+            messages = []


Suggested change

messages = []

tomvothecoder · 2024-02-16T17:24:15Z

e3sm_to_cmip/__main__.py


        elif self.freq and self.tables_path and self.input_path:
+


Suggested change

tomvothecoder · 2024-02-16T17:26:00Z

e3sm_to_cmip/__main__.py

+                        messages.append(hand_msg)
+                    else:
+                        stat_msg = f"Table={handler['table']}:Variable={handler['name']}:DataSupport=FALSE"
+                    print_message(stat_msg, status="info")


Suggested change

print_message(stat_msg, status="info")

logger.info(stat_msg)

Can we use the logger object defined at the top of this module instead of print_message()?

Also should we still print out stat_msg if it is None?

Or should it be within the else: statement to avoid printing None?

tomvothecoder · 2024-02-16T17:27:08Z

e3sm_to_cmip/__main__.py

+                    # We test here against the input "freq", because atmos mon data satisfies BOTH
+                    # CMIP6_day.json AND CMIP6_mon.json, but we only want the latter in the "hand_msg" output.
+                    # The variables-check below is insufficient as vars "hass" and "rlut" have multiple freqs.
+


Suggested change

tomvothecoder · 2024-02-16T17:28:36Z

e3sm_to_cmip/__main__.py

+                    # We test here against the input "freq", because atmos mon data satisfies BOTH
+                    # CMIP6_day.json AND CMIP6_mon.json, but we only want the latter in the "hand_msg" output.
+                    # The variables-check below is insufficient as vars "hass" and "rlut" have multiple freqs.


Not sure if these comments are within 80 characters per line

In total, no. But does one begin measuring at the "#" or at column 1? (Never knew very long comment lines to be an issue.)

tomvothecoder · 2024-02-16T17:31:46Z

e3sm_to_cmip/__main__.py

+                    if self.freq == "day" and handler['table'] == "CMIP6_Amon.json":
+                        continue
+
+                    hand_msg = None


Suggested change

hand_msg = None

Don't need to set the default value of hand_msg to None.

Right - true of "stat_msg" as well, "Defensive Programming"? (someone later changes code in a manner that causes a exception because "None" was not assigned?)

Unless a portion of code is in a tight-loop (may be executed millions of times), I an comfortable with defensive programming techniques. Too much minimalism has its drawbacks - and most compilers (I assume) elide unnecessary codes.

tomvothecoder · 2024-02-16T17:32:42Z

e3sm_to_cmip/__main__.py

+                        continue
+
+                    hand_msg = None
+                    stat_msg = None


Suggested change

stat_msg = None

Probably don't need to set the default value of stat_msg to None if we don't want to print out None values.

Revised e3sm_to_cmip --info handling. Details within.

530ea14

TonyB9000 requested review from tomvothecoder and chengzhuzhang July 21, 2023 23:06

chengzhuzhang approved these changes Jul 22, 2023

View reviewed changes

TonyB9000 added 4 commits July 24, 2023 15:12

consolidate info_mode 3 multi-support-var assessment to single status

093800c

missed one do_pbar-hack

3168d80

corrected message logic to isolate the successful handler info

6aaaa35

addressed rlut-ambiquity in info mode 3

9374ae5

rebased with __main__ conflict resolved/reverted

10c90da

tomvothecoder reviewed Feb 16, 2024

View reviewed changes

minor code cleanup

4586955

TonyB9000 merged commit fd821e1 into master Feb 16, 2024
3 checks passed

TonyB9000 deleted the fix_e3c_info_mode branch February 16, 2024 20:52

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Revised e3sm_to_cmip --info handling. Details within. #199

Revised e3sm_to_cmip --info handling. Details within. #199

TonyB9000 commented Jul 21, 2023 •

edited by tomvothecoder

Loading

chengzhuzhang left a comment

TonyB9000 commented Jul 24, 2023

TonyB9000 commented Jul 24, 2023

tomvothecoder commented Jul 24, 2023 •

edited

Loading

TonyB9000 commented Jul 24, 2023

TonyB9000 commented Feb 15, 2024

TonyB9000 commented Feb 15, 2024

tomvothecoder left a comment

tomvothecoder Feb 16, 2024

tomvothecoder Feb 16, 2024

tomvothecoder Feb 16, 2024

tomvothecoder Feb 16, 2024

tomvothecoder Feb 16, 2024

tomvothecoder Feb 16, 2024

tomvothecoder Feb 16, 2024

tomvothecoder Feb 16, 2024

tomvothecoder Feb 16, 2024

TonyB9000 Feb 16, 2024

tomvothecoder Feb 16, 2024

TonyB9000 Feb 16, 2024

tomvothecoder Feb 16, 2024

		file_path = next(Path(self.input_path).glob("*.nc"))

		messages = []

Revised e3sm_to_cmip --info handling. Details within. #199

Revised e3sm_to_cmip --info handling. Details within. #199

Conversation

TonyB9000 commented Jul 21, 2023 • edited by tomvothecoder Loading

chengzhuzhang left a comment

Choose a reason for hiding this comment

TonyB9000 commented Jul 24, 2023

TonyB9000 commented Jul 24, 2023

tomvothecoder commented Jul 24, 2023 • edited Loading

TonyB9000 commented Jul 24, 2023

TonyB9000 commented Feb 15, 2024

TonyB9000 commented Feb 15, 2024

tomvothecoder left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

TonyB9000 commented Jul 21, 2023 •

edited by tomvothecoder

Loading

tomvothecoder commented Jul 24, 2023 •

edited

Loading