Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Splitattrs ncsave redo #5410

Merged
Merged
Show file tree
Hide file tree
Changes from 20 commits
Commits
Show all changes
38 commits
Select commit Hold shift + click to select a range
83eee13
Add docs and future switch, no function yet.
pp-mo Jul 21, 2023
ce5cdef
Typing enables code completion for Cube.attributes.
pp-mo Jul 30, 2023
4d660b6
Make roundtrip checking more precise + improve some tests accordingly…
pp-mo Jul 30, 2023
9a80cd2
Rework all tests to use common setup + results-checking code.
pp-mo Jul 30, 2023
4ecef96
Saver supports split-attributes saving (no tests yet).
pp-mo Jul 21, 2023
78aaebb
Tiny docs fix.
pp-mo Aug 2, 2023
54f344c
Explain test routines better.
pp-mo Aug 3, 2023
edc9899
Fix init of FUTURE object.
pp-mo Aug 3, 2023
d242506
Remove spurious re-test of FUTURE.save_split_attrs.
pp-mo Aug 3, 2023
8d7ad2a
Don't create Cube attrs of 'None' (n.b. but no effect as currently us…
pp-mo Aug 4, 2023
8aa5311
Remove/repair refs to obsolete routines.
pp-mo Aug 4, 2023
f87550b
Check all warnings from save operations.
pp-mo Aug 4, 2023
30d62c2
Remove TestSave test numbers.
pp-mo Aug 4, 2023
fb13770
More save cases: no match with missing, and different cube attribute …
pp-mo Aug 4, 2023
5a429d0
Run save/roundtrip tests both with+without split saves.
pp-mo Aug 5, 2023
dd53275
Fix.
pp-mo Aug 6, 2023
80a4039
Review changes.
pp-mo Aug 7, 2023
fb343ae
Fix changed warning messages.
pp-mo Aug 8, 2023
b1778c6
Move warnings checking from 'run' to 'check' phase.
pp-mo Aug 8, 2023
067f07d
Simplify and improve warnings checking code.
pp-mo Aug 8, 2023
891da48
Fix wrong testcase.
pp-mo Aug 8, 2023
8ecadca
Minor review changes.
pp-mo Aug 8, 2023
4edf778
Fix reverted code.
pp-mo Aug 8, 2023
159914c
Use sets to simplify demoted-attributes code.
pp-mo Aug 8, 2023
deb1db3
WIP
pp-mo Aug 18, 2023
e5d5ff9
Working with iris 3.6.1, no errors TestSave or TestRoundtrip.
pp-mo Aug 18, 2023
42fce92
Interim save (incomplete?).
pp-mo Aug 21, 2023
19a2956
Different results form for split tests; working for roundtrip.
pp-mo Aug 21, 2023
59d05dc
Check that all param lists are sorted.
pp-mo Aug 21, 2023
1770d97
Check matrix result-files compatibility; add test_save_matrix.
pp-mo Aug 21, 2023
b67f510
test_load_matrix added; two types of load result.
pp-mo Aug 21, 2023
2576826
Finalise special-case attributes.
pp-mo Aug 21, 2023
e201a8e
Small docs tweaks.
pp-mo Aug 21, 2023
4602f06
Add some more testcases,
pp-mo Aug 21, 2023
2377c89
Ensure valid sort-order for globals of possibly different types.
pp-mo Aug 21, 2023
8987a26
Initialise matrix results with legacy values from v3.6.1 -- all match…
pp-mo Aug 21, 2023
cbd7167
Add full current matrix results, i.e. snapshot current behaviours.
pp-mo Aug 21, 2023
eea99d1
Review changes : rename some matrix testcases, for clarity.
pp-mo Oct 10, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
84 changes: 30 additions & 54 deletions lib/iris/fileformats/netcdf/saver.py
Original file line number Diff line number Diff line change
Expand Up @@ -2626,7 +2626,7 @@ def save(
dictionaries on each cube in the saved cube list will be compared, and common
attributes saved as NetCDF global attributes where appropriate.

Or, **when split-attribute saving is enabled**, then `cube.attributes.locals``
Or, **when split-attribute saving is enabled**, then ``cube.attributes.locals``
are always saved as attributes of data-variables, and ``cube.attributes.globals``
are saved as global (dataset) attributes, where possible.
Since the 2 types are now distinguished : see :class:`~iris.cube.CubeAttrsDict`.
Expand Down Expand Up @@ -2835,28 +2835,29 @@ def attr_values_equal(val1, val2):
return match

cube0 = cubes[0]
invalid_globals = [
attrname
for attrname in global_names
if not all(
attr_values_equal(
cube.attributes.globals.get(attrname),
cube0.attributes.globals.get(attrname),
invalid_globals = set(
[
attrname
for attrname in global_names
if not all(
attr_values_equal(
cube.attributes.globals.get(attrname),
cube0.attributes.globals.get(attrname),
)
for cube in cubes[1:]
)
for cube in cubes[1:]
)
]
]
)

# Establish all the global attributes which we will write to the file (at end).
global_attributes = {
attr: cube0.attributes.globals.get(attr)
for attr in global_names
if attr not in invalid_globals
for attr in global_names - invalid_globals
}
if invalid_globals:
# Some cubes have different global attributes: modify cubes as required.
warnings.warn(
f"Saving the cube global attributes {invalid_globals} as local "
f"Saving the cube global attributes {sorted(invalid_globals)} as local "
"(i.e. data-variable) attributes, where possible, since they are not "
"the same on all input cubes."
)
Expand All @@ -2865,39 +2866,30 @@ def attr_values_equal(val1, val2):
# We iterate over cube *index*, so we can replace the list entries with
# with cube *copies* -- just to avoid changing our call args.
cube = cubes[i_cube]
demote_attrs = [
attr
for attr in cube.attributes.globals
if attr in invalid_globals
]
demote_attrs = set(cube.attributes.globals) & invalid_globals
if any(demote_attrs):
# This cube contains some 'demoted' global attributes.
# Replace the input cube with a copy, so we can modify attributes.
cube = cube.copy()
cubes[i_cube] = cube
# Catch any demoted attrs where there is already a local version
blocked_attrs = [
attrname
for attrname in demote_attrs
if attrname in cube.attributes.locals
]
blocked_attrs = demote_attrs & set(cube.attributes.locals)
if blocked_attrs:
warnings.warn(
f"Global cube attributes {blocked_attrs} "
f"Global cube attributes {sorted(blocked_attrs)} "
f'of cube "{cube.name()}" were not saved, overlaid '
"by existing local attributes with the same names."
)
for attr in demote_attrs:
if attr not in blocked_attrs:
cube.attributes.locals[
attr
] = cube.attributes.globals[attr]
cube.attributes.globals.pop(attr)
demote_attrs -= blocked_attrs
if demote_attrs:
# This cube contains some 'demoted' global attributes.
# Replace input cube with a copy, so we can modify attributes.
cube = cube.copy()
cubes[i_cube] = cube
for attr in demote_attrs:
# move global to local
value = cube.attributes.globals.pop(attr)
cube.attributes.locals[attr] = value

else:
trexfeathers marked this conversation as resolved.
Show resolved Hide resolved
# Determine the attribute keys that are common across all cubes and
# thereby extend the collection of local_keys for attributes
# that should be attributes on data variables.
# Legacy mode: calculate "local_keys" to control which attributes are local
# and which global.
if local_keys is None:
local_keys = set()
else:
Expand All @@ -2919,22 +2911,6 @@ def attr_values_equal(val1, val2):
common_keys.difference_update(different_value_keys)
local_keys.update(different_value_keys)

common_attr_values = None
for cube in cubes:
cube_attributes = cube.attributes
keys = set(cube_attributes)
if common_attr_values is None:
common_attr_values = cube_attributes.copy()
common_keys = keys.copy()
local_keys.update(keys.symmetric_difference(common_keys))
common_keys.intersection_update(keys)
different_value_keys = []
for key in common_keys:
if np.any(common_attr_values[key] != cube_attributes[key]):
different_value_keys.append(key)
common_keys.difference_update(different_value_keys)
local_keys.update(different_value_keys)

def is_valid_packspec(p):
"""Only checks that the datatype is valid."""
if isinstance(p, dict):
Expand Down
Loading