Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Model maps shown for EMEP should use the frequency of the Base_*.nc file rather than the colocated data and then resampling #1260

Open
lewisblake opened this issue Jul 12, 2024 · 2 comments · May be fixed by #1312
Assignees
Labels
enhancement New feature or request performance Excessive use of time or memory
Milestone

Comments

@lewisblake
Copy link
Member

Is your feature request related to a problem? Please describe.
An inefficiency in the creation of the model maps is that for some models, such as EMEP, we already have yearly files available, so reading in daily data and then temporally averaging is unnecessarily expensive.

Describe the solution you would like to see
In the case of making maps for the EMEP model, the maps_freq argument set in a config file should be used to read the Base_{maps_freq}.nc file already available, regardless of the colocation frequency main_freq. Some information in the logger should be present to inform that this option is being used.

Additional context
Blocked by #1259

@lewisblake lewisblake added enhancement New feature or request performance Excessive use of time or memory labels Jul 12, 2024
@lewisblake lewisblake added this to the m2024-09 milestone Jul 12, 2024
@heikoklein
Copy link
Member

This is critical for the trends-evaluation, which does collocation in monthly data which takes 45min for 4 periods without model-maps. Model-maps for one period takes 6h alone, since daily data is read.

@thorbjoernl thorbjoernl removed their assignment Aug 16, 2024
@dulte dulte linked a pull request Aug 16, 2024 that will close this issue
9 tasks
@dulte dulte linked a pull request Aug 16, 2024 that will close this issue
9 tasks
@dulte
Copy link
Collaborator

dulte commented Aug 16, 2024

I've started on the problem in #1312. If I understand the problem correctly, this should solve it, even though it is not in the most elegant way:

I'm testing out simply changing main_freq to maps_freq when the maps are calculated(or the coarsest freq if that option is chosen). It of course then colocates the obs wrong. It seems that ModelMapsEngine creates a colocator, but only uses the model data, in which case the obs are irrelevant. And as I understand it, the obs shown on the map are created elsewhere, so this does not matter anyway(?). The obs shown will also be just the July data for that year, and not a mean of the year.

The freq change is done for all models, no special case for EMEP. I can change that to check if the model is EMEP, as we are not always sure to have yearly data

I think my code is a bit misplaced, but due to the overuse of OOP, it is not easy to know where things are happening. But we can discuss that later

@heikoklein heikoklein modified the milestones: m2024-09, m2024-10 Sep 10, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request performance Excessive use of time or memory
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants