pypromicev1.4.0 #291
Merged
pypromicev1.4.0 #291
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This reverts commit af09818.
* Bedrock stations shouldn’t depend on the noisy GPS signal for elevation. * Added station dimension values for WEG_B * Added corresponding unittest
Added eccodes installation
* Removed station_configurations.toml from repository * Updated bufr_utilities.set_station to validate wmo id * Implemented StationConfig io tests * Extracted StationConfiguration utils from get_bufr * Added support for loading multiple station configuration files Other * Made ArgumentParser instantiation inline
* Added detailed descriptions with references to the attributes in BUFRVariables * Change the attribute order to align with the exported schema * Changed variable roundings to align with the scales defined in the BUFR schemas: * Latitude and longitude is set to 5. Was 6 * heightOfStationGroundAboveMeanSeaLevel is set to 1. Was 2 * heightOfBarometerAboveMeanSeaLevel is set to to 1. Was 2 * pressure is set to -1. Was 1. Note: The BUFRVariable unit is Pa and not hPA * airTemperature is set to 2. Was 1. * heightOfSensorAboveLocalGroundOrDeckOfMarinePlatformTempRH is set to 2. Was 4 * heightOfSensorAboveLocalGroundOrDeckOfMarinePlatformWSPD is set to 2. Was 4 * Added unit tests to test the roundings * Updated existing unit tests to align with corrected precision
* The station position determination (AWS_latest_locations) is separated from the bufr file export * Updated the unit tests Corrected minimum data check to allow p_i or t_i to be nan Renamed process_station parameters for readability * Rename now_timestamp -> target_timestamp * Rename time_limit -> linear_regression_time_limit Applied black
Added missing "test" prefix required by the unittest framework.
* processes surface heights variables: `z_surf_combined`, `z_ice_surf`, `snow_height`, and thermistors' depths: `d_t_i_1-11` * `variable.csv` was updated accordingly * some clean-up of turbulent fluxes calculation, including renaming functions * handling empty station configuration files and making errors understandable * updated join_l3 so that surface height and thermistor depths in historical data are no longer ignored and to adjust the surface height between the merged datasets * calculated either from `gps_lat, gps_lon, gps_alt` or `lat, lon, alt`, static values called `latitude`, `longitude` and `altitude` are saved as attributes along with `latitude_origin`, `longitude_origin` and `altitude_origin` which states whether they come from gappy observations `gps_lat, gps_lon, gps_alt` or from gap-filled postprocess `lat, lon, alt` * changed "H" to "h" in pandas and added ".iloc" when necessary to remove deprecation warnings * made `make_metadata_csv.py` to update latest location in `aws-l3/AWS_station_metadata.csv` and `aws-l3/AWS_sites_metadata.csv` --------- Co-authored-by: Penny How <[email protected]> * L2toL3 test added (#282) * 3.8 and 3.9 tests removed, tests only for 3.10 and 3.11 * echo syntax changed * updated input file paths ---------
get_l2 were previously also loading vars and meta in addition to AWS. AWS is populating meta with source information during instantiation.
…tadata files * Refactor getVar getMeta * Use pypromice.resources instaed of pkg_resources
The format string was previously selected from the last l0 file.
* Added test case for output meta data * Added json formatted source string to attributes * Added title string to attributes * Updated ID string to include level * Added utility function for fetching git commit id
* Added test station configuration * Cleanup test data files
* Added data issues path to process_test.yml
Validate attribute keys in e2e test
* passing adjustment_dir to L2toL3.py * fixing attributes in join_l3 - station_attribute containing info from merged dataset was lost when concatenating the datasets - The key "source" is not present in the attributes of the old GC-Net files so `station_source = json.loads(station_attributes["source"])` was throwing an error * give data_issues_path to get_l2tol3 in test_process * using data_adjustments_dir as input in AWS.getL3 * adding path to dummy data_issues folder to process_test * making sure data_issues_path is Path in get_l2tol3
BaptisteVandecrux
added a commit
that referenced
this pull request
Aug 20, 2024
This reverts commit 735a8ba.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Bringing all changes from develop to main branch:
Producing data in its new structure: level 2 statin data + level 3 site data
fixing #246, #247, #175
Merging of data from different stations at the same site + historical GC-Net station data are appended to GEUS GC-Net station data
fixing #175
Calculation of gap-free and smoothed
lat,lonandaltand mean coords as attributesfixing #236, #173, #124
Calculation of a continuous surface height, ice surface height and snow height variables
z_surf_combined,z_ice_surfandsnow_heightfixing #10
Calculation of the thermistors' depth
fixing #248
Reading station configutation files from
AWS-L0/metadata/station_configurationfixing #217
Updating metadata CSV automatically
fixing #142
Also: