diff --git a/CONTRIBUTING.rst b/CONTRIBUTING.rst index 794f4488..36293d7a 100644 --- a/CONTRIBUTING.rst +++ b/CONTRIBUTING.rst @@ -307,17 +307,17 @@ To run Ruff locally : In the table below, some CI tool are mentioned for your information, but does not need to be installed on your computer. They are automatically run when you push your changes to the main repository via a GitHub Pull Request. -+-----------------------------------------------------------------------------------------------+------------------------------------------------------------------+------------+-------+-------------------------------------------+ -| Tool | Aim | pre-commit | CI/CD | Python version | -+===============================================================================================+==================================================================+============+=======+===========================================+ -| `pre-commit.ci `__ | Run pre-commit (as defined in `.pre-commit-config.yaml` ) | | πŸ‘ | | -+-----------------------------------------------------------------------------------------------+------------------------------------------------------------------+------------+-------+-------------------------------------------+ -| `CodeBeat `__ | Automated code review and analysis tools | - | πŸ‘ | all versions according to GitHub workflow | -+-----------------------------------------------------------------------------------------------+------------------------------------------------------------------+------------+-------+-------------------------------------------+ -| `CodeScene `__ | Automated code review and analysis tools | - | πŸ‘ | ? | -+-----------------------------------------------------------------------------------------------+------------------------------------------------------------------+------------+-------+-------------------------------------------+ -| `CodeFactor `__ | Automated code review and analysis tools | - | πŸ‘ | ? | -+-----------------------------------------------------------------------------------------------+------------------------------------------------------------------+------------+-------+-------------------------------------------+ ++-----------------------------------------------------------------------------------------------+------------------------------------------------------------------+-------------------------------------------+ +| Tool | Aim | Python version | ++===============================================================================================+==================================================================+===========================================+ +| `pre-commit.ci `__ | Run pre-commit (as defined in `.pre-commit-config.yaml` ) | | ++-----------------------------------------------------------------------------------------------+------------------------------------------------------------------+-------------------------------------------+ +| `CodeBeat `__ | Automated code review and analysis tools | all versions according to GitHub workflow | ++-----------------------------------------------------------------------------------------------+------------------------------------------------------------------+-------------------------------------------+ +| `CodeScene `__ | Automated code review and analysis tools | ? | ++-----------------------------------------------------------------------------------------------+------------------------------------------------------------------+-------------------------------------------+ +| `CodeFactor `__ | Automated code review and analysis tools | ? | ++-----------------------------------------------------------------------------------------------+------------------------------------------------------------------+-------------------------------------------+ @@ -379,17 +379,17 @@ The reader test succeeds if both files (ground truth and transformation of the r The Continuous Integration (CI) on GitHub runs tests and analyzes code coverage. The following tools are used: -+-----------------------------------------------------------------------------------------------+------------------------------------------------------------------+------------+-------+-------------------------------------------+ -| Tool | Aim | pre-commit | CI/CD | Version | -+===============================================================================================+==================================================================+============+=======+===========================================+ -| `Pytest `__ | Execute unit tests and functional tests | - | πŸ‘ | | -+-----------------------------------------------------------------------------------------------+------------------------------------------------------------------+------------+-------+-------------------------------------------+ -| Coverage | Measure the code coverage of the project's unit tests | - | πŸ‘ | all versions according to GitHub workflow | -+-----------------------------------------------------------------------------------------------+------------------------------------------------------------------+------------+-------+-------------------------------------------+ -| `CodeCov `__ | Uses the "coverage" package to generate a code coverage report. | - | πŸ‘ | all versions according to GitHub workflow | -+-----------------------------------------------------------------------------------------------+------------------------------------------------------------------+------------+-------+-------------------------------------------+ -| `Coveralls `__ | Uses the "coverage" to track the quality of your code over time. | - | πŸ‘ | all versions according to GitHub workflow | -+-----------------------------------------------------------------------------------------------+------------------------------------------------------------------+------------+-------+-------------------------------------------+ ++-----------------------------------------------------------------------------------------------+------------------------------------------------------------------+-------------------------------------------+ +| Tool | Aim | Version | ++===============================================================================================+==================================================================+===========================================+ +| `Pytest `__ | Execute unit tests and functional tests | | ++-----------------------------------------------------------------------------------------------+------------------------------------------------------------------+-------------------------------------------+ +| Coverage | Measure the code coverage of the project's unit tests | all versions according to GitHub workflow | ++-----------------------------------------------------------------------------------------------+------------------------------------------------------------------+-------------------------------------------+ +| `CodeCov `__ | Uses the "coverage" package to generate a code coverage report. | all versions according to GitHub workflow | ++-----------------------------------------------------------------------------------------------+------------------------------------------------------------------+-------------------------------------------+ +| `Coveralls `__ | Uses the "coverage" to track the quality of your code over time. | all versions according to GitHub workflow | ++-----------------------------------------------------------------------------------------------+------------------------------------------------------------------+-------------------------------------------+ @@ -426,8 +426,6 @@ a Pull Request (PR) to the DISDRODB main branch. - Add screenshots or GIFs for any UI changes. This will help the person reviewing your code to understand what you’ve changed and how it works. - - *Hint: use * \ `Kap `__\ * or * \ `Licecap `__\ * to record your screen.* - - Please use the pertinent template for the pull request, and fill it out accurately. - It’s OK to have multiple small commits as you work on the PR - GitHub diff --git a/docs/source/maintainers_guidelines.rst b/docs/source/maintainers_guidelines.rst index cfd015f8..f9a1e48a 100644 --- a/docs/source/maintainers_guidelines.rst +++ b/docs/source/maintainers_guidelines.rst @@ -138,7 +138,7 @@ Currently, on each Pull Request, GitHub Actions are configured as follow: +----------------------------------------------------------------------------------------------------+------------------------------------------------------------------+----------------------------------------------------------------------------------------------+-------------------------------------------+ -| | Aim | Status | Python version | +| Tools | Aim | Project page | Python version | +====================================================================================================+==================================================================+==============================================================================================+===========================================+ | `Pytest `__ | Execute unit tests and functional tests | | | +----------------------------------------------------------------------------------------------------+------------------------------------------------------------------+----------------------------------------------------------------------------------------------+-------------------------------------------+ diff --git a/docs/source/metadata_csv/Deployment_Info.csv b/docs/source/metadata_csv/Deployment_Info.csv index 25a8ddbe..9bac0a46 100644 --- a/docs/source/metadata_csv/Deployment_Info.csv +++ b/docs/source/metadata_csv/Deployment_Info.csv @@ -1,9 +1,9 @@ Keys, Description -latitude, WGS84 latitude in degree north [-90,90]. If the disdrometer is moving, specify -9999 -longitude, WGS84 longitude in degree east [-180,180]. If the disdrometer is moving, specify -9999 -altitude, Elevation above the sea level in meters. If the disdrometer is moving, specify -9999 -deployment_status, Deployment status. Possible values: 'terminated' or 'ongoing' -deployment_mode, Deployment mode. Possible values: 'land', 'ship', 'truck', 'cable' -platform_type, Platform type. Possible values: ' fixed' or 'mobile' -platform_protection, Platform protection. Possible values: 'N/A', 'shielded', 'unshielded’ -platform_orientation, Platform orientation in 0-360 degrees from the North direction (clockwise) +latitude, "WGS84 latitude in degree north [-90,90]. If the disdrometer is moving, specify -9999" +longitude, "WGS84 longitude in degree east [-180,180]. If the disdrometer is moving, specify -9999" +altitude, "Elevation above the sea level in meters. If the disdrometer is moving, specify -9999" +deployment_status, "Deployment status. Possible values: 'terminated' or 'ongoing'" +deployment_mode, "Deployment mode. Possible values: 'land', 'ship', 'truck', 'cable'" +platform_type, "Platform type. Possible values: ' fixed' or 'mobile'" +platform_protection, "Platform protection. Possible values: 'N/A', 'shielded', 'unshielded’" +platform_orientation, "Platform orientation in 0-360 degrees from the North direction (clockwise)" diff --git a/docs/source/metadata_csv/Description.csv b/docs/source/metadata_csv/Description.csv index af843d5d..3b4de152 100644 --- a/docs/source/metadata_csv/Description.csv +++ b/docs/source/metadata_csv/Description.csv @@ -7,6 +7,6 @@ summary, Summary information of the station comment, Comment on the station measurements history, History of the raw data file station_id,ID of the station -location, Village, town or region where the disdrometer is located +location, "Village, town or region where the disdrometer is located" country, Country where the disdrometer is located continent,Continent where the disdrometer is located diff --git a/docs/source/metadata_csv/Sensor_Info.csv b/docs/source/metadata_csv/Sensor_Info.csv index f4e7f176..2b9fed02 100644 --- a/docs/source/metadata_csv/Sensor_Info.csv +++ b/docs/source/metadata_csv/Sensor_Info.csv @@ -1,6 +1,6 @@ Keys, Description sensor_long_name, Sensor long name -sensor_manufacturer, Sensor manufacturer. Examples: Thies Clima, OTT Hydromet, Vaisala, Campbell, … +sensor_manufacturer, "Sensor manufacturer. Examples: Thies Clima, OTT Hydromet, Vaisala, Campbell, …" sensor_wavelength, Sensor wavelength sensor_serial_number, Sensor serial number firmware_iop, TO BE DEFINED [Available for OTT Parsivels] diff --git a/docs/source/metadata_csv/Source_Info.csv b/docs/source/metadata_csv/Source_Info.csv index 651937f5..0c0691db 100644 --- a/docs/source/metadata_csv/Source_Info.csv +++ b/docs/source/metadata_csv/Source_Info.csv @@ -1,4 +1,4 @@ Keys, Description source, Source information -source_convention, Raw data file convention (i.e. ARM v1.XXX, NASA v1.XX, …) +source_convention, "Raw data file convention (i.e. ARM v1.XXX, NASA v1.XX, …)" source_processing_date, Date of source raw data file creation diff --git a/docs/source/readers.rst b/docs/source/readers.rst index 40944131..54a9d5b2 100644 --- a/docs/source/readers.rst +++ b/docs/source/readers.rst @@ -379,7 +379,7 @@ There are 7 metadata keys for which is mandatory to specify the value : * the ``station_name`` must be the same as the name of the metadata YAML file without the .yml extension * the ``sensor_name`` must be one of the implemented sensor configurations. See ``disdrodb.available_sensor_name()``. If the sensor which produced your data is not within the available sensors, you first need to add the sensor - configurations. For this task, read the section `Add new sensor configs` TODO ADD LINK. + configurations. For this task, read the section `Add new sensor configs `_ * the ``raw_data_format`` must be either 'txt' or 'netcdf'. 'txt' if the source data are text/ASCII files. 'netcdf' if source data are netCDFs. * the ``platform_type`` must be either 'fixed' or 'mobile'. If 'mobile', the DISDRODB L0 processing accepts latitude/longitude/altitude coordinates to vary with time. * the ``reader`` name is essential to enable to select the correct reader when processing the station. @@ -422,7 +422,7 @@ Step 4 : Analyse the data and define the reader components ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Once the DISDRODB directory structure, reader and metadata are set up, you can start analyzing the content of your data. -To facilitate the task, we provide you with the `reader_preparation.ipynb Jupyter Notebook `. +To facilitate the task, we provide you with the `reader_preparation.ipynb Jupyter Notebook `_. We highly suggest to copy the notebook and adapt it to your own data. However, before starting adapting the Jupyter Notebook to your own data, @@ -486,7 +486,7 @@ For example, to process all stations of the EPFL_2008 campaign, you would run: .. note:: - * For more details and options related to DISDRODB L0 processing, read the section `Run DISDRODB L0 Processing` TODO ADD SECTION LINK + * For more details and options related to DISDRODB L0 processing, read the section `Run DISDRODB L0 Processing `_. The DISDRODB L0 processing generates the DISDRODB `Processed` directories tree illustrated here below. @@ -541,13 +541,6 @@ that the reader you implemented will provide the expected results also when someone else will add changes to the disdrodb codebase in the future. -All readers are tested as follow : - - -.. image:: /static/reader_testing.png - - - .. note:: The objective is to run every reader sequentially. Therefore, make sure to provide a very small test sample in order to limit the computing time. @@ -556,34 +549,28 @@ All readers are tested as follow : A typical test file is composed of 2 stations, with two files and a couple of timesteps with measurements. -The `GitHub readers testing resources `_ must have the following structure: - - -| πŸ“ disdrodb-testing -| β”œβ”€β”€ πŸ“ disdrodb -| β”œβ”€β”€ πŸ“ L0 -| β”œβ”€β”€ πŸ“ readers -| β”œβ”€β”€ πŸ“ -| β”œβ”€β”€ πŸ“ -| β”œβ”€β”€ πŸ“œ raw.zip -| β”œβ”€β”€ πŸ“ data -| β”œβ”€β”€ πŸ“ -| β”œβ”€β”€ πŸ“œ \*.\* : raw files -| β”œβ”€β”€ πŸ“ issue -| β”œβ”€β”€ πŸ“œ .yml -| β”œβ”€β”€ πŸ“ metadata -| β”œβ”€β”€ πŸ“œ .yml -| β”œβ”€β”€ πŸ“œ processed.zip -| β”œβ”€β”€ πŸ“ info -| β”œβ”€β”€ πŸ“œ .yml -| β”œβ”€β”€ πŸ“ L0A -| β”œβ”€β”€ πŸ“ `station_name> -| β”œβ”€β”€ πŸ“œ \*.\parquet -| β”œβ”€β”€ πŸ“ L0B -| β”œβ”€β”€ πŸ“ -| β”œβ”€β”€ πŸ“œ \*.\nc -| β”œβ”€β”€ πŸ“ metadata -| β”œβ”€β”€ πŸ“œ .yml +You should place you data and config files under the following directory tree: + +| πŸ“ disdrodb/tests/pytest_files/check_readers/DISDRODB +| β”œβ”€β”€ πŸ“ Raw +| β”œβ”€β”€ πŸ“ +| β”œβ”€β”€ πŸ“ +| β”œβ”€β”€ πŸ“ issue +| β”œβ”€β”€ πŸ“œ .yml +| β”œβ”€β”€ πŸ“ metadata +| β”œβ”€β”€ πŸ“œ .yml +| β”œβ”€β”€ πŸ“ data +| β”œβ”€β”€ πŸ“ +| β”œβ”€β”€ πŸ“œ .\* +| β”œβ”€β”€ πŸ“ ground_truth +| β”œβ”€β”€ πŸ“ +| β”œβ”€β”€ πŸ“œ .\* + + + +The `data` folder must contain your raw data files, while the `ground_truth` folder must contain the corresponding ground truth files. + +Once the reader is run with the raw data, the output files is compared to the ground truth files. If the files are identical, the reader is considered valid. @@ -607,18 +594,8 @@ The `GitHub readers testing resources `_ - - - .. code-block:: bash - git clone https://github.com/EPFL-ENAC/LTE-disdrodb-testing.git -2. Add your file according structure described above -3. Commit and push your changes -4. Test your test with pytest (have a look `here `__) diff --git a/docs/source/sensor_configs.rst b/docs/source/sensor_configs.rst index 6c2fd8ed..17f2c623 100644 --- a/docs/source/sensor_configs.rst +++ b/docs/source/sensor_configs.rst @@ -28,9 +28,9 @@ For each sensor, the following list of configuration YAML files are required: | β”œβ”€β”€ πŸ“œ \*.yml : YAML files defining sensor characteristics (e.g. diameter and velocity bins) | β”œβ”€β”€ πŸ“œ bins_diameter.yml : Information related to sensor diameter bins | β”œβ”€β”€ πŸ“œ bins_velocity.yml : Information related to sensor velocity bins -| β”œβ”€β”€ πŸ“œ L0A_encodings.yml : Variables encodings for the L0A product -| β”œβ”€β”€ πŸ“œ L0B_encodings.yml : Variables encodings for the L0B product -| β”œβ”€β”€ πŸ“œ L0_data_format.yml : Information related to the variables logged by the sensor +| β”œβ”€β”€ πŸ“œ l0a_encodings.yml : Variables encodings for the l0a product +| β”œβ”€β”€ πŸ“œ l0b_encodings.yml : Variables encodings for the l0b product +| β”œβ”€β”€ πŸ“œ raw_data_format.yml : Information related to the variables logged by the sensor | β”œβ”€β”€ πŸ“œ variables.yml : Variables logged by the sensor | β”œβ”€β”€ πŸ“œ variable_description.yml : Variables description | β”œβ”€β”€ πŸ“œ variable_long_name.yml: Variables long_name @@ -69,25 +69,25 @@ If the sensor (i.e. impact disdrometers) does not measure the drop fall velocity the YAML files must be defined empty ! -L0A_encodings.yml file +l0a_encodings.yml file ~~~~~~~~~~~~~~~~~~~~~~~ This file list the variables that are allow to be saved into the -DISDRODB L0A Apache Parquet format. +DISDRODB l0a Apache Parquet format. The file also specified the type (i.e. integer/floating precision/string) each variable is saved in the Apache Parquet binary format. In addition to the specified variables, also the following variables are allowed -to be saved into the DISDRODB L0A files: +to be saved into the DISDRODB l0a files: * the ``time`` column (in UTC format) * the ``latitude`` and ``longitude`` columns if the disdrometer station is mobile. -L0B_encodings.yml file +l0b_encodings.yml file ~~~~~~~~~~~~~~~~~~~~~~~ This file list the variables that are allow to be saved into the -DISDRODB L0B netCDF format. +DISDRODB l0b netCDF format. For each variable, you need to specify the compression options, the data type, the _FillValue to store i.e. NaN values (if integer data type), the chunk size @@ -153,19 +153,19 @@ variable_description.yml file ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ This file contains a description for each variable. -The ``description`` will be attached as a variable attribute to the DISDRODB L0B netCDF +The ``description`` will be attached as a variable attribute to the DISDRODB l0b netCDF variable_units.yml file ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ This file specify the measurement unit for each variable. -The ``units`` will be attached as a variable attribute to the DISDRODB L0B netCDF +The ``units`` will be attached as a variable attribute to the DISDRODB l0b netCDF variable_long_name.yml file ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ This file specify the long_name for each variable. -The ``long_name`` will be attached as a variable attribute to the DISDRODB L0B netCDF. +The ``long_name`` will be attached as a variable attribute to the DISDRODB l0b netCDF. See the `CF Conventions guidelines for long_name `_ for more information.