Update CI to produce unit test .mat and .hdf5 data (#289)

* Move content of benchmark_scripts readme into doc/developers

* Update CI attempt 1

* Actually use the right syntax

* Force python version to prevent MacOS using python2.7

* Rename files to better match the data they produce

* Aplease windows syntax

* Remove pip cache due to known windows bug: actions/setup-python#436

.github/workflows/ci.yml

@@ -54,18 +54,18 @@ jobs:
       - name: Set up MATLAB
         uses: matlab-actions/[email protected]
 
-      - name: Produce MATLAB unit test data
-        uses: matlab-actions/run-command@v1
+      - name: Setup Python
+        uses: actions/setup-python@v4
         with:
-          command: cd('tdms/tests/unit/benchmark_scripts/'), setup_unit_tests
+          python-version: '3.9'
 
       # -------------------------------------------------------------------------------
       # Ubuntu
       - name: Install dependencies for Ubuntu
         if: ${{ contains(matrix.os, 'ubuntu') }}
         run: |
           sudo apt-get update
-          sudo apt-get install libfftw3-dev libhdf5-dev libgomp1 python3
+          sudo apt-get install libfftw3-dev libhdf5-dev libgomp1
 
       # -------------------------------------------------------------------------------
       # Windows
@@ -125,6 +125,20 @@ jobs:
         run: |
           cmake --build . --config ${{ matrix.build_type }}
 
+      # -------------------------------------------------------------------------------
+      # Unit tests
+      - name: Produce MATLAB unit test data
+        if: matrix.build_testing == 'ON'
+        uses: matlab-actions/run-command@v1
+        with:
+          command: cd('tdms/tests/unit/benchmark_scripts/'), setup_unit_tests
+
+      - name: Produce hdf5 unit test data
+        if: matrix.build_testing == 'ON'
+        run: |
+          pip install -r ${{ github.workspace }}/tdms/tests/requirements.txt
+          python ${{ github.workspace }}/tdms/tests/unit/benchmark_scripts/create_hdf5_test_file.py
+
       - name: Run TDMS unit tests
         if: matrix.build_testing == 'ON'
         working-directory: ${{ runner.workspace }}/build
@@ -133,6 +147,8 @@ jobs:
           export OMP_NUM_THREADS=1
           ctest -C ${{ matrix.build_type }} --output-on-failure --extra-verbose
 
+      # -------------------------------------------------------------------------------
+      # Upload build artefact for system tests
       - name: Tar the build result to maintain permissions
         # https://github.com/actions/upload-artifact#maintaining-file-permissions-and-case-sensitive-files
         # https://github.com/actions/upload-artifact/issues/38

doc/developers.md

@@ -225,10 +225,10 @@ A number of unit tests require the presence of a `.mat` or `.hdf5` file to read/
 The locations of these files are coded into `tests/include/unit_test_utils.h`, but the files themselves are not committed to the repository.
 You can create the data files locally by running the `setup_unit_tests.m` and `create_hdf5_data.py` scripts locally when you wish to run the unit tests - the CI does this automatically on GitHub.
 Additional data can be "added" to that available to the unit tests by editing the content of the following scripts:
-- `create_hdf5_data.py` : Produces `hdf5_test_file.hdf5`; used when testing read/writing from `.hdf5` files.
+- `create_hdf5_test_file.py` : Produces `hdf5_test_file.hdf5`; used when testing read/writing from `.hdf5` files.
 - `create_structure_array.m` : Produces `structure_array.mat`; used when testing reading/writing MATLAB `struct` arrays.
-- `create_tdms_object_data.m` : Produces `class_data.mat`; used when testing reading/writing `tdms` objects to/from `.mat` files.
-- `create_bad_tdms_object_data.m` : Produces `bad_class_data.mat`; used for testing failure cases when reading/writing `tdms` objects to/from `.mat` files.
+- `create_class_data.m` : Produces `class_data.mat`; used when testing reading/writing `tdms` objects to/from `.mat` files.
+- `create_bad_class_data.m` : Produces `bad_class_data.mat`; used for testing failure cases when reading/writing `tdms` objects to/from `.mat` files.
 
 The `benchmark_` scripts perform band-limited interpolation (BLi) using `MATLAB`'s `interp` function.
 `TDMS`'s interpolation schemes are based off this `MATLAB` function (specficially, in the coefficients the scheme uses to interpolate), and are thus used to benchmark the accuracy of the scheme.

tdms/tests/unit/benchmark_scripts/setup_unit_tests.m

@@ -3,7 +3,7 @@
 end
 
 create_structure_array;
-create_tdms_object_data;
-create_bad_tdms_object_data;
+create_class_data;
+create_bad_class_data;
 
 exit;