0.2.0-alpha

Release notes

• Changes to src/utils/hdfs_mods.py. Several modifications were made involving the imports and usage of the read_hdfs_csv, write_hdfs_csv, and hdfs_load_json functions.
• Added new functionality for data validation, including validation of postcodes and checking for real postcodes.
• Introduced utility functions for reading and writing CSV files from HDFS and loading JSON files from HDFS.
• Implemented schema loading and data shape checking functions.
• Updated test cases to cover the new functionality.

Dummy file additions

Addition of a new file: src/data_visualisations.py which is currently empty, just there as a placeholder. This will later introduct new functionality related to data visualisations.

Addition of a new file src/outlier_detection.py which is currently empty, just there as a placeholder. Will contain functionality related to outlier detection.

Tidying up

• Deleted init.py files in tests/test_data_validation, tests/test_data_visualisations, and tests/test_outlier_detection directories.
• Renamed the tests/test_utils/test_hdfs_mods.py file to tests/test_utils/test_hdfs.py.

What's Changed

• Update README.md by @jwestw in #24
• 118 data schema by @RoddyMacrae in #20
• 116 badge readme by @jwestw in #15
• 147 transmute spp snapshot by @gallmand in #29
• 124 validate postcodes by @jwestw in #27
• 176 pipeline logging by @ali-i3 in #28
• 208 responses dict by @jwestw in #30
• 142 jenkins file rap by @RoddyMacrae in #16
• 120 check num cols by @RoddyMacrae in #21
• 119 file present by @RoddyMacrae in #19
• Create release for 0.2.0-alpha by @jwestw in #33

New Contributors

• @gallmand made their first contribution in #29

Full Changelog: 0.1.0-alpha...0.2.0-alpha