0.3.1-alpha-release: Merge pull request #116 from ONSdigital/495-export-config

495 export config

v0.3.0

This release includes:

• Expansion imputation
• Further outputs
• Post code fix

What's Changed

• Rdrp 467 by @zorge69 in #110
• 324 gb sas by @zorge69 in #105
• Draft: RDRP-488: Stage updated snapshot file by @coatet in #112
• RDRP-463: Improve postcode validation by @JenCheshire in #107
• Sf expansion 481 by @AnneONS in #109
• 482 numeric pg by @zorge69 in #104
• Rdrp 468 intram region by @zorge69 in #113

Full Changelog: 0.2.7-alpha-release...0.3.0-alpha-release

Imputation through the whole pipeline

The results of TMI imputation (long forms only) now feed through to the outputs.

Outlier-detection and Estimation modules have been updated to deal with short forms in long-form format.

Output schemas have been moved from the outputs folder to the config folder.

Run-id bug fix

Increased version to 0.2.5

Fixed the bug where the run id was not updating in the DAP prod environment.

The short form output is now complete.

What's Changed

• Postcode fix 2 by @ali-i3 in #81
• 407_validated_itl2 by @okudum in #84
• 399 creating short form output and schema by @ali-i3 in #78
• 413 keep lastupdateddate col (staging fix) by @AnneONS in #83
• 434_output_staging by @okudum in #88
• run id fix by @ali-i3 in #90

Full Changelog: 0.2.3-alpha-release...0.2.5-alpha-release

To test postcode validation in DAP

0.2.4-alpha-release

Increased version to 0.2.4 to build a new release

correct config paths for DAP

The pipeline as of 13 Sept 23. It includes:

• staging of spp snapshot data
• staging of mappers
• postcode validation
• outlier implementation
• estimation
• some short from output

0.2.2-alpha-z-release

Moved postcodes to src/

Zipped postcodes added to the repo

0.2.2-alpha-release

Increase version to 0.2.2

0.2.0-alpha

Release notes

• Changes to src/utils/hdfs_mods.py. Several modifications were made involving the imports and usage of the read_hdfs_csv, write_hdfs_csv, and hdfs_load_json functions.
• Added new functionality for data validation, including validation of postcodes and checking for real postcodes.
• Introduced utility functions for reading and writing CSV files from HDFS and loading JSON files from HDFS.
• Implemented schema loading and data shape checking functions.
• Updated test cases to cover the new functionality.

Dummy file additions

Addition of a new file: src/data_visualisations.py which is currently empty, just there as a placeholder. This will later introduct new functionality related to data visualisations.

Addition of a new file src/outlier_detection.py which is currently empty, just there as a placeholder. Will contain functionality related to outlier detection.

Tidying up

• Deleted init.py files in tests/test_data_validation, tests/test_data_visualisations, and tests/test_outlier_detection directories.
• Renamed the tests/test_utils/test_hdfs_mods.py file to tests/test_utils/test_hdfs.py.

What's Changed

• Update README.md by @jwestw in #24
• 118 data schema by @RoddyMacrae in #20
• 116 badge readme by @jwestw in #15
• 147 transmute spp snapshot by @gallmand in #29
• 124 validate postcodes by @jwestw in #27
• 176 pipeline logging by @ali-i3 in #28
• 208 responses dict by @jwestw in #30
• 142 jenkins file rap by @RoddyMacrae in #16
• 120 check num cols by @RoddyMacrae in #21
• 119 file present by @RoddyMacrae in #19
• Create release for 0.2.0-alpha by @jwestw in #33

New Contributors

• @gallmand made their first contribution in #29

Full Changelog: 0.1.0-alpha...0.2.0-alpha

Test Jenkins build, and build ver. 0.1.0

release-0.1.0

Changed .jenkins file so that it builds develop branch and tags start…