From 6ec9aa8709d7d40d4dde6c8f8205020af5d2ac6e Mon Sep 17 00:00:00 2001 From: Jozsef K Date: Mon, 28 Oct 2024 14:49:22 +0000 Subject: [PATCH 1/4] added new readme --- features/README.md | 23 ++++++++++++++++++++++- 1 file changed, 22 insertions(+), 1 deletion(-) diff --git a/features/README.md b/features/README.md index 30404ce4..5e063200 100644 --- a/features/README.md +++ b/features/README.md @@ -1 +1,22 @@ -TODO \ No newline at end of file +# `enviroment` setup + +This scipt sets up an enviorment (docker container) for testing and after testing has been completed removes the container and temporary directories. + +## `before_all` function + +The script sets up a container representing a fake backend to run tests. These test confirms that the request are routing as required, also checks the fixtures unzipped data paths are correct/exist. + +## `before_scenario` function + +This function will run bfeore each scenario would be ran. Creates a temporary directory and changes current working directory to the new temporary directory. This allows test files to be placed in there and later safely can be deleted. + +Set the `UUID` (Universal Unique Identifier) in the docker logs. +Sets each scenario with a `custom Session` so some of the default set headers will be removed. + +## `after_scenario` function + +This function removes the temporary directories and all contents, then changes directory out of the temporary directory. + +## `after_all` function + +This function stops and removes the `docker container`, also remove temporary directories for test output files. \ No newline at end of file From 318d55e9e4ec4286cc673c429e33b8434647974d Mon Sep 17 00:00:00 2001 From: Jozsef K Date: Mon, 28 Oct 2024 15:18:22 +0000 Subject: [PATCH 2/4] f --- dpypelines/pipeline/dataset_ingress_v1.md | 2 +- dpypelines/pipeline/generic_file_ingress_v1.md | 2 +- myscript.py | 18 ------------------ tests/README.md | 8 +++++++- 4 files changed, 9 insertions(+), 21 deletions(-) delete mode 100644 myscript.py diff --git a/dpypelines/pipeline/dataset_ingress_v1.md b/dpypelines/pipeline/dataset_ingress_v1.md index ffc626b6..c3f9286e 100644 --- a/dpypelines/pipeline/dataset_ingress_v1.md +++ b/dpypelines/pipeline/dataset_ingress_v1.md @@ -40,7 +40,7 @@ Set the required environment variables by opening your terminal and entering the ```bash export DISABLE_NOTIFICATIONS=true export SKIP_DATA_UPLOAD=true -export UPLOAD_SERVICE_URL=not-used +export UPLOAD_SERVICE_URL=not-used/not-required ``` Then run `myscript.py` as [described above](#running-locally). This will run the full transform but will stop short of uploading outputs to the [dp-upload-service](https://github.com/ONSdigital/dp-upload-service). diff --git a/dpypelines/pipeline/generic_file_ingress_v1.md b/dpypelines/pipeline/generic_file_ingress_v1.md index bc9f4678..f6ab4e1b 100644 --- a/dpypelines/pipeline/generic_file_ingress_v1.md +++ b/dpypelines/pipeline/generic_file_ingress_v1.md @@ -40,7 +40,7 @@ Set the required environment variables by opening your terminal and entering the ```bash export DISABLE_NOTIFICATIONS=true export SKIP_DATA_UPLOAD=true -export UPLOAD_SERVICE_URL=not-used +export UPLOAD_SERVICE_URL=not-used/not-required ``` Then run `myscript.py` as [described above](#running-locally). This will run the `generic_file_ingress_v1` function but will stop short of uploading outputs to the [dp-upload-service](https://github.com/ONSdigital/dp-upload-service). diff --git a/myscript.py b/myscript.py deleted file mode 100644 index 32801428..00000000 --- a/myscript.py +++ /dev/null @@ -1,18 +0,0 @@ -#from dpypelines import s3_tar_received - -#s3_tar_received.start('joes-bucket-will-be-deleted/config-no-options.tar') - -import sys -from behave.__main__ import run_behave -from behave.configuration import Configuration - -if __name__ == "__main__": - # args = sys.argv[1:] if len(sys.argv) > 1 else [] - args = [ - "--verbose", - "features/dataset_ingress.feature", # Feature file path - "-n", - "Generic ingress runs without errors", # Scenario text - ] - configuration = Configuration(args) - sys.exit(run_behave(configuration)) \ No newline at end of file diff --git a/tests/README.md b/tests/README.md index 46409041..d28a64ef 100644 --- a/tests/README.md +++ b/tests/README.md @@ -1 +1,7 @@ -# TODO +# `tests` Directory + +This directory contains the `unit tests` and the required `test-cases`. + +`unit tests` are small test to verify each function is working ad=s intended. + +`test-cases` are test files that are used to run unit tests. \ No newline at end of file From b628fbc878b5c2c5bbc8fcf10ac4ad230ca699a8 Mon Sep 17 00:00:00 2001 From: Jozsef K Date: Thu, 31 Oct 2024 08:22:46 +0000 Subject: [PATCH 3/4] f --- requirements.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/requirements.md b/requirements.md index 9a8db8a4..f67529f6 100644 --- a/requirements.md +++ b/requirements.md @@ -4,7 +4,7 @@ This document outlines the [input requirements](#inputs) for files to be submitt ## Inputs -The DTV pipeline is triggered when a single `.tar` file is uploaded to the designated AWS S3 bucket. This `.tar` file should contain the file(s) to be transformed, any supplementary distribution files, and a `manifest.json` file for configuring the pipeline. +The DTV pipeline is triggered when a single or multiple `.tar` file/files is uploaded to the designated AWS S3 bucket. This `.tar` file should contain the file(s) to be transformed, any supplementary distribution files, and a `manifest.json` file for configuring the pipeline. Business areas wishing to submit datasets to the DTV pipeline will add the relevant files to an MS Sharepoint app/folder. The primary issue to be resolved is how the pipeline will be triggered when multiple files to be processed in one submission are added to Sharepoint. From a39a97c38d7e10f586276d18166639255a1e28b8 Mon Sep 17 00:00:00 2001 From: Jozsef K Date: Fri, 1 Nov 2024 09:45:50 +0000 Subject: [PATCH 4/4] comments addressed --- dpypelines/pipeline/dataset_ingress_v1.md | 2 +- .../pipeline/generic_file_ingress_v1.md | 2 +- features/README.md | 23 ++----------------- requirements.md | 2 +- tests/README.md | 2 +- 5 files changed, 6 insertions(+), 25 deletions(-) diff --git a/dpypelines/pipeline/dataset_ingress_v1.md b/dpypelines/pipeline/dataset_ingress_v1.md index c3f9286e..ffc626b6 100644 --- a/dpypelines/pipeline/dataset_ingress_v1.md +++ b/dpypelines/pipeline/dataset_ingress_v1.md @@ -40,7 +40,7 @@ Set the required environment variables by opening your terminal and entering the ```bash export DISABLE_NOTIFICATIONS=true export SKIP_DATA_UPLOAD=true -export UPLOAD_SERVICE_URL=not-used/not-required +export UPLOAD_SERVICE_URL=not-used ``` Then run `myscript.py` as [described above](#running-locally). This will run the full transform but will stop short of uploading outputs to the [dp-upload-service](https://github.com/ONSdigital/dp-upload-service). diff --git a/dpypelines/pipeline/generic_file_ingress_v1.md b/dpypelines/pipeline/generic_file_ingress_v1.md index f6ab4e1b..bc9f4678 100644 --- a/dpypelines/pipeline/generic_file_ingress_v1.md +++ b/dpypelines/pipeline/generic_file_ingress_v1.md @@ -40,7 +40,7 @@ Set the required environment variables by opening your terminal and entering the ```bash export DISABLE_NOTIFICATIONS=true export SKIP_DATA_UPLOAD=true -export UPLOAD_SERVICE_URL=not-used/not-required +export UPLOAD_SERVICE_URL=not-used ``` Then run `myscript.py` as [described above](#running-locally). This will run the `generic_file_ingress_v1` function but will stop short of uploading outputs to the [dp-upload-service](https://github.com/ONSdigital/dp-upload-service). diff --git a/features/README.md b/features/README.md index 5e063200..96d54c18 100644 --- a/features/README.md +++ b/features/README.md @@ -1,22 +1,3 @@ -# `enviroment` setup +# `environment` setup -This scipt sets up an enviorment (docker container) for testing and after testing has been completed removes the container and temporary directories. - -## `before_all` function - -The script sets up a container representing a fake backend to run tests. These test confirms that the request are routing as required, also checks the fixtures unzipped data paths are correct/exist. - -## `before_scenario` function - -This function will run bfeore each scenario would be ran. Creates a temporary directory and changes current working directory to the new temporary directory. This allows test files to be placed in there and later safely can be deleted. - -Set the `UUID` (Universal Unique Identifier) in the docker logs. -Sets each scenario with a `custom Session` so some of the default set headers will be removed. - -## `after_scenario` function - -This function removes the temporary directories and all contents, then changes directory out of the temporary directory. - -## `after_all` function - -This function stops and removes the `docker container`, also remove temporary directories for test output files. \ No newline at end of file +This script sets up an environment for acceptance testing and removes it once testing has been completed. \ No newline at end of file diff --git a/requirements.md b/requirements.md index f67529f6..a94f8b77 100644 --- a/requirements.md +++ b/requirements.md @@ -4,7 +4,7 @@ This document outlines the [input requirements](#inputs) for files to be submitt ## Inputs -The DTV pipeline is triggered when a single or multiple `.tar` file/files is uploaded to the designated AWS S3 bucket. This `.tar` file should contain the file(s) to be transformed, any supplementary distribution files, and a `manifest.json` file for configuring the pipeline. +The DTV pipeline is triggered when one or more .tar files is uploaded to the designated AWS S3 bucket. Each .tar file should contain the file(s) to be transformed, any supplementary distribution files, and a manifest.json file for configuring the pipeline. Business areas wishing to submit datasets to the DTV pipeline will add the relevant files to an MS Sharepoint app/folder. The primary issue to be resolved is how the pipeline will be triggered when multiple files to be processed in one submission are added to Sharepoint. diff --git a/tests/README.md b/tests/README.md index d28a64ef..e3a3b352 100644 --- a/tests/README.md +++ b/tests/README.md @@ -2,6 +2,6 @@ This directory contains the `unit tests` and the required `test-cases`. -`unit tests` are small test to verify each function is working ad=s intended. +`unit tests` are small tests to verify each function is working as intended. `test-cases` are test files that are used to run unit tests. \ No newline at end of file