companies-house-streaming-etl

Proof of concept for use of the Companies House Streaming API to be integrated with our ETL

• See API documentation here

Before merging a PR and publishing with Jenkins

Ensure the dependencies have been updated, see here Increment .version to create an updated version tag in ECR

To run the streamer locally:

First get an API key:

• Follow instruction here to create an account
• Make a new application and find the key in the manage-applications page

Create a yaml file at companies_house_streaming_etl/config.yml it should like this with your api key

stream_key: ...

• stream_key Your api key obtained following the instructions above

First please make sure that you:

• are using the Python version specified in .python-version
• have the Poetry Python dependency manager installed by following instructions here

Install poetry dependencies with:

poetry install

If you want to run without docker:

In a terminal, run the streamer by executing the following:

CH_DEBUG="true" CH_WRITE_LOCATION="local" CH_WRITE_BUCKET="n/a" CH_WRITE_PREFIX="n/a" poetry run run-streamer

if you want to run with Docker (useful for testing lambda deployment)

If dependencies have changed

Ensure that the requirements file has been updated too (this is done to avoid complications with poetry in docker)

poetry update

poetry export -f requirements.txt --without-hashes > requirements.txt

make the docker image

make build

run the docker image in a container

make run-local

License

This code is open source software licensed under the Apache 2.0 License.