OSC Data Extractor Pre-Steps

OS-Climate Data Extraction Tool

This code provides you with an api and a streamlit app to which you can provide a pdf document and the output will be the text content in a json format. In the backend it is using a python module for extracting text from pdfs, which might be extended in the future to other file types. The json file is needed for later usage in the context of transformer models to extract relevant information, but it can also be used independently.

Quick start

Install via PyPi

You can simply install the package via:

$ pip install osc-transformer-presteps

Afterwards you can use the tooling as a CLI tool by simply typing:

$ osc-transformer-presteps

We are using typer to have a nice CLI tool here. All details and help will be shown in the CLI tool itself and are not described here in more detail.

Install via Github Repository

For a quick start with the tool install python and clone the repository to your local environment:

$ git clone https://github.com/os-climate/osc-transformer-presteps

Afterwards update your python to the requirements (possible for example via pdm update) and start a local api server via:

$ python ./src/run_server.py

Note:

• We assume that you are located in the cloned repository.
• To check if it is running open "http://localhost:8000/liveness" and you should see the message {"message": "OSC Transformer Pre-Steps Server is running."}.

Finally, run the following code to start a streamlit app which opens up the possibility to "upload" a file and extract data from pdf to json via this UI. Note that the UI needs the running server so you have to open the streamlit and the server in two different terminals.:

$ streamlit run ./src/osc_transformer_presteps/streamlit/app.py

Note: Check also docs/demo. There you can find local_extraction_demo.py which will start an extraction without any API call and then there is post_request_demo.py which will send a file to the API (of course you have to start server as above first).

Developer Notes

For adding new dependencies use pdm. First install via pip:

$ pip install pdm

And then you could add new packages via pdm add. For example numpy via:

$ pdm add numpy

For running linting tools just to the following:

$ pip install tox
$ tox -e lint
$ tox -e test