Packaging (#48)

This is a major restructuring of the source code for python packaging. The way importing works has therefore also changed, see the notebooks for examples. Some minor bug fixes: * Small fix in plot_utils * Fix AHN fuser when no ground points are found Co-authored-by: chrise96 <[email protected]>
Amsterdam-AI-Team · Feb 18, 2022 · f7e12e4 · f7e12e4
1 parent 505cfaa
commit f7e12e4
Show file tree

Hide file tree

Showing 51 changed files with 412 additions and 171 deletions.
diff --git a/.gitignore b/.gitignore
@@ -3,9 +3,13 @@
 *.las
 *.LAZ
 *.csv
+*.txt
 *.gml
 *.npz
 *.TIF
+*.tif
+*.qgz
+*.html
 *.png
 
 # Byte-compiled / optimized / DLL files

diff --git a/README.md b/README.md
@@ -39,31 +39,52 @@ For a quick dive into this repository take a look at our [complete solution note
    * [`examples`](./media/examples)
  * [`notebooks`](./notebooks) _Jupyter notebook tutorials_
  * [`scripts`](./scripts) _Python scripts_
- * [`src`](./src) _Python source code_
-   * [`fusion`](./src/fusion) _Data fusion code_
-   * [`preprocessing`](./src/preprocessing) _Pre-processing code_
-   * [`region_growing`](./src/region_growing) _Region growing code_
-   * [`scrapers`](./src/scrapers) _Data scrapers_
-   * [`utils`](./src/utils) _Utility functions_
-
-
+ * [`src/upcp`](./src/upcp) _Python source code_
+   * [`analysis`](./src/upcp/analysis) _Dataset analysis code_
+   * [`fusion`](./src/upcp/fusion) _Data fusion code_
+   * [`preprocessing`](./src/upcp/preprocessing) _Pre-processing code_
+   * [`region_growing`](./src/upcp/region_growing) _Region growing code_
+   * [`scrapers`](./src/upcp/scrapers) _Data scrapers_
+   * [`utils`](./src/upcp/utils) _Utility functions_
 
 ---
 
 ## Installation
 
-1. Clone this repository:
+This code has been tested with `Python 3.8` on `Linux` and `MacOS`, and should likely work under Windows as well.
+
+1.  To use this code in development mode simply clone the repository and install the dependencies.
+
     ```bash
+    # Clone the repository
     git clone https://github.com/Amsterdam-AI-Team/Urban_PointCloud_Processing.git
+
+    # Install dependencies
+    cd Urban_PointCloud_Processing
+    python -m pip install -r requirements.txt
+
+    # Optionally install JupyterLab to use the notebooks
+    python -m pip install jupyterlab ipympl ipywidgets
     ```
 
-2. Install all dependencies:
+2.  Alternatively, the code can be installed as a Python package from source:
+
     ```bash
-    pip install -r requirements.txt
+    # Install the latest release as Wheel
+    python -m pip install
+    https://github.com/Amsterdam-AI-Team/Urban_PointCloud_Processing/archive/refs/tags/upcp-0.1-py3-none-any.whl
+
+    # Alternatively, install the latest version from source
+    python -m pip install git+https://github.com/Amsterdam-AI-Team/Urban_PointCloud_Processing.git#egg=upcp
+
+    # Or, after making changes in the code
+    cd Urban_PointCloud_Processing
+    python -m pip install .
     ```
-    Additionally, install `cccorelib` and `pycc` by following the [instructions on their GitHub page](https://github.com/tmontaigu/CloudCompare-PythonPlugin/blob/master/docs/building.rst#building-as-indenpendent-wheels).
 
-3. Check out the [notebooks](notebooks) for a demonstration.
+    If you use the latter and want your code changes to take effect without re-installing the package, use the `--editable` flag for pip.
+
+**Additionally**, install `cccorelib` and `pycc` by following the [instructions on their GitHub page](https://github.com/tmontaigu/CloudCompare-PythonPlugin/blob/master/docs/building.rst#building-as-independent-wheels). Please note, these two packages are not available on the Python Package Index (PyPi).
 
 ---
 
@@ -88,7 +109,6 @@ If you use (parts of) this repositiory in your work, please cite [our paper](htt
 }
 ```
 
-
 ---
 
 ## Acknowledgements

diff --git a/notebooks/0. Complete solution.ipynb b/notebooks/0. Complete solution.ipynb
@@ -17,24 +17,33 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "responsible-network",
+   "id": "055462da-8dd8-4e26-a4a5-773fd613b6d9",
    "metadata": {},
    "outputs": [],
    "source": [
+    "# Uncomment to load the local package rather than the pip-installed version.\n",
     "# Add project src to path.\n",
-    "import set_path\n",
-    "\n",
+    "import set_path"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "d6514388-d99e-4c90-a8fb-e3256bb2c8a9",
+   "metadata": {},
+   "outputs": [],
+   "source": [
     "# Import modules.\n",
     "import logging\n",
     "\n",
-    "import src.fusion as fusion\n",
-    "import src.region_growing as growing\n",
-    "import src.utils.ahn_utils as ahn_utils\n",
-    "import src.utils.bgt_utils as bgt_utils\n",
-    "import src.utils.plot_utils as plot_utils\n",
-    "import src.utils.log_utils as log_utils\n",
-    "from src.pipeline import Pipeline\n",
-    "from src.labels import Labels"
+    "import upcp.fusion as fusion\n",
+    "import upcp.region_growing as growing\n",
+    "import upcp.utils.ahn_utils as ahn_utils\n",
+    "import upcp.utils.bgt_utils as bgt_utils\n",
+    "import upcp.utils.plot_utils as plot_utils\n",
+    "import upcp.utils.log_utils as log_utils\n",
+    "from upcp.pipeline import Pipeline\n",
+    "from upcp.labels import Labels"
    ]
   },
   {
@@ -45,12 +54,12 @@
    "outputs": [],
    "source": [
     "# Set-up logging.\n",
-    "logfile = 'pipeline.log'\n",
+    "logfile = 'upcp.pipeline.log'\n",
     "log_utils.reset_logger()\n",
     "\n",
     "# INFO messages will be printed to console.\n",
     "log_utils.add_console_logger()\n",
-    "# All messages will be printed to a file. Use `clear_log` flag to clear the log file, if desired.\n",
+    "# DEBUG messages will be printed to a file. Use `clear_log` flag to clear the log file, if desired.\n",
     "log_utils.add_file_logger(logfile, clear_log=True)"
    ]
   },
@@ -389,13 +398,8 @@
     "# The prefix will be replaced for processed files (otherwise the input might be overwritten).\n",
     "out_prefix = 'processed_'\n",
     "\n",
-    "# Re-set logging to suppress console output.\n",
-    "log_utils.reset_logger()\n",
-    "\n",
-    "# Only ERROR messages will be printed to console.\n",
-    "log_utils.add_console_logger(level=logging.ERROR)\n",
-    "# All messages will be printed to a file. Use `clear_log` flag to clear the log file, if desired.\n",
-    "log_utils.add_file_logger(logfile, clear_log=True)"
+    "# Suppress logging other than ERROR messages to console.\n",
+    "log_utils.set_console_level(level=logging.ERROR)"
    ]
   },
   {
@@ -480,7 +484,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.8.2"
+   "version": "3.8.12"
   }
  },
  "nbformat": 4,

diff --git a/notebooks/1. AHN preprocessing.ipynb b/notebooks/1. AHN preprocessing.ipynb
@@ -5,31 +5,40 @@
    "id": "c3fd9730-0dc5-4520-a485-b2445c984820",
    "metadata": {},
    "source": [
-    "# Pre-processing of LAS tiles\n",
+    "python3 -m build# Pre-processing of LAS tiles\n",
     "\n",
     "Pre-processing consists of two parts:\n",
     "1. Clipping of AHN point clouds to match target point clouds;\n",
     "2. Ground and building surface extraction."
    ]
   },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "57f542f8-21bf-4382-bfb9-11b11b05802b",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Uncomment to load the local package rather than the pip-installed version.\n",
+    "# Add project src to path.\n",
+    "import set_path"
+   ]
+  },
   {
    "cell_type": "code",
    "execution_count": null,
    "id": "8adb4173-257d-49f2-9b20-c5af54f642b5",
    "metadata": {},
    "outputs": [],
    "source": [
-    "# Helper script to allow importing from parent folder.\n",
-    "import set_path\n",
-    "\n",
     "# Import modules.\n",
     "import numpy as np\n",
     "import time\n",
     "\n",
-    "import src.preprocessing.ahn_preprocessing as ahn_preprocessing\n",
-    "import src.utils.ahn_utils as ahn_utils\n",
-    "import src.utils.las_utils as las_utils\n",
-    "import src.utils.plot_utils as plot_utils"
+    "import upcp.preprocessing.ahn_preprocessing as ahn_preprocessing\n",
+    "import upcp.utils.ahn_utils as ahn_utils\n",
+    "import upcp.utils.las_utils as las_utils\n",
+    "import upcp.utils.plot_utils as plot_utils"
    ]
   },
   {
@@ -207,7 +216,7 @@
  ],
  "metadata": {
   "kernelspec": {
-   "display_name": "Python 3",
+   "display_name": "Python 3 (ipykernel)",
    "language": "python",
    "name": "python3"
   },
@@ -221,7 +230,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.7.10"
+   "version": "3.8.12"
   }
  },
  "nbformat": 4,

diff --git a/notebooks/1. Generate reference data.ipynb b/notebooks/1. Generate reference data.ipynb
@@ -15,17 +15,26 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "productive-median",
+   "id": "03f66669-5b3f-402e-a9b7-a1279f778fcd",
    "metadata": {},
    "outputs": [],
    "source": [
+    "# Uncomment to load the local package rather than the pip-installed version.\n",
     "# Add project src to path.\n",
-    "import set_path\n",
-    "\n",
-    "import src.scrapers.ams_bgt_scraper as ams_bgt_scraper\n",
-    "import src.scrapers.ndw_scraper as ndw_scraper\n",
-    "import src.utils.las_utils as las_utils\n",
-    "import src.utils.csv_utils as csv_utils"
+    "import set_path"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "productive-median",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import upcp.scrapers.ams_bgt_scraper as ams_bgt_scraper\n",
+    "import upcp.scrapers.ndw_scraper as ndw_scraper\n",
+    "import upcp.utils.las_utils as las_utils\n",
+    "import upcp.utils.csv_utils as csv_utils"
    ]
   },
   {
@@ -245,7 +254,7 @@
  ],
  "metadata": {
   "kernelspec": {
-   "display_name": "Python 3",
+   "display_name": "Python 3 (ipykernel)",
    "language": "python",
    "name": "python3"
   },
@@ -259,7 +268,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.8.5"
+   "version": "3.8.12"
   }
  },
  "nbformat": 4,

diff --git a/notebooks/2. Ground and Buildings.ipynb b/notebooks/2. Ground and Buildings.ipynb
@@ -17,25 +17,34 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "c4489f95-98a5-4ed2-87e7-36a0554960a2",
+   "id": "9b93f546-68ee-4b1d-b0eb-3f3479292aa7",
    "metadata": {},
    "outputs": [],
    "source": [
+    "# Uncomment to load the local package rather than the pip-installed version.\n",
     "# Add project src to path.\n",
-    "import set_path\n",
-    "\n",
+    "import set_path"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "c4489f95-98a5-4ed2-87e7-36a0554960a2",
+   "metadata": {},
+   "outputs": [],
+   "source": [
     "# Import modules.\n",
     "import logging\n",
     "\n",
-    "import src.fusion as fusion\n",
-    "from src.pipeline import Pipeline\n",
-    "import src.region_growing as growing\n",
-    "import src.utils.ahn_utils as ahn_utils\n",
-    "import src.utils.bgt_utils as bgt_utils\n",
-    "import src.utils.las_utils as las_utils\n",
-    "import src.utils.log_utils as log_utils\n",
-    "import src.utils.csv_utils as csv_utils\n",
-    "from src.labels import Labels\n",
+    "import upcp.fusion as fusion\n",
+    "from upcp.pipeline import Pipeline\n",
+    "import upcp.region_growing as growing\n",
+    "import upcp.utils.ahn_utils as ahn_utils\n",
+    "import upcp.utils.bgt_utils as bgt_utils\n",
+    "import upcp.utils.las_utils as las_utils\n",
+    "import upcp.utils.log_utils as log_utils\n",
+    "import upcp.utils.csv_utils as csv_utils\n",
+    "from upcp.labels import Labels\n",
     "\n",
     "# INFO messages will be printed to console.\n",
     "log_utils.reset_logger()\n",
@@ -171,6 +180,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
+   "id": "c1bac19b",
    "metadata": {},
    "outputs": [],
    "source": [
@@ -184,6 +194,7 @@
   },
   {
    "cell_type": "markdown",
+   "id": "03a05342",
    "metadata": {},
    "source": [
     "### [Alternative] Building fuser using BGT data\n",
@@ -295,6 +306,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
+   "id": "755a8b87",
    "metadata": {},
    "outputs": [],
    "source": [
@@ -306,6 +318,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
+   "id": "42d2419e",
    "metadata": {},
    "outputs": [],
    "source": [
@@ -350,7 +363,7 @@
  ],
  "metadata": {
   "kernelspec": {
-   "display_name": "Python 3",
+   "display_name": "Python 3 (ipykernel)",
    "language": "python",
    "name": "python3"
   },
@@ -364,7 +377,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.7.10"
+   "version": "3.8.12"
   }
  },
  "nbformat": 4,