Reproducible scientific workflows for accessing, processing, and visualizing USGS 3DEP lidar data

The U.S. Geological Survey 3D Elevation Program (3DEP) is acquiring high-quality light detecting and ranging (lidar) data over the conterminous United States, Hawaii, and US Territories to meet the growing need for high-resolution 3-D representations of Earth's surface, vegetation, and other constructed features.

This repository contains a suite of customizable and reproducible workflows implemented as Jupyter Notebooks for programmatically accessing, processing, and visualizing USGS 3DEP lidar point cloud data and creating a range of derivative products (DEMs, topographic difference maps, canopy height models, and colorized point clouds). Each notebook is designed to address a specific use case/geospatial application and leverages open source geospatial software (e.g. PDAL,GDAL) along with detailed documentation that lower the barrier of entry for users with minimal programming experience.

Installation

The Jupyter Notebooks can be run locally or on the Google Colaboratory cloud platform (Requires Google account with Google Drive access).

Dependencies

• gdal
• geopandas
• ipyleaflet
• matplotlib
• pdal
• pyproj
• requests
• rioxarray

Option 1 (Recommended): Google Colaboratory Installation

For quickly getting started and ease-of-use, it is recommended to launch and execute these notebooks on the Google Colaboratory ("Colab") cloud platform. Dependencies will be installed on a virtual environment on Google's cloud servers and the code will be executed in the browser. A major benefit is direct access to Google high-end CPU/GPUs and avoiding software dependency issues. All products will be saved to your personal Google Drive. Google Colab users are allocated 12GB of RAM (which can be upgraded to 25GB with Colab Pro). User's are advised that accessing and processing large point clouds (>200,000,000 points) may consume this RAM allocation and cause the runtime to fail. Those seeking to process large quantities of data may benefit from running the notebooks locally (Option 2).

To experiment and run one of the below Jupyter Notebooks on Google Colab click the badge beneath the corresponding Jupyter Notebook in the USGS 3DEP Jupyter Notebooks section, below.

Option 2: Local Installation

In order to run the Jupyter Notebook on your local machine, follow these steps:

Make a new directory on your local file system where the 3DEP Jupyter Notebooks (and all 3DEP data, if desired) will be saved. In this case, the directory will be called 3DEP. Change into the new directory and git clone the Github repository containing the Jupyter Notebooks and other relevant files to your local file system.

mkdir 3DEP
cd 3DEP
git clone https://github.com/cmspeed/OT_3DEP_Workflows

Anaconda is recommended for Python package installation and management. Package versions in Anaconda are managed by the package management system conda. Anaconda installers for MacOS/Linux/Windows can be downloaded from https://docs.anaconda.com/anaconda/install/. Follow the instructions to install the appropriate version of Anaconda for your operating system.

After installing Anaconda, create a conda virtual environment with the required dependencies (contained in environment.yml). Note: Exectuting the following command will automatically create the conda environment with name 3dep and all of the required dependencies installed. If you would prefer a different name, replace 3dep with another name in the following command:

cd OT_3DEP_Workflows
conda env create -n 3dep --file environment.yml

Activate the conda environment with all of the necessary dependencies installed. (If different, substitute '3dep' for the name of your new conda environment.)

conda activate 3dep

Now, launch the chosen Jupyter Notebook. If unsure how to launch a Notebook, refer to this guide (https://jupyter-notebook-beginner-guide.readthedocs.io/en/latest/execute.html).

USGS 3DEP Jupyter Notebooks

1. Generate and visualize DEMs (DTM and DSM) from USGS 3D Elevation Program (3DEP) lidar data for user-defined area of interest
   

2. Generate and visualize DEMs (DTM and DSM) from USGS 3D Elevation Program (3DEP) lidar data for USGS 7.5’ Quadrangles
   

3. Generate and visualize DEMs (DTM and DSM) from USGS 3D Elevation Program (3DEP) lidar data for USGS Hydrologic Units
   

4. Generate and visualize DEMs (DTM and DSM) from USGS 3D Elevation Program (3DEP) lidar data for user-defined corridors
   

5. Generate Canopy Height Model (CHM) using USGS 3D Elevation Program (3DEP) lidar data for user-defined area of interest
   

6. Topographic Differencing using USGS 3D Elevation Program (3DEP) lidar data for user-defined area of interest
   

7. Generate colorized (RGB) point clouds using USGS 3D Elevation Program (3DEP) lidar data and National Agriculture Imagery Program (NAIP) Imagery

Issues and Feature Improvements

Reporting issues and/or providing suggestions for improving these workflows or developing new ones is highly encouranged. Issues or suggested feature improvements may be submitted as an issue or by email to OpenTopography Support.

Funding

Funding for the creation and distribution of these Jupyter Notebook-based workflows was provided as by the USGS Community for Data Integration (CDI) through the funded grant Enhancing usability of 3DEP data and web services with Jupyter Notebooks.

OpenTopography is supported by the National Science Foundation (NSF) under Award Numbers 1948997, 1948994 & 1948857.

Additional Resources

• The USGS 3DEP lidar point ploud pata are accessible in Entwine Point Tile (EPT) format from this Amazon Web Services S3 Bucket.

• The USGS hydrologic unit boundaries are accessed via the USGS Watershed Dataset Map Service.

• The USGS 7.5' quadrangle boundaries are accessed via the USGS Map Indicies Service.

• Documentation for open-source Python libararies used by these workflows include PDAL and GDAL.

• The UNAVCO Student Internship Program (USIP).

• Access USGS 3DEP via the OpenTopography portal (currently restricted to academic use).