Conversions_ALOS2-SLC_to_NISAR-L2_cryo.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "a6195696-d4f9-425c-b7a9-79c7aac18903",
   "metadata": {},
   "source": [
    "# <img src=\"https://upload.wikimedia.org/wikipedia/commons/6/60/NISAR_artist_concept.jpg\" width=400 align=\"left\"/>\n",
    "<img src=\"https://upload.wikimedia.org/wikipedia/commons/9/9b/NISAR_Mission_Logo.png\" width=400 align=\"left\"/><br><br><br><br><br>\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3e7ba3b7-d7e0-4f8d-be90-f4fea8db69c0",
   "metadata": {},
   "source": [
    "# Prepare ALOS2 data and generate L2 NISAR-like InSAR data products\n",
    "\n",
    "(*This workflow was adapted from Alex C.'s GCOV workflow, for InSAR products. For questions, reach out to Catalina T.*)\n",
    "\n",
    "In preparation for NISAR, this notebook **converts ALOS-2 SLC data from ASF to NISAR RSLC and generates NISAR L1 and L2 sample data products using ISCE3 InSAR (e.g. RUNW, GUNW)**:\n",
    "1. Downloads desired ALOS-2 pair from ASF (note these data are restricted under NASA-JAXA agreement)\n",
    "2. Converts ALOS2 SLC to NISAR RSLC using `alos2_to_nisar_l1.py`\n",
    "3. Processes a DEM for the study area using the `stage_dem.py` from isce3\n",
    "4. Processes NISAR RSLC to GUNW using `insar.py` from isce3 and a `runconfig_insar.yaml` ← change parameters for custom products\n",
    "- `runconfig_insar_VB_template.yaml` ← standard unwrapping 80m\n",
    "- `runconfig_insar_20m_unw.yaml` ← unwrap at 20m\n",
    "\n",
    "Notes:\n",
    "- This workflow example is designed to process a single pair at a time. Granule ID should be known ahead of time and used for folder structure. e.g. pair ALOS2456335530-221104_ALOS2458405530-221118\n",
    "- For a PCM job, workflow may need to be adjusted to optimize pairing up and processing.\n",
    "- Current workflow runs on `/scratch` but if it's possible to stage all files on `s3` and link from there, that might be better since `s3` is cheaper and backed up compared to `/scratch`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "b3a41cd0-91b7-4afe-88ad-dbce4b3e941a",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import sys\n",
    "from pathlib import Path\n",
    "import subprocess\n",
    "import fnmatch\n",
    "import zipfile\n",
    "import h5py\n",
    "from string import Template"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "15590b4e-2359-4f57-9953-6b2c7aef75b5",
   "metadata": {},
   "source": [
    "#### Choose your place to save your outputs. \n",
    "\n",
    "`/scratch` can be used for short-term storage, but the cost is much higher and the space is not backed up. Files for long-term storage should be moved to an S3 bucket."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "f2b3d3d1-4f46-403c-b680-628cfcb7c682",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "scratch_folder = Path('/scratch/taglialatela_cryo_test/sample_products/ALOS2/')   ##change to your folder"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ec3525da-c42f-4fab-9f71-6654b05bd3bf",
   "metadata": {},
   "source": [
    "#### Choose your AOI\n",
    "\n",
    "New directories will be made to store files for this AOI"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "602162d6-d82a-4ff8-a14b-923cfa434733",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "aoi = 'antarctica'\n",
    "\n",
    "aoi_dir = scratch_folder/aoi\n",
    "PAIR_dir = aoi_dir / 'ALOS2456335530-221104-ALOS2458405530-221118'   #manual update based on granuleIDs of pair to process\n",
    "ALOS2_dir1 = aoi_dir / PAIR_dir / 'ALOS2_ref'\n",
    "ALOS2_dir2 = aoi_dir / PAIR_dir / 'ALOS2_sec'\n",
    "DEM_dir = aoi_dir / PAIR_dir / 'DEM'\n",
    "INSAR_dir = aoi_dir / PAIR_dir / 'INSAR'\n",
    "unwrap_80m_dir = aoi_dir / PAIR_dir / INSAR_dir / 'unwrap_80m'\n",
    "output_insar_dir = aoi_dir / PAIR_dir / INSAR_dir / unwrap_80m_dir / 'output_insar'\n",
    "scratch_insar_dir = aoi_dir / PAIR_dir / INSAR_dir / unwrap_80m_dir / 'scratch_insar'\n",
    "RSLC_dir1 = aoi_dir /  PAIR_dir / 'RSLC_ref'\n",
    "RSLC_dir2 = aoi_dir /  PAIR_dir / 'RSLC_sec'\n",
    "\n",
    "Path(aoi_dir).mkdir(parents=True, exist_ok=True)\n",
    "Path(PAIR_dir).mkdir(parents=True, exist_ok=True)\n",
    "Path(ALOS2_dir1).mkdir(parents=True, exist_ok=True)\n",
    "Path(ALOS2_dir2).mkdir(parents=True, exist_ok=True)\n",
    "Path(DEM_dir).mkdir(parents=True, exist_ok=True)\n",
    "Path(INSAR_dir).mkdir(parents=True, exist_ok=True)\n",
    "Path(unwrap_80m_dir).mkdir(parents=True, exist_ok=True)\n",
    "Path(output_insar_dir).mkdir(parents=True, exist_ok=True)\n",
    "Path(scratch_insar_dir).mkdir(parents=True, exist_ok=True)\n",
    "Path(RSLC_dir1).mkdir(parents=True, exist_ok=True)\n",
    "Path(RSLC_dir2).mkdir(parents=True, exist_ok=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5e80d13b-1dc0-4c12-8ec5-46e8c4ef43fd",
   "metadata": {
    "tags": []
   },
   "source": [
    "_**Skip the download steps if already downloaded** these files previsouly. e.g. provide s3 link to ALOS2 data in NISAR ODS instead._\n",
    "\n",
    "#### Copy the ALOS2 URLs into a list\n",
    "\n",
    "Workflow build to run a single pair (2 frames) at a time. Here we define the 2 frames for the pair we want to process, by providing the ASF link for download. Corresponding links are in the ASF-ALOS2 spreadsheet. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "cbf4a616-679b-4961-976c-0304a7936912",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "ALOS_SLC_ref = 'https://cumulus.asf.alaska.edu/L1.1/A4/0000487643_001001_ALOS2456335530-221104.zip'\n",
    "ALOS_SLC_sec = 'https://cumulus.asf.alaska.edu/L1.1/A4/0000489662_001001_ALOS2458405530-221118.zip'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "93089f0b-7724-42c1-b26e-8fb01f10d618",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "https://cumulus.asf.alaska.edu/L1.1/A4/0000487643_001001_ALOS2456335530-221104.zip\n",
      "https://cumulus.asf.alaska.edu/L1.1/A4/0000489662_001001_ALOS2458405530-221118.zip\n"
     ]
    }
   ],
   "source": [
    "print(ALOS_SLC_ref)\n",
    "print(ALOS_SLC_sec)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c32a3160-3d4b-424d-be88-5ad1a0a28b58",
   "metadata": {
    "tags": []
   },
   "source": [
    "#### Download the ALOS2 files using wget\n",
    "\n",
    "Download 1 pair at a time. \n",
    "\n",
    "This seems to take some time. Single zip around 6 GB"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "33df314c-cdf1-4afd-94b3-5638006dad27",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "filename1 = ALOS2_dir1 / ALOS_SLC_ref.split('/')[-1]\n",
    "if os.path.isfile(filename1)==False:\n",
    "    command = f\"wget -P {ALOS2_dir1} -q {ALOS_SLC_ref}\"\n",
    "    output = subprocess.check_output(command, shell=True)    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "30a76119-86cd-4070-97ab-4440273e373a",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "filename2 = ALOS2_dir2 / ALOS_SLC_sec.split('/')[-1]\n",
    "if os.path.isfile(filename2)==False:\n",
    "    command = f\"wget -P {ALOS2_dir2} -q {ALOS_SLC_sec}\"\n",
    "    output = subprocess.check_output(command, shell=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e2a9e0ef-aef8-4782-b100-c5059d254567",
   "metadata": {},
   "source": [
    "#### Unzip the ALOS2 files\n",
    "\n",
    "Note: `alos2_to_nisar_l1.py` assumes there is only one directory or file it is pointing to. The zip file should be unzipped individually and left in their own directory.\n",
    "\n",
    "If you don't want to keep the zip files, you should uncomment the `os.remove` line"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a0190501-2df3-4329-8b55-39b815ca9d95",
   "metadata": {},
   "source": [
    "##### (a) Unzip the *reference* ALOS2 data in its own folder:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "b838d660-f2b2-40e9-ae9b-e0b05d74d05a",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/scratch/taglialatela_cryo_test/sample_products/ALOS2/antarctica/ALOS2456335530-221104-ALOS2458405530-221118/ALOS2_ref/0000487643_001001_ALOS2456335530-221104.zip\n"
     ]
    }
   ],
   "source": [
    "zipfiles = [os.path.join(dirpath,f)\n",
    "                for dirpath,dirnames, files in os.walk(ALOS2_dir1)\n",
    "                for f in fnmatch.filter(files,'*.zip')]\n",
    "for file in zipfiles:\n",
    "    print(file)\n",
    "    if os.path.isdir(ALOS2_dir1 / file[:-4]):\n",
    "        ##check full unzipping\n",
    "        if sum(os.path.getsize(ALOS2_dir1 / file[:-4] / f) for f in os.listdir(ALOS2_dir1 / file[:-4]) if os.path.isfile(ALOS2_dir1 / file[:-4] /f)) < 6509714637:\n",
    "            with zipfile.ZipFile(file, 'r') as zip_ref:\n",
    "                zip_ref.extractall(ALOS2_dir1/file.split('/')[-1][:-4])\n",
    "            os.remove(file)\n",
    "    else:\n",
    "        with zipfile.ZipFile(file, 'r') as zip_ref:\n",
    "            zip_ref.extractall(ALOS2_dir1/file.split('/')[-1][:-4])\n",
    "            os.remove(file)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a220da6d-00bb-4b9b-8f70-a034d03501de",
   "metadata": {},
   "source": [
    "##### (b) Unzip the *secondary* ALOS2 data in its own folder:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "2f9a0c38-8800-4203-aeb7-5f5766233c8e",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/scratch/taglialatela_cryo_test/sample_products/ALOS2/antarctica/ALOS2456335530-221104-ALOS2458405530-221118/ALOS2_sec/0000489662_001001_ALOS2458405530-221118.zip\n"
     ]
    }
   ],
   "source": [
    "zipfiles = [os.path.join(dirpath,f)\n",
    "                for dirpath,dirnames, files in os.walk(ALOS2_dir2)\n",
    "                for f in fnmatch.filter(files,'*.zip')]\n",
    "for file in zipfiles:\n",
    "    print(file)\n",
    "    # if os.path.isdir(ALOS2_dir2 / file[:-4]):\n",
    "        # ##check full unzipping\n",
    "        # if sum(os.path.getsize(ALOS2_dir2 / file[:-4] / f) for f in os.listdir(ALOS2_dir2 / file[:-4]) if os.path.isfile(ALOS2_dir1 / file[:-4] /f)) < 6509714637:\n",
    "        #     with zipfile.ZipFile(file, 'r') as zip_ref:\n",
    "        #         zip_ref.extractall(ALOS2_dir2/file.split('/')[-1][:-4])\n",
    "        #     # os.remove(file)\n",
    "    # else:\n",
    "    with zipfile.ZipFile(file, 'r') as zip_ref:\n",
    "        zip_ref.extractall(ALOS2_dir2/file.split('/')[-1][:-4])\n",
    "        os.remove(file)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "af65b7df-5d79-42a9-b287-4002d325c12b",
   "metadata": {},
   "source": [
    "##### (c) Get a list of folders for each ALOS2 file\n",
    "\n",
    "_If previously downloaded ALOS2 data and stored in NISAR ODS (and you skipped the steps above), this is where you can provide the path to the s3 bucket (untested) for ALOS2 unzipped data._"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "7199ad2b-319d-4162-820d-5cef2abfd3ec",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['/scratch/taglialatela_cryo_test/sample_products/ALOS2/antarctica/ALOS2456335530-221104-ALOS2458405530-221118/ALOS2_ref/0000487643_001001_ALOS2456335530-221104']\n",
      "['/scratch/taglialatela_cryo_test/sample_products/ALOS2/antarctica/ALOS2456335530-221104-ALOS2458405530-221118/ALOS2_sec/0000489662_001001_ALOS2458405530-221118']\n"
     ]
    }
   ],
   "source": [
    "ALOS2folder1 = [os.path.join(dirpath,f)\n",
    "                for dirpath,dirnames, files in os.walk(ALOS2_dir1)\n",
    "                for f in fnmatch.filter(dirnames,'*')]      # all directories\n",
    "                # for f in fnmatch.filter(dirnames,'*ALOS2*')]   # ALOS2 data directories only\n",
    "ALOS2folder2 = [os.path.join(dirpath,f)\n",
    "                for dirpath,dirnames, files in os.walk(ALOS2_dir2)\n",
    "                for f in fnmatch.filter(dirnames,'*')]         # all directories\n",
    "print(ALOS2folder1)\n",
    "print(ALOS2folder2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7d98b8a4-6741-46dc-9cd4-2d56167cbbd0",
   "metadata": {},
   "source": [
    "### Loop through each ALOS2 file and:\n",
    "##### 1. **Convert ALOS2 SLC to NISAR RSLC**\n",
    "\n",
    "For ALOS-2 data we can only start from SLC (L1.1) and repackage them in a NISAR RSLC HDF5 format. This is the only SLC format recognized by the NISAR workflow in ISCE3. The conversion script is available here https://github.com/isce-framework/isce3/blob/develop/share/nisar/examples/alos2_to_nisar_l1.py\n",
    "\n",
    "##### 2. **Get a DEM for the study area** using the `stage_dem.py` from isce3\n",
    "##### 3. **Process NISAR RSLC to GUNW** using `insar.py` from isce3 and a `runconfig_insar.yaml` (<-- To create custom L2 products or troubleshoot, edit paramters in this file.)\n",
    "\n",
    "--------------"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5a79da07-cdce-42c5-beb7-3c8e499f719a",
   "metadata": {},
   "source": [
    "**(1a) Process Ref frame first:**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "69479fc2-23a0-4148-a43d-085cb6ed7e49",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "0000487643_001001_ALOS2456335530-221104\n",
      "\n",
      "conda run -n isce3_src /home/jovyan/isce3/share/nisar/examples/alos2_to_nisar_l1.py -i /scratch/taglialatela_cryo_test/sample_products/ALOS2/antarctica/ALOS2456335530-221104-ALOS2458405530-221118/ALOS2_ref/0000487643_001001_ALOS2456335530-221104 -o /scratch/taglialatela_cryo_test/sample_products/ALOS2/antarctica/ALOS2456335530-221104-ALOS2458405530-221118/RSLC_ref/0000487643_001001_ALOS2456335530-221104.h5\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING: overwriting environment variables set in the machine\n",
      "overwriting variable {'LD_LIBRARY_PATH'}\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Convert ALOS2 to NISAR-like format:\n",
    "ALOS2_id1 = ALOS2folder1[0].split('/')[-1]\n",
    "print('')\n",
    "print(ALOS2_id1)  \n",
    "\n",
    "# Comment the next section out if you've already converted, and just need to define paramters for later steps in workflow\n",
    "# command = f\"conda run -n isce3_src /home/jovyan/isce3/share/nisar/examples/alos2_to_nisar_l1.py -i {ALOS2folder} -o {RSLC_dir/ALOS2_id}.h5\"\n",
    "command = f\"conda run -n isce3_src /home/jovyan/isce3/share/nisar/examples/alos2_to_nisar_l1.py -i {ALOS2folder1[0]} -o {RSLC_dir1/ALOS2_id1}.h5\"\n",
    "print('')\n",
    "print(command)    \n",
    "if os.path.isfile(f\"{RSLC_dir1/ALOS2_id1}.h5\")==False or os.path.getsize(f\"{RSLC_dir1/ALOS2_id1}.h5\")<6000000000:\n",
    "    try: os.remove(f\"{RSLC_dir1/ALOS2_id1}.h5\")\n",
    "    except:''\n",
    "    output = subprocess.check_output(command, shell=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f48a3bfe-9268-40c5-b31a-4676005a2b95",
   "metadata": {},
   "source": [
    "**(1b) Process secondary frame next** (no DEM step here since we only need one, and we used the ref data to make it):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "c1019cdd-c9ef-498e-8da0-813a431a1a9d",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "0000489662_001001_ALOS2458405530-221118\n",
      "\n",
      "conda run -n isce3_src /home/jovyan/isce3/share/nisar/examples/alos2_to_nisar_l1.py -i /scratch/taglialatela_cryo_test/sample_products/ALOS2/antarctica/ALOS2456335530-221104-ALOS2458405530-221118/ALOS2_sec/0000489662_001001_ALOS2458405530-221118 -o /scratch/taglialatela_cryo_test/sample_products/ALOS2/antarctica/ALOS2456335530-221104-ALOS2458405530-221118/RSLC_sec/0000489662_001001_ALOS2458405530-221118.h5\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING: overwriting environment variables set in the machine\n",
      "overwriting variable {'LD_LIBRARY_PATH'}\n",
      "\n"
     ]
    }
   ],
   "source": [
    "ALOS2_id2 = ALOS2folder2[0].split('/')[-1]\n",
    "print('')\n",
    "print(ALOS2_id2)  \n",
    "    \n",
    "# command = f\"conda run -n isce3_src /home/jovyan/isce3/share/nisar/examples/alos2_to_nisar_l1.py -i {ALOS2folder} -o {RSLC_dir/ALOS2_id}.h5\"\n",
    "command = f\"conda run -n isce3_src /home/jovyan/isce3/share/nisar/examples/alos2_to_nisar_l1.py -i {ALOS2folder2[0]} -o {RSLC_dir2/ALOS2_id2}.h5\"\n",
    "print('')\n",
    "print(command)    \n",
    "if os.path.isfile(f\"{RSLC_dir2/ALOS2_id2}.h5\")==False or os.path.getsize(f\"{RSLC_dir2/ALOS2_id2}.h5\")<6000000000:\n",
    "    try: os.remove(f\"{RSLC_dir2/ALOS2_id2}.h5\")\n",
    "    except:''\n",
    "    output = subprocess.check_output(command, shell=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "073f0952-0dad-4f4a-8c6c-945cff6eacdb",
   "metadata": {},
   "source": [
    "**(2) Get a DEM for the study area** using the `stage_dem.py` from isce3 and the reference RSLC we made above. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "382af8c1-4a54-4913-b5b5-fafabd774593",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "conda run -n isce3_src /home/jovyan/isce3/python/packages/nisar/workflows/stage_dem.py -p /scratch/taglialatela_cryo_test/sample_products/ALOS2/antarctica/ALOS2456335530-221104-ALOS2458405530-221118/RSLC_ref/0000487643_001001_ALOS2456335530-221104.h5 -o /scratch/taglialatela_cryo_test/sample_products/ALOS2/antarctica/ALOS2456335530-221104-ALOS2458405530-221118/DEM/0000487643_001001_ALOS2456335530-221104.vrt\n"
     ]
    }
   ],
   "source": [
    "## Get NISAR DEM\n",
    "command = f\"conda run -n isce3_src /home/jovyan/isce3/python/packages/nisar/workflows/stage_dem.py -p {RSLC_dir1/ALOS2_id1}.h5 -o {DEM_dir/ALOS2_id1}.vrt\"\n",
    "print('')\n",
    "print(command)    \n",
    "\n",
    "if os.path.isfile(f\"{DEM_dir/ALOS2_id1}.vrt\")==False:\n",
    "    output = subprocess.check_output(command, shell=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5733498e-77bb-4b6d-be59-cde144a86254",
   "metadata": {},
   "source": [
    "### Generate NISAR-like INSAR products**\n",
    "\n",
    "Set up the runconfig with right paths for pair to be processed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "deb984ea-05cb-47e2-98b1-9dee3c26a429",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/scratch/taglialatela_cryo_test/sample_products/ALOS2/antarctica/ALOS2456335530-221104-ALOS2458405530-221118/RSLC_ref\n",
      "0000487643_001001_ALOS2456335530-221104\n",
      "/scratch/taglialatela_cryo_test/sample_products/ALOS2/antarctica/ALOS2456335530-221104-ALOS2458405530-221118/DEM\n",
      "/scratch/taglialatela_cryo_test/sample_products/ALOS2/antarctica/ALOS2456335530-221104-ALOS2458405530-221118\n",
      "/scratch/taglialatela_cryo_test/sample_products/ALOS2/antarctica/ALOS2456335530-221104-ALOS2458405530-221118/INSAR/unwrap_80m/output_insar\n"
     ]
    }
   ],
   "source": [
    "print(RSLC_dir1)\n",
    "print(ALOS2_id1)\n",
    "print(DEM_dir)\n",
    "print(PAIR_dir)\n",
    "print(output_insar_dir)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "83c1be2c-4885-40b9-96de-3bed6c210a99",
   "metadata": {},
   "source": [
    "Provide path to the runconfig used by isce3 code"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "97005942-4a25-4178-b21b-66600de1e4de",
   "metadata": {},
   "outputs": [],
   "source": [
    "filein = open('/home/jovyan/cryo_sample_products/runconfig_insar_VB_template.yaml')\n",
    "template = Template(filein.read())\n",
    "replacements = {'inputrefrslc':  f\"{RSLC_dir1/ALOS2_id1}.h5\",\n",
    "                'inputsecrslc':  f\"{RSLC_dir2/ALOS2_id2}.h5\",\n",
    "                'demfile': f\"{DEM_dir/ALOS2_id1}.vrt\",\n",
    "                'prodpath': f\"{output_insar_dir}\",\n",
    "                'scratchpath': f\"{scratch_insar_dir}\",\n",
    "                'sasoutpath':  f\"{output_insar_dir/ALOS2_id1}_product.h5\",\n",
    "                'logpath': f\"{output_insar_dir}/insar.log\"\n",
    "                }\n",
    "makeoutput = template.substitute(replacements)\n",
    "file = open('%s/%s.yaml' %(output_insar_dir,ALOS2_id1),'w')\n",
    "file.write(makeoutput)\n",
    "file.close()\n",
    "filein.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aafded4c-555b-487f-82c5-53f26124aa98",
   "metadata": {},
   "source": [
    "##### NISAR RSLC --> NISAR L1 and L2 insar products"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "ceb619ba-f17d-4f23-810d-24a4d49a33ba",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "conda run -n isce3_src python /home/jovyan/isce3/python/packages/nisar/workflows/insar.py /scratch/taglialatela_cryo_test/sample_products/ALOS2/antarctica/ALOS2456335530-221104-ALOS2458405530-221118/INSAR/unwrap_80m/output_insar/0000487643_001001_ALOS2456335530-221104.yaml\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING: overwriting environment variables set in the machine\n",
      "overwriting variable {'LD_LIBRARY_PATH'}\n",
      "/home/jovyan/isce3/build/packages/nisar/workflows/offsets_product.py:86: UserWarning: swmr=True only affects read ('r') mode. For swmr write mode, set f.swmr_mode = True after opening the file.\n",
      "  with h5py.File(output_hdf5, 'a', libver='latest', swmr=True) as dst_h5:\n",
      "/home/jovyan/isce3/build/packages/nisar/workflows/rubbersheet.py:73: UserWarning: swmr=True only affects read ('r') mode. For swmr write mode, set f.swmr_mode = True after opening the file.\n",
      "  with h5py.File(output_hdf5, 'r+', libver='latest', swmr=True) as dst_h5:\n",
      "/home/jovyan/isce3/build/packages/nisar/workflows/unwrap.py:65: UserWarning: swmr=True only affects read ('r') mode. For swmr write mode, set f.swmr_mode = True after opening the file.\n",
      "  with h5py.File(output_hdf5, \"a\", libver=\"latest\", swmr=True) as dst_h5,\\\n",
      "/home/jovyan/isce3/build/packages/nisar/workflows/ionosphere.py:1055: UserWarning: swmr=True only affects read ('r') mode. For swmr write mode, set f.swmr_mode = True after opening the file.\n",
      "  with h5py.File(iono_output, 'a', libver='latest', swmr=True) as dst_h5:\n",
      "/home/jovyan/isce3/build/packages/nisar/workflows/geocode_insar.py:1078: UserWarning: swmr=True only affects read ('r') mode. For swmr write mode, set f.swmr_mode = True after opening the file.\n",
      "  with h5py.File(output_hdf5, \"a\", libver='latest', swmr=True) as dst_h5:\n",
      "/home/jovyan/.local/envs/isce3_src/lib/python3.11/site-packages/pysolid/grid.py:93: UserWarning: Input line 1 contained no data and will not be counted towards `max_rows=580`.  This differs from the behaviour in NumPy <=1.22 which counted lines rather than rows.  If desired, the previous behaviour can be achieved by using `itertools.islice`.\n",
      "Please see the 1.23 release notes for an example on how to do this.  If you wish to ignore this warning, use `warnings.filterwarnings`.  This warning is expected to be removed in the future and is given only once per `loadtxt` call.\n",
      "  fc = np.loadtxt(txt_file,\n",
      "/home/jovyan/isce3/build/packages/nisar/workflows/h5_prep.py:1191: UserWarning: swmr=True only affects read ('r') mode. For swmr write mode, set f.swmr_mode = True after opening the file.\n",
      "  with h5py.File(gunw_hdf5, 'a', libver='latest', swmr=True) as hdf:\n",
      "\n"
     ]
    }
   ],
   "source": [
    "command = f\"conda run -n isce3_src python /home/jovyan/isce3/python/packages/nisar/workflows/insar.py {output_insar_dir/ALOS2_id1}.yaml\"\n",
    "print('')\n",
    "print(command)\n",
    "# if os.path.isfile(f\"{output_insar_dir/ALOS2_id1}_product.h5\")==False:\n",
    "if os.path.isfile(f\"{output_insar_dir}_product.h5\")==False:  #this might be right syntax to get e.g. ROFF_product.h5 which might be standard naming\n",
    "    output = subprocess.check_output(command, shell=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a5a797e8-7d3b-41c3-b8f7-3fe9eb3075ca",
   "metadata": {},
   "source": [
    "See final list of NISAR data files"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "c7165d81-6957-4e14-b953-99943bda4232",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['/scratch/taglialatela_cryo_test/sample_products/ALOS2/antarctica/ALOS2456335530-221104-ALOS2458405530-221118/INSAR/unwrap_80m/output_insar/RUNW_0000487643_001001_ALOS2456335530-221104_product.h5',\n",
       " '/scratch/taglialatela_cryo_test/sample_products/ALOS2/antarctica/ALOS2456335530-221104-ALOS2458405530-221118/INSAR/unwrap_80m/output_insar/RIFG_0000487643_001001_ALOS2456335530-221104_product.h5',\n",
       " '/scratch/taglialatela_cryo_test/sample_products/ALOS2/antarctica/ALOS2456335530-221104-ALOS2458405530-221118/INSAR/unwrap_80m/output_insar/GOFF_0000487643_001001_ALOS2456335530-221104_product.h5',\n",
       " '/scratch/taglialatela_cryo_test/sample_products/ALOS2/antarctica/ALOS2456335530-221104-ALOS2458405530-221118/INSAR/unwrap_80m/output_insar/ROFF_0000487643_001001_ALOS2456335530-221104_product.h5',\n",
       " '/scratch/taglialatela_cryo_test/sample_products/ALOS2/antarctica/ALOS2456335530-221104-ALOS2458405530-221118/INSAR/unwrap_80m/output_insar/GUNW_0000487643_001001_ALOS2456335530-221104_product.h5']"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "NISAR_GUNWs = [os.path.join(dirpath,f)\n",
    "                for dirpath,dirnames, files in os.walk(output_insar_dir)\n",
    "                for f in fnmatch.filter(files,'*.h5')]\n",
    "NISAR_GUNWs"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "isce3_src",
   "language": "python",
   "name": "isce3_src"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}