templates/2_Pixie_Cluster_Pixels.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Pixie: pixel clustering notebook"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "import"
    ]
   },
   "outputs": [],
   "source": [
    "# import required packages\n",
    "import json\n",
    "import os\n",
    "import subprocess\n",
    "from datetime import datetime as dt\n",
    "\n",
    "import feather\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import xarray as xr\n",
    "from matplotlib import rc_file_defaults\n",
    "from alpineer import io_utils, load_utils\n",
    "\n",
    "from ark.analysis import visualize\n",
    "from ark.phenotyping import pixel_cluster_utils\n",
    "from ark.utils import data_utils, example_dataset, plot_utils\n",
    "from ark.utils.metacluster_remap_gui import (MetaClusterData, MetaClusterGui,\n",
    "                                             colormap_helper,\n",
    "                                             metaclusterdata_from_files)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 0: Set root directory and download the example dataset\n",
    "\n",
    "Here we are using the example data located in `/data/example_dataset/input_data`. To modify this notebook to run using your own data, simply change `base_dir` to point to your own sub-directory within the data folder.\n",
    "\n",
    "* `base_dir`: the path to all of your imaging data. Should contain a directory for your images, segmentations, and cell table (generated from `1_Segment_Image_Data.ipynb`). This directory will also store all of the directories/files created during pixel clustering."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "base_dir"
    ]
   },
   "outputs": [],
   "source": [
    "base_dir = \"../data/example_dataset\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you would like to test the features in Ark with an example dataset, run the cell below. It will download a dataset consisting of 11 FOVs with 22 channels. You may find more information about the example dataset in the [README](../README.md#example-dataset).\n",
    "\n",
    "If you are using your own data, skip the cell below.\n",
    "\n",
    "* `overwrite_existing`: If set to `False`, it will not overwrite existing data in the `data/example_dataset`. Recommended setting to `False` if you are running Notebooks 1,2,3 and 4 in succession. Set to `True` if you are just running Notebook 2."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "ex_data_download"
    ]
   },
   "outputs": [],
   "source": [
    "example_dataset.get_example_dataset(dataset=\"cluster_pixels\", save_dir = base_dir, overwrite_existing = False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## 1: set file paths and parameters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* `tiff_dir`: the path to the directory containing your imaging data\n",
    "* `img_sub_folder`: if `tiff_dir` contains an additional subfolder structure, override `None` with the appropriate value\n",
    "* `segmentation_dir`: the path to the directory containing your segmentations (generated from `1_Segment_Image_Data.ipynb`). Set this argument to `None` if you do not have segmentation labels or wish to run pixel clustering without them. However, note that you will not be able to run cell clustering as that process is heavily dependent on them.\n",
    "* `seg_suffix`: the suffix plus the file extension of the segmented images for each FOV. Note that these should be the same for all FOVs and that normally, the value should be `'_whole_cell.tiff'`. This argument will be ignored if `segmentation_dir` is set to `None`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "file_path"
    ]
   },
   "outputs": [],
   "source": [
    "tiff_dir = os.path.join(base_dir, \"image_data\")\n",
    "img_sub_folder = None\n",
    "segmentation_dir = os.path.join(\"segmentation\", \"deepcell_output\")\n",
    "seg_suffix = '_whole_cell.tiff'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* `fovs` (optional): set a specific set of fovs to load, default loads all the fovs in `tiff_dir`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "load_fovs"
    ]
   },
   "outputs": [],
   "source": [
    "# either get all fovs in the folder...\n",
    "fovs = io_utils.list_folders(tiff_dir)\n",
    "\n",
    "# ... or optionally, select a specific set of fovs manually\n",
    "# fovs = [\"fov14\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Define multiprocessing parameters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Turning on multiprocessing provides a speed boost, however it is not always cross-platform compatible. If you receive errors such as hanging cells without progress updates, try setting `multiprocess` back to `False`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "set_multi"
    ]
   },
   "outputs": [],
   "source": [
    "# set to True to turn on multiprocessing\n",
    "multiprocess = False\n",
    "\n",
    "# define the number of FOVs to process in parallel, ignored if multiprocessing is set to False\n",
    "batch_size = 5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## 2: Preprocess"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Set a prefix to be applied to all data directories/files created by pixel and cell clustering. If the prefix is not set, a default of the datetime at the start of the run is used."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "pixel_prefix"
    ]
   },
   "outputs": [],
   "source": [
    "# explicitly set pixel_cluster_prefix to override datetime default\n",
    "pixel_cluster_prefix = \"example\"\n",
    "\n",
    "if pixel_cluster_prefix is None:\n",
    "    pixel_cluster_prefix = dt.now().strftime('%Y-%m-%dT%H:%M:%S')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following data directories/files will be created for preprocessing with names prefixed by `pixel_cluster_prefix`:\n",
    "\n",
    "* `pixel_output_dir`: the name of the folder to store the pixel clustering directories/files\n",
    "* `preprocessed_dir`: the name of the directory to save the preprocessed pixel data\n",
    "* `subsetted_dir`: the name of the directory to save the subsetted pixel data\n",
    "* `norm_vals_name`: file name to store the values used to normalize each channel on the full preprocessed dataset when assigning pixel SOM cluster labels."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "dir_set"
    ]
   },
   "outputs": [],
   "source": [
    "# define the base output pixel folder using the specified pixel cluster prefix\n",
    "pixel_output_dir = os.path.join(\"pixie\", \"%s_pixel_output_dir\" % pixel_cluster_prefix)\n",
    "if not os.path.exists(os.path.join(base_dir, pixel_output_dir)):\n",
    "    os.makedirs(os.path.join(base_dir, pixel_output_dir))\n",
    "\n",
    "# define the preprocessed pixel data folders\n",
    "pixel_data_dir = os.path.join(pixel_output_dir, 'pixel_mat_data')\n",
    "pixel_subset_dir = os.path.join(pixel_output_dir, 'pixel_mat_subset')\n",
    "norm_vals_name = os.path.join(pixel_output_dir, 'channel_norm_post_rowsum.feather')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For certain channels, such as membraneous tumor markers, the subcellular localization of the marker isn't important. Instead, what matters is that cells which are positive for the marker show up as positive. In these cases, we have sometimes found it useful to add additional blurring to these markers before clustering. This ensures that more of the pixels within the cell are positive for the marker, instead of only a few pixels at the border, especially for cells which are under-segmented. However, higher blur levels will also cause more of the pixels in neighboring cells to show up as positive. Therefore, this works best when you have other, robust markers (like CD45) which you can use to determine which cells are false positives for the blurred channel. If you have markers in your panel which fit this description, you can add them in the cell below. Then, when specifying the list of markers to include for clustering, make sure to add `{marker_name}_smoothed`, as that is what the tiff will be called"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "smooth_channels"
    ]
   },
   "outputs": [],
   "source": [
    "# set an optional list of markers for additional blurring\n",
    "blurred_channels = [\"ECAD\"]\n",
    "smooth_vals = 6\n",
    "\n",
    "pixel_cluster_utils.smooth_channels(\n",
    "    fovs=fovs,\n",
    "    tiff_dir=tiff_dir,\n",
    "    img_sub_folder=img_sub_folder,\n",
    "    channels=blurred_channels,\n",
    "    smooth_vals=smooth_vals,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sometimes, markers will have background staining that you'll want to filter out for the clustering process. Define the name of the marker you want to filter using `filter_channel`. If the marker is only present in the nucleus, set `exclude = False` to filter it out from the membrane. Conversely, if the marker is only present in the membrane, set `exclude = True` to filter it out from the nucleus.\n",
    "\n",
    "When specifying the list of markers to include for clustering, make sure to add `{marker_name}_nuc_exclude` or `{marker_name}_nuc_include` depending on what type of signal was filtered out.\n",
    "\n",
    "Skip this cell if you don't want to run marker filtering."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "filter_channels"
    ]
   },
   "outputs": [],
   "source": [
    "filter_channel = 'CD163'\n",
    "nuclear_exclude = True\n",
    "\n",
    "pixel_cluster_utils.filter_with_nuclear_mask(\n",
    "    fovs=fovs,\n",
    "    tiff_dir=tiff_dir,\n",
    "    seg_dir=os.path.join(base_dir, segmentation_dir),\n",
    "    channel=filter_channel,\n",
    "    nuc_seg_suffix=\"_nuclear.tiff\",\n",
    "    img_sub_folder=img_sub_folder,\n",
    "    exclude=nuclear_exclude\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Set the following arguments\n",
    "\n",
    "* `channels`: set a subset to run pixel clustering over.\n",
    "* `blur_factor`: the sigma to use for the Gaussian filter when running the Gaussian blur. Higher values are more aggressive in smoothing signal.\n",
    "* `subset_proportion`: the fraction of pixels to take from each fov. Sampling is random."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "channel_set"
    ]
   },
   "outputs": [],
   "source": [
    "channels = [\"CD3\", \"CD4\", \"CD8\", \"CD14\", \"CD20\", \"CD31\", \"CD45\", \"CD68\",\n",
    "            \"CD163_nuc_exclude\", \"CK17\", \"Collagen1\", \"Fibronectin\", \n",
    "            \"ECAD_smoothed\", \"HLADR\", \"SMA\", \"Vim\"]\n",
    "blur_factor = 2\n",
    "subset_proportion = 0.1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "During pixel preprocessing, the following is done for each FOV:\n",
    "\n",
    "* Gaussian blur each channel separately\n",
    "* Remove empty pixels\n",
    "* For the remaining pixels, normalize each channel by the sum of all the channels, this creates the full preprocessed dataset\n",
    "* Subset a `subset_proportion` fraction of non-empty, normalized pixels, this creates the subsetted dataset\n",
    "\n",
    "Note: if you get integer overflow errors loading in your data, try changing the `dtype` argument to a larger type."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "gen_pixel_mat"
    ]
   },
   "outputs": [],
   "source": [
    "# run pixel data preprocessing\n",
    "pixel_cluster_utils.create_pixel_matrix(\n",
    "    fovs,\n",
    "    channels,\n",
    "    base_dir,\n",
    "    tiff_dir,\n",
    "    os.path.join(base_dir, segmentation_dir),\n",
    "    img_sub_folder=img_sub_folder,\n",
    "    seg_suffix=seg_suffix,\n",
    "    pixel_output_dir=pixel_output_dir,\n",
    "    data_dir=pixel_data_dir,\n",
    "    subset_dir=pixel_subset_dir,\n",
    "    norm_vals_name=norm_vals_name,\n",
    "    blur_factor=blur_factor,\n",
    "    subset_proportion=subset_proportion,\n",
    "    multiprocess=multiprocess,\n",
    "    batch_size=batch_size\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3: pixel clustering"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.1: train pixel SOM"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following data directories/files will be created for pixel clustering:\n",
    "\n",
    "* `pixel_som_weights_name`: file name to place the pixel SOM weights\n",
    "* `pc_chan_avg_som_cluster_name`: file name to store the average channel expression across all pixel SOM clusters\n",
    "* `pc_chan_avg_meta_cluster_name`: same as above except for pixel meta clusters\n",
    "* `pixel_meta_cluster_remap_name`: for the meta cluster remapping process, the file to store the new SOM to meta mappings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "pixel_som_path_set"
    ]
   },
   "outputs": [],
   "source": [
    "pixel_som_weights_name = os.path.join(pixel_output_dir, 'pixel_som_weights.feather')\n",
    "pc_chan_avg_som_cluster_name = os.path.join(pixel_output_dir, 'pixel_channel_avg_som_cluster.csv')\n",
    "pc_chan_avg_meta_cluster_name = os.path.join(pixel_output_dir, 'pixel_channel_avg_meta_cluster.csv')\n",
    "pixel_meta_cluster_remap_name = os.path.join(pixel_output_dir, 'pixel_meta_cluster_mapping.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Train the pixel SOM on the subsetted data. Training is done using the `FlowSOM` algorithm.\n",
    "\n",
    "Note that each channel is normalized by their 99.9% value across the entire subsetted dataset before training. These values get saved to `norm_vals_name`.\n",
    "\n",
    "For a full set of parameters you can customize for `train_pixel_som`, please consult: <a href=https://ark-analysis.readthedocs.io/en/latest/_markdown/ark.phenotyping.html#ark.phenotyping.pixel_cluster_utils.train_pixel_som>pixel training docs</a>."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "train_pixel_som"
    ]
   },
   "outputs": [],
   "source": [
    "# create the pixel-level SOM weights\n",
    "pixel_pysom = pixel_cluster_utils.train_pixel_som(\n",
    "    fovs,\n",
    "    channels,\n",
    "    base_dir,\n",
    "    subset_dir=pixel_subset_dir,\n",
    "    norm_vals_name=norm_vals_name,\n",
    "    som_weights_name=pixel_som_weights_name,\n",
    "    num_passes=1,\n",
    "    seed=42\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.2: Assign pixel SOM clusters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use the SOM weights learned from `train_pixel_som` to assign pixel clusters to the full preprocessed dataset.\n",
    "\n",
    "Note that each channel is normalized by the respective value stored in `norm_vals_name` (computed in `train_pixel_som`) prior to cluster assignment.\n",
    "\n",
    "`generate_som_avg_files` will then compute the average channel expression across all pixel SOM clusters as well as the number of pixels in each pixel SOM cluster (the data placed in `pc_chan_avg_som_cluster_name`). This is needed for pixel consensus clustering."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "cluster_pixel_mat"
    ]
   },
   "outputs": [],
   "source": [
    "# use pixel SOM weights to assign pixel clusters\n",
    "pixel_cluster_utils.cluster_pixels(\n",
    "    fovs,\n",
    "    channels,\n",
    "    base_dir,\n",
    "    pixel_pysom,\n",
    "    data_dir=pixel_data_dir,\n",
    "    multiprocess=multiprocess,\n",
    "    batch_size=batch_size\n",
    ")\n",
    "\n",
    "# generate the SOM cluster summary files\n",
    "pixel_cluster_utils.generate_som_avg_files(\n",
    "    fovs,\n",
    "    channels,\n",
    "    base_dir,\n",
    "    pixel_pysom,\n",
    "    data_dir=pixel_data_dir,\n",
    "    pc_chan_avg_som_cluster_name=pc_chan_avg_som_cluster_name\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.3: run pixel consensus clustering"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With the SOM cluster labels assigned to the full preprocessed pixel data, assign consensus cluster labels. The consensus clusters are trained on the average channel expression across all pixel SOM clusters (the data stored in `pc_chan_avg_som_cluster_name`). These values are z-scored and capped at the value specified in the `cap` argument prior to training: this helps improve the meta clustering process.\n",
    "\n",
    "After consensus clustering, the following are also computed by `generate_meta_avg_files`:\n",
    "\n",
    "* The average channel expression across all pixel meta clusters, and the number of pixels per meta cluster (the data placed in `pc_chan_avg_meta_cluster_name`)\n",
    "* The meta cluster mapping for each pixel SOM cluster in `pc_chan_avg_som_cluster_name` (data is resaved, same data except with an associated meta cluster column)\n",
    "\n",
    "For a full set of parameters you can customize for `pixel_consensus_cluster`, please consult: <a href=https://ark-analysis.readthedocs.io/en/latest/_markdown/ark.phenotyping.html#ark.phenotyping.pixel_cluster_utils.pixel_consensus_cluster>pixel consensus clustering docs</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* `max_k`: the number of consensus clusters desired\n",
    "* `cap`: used to clip z-scored values prior to consensus clustering (in the range `[-cap, cap]`)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "pixel_consensus_cluster"
    ]
   },
   "outputs": [],
   "source": [
    "max_k = 20\n",
    "cap = 3\n",
    "\n",
    "# run hierarchical clustering based on pixel SOM cluster assignments\n",
    "pixel_cc = pixel_cluster_utils.pixel_consensus_cluster(\n",
    "    fovs,\n",
    "    channels,\n",
    "    base_dir,\n",
    "    max_k=max_k,\n",
    "    cap=cap,\n",
    "    data_dir=pixel_data_dir,\n",
    "    pc_chan_avg_som_cluster_name=pc_chan_avg_som_cluster_name,\n",
    "    multiprocess=multiprocess,\n",
    "    batch_size=batch_size\n",
    ")\n",
    "\n",
    "# generate the meta cluster summary files\n",
    "pixel_cluster_utils.generate_meta_avg_files(\n",
    "    fovs,\n",
    "    channels,\n",
    "    base_dir,\n",
    "    pixel_cc,\n",
    "    data_dir=pixel_data_dir,\n",
    "    pc_chan_avg_som_cluster_name=pc_chan_avg_som_cluster_name,\n",
    "    pc_chan_avg_meta_cluster_name=pc_chan_avg_meta_cluster_name\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4: visualize results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.1: use the interactive reclustering results to relabel pixel meta clusters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The visualization shows the z-scored average channel expression per pixel SOM and meta cluster. The heatmaps are faceted by pixel SOM clusters on the left and pixel meta clusters on the right.\n",
    "\n",
    "## Usage\n",
    "\n",
    "### Quickstart\n",
    "- **Select**: Left Click\n",
    "- **Remap**: **New metacluster** button or Right Click\n",
    "- **Edit Metacluster Name**: Textbox at bottom right of the heatmaps.\n",
    "\n",
    "### Selection and Remapping details\n",
    "- To select a SOM cluster, click on its respective position in the **selected** bar. Click on it again to deselect.\n",
    "- To select a meta cluster, click on its corresponding color in the **metacluster** bar. Click on it again to deselect.\n",
    "- To remap the selected clusters, click the **New metacluster** button (alternatively, right click anywhere). Note that remapping an entire metacluster deletes it.\n",
    "- To clear the selected SOM/meta clusters, use the **Clear Selection** button.\n",
    "- **After remapping a meta cluster, make sure to deselect the newly created one to prevent unwanted combinations.**\n",
    "\n",
    "### Other features and notes\n",
    "- You will likely need to zoom out to see the entire visualization. To toggle Zoom, use Ctrl -/Ctrl + on Windows or ⌘ +/⌘ - on Mac.\n",
    "- The bars at the top show the number of pixels in each SOM cluster.\n",
    "- The text box at the bottom right allows you to rename a particular meta cluster. This can be useful as remapping may cause inconsistent numbering.\n",
    "- Adjust the z-score limit using the slider on the bottom left to adjust your dynamic range.\n",
    "- When meta clusters are combined or a meta cluster is renamed, the change is immediately saved to `pixel_meta_cluster_remap_name`.\n",
    "- You won't be able to advance in the notebook until you've clicked `New metacluster` or renamed a meta cluster at least once. If you don't want to make changes, just click `New metacluster` to trigger a save before continuing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "pixel_interactive"
    ]
   },
   "outputs": [],
   "source": [
    "%matplotlib widget\n",
    "rc_file_defaults()\n",
    "plt.ion()\n",
    "\n",
    "pixel_mcd = metaclusterdata_from_files(\n",
    "    os.path.join(base_dir, pc_chan_avg_som_cluster_name),\n",
    "    cluster_type='pixel'\n",
    ")\n",
    "pixel_mcd.output_mapping_filename = os.path.join(base_dir, pixel_meta_cluster_remap_name)\n",
    "pixel_mcg = MetaClusterGui(pixel_mcd, width=17)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Relabel the pixel SOM and meta clusters using the mapping, and recompute the meta cluster average files with the new meta cluster names."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "pixel_apply_remap"
    ]
   },
   "outputs": [],
   "source": [
    "# rename the meta cluster values in the pixel dataset\n",
    "pixel_cluster_utils.apply_pixel_meta_cluster_remapping(\n",
    "    fovs,\n",
    "    channels,\n",
    "    base_dir,\n",
    "    pixel_data_dir,\n",
    "    pixel_meta_cluster_remap_name,\n",
    "    multiprocess=multiprocess,\n",
    "    batch_size=batch_size\n",
    ")\n",
    "\n",
    "# recompute the mean channel expression per meta cluster and apply these new names to the SOM cluster average data\n",
    "pixel_cluster_utils.generate_remap_avg_files(\n",
    "    fovs,\n",
    "    channels,\n",
    "    base_dir,\n",
    "    pixel_data_dir,\n",
    "    pixel_meta_cluster_remap_name,\n",
    "    pc_chan_avg_som_cluster_name,\n",
    "    pc_chan_avg_meta_cluster_name\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Generate the color scheme returned by the interactive reclustering process. This will be for visualizing the pixel cluster overlay."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "pixel_cmap_gen"
    ]
   },
   "outputs": [],
   "source": [
    "raw_cmap, _ = colormap_helper.generate_meta_cluster_colormap_dict(\n",
    "    pixel_mcd.output_mapping_filename,\n",
    "    pixel_mcg.im_cl.cmap\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.2: pixel cluster overlay (pixel meta clusters only)\n",
    "\n",
    "Select a small subset of your FOVs to view within this notebook. Or if you wish to generate and save a significant amount of FOVs, the masks will be created and saved in batches.\n",
    "\n",
    "Files will be written as `{fov_name}_pixel_mask.tiff` in `pixel_output_dir`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "pixel_overlay_fovs"
    ]
   },
   "outputs": [],
   "source": [
    "# select fovs to display\n",
    "subset_pixel_fovs = ['fov0', 'fov1']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "pixel_mask_gen_save"
    ]
   },
   "outputs": [],
   "source": [
    "# define the path to the channel file\n",
    "if img_sub_folder is None:\n",
    "    chan_file = os.path.join(\n",
    "        io_utils.list_files(os.path.join(tiff_dir, fovs[0]), substrs=['.tiff'])[0]\n",
    "    )\n",
    "else:\n",
    "    chan_file = os.path.join(\n",
    "        img_sub_folder, io_utils.list_files(os.path.join(tiff_dir, fovs[0], img_sub_folder), substrs=['.tiff'])[0]\n",
    "    )\n",
    "\n",
    "# generate and save the pixel cluster masks for each fov in subset_pixel_fovs\n",
    "data_utils.generate_and_save_pixel_cluster_masks(\n",
    "    fovs=subset_pixel_fovs,\n",
    "    base_dir=base_dir,\n",
    "    save_dir=os.path.join(base_dir, pixel_output_dir),\n",
    "    tiff_dir=tiff_dir,\n",
    "    chan_file=chan_file,\n",
    "    pixel_data_dir=pixel_data_dir,\n",
    "    pixel_cluster_col='pixel_meta_cluster',\n",
    "    sub_dir='pixel_masks',\n",
    "    name_suffix='_pixel_mask',\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load a subset of the pixel cluster masks that you would like to preview."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "pixel_overlay_gen"
    ]
   },
   "outputs": [],
   "source": [
    "for pixel_fov in subset_pixel_fovs:\n",
    "    pixel_cluster_mask = load_utils.load_imgs_from_dir(\n",
    "        data_dir=os.path.join(base_dir, pixel_output_dir, \"pixel_masks\"),\n",
    "        files=[pixel_fov + \"_pixel_mask.tiff\"],\n",
    "        trim_suffix=\"_pixel_mask\",\n",
    "        match_substring=\"_pixel_mask\",\n",
    "        xr_dim_name=\"pixel_mask\",\n",
    "        xr_channel_names=None,\n",
    "    )\n",
    "\n",
    "    plot_utils.plot_pixel_cell_cluster_overlay(\n",
    "        pixel_cluster_mask,\n",
    "        [pixel_fov],\n",
    "        os.path.join(base_dir, pixel_meta_cluster_remap_name),\n",
    "        metacluster_colors=raw_cmap\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5: Save parameters for use in cell clustering"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following parameters are saved:\n",
    "\n",
    "* `fovs`: the subset of fovs in `pixel_data_dir` (may not be the same as the original fov list specified due to potential removal of corrupted fovs)\n",
    "* `channels`: the subset of channels\n",
    "* `segmentation_dir`: the path to the directory containing your segmentated images per FOV (generated from `1_Segment_Image_Data.ipynb`)\n",
    "* `seg_suffix`: the suffix plus the file extension of the segmented images for each FOV\n",
    "* `pixel_data_dir`: the name of the directory containing tne full pixel data with the pixel SOM and consensus cluster assignments\n",
    "* `pc_chan_avg_som_cluster_name`: the name of the file containing the average channel expression per pixel SOM cluster\n",
    "* `pc_chan_avg_meta_cluster_name`: same as above except for pixel meta clusters\n",
    "\n",
    "The save file will be `{pixel_cluster_prefix}_cell_clustering_params.json` and will be placed in `pixel_output_dir`. Note that the `pixel_output_dir` you use in `2_Pixie_Cluster_Pixels.ipynb` should be the same as in `3_Pixie_Cluster_Cells.ipynb`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "cell_param_save"
    ]
   },
   "outputs": [],
   "source": [
    "# define the params dict\n",
    "cell_clustering_params = {\n",
    "    'fovs': io_utils.remove_file_extensions(io_utils.list_files(os.path.join(base_dir, pixel_data_dir), substrs='.feather')),\n",
    "    'channels': channels,\n",
    "    'segmentation_dir': segmentation_dir,\n",
    "    'seg_suffix': seg_suffix,\n",
    "    'pixel_data_dir': pixel_data_dir,\n",
    "    'pc_chan_avg_som_cluster_name': pc_chan_avg_som_cluster_name,\n",
    "    'pc_chan_avg_meta_cluster_name': pc_chan_avg_meta_cluster_name\n",
    "}\n",
    "\n",
    "# save the params dict\n",
    "with open(os.path.join(base_dir, pixel_output_dir, 'cell_clustering_params.json'), 'w') as fh:\n",
    "    json.dump(cell_clustering_params, fh)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 5.1 Save Images a Mantis Project\n",
    "\n",
    "Mantis Viewer is a visualization tool for multi-dimensional imaging in pathology. Learn more about Mantis Viewer in the [README](../README.md#mantis-viewer)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "pixel_mantis_project"
    ]
   },
   "outputs": [],
   "source": [
    "plot_utils.create_mantis_dir(\n",
    "    fovs=subset_pixel_fovs,\n",
    "    mantis_project_path=os.path.join(base_dir, pixel_output_dir, \"mantis\"),\n",
    "    img_data_path=tiff_dir,\n",
    "    mask_output_dir=os.path.join(base_dir, pixel_output_dir, \"pixel_masks\"),\n",
    "    mapping = os.path.join(base_dir, pixel_meta_cluster_remap_name),\n",
    "    seg_dir=os.path.join(base_dir, segmentation_dir),\n",
    "    mask_suffix=\"_pixel_mask\",\n",
    "    seg_suffix_name=seg_suffix\n",
    ")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.15"
  },
  "toc-autonumbering": false,
  "toc-showcode": true,
  "toc-showmarkdowntxt": true,
  "vscode": {
   "interpreter": {
    "hash": "b0ac63cc785e7c4cb24da0642dbfe09cfb2d314979108d7192228f9c45902a86"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}