templates/3_Pixie_Cluster_Cells.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "# Pixie: cell clustering notebook"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "NOTE: this notebook should be run after `2_Pixie_Cluster_Pixels.ipynb`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "tags": [
     "import"
    ]
   },
   "outputs": [],
   "source": [
    "# import required packages\n",
    "import json\n",
    "import os\n",
    "import subprocess\n",
    "from datetime import datetime as dt\n",
    "\n",
    "import feather\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import scipy.stats as stats\n",
    "import seaborn as sns\n",
    "import xarray as xr\n",
    "from matplotlib import rc_file_defaults\n",
    "from alpineer import io_utils, load_utils\n",
    "\n",
    "from ark.analysis import visualize\n",
    "from ark.phenotyping import cell_cluster_utils\n",
    "from ark.utils import data_utils, example_dataset, plot_utils\n",
    "from ark.utils.metacluster_remap_gui import (MetaClusterData, MetaClusterGui,\n",
    "                                             colormap_helper,\n",
    "                                             metaclusterdata_from_files)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## 0: Set root directory and download the example dataset\n",
    "\n",
    "Here we are using the example data located in `/data/example_dataset/input_data`. To modify this notebook to run using your own data, simply change `base_dir` to point to your own sub-directory within the data folder, rather than `'example_dataset'`. \n",
    "\n",
    "* `base_dir`: the path to all of your imaging data. Should contain a directory for your images, segmentations, and cell table (generated from `1_Segment_Image_Data.ipynb`). This directory will also store all of the directories/files created during pixel clustering."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "base_dir"
    ]
   },
   "outputs": [],
   "source": [
    "# define the home directory containing pixel_output_dir\n",
    "base_dir = \"../data/example_dataset/\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you would like to test the features in Ark with an example dataset, run the cell below. It will download a dataset consisting of 11 FOVs with 22 channels. You may find more information about the example dataset in the [README](../README.md#example-dataset).\n",
    "\n",
    "If you are using your own data, skip the cell below.\n",
    "\n",
    "* `overwrite_existing`: If set to `False`, it will not overwrite existing data in the `data/example_dataset`. Recommended setting to `False` if you are running Notebooks 1,2,3 and 4 in succession. Set to `True` if you are just running Notebook 3."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "ex_data_download"
    ]
   },
   "outputs": [],
   "source": [
    "example_dataset.get_example_dataset(dataset=\"cluster_cells\", save_dir = base_dir, overwrite_existing = False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## 1: Load parameters for cell clustering (computed by `2_Pixie_Cluster_Pixels.ipynb`)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`cell_clustering_params_name` should be in the form `{pixel_cluster_prefix}_cell_clustering_params.json` contained in `{pixel_cluster_prefix}_pixel_output_dir`. Make sure to set `base_dir` and `pixel_output_dir` to the same value used in `2_Pixie_Cluster_Pixels.ipynb`.\n",
    "\n",
    "NOTE: `{pixel_cluster_prefix}` is set in `2_Pixie_Cluster_Pixels.ipynb`. If you did not explicity set a `{pixel_cluster_prefix}` in `2_Pixie_Cluster_Pixels.ipynb`, the prefix defaults to the timestamp of the run. Please check the run directory (`base_dir` as set in `2_Pixie_Cluster_Pixels.ipynb`) to see the prefix that was used."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "tags": [
     "dir_set"
    ]
   },
   "outputs": [],
   "source": [
    "# define the name of the folder containing the pixel cluster data\n",
    "pixel_output_dir = 'example_pixel_output_dir'\n",
    "\n",
    "# define the name of the cell clustering params file\n",
    "cell_clustering_params_name = 'cell_clustering_params.json'\n",
    "\n",
    "# define the name of the directory with the extracted image data\n",
    "tiff_dir = os.path.join(base_dir, \"image_data\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following params are loaded:\n",
    "\n",
    "* `fovs`: the subset of fovs used for pixel clustering\n",
    "* `channels`: the subset of channels used for pixel clustering\n",
    "* `segmentation_dir`: the path to the directory containing your segmented images (generated from `1_Segment_Image_Data.ipynb`)\n",
    "* `seg_suffix`: the suffix plus the file extension of the segmented images for each FOV\n",
    "* `pixel_data_dir`: the name of the directory containing pixel data with the pixel SOM and consensus cluster assignments\n",
    "* `pc_chan_avg_som_cluster_name`: the name of the file containing the average channel expression per pixel SOM cluster, used for the visualization of weighted channel average per cell\n",
    "* `pc_chan_avg_meta_cluster_name`: the name of the file containing the average channel expression per pixel meta cluster, used for the visualization of weighted channel average per cell\n",
    "\n",
    "Additionally, define the following param:\n",
    "\n",
    "* `cell_table_path`: path to the cell table created by `1_Segment_Image_Data.ipynb`, should be placed in `segmentation_dir` by default. You can use either the normalized or arcsinh versions (the only columns from the cell table used are `fov`, `label`, and `cell_size`, so the clustering pipeline supports both)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "tags": [
     "param_load"
    ]
   },
   "outputs": [],
   "source": [
    "# load the params\n",
    "with open(os.path.join(base_dir, \"pixie\", pixel_output_dir, cell_clustering_params_name)) as fh:\n",
    "    cell_clustering_params = json.load(fh)\n",
    "    \n",
    "# assign the params to variables\n",
    "fovs = cell_clustering_params['fovs']\n",
    "channels = cell_clustering_params['channels']\n",
    "segmentation_dir = cell_clustering_params['segmentation_dir']\n",
    "seg_suffix = cell_clustering_params['seg_suffix']\n",
    "pixel_data_dir = cell_clustering_params['pixel_data_dir']\n",
    "pc_chan_avg_som_cluster_name = cell_clustering_params['pc_chan_avg_som_cluster_name']\n",
    "pc_chan_avg_meta_cluster_name = cell_clustering_params['pc_chan_avg_meta_cluster_name']\n",
    "\n",
    "# define the cell table path\n",
    "cell_table_path = os.path.join(base_dir, 'segmentation', 'cell_table', 'cell_table_size_normalized.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2: Preprocess"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Set a prefix to be applied to all data directories/files created by pixel and cell clustering. If the prefix is not set, a default of the datetime at the start of the run is used."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "tags": [
     "cluster_prefix"
    ]
   },
   "outputs": [],
   "source": [
    "# explicitly set cell_cluster_prefix to override datetime default\n",
    "cell_cluster_prefix = \"example\"\n",
    "\n",
    "if cell_cluster_prefix is None:\n",
    "    cell_cluster_prefix = dt.now().strftime('%Y-%m-%dT%H:%M:%S')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following folders/files will be created with names prefixed by `cell_cluster_prefix`:\n",
    "\n",
    "* `cell_output_dir`: the name of the folder to store the cell clustering directories/files\n",
    "* `cell_som_weights_name`: file name to place the cell SOM weights\n",
    "* `cluster_counts_name`: file name to store the counts of each pixel SOM/meta clusters per cell\n",
    "* `cluster_counts_size_norm_name`: same as above, except with each value normalized by the respective cell's size. The data will also contain the SOM and meta cluster labels assigned.\n",
    "* `weighted_cell_channel_name`: file name to store the weighted cell channel expression for each cell. Refer to <a href=https://ark-analysis.readthedocs.io/en/latest/_markdown/ark.phenotyping.html#ark.phenotyping.cell_cluster_utils.compute_p2c_weighted_channel_avg>cell channel weighting docs</a> for how the weighting is computed.\n",
    "* `cell_som_cluster_count_avg_name`: file name to store the average number of pixel SOM/meta clusters per cell SOM cluster\n",
    "* `cell_meta_cluster_count_avg_name`: same as above except for cell meta clusters\n",
    "* `cell_som_cluster_channel_avg_name`: file name to store the average weighted channel expression per cell SOM cluster\n",
    "* `cell_meta_cluster_channel_avg_name`: same as above except for cell meta clusters\n",
    "* `cell_meta_cluster_remap_name`: for the meta cluster remapping process, the file to store the new SOM to meta mappings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "cell_cluster_files"
    ]
   },
   "outputs": [],
   "source": [
    "# define the base output cell folder\n",
    "cell_output_dir = '%s_cell_output_dir' % cell_cluster_prefix\n",
    "if not os.path.exists(os.path.join(base_dir, \"pixie\", cell_output_dir)):\n",
    "    os.mkdir(os.path.join(base_dir, \"pixie\", cell_output_dir))\n",
    "    \n",
    "# define the paths to cell clustering files, explicitly set the variables to use custom names\n",
    "cell_som_weights_name = os.path.join(\"pixie\", cell_output_dir, 'cell_som_weights.feather')\n",
    "cluster_counts_name = os.path.join(\"pixie\", cell_output_dir, 'cluster_counts.feather')\n",
    "cluster_counts_size_norm_name = os.path.join(\"pixie\", cell_output_dir, 'cluster_counts_size_norm.feather')\n",
    "weighted_cell_channel_name = os.path.join(\"pixie\", cell_output_dir, 'weighted_cell_channel.feather')\n",
    "cell_som_cluster_count_avg_name = os.path.join(\"pixie\", cell_output_dir, 'cell_som_cluster_count_avg.csv')\n",
    "cell_meta_cluster_count_avg_name = os.path.join(\"pixie\", cell_output_dir, 'cell_meta_cluster_count_avg.csv')\n",
    "cell_som_cluster_channel_avg_name = os.path.join(\"pixie\", cell_output_dir, 'cell_som_cluster_channel_avg.csv')\n",
    "cell_meta_cluster_channel_avg_name = os.path.join(\"pixie\", cell_output_dir, 'cell_meta_cluster_channel_avg.csv')\n",
    "cell_meta_cluster_remap_name = os.path.join(\"pixie\", cell_output_dir, 'cell_meta_cluster_mapping.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Additionally, define the following pixel cluster aggregation variable:\n",
    "\n",
    "* `pixel_cluster_col`: whether to use pixel SOM or meta cluster counts for training/aggregating. Set to `'pixel_som_cluster'` or `'pixel_meta_cluster_rename'` respectively. Note that if you did not explicitly rename your pixel meta clusters in `2_Pixie_Cluster_Pixels.ipynb`, the default numeric names will be used."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "tags": [
     "pixel_cluster_col"
    ]
   },
   "outputs": [],
   "source": [
    "# define the type of pixel cluster to aggregate on\n",
    "pixel_cluster_col = 'pixel_meta_cluster_rename'\n",
    "\n",
    "# depending on which pixel_cluster_col is selected, choose the pixel channel average table accordingly\n",
    "if pixel_cluster_col == 'pixel_som_cluster':\n",
    "    pc_chan_avg_name = pc_chan_avg_som_cluster_name\n",
    "elif pixel_cluster_col == 'pixel_meta_cluster_rename':\n",
    "    pc_chan_avg_name = pc_chan_avg_meta_cluster_name"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Generate the input data for the SOM. This will be the counts of each pixel cluster column per cell, normalized by the respective cell size."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "generate_som_input_data"
    ]
   },
   "outputs": [],
   "source": [
    "# generate the preprocessed data before \n",
    "cluster_counts, cluster_counts_size_norm = cell_cluster_utils.create_c2pc_data(\n",
    "    fovs, os.path.join(base_dir, pixel_data_dir), cell_table_path, pixel_cluster_col\n",
    ")\n",
    "\n",
    "# define the count columns found in cluster_counts_norm\n",
    "cell_som_cluster_cols = cluster_counts_size_norm.filter(\n",
    "    regex=f'{pixel_cluster_col}.*'\n",
    ").columns.values\n",
    "\n",
    "# write the unnormalized input data to cluster_counts_name for reference\n",
    "feather.write_dataframe(\n",
    "    cluster_counts,\n",
    "    os.path.join(base_dir, cluster_counts_name),\n",
    "    compression='uncompressed'\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Generate the weighted cell channel expression file. This data will be needed to compute the weighted average channel expression per cell cluster (the data stored in `cell_som_cluster_channel_avg_name` and `cell_meta_cluster_channel_avg_name`). See documentation of `compute_p2c_weighted_channel_avg` for how weighted cell channel average is computed: <a href=https://ark-analysis.readthedocs.io/en/latest/_markdown/ark.phenotyping.html#ark.phenotyping.cell_cluster_utils.compute_p2c_weighted_channel_avg>cell channel weighting docs</a>."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "generate_weighted_channel_data"
    ]
   },
   "outputs": [],
   "source": [
    "# depending on which pixel_cluster_col is selected, choose the pixel channel average table accordingly\n",
    "if pixel_cluster_col == 'pixel_som_cluster':\n",
    "    pc_chan_avg_name = pc_chan_avg_som_cluster_name\n",
    "elif pixel_cluster_col == 'pixel_meta_cluster_rename':\n",
    "    pc_chan_avg_name = pc_chan_avg_meta_cluster_name\n",
    "\n",
    "# generate the weighted cell channel expression data\n",
    "pixel_channel_avg = pd.read_csv(os.path.join(base_dir, pc_chan_avg_name))\n",
    "weighted_cell_channel = cell_cluster_utils.compute_p2c_weighted_channel_avg(\n",
    "    pixel_channel_avg,\n",
    "    channels,\n",
    "    cluster_counts,\n",
    "    fovs=fovs,\n",
    "    pixel_cluster_col=pixel_cluster_col\n",
    ")\n",
    "\n",
    "# write the data to weighted_cell_channel_name\n",
    "feather.write_dataframe(\n",
    "    weighted_cell_channel,\n",
    "    os.path.join(base_dir, weighted_cell_channel_name),\n",
    "    compression='uncompressed'\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3: Cell clustering"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.1: train cell SOM"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Train the cell SOM on the normalized number of pixel SOM/meta clusters per cell (the data stored in `cluster_counts_size_norm_name`).  Training is done using the `FlowSOM` algorithm. Note that each of the pixel SOM/meta cluster columns are normalized by their 99.9% value prior to training.\n",
    "\n",
    "For a full set of parameters you can customize for `train_cell_som`, please consult: <a href=https://ark-analysis.readthedocs.io/en/latest/_markdown/ark.phenotyping.html#ark.phenotyping.cell_cluster_utils.train_cell_som>cell training docs</a>."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "train_cell_som"
    ]
   },
   "outputs": [],
   "source": [
    "# create the cell-level SOM weights\n",
    "cell_pysom = cell_cluster_utils.train_cell_som(\n",
    "    fovs,\n",
    "    base_dir,\n",
    "    cell_table_path=cell_table_path,\n",
    "    cell_som_cluster_cols=cell_som_cluster_cols,\n",
    "    cell_som_input_data=cluster_counts_size_norm,\n",
    "    som_weights_name=cell_som_weights_name,\n",
    "    num_passes=1,\n",
    "    seed=42\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.2: assign cell SOM clusters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use the weights learned from `train_cell_som` to assign cell clusters to the cell dataset. Note that this is done on the normalized pixel SOM/meta cluster counts table. As with `train_pixel_som`, each of the columns are normalized by their 99.9% value prior to assigning a cell SOM cluster label.\n",
    "\n",
    "`generate_som_avg_files` will then compute the average number of pixel SOM/meta clusters per cell SOM cluster as well as the number of cells in each cell SOM cluster (the data placed in `cell_som_cluster_count_avg_name`). This is needed for cell consensus clustering."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "cluster_cell_data"
    ]
   },
   "outputs": [],
   "source": [
    "# use cell SOM weights to assign cell clusters\n",
    "cluster_counts_size_norm = cell_cluster_utils.cluster_cells(\n",
    "    base_dir,\n",
    "    cell_pysom,\n",
    "    cell_som_cluster_cols=cell_som_cluster_cols\n",
    ")\n",
    "\n",
    "# generate the SOM cluster summary files\n",
    "cell_cluster_utils.generate_som_avg_files(\n",
    "    base_dir,\n",
    "    cluster_counts_size_norm,\n",
    "    cell_som_cluster_cols=cell_som_cluster_cols,\n",
    "    cell_som_expr_col_avg_name=cell_som_cluster_count_avg_name\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.3: run cell consensus clustering"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With the SOM cluster labels assigned to the cell data, assign consensus cluster labels. The consensus clusters are trained on the average number of pixel SOM/meta clusters across all cell SOM clusters (the data stored in `cell_som_cluster_count_avg_name`). These values are z-scored and capped at the value specified in the `cap` argument prior to training: this helps improve the meta clustering process.\n",
    "\n",
    "After consensus clustering, the following are computed by `generate_meta_avg_files`:\n",
    "\n",
    "* The average number of pixel SOM/meta clusters across all cell meta clusters, and the number of cells per meta cluster (the data placed in `cell_meta_cluster_count_avg_name`)\n",
    "* The meta cluster mapping for each cell SOM cluster in `cell_som_cluster_count_avg_name` (data is resaved, same data except with an associated meta cluster column)\n",
    "\n",
    "`generate_wc_avg_files` also creates the following:\n",
    "\n",
    "* The weighted channel average across all cell clusters (the data placed in `cell_som_cluster_channel_avg_name` and `cell_meta_cluster_channel_avg_name`). This will be done for both `'cell_som_cluster'` and `'cell_meta_cluster'`.\n",
    "\n",
    "For a full set of parameters you can customize for `cell_consensus_cluster`, please consult: <a href=https://ark-analysis.readthedocs.io/en/latest/_markdown/ark.phenotyping.html#ark.phenotyping.cell_cluster_utils.cell_consensus_cluster>cell consensus clustering docs</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* `max_k`: the number of consensus clusters desired\n",
    "* `cap`: used to clip z-scored values prior to consensus clustering (in the range `[-cap, cap]`)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "cell_consensus_cluster",
     "cell_mantis_project"
    ]
   },
   "outputs": [],
   "source": [
    "max_k = 20\n",
    "cap = 3\n",
    "\n",
    "# run hierarchical clustering based on cell SOM cluster assignments\n",
    "cell_cc, cluster_counts_size_norm = cell_cluster_utils.cell_consensus_cluster(\n",
    "    base_dir,\n",
    "    cell_som_cluster_cols=cell_som_cluster_cols,\n",
    "    cell_som_input_data=cluster_counts_size_norm,\n",
    "    cell_som_expr_col_avg_name=cell_som_cluster_count_avg_name,\n",
    "    max_k=max_k,\n",
    "    cap=cap\n",
    ")\n",
    "\n",
    "# generate the meta cluster summary files\n",
    "cell_cluster_utils.generate_meta_avg_files(\n",
    "    base_dir,\n",
    "    cell_cc,\n",
    "    cell_som_cluster_cols=cell_som_cluster_cols,\n",
    "    cell_som_input_data=cluster_counts_size_norm,\n",
    "    cell_som_expr_col_avg_name=cell_som_cluster_count_avg_name,\n",
    "    cell_meta_expr_col_avg_name=cell_meta_cluster_count_avg_name\n",
    ")\n",
    "\n",
    "# generate weighted channel summary files\n",
    "cell_cluster_utils.generate_wc_avg_files(\n",
    "    fovs,\n",
    "    channels,\n",
    "    base_dir,\n",
    "    cell_cc,\n",
    "    cell_som_input_data=cluster_counts_size_norm,\n",
    "    weighted_cell_channel_name=weighted_cell_channel_name,\n",
    "    cell_som_cluster_channel_avg_name=cell_som_cluster_channel_avg_name,\n",
    "    cell_meta_cluster_channel_avg_name=cell_meta_cluster_channel_avg_name\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4: visualize results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.1: use the interactive reclustering results to relabel cell meta clusters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The visualization shows the z-scored average pixel cluster count expression per cell SOM and meta cluster. The heatmaps are faceted by cell SOM clusters on the left and cell meta clusters on the right.\n",
    "\n",
    "## Usage\n",
    "\n",
    "### Quickstart\n",
    "- **Select**: Left Click\n",
    "- **Remap**: **New metacluster button** or Right Click\n",
    "- **Edit Metacluster Name**: Textbox at bottom right of the heatmaps.\n",
    "\n",
    "### Selection and Remapping details\n",
    "- To select a SOM cluster, click on its respective position in the **selected** bar. Click on it again to deselect.\n",
    "- To select a meta cluster, click on its corresponding color in the **metacluster** bar. Click on it again to deselect.\n",
    "- To remap the selected clusters, click the **New metacluster** button (alternatively, right click anywhere). Note that remapping an entire metacluster deletes it.\n",
    "- To clear the selected SOM/meta clusters, use the **Clear Selection** button.\n",
    "- **After remapping a meta cluster, make sure to deselect the newly created one to prevent unwanted combinations.**\n",
    "\n",
    "### Other features and notes\n",
    "- You will likely need to zoom out to see the entire visualization. To toggle Zoom, use Ctrl -/Ctrl + on Windows or ⌘ +/⌘ - on Mac.\n",
    "- The bars at the top show the number of cells in each SOM cluster.\n",
    "- The text box at the bottom right allows you to rename a particular meta cluster. This can be useful as remapping may cause inconsistent numbering.\n",
    "- Adjust the z-score limit using the slider on the bottom left to adjust your dynamic range.\n",
    "- When meta clusters are combined or a meta cluster is renamed, the change is immediately saved to `cell_meta_cluster_remap_name`.\n",
    "- You won't be able to advance until you've clicked `New metacluster` or renamed a meta cluster at least once. If you do not want to make changes, just click `New metacluster` to trigger a save before continuing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "cell_interactive"
    ]
   },
   "outputs": [],
   "source": [
    "%matplotlib widget\n",
    "rc_file_defaults()\n",
    "plt.ion()\n",
    "\n",
    "cell_mcd = metaclusterdata_from_files(\n",
    "    os.path.join(base_dir, cell_som_cluster_count_avg_name),\n",
    "    cluster_type='cell',\n",
    "    prefix_trim=pixel_cluster_col + '_'\n",
    ")\n",
    "cell_mcd.output_mapping_filename = os.path.join(base_dir, cell_meta_cluster_remap_name)\n",
    "cell_mcg = MetaClusterGui(cell_mcd, width=17)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Relabel the cell meta clusters using the mapping."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "cell_apply_remap"
    ]
   },
   "outputs": [],
   "source": [
    "# rename the meta cluster values in the cell dataset\n",
    "cluster_counts_size_norm = cell_cluster_utils.apply_cell_meta_cluster_remapping(\n",
    "    base_dir,\n",
    "    cluster_counts_size_norm,\n",
    "    cell_meta_cluster_remap_name\n",
    ")\n",
    "\n",
    "# recompute the mean column expression per meta cluster and apply these new names to the SOM cluster average data\n",
    "cell_cluster_utils.generate_remap_avg_count_files(\n",
    "    base_dir,\n",
    "    cluster_counts_size_norm,\n",
    "    cell_meta_cluster_remap_name,\n",
    "    cell_som_cluster_cols,\n",
    "    cell_som_cluster_count_avg_name,\n",
    "    cell_meta_cluster_count_avg_name,\n",
    ")\n",
    "\n",
    "# recompute the mean weighted channel expression per meta cluster and apply these new names to the SOM channel average data\n",
    "cell_cluster_utils.generate_remap_avg_wc_files(\n",
    "    fovs,\n",
    "    channels,\n",
    "    base_dir,\n",
    "    cluster_counts_size_norm,\n",
    "    cell_meta_cluster_remap_name,\n",
    "    weighted_cell_channel_name,\n",
    "    cell_som_cluster_channel_avg_name,\n",
    "    cell_meta_cluster_channel_avg_name\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Generate the color scheme returned by the interactive reclustering process. This will be for visualizing the weighted channel average heatmaps and the cell cluster overlay."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "cell_cmap_gen"
    ]
   },
   "outputs": [],
   "source": [
    "raw_cmap, renamed_cmap = colormap_helper.generate_meta_cluster_colormap_dict(\n",
    "    cell_mcd.output_mapping_filename,\n",
    "    cell_mcg.im_cl.cmap\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.2: weighted cell SOM cluster average heatmap over channels (z-scored)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "cell_som_heatmap"
    ]
   },
   "outputs": [],
   "source": [
    "cell_cluster_utils.generate_weighted_channel_avg_heatmap(\n",
    "    os.path.join(base_dir, cell_som_cluster_channel_avg_name),\n",
    "    'cell_som_cluster',\n",
    "    channels,\n",
    "    raw_cmap,\n",
    "    renamed_cmap\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.3: weighted cell meta cluster average heatmap over channels (z-scored)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "cell_meta_heatmap"
    ]
   },
   "outputs": [],
   "source": [
    "cell_cluster_utils.generate_weighted_channel_avg_heatmap(\n",
    "    os.path.join(base_dir, cell_meta_cluster_channel_avg_name),\n",
    "    'cell_meta_cluster_rename',\n",
    "    channels,\n",
    "    raw_cmap,\n",
    "    renamed_cmap\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.4: cell cluster overlay (cell meta clusters only)\n",
    "\n",
    "This will take your FOVs and generate cell cluster images in batches. Run this cell if you wish to create a significant amount of cell cluster mask images for downstream analysis."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "cell_overlay_fovs"
    ]
   },
   "outputs": [],
   "source": [
    "# select fovs to display\n",
    "subset_cell_fovs = ['fov0', 'fov1']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "cell_mask_gen_save"
    ]
   },
   "outputs": [],
   "source": [
    "# generate and save the cell cluster masks for each fov in subset_cell_fovs\n",
    "data_utils.generate_and_save_cell_cluster_masks(\n",
    "    fovs=subset_cell_fovs,\n",
    "    base_dir=base_dir,\n",
    "    save_dir=os.path.join(base_dir, \"pixie\", cell_output_dir),\n",
    "    seg_dir=os.path.join(base_dir, segmentation_dir),\n",
    "    cell_data=cluster_counts_size_norm,\n",
    "    seg_suffix=seg_suffix,\n",
    "    sub_dir='cell_masks',\n",
    "    name_suffix='_cell_mask'\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load a subset of the cell cluster masks that you would like to preview."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "cell_overlay_gen"
    ]
   },
   "outputs": [],
   "source": [
    "for cell_fov in subset_cell_fovs:\n",
    "    cell_cluster_mask = load_utils.load_imgs_from_dir(\n",
    "        data_dir = os.path.join(base_dir, \"pixie\", cell_output_dir, \"cell_masks\"),\n",
    "        files=[cell_fov + \"_cell_mask.tiff\"],\n",
    "        trim_suffix=\"_cell_mask\",\n",
    "        match_substring=\"_cell_mask\",\n",
    "        xr_dim_name=\"cell_mask\",\n",
    "        xr_channel_names=None,\n",
    "    )\n",
    "\n",
    "    plot_utils.plot_pixel_cell_cluster_overlay(\n",
    "        cell_cluster_mask,\n",
    "        [cell_fov],\n",
    "        os.path.join(base_dir, cell_meta_cluster_remap_name),\n",
    "        metacluster_colors=raw_cmap\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.5: append consensus cluster labels to cell table"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The cell table with consensus cluster labels is saved to `{cell_table_path}_cell_labels.csv`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "cell_append_meta"
    ]
   },
   "outputs": [],
   "source": [
    "cell_cluster_utils.add_consensus_labels_cell_table(\n",
    "    base_dir, cell_table_path, cluster_counts_size_norm\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.6: save the full results of Pixie cell clustering"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`cluster_counts_size_norm` with the SOM, meta, and renamed meta cluster labels, is saved to `cluster_counts_size_norm_name` as a `.feather` file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "pixie_cell_save"
    ]
   },
   "outputs": [],
   "source": [
    "feather.write_dataframe(\n",
    "    cluster_counts_size_norm,\n",
    "    os.path.join(base_dir, cluster_counts_size_norm_name),\n",
    "    compression='uncompressed'\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.7: Save Images a Mantis Viewer Project\n",
    "\n",
    "Mantis Viewer is a visualization tool for multi-dimensional imaging in pathology. Learn more about Mantis Viewer in the [README](../README.md#mantis-viewer)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "cell_mantis_project"
    ]
   },
   "outputs": [],
   "source": [
    "plot_utils.create_mantis_dir(\n",
    "    fovs=subset_cell_fovs,\n",
    "    mantis_project_path=os.path.join(base_dir, \"pixie\", cell_output_dir, \"mantis\"),\n",
    "    img_data_path=tiff_dir,\n",
    "    mask_output_dir=os.path.join(base_dir, \"pixie\", cell_output_dir, \"cell_masks\"),\n",
    "    mapping = os.path.join(base_dir, cell_meta_cluster_remap_name),\n",
    "    seg_dir=os.path.join(base_dir, segmentation_dir),\n",
    "    mask_suffix=\"_cell_mask\",\n",
    "    seg_suffix_name=seg_suffix\n",
    ")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.15"
  },
  "vscode": {
   "interpreter": {
    "hash": "b0ac63cc785e7c4cb24da0642dbfe09cfb2d314979108d7192228f9c45902a86"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}