egocentric align

sgoldenlab · Dec 9, 2024 · 9c62b82 · 9c62b82
1 parent 5e93a6d
commit 9c62b82
Show file tree

Hide file tree

Showing 20 changed files with 982 additions and 195 deletions.
diff --git a/docs/_static/img/egocentric_nb_1.webm b/docs/_static/img/egocentric_nb_1.webm
diff --git a/docs/_static/img/egocentric_nb_2.webm b/docs/_static/img/egocentric_nb_2.webm
diff --git a/docs/_static/img/egocentric_nb_3.webm.webm b/docs/_static/img/egocentric_nb_3.webm.webm
diff --git a/docs/nb/egocentric_align.ipynb b/docs/nb/egocentric_align.ipynb
@@ -0,0 +1,301 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "4f019f1c",
+   "metadata": {},
+   "source": [
+    "# Egocentric data and video alignment"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "cb011978",
+   "metadata": {},
+   "source": [
+    "In this notebook, we will egocentrically align pose estimation and associated video data.\n",
+    "\n",
+    "Egocentric alignment is a crucial preprocessing step with various applications.\n",
+    "\n",
+    "One primary use case is in unsupervised deep learning scenarios, where images of animals are used as inputs to train algorithms.\n",
+    "Standardizing the orientation of the subject (e.g., an animal) ensures that the model and subsequent analyses do not incorrectly interpret the same behavior in different orientations\n",
+    "(e.g., grooming while facing north versus south) as distinct behaviors. By aligning the animal to a consistent frame of reference, we eliminate orientation as a confounding variable.\n",
+    "\n",
+    "While egocentric alignment is an essential first step, it is often insufficient by itself for comprehensive analyses. Additional preprocessing steps are typically required, such as:\n",
+    "\n",
+    "* Background subtraction to isolate the animal from its surroundings (see other relevant methods and notebooks).\n",
+    "* Geometric segmentation to slice out and focus on the subject (again, see other relevant methods and notebooks).\n",
+    "\n",
+    "In this notebook, we will focus exclusively on performing egocentric alignment. Further preprocessing steps are outlined in related materials."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "id": "4654f323",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from simba.data_processors.egocentric_aligner import EgocentricalAligner\n",
+    "from ipywidgets import Video\n",
+    "from IPython.display import HTML"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "id": "4d4919e1",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# WE DEFINE HOW WE SHOULD EGOCENTRICALLY ALIGN THE DATA AND VIDEOs\n",
+    "\n",
+    "ANCHOR_POINT_1 = 'center'       # Name of the body-part which is the \"primary\" anchor point around which the alignment centers. In rodents, this is often the center of the tail-base of the animal.\n",
+    "ANCHOR_POINT_2 = 'nose'         # The name of the secondary anchor point defining the alignment direction. This is often the anterior body-part, in rodents it can be the nose or nape.\n",
+    "DIRECTION = 0                   # The egocentric alignment angle, in degrees. For example, `0` and the animals `ANCHOR_POINT_2` is directly to the east (right) of `ANCHOR_POINT_1`. `180` and the animals `ANCHOR_POINT_2` is directly to the west (left) of `ANCHOR_POINT_1`.\n",
+    "ANCHOR_LOCATION = (250, 250)    # The pixel location in the video where `ANCHOR_POINT_1` should be placed. For example, if the videos are 500x500, 250x250 will place the anchor right in the middle.\n",
+    "GPU = True                      # If we have an NVIDEA GPU availalable, we can use it to speed up processing. Otherwise set this to `False`.\n",
+    "FILL_COLOR = (0, 0, 0)          # We are rotating videos, while at the same time retaining the original video size. Therefore, there will be some \"new\" areas exposed in the video (see below for more info). This is what color to color these new areas.\n",
+    "VERBOSE = False                 # If True, prints progress (like which frame and video is being processed etc). However, this information will flood this notebook is I am going to turn it off."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "id": "8627686d",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# WE DEFINE THE PATHS TO THE DIRECTORIES HOLDING THE DATA AND VIDEOS, AND DIRECTORY WHERE WE SHOULD STORE THE RESULTS.\n",
+    "\n",
+    "DATA_DIRECTORY = r'C:\\Users\\sroni\\OneDrive\\Desktop\\rotate_ex\\data' #DIRECTORY WHICH IS HOLDING POSE-ESTIMATION DATA\n",
+    "VIDEOS_DIRECTORY = r'C:\\Users\\sroni\\OneDrive\\Desktop\\rotate_ex\\videos' #DIRECTORY WHICH IS VIDEOS, ONE FOR EACH FILE IN THE DATA_DIRECTORY. NOTE: IF YOU SET THIS TO None, THEN THE ALIGNMENT WILL BE PERFORMED ON THE DATA ONLY.\n",
+    "SAVE_DIRECTORY = r\"C:\\Users\\sroni\\OneDrive\\Desktop\\rotate_ex\\rotated\" #DIRECTORY WHERE WE SHOULD SAVE THE ROTATED POSE-ESTIMATION AND ROTATED VIDEOS."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "id": "a3680603",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "SIMBA COMPLETE: Video concatenated (elapsed time: 16.214s) \tcomplete\n",
+      "SIMBA COMPLETE: Egocentric rotation video C:\\Users\\sroni\\OneDrive\\Desktop\\rotate_ex\\rotated\\501_MA142_Gi_Saline_0513.mp4 complete (elapsed time: 102.2651s) \tcomplete\n",
+      "SIMBA COMPLETE: Video concatenated (elapsed time: 15.0453s) \tcomplete\n",
+      "SIMBA COMPLETE: Egocentric rotation video C:\\Users\\sroni\\OneDrive\\Desktop\\rotate_ex\\rotated\\501_MA142_Gi_Saline_0515.mp4 complete (elapsed time: 101.7265s) \tcomplete\n",
+      "SIMBA COMPLETE: Video concatenated (elapsed time: 14.7956s) \tcomplete\n",
+      "SIMBA COMPLETE: Egocentric rotation video C:\\Users\\sroni\\OneDrive\\Desktop\\rotate_ex\\rotated\\501_MA142_Gi_Saline_0517.mp4 complete (elapsed time: 105.1418s) \tcomplete\n",
+      "SIMBA COMPLETE: Video concatenated (elapsed time: 16.5156s) \tcomplete\n",
+      "SIMBA COMPLETE: Egocentric rotation video C:\\Users\\sroni\\OneDrive\\Desktop\\rotate_ex\\rotated\\502_MA141_Gi_Saline_0513.mp4 complete (elapsed time: 107.4481s) \tcomplete\n",
+      "SIMBA COMPLETE: Video concatenated (elapsed time: 14.7832s) \tcomplete\n",
+      "SIMBA COMPLETE: Egocentric rotation video C:\\Users\\sroni\\OneDrive\\Desktop\\rotate_ex\\rotated\\502_MA141_Gi_Saline_0515.mp4 complete (elapsed time: 103.6864s) \tcomplete\n",
+      "SIMBA COMPLETE: Video concatenated (elapsed time: 14.9907s) \tcomplete\n",
+      "SIMBA COMPLETE: Egocentric rotation video C:\\Users\\sroni\\OneDrive\\Desktop\\rotate_ex\\rotated\\502_MA141_Gi_Saline_0517.mp4 complete (elapsed time: 106.1234s) \tcomplete\n",
+      "SIMBA COMPLETE: Egocentrically aligned data for 6 files saved in C:\\Users\\sroni\\OneDrive\\Desktop\\rotate_ex\\rotated (elapsed time: 627.5204s) \tcomplete\n"
+     ]
+    }
+   ],
+   "source": [
+    "# NOW WE ARE GOOD TO GO, USING THE INFORMATION ABOVE, WE DEFINE AN INSTANCE OF AN SimBA EGOCENTRIALIGNER AND RUN IT\n",
+    "# ON THE 6 VIDEOS AND VIDEO DATA INSIDE THE DATA AND VIDEO DIRECTORIES, RESPECTIVELY.\n",
+    "aligner = EgocentricalAligner(anchor_1=ANCHOR_POINT_1,\n",
+    "                              anchor_2=ANCHOR_POINT_2,\n",
+    "                              data_dir=DATA_DIRECTORY,\n",
+    "                              videos_dir=VIDEOS_DIRECTORY,\n",
+    "                              save_dir=SAVE_DIRECTORY,\n",
+    "                              direction=DIRECTION,\n",
+    "                              gpu=GPU,\n",
+    "                              anchor_location=ANCHOR_LOCATION,\n",
+    "                              fill_clr=FILL_COLOR,\n",
+    "                              verbose=VERBOSE)\n",
+    "aligner.run()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "id": "54298ed3",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/html": [
+       " <video width=\"600\" height=\"600\" controls> <source src=\"https://raw.githubusercontent.com/sgoldenlab/simba/master/docs/_static/img/egocentric_nb_1.webm\" type=\"video/webm\"> </video>\n"
+      ],
+      "text/plain": [
+       "<IPython.core.display.HTML object>"
+      ]
+     },
+     "execution_count": 6,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "#EXAMPLE VIDEO EXPECTED RESULTS SNIPPET\n",
+    "video_url = 'https://raw.githubusercontent.com/sgoldenlab/simba/master/docs/_static/img/egocentric_nb_1.webm'\n",
+    "HTML(f''' <video width=\"600\" height=\"600\" controls> <source src=\"{video_url}\" type=\"video/webm\"> </video>\n",
+    "''')"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "id": "e67378a7",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "#NOW, LET'S CHANGE A FEW SETTINGS, TO GET A FEELING FOR HOW IT BEHAVES …\n",
+    "ANCHOR_LOCATION = (500, 100)\n",
+    "FILL_COLOR = (255, 0, 0)\n",
+    "DIRECTION = 180"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 8,
+   "id": "8ead691a",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "SIMBA COMPLETE: Video concatenated (elapsed time: 16.2393s) \tcomplete\n",
+      "SIMBA COMPLETE: Egocentric rotation video C:\\Users\\sroni\\OneDrive\\Desktop\\rotate_ex\\rotated\\501_MA142_Gi_Saline_0513.mp4 complete (elapsed time: 108.1662s) \tcomplete\n",
+      "SIMBA COMPLETE: Video concatenated (elapsed time: 15.0821s) \tcomplete\n",
+      "SIMBA COMPLETE: Egocentric rotation video C:\\Users\\sroni\\OneDrive\\Desktop\\rotate_ex\\rotated\\501_MA142_Gi_Saline_0515.mp4 complete (elapsed time: 101.4825s) \tcomplete\n",
+      "SIMBA COMPLETE: Video concatenated (elapsed time: 14.7555s) \tcomplete\n",
+      "SIMBA COMPLETE: Egocentric rotation video C:\\Users\\sroni\\OneDrive\\Desktop\\rotate_ex\\rotated\\501_MA142_Gi_Saline_0517.mp4 complete (elapsed time: 102.9438s) \tcomplete\n",
+      "SIMBA COMPLETE: Video concatenated (elapsed time: 16.4037s) \tcomplete\n",
+      "SIMBA COMPLETE: Egocentric rotation video C:\\Users\\sroni\\OneDrive\\Desktop\\rotate_ex\\rotated\\502_MA141_Gi_Saline_0513.mp4 complete (elapsed time: 109.8081s) \tcomplete\n",
+      "SIMBA COMPLETE: Video concatenated (elapsed time: 14.8025s) \tcomplete\n",
+      "SIMBA COMPLETE: Egocentric rotation video C:\\Users\\sroni\\OneDrive\\Desktop\\rotate_ex\\rotated\\502_MA141_Gi_Saline_0515.mp4 complete (elapsed time: 103.8362s) \tcomplete\n",
+      "SIMBA COMPLETE: Video concatenated (elapsed time: 15.0489s) \tcomplete\n",
+      "SIMBA COMPLETE: Egocentric rotation video C:\\Users\\sroni\\OneDrive\\Desktop\\rotate_ex\\rotated\\502_MA141_Gi_Saline_0517.mp4 complete (elapsed time: 107.5951s) \tcomplete\n",
+      "SIMBA COMPLETE: Egocentrically aligned data for 6 files saved in C:\\Users\\sroni\\OneDrive\\Desktop\\rotate_ex\\rotated (elapsed time: 634.8647s) \tcomplete\n"
+     ]
+    }
+   ],
+   "source": [
+    "# ... WE CREATE A NEW INSTANCE BASED ON THE UPDATED INFORMATION ABOVE, AND RUN IT.\n",
+    "aligner = EgocentricalAligner(anchor_1=ANCHOR_POINT_1,\n",
+    "                              anchor_2=ANCHOR_POINT_2,\n",
+    "                              data_dir=DATA_DIRECTORY,\n",
+    "                              videos_dir=VIDEOS_DIRECTORY,\n",
+    "                              save_dir=SAVE_DIRECTORY,\n",
+    "                              direction=DIRECTION,\n",
+    "                              gpu=GPU,\n",
+    "                              anchor_location=ANCHOR_LOCATION,\n",
+    "                              fill_clr=FILL_COLOR,\n",
+    "                              verbose=VERBOSE)\n",
+    "aligner.run()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 10,
+   "id": "dd458e84",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/html": [
+       " <video width=\"600\" height=\"600\" controls> <source src=\"https://raw.githubusercontent.com/sgoldenlab/simba/master/docs/_static/img/egocentric_nb_2.webm\" type=\"video/webm\"> </video>\n"
+      ],
+      "text/plain": [
+       "<IPython.core.display.HTML object>"
+      ]
+     },
+     "execution_count": 10,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "#EXAMPLE VIDEO EXPECTED RESULTS SNIPPET\n",
+    "video_url = 'https://raw.githubusercontent.com/sgoldenlab/simba/master/docs/_static/img/egocentric_nb_2.webm'\n",
+    "HTML(f''' <video width=\"600\" height=\"600\" controls> <source src=\"{video_url}\" type=\"video/webm\"> </video>\n",
+    "''')"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "0f659939",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# FINALLY, LET'S USE A DIFFERENT EXPERIMENT, WITH DIFFERENT BODY-PART NAMES TO SEE HOW IT BEHAVES\n",
+    "ANCHOR_POINT_1 = 'center'       \n",
+    "ANCHOR_POINT_2 = 'nose'      \n",
+    "DIRECTION = 0    \n",
+    "ANCHOR_LOCATION = (600, 300)\n",
+    "FILL_COLOR = (128, 120, 128) \n",
+    "\n",
+    "DATA_DIRECTORY = r'C:\\troubleshooting\\open_field_below\\project_folder\\csv\\outlier_corrected_movement_location'\n",
+    "VIDEOS_DIRECTORY = r'C:\\troubleshooting\\open_field_below\\project_folder\\videos'\n",
+    "SAVE_DIRECTORY = r\"C:\\troubleshooting\\open_field_below\\project_folder\\videos\\rotated\""
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "9c654f6a",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# ... AGAIN WE CREATE A NEW INSTANCE BASED ON THE UPDATED INFORMATION ABOVE, AND RUN IT.\n",
+    "aligner = EgocentricalAligner(anchor_1=ANCHOR_POINT_1,\n",
+    "                              anchor_2=ANCHOR_POINT_2,\n",
+    "                              data_dir=DATA_DIRECTORY,\n",
+    "                              videos_dir=VIDEOS_DIRECTORY,\n",
+    "                              save_dir=SAVE_DIRECTORY,\n",
+    "                              direction=DIRECTION,\n",
+    "                              gpu=GPU,\n",
+    "                              anchor_location=ANCHOR_LOCATION,\n",
+    "                              fill_clr=FILL_COLOR,\n",
+    "                              verbose=VERBOSE)\n",
+    "aligner.run()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "5a70d3f8",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "#EXAMPLE VIDEO EXPECTED RESULTS SNIPPET\n",
+    "video_url = 'https://raw.githubusercontent.com/sgoldenlab/simba/master/docs/_static/img/egocentric_nb_2.webm'\n",
+    "HTML(f''' <video width=\"600\" height=\"600\" controls> <source src=\"{video_url}\" type=\"video/webm\"> </video>\n",
+    "''')"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "simba",
+   "language": "python",
+   "name": "simba"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.6.13"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}
diff --git a/docs/nb/geometry_example_1.ipynb b/docs/nb/geometry_example_1.ipynb
@@ -9,7 +9,6 @@
    ]
   },
   {
-   "attachments": {},
    "cell_type": "markdown",
    "id": "85aedd1a",
    "metadata": {},
@@ -335,7 +334,7 @@
  ],
  "metadata": {
   "kernelspec": {
-   "display_name": "Python 3 (ipykernel)",
+   "display_name": "Python 3",
    "language": "python",
    "name": "python3"
   },
@@ -349,7 +348,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.10.15"
+   "version": "3.6.13"
   }
  },
  "nbformat": 4,

diff --git a/docs/nb/geometry_example_6.ipynb b/docs/nb/geometry_example_6.ipynb
@@ -503,7 +503,7 @@
  ],
  "metadata": {
   "kernelspec": {
-   "display_name": "Python 3 (ipykernel)",
+   "display_name": "Python 3",
    "language": "python",
    "name": "python3"
   },
@@ -517,7 +517,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.10.15"
+   "version": "3.6.13"
   }
  },
  "nbformat": 4,

diff --git a/docs/nb/geometry_example_7.ipynb b/docs/nb/geometry_example_7.ipynb
@@ -690,7 +690,7 @@
  ],
  "metadata": {
   "kernelspec": {
-   "display_name": "Python 3 (ipykernel)",
+   "display_name": "Python 3",
    "language": "python",
    "name": "python3"
   },
@@ -704,7 +704,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.10.15"
+   "version": "3.6.13"
   }
  },
  "nbformat": 4,

diff --git a/docs/simba.data_processors.rst b/docs/simba.data_processors.rst
@@ -180,3 +180,12 @@ Heuristic freezing detector
    :members:
    :undoc-members:
 
+
+Data GPU methods
+----------------------------------------------
+
+.. automodule:: simba.data_processors.cuda.data
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
diff --git a/docs/simba.video_processing.rst b/docs/simba.video_processing.rst
@@ -51,9 +51,16 @@ Interactive brightness / contrast
    :show-inheritance:
 
 
-Batch process executor
+Batch video process executor
 -----------------------------------------------------------------------
 
 .. automodule:: simba.video_processors.batch_process_create_ffmpeg_commands
    :members:
    :show-inheritance:
+
+Egocentrically rotate videos
+-----------------------------------------------------------------------
+
+.. automodule:: simba.video_processors.egocentric_video_rotator
+   :members:
+   :show-inheritance: