Add tutorial on customizing insert

notebooks/04_PopulateConfigFile.ipynb

@@ -0,0 +1,249 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Customizing Data Insertion into Spyglass\n",
+    "\n",
+    "If you would like to insert data into Spyglass that does not\n",
+    "follow the naming or organizational format expected by Spyglass, \n",
+    "or you would like to override what values are ingested into Spyglass from \n",
+    "your NWB files, including missing values, follow this guide.\n",
+    "\n",
+    "## General Approach\n",
+    "\n",
+    "When an NWB file is ingested into Spyglass, metadata about the session\n",
+    "is first read from the NWB file and inserted into\n",
+    "tables in the `common` module (e.g. `Institution`, `Lab`, `Electrode`, etc).\n",
+    "However, not every NWB file has all the information required by Spyglass or\n",
+    "the information in the NWB file is not in a format that Spyglass expects. For\n",
+    "example, many NWB files do not contain information about the\n",
+    "`DataAcquisitionDevice` or `Probe` because the NWB data standard does not yet\n",
+    "have an official\n",
+    "standard for specifying them. For these cases, we provide a way to customize\n",
+    "how data is ingested into Spyglass.\n",
+    "\n",
+    "Let's say that you want to ingest an NWB file into Spyglass where the lab name\n",
+    "in the file is written as \"Loren Frank Lab\" or it is not specified, but you \n",
+    "know the data comes from the Loren Frank Lab. Let's say that in Spyglass,\n",
+    "the lab name that is associated with sessions from the Loren Frank Lab is \n",
+    "\"Frank Lab\" and you would like to use the same name in order to facilitate\n",
+    "data search in Spyglass. To change the lab name when you insert your new data \n",
+    "to Spyglass, you could either 1) edit the NWB file directly and then \n",
+    "insert it into Spyglass, or 2) define an override value \"Frank Lab\" to be \n",
+    "used instead of the value specified in the NWB file (or lack thereof).\n",
+    "\n",
+    "Note that if this is your own data and you want to make changes to\n",
+    "information about how the data is interpreted, e.g., the units of measurement\n",
+    "are incorrect, we recommend that you edit the NWB file directly because the \n",
+    "file or derivatives of it might eventually be shared outside of Spyglass and \n",
+    "they will not reflect any modifications that you have made to \n",
+    "the data only in Spyglass."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Define a Configuration YAML File\n",
+    "\n",
+    "To override values in the NWB file during insertion into Spyglass, \n",
+    "you will need to create a configuration \n",
+    "YAML file that lives in the same directory as your NWB file, named: \n",
+    "`<name_of_nwb_file>_spyglass_config.yaml`\n",
+    "\n",
+    "An example configuration YAML file can be found at\n",
+    "`examples/config_yaml/​​sub-AppleBottom_ses-AppleBottom-DY20-g3_behavior+ecephys_spyglass_config.yaml`.\n",
+    "This file is associated with the NWB file\n",
+    "`sub-AppleBottom_ses-AppleBottom-DY20-g3_behavior+ecephys.nwb`.\n",
+    "\n",
+    "This is the general format for entries in this configuration file:\n",
+    "\n",
+    "```yaml\n",
+    "TableName:\n",
+    "- primary_key1: value1\n",
+    "```\n",
+    "\n",
+    "For example:\n",
+    "\n",
+    "```yaml\n",
+    "Lab:\n",
+    "- lab_name: Frank Lab\n",
+    "```\n",
+    "\n",
+    "In this example, the NWB file that corresponds to this config YAML will become\n",
+    "associated with the entry in the `Lab` table with the value `Frank Lab` for \n",
+    "the primary key `lab_name`. This entry must already exist. More specifically,\n",
+    "when the NWB file is ingested into Spyglass, \n",
+    "a new `Session` entry will be created for the NWB file that has a foreign key to\n",
+    "the `Lab` entry with `lab_name` = \"Frank Lab\", ignoring whatever lab value is \n",
+    "in the NWB file, even if one does not exist.\n",
+    "\n",
+    "TODO implement this for `Lab`.\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Create Entries to Reference in the Configuration YAML\n",
+    "\n",
+    "As mentioned earlier, the table entry that you want to associate with your NWB\n",
+    "file must already exist in the database. This entry would typically be a value\n",
+    "that is independent of any particular NWB file, such as\n",
+    "`DataAcquisitionDevice`, `Lab`, `Probe`, and `BrainRegion`. \n",
+    "\n",
+    "If the entry does not already exist, you can either:\n",
+    "1) create the entry programmatically using DataJoint `insert` commands, or\n",
+    "2) define the entry in a YAML file called `entries.yaml` that is automatically\n",
+    "   processed when Spyglass is imported. You can think of `entries.yaml` as a\n",
+    "   place to define information that the database should come pre-equipped prior\n",
+    "   to ingesting your NWB files. The `entries.yaml` file should be placed in the\n",
+    "   `spyglass` base directory (next to `README.md`). An example can be found in\n",
+    "   `examples/config_yaml/entries.yaml`. This file should have the following\n",
+    "   structure:\n",
+    "\n",
+    "    ```yaml\n",
+    "    TableName:\n",
+    "    - TableEntry1Field1: Value\n",
+    "      TableEntry1Field2: Value\n",
+    "    - TableEntry2Field1: Value\n",
+    "      TableEntry2Field2: Value\n",
+    "    ```\n",
+    "\n",
+    "    For example,\n",
+    "\n",
+    "    ```yaml\n",
+    "    DataAcquisitionDeviceSystem:\n",
+    "      data_acquisition_device_system: SpikeGLX\n",
+    "    DataAcquisitionDevice:\n",
+    "    - data_acquisition_device_name: Neuropixels_SpikeGLX\n",
+    "      data_acquisition_device_system: SpikeGLX\n",
+    "      data_acquisition_device_amplifier: Intan\n",
+    "    ```\n",
+    "\n",
+    "    Only `dj.Manual`, `dj.Lookup`, and `dj.Part` tables can be populated\n",
+    "    using this approach.\n",
+    "\n",
+    "Once the entry that you want to associate with your NWB file exists in the\n",
+    "database, you can write the configuration YAML file and then ingest your\n",
+    "NWB file. As an another example, let's say that you want to associate your NWB\n",
+    "file with the `DataAcquisitionDevice` entry with `data_acquisition_device_name`\n",
+    "= \"Neuropixels_SpikeGLX\" that was defined above. You would write the following\n",
+    "configuration YAML file:\n",
+    "\n",
+    "```yaml\n",
+    "DataAcquisitionDevice:\n",
+    "- data_acquisition_device_name: Neuropixels_SpikeGLX\n",
+    "```\n",
+    "\n",
+    "The example in\n",
+    "`examples/config_yaml/​​sub-AppleBottom_ses-AppleBottom-DY20-g3_behavior+ecephys_spyglass_config.yaml`\n",
+    "includes additional examples."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Example Ingestion with Real Data\n",
+    "\n",
+    "For this example, you will need to download the NWB file `sub-JDS-NFN-AM2_behavior+ecephys.nwb`\n",
+    "from dandiset 000447 here: \n",
+    "https://dandiarchive.org/dandiset/000447/0.230316.2133/files?location=sub-JDS-NFN-AM2&page=1\n",
+    "\n",
+    "Click the download arrow button to download the file to your computer. Add it to the folder\n",
+    "containing your raw NWB data to be ingested into Spyglass.\n",
+    "\n",
+    "This file does not specify a data acquisition device. Let's say that the\n",
+    "data was collected from a SpikeGadgets system with an Intan amplifier. This\n",
+    "matches an existing entry in the `DataAcquisitionDevice` table with name\n",
+    "\"data_acq_device0\". We will create a configuration YAML file to associate\n",
+    "this entry with the NWB file."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# this configuration yaml file should be placed next to the downloaded NWB file\n",
+    "yaml_config_path = \"sub-JDS-NFN-AM2_behavior+ecephys_spyglass_config.yaml\"\n",
+    "with open(yaml_config_path, \"w\") as config_file:\n",
+    "    lines = [\n",
+    "        \"DataAcquisitionDevice\",\n",
+    "        \"- data_acquisition_device_name: data_acq_device0\",\n",
+    "    ]\n",
+    "    config_file.writelines(line + '\\n' for line in lines)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Then call `insert_sessions` as usual."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import spyglass.data_import as sgi\n",
+    "\n",
+    "nwb_file_name = \"sub-JDS-NFN-AM2_behavior+ecephys.nwb\"\n",
+    "sgi.insert_sessions(nwb_file_name)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Confirm the session was inserted with the correct `DataAcquisitionDevice`"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import spyglass.common as sgc\n",
+    "from spyglass.utils.nwb_helper_fn import get_nwb_copy_filename\n",
+    "\n",
+    "nwb_copy_file_name = get_nwb_copy_filename(nwb_file_name)\n",
+    "\n",
+    "sgc.Session.DataAcquisitionDevice & {\"nwb_file_name\": nwb_copy_file_name}"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.9.18"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}