Re-organize docs (#1029)

* Docs Overhaul. Ignore pre-commit

* Note pre-commit exception

* Use most recent tag to deploy docs on manual dispatch

* Use marketplace git describe for tag

* Fix typo

* Upgrade workflow deps

* Test tag fetch

* Minor fixes

* Fix failing tests

* 🐛 : Debug tests

* Update C

* Add params edit snippet

* Mention optional dependencies in 00_Setup

* SpikesortV0: mention version

* Add nwb-obj/tbl mappings from @samuelbray32

* Revert version bump

.github/workflows/publish-docs.yml

@@ -2,7 +2,7 @@ name: Publish docs
 on:
   push:
     tags: # See PEP 440 for valid version format
-      - "*.*.*" # For docs bump, use X.X.XaX
+      - "*.*.*" # For docs bump, use workflow_dispatch
     branches:
       - test_branch
   workflow_dispatch: # Manually trigger with 'Run workflow' button
@@ -23,13 +23,17 @@ jobs:
       issues: write
     steps:
       - name: Checkout repository
-        uses: actions/checkout@v3
+        uses: actions/checkout@v4
         with:
           token: ${{ secrets.GITHUB_TOKEN }}
           fetch-depth: 0
 
+      - name: Git describe # Get tags
+        id: ghd # see Deploy below. Will fail if no tags on branch
+        uses: proudust/gh-describe@v2
+
       - name: Set up Python runtime
-        uses: actions/setup-python@v4
+        uses: actions/setup-python@v5
         with:
           python-version: 3.9
           token: ${{ secrets.GITHUB_TOKEN }}
@@ -40,8 +44,8 @@ jobs:
           git config user.name 'github-actions[bot]' && git config user.email 'github-actions[bot]@users.noreply.github.com'
 
       - name: Deploy
-        run: |
-          FULL_VERSION=${{ github.ref_name }}
+        run: | # github.ref_name is branch name if dispatch
+          FULL_VERSION=${{ steps.ghd.outputs.tag }}
           export MAJOR_VERSION=${FULL_VERSION:0:3}
           echo "OWNER: ${REPO_OWNER}. BUILD: ${MAJOR_VERSION}"
           bash ./docs/build-docs.sh push $REPO_OWNER

.github/workflows/test-package-build.yml

@@ -64,7 +64,7 @@ jobs:
         with:
           name: archive
           path: archive/
-      - uses: actions/setup-python@v4
+      - uses: actions/setup-python@v5
         with:
           python-version: 3.9
       - name: Display Python version

.markdownlint.yaml

@@ -1,8 +1,10 @@
 # https://github.com/DavidAnson/markdownlint
 # https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md
-MD007: false # permit indenting 4 spaces instead of 2
+MD007: # permit indenting 4 spaces instead of 2
+  indent: 4
+  start_indent: 4
 MD013:
-    line_length: "80" # Line length limits
+    line_length: 80 # Line length limits
     tables: false # disable for tables
     code_blocks: false # disable for code blocks
 MD025: false # permit adjacent headings

CHANGELOG.md

@@ -2,7 +2,7 @@
 
 ## [0.5.3] (Unreleased)
 
-### Release Notes
+## Release Notes
 
 <!-- Running draft to be removed immediately prior to release. -->
 
@@ -36,10 +36,14 @@ PositionGroup.alter()
 - Ensure integrity of group tables #1026
 - Convert list of LFP artifact removed interval list to array #1046
 - Merge duplicate functions in decoding and spikesorting #1050
+- Revise docs organization.
+    - Misc -> Features/ForDevelopers. #1029
+    - Installation instructions -> Setup notebook. #1029
 
 ### Pipelines
 
 - Common
+
     - `PositionVideo` table now inserts into self after `make` #966
     - Don't insert lab member when creating lab team #983
     - Files created by `AnalysisNwbfile.create()` receive new object_id #999
@@ -51,10 +55,14 @@ PositionGroup.alter()
     - `PositionIntervalMap` now inserts null entries for missing intervals #870
     - `AnalysisFileLog` now truncates table names that exceed field length #1021
     - Disable logging with `AnalysisFileLog` #1024
+
 - Decoding:
+
     - Default values for classes on `ImportError` #966
     - Add option to upsample data rate in `PositionGroup` #1008
+
 - Position
+
     - Allow dlc without pre-existing tracking data #973, #975
     - Raise `KeyError` for missing input parameters across helper funcs #966
     - `DLCPosVideo` table now inserts into self after `make` #966
@@ -67,7 +75,9 @@ PositionGroup.alter()
         `get_video_info` to reflect actual use #870
     - Fix `red_led_bisector` `np.nan` handling issue from #870. Fixed in #1034
     - Fix `one_pt_centoid` `np.nan` handling issue from #870. Fixed in #1034
+
 - Spikesorting
+
     - Allow user to set smoothing timescale in `SortedSpikesGroup.get_firing_rate`
         #994
     - Update docstrings #996

CITATION.cff

@@ -2,7 +2,7 @@
 # Visit https://bit.ly/cffinit to generate yours today!
 
 cff-version: 1.2.0
-title: spyglass
+title: 'Spyglass: a data analysis framework for reproducible and shareable neuroscience research'
 message: 'If you use this software, please cite it as below.'
 type: software
 authors:
@@ -84,18 +84,18 @@ authors:
     email: [email protected]
     affiliation: 'University of California, San Francisco'
     orcid: 'https://orcid.org/0000-0001-5559-2910'
-  - given-names: Shin
-    family-names: Donghoon
+  - given-names: Donghoon
+    family-names: Shin
     email: [email protected]
     affiliation: 'University of California, San Francisco'
     orcid: 'https://orcid.org/0009-0000-8916-7314'
-  - given-names: Chiang
-    family-names: Sharon
+  - given-names: Sharon
+    family-names: Chiang
     email: [email protected]
     affiliation: 'University of California, San Francisco'
     orcid: 'https://orcid.org/0000-0002-4548-4550'
-  - given-names: Holobetz
-    family-names: Cristofer
+  - given-names: Cristofer
+    family-names: Holobetz
     email: [email protected]
     affiliation: 'University College London'
     orcid: 'https://orcid.org/0009-0009-8567-3290'

README.md

@@ -5,14 +5,20 @@
 
 ![Spyglass Figure](docs/src/images/fig1.png)
 
-[Demo](https://spyglass.hhmi.2i2c.cloud/hub/user-redirect/git-pull?repo=https%3A%2F%2Fgithub.com%2FLorenFrankLab%2Fspyglass-demo&urlpath=lab%2Ftree%2Fspyglass-demo%2Fnotebooks%2F01_Insert_Data.ipynb&branch=main) | [Installation](https://lorenfranklab.github.io/spyglass/latest/installation/) | [Docs](https://lorenfranklab.github.io/spyglass/) | [Tutorials](https://github.com/LorenFrankLab/spyglass/tree/master/notebooks) | [Citation](#citation)
+[Demo](https://spyglass.hhmi.2i2c.cloud/hub/user-redirect/git-pull?repo=https%3A%2F%2Fgithub.com%2FLorenFrankLab%2Fspyglass-demo&urlpath=lab%2Ftree%2Fspyglass-demo%2Fnotebooks%2F02_Insert_Data.ipynb&branch=main)
+|
+[Installation](https://lorenfranklab.github.io/spyglass/latest/notebooks/00_Setup/)
+| [Docs](https://lorenfranklab.github.io/spyglass/) |
+[Tutorials](https://github.com/LorenFrankLab/spyglass/tree/master/notebooks) |
+[Citation](#citation)
 
 `spyglass` is a data analysis framework that facilitates the storage, analysis,
 visualization, and sharing of neuroscience data to support reproducible
 research. It is designed to be interoperable with the NWB format and integrates
 open-source tools into a coherent framework.
 
-Try out a demo [here](https://spyglass.hhmi.2i2c.cloud/hub/user-redirect/git-pull?repo=https%3A%2F%2Fgithub.com%2FLorenFrankLab%2Fspyglass-demo&urlpath=lab%2Ftree%2Fspyglass-demo%2Fnotebooks%2F01_Insert_Data.ipynb&branch=main)!
+Try out a demo
+[here](https://spyglass.hhmi.2i2c.cloud/hub/user-redirect/git-pull?repo=https%3A%2F%2Fgithub.com%2FLorenFrankLab%2Fspyglass-demo&urlpath=lab%2Ftree%2Fspyglass-demo%2Fnotebooks%2F02_Insert_Data.ipynb&branch=main)!
 
 Features of Spyglass include:
 
@@ -60,16 +66,16 @@ Documentation can be found at -
 ## Installation
 
 For installation instructions see -
-[https://lorenfranklab.github.io/spyglass/latest/installation/](https://lorenfranklab.github.io/spyglass/latest/installation/)
+[https://lorenfranklab.github.io/spyglass/latest/notebooks/00_Setup/](https://lorenfranklab.github.io/spyglass/latest/notebooks/00_Setup/)
 
 Typical installation time is: 5-10 minutes
 
 ## Tutorials
 
-The tutorials for `spyglass` is currently in the form of Jupyter Notebooks and
+The tutorials for `spyglass` are currently in the form of Jupyter Notebooks and
 can be found in the
 [notebooks](https://github.com/LorenFrankLab/spyglass/tree/master/notebooks)
-directory. We strongly recommend opening them in the context of `jupyterlab`.
+directory. We strongly recommend running the notebooks yourself.
 
 ## Contributing
 
@@ -85,17 +91,31 @@ License and Copyright notice can be found at
 
 ## System requirements
 
-Spyglass has been tested on Linux Ubuntu 20.04 and MacOS 10.15. It has not been tested on Windows and likely will not work.
+Spyglass has been tested on Linux Ubuntu 20.04 and MacOS 10.15. It has not been
+tested on Windows and likely will not work.
 
-No specific hardware requirements are needed to run spyglass. However, the amount of data that can be stored and analyzed is limited by the available disk space and memory. GPUs are required for some of the analysis tools, such as DeepLabCut.
+No specific hardware requirements are needed to run spyglass. However, the
+amount of data that can be stored and analyzed is limited by the available disk
+space and memory. GPUs are required for some of the analysis tools, such as
+DeepLabCut.
 
-See [pyproject.toml](pyproject.toml), [environment.yml](environment.yml), or [environment_dlc.yml](environment_dlc.yml) for software dependencies.
+See [pyproject.toml](pyproject.toml), [environment.yml](environment.yml), or
+[environment_dlc.yml](environment_dlc.yml) for software dependencies.
 
-See [spec-file.txt](https://github.com/LorenFrankLab/spyglass-demo/blob/main/spec-file/spec-file.txt) for the conda environment used in the demo.
+See
+[spec-file.txt](https://github.com/LorenFrankLab/spyglass-demo/blob/main/spec-file/spec-file.txt)
+for the conda environment used in the demo.
 
 ## Citation
 
-> Lee, K.H.\*, Denovellis, E.L.\*, Ly, R., Magland, J., Soules, J., Comrie, A.E., Gramling, D.P., Guidera, J.A., Nevers, R., Adenekan, P., Brozdowski, C., Bray, S., Monroe, E., Bak, J.H., Coulter, M.E., Sun, X., Broyles, E., Shin, D., Chiang, S., Holobetz, C., Tritt, A., Rübel, O., Nguyen, T., Yatsenko, D., Chu, J., Kemere, C., Garcia, S., Buccino, A., Frank, L.M., 2024. Spyglass: a data analysis framework for reproducible and shareable neuroscience research. bioRxiv. [10.1101/2024.01.25.577295](https://doi.org/10.1101/2024.01.25.577295).
+> Lee, K.H.\*, Denovellis, E.L.\*, Ly, R., Magland, J., Soules, J., Comrie,
+> A.E., Gramling, D.P., Guidera, J.A., Nevers, R., Adenekan, P., Brozdowski, C.,
+> Bray, S., Monroe, E., Bak, J.H., Coulter, M.E., Sun, X., Broyles, E., Shin,
+> D., Chiang, S., Holobetz, C., Tritt, A., Rübel, O., Nguyen, T., Yatsenko, D.,
+> Chu, J., Kemere, C., Garcia, S., Buccino, A., Frank, L.M., 2024. Spyglass: a
+> data analysis framework for reproducible and shareable neuroscience research.
+> bioRxiv.
+> [10.1101/2024.01.25.577295](https://doi.org/10.1101/2024.01.25.577295).
 
 *\* Equal contribution*
 

docs/README.md

@@ -16,11 +16,9 @@ The remainder of `mkdocs.yml` specifies the site's
 ## GitHub
 
 Whenever a new tag is pushed, GitHub actions will run
-`.github/workflows/publish-docs.yml`. Progress can be monitored in the 'Actions'
-tab within the repo.
-
-Releases should be tagged with `X.Y.Z`. A tag to redeploy docs should use the
-current version, with an alpha release suffix, e.g. `X.Y.Za1`.
+`.github/workflows/publish-docs.yml`. From the repository, select the Actions
+tab, and then the 'Publish Docs' workflow on the left to monitor progress. The
+process can also be manually triggered by selecting 'Run workflow' on the right.
 
 To deploy on your own fork without a tag, follow turn on github pages in
 settings, following a `documentation` branch, and then push to `test_branch`.
@@ -47,7 +45,7 @@ the root notebooks directory may not be reflected when rebuilding.
 Use a browser to navigate to `localhost:8000/` to inspect the site. For
 auto-reload of markdown files during development, use
 `mkdocs serve -f ./docs/mkdosc.yaml`. The `mike` package used in the build
-script manages versioning, but does not support dynamic versioning.
+script manages versioning, but does not support dynamic reloading.
 
 The following items can be commented out in `mkdocs.yml` to reduce build time:
 

docs/build-docs.sh

@@ -14,12 +14,25 @@ mv ./docs/src/notebooks/README.md ./docs/src/notebooks/index.md
 cp -r ./notebook-images ./docs/src/notebooks/
 cp -r ./notebook-images ./docs/src/
 
-if [ -z "$MAJOR_VERSION" ]; then # Get version from file
-  version_line=$(grep "__version__ =" ./src/spyglass/_version.py)
-  version_string=$(echo "$version_line" | awk -F"[\"']" '{print $2}')
+# Function for checking major version format: #.#
+check_format() {
+    local version="$1"
+    if [[ $version =~ ^[0-9]+\.[0-9]+$ ]]; then
+        return 0
+    else
+        return 1
+    fi
+}
+
+# Check if the MAJOR_VERSION not defined or does not meet format criteria
+if [ -z "$MAJOR_VERSION" ] || ! check_format "$MAJOR_VERSION"; then
+  full_version=$(git describe --tags --abbrev=0)
   export MAJOR_VERSION="${version_string:0:3}"
 fi
-echo "$MAJOR_VERSION" # May be available as env var
+if ! check_format "$MAJOR_VERSION"; then
+  export MAJOR_VERSION="dev" # Fallback to dev if still not valid
+fi
+echo "$MAJOR_VERSION"
 
 # Get ahead of errors
 export JUPYTER_PLATFORM_DIRS=1

docs/mkdocs.yml

@@ -42,15 +42,14 @@ theme:
 
 nav:
   - Home: index.md
-  - Installation: installation.md
   - Tutorials:
       - Overview: notebooks/index.md
       - Intro:
           - Setup: notebooks/00_Setup.ipynb
-          - Insert Data: notebooks/01_Insert_Data.ipynb
-          - Data Sync: notebooks/02_Data_Sync.ipynb
-          - Merge Tables: notebooks/03_Merge_Tables.ipynb
-          - Config Populate: notebooks/04_PopulateConfigFile.ipynb
+          - Concepts: notebooks/01_Concepts.ipynb
+          - Insert Data: notebooks/02_Insert_Data.ipynb
+          - Data Sync: notebooks/03_Data_Sync.ipynb
+          - Merge Tables: notebooks/04_Merge_Tables.ipynb
           - Export: notebooks/05_Export.ipynb
       - Spikes:
           - Spike Sorting V0: notebooks/10_Spike_SortingV0.ipynb
@@ -70,18 +69,22 @@ nav:
           - Decoding Clusterless: notebooks/41_Decoding_Clusterless.ipynb
           - Decoding Sorted Spikes: notebooks/42_Decoding_SortedSpikes.ipynb
       - MUA Detection: notebooks/50_MUA_Detection.ipynb
-  - Miscellaneous:
-      - Overview: misc/index.md
-      - Common Errors: misc/common_errs.md
-      - Database Management: misc/database_management.md
-      - Export: misc/export.md
-      - FigURL: misc/figurl_views.md
-      - Insert Data: misc/insert_data.md
-      - Merge Tables: misc/merge_tables.md
-      - Mixin: misc/mixin.md
-      - Session Groups: misc/session_groups.md
+  - Features:
+    - Overview: Features/index.md
+    - FigURL: Features/FigURL.md
+    - Merge Tables: Features/Merge.md
+    - Export: Features/Export.md
+    - Session Groups: Features/SessionGroups.md
+    - Centralized Code: Features/Mixin.md
+  - For Developers:
+    - Overview: ForDevelopers/index.md
+    - How to Contribute: ForDevelopers/Contribute.md
+    - Database Management: ForDevelopers/Management.md
+    - Code Reuse: ForDevelopers/Reuse.md
+    - Table Types: ForDevelopers/TableTypes.md
+    - Understanding a Schema: ForDevelopers/Schema.md
+    - Using NWB: ForDevelopers/UsingNWB.md
   - API Reference: api/ # defer to gen-files + literate-nav
-  - How to Contribute: contribute.md
   - Change Log: CHANGELOG.md
   - Copyright: LICENSE.md
 

docs/src/misc/mixin.md → docs/src/Features/Mixin.md

@@ -6,7 +6,7 @@ functionalities that have been added to DataJoint tables. This includes...
 - Fetching NWB files
 - Long-distance restrictions.
 - Delete functionality, including permission checks and part/master pairs
-- Export logging. See [export doc](export.md) for more information.
+- Export logging. See [export doc](./Export.md) for more information.
 
 To add this functionality to your own tables, simply inherit from the mixin:
 
@@ -53,8 +53,8 @@ to a `_nwb_table` attribute.
 In complicated pipelines like Spyglass, there are often tables that 'bury' their
 foreign keys as secondary keys. This is done to avoid having to pass a long list
 of foreign keys through the pipeline, potentially hitting SQL limits (see also
-[Merge Tables](./merge_tables.md)). This burrying makes it difficult to restrict
-a given table by familiar attributes.
+[Merge Tables](./Merge.md)). This burrying makes it difficult to restrict a
+given table by familiar attributes.
 
 Spyglass provides a function, `restrict_by`, to handle this. The function takes
 your restriction and checks parents/children until the restriction can be
@@ -122,7 +122,7 @@ If the user shares a lab team with the session experimenter, the deletion is
 permitted.
 
 This is not secure system and is not a replacement for database backups (see
-[database management](./database_management.md)). A user could readily
+[database management](../ForDevelopers/Management.md)). A user could readily
 curcumvent the default permission checks by adding themselves to the relevant
 team or removing the mixin from the class declaration. However, it provides a
 reasonable level of security for the average user.
@@ -134,11 +134,11 @@ entry without deleting the corresponding master. This is useful for enforcing
 the custom of adding/removing all parts of a master at once and avoids orphaned
 masters, or null entry masters without matching data.
 
-For [Merge tables](./merge_tables.md), this is a significant problem. If a user
-wants to delete all entries associated with a given session, she must find all
-part table entries, including Merge tables, and delete them in the correct
-order. The mixin provides a function, `delete_downstream_parts`, to handle this,
-which is run by default when calling `delete`.
+For [Merge tables](./Merge.md), this is a significant problem. If a user wants
+to delete all entries associated with a given session, she must find all part
+table entries, including Merge tables, and delete them in the correct order. The
+mixin provides a function, `delete_downstream_parts`, to handle this, which is
+run by default when calling `delete`.
 
 `delete_downstream_parts`, also aliased as `ddp`, identifies all part tables
 with foreign key references downstream of where it is called. If `dry_run=True`,

docs/src/Features/index.md

@@ -0,0 +1,12 @@
+# Features
+
+This directory contains a series of explainers on tools that have been added to
+Spyglass.
+
+- [Export](./Export.md) - How to export an analysis.
+- [FigURL](./FigURL.md) - How to use FigURL to share figures.
+- [Merge Tables](./Merge.md) - Tables for pipeline versioning.
+- [Mixin](./Mixin.md) - Spyglass-specific functionalities to DataJoint tables,
+    including fetching NWB files, long-distance restrictions, and permission
+    checks on delete operations.
+- [Session Groups](./SessionGroups.md) - How to operate on sets of sessions.