More robust delete_downstream_merge

[CBroz1]

Description

Now leveraging the underlying datajoint dependency on networkx to find the shortest pipeline paths
between a given table and various merges, or Session and a given table, for delete_downstream_merge and check_permissions in SpyglassMixin. By running a join on all tables across two points, each of these funcs should be more robust to edge cases.

Initial drafts proved to be very slow, building these chains across arbitrary pipeline points by hand. Even with networkx method, I opted to cache connections to improve speed when folks assign tables (e.g., from spy import Table; t=Table(); t.delete_downstream)

• Fixes Cautious delete restriction #791 : Join on all tables between target and merges.
  • common/common_usage.py : proposed new usage tracking function to monitor how long the cautious delete process takes, and where is it is used from
  • dj_merge_tables: Migrate delete_downstream_merge out into the mixin to cache calculated links better
  • dj_mixin.py: add cache of merge tables found and pipeline connections I dubbed 'chains' from self to merge tables. In practice, the number of merge tables I'm able to find depends on whether or not the user has imported them. The reload_cache flag allows a user to see a blocking merge part, import the relevant table, and then reload the cache
  • Future versions could intercept the datajoint delete error and load these tables.
  • Edited docs and notebooks to reflect new functionality
• Ran jupytext on notebook edits from previous PRs
• New black version 24.0 means a lot of minor changes elsewhere.

Checklist:

• This PR should be accompanied by a release: Maybe not this one, but soon?
• (If release) I have updated the CITATION.cff
• I have updated the CHANGELOG.md
• I have added/edited docs/notebooks to reflect the changes

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[edeno]

There are still some linter issues but otherwise looks good.

[edeno]

Join on all tables between target and merges

Is this potentially expensive with a large database?

[CBroz1]

Join on all tables between target and merges

Is this potentially expensive with a large database?

It is, yes. Using networkx is a huge speed improvement over doing the search myself to find the chain, but the join process could be expensive. With narrow restrictions on the parent, I found it pretty quick to run the join, especially with the cached chain. My new usage table is designed to monitor how this gets used, and how long it takes. If still cumbersome, I can look into replacing TableChain.join's python join process with something that would be more SQL-native

[samuelbray32]

Tested in the lab database for conditions that were problems before and they looked good. Thanks @CBroz1 !

Cases:

• downstream table doesn't contain original restrictions
• projection of key name between tables