MRG: add generic support for any type of sketch collection as query or database

[ctb]

This PR adds MultiCollection, a wrapper around multiple Collection structs, as an implementation for many important features, including loading of standalone manifests and zip files from pathlists. As part of this it also adds direct sketch loading from RocksDB/RevIndex.

This PR:

• adds more docs/comments
• fixes support standalone manifests containing zip files #266
• fixes recommend (and test) 'standalone manifests' #235
• fixes multisearch continues past "no query signatures loaded, exiting" message #280
• adds more general support for loading from RocksDB

It does, however, break some functionality in index, because RevIndexes with external storage cannot be created from multiple CollectionSets, so we had to use specialized loading code.

Note also that there is a bug around loading multisketch files from pathlists in (see #445); this PR does not adjust the code to deal with this.

TODO:

• initial benchmarks of a few situations - GTDB as normal, but also "picklist"-style manifest subsetting
• benchmark manysearch as well (see 10-fold manysearch performance slowdown between v0.9.5 and v0.9.6 #463, 'unreleased with 430 and 464')
• figure out fastmultigather memory usage reason
• update docs to mention the bug in WIP: debug multisigfile test #445
• update/merge/fix reliance on unreleased sourmash core code (see: MRG: add Manifest::intersect_manifest to Rust core sourmash#3305)
• update docs in re standalone manifests as well as RocksDB (see: MRG: documentation updates based on new collection loading #444)
• maybe: test manifests-containing-pathlists-containing ... chain all in one?
• figure out exactly what index+external storage works on 😆
• resolve @ctb comments
• refactor picklist code a bit
• fix test_fastgather.py::test_indexed_against
• fix MultiCollection / SmallSignature struct to not clone Collections each time
• add tests for pathlists containing standalone manifests
• fix index to take more/better inputs

From #436 -

• add tests for standalone manifests containing zip files
• add tests for pathlists containing zip files
• add tests for fastgather loading a query sketch from RocksDB
• add tests for fastgather loading against sketches from RocksDB
• add tests for multisearch loading query & against sketches from RocksDB
• add tests for pairwise loading sketches from RocksDB
• add check for warning about loading all sketches from a RocksDB

From #437 -

• regularize the code for multisearch error exit/better reporting, add tests, etc.

Punting to issues, to be created before merge:

• systematic benchmarks of picklist stuff
• fix test_fastgather.py::test_against_multisigfile (see WIP: debug multisigfile test #445)
• consider supporting env_logger, perhaps via features?
• add tests for relative paths in pathlists
• add tests for relative paths in manifests ref sourmash behavior
• write issue in sourmash talking about luiz point about traits vs structs for Collection and Storage: consider how to support more flexible Collection in RevIndex for external storage sourmash#3321

[ctb]

The benchmarks in #463 show that loading is not significantly slower nor is more memory used, with this PR - if anything, less memory and faster loading, which could make sense given the changes - here is a screenshot of the benchmarks as of 5380325, which includes the fixes for #463 in #464.

The only remaining big blocker for merging this PR is the memory performance of fastmultigather, seen above in #430 (comment). I'm going to rerun this to verify the bad behavior before tackling a fix ;).

[ctb]

Updated benchmarks as of 2563b0b, via code in sourmash-bio/sourmash#3232. (Previous benchmarks for this PR here: #430 (comment))

Indexing:

GTDB rs214 took 3 hr 10m, and 14.3 GB of RAM. The index is 19 GB in size.

• This is a decrease in time from the last release, by more than an hour - wow!
• This is a large increase in size, presumably due to using internal storage

For SRR1976948, we see:

prefix	s	max_rss
fastmultigather_rocksdb	151.352	590.32
fastgather	166.86	8287.86
fastmultigather	467.033	25236.7
pygather	2708.01	13832.3

The times for this PR are slower by a bit from , but not a huge amount.

fastmultigather still requires 25 GB. Sigh. I think that's the last remaining blocker for merge.

but! fastgather memory has decreased by about 1/3, probably due to sourmash-bio/sourmash#3342!