[python] Review TileDB-SOMA for spec compliance

[atolopko-czi]

Once TileDB-SOMA RC0 is code-complete, review the TileDB-SOMA public API to ensure compliance with the SOMA abstract specification w.r.t class names, class properties, and method names , and method signatures.

The review team is:
@atolopko-czi , @mlin , @thetorpedodog , @aaronwolen

• Any discrepancies found can simply be noted on this issue as comments. @atolopko-czi will coalesce into one or more issues for any fixes that may be needed.
• Review discrepancies with the larger team if it is not clear whether it is the implementation or the spec that should be corrected.
• Python-perspective review & impl fixes, as needed (@atolopko-czi)
  - [ ] R-perspective review & impl fixes, as needed (@aaronwolen)

[atolopko-czi]

@johnkerl @@thetorpedodog We still have many "⚠️" notes in the spec. Do we need to address these before performing a spec compliance check or proceed as is?

• "Issue: are there parts of the Arrow type system that we wish to explicitly exclude from SOMA?"
• "Issue: it would be a good idea to factor SOMAExperiment and SOMAMeasurement into separate sections."
• "[SOMADataFrame] To be further specified: all methods need specification"
• "[SOMADenseNDArray] To be further specified: this is incomplete."
• "[Operation: read()] Issue: support for other formats, such as Arrow Table, is under discussion."
• "[General Utilities] To be further specified"
• "[Value Filters] To be further specified"
• "[Other Issues (open issues with this doc)] Issues to be resolved: ..."

[atolopko-czi]

Tracking review progress and action items here.

[johnkerl]

cc @bkmartinjr

[atolopko-czi]

@thetorpedodog @mlin Found and fixed some discrepancies in spec in single-cell-data/SOMA#145, but looking for feedback on how to handle the following additional discrepancies:

• Collection.add_new_collection: cls param: cls (impl) vs type (spec) [use kind in both places, consistently throughout]
• Should spec mention closed and mode properties? (e.g. it does mention context property) [ignore]
• DataFrame.write: values param only admits pa.Table (impl) vs spec admits pa.Table or pa.RecordBatch [issue to add to impl; also for SparseNDArray]

[bkmartinjr]

Regarding the above open issues - my general attitude is we should move away from inline "issue" tracking and start using GH issues (to capture discussion, etc).

On the specific items listed above:

"Issue: are there parts of the Arrow type system that we wish to explicitly exclude from SOMA?"

I believe this has been resolved by the decision to have create methods raise an error when an unsupported type is requested (i.e., where the underlying storage engine is unable or unwilling to support a given schema).

Propose we remove this issue from the document.

"Issue: it would be a good idea to factor SOMAExperiment and SOMAMeasurement into separate sections."

As this is editorial, and non-critical, it could be removed. At most, it belongs in a low-priority issue on the repo.

"[SOMADataFrame] To be further specified: all methods need specification"

I believe all methods do have at lease a cursory specification, so this can be removed.

"[SOMADenseNDArray] To be further specified: this is incomplete."

Ditto

"[Operation: read()] Issue: support for other formats, such as Arrow Table, is under discussion."

These are largely resolved, so I would remove and any new discussion should be directed to a GH issue.

"[General Utilities] To be further specified"

remove

"[Value Filters] To be further specified"

remove

"[Other Issues (open issues with this doc)] Issues to be resolved: ..."

item (1) seems resolved.

items (2) and (3) could be moved to a GH issue and this section could be removed.

[atolopko-czi]

Regarding

@johnkerl @@thetorpedodog We still have many "⚠️" notes in the spec. Do we need to address these before performing a spec compliance check or proceed as is?

and

Regarding the above open issues - my general attitude is we should move away from inline "issue" tracking and start using GH issues (to capture discussion, etc).

Removed 'em all, creating follow-up github issues for 2 specifically:

• Abstract spec: Factor SOMAExperiment and SOMAMeasurement into separate sections SOMA#147
• Support value filters in NDArray read operations SOMA#148

[maniarathi]

Couple more discrepancies I noted -- the SOMACollection spec says that has() and iterator are available operations, but this is currently not true. I am not sure whether there is a plan to add these in the future or not, but right now it will output a has no attribute error if those methods are called on a Collection object.

[atolopko-czi]

Couple more discrepancies I noted -- the SOMACollection spec says that has() and iterator are available operations, but this is currently not true. I am not sure whether there is a plan to add these in the future or not, but right now it will output a has no attribute error if those methods are called on a Collection object.

These are handled by the Collection's Python dunder methods, __iter__() and __contains__(), which work:

In [13]: [k for k in c]
Out[13]: ['census_info', 'census_data']

In [14]: 'census_info' in c
Out[14]: True

So while Collection does not provide an explicit has() or iterator() method, I think we're covered by these Python idiomatic patterns.

[atolopko-czi]

Split off the R-related portion of the issue to #1062