Migration of Cineast Features

[ppanopticon]

Task Description

As discussed during our last meeting, I have removed the 'migration' features from the code base. The purpose of this issue is to discuss and decide, which of the legacy Cineast features should be kept and migrated and to migrate them eventually. I simply start with a list of all the features that have had a migration feature:

Keep / Migrate

• CLD
• EHD
• HueHistogram
• MedianColor (demonstrator, like AverageColor)

Add to FES (@faberf)

• SkeletonPose (demonstrator)
• VisualTextCoEmbedding

Drop / Remove

• AverageColorGrid8
• AverageColorGrid8Reduced15 (basically an existing feature with a pre-processing step)
• AverageFuzzyHist
• AverageFuzzyHistNormalized
• DominantColor
• DominantEdgeGrid
• DominantEdgeGrid8
• DominantEdgeGrid16
• EdgeARP88
• EdgeGrid16
• InceptionResNetV
• MedianFuzzyHist
• CLDReduced15
• ConceptMasksADE20
• HOGMF25k512 (obsolete)
• OpenCLIP (better option available)
• ProvidedOCR (not needed as a feature, create an import script)
• SURFMF25K512 (obsolete)
• WhisperASR (better option available)

I have already singled-out some of the feature I would probably drop. Maybe @lucaro can weigh-in on the others and describe what they bring to the table.

Dependencies

None

Boundary Conditions

Only features that are useful should be migrated (e.g., in terms for functionality or didactically to illustrate how features can be built). No need to keep legacy features that bring no advantage.

[lucaro]

Interestingly enough, some of the ones you already crossed out are among the ones I would have considered to be more interesting to keep. I think it would be good to have some interest-point bag-of-word-based feature support, as there are specific use cases, such as sub-image or near-duplicate matching, where they are still very useful. It is debatable what exact method for interest-point detection and description should be employed.
Similarly, I think the pose thing is an interesting demonstrator. I don't know what you take as a baseline for the claim of sub-par performance, as there isn't really a whole lot to compare it to.

For the other ones, everything that ends in 'Reduced' does not need to be a dedicated feature, as it was a version of another feature that applied a color reduction operation first. While this can be useful for doing things like aligning color palettes for, e.g., sketch-based retrieval, this can now be a dedicated pre-processing step.
The 'VisualTextCoEmbedding' and the 'InceptionResNetV2' are just dense models that could be added to the feature extraction server and do not need any additional support on the engine side of things.
Of the others, I'd probably keep the CLD and EHD, mostly because they are part of the MPEG7 standard and can be somewhat useful for low-level visual description and maybe the HueHistogram, not because it is particularly useful for retrieval but because it can be nice for exploration and visualization.
The other ones might have their uses in very specific use cases that probably nobody has anymore, so I don't think it would be a huge loss to not have them anymore. If anyone ever has such a use case, adding stuff back in is always an option.

[ppanopticon]

Thanks for the comment. I agree on your point on the BoW features for near-duplicate matching. However, don't think those two features are particularly useful. If anything, I'd add something new for this purpose. Talking about near-duplicate detection: Didn't we once have a Master's Thesis dealing with this topic?

What about DominantColor? Having information about dominant colors in an image could certainly be useful, even in the future.

[ppanopticon]

Ohh and one remark regarding SkeletonPose (which I moved to the list of migrated features). I'm not sure this will be compatible with the new PgVector database backend.

[lucaro]

Regarding the dominant color, I guess what would be more useful is to have this as a label rather than a vector. So, what I would see as useful is defining a palette of colors and having a feature that assigns one (or none) of these. The Cineast DominantColor feature did produce a vector with the most dominant color and then used knn on it, that's probably less useful.

For the SkeletonPose: yes, this will probably not be easily supported by the pgVector connection. I foresee that this would require a new type of query object that has to be supported by the connection. Even if the feature by itself is probably only useful in some somewhat narrow use cases, it might still be interesting to have, if only as a demonstration, of how custom querying logic can be supported in a connection.

[lucaro]

Regarding the ConceptMaskADE20k feature, that one was used for query by semantic sketch. I guess having such a query mode back would also be interesting in the medium term. From the implementation side of things, I'd, however, rather use something like https://github.com/apple/ml-4m/ in the backend.

[ppanopticon]

I'd suggest to open separate issues for new feature requests.

Regarding DominantColor: I agree with your assessment. Is this something that can be derived using the existing feature? Or is this something completely different.

[lucaro]

It's probably easier to build this freshly than trying to adapt the old one. I guess one would have to first have to come up with a concise definition of a dominant color in this context, as color is not as straightforward as one might think 😉. I can take care of this at some point.

[ppanopticon]

I see! So I've updated the TODO-List - which is quite short at this point.

@faberf Any chance you might be able to add these features to the FES in the near future?

[ppanopticon]

Closed

Opened #100 and #101 for migration of SkeletonPose and VisualTextCoEmbedding.