Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[features] Fix ShapeContext3DEstimation #6050

Open
mvieth opened this issue May 22, 2024 · 4 comments
Open

[features] Fix ShapeContext3DEstimation #6050

mvieth opened this issue May 22, 2024 · 4 comments
Labels
good first issue Skills/areas of expertise needed to tackle the issue help wanted kind: bug Type of issue module: features

Comments

@mvieth
Copy link
Member

mvieth commented May 22, 2024

Currently, the x-axis (and consequently also the y-axis) are chosen randomly, which makes this feature useless without further processing.
The paper by Frome et al says in section 2.1:

We have a degree of freedom in the azimuth direction that we must remove in
order to compare shape contexts calculated in different coordinate systems. To
account for this, we choose some direction to be Φ0 in an initial shape context,
and then rotate the shape context about its north pole into L positions, such
that each Φl division is located at the original 0 position in one of the rotations.
For descriptor data sets derived from our reference scans, L rotations for each
basis point are included, whereas in the query data sets, we include only one
position per basis point

It seems like the PCL implementation is missing this additional step. There is a so far unimplemented function, seemingly for this purpose: https://github.com/PointCloudLibrary/pcl/blob/master/features/include/pcl/features/3dsc.h#L224 The task is to implement this function and call it in computeFeature, after computePoint.

When the implementation is fixed, the test in test_shot_estimation.cpp probably has to be updated. Ideally, a new test should be added where this feature is computed twice on the same cloud, and the outputs should be very similar.

@mvieth mvieth added kind: bug Type of issue good first issue Skills/areas of expertise needed to tackle the issue help wanted module: features labels May 22, 2024
@kai-waang
Copy link
Contributor

The task is to implement this function and call it in computeFeature, after computePoint.

computeFeature is expected to return a PointCloud<pcl::ShapeContext1980>, but if we use shiftAlongAzimuth to do the rotation, then as described in the comment, the desc will be a vector at a length of descriptor_length_ * azimuth_bins_ (1980 * 12 = 23760 at default).

As a result, every point in the output cloud that supposed to store the feature at a size 1980 may overflow. So how to deal with this?

@mvieth
Copy link
Member Author

mvieth commented Aug 12, 2024

Maybe shiftAlongAzimuth was originally intended to be used in a different way. I think the size should be unchanged (1980).

@kai-waang
Copy link
Contributor

In this commit, shiftAlongAzimuth was first introduced and called in computeFeature. The final descriptor was accepted by either a pcl::SHOT or a pcl::ShapeContext, which has now been deleted and replaced (see this). In this commit, shiftAlongAzimuth was deleted due to some bugs and is no longer called from computeFeature.

The pcl::SHOT or pcl::ShapeContext struct has a descriptor field of type std::vector, but now pcl::ShapeContext1980 only contains a fixed-sized array.

A possible solution is to add a new point type to store point-wise features(1980x12) for pcl::ShapeContext1980, acting as a descriptor dataset. But this seems not the best solution.

Ideally, a new test should be added where this feature is computed twice on the same cloud, and the outputs should be very similar.

Even if we implement the shift feature, determining if two features are very similar is challenging since rotation does not guarantee the order of L descriptors. Matching a query to an existing dataset requires comparing all L descriptors. One approach to measure similarity is by calculating the distance between two features; for example, we could calculate the minimum distances from a query to features from two runs and check if they are close enough.

@mvieth
Copy link
Member Author

mvieth commented Aug 16, 2024

Okay, this seems to be not as straightforward as I initially thought. I also had not checked how shiftAlongAzimuth was implemented and used in the past.

There are some assumptions that feature implementations have to meet in PCL to work well with other PCL classes (e.g. registration methods):

  • feature similarity is usually measured by (euclidean) distance (as you also mentioned). This is especially the case when kdtrees are used to find features similar to a query feature. So a new 1980x12 feature type with a custom similarity function is not really possible
  • many classes assume a one-to-one correspondence between points in the cloud and computed features. That means we can't make computeFeature return 12 times pcl::ShapeContext1980 (different rotations) for every input point.
  • there are probably more, but these are the two which I could think of that are most relevant for this problem

By the way, the 3D Shape Context feature has an extension, the Unique Shape Context (USC, https://dl.acm.org/doi/10.1145/1877808.1877821). That one does not use random x- and y-axes, but computes a reference frame instead.

I think the best solution for the 3D Shape Context would be to deviate slightly from the paper, and not do L rotations, but just do one specific rotation on the result of computePoint. That rotation should be done so that the feature has a certain property, which would result in a small euclidean distance of two features computed on the same point with two different random x-axes. One idea could be to rotate such that the bin with the highest value in the feature is always in a specific azimuth direction. Then the initial, random x-axis would not matter any more.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
good first issue Skills/areas of expertise needed to tackle the issue help wanted kind: bug Type of issue module: features
Projects
None yet
Development

No branches or pull requests

2 participants