Monocular-only localization of iphone generated map

[bill-healey]

I am trying to use a relatively cheap monocular drone camera to localize using an iphone generated map.

I've calibrated the monocular camera, set Rtab-map to localization only, and disabled odometry, and set 3D to 2D (PnP) motion estimation. RTAB-Map is able to load the iphone database and show the point cloud, but I'm unable to get the moncular camera to localize within the map. It looks like loop detection is running but it cannot find a match.

Has anyone been able to do this successfully, and are there settings I should change to make the match more likely even if potentially incorrect? I tried reducing the minimum correspondences to 6 but still wasn't able to get localization to work.

[matlabbe]

If the field of view and resolution are different, you may have to use visual features more robust to scale/shear, like SURF or SIFT (or even SuperPoint). If you built rtabmap with nonfree and xfeatures2d modules from opencv, you should be able to try:

# SURF
Kp/DetectorStrategy: 0
Vis/FeatureType: 0

# SIFT
Kp/DetectorStrategy: 1
Vis/FeatureType: 1

To convert your database taken from iPhone to use one of this feature, you can do:

rtabmap-reprocess --Kp/DetectorStrategy 1 --Vis/FeatureType 1 rtabmap.db output.db

[bill-healey]

Great, that definitely helped and I am now getting some localizations!

The monocular camera is lower resolution and has a wider (more fisheye) lens. Any other settings/reprocessing that you would recommend experimenting with?

[matlabbe]

Depending if you are using SURF or SIFT, they have also other parameters that can be tuned (to generate more features on low resolution for example):

rtabmap --params | grep SIFT
Param: SIFT/ContrastThreshold = "0.04"                     [The contrast threshold used to filter out weak features in semi-uniform (low-contrast) regions. The larger the threshold, the less features are produced by the detector.]
Param: SIFT/EdgeThreshold = "10"                           [The threshold used to filter out edge-like features. Note that the its meaning is different from the contrastThreshold, i.e. the larger the edgeThreshold, the less features are filtered out (more features are retained).]
Param: SIFT/NFeatures = "0"                                [The number of best features to retain. The features are ranked by their scores (measured in SIFT algorithm as the local contrast).]
Param: SIFT/NOctaveLayers = "3"                            [The number of layers in each octave. 3 is the value used in D. Lowe paper. The number of octaves is computed automatically from the image resolution.]
Param: SIFT/RootSIFT = "false"                             [Apply RootSIFT normalization of the descriptors.]
Param: SIFT/Sigma = "1.6"                                  [The sigma of the Gaussian applied to the input image at the octave #0. If your image is captured with a weak camera with soft lenses, you might want to reduce the number.]

rtabmap --params | grep SURF
Param: SURF/Extended = "false"                             [Extended descriptor flag (true - use extended 128-element descriptors; false - use 64-element descriptors).]
Param: SURF/GpuKeypointsRatio = "0.01"                     [Used with SURF GPU.]
Param: SURF/GpuVersion = "false"                           [GPU-SURF: Use GPU version of SURF. This option is enabled only if OpenCV is built with CUDA and GPUs are detected.]
Param: SURF/HessianThreshold = "500"                       [Threshold for hessian keypoint detector used in SURF.]
Param: SURF/OctaveLayers = "2"                             [Number of octave layers within each octave.]
Param: SURF/Octaves = "4"                                  [Number of pyramid octaves the keypoint detector will use.]
Param: SURF/Upright = "false"                              [Up-right or rotated features flag (true - do not compute orientation of features; false - compute orientation).]

[NikolasE]

I'm testing something similar, could you maybe share your parameters or launch file?

[bill-healey]

While I was able to get a some localizations, it unfortunately wasn't enough to really be usable. It was also very slow with my settings and wasn't able to keep up anywhere near realtime even with my desktop PC/GPU doing the processing. I think a better approach might be to use one of the many neural depth projects that infer depth from a non-depth RGB image then use RGBD for SLAM.

The RTAB-Map iphone app is also great at both localization and mapping due to all the ARKit sensors and RTAB's loop closure so I'm also considering just using that directly but it does require mounting the phone on the robot.