sensor fusion between 2da lidar and realsense camera

[hardikbvaghela]

hello
i want to ask you about the sensor fusion how can i fuse both camera and lidar and get the obstacle distance and angle measurement on my screen?

[palanc]

Hi @hardikbvaghela - Here are all of our tutorials: https://mushr.io/tutorials/

And here are some that might be specifically relevant to your question:
https://mushr.io/tutorials/realsense/
https://mushr.io/tutorials/mapping/

[hardikbvaghela]

Thank you so much for your response. but I want to know if it's a 2D ydlidar. so it gives a 2d lidar point cloud. but if want to detect the pedestrian with this lidar so how can I detect it because it gives the same color point clouds surrounding the environment. so how can I detect pedestrians with this 2dlidar? I can attach an image for my Rviz. please guide me.

[schmittlema]

yes it is a 2D lidar. Lidar returns distances not classifications. The color on the pointcloud is just hardcoded for visualization. You ideally would use the camera to detect people. For example like this. Then you can use lidar (or more easily the depth image) to get the position of the person. If you want to detect people soley based on lidar there exist models for that but it is more tricky/difficult especially with 2dlidar

[schmittlema]

Hi hardikvaghela. This will take a bit of work. First you will need to classify obstacles in the image space. Most people use some neural network for this. Assuming you have a good classification using the RGB of the camera, you can use the depth image to get associated depth of the obstacles. So if some pixel is classified as a wall, the same pixel in the depth image should give you the depth of the wall. Then you can convert them to 3D points using the following

For each point in the laserscan there is a distance and associated angle. You can convert that to x, y points via the following polar --> cartesian equations
x = r cos(theta)
y = r sin(theta)
Where r is the distance of the scan point and theta is the angle.