Registering a thermal image with the depth image of the intel realsense D415

[MCR100]

Hi everyone.

For one of my project, I want to align/register a thermal image (From FLIR camera) of an object onto a depth image of the same object acquired using D415. I want to do it using the extrinsic camera parameters without using the scene features (independent of scene structure). Is it possible to utilize rs.align or is there another ways?

[MartyG-RealSense]

Hi @MCR100 #10363 should be a useful reference for aligning between a thermal camera and a RealSense camera, using the RealSense SDK's software-device interface.

[MCR100]

Hi @MartyG-RealSense , thank you for the reply. I almost followed the same method. I have calculated the extrinsic parameters between the two cameras using stereo rectification using the chessboard as described. But when I try to overlap after performing the stereorectification (https://amroamroamro.github.io/mexopencv/matlab/cv.stereoRectify.html) on both the images, they are not aligning correctly. Showing some disparity. Am I doing anything wrong?

[MartyG-RealSense]

When aligning, are you using the RealSense SDK's align_to instruction like rs-align does?

If you are using align_to, are you using the intrinsics of the stream that you are aligning to? If you are aligning depth to the thermal image then you should use the thermal intrinsics. If you are aligning depth to thermal and you use the depth intrinsics then it may result in an inaccurate aligned image.

[MCR100]

Hi @MartyG-RealSense . Thank you for the reply, sorry it took a while to get back. still couldn't figure out the method. So using checkerboard I was able to calculate the intrinsic parameters, the relative rotation and translation between the intelrealsense and thermal camera. Using the stereocalibration method I have rectified the thermal image. To put it in simple words, Using this method I can align the thermal image with the RGB image captured using intelrealsense D415 without much deviation. My question is how do I align the thermal image with the depth image now. Since i already aligned thermal with intelrealsense RGB, is it possible to use rs-align? I am working on python.

[MartyG-RealSense]

Hi @MCR100 It's great to hear that you made significant progress!

rs-align is a C++ language example, though the Python example align_depth2color.py uses the same alignment mechanism (align_to) as rs-align.

https://github.com/IntelRealSense/librealsense/blob/master/wrappers/python/examples/align-depth2color.py

I would think that in order to use your thermal-RGB merged image as the color frame, you would have to define a color_frame that uses that merged image instead of the original thermal-less RGB image.

https://github.com/IntelRealSense/librealsense/blob/master/wrappers/python/examples/align-depth2color.py#L75

[MCR100]

Thanks @MartyG-RealSense for the quick response. Yes as you said I have to define a "color_frame" with the thermal image.

Yes I am using the wrapper rs_align. Given below is the part of the code.

#------------------ENABLING_STREAM-----------------------------------------------
config.enable_stream(rs.stream.depth, 640, 480, rs.format.z16, 30)
config.enable_stream(rs.stream.color, 640, 480, rs.format.bgr8, 30)

align_to = rs.stream.color
align = rs.align(align_to)

pipeline.start(config)

#------------------ACQUIRING_THE_FRAME-----------------------------------------------
for x in range(5):
pipeline.wait_for_frames()

frames = pipeline.wait_for_frames()

#------------------ Align the depth frame to color frame------------------
frames = align.process(frames)

so before processing the alignment (align.process(frames)), I have to replace the color image with thermal image. something like frames.color_frame.set_data(thermal_data) Do you have any references that would help me to show how this modification can be done? As I have already aligned the thermal with intel RGB, doesn't require to modify the intrinsics or extrinsic
Appreciate your help.

[MartyG-RealSense]

There was a case at #5093 (comment) where instead of aligning two streams into one and then aligning that with another, alignment was defined for the three frame types separately and then they were all combined together by an aligned_frames = align.process(frames) instruction.

[MCR100]

Thanks @MartyG-RealSense for the reply. I am going to try that method and see. Regarding the frames, currently thermal image is there only as a data (numpy. ndarray). How to convert that into a class frame?

[MartyG-RealSense]

Other RealSense users have previously attempted to convert a numpy array to rs2::frame but have not succeeded.

cv::mat to rs2::frame is feasible, as advised by a RealSense team member at #2634 (though the scripting is C++).

[MCR100]

@MartyG-RealSense , is there any way to replace the color data [frames.get_color_frame().get_data()] with my values. Thats all I actually wanted and that would make things less complex.

[MartyG-RealSense]

The software-device interface would be the best way to import custom data instead of using an SDK stream, but it is best suited to C++ rather than Python.

Would it be be possible to perform a normal depth to color alignment (so that the depth stream field of view size changes to match the color field of view size) and then align the thermal image to the color image afterwards after the depth-color alignment has been performed?

[MCR100]

Hi @MartyG-RealSense, thanks again for the suggestions.

Yes aligning the depth to color and then thermal to the aligned color is possible, but the calibration between the thermal and the color will become bit complex and I have to redo the calibration process I have done previously.

However, I have used the software_device interface in python to replace the color image data and created thermal color frame. But as Arvid2 Nilsson mentioned in https://support.intelrealsense.com/hc/en-us/community/posts/1500000934242-exploring-equivalent-API-in-python-for-rs2-software-device-create-matcher-RS2-MATCHER-DLR-C-defined-in-c-, can only inject and get one stream at a time through the syncher. So I have the color(now thermal) and depth as separate frames.

Is it possible to create a composite frameset using those two?. I have tried #5847, but the processing_block is not getting invoked. below is the merging part of my code,

def bundler_callback(f, src):
global fs
print('bundler function called')
bundle.push_back(f)
if len(bundle) == 2:
fs = src.frame_source.allocate_composite_frame(bundle)
src.frame_ready(fs)
bundle.clear()

bundle =list()
bundler = rs.processing_block(bundler_callback)
q = rs.frame_queue()

pipe = rs.pipeline()
conf = rs.config()
conf.enable_stream(rs.stream.depth, 640, 480, rs.format.z16, 30)
conf.enable_stream(rs.stream.color, 640, 480, rs.format.bgr8, 30)
pipe.start(conf,bundler_callback)

bundler.start(q)

bundler.invoke(updated_color_frame)
bundler.invoke(updated_depth_frame)

fs = q.wait_for_frame()

pipe.stop()

[MartyG-RealSense]

I note that in your 4th line you are using bundle.push_back(f), whilst the original script at #5847 (comment) uses an append instruction: bundle.append(f)

[MCR100]

@MartyG-RealSense , yes I tried with bundle.append(f) too. The issue is processing_block is not getting invoked, if so as mentioned in #5847, the print call (print('bundler function called')) would have printed or atleast an error would have generated. Nothing happened. Am I missing something?

Is there any references other than #5847 and #5417 where custom_processing block has been used in python?
Or
Is there any alternative way to create a composite frame using two separate frames?

Thanks

[MartyG-RealSense]

I could not locate any further references for custom processing block in Python either, unfortunately.

I am not sure if this will be relevant to your particular situation, but a RealSense user at #6358 attempted to read depth and color composite frames into a list with frame appending and then performed alignment on them.

[MCR100]

@MartyG-RealSense No in the example you have given, the user is extracting the data after alignment. So basically two options are there and two of them have issues in python.
1 - Software device : Works but only for one stream at a time
2- Custom processing block: Couldn't invoke the block

So if I can one get one reference where one of the above completely works with python and give a composite frame then it would be solved but couldn't find any. Any other suggestions?

[MartyG-RealSense]

The C++ example for software_device that uses syncer can handle two stream types simultaneously. Logically the same should apply for Python, though my knowledge of syncer is admittedly limited.

https://github.com/IntelRealSense/librealsense/blob/master/examples/software-device/rs-software-device.cpp#L150-L153

https://github.com/IntelRealSense/librealsense/blob/master/examples/software-device/rs-software-device.cpp#L186-L188

[MCR100]

@MartyG-RealSense . But both of them have issues as I mentioned above. Using custom processing block #5847 (comment) is the easiest and shortest way compared to software device .

As mentioned in #5847 (comment) and #9900 (comment). The invoke is not working in python. I have been trying for the past month. Is it possible to get the help from your programming team or @dorodnic . Implementing that custom processing will save a huge chunk of my work.

[MartyG-RealSense]

I have highlighted to my Intel RealSense colleagues your Python bundler.invoke problem and the previous one at #9900 (comment) to seek their advice.

[MCR100]

Thank you @MartyG-RealSense, looking forward to your reply.

[MartyG-RealSense]

Hi @MCR100 One of my colleagues has provided a response to your issue.

---

If I understand correctly, the customer already aligned the thermal with Intel RGB, so it can be simplified by aligning depth to Intel RGB as well.

https://github.com/IntelRealSense/librealsense/blob/master/wrappers/python/examples/align-depth2color.py#L74

The depth frame is aligned to RGB with the same resolution and viewpoint as the Intel RGB, pixel to pixel matched.

aligned_depth_frame = aligned_frames.get_depth_frame()

and the Intel RGB image is unchanged during the alignment in this case:

color_frame = aligned_frames.get_color_frame()

So at this point, if the thermal is aligned with the Intel RGB, with the same resolution and same viewpoint as Intel RGB, then
the aligned depth frame should also be aligned with the aligned thermal. No additional alignment is needed.

[MartyG-RealSense]

You are very welcome. It's great news that you achieved a solution. Thanks so much for the update and for sharing your solution with the RealSense community. As you suggested, I will close the issue. Thanks again!