Optimizing Get Frame Time for LiDAR L515

[littlefred117]

Hi,

So I am trying to optimize the time it takes for the LiDAR L515 to get the depth frames. In my current code, it takes around 1.92 seconds to get the depth frames (how long it takes for the first "done" message to be printed):

• import pyrealsense2 as rs
  import numpy as np

Initialize the RealSense pipeline

pipeline = rs.pipeline()
config = rs.config()
config.enable_stream(rs.stream.depth, 320, 240, rs.format.z16, 30)

Start the pipeline with the given configuration

pipeline.start(config)

framecount = 0
accumulated_points = None

try:
while framecount < 2:
# Wait for a frame
frames = pipeline.wait_for_frames()

    # Get depth and color frames
    depth_frame = frames.get_depth_frame()
    
    

    if not depth_frame:
        continue

    # Get the raw depth and color data as numpy arrays
    depth_data = np.asanyarray(depth_frame.get_data())
    

    # Generate the point cloud from the depth data
    points = rs.pointcloud()
    depth_frame = depth_frame.as_depth_frame()
    points.map_to(depth_frame)
    point_cloud = points.calculate(depth_frame)
    

    # Get the point cloud data as a numpy array
    point_cloud_data = np.asanyarray(point_cloud.get_vertices())


    # Save the point cloud data to a file
    np.savetxt('point_cloud2.xyz', point_cloud_data, delimiter=' ')
    print("done")

    #Increase frame count
    framecount += 1

finally:
# Stop the pipeline
pipeline.stop()
print("finished")

However, I read on this datasheet,
Intel_RealSense_LiDAR_L515_Datasheet_Rev003 (1).pdf, on page 6 that you can "a short exposure time of <100ns per depth
point, even rapidly moving objects can be captured with minimal motion blur". I am considering adding something like this to my code, in order to shorten the time:

depth_sensor = profile.get_device().first_depth_sensor()
depth_sensor.set_option(rs.option.exposure, 100) # Set the exposure time in microseconds
depth_sensor.set_option(rs.option.gain, 16) # Set the gain value
depth_sensor.set_option(rs.option.laser_power, 16) # Set the laser power

However, I am getting issues with adding these options, so I was wondering if anyone knew how to properly implement this in order to optimize time. Thanks

[MartyG-RealSense]

Hi @littlefred117 The L515 does not have the Gain option that the 400 Series cameras have. Instead, it has Receiver Gain, which can be controlled with Python code like this:

depth_sensor.set_option(rs.option.receiver_gain, 16)

L515 does not have depth exposure controls, though it does have exposure control for its RGB. L515's depth has a short exposure time of less than 100ns per depth point

In the absence of depth exposure controls, the laser_power and receiver_gain options are the primary method of adjusting how well the camera performs in a particular environment or when scanning particular types of objects. For example, the Short Range Visual Preset camera configuration works well in a lot of situations by lowering both the laser_power and receiver_gain values. Your script instruction for laser_power is correct.

---

The official L515 user guide, available at the link below as a downloadable PDF document, has details about the effect that each L515 preset has. Plese see pages 9 and 10 of the guide.

https://support.intelrealsense.com/hc/en-us/article_attachments/360081846174

[littlefred117]

Hello, I am currently using pyrealsense2 and depth_sensor.set_option(rs.option.receiver_gain, 16) is not a supported function. What should I use in pyrealsense2 to get a similar result?

[MartyG-RealSense]

Hi @littlefred117 If you are using the L515 camera model then rs.option.receiver_gain should be supported. It is an L515-only option.

Have you defined depth_sensor in your script, please?

depth_sensor = profile.get_device().first_depth_sensor()
depth_sensor.set_option(rs.option.receiver_gain, 16)

[littlefred117]

Hi, so I ended up following your advice and changed my code. Right now my "processing time" is around 0.85-0.91 seconds. I am wondering if there are any other methods I could use to get the time down even more.

Code:
`import pyrealsense2 as rs
import numpy as np
import matplotlib.pyplot as plt
import time

pipeline = rs.pipeline()
config = rs.config()

pipeline_wrapper = rs.pipeline_wrapper(pipeline)
pipeline_profile = config.resolve(pipeline_wrapper)
device = pipeline_profile.get_device()

config.enable_stream(rs.stream.depth, rs.format.z16, 30)
#config.enable_stream(rs.stream.color, rs.format.bgr8, 30)

Start streaming

pipeline.start(config)

Get stream profile and camera intrinsics

profile = pipeline.get_active_profile()

Set laser power, gain, and confidence

sensor_dep = profile.get_device().first_depth_sensor()
sensor_dep.set_option(rs.option.laser_power, 50)
sensor_dep.set_option(rs.option.receiver_gain, 18)
sensor_dep.set_option(rs.option.confidence_threshold, 3)

Get camera intrinsics

depth_profile = rs.video_stream_profile(profile.get_stream(rs.stream.depth))
depth_intrinsics = depth_profile.get_intrinsics()
w, h = depth_intrinsics.width, depth_intrinsics.height

Processing blocks

pc = rs.pointcloud()
decimate = rs.decimation_filter()
decimate.set_option(rs.option.filter_magnitude, 2 ** 1)
#colorizer = rs.colorizer()

start_time = time.time()

frames = pipeline.wait_for_frames()

end_time = time.time()

print(end_time-start_time)

depth_frame = frames.get_depth_frame()
#color_frame = frames.get_color_frame()

depth_frame = decimate.process(depth_frame)

Grab new intrinsics (may be changed by decimation)

depth_intrinsics = rs.video_stream_profile(
depth_frame.profile).get_intrinsics()
w, h = depth_intrinsics.width, depth_intrinsics.height

depth_image = np.asanyarray(depth_frame.get_data())
#color_image = np.asanyarray(color_frame.get_data())

#depth_colormap = np.asanyarray(colorizer.colorize(depth_frame).get_data())

#mapped_frame, color_source = color_frame, color_image

points = pc.calculate(depth_frame)
v = points.get_vertices()
verts = np.asanyarray(v).view(np.float32).reshape(-1, 3) # xyz

fig = plt.figure()
ax = fig.add_subplot(projection='3d')

ax.scatter(verts[:,0],verts[:,1],verts[:,2])
plt.show()`

[MartyG-RealSense]

You could test whether disabling the decimation post-processing filter inproves performance. Post-processing filters are calculated on the computer and not the camera hardware, so a filter will impose a processing burden on the CPU.

[littlefred117]

Ok, by post processing do you mean the decimate in this code block:

Processing blocks

pc = rs.pointcloud()
decimate = rs.decimation_filter()
decimate.set_option(rs.option.filter_magnitude, 2 ** 1)
#colorizer = rs.colorizer()

start_time = time.time()

frames = pipeline.wait_for_frames()

end_time = time.time()

print(end_time-start_time)

depth_frame = frames.get_depth_frame()
#color_frame = frames.get_color_frame()

depth_frame = decimate.process(depth_frame)

Grab new intrinsics (may be changed by decimation)

depth_intrinsics = rs.video_stream_profile(
depth_frame.profile).get_intrinsics()
w, h = depth_intrinsics.width, depth_intrinsics.height

[MartyG-RealSense]

Yes, I mean the decimate instructions in the code block.