Skip to content

Latest commit

 

History

History
executable file
·
196 lines (123 loc) · 7.19 KB

README.md

File metadata and controls

executable file
·
196 lines (123 loc) · 7.19 KB

UnrealEngine4 Wrapper for RealSense SDK 2.0

alt text

Level with examples demonstrating streaming depth, color (IR and external RGB) and pointcloud in Unreal Engine 4.

Overview

UnrealEngine4 wrapper shows how to use RealSense features via Blueprints (UE4 version 4.19+). At the moment it provides next functionality:

  • enumerate devices/sensors/options/profiles
  • load and save configuration presets
  • tune sensor options
  • access live sensor streams via textures
  • record and playback sensor streams
  • process raw data in shader
  • align depth/color streams
  • generate and render point cloud mesh

Getting Started

Download and install RealSense SDK 2.0

Download wrapper and generate project files for RealSenseUE.uproject

Copy C:\Program Files (x86)\Intel RealSense SDK 2.0\bin\x64\realsense2.dll to either:

  • The engine bin folder C:\Program Files\Epic Games\UE_4.19\Engine\Binaries\Win64\ OR
  • The plugin build folder RealSense-UE4-Wrapper\Plugins\RealSense\Binaries\Win64\

In case RealSense SDK was installed in custom folder:

  • edit file RealSense-UE4-Wrapper\Plugins\RealSense\Source\RealSense\RealSense.Build.cs
  • update line string RealSenseDirectory = "REALSENSE_SDK_PATH_HERE"

Build RealSenseUE.sln using DevelopmentEditor configuration

Start and play RealSenseUE.uproject to see some live data

Next steps

Check out prefab BP_RealSenseAPI, it shows functionality exposed to Blueprints and a few examples:

  • ExEnumerateHardware - get connected devices/sensors and print some info
  • ExModifyOption - modify specific sensor option
  • ExPresetLoadSave - load and save device configuration preset
  • ExStreamConfiguration - enable/disable streams, set video mode
  • ExRecordPlayback - record and playback demo

Check out prefab BP_RealSenseInspector, it simplifies stream configuration and management, also binds live data to scene objects. BP_RealSenseAPI should be placed to scene as well if new one is being created.

Hardware inspector

Hardware inspector allows to check connected hardware and tune sensor options via editor. This feature is for debugging only and changes are not saved automatically. It's possible to load and save configuration presets using corresponding buttons in hardware inspector dropdown or via blueprint functions.

Preset files can be downloaded here.

alt text

Picture: hardware inspector

Device configuration

DeviceSerial allows to select specific device by serial. If empty default device will be used.

PipelineMode switches between CaptureOnly/Record/Playback:

  • CaptureOnly - fetch and render live data
  • Record - fetch and render live data but also record it to specified file
  • Playback - load previously recorded file and loop it content

EnablePolling switches poll/wait update mode. Poll mode is checking for data with specified rate, wait mode is blocking until data is available.

alt text

Picture: mode inspector

Stream configuration

Each stream has it's own dropdown with enable/disable flag and video mode selector. These options should be configured before starting RealSenseInspector. Restarting is required to change video mode. Enable/disable flag can be used to pause/resume already running stream.

Depth can provide two textures: raw and colorized. Colorized depth is soft-processed based on Equalize/MinMax/Colormap settings. Avoid using colorized depth on high resolutions/rates.

In case running both depth and infrared they should be configured to same video mode.

alt text

Picture: depth stream inspector

Point cloud configuration

Point cloud support is experimental and has a lot of flaws. Current implementation uses third-party plugin for dynamic mesh generation and should be optimized.

  • PclScale - allows to scale the whole cloud
  • PclVoxelSize - single point/voxel size in mm
  • PclDensity - percentage of rendered points
  • PclRenderRate - how often the PCL mesh is updated (0.5 means two times per second)

alt text

Picture: point cloud inspector

Blueprint API

RealSenseContext is global object initialized at plugin load. It wraps native RealSense library and provides device management functions.

alt text

Picture: get context blueprint

After getting the context devices can be enumerated. Each device contains a few sensors, each sensor has some options. Sensor profile contains video mode and frame rate.

alt text

Picture: enumerate devices blueprint

alt text

Picture: enumerate sensors blueprint

alt text

Picture: enumerate options blueprint

Sensor options can be tuned one by one or using preset load/save functions.

alt text

Picture: get and set options value

alt text

Picture: save and load preset blueprint

RealSenseInspector provides stream configuration and management functions. It updates textures and PCL mesh.

alt text

Picture: find Inspector blueprint

Properties described in StreamConfiguration section also can be modified from Blueprints.

alt text

Picture: modify stream mode blueprint

After configuration RealSenseInspector should be started with any PipelineMode.

alt text

Picture: set mode blueprint

Time to get texture objects and PCL mesh.

alt text

Picture: get stream texture blueprint

alt text

Picture: transform PCL mesh blueprint

Native code

It's also possible to access wrapper from native code, check RealSenseTestActor.cpp for example:

#include "RealSensePlugin.h"
#include "RealSenseContext.h"
#include "RealSenseDevice.h"
#include "RealSenseNative.h"

ARealSenseTestActor::ARealSenseTestActor()
{
	auto Context = IRealSensePlugin::Get().GetContext();
	Context->QueryDevices();
	for (auto Device : Context->Devices)
	{
		UE_LOG(LogRealSenseDemo, Display, TEXT("Wrapper device %s"), *(Device->Name));
	}

	rs2::context_ref RsContext(Context->GetHandle());
	auto DeviceList = RsContext.query_devices();
	for (auto Device : DeviceList)
	{
		FString DevName(ANSI_TO_TCHAR(Device.get_info(RS2_CAMERA_INFO_NAME)));
		UE_LOG(LogRealSenseDemo, Display, TEXT("Native device %s"), *DevName);
	}
}

Known Issues

  • Frame drop/timeout on high resolutions might be laggy (depends on the hosts hardware)
  • High CPU usage (RealSenseSDK issue)
  • ColorizedDepth is CPU bound
  • PointCloud (PCL) has sometimes texture corruptions at thhe egdes (RealSenseSDK issue)
  • PointCloud (PCL) update/rendering performance is somtime slow
  • rebuilding project files can sometimes fail after switching Unreal Engine versions. Cleaning the solution or Deleting the intermediate directory should fix this.

License

This project is licensed under the Apache License, Version 2.0.