RealRobot

Real robot interface to connect with ManiSkill2

Runs camera capturing and visualization as separate processes to make it closer to using ROS

❗ This repo is still under heavy development, so API might be changed without notice

Installation

From PyPI

python3 -m pip install -U real-robot

If you need to install the optional xarm dependency, you must install from GitHub repo

python3 -m pip install -U real-robot[xarm]@git+https://github.com/KolinGuo/RealRobot.git

---

Calibrated camera poses are stored in real_robot/assets/hec_camera_poses/ and loaded in real_robot/sensors/camera.py.

Helpful Tools

• Capture Color/Depth/IR images from RS camera, do

  python3 -m real_robot.tools.rs_capture

• Get detailed device information of connected RealSense device (e.g., supported stream configs, sensor intrinsics, extrinsics between sensors), do

  python3 -m real_robot.tools.enumerate_rs_devices > info.log

• Convert recorded ROS bag file and save rgb & depth images into .npz, do

  python3 -m real_robot.tools.convert_rosbag_to_frames <bag_path>

Notes

To fix an unindexed rosbag recorded from RSDevice, do

apt install -y python3-rosbag
rosbag reindex <bag_path>

Known Issue

• When controlling xArm7 with motion_mode="cartesian_online" and wait=True, sometimes the robot will be stuck indefinitely in wait_move() due to get_state() returns wrong state (i.e., returns state=1 thinking it's still in motion). Simple solution can be just to control briefly via UFACTORY Studio to get it unstuck.

Changelog

0.1.0

New features

• Added SharedObject to create/mount objects stored in SharedMemory
• Enabled RSDevice to run as a separate process (now Camera will create RSDevice as a separate process)
• Enabled RSDevice to record camera streams as a rosbag file
• Enabled XArm7 to run as a separate process (for streaming robot states)
• Enabled CV2Visualizer and O3DGUIVisualizer to run as separate processes (for visualization)
• Added a default FileHandler to all Logger created through real_robot.utils.logger.get_logger
• Allow enabling selected camera streams from RSDevice and sensors.camera.Camera
• Added sensors.simsense_depth.SimsenseDepth class to generate depth image from stereo IR images

API changes

• real_robot.agents.xarm
  • Change XArm7 parameters for clarity (safety_boundary => safety_boundary_mm, boundary_clip_eps => boundary_clip_mm)
  • Add get_gripper_position() to get gripper opening width in mm or m
  • Add gripper_speed parameter to set_action() to control gripper speed
• real_robot.utils.visualization.visualizer
  • Rename Visualizer method show_observation() => show_obs()
• real_robot.sensors.camera
  • CameraConfig now accepts a config parameter
  • Rename CameraConfig parameter parent_pose_fn => parent_pose_so_name
• real_robot.utils.realsense
  • RSDevice now accepts device_sn instead of an rs.device
  • RSDevice now accepts config as parameter (width, height, fps) instead of rs.config

Other changes

• Switch from gym to gymnasium
• Rename all control_mode by removing pd_ prefix for clarity. No PD controller is used.
• real_robot.agents.xarm
  • XArm7 will not clear "Safety Boundary Limit" error automatically in set_action()
  • For motion_mode == "position", switch from using set_tool_position() to set_position()
  • Enable gripper and set to maximum speed in reset()
• Remove all Loggers created as global variables (they will be created at import, which might not be saved under REAL_ROBOT_LOG_DIR)
• Bugfix in xArm-Python-SDK: enable wait=True for modes other than position mode

0.0.2

• Added motion_mode to XArm7 agent
• Added several control_mode: pd_ee_pos, pd_ee_pose_axangle, pd_ee_delta_pose_axangle, pd_ee_pose_quat, pd_ee_delta_pose_quat