controllers.md

Choosing the Right Controller

While developing the simulation a need arose for the optimal control of the grippers. For a 2 finger gripper this is a rudimentary task, since by activating/closing both DoFs of the gripper the grasp is performed correctly. Nonetheless, with a more complex gripper (a 3-5 fingers) the closing routine is not as straight forward. Consequently, a "closing mask" was implemented to be used in tandem with custom gripper controllers to control different grippers in the same simulation. First, the "close_dir" of every gripper was defined by the visualization of the grippers within the simulation. The close_dir is just a n-dimesional vector stating the direction a DoF of the gripper must move to in order to close the gripper.

isaac_sim_grasping/managers.py

Lines 103 to 115 in 4e96519

	self.close_dir= {
	"fetch_gripper" : [1,1],
	"franka_panda": [-1, -1], # NOTE, franka_panda gripper by default is closed, so need to open before, Opendir = [1, 1]
	"sawyer": [1, 1],
	"wsg_50": [1, -1], # NOTE, wsg_50 gripper by default is closed, so need to open before, Opendir = [-1, 1]
	"Barrett": [0, 0, 1, 1, 1, 0, 0, 0],
	"jaco_robot": [1, 1, 1],
	"robotiq_3finger": [0, 0, 1, 1, 1, 1, 1, 1, 0, 0, 0],
	"Allegro": [0, 0.5, 0, 0, 1, 0, 1, 1, 0, 1, 0, 0, 0, 0, 0, 0],
	"HumanHand": [0, 0, 0, 0, 0, 1, 1, 1, 1, 0.75, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
	"shadow_hand": [0, 0, 0, 0, 0, 1, 0, 1, 1, 0, 1, 1, 1, 1, 0, 0, 1, 0, 0, 1, 0, 1],
	"h5_hand": [1,-1,0,0]
	}

For example, the fetch gripper, being a 2 finger gripper, must only move boths of its DoFs to the positive direction to perform the grasp. On the other hand, a more complex gripper, such as the Allegro gripper, has multiple DoFs that remain static and just a handfull of DoFs that move to close the gripper; we denoted the moving DoFs as "driving DoFs". It is worth noting that this is a very subjective way of creating a closing routine. Many manufacturers provide different closing routines for the different types of objects the gripper may grasp, for example the Robotiq 3-finger gripper which has multiple grasping routines, such as the a fingertip grip or an encompassing grip. Nonetheless, a simple controller can be a good starting point for the grasping tests. In the future, more controllers could be developed without necessarily using the closing masks.

Available controllers

For our simulation, 2 controllers were developed as starting points. One using the position control of the grippers and another using a force control approach. All controllers take as input the same parameters at initialization and during simulation (forward function):

Initialization:

• close_mask: close direction of the DoFs
• test_time: total time of the tests
• max_efforts: max effort(force/torque) of every DoF
• robots: ArticulationView of all the grippers within the simulation Note: every controller must have is own name (Isaac Sim Convention) and the self.type with a description of the controller used (Saved to output file).

Forward function:

• action: Isaac Sim convention (string describing if it is a closing or opening action)
• time: current time since the test started.
• current_dofs: DoF values of the robot.
• close_position: Final DoF values that describe the closed position of the gripper. It is calculated within view.py view.post_reset() function using the close_mask.

isaac_sim_grasping/views.py

Lines 97 to 116 in 4e96519

	# Get max efforts and dofs
	dc = self.world.dc_interface
	articulation = dc.get_articulation(self.work_path+"/gripper")
	self.dof_props = dc.get_articulation_dof_properties(articulation)
	self.close_positions = np.zeros_like(self.dofs)
	max_efforts = np.zeros_like(self.dofs)
	close_mask = self.manager.close_mask
	#print(self.dof_props)
	for i in range(len(self.dof_props)):
	if (close_mask[i]==0):
	max_efforts[:,i] = self.dof_props[i][6]
	self.close_positions[:,i]=(self.dofs[:,i])
	elif (close_mask[i]>0):
	max_efforts[:,i]= self.dof_props[i][6]
	self.close_positions[:,i]=(self.dof_props[i][3])
	elif (close_mask[i]<0):
	max_efforts[:,i]= -self.dof_props[i][6]
	self.close_positions[:,i]=(self.dof_props[i][2])
	else:
	raise ValueError("clos_dir arrays for grippers can only have 1,-1 and 0 values indicating closing direction")

Note: For new controllers the reference must be added to the controller_dict.

isaac_sim_grasping/controllers.py

Lines 164 to 169 in 4e96519

	controller_dict = {
	'default': PositionController,
	'force' : ForceController,
	'position': PositionController,
	'transfer_default': TransferPositionController
	}

Position-based Controller

isaac_sim_grasping/controllers.py

Lines 54 to 100 in 4e96519

	class PositionController(BaseGripperController):
	""" Position Controller Version 0
	Args:
	close_mask: dofs directions to close grippers
	test_time: total test time
	max_effors: max_effort for Gripper "driving dofs"
	robots: ArticulationView for Robots in simulation
	"""
	def __init__(self, close_mask, test_time, max_efforts, robots):
	name = "Controller"
	super().__init__(name=name)
	self.type = 'position_controller_v0'
	self.effort = max_efforts
	self.close_mask = close_mask
	return
	def close(self,time):
	return
	def open(self,time):
	return
	def forward(self, action, time, grippers, close_position):
	current_dofs = grippers.get_joint_positions()
	pos = np.zeros_like(close_position)
	for i in range(len(self.close_mask)):
	if (self.close_mask[i]==0):
	pos[:,i] = close_position[:,i]
	else:
	pos[:,i] = close_position[:,i] * abs(self.close_mask[i])
	if action == "h5_hand":
	pos[:,2]= -1*current_dofs[:,0]
	pos[:,3]= -1*current_dofs[:,1]
	tmp= pos[:,2:]
	grippers.set_joint_positions(tmp, joint_indices = [2,3])
	actions = ArticulationActions(joint_positions = pos)
	# A controller has to return an ArticulationAction
	self.last_actions = actions
	#print(pos)
	return actions

The simplest of both controllers is the position based controller denominated as "PositionController" and accessible in the simulation with the "position" keyword. The controller simply gets the "close_position" and transforms it to an ArticulationActions object for the use in Isaac Sim. The resulting behavior is the movement of the gripper DoF to the final "closed position" denoted by the close_mask and the DoF range.

Force-based Controller

isaac_sim_grasping/controllers.py

Lines 10 to 52 in 4e96519

	class ForceController(BaseGripperController):
	""" Force Controller Version 0
	Args:
	close_mask: dofs directions to close grippers
	test_time: total test time
	max_effors: max_effort for Gripper "driving dofs"
	robots: ArticulationView for Robots in simulation
	"""
	def __init__(self, close_mask, test_time, max_efforts,robots):
	name = "Controller"
	super().__init__(name=name)
	self.close_mask = close_mask
	self.type = 'force_control_v0'
	self.total_time = test_time
	ind = np.squeeze(np.argwhere(max_efforts[0]!=0))
	robots.set_max_efforts(np.zeros_like(ind),joint_indices=ind.reshape((1,len(ind))))
	self.max_efforts = max_efforts
	return
	def close(self,time):
	joint_command = ((self.total_time-2*time)/self.total_time)*self.max_efforts
	#print(joint_command)
	return joint_command
	def open(self,time):
	return
	def forward(self, action, time, current_dofs, close_position):
	command = self.close(time)
	new_pos = np.zeros_like(close_position)
	for i in range(len(self.close_mask)):
	if self.close_mask[i] == 0:
	new_pos[:,i] = close_position[:,i]
	else:
	new_pos[:,i] = current_dofs[:,i]
	actions = ArticulationActions(joint_positions = new_pos, joint_efforts=command)
	# A controller has to return an ArticulationAction
	self.last_actions = actions
	return actions

The force controller is denominated as "ForceController" and accessible in the simulation with the "force" keyword. The controller works by exerting forces in the driving DoFs of the gripper. The exerted forces are constantly decreasing from 100% of the max DoF effort (described in the robot .usd) to -100% of the max effort. In simple words, for the first half of the test the gripper is closing with a decreasing amount of effort and for the last half the gripper will be opening with an increasing amount of effort. Note that for the proper exertion of the forces in the DoFs, they first needed to be disabled by setting their max effort as 0 and then applying the ArticulationActions manually through the controller.

Position-based Controller for Transferred Grasps

isaac_sim_grasping/controllers.py

Lines 103 to 157 in 4e96519

	class TransferPositionController(BaseGripperController):
	""" Transfer Position Controller Version 0
	Args:
	close_mask: dofs directions to close grippers
	test_time: total test time
	max_effors: max_effort for Gripper "driving dofs"
	robots: ArticulationView for Robots in simulation
	"""
	def __init__(self, close_mask, test_time, max_efforts, robots):
	name = "Controller"
	super().__init__(name=name)
	self.type = 'transfer_position_controller_v0'
	self.effort = max_efforts
	self.close_mask = close_mask
	self.touch_dofs = np.zeros_like(max_efforts)
	return
	def close(self,time):
	return
	def open(self,time):
	return
	def forward(self, action, time, grippers, close_position):
	current_dofs = grippers.get_joint_positions()
	pos = np.zeros_like(close_position)
	time = np.squeeze(time)
	uninit = np.argwhere(time==0)
	init = np.argwhere(time!=0)
	self.touch_dofs[uninit]= current_dofs[uninit]
	#2 behaviors ready and not ready grasps (view.py gives 0 as time for each workstation that is not set up)
	for i in range(len(self.close_mask)):
	if (self.close_mask[i]== 0):
	pos[:,i] = close_position[:,i]
	elif ((abs(self.close_mask[i])-1)==0):
	if(len(init)>0):
	pos[init,i] = self.touch_dofs[init,i]
	if(len(uninit)>0):
	pos[uninit,i] = close_position[uninit,i]
	else:
	pos[:,i] = close_position[:,i] * (abs(self.close_mask[i])-1)
	if action == "h5_hand":
	pos[:,2]= -1*current_dofs[:,0]
	pos[:,3]= -1*current_dofs[:,1]
	grippers.set_joint_positions(pos[:,2:], joint_indices = [2,3])
	actions = ArticulationActions(joint_positions = pos)
	# A controller has to return an ArticulationAction
	self.last_actions = actions
	return actions

Our simulation was able to evaluate the object fall-off time for a large amount of generated grasps. The successful grasps of one gripper can represent successful grasps in others and increase the overall amount of successful grasps in the dataset. To test this hypothesis, we implemented the grasp transfer of successful grasps from one gripper to others and evaluated the transferred grasps using Isaac Sim. We first utilize the alignment between grippers to transfer grasps. We transform a source gripper pose to its aligned pose, and then transform the aligned pose to the target gripper. To transfer a grasp from a source gripper to a target gripper, the joint configuration of the source gripper cannot easily be mapped to the target gripper. Therefore, we first set the joint configuration of the target gripper to the open configuration. Then the controller closes all fingers until it comes into contact with the object. After that, the controller set the root joints of the fingers to reach their closing position. The gripper performs a closing motion. The joint values at the moment of contact between the gripper and object are recorded, and the object fall-off time is simulated.