TypeError: expected Tensor as element 0 in argument 0, but got NoneType

[YuhaiW]

Hello!
When I run the quick demo:
cd SeqDex/dexteroushand
python train_rlgames.py --task BlockAssemblyGUI --num_envs=1 --play

there has an error:

num_object_shapes:  958 num_object_bodies:  138 num_object_shapes:  959 num_object_bodies:  139 
num_object_shapes:  960 num_object_bodies:  140 num_object_shapes:  961 num_object_bodies:  141 
num_object_shapes:  962 num_object_bodies:  142 num_object_shapes:  963 num_object_bodies:  143 
num_object_shapes:  964 num_object_bodies:  144 num_object_shapes:  965 num_object_bodies:  145 
num_object_shapes:  966 num_object_bodies:  146 Started convex decomposition for mesh '../assets/urdf/blender/assets_for_insertion/urdf/../origin_obj/8x8x1_real/8x8x1_real.stl' Finished convex decomposition for mesh '../assets/urdf/blender/assets_for_insertion/urdf/../origin_obj/8x8x1_real/8x8x1_real.stl': 183 hulls num_extra_lego_asset_shapes:  1149 num_extra_lego_asset_bodies:  147 [Warning] [carb.gym.plugin] User aggregate size exceeded [Error] [carb.gym.plugin] cudaExternamMemoryGetMappedBuffer failed on rgbImage buffer with error 101 [Error] [carb.gym.plugin] cudaExternamMemoryGetMappedBuffer failed on depthImage buffer with error 101 [Error] [carb.gym.plugin] cudaExternamMemoryGetMappedBuffer failed on segmentationImage buffer with error 101 [Error] [carb.gym.plugin] cudaExternamMemoryGetMappedBuffer failed on optical flow buffer with error 101 *** Can't create empty tensor *** Can't create empty tensor mount_rigid_body_index:  7 [Error] [carb.gym.plugin] cudaExternamMemoryGetMappedBuffer failed on rgbImage buffer with error 101 [Error] [carb.gym.plugin] cudaExternamMemoryGetMappedBuffer failed on depthImage buffer with error 101 [Error] [carb.gym.plugin] cudaExternamMemoryGetMappedBuffer failed on segmentationImage buffer with error 101 [Error] [carb.gym.plugin] cudaExternamMemoryGetMappedBuffer failed on optical flow buffer with error 101 *** Can't create empty tensor [Error] [carb.gym.plugin] cudaExternamMemoryGetMappedBuffer failed on rgbImage buffer with error 101 [Error] [carb.gym.plugin] cudaExternamMemoryGetMappedBuffer failed on depthImage buffer with error 101 [Error] [carb.gym.plugin] cudaExternamMemoryGetMappedBuffer failed on segmentationImage buffer with error 101 [Error] [carb.gym.plugin] cudaExternamMemoryGetMappedBuffer failed on optical flow buffer with error 101 *** Can't create empty tensor *** Can't create empty tensor Traceback (most recent call last): File "train_rlgames.py", line 75, in <module> task, env = parse_task(args, cfg, cfg_train, sim_params, agent_index) File "/home/yuhai/SeqDex/dexteroushandenvs/utils/parse_task.py", line 166, in parse_task task = eval(args.task)( File "/home/yuhai/SeqDex/dexteroushandenvs/tasks/block_assembly/allegro_hand_block_assembly_gui.py", line 187, in __init__ super().__init__(cfg=self.cfg, enable_camera_sensors=self.enable_camera_sensors) File
 "/home/yuhai/SeqDex/dexteroushandenvs/tasks/hand_base/base_task.py", line 83, in __init__ self.create_sim() File 
"/home/yuhai/SeqDex/dexteroushandenvs/tasks/block_assembly/allegro_hand_block_assembly_gui.py", line 498, in create_sim self._create_envs(self.num_envs, self.cfg["env"]['envSpacing'], int(np.sqrt(self.num_envs))) File 
"/home/yuhai/SeqDex/dexteroushandenvs/tasks/block_assembly/allegro_hand_block_assembly_gui.py", line 1301, in _create_envs self.camera_rgbd_image_tensors = torch.stack(self.camera_tensors, 
dim=0).view(self.num_envs, -1) TypeError: expected Tensor as element 0 in argument 0, but got NoneType

My machine info is there:

I asked ChatGPT,
and here is the answer
User aggregate size exceeded: This warning suggests that the simulation is attempting to allocate more resources than allowed or available. This might be a configuration issue or a limitation of the system.

Can't create empty tensor: This error is likely a result of the previous CUDA errors. Since the GPU memory buffers could not be created, the tensors that are supposed to hold this data cannot be formed, hence the "empty tensor" error.

TypeError: The final error in the traceback shows that there was an attempt to stack a list of tensors, but at least one element in the list was None instead of a tensor object. This is consistent with the previous errors because if the GPU buffers failed to initialize, then the expected tensors would not be created, leading to this error when attempting to stack them.```

To address these errors, consider the following steps:
```Check System Resources: Ensure that your system has enough resources (RAM, GPU memory) to allocate for the simulation.
Review Configuration: Look into the configuration of the simulation (in this case, the cfg object) to ensure that the resource limits are set correctly.
Update GPU Drivers: Make sure your GPU drivers are up to date and compatible with the version of CUDA you're using.
CUDA Compatibility: Verify that the GPU is compatible with the CUDA version installed and that PyTorch (or whichever framework you're using) supports that CUDA version.
Restart the System: Sometimes, simply restarting the system can clear up any issues with GPU memory allocation.
Error Handling: Modify the code to handle the case where tensors could not be created, potentially by adding checks before trying to stack them.```

[j96w]

Hi! This error may be due to the desktop does not have a monitor, so it can't create a camera buffer, and then the camera handle will be Nonetype. It needs a monitor to use IsaacGym's GUI, could you take a try with that?

Hope this can help you.

[YuhaiW]

Hi,
Thank you for your response!
But we do have a monitor, and when I run the sample from Isaacgymenv, there is no error.
here is how I run Isaacgymenv's example:
(gym) yuhai@cappuccino:~/IsaacGymEnvs/isaacgymenvs$ python train.py task=Cartpole wandb_activate=True wandb_entity=yuhaiwan wandb_project=issac-gym seed=42

[YuhaiW]

I followed the installation step
git clone https://github.com/sequential-dexterity/SeqDex.git
cd SeqDex
pip install -r requirements.txt
pip install -e .
there is no error
and then, Trained checkpoint. Download from Link.
(base) yuhai@cappuccino:~/Downloads$ unzip checkpoint.zip
(base) yuhai@cappuccino:~/Downloads$ mv checkpoint /home/yuhai/SeqDex/dexteroushandenvs/checkpoint
and then I run the quick demo
in this file:

I run:
python train_rlgames.py --task BlockAssemblyGUI --num_envs=1 --play

And at this time it has been stuck here over 5 min:

then an error has occurred:
Started convex decomposition for mesh '../assets/urdf/blender/assets_for_insertion/urdf/../origin_obj/8x8x1_real/8x8x1_real.stl' Finished convex decomposition for mesh '../assets/urdf/blender/assets_for_insertion/urdf/../origin_obj/8x8x1_real/8x8x1_real.stl': 183 hulls num_extra_lego_asset_shapes: 1149 num_extra_lego_asset_bodies: 147 [Warning] [carb.gym.plugin] User aggregate size exceeded [Error] [carb.gym.plugin] cudaExternamMemoryGetMappedBuffer failed on rgbImage buffer with error 101 [Error] [carb.gym.plugin] cudaExternamMemoryGetMappedBuffer failed on depthImage buffer with error 101 [Error] [carb.gym.plugin] cudaExternamMemoryGetMappedBuffer failed on segmentationImage buffer with error 101 [Error] [carb.gym.plugin] cudaExternamMemoryGetMappedBuffer failed on optical flow buffer with error 101 *** Can't create empty tensor *** Can't create empty tensor mount_rigid_body_index: 7 [Error] [carb.gym.plugin] cudaExternamMemoryGetMappedBuffer failed on rgbImage buffer with error 101 [Error] [carb.gym.plugin] cudaExternamMemoryGetMappedBuffer failed on depthImage buffer with error 101 [Error] [carb.gym.plugin] cudaExternamMemoryGetMappedBuffer failed on segmentationImage buffer with error 101 [Error] [carb.gym.plugin] cudaExternamMemoryGetMappedBuffer failed on optical flow buffer with error 101 *** Can't create empty tensor [Error] [carb.gym.plugin] cudaExternamMemoryGetMappedBuffer failed on rgbImage buffer with error 101 [Error] [carb.gym.plugin] cudaExternamMemoryGetMappedBuffer failed on depthImage buffer with error 101 [Error] [carb.gym.plugin] cudaExternamMemoryGetMappedBuffer failed on segmentationImage buffer with error 101 [Error] [carb.gym.plugin] cudaExternamMemoryGetMappedBuffer failed on optical flow buffer with error 101 *** Can't create empty tensor *** Can't create empty tensor Traceback (most recent call last): File "train_rlgames.py", line 75, in <module> task, env = parse_task(args, cfg, cfg_train, sim_params, agent_index) File "/home/yuhai/SeqDex/dexteroushandenvs/utils/parse_task.py", line 166, in parse_task task = eval(args.task)( File "/home/yuhai/SeqDex/dexteroushandenvs/tasks/block_assembly/allegro_hand_block_assembly_gui.py", line 187, in __init__ super().__init__(cfg=self.cfg, enable_camera_sensors=self.enable_camera_sensors) File "/home/yuhai/SeqDex/dexteroushandenvs/tasks/hand_base/base_task.py", line 83, in __init__ self.create_sim() File "/home/yuhai/SeqDex/dexteroushandenvs/tasks/block_assembly/allegro_hand_block_assembly_gui.py", line 498, in create_sim self._create_envs(self.num_envs, self.cfg["env"]['envSpacing'], int(np.sqrt(self.num_envs))) File "/home/yuhai/SeqDex/dexteroushandenvs/tasks/block_assembly/allegro_hand_block_assembly_gui.py", line 1301, in _create_envs self.camera_rgbd_image_tensors = torch.stack(self.camera_tensors, dim=0).view(self.num_envs, -1) TypeError: expected Tensor as element 0 in argument 0, but got NoneType

[j96w]

I see. So this error is camera can't be created. We try to reproduce the error, but it works well on our end. We guess this is probably related to some bugs in using isaacgym with multiple GPUs. Sometimes when using multiple GPUs, an image driver that is not cuda may be specified, such as llvmpipe in this issue. So maybe you can try specifying the GPU as they did in this issue. Let me know whether it helps

[j96w]

Hey @YuhaiW , did you figure out the issue? Feel free to share more details and we can try to reproduce the error.

[YuhaiW]

Hi Chen, I will go back to lab in a few days. And then I will share you more information. Thank you. Yuhai

…

On Tue, Dec 19, 2023 at 02:48 Chen Wang ***@***.***> wrote: Hey @YuhaiW <https://urldefense.com/v3/__https://github.com/YuhaiW__;!!LIr3w8kk_Xxm!oQAIFWmhosGN5-6xyRRvGSQjRysPSyfSSaxjr87JeRv1k4slBiIQ4x1gT-EmfVB2yT6TnhfET9VQLydDGA3O3dlp$> , did you figure out the issue? Feel free to share more details and we can try to reproduce the error. — Reply to this email directly, view it on GitHub <https://urldefense.com/v3/__https://github.com/sequential-dexterity/SeqDex/issues/4*issuecomment-1862271984__;Iw!!LIr3w8kk_Xxm!oQAIFWmhosGN5-6xyRRvGSQjRysPSyfSSaxjr87JeRv1k4slBiIQ4x1gT-EmfVB2yT6TnhfET9VQLydDGEFYm-i5$>, or unsubscribe <https://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/A5EQLKPIPVP7E7BSARYZ4EDYKFBDPAVCNFSM6AAAAAA63ROIQSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNRSGI3TCOJYGQ__;!!LIr3w8kk_Xxm!oQAIFWmhosGN5-6xyRRvGSQjRysPSyfSSaxjr87JeRv1k4slBiIQ4x1gT-EmfVB2yT6TnhfET9VQLydDGKIx9ZYw$> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[YuhaiW]

Hi,
I run the code again, I am using --graphics_device_id 0 to make sure using only 1 GPU
(gym) yuhai@cappuccino:~/SeqDex/dexteroushandenvs$ python train_rlgames.py --task BlockAssemblyGUI --num_envs=1 --play --graphics_device_id 0
I think the original error has been fixed, but there is a new error happened,
Importing module 'gym_38' (/home/yuhai/isaacgym/python/isaacgym/_bindings/linux-x86_64/gym_38.so) Setting GYM_USD_PLUG_INFO_PATH to /home/yuhai/isaacgym/python/isaacgym/_bindings/linux-x86_64/usd/plugInfo.json PyTorch version 2.1.1+cu121 Device count 2 /home/yuhai/isaacgym/python/isaacgym/_bindings/src/gymtorch Using /home/yuhai/.cache/torch_extensions/py38_cu121 as PyTorch extensions root... Emitting ninja build file /home/yuhai/.cache/torch_extensions/py38_cu121/gymtorch/build.ninja... Building extension module gymtorch... Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N) ninja: no work to do. Loading extension module gymtorch... Loading config: cfg/lego/ppo_continuous.yaml Setting seed: 22 Task type: RLgames Obs type: partial_contact Not connected to PVD +++ Using GPU PhysX Physics Engine: PhysX Physics Device: cuda:0 GPU Pipeline: enabled Num dofs: 23 num_arm_hand_shapes: 31 num_arm_hand_bodies: 24 /home/yuhai/SeqDex/dexteroushandenvs/tasks/block_assembly/allegro_hand_block_assembly_gui.py:711: DeprecationWarning: an integer is required (got type isaacgym._bindings.linux-x86_64.gym_38.DofDriveMode). Implicit conversion to integers using __int__ is deprecated, and may be removed in a future version of Python. lego_asset_options.default_dof_drive_mode = gymapi.DOF_MODE_NONE num_object_shapes: 1 num_object_bodies: 1 num_object_shapes: 2 num_object_bodies: 2 num_object_shapes: 3 num_object_bodies: 3 num_object_shapes: 4 num_object_bodies: 4 num_object_shapes: 5 num_object_bodies: 5 num_object_shapes: 6 num_object_bodies: 6 num_object_shapes: 7 num_object_bodies: 7 num_object_shapes: 8 num_object_bodies: 8 num_object_shapes: 9 num_object_bodies: 9 num_object_shapes: 10 num_object_bodies: 10 num_object_shapes: 11 num_object_bodies: 11 num_object_shapes: 12 num_object_bodies: 12 num_object_shapes: 13 num_object_bodies: 13 num_object_shapes: 14 num_object_bodies: 14 num_object_shapes: 15 num_object_bodies: 15 num_object_shapes: 16 num_object_bodies: 16 num_object_shapes: 17 num_object_bodies: 17 num_object_shapes: 18 num_object_bodies: 18 num_object_shapes: 19 num_object_bodies: 19 num_object_shapes: 20 num_object_bodies: 20 num_object_shapes: 21 num_object_bodies: 21 num_object_shapes: 22 num_object_bodies: 22 num_object_shapes: 23 num_object_bodies: 23 num_object_shapes: 24 num_object_bodies: 24 num_object_shapes: 25 num_object_bodies: 25 num_object_shapes: 26 num_object_bodies: 26 num_object_shapes: 27 num_object_bodies: 27 num_object_shapes: 28 num_object_bodies: 28 num_object_shapes: 29 num_object_bodies: 29 num_object_shapes: 30 num_object_bodies: 30 num_object_shapes: 31 num_object_bodies: 31 num_object_shapes: 32 num_object_bodies: 32 num_object_shapes: 33 num_object_bodies: 33 num_object_shapes: 34 num_object_bodies: 34 num_object_shapes: 35 num_object_bodies: 35 num_object_shapes: 36 num_object_bodies: 36 num_object_shapes: 37 num_object_bodies: 37 num_object_shapes: 38 num_object_bodies: 38 num_object_shapes: 39 num_object_bodies: 39 num_object_shapes: 40 num_object_bodies: 40 num_object_shapes: 41 num_object_bodies: 41 num_object_shapes: 42 num_object_bodies: 42 num_object_shapes: 43 num_object_bodies: 43 num_object_shapes: 44 num_object_bodies: 44 num_object_shapes: 45 num_object_bodies: 45 num_object_shapes: 46 num_object_bodies: 46 num_object_shapes: 47 num_object_bodies: 47 num_object_shapes: 48 num_object_bodies: 48 num_object_shapes: 49 num_object_bodies: 49 num_object_shapes: 50 num_object_bodies: 50 num_object_shapes: 51 num_object_bodies: 51 num_object_shapes: 52 num_object_bodies: 52 num_object_shapes: 53 num_object_bodies: 53 num_object_shapes: 54 num_object_bodies: 54 num_object_shapes: 55 num_object_bodies: 55 num_object_shapes: 56 num_object_bodies: 56 num_object_shapes: 57 num_object_bodies: 57 num_object_shapes: 58 num_object_bodies: 58 num_object_shapes: 59 num_object_bodies: 59 num_object_shapes: 60 num_object_bodies: 60 num_object_shapes: 61 num_object_bodies: 61 num_object_shapes: 62 num_object_bodies: 62 num_object_shapes: 63 num_object_bodies: 63 num_object_shapes: 64 num_object_bodies: 64 num_object_shapes: 65 num_object_bodies: 65 num_object_shapes: 66 num_object_bodies: 66 num_object_shapes: 67 num_object_bodies: 67 num_object_shapes: 68 num_object_bodies: 68 num_object_shapes: 69 num_object_bodies: 69 num_object_shapes: 70 num_object_bodies: 70 num_object_shapes: 71 num_object_bodies: 71 num_object_shapes: 72 num_object_bodies: 72 num_object_shapes: 73 num_object_bodies: 73 num_object_shapes: 74 num_object_bodies: 74 num_object_shapes: 75 num_object_bodies: 75 num_object_shapes: 76 num_object_bodies: 76 num_object_shapes: 77 num_object_bodies: 77 num_object_shapes: 78 num_object_bodies: 78 num_object_shapes: 79 num_object_bodies: 79 num_object_shapes: 80 num_object_bodies: 80 num_object_shapes: 81 num_object_bodies: 81 num_object_shapes: 82 num_object_bodies: 82 num_object_shapes: 83 num_object_bodies: 83 num_object_shapes: 84 num_object_bodies: 84 num_object_shapes: 85 num_object_bodies: 85 num_object_shapes: 86 num_object_bodies: 86 num_object_shapes: 87 num_object_bodies: 87 num_object_shapes: 88 num_object_bodies: 88 num_object_shapes: 89 num_object_bodies: 89 num_object_shapes: 90 num_object_bodies: 90 num_object_shapes: 91 num_object_bodies: 91 num_object_shapes: 92 num_object_bodies: 92 num_object_shapes: 93 num_object_bodies: 93 num_object_shapes: 94 num_object_bodies: 94 num_object_shapes: 95 num_object_bodies: 95 num_object_shapes: 96 num_object_bodies: 96 num_object_shapes: 97 num_object_bodies: 97 num_object_shapes: 98 num_object_bodies: 98 num_object_shapes: 99 num_object_bodies: 99 num_object_shapes: 100 num_object_bodies: 100 num_object_shapes: 101 num_object_bodies: 101 num_object_shapes: 102 num_object_bodies: 102 num_object_shapes: 103 num_object_bodies: 103 num_object_shapes: 104 num_object_bodies: 104 num_object_shapes: 105 num_object_bodies: 105 num_object_shapes: 106 num_object_bodies: 106 Using VHACD cache directory '/home/yuhai/.isaacgym/vhacd' Found existing convex decomposition for mesh '../assets/urdf/blender/urdf/../origin_obj/1x2/1x2.stl' num_object_shapes: 126 num_object_bodies: 107 Found existing convex decomposition for mesh '../assets/urdf/blender/urdf/../origin_obj/1x2_curve/1x2_curve.stl' num_object_shapes: 171 num_object_bodies: 108 Found existing convex decomposition for mesh '../assets/urdf/blender/urdf/../origin_obj/1x3_curve_soft/1x3_curve_soft.stl' num_object_shapes: 205 num_object_bodies: 109 Found existing convex decomposition for mesh '../assets/urdf/blender/urdf/../origin_obj/1x3_curve/1x3_curve.stl' num_object_shapes: 227 num_object_bodies: 110 Found existing convex decomposition for mesh '../assets/urdf/blender/urdf/../origin_obj/1x1/1x1.stl' num_object_shapes: 240 num_object_bodies: 111 Found existing convex decomposition for mesh '../assets/urdf/blender/urdf/../origin_obj/1x3/1x3.stl' num_object_shapes: 267 num_object_bodies: 112 Found existing convex decomposition for mesh '../assets/urdf/blender/urdf/../origin_obj/1x4/1x4.stl' num_object_shapes: 296 num_object_bodies: 113 Found existing convex decomposition for mesh '../assets/urdf/blender/urdf/../origin_obj/2x2_curve_soft/2x2_curve_soft.stl' num_object_shapes: 319 num_object_bodies: 114 Found existing convex decomposition for mesh '../assets/urdf/blender/urdf/../origin_obj/1x2/1x2.stl' num_object_shapes: 339 num_object_bodies: 115 Found existing convex decomposition for mesh '../assets/urdf/blender/urdf/../origin_obj/1x2_curve/1x2_curve.stl' num_object_shapes: 384 num_object_bodies: 116 Found existing convex decomposition for mesh '../assets/urdf/blender/urdf/../origin_obj/1x3_curve_soft/1x3_curve_soft.stl' num_object_shapes: 418 num_object_bodies: 117 Found existing convex decomposition for mesh '../assets/urdf/blender/urdf/../origin_obj/1x3_curve/1x3_curve.stl' num_object_shapes: 440 num_object_bodies: 118 Found existing convex decomposition for mesh '../assets/urdf/blender/urdf/../origin_obj/1x1/1x1.stl' num_object_shapes: 453 num_object_bodies: 119 Found existing convex decomposition for mesh '../assets/urdf/blender/urdf/../origin_obj/1x3/1x3.stl' num_object_shapes: 480 num_object_bodies: 120 Found existing convex decomposition for mesh '../assets/urdf/blender/urdf/../origin_obj/1x4/1x4.stl' num_object_shapes: 509 num_object_bodies: 121 Found existing convex decomposition for mesh '../assets/urdf/blender/urdf/../origin_obj/2x2_curve_soft/2x2_curve_soft.stl' num_object_shapes: 532 num_object_bodies: 122 Found existing convex decomposition for mesh '../assets/urdf/blender/urdf/../origin_obj/1x2/1x2.stl' num_object_shapes: 552 num_object_bodies: 123 Found existing convex decomposition for mesh '../assets/urdf/blender/urdf/../origin_obj/1x2_curve/1x2_curve.stl' num_object_shapes: 597 num_object_bodies: 124 Found existing convex decomposition for mesh '../assets/urdf/blender/urdf/../origin_obj/1x3_curve_soft/1x3_curve_soft.stl' num_object_shapes: 631 num_object_bodies: 125 Found existing convex decomposition for mesh '../assets/urdf/blender/urdf/../origin_obj/1x3_curve/1x3_curve.stl' num_object_shapes: 653 num_object_bodies: 126 Found existing convex decomposition for mesh '../assets/urdf/blender/urdf/../origin_obj/1x1/1x1.stl' num_object_shapes: 666 num_object_bodies: 127 Found existing convex decomposition for mesh '../assets/urdf/blender/urdf/../origin_obj/1x3/1x3.stl' num_object_shapes: 693 num_object_bodies: 128 Found existing convex decomposition for mesh '../assets/urdf/blender/urdf/../origin_obj/1x4/1x4.stl' num_object_shapes: 722 num_object_bodies: 129 Found existing convex decomposition for mesh '../assets/urdf/blender/urdf/../origin_obj/2x2_curve_soft/2x2_curve_soft.stl' num_object_shapes: 745 num_object_bodies: 130 Found existing convex decomposition for mesh '../assets/urdf/blender/urdf/../origin_obj/1x2/1x2.stl' num_object_shapes: 765 num_object_bodies: 131 Found existing convex decomposition for mesh '../assets/urdf/blender/urdf/../origin_obj/1x2_curve/1x2_curve.stl' num_object_shapes: 810 num_object_bodies: 132 Found existing convex decomposition for mesh '../assets/urdf/blender/urdf/../origin_obj/1x3_curve_soft/1x3_curve_soft.stl' num_object_shapes: 844 num_object_bodies: 133 Found existing convex decomposition for mesh '../assets/urdf/blender/urdf/../origin_obj/1x3_curve/1x3_curve.stl' num_object_shapes: 866 num_object_bodies: 134 Found existing convex decomposition for mesh '../assets/urdf/blender/urdf/../origin_obj/1x1/1x1.stl' num_object_shapes: 879 num_object_bodies: 135 Found existing convex decomposition for mesh '../assets/urdf/blender/urdf/../origin_obj/1x3/1x3.stl' num_object_shapes: 906 num_object_bodies: 136 Found existing convex decomposition for mesh '../assets/urdf/blender/urdf/../origin_obj/1x4/1x4.stl' num_object_shapes: 935 num_object_bodies: 137 Found existing convex decomposition for mesh '../assets/urdf/blender/urdf/../origin_obj/2x2_curve_soft/2x2_curve_soft.stl' num_object_shapes: 958 num_object_bodies: 138 num_object_shapes: 959 num_object_bodies: 139 num_object_shapes: 960 num_object_bodies: 140 num_object_shapes: 961 num_object_bodies: 141 num_object_shapes: 962 num_object_bodies: 142 num_object_shapes: 963 num_object_bodies: 143 num_object_shapes: 964 num_object_bodies: 144 num_object_shapes: 965 num_object_bodies: 145 num_object_shapes: 966 num_object_bodies: 146 Found existing convex decomposition for mesh '../assets/urdf/blender/assets_for_insertion/urdf/../origin_obj/8x8x1_real/8x8x1_real.stl' num_extra_lego_asset_shapes: 1149 num_extra_lego_asset_bodies: 147 [Warning] [carb.gym.plugin] User aggregate size exceeded [Error] [carb.gym.plugin] cudaExternamMemoryGetMappedBuffer failed on rgbImage buffer with error 101 [Error] [carb.gym.plugin] cudaExternamMemoryGetMappedBuffer failed on depthImage buffer with error 101 [Error] [carb.gym.plugin] cudaExternamMemoryGetMappedBuffer failed on segmentationImage buffer with error 101 [Error] [carb.gym.plugin] cudaExternamMemoryGetMappedBuffer failed on optical flow buffer with error 101 *** Can't create empty tensor *** Can't create empty tensor mount_rigid_body_index: 7 [Error] [carb.gym.plugin] cudaExternamMemoryGetMappedBuffer failed on rgbImage buffer with error 101 [Error] [carb.gym.plugin] cudaExternamMemoryGetMappedBuffer failed on depthImage buffer with error 101 [Error] [carb.gym.plugin] cudaExternamMemoryGetMappedBuffer failed on segmentationImage buffer with error 101 [Error] [carb.gym.plugin] cudaExternamMemoryGetMappedBuffer failed on optical flow buffer with error 101 *** Can't create empty tensor [Error] [carb.gym.plugin] cudaExternamMemoryGetMappedBuffer failed on rgbImage buffer with error 101 [Error] [carb.gym.plugin] cudaExternamMemoryGetMappedBuffer failed on depthImage buffer with error 101 [Error] [carb.gym.plugin] cudaExternamMemoryGetMappedBuffer failed on segmentationImage buffer with error 101 [Error] [carb.gym.plugin] cudaExternamMemoryGetMappedBuffer failed on optical flow buffer with error 101 *** Can't create empty tensor *** Can't create empty tensor Traceback (most recent call last): File "train_rlgames.py", line 75, in <module> task, env = parse_task(args, cfg, cfg_train, sim_params, agent_index) File "/home/yuhai/SeqDex/dexteroushandenvs/utils/parse_task.py", line 166, in parse_task task = eval(args.task)( File "/home/yuhai/SeqDex/dexteroushandenvs/tasks/block_assembly/allegro_hand_block_assembly_gui.py", line 187, in __init__ super().__init__(cfg=self.cfg, enable_camera_sensors=self.enable_camera_sensors) File "/home/yuhai/SeqDex/dexteroushandenvs/tasks/hand_base/base_task.py", line 83, in __init__ self.create_sim() File "/home/yuhai/SeqDex/dexteroushandenvs/tasks/block_assembly/allegro_hand_block_assembly_gui.py", line 498, in create_sim self._create_envs(self.num_envs, self.cfg["env"]['envSpacing'], int(np.sqrt(self.num_envs))) File "/home/yuhai/SeqDex/dexteroushandenvs/tasks/block_assembly/allegro_hand_block_assembly_gui.py", line 1301, in _create_envs self.camera_rgbd_image_tensors = torch.stack(self.camera_tensors, dim=0).view(self.num_envs, -1) TypeError: expected Tensor as element 0 in argument 0, but got NoneType