update-vzg failed with index out of bounds error

[tangboyun]

Hello everyone,
We got 2 datasets each had 10 regions and wanted to do cell segmentation. Two of the twenty regions failed at 'update-vzg' step with the index out-of-bounds error.

The first Region:

/home/tangboyun/vizgen/out_put/202307311058_mouseembryobrainMDBP0904_VMSC03601/region_2/202307311058_mouseembryobrainMDBP0904_VMSC03601_region_2.vzg unpacked!
2023-09-01 02:20:52,269 - . - INFO - Running cell assembly in 20 processes
2023-09-01 02:21:10,321 - ./task-13 - ERROR - exception of type <class 'IndexError'> thrown: index 16 is out of bounds for axis 0 with size 16
Traceback (most recent call last):
  File "/home/tangboyun/.local/lib/python3.10/site-packages/vpt/app/context.py", line 153, in _context_wrapper
    return task.proc(task.args)
  File "/home/tangboyun/.local/lib/python3.10/site-packages/vpt/update_vzg/run_update_vzg.py", line 34, in _process
    results = cell_transfer.process_cells(raw_cells, fov_idx)
  File "/home/tangboyun/.local/lib/python3.10/site-packages/vpt/update_vzg/polygon_transfer.py", line 46, in process_cells
    poly_relevant_reduce = polygon.points_transform(
  File "/home/tangboyun/.local/lib/python3.10/site-packages/vpt/update_vzg/polygons/polygonset.py", line 145, in points_transform
    self._make_fan_packed_lods(points, polys, expBbox, cellSize, normScale)
  File "/home/tangboyun/.local/lib/python3.10/site-packages/vpt/update_vzg/polygons/polygonset.py", line 224, in _make_fan_packed_lods
    self._fan_packed_lod(points, LodLevel.Min, 9,
  File "/home/tangboyun/.local/lib/python3.10/site-packages/vpt/update_vzg/polygons/polygonset.py", line 244, in _fan_packed_lod
    (polyType, fanPoly.get_packed_bytes(polys, cellSize,
  File "/home/tangboyun/.local/lib/python3.10/site-packages/vpt/update_vzg/polygons/packedfanpolygon.py", line 81, in get_packed_bytes
    self._pack_indices(polyList, packedPolyBtr)
  File "/home/tangboyun/.local/lib/python3.10/site-packages/vpt/update_vzg/polygons/packedfanpolygon.py", line 103, in _pack_indices
    indices[idx] = np.uint8(np.uint8(includeIdx[i]) << 7) +\
IndexError: index 16 is out of bounds for axis 0 with size 16

The second region：

/home/tangboyun/vizgen/out_put/202308181431_mouseembryobrainMDBP0904_VMSC03601/region_4/202308181431_mouseembryobrainMDBP0904_VMSC03601_region_4.vzg unpacked!
2023-09-01 07:03:33,499 - . - INFO - Running cell assembly in 20 processes
2023-09-01 07:04:36,810 - ./task-11 - ERROR - exception of type <class 'IndexError'> thrown: index 16 is out of bounds for axis 0 with size 16
Traceback (most recent call last):
  File "/home/tangboyun/.local/lib/python3.10/site-packages/vpt/app/context.py", line 153, in _context_wrapper
    return task.proc(task.args)
  File "/home/tangboyun/.local/lib/python3.10/site-packages/vpt/update_vzg/run_update_vzg.py", line 34, in _process
    results = cell_transfer.process_cells(raw_cells, fov_idx)
  File "/home/tangboyun/.local/lib/python3.10/site-packages/vpt/update_vzg/polygon_transfer.py", line 46, in process_cells
    poly_relevant_reduce = polygon.points_transform(
  File "/home/tangboyun/.local/lib/python3.10/site-packages/vpt/update_vzg/polygons/polygonset.py", line 145, in points_transform
    self._make_fan_packed_lods(points, polys, expBbox, cellSize, normScale)
  File "/home/tangboyun/.local/lib/python3.10/site-packages/vpt/update_vzg/polygons/polygonset.py", line 224, in _make_fan_packed_lods
    self._fan_packed_lod(points, LodLevel.Min, 9,
  File "/home/tangboyun/.local/lib/python3.10/site-packages/vpt/update_vzg/polygons/polygonset.py", line 244, in _fan_packed_lod
    (polyType, fanPoly.get_packed_bytes(polys, cellSize,
  File "/home/tangboyun/.local/lib/python3.10/site-packages/vpt/update_vzg/polygons/packedfanpolygon.py", line 81, in get_packed_bytes
    self._pack_indices(polyList, packedPolyBtr)
  File "/home/tangboyun/.local/lib/python3.10/site-packages/vpt/update_vzg/polygons/packedfanpolygon.py", line 103, in _pack_indices
    indices[idx] = np.uint8(np.uint8(includeIdx[i]) << 7) +\
IndexError: index 16 is out of bounds for axis 0 with size 16

We've tried to redo the whole segmentation for these two regions and found these errors were repeatable. We tried to upgrade the vpt to the latest v1.1.2, but still got the same error:

2023-09-01 09:03:55,721 - . - INFO - installed versions:
python 3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0]
vpt 1.1.2
vpt_core 1.0.0
geopandas 0.12.1
shapely 2.0.0
s3fs 2021.10.0
gcsfs 2021.10.0
rasterio 1.3.0
cellpose 1.0.2
stardist 0.8.3
2023-09-01 09:03:55,730 - . - INFO - Unpacking vzg file
2023-09-01 09:04:14,913 - . - INFO - /home/tangboyun/vizgen/out_put/202307311058_mouseembryobrainMDBP0904_VMSC03601/region_2/202307311058_mouseembryobrainMDBP0904_VMSC03601_region_2.vzg unpacked!
2023-09-01 09:04:14,914 - . - INFO - Dataset folder: /home/tangboyun/vzg_build_temp/vzg_2023-09-01T09_03_55_730223/202307311058_mouseembryobrainMDBP0904_VMSC03601_region_2
2023-09-01 09:04:14,914 - . - INFO - Number of input genes: 315
2023-09-01 09:04:21,565 - . - INFO - Running cell assembly in 10 processes for feature cell
2023-09-01 09:04:23,343 - ./task-0 - INFO - running cells processing for fovs
2023-09-01 09:04:23,347 - ./task-9 - INFO - running cells processing for fovs
2023-09-01 09:04:23,347 - ./task-8 - INFO - running cells processing for fovs
2023-09-01 09:04:23,348 - ./task-1 - INFO - running cells processing for fovs
2023-09-01 09:04:23,349 - ./task-2 - INFO - running cells processing for fovs
2023-09-01 09:04:23,350 - ./task-3 - INFO - running cells processing for fovs
2023-09-01 09:04:23,351 - ./task-6 - INFO - running cells processing for fovs
2023-09-01 09:04:23,351 - ./task-5 - INFO - running cells processing for fovs
2023-09-01 09:04:23,352 - ./task-7 - INFO - running cells processing for fovs
2023-09-01 09:04:23,353 - ./task-4 - INFO - running cells processing for fovs
2023-09-01 09:12:16,430 - ./task-4 - ERROR - exception of type <class 'IndexError'> thrown: index 16 is out of bounds for axis 0 with size 16
2023-09-01 09:12:19,786 - distributed.nanny - WARNING - Worker process still alive after 3.199999389648438 seconds, killing
2023-09-01 09:12:19,786 - distributed.nanny - WARNING - Worker process still alive after 3.1999995422363288 seconds, killing
2023-09-01 09:12:19,786 - distributed.nanny - WARNING - Worker process still alive after 3.1999995422363288 seconds, killing
2023-09-01 09:12:19,787 - distributed.nanny - WARNING - Worker process still alive after 3.199999389648438 seconds, killing
2023-09-01 09:12:19,874 - . - ERROR - exception of type <class 'IndexError'> thrown: index 16 is out of bounds for axis 0 with size 16
2023-09-01 09:12:19,874 - . - ERROR - vpt has encountered a runtime error (index 16 is out of bounds for axis 0 with size 16) and will now exit

2023-09-01 09:37:28,643 - . - INFO - installed versions:
python 3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0]
vpt 1.1.2
vpt_core 1.0.0
geopandas 0.12.1
shapely 2.0.0
s3fs 2021.10.0
gcsfs 2021.10.0
rasterio 1.3.0
cellpose 1.0.2
stardist 0.8.3
2023-09-01 09:37:28,652 - . - INFO - Unpacking vzg file
2023-09-01 09:37:48,139 - . - INFO - /home/tangboyun/vizgen/out_put/202308181431_mouseembryobrainMDBP0904_VMSC03601/region_4/202308181431_mouseembryobrainMDBP0904_VMSC03601_region_4.vzg unpacked!
2023-09-01 09:37:48,140 - . - INFO - Dataset folder: /home/tangboyun/vzg_build_temp/vzg_2023-09-01T09_37_28_652080/202308181431_mouseembryobrainMDBP0904_VMSC03601_region_4
2023-09-01 09:37:48,140 - . - INFO - Number of input genes: 315
2023-09-01 09:37:56,756 - . - INFO - Running cell assembly in 10 processes for feature cell
2023-09-01 09:37:58,424 - ./task-5 - INFO - running cells processing for fovs
2023-09-01 09:37:58,432 - ./task-7 - INFO - running cells processing for fovs
2023-09-01 09:37:58,437 - ./task-2 - INFO - running cells processing for fovs
2023-09-01 09:37:58,439 - ./task-8 - INFO - running cells processing for fovs
2023-09-01 09:37:58,439 - ./task-0 - INFO - running cells processing for fovs
2023-09-01 09:37:58,440 - ./task-3 - INFO - running cells processing for fovs
2023-09-01 09:37:58,442 - ./task-1 - INFO - running cells processing for fovs
2023-09-01 09:37:58,443 - ./task-6 - INFO - running cells processing for fovs
2023-09-01 09:37:58,443 - ./task-4 - INFO - running cells processing for fovs
2023-09-01 09:37:58,445 - ./task-9 - INFO - running cells processing for fovs
2023-09-01 09:38:48,284 - ./task-7 - INFO - Done fov 7
2023-09-01 09:44:48,868 - ./task-3 - ERROR - exception of type <class 'IndexError'> thrown: index 16 is out of bounds for axis 0 with size 16
2023-09-01 09:44:52,222 - distributed.nanny - WARNING - Worker process still alive after 3.199999389648438 seconds, killing
2023-09-01 09:44:52,222 - distributed.nanny - WARNING - Worker process still alive after 3.1999995422363288 seconds, killing
2023-09-01 09:44:52,223 - distributed.nanny - WARNING - Worker process still alive after 3.1999992370605472 seconds, killing
2023-09-01 09:44:52,223 - distributed.nanny - WARNING - Worker process still alive after 3.1999992370605472 seconds, killing
2023-09-01 09:44:52,224 - distributed.nanny - WARNING - Worker process still alive after 3.199999389648438 seconds, killing
2023-09-01 09:44:52,224 - distributed.nanny - WARNING - Worker process still alive after 3.199999389648438 seconds, killing
2023-09-01 09:44:52,401 - . - ERROR - exception of type <class 'IndexError'> thrown: index 16 is out of bounds for axis 0 with size 16
2023-09-01 09:44:52,401 - . - ERROR - vpt has encountered a runtime error (index 16 is out of bounds for axis 0 with size 16) and will now exit