.vzg update- runtime error ('Type')

[katielong3768]

Hi everyone-

I am trying to run through the example dataset as described here:
https://vizgen.github.io/vizgen-postprocessing/analysis_vignettes/index.html

I am having an issue at the step 4, updating the .vzg file. When I run the following command:
vpt --verbose --processes 2 update-vzg
--input-vzg 202305010900_U2OS_small_set_VMSC00000/region_0/202305010900_U2OS_small_set_VMSC00000_region_0.vzg
--input-boundaries analysis_outputs/cellpose_micron_space.parquet
--input-entity-by-gene analysis_outputs/cell_by_gene.csv
--input-metadata analysis_outputs/cell_metadata.csv
--output-vzg analysis_outputs/202305010900_U2OS_small_set_VMSC00000_region_0_CellPose_PolyT.vzg

It returns back:
2023-11-20 17:51:34,498 - . - INFO - installed versions:
python 3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0]
vpt 1.1.2
vpt_core 1.0.0
geopandas 0.12.1
shapely 2.0.0
s3fs 2021.10.0
gcsfs 2021.10.0
rasterio 1.3.0
cellpose Not Installed
stardist Not Installed
2023-11-20 17:51:34,503 - . - INFO - Unpacking vzg file
2023-11-20 17:51:34,991 - . - INFO - 202305010900_U2OS_small_set_VMSC00000/region_0/202305010900_U2OS_small_set_VMSC00000_region_0.vzg unpacked!
2023-11-20 17:51:34,991 - . - INFO - Dataset folder: /home/user/vzg_build_temp/vzg_2023-11-20T17_51_34_503159/202305010900_U2OS_small_set_VMSC00000_region_0
2023-11-20 17:51:34,991 - . - INFO - Number of input genes: 140
2023-11-20 17:51:34,999 - . - ERROR - exception of type <class 'KeyError'> thrown: 'Type'
2023-11-20 17:51:34,999 - . - ERROR - vpt has encountered a runtime error ('Type') and will now exit

What is the nature of the runtime error ('Type') and what could be done to resolve this?

Thanks!
Katie

[JessicaChevallier]

Hi,

I'm getting the same error when updating the .vzg file. Were you able to fix it?

Thank you!
Jessica

[wangbo00129]

I've got the same error and the key for failing might be the step 1, which outputs: 2024-01-31 09:29:55,524 - . - INFO - installed versions:
python 3.10.13 (main, Sep 11 2023, 13:44:35) [GCC 11.2.0]
vpt 1.1.2
vpt_core 1.0.0
geopandas 0.12.1
shapely 2.0.0
s3fs 2021.10.0
gcsfs 2021.10.0
rasterio 1.3.0
cellpose 1.0.2
stardist 0.8.3
2024-01-31 09:29:55,525 - . - INFO - run_segmentation started
2024-01-31 09:29:55,638 - . - INFO - prepare segmentation started
2024-01-31 09:29:55,806 - . - INFO - prepare segmentation finished
2024-01-31 09:29:55,811 - . - INFO - Run segmentation on tile 0 started
2024-01-31 09:29:55,812 - . - INFO - Tile 0 [0, 0, 2800, 2800]
2024-01-31 09:29:58,733 - . - INFO - raw segmentation result contains 0 rows
2024-01-31 09:29:58,734 - . - INFO - fuze across z
2024-01-31 09:29:58,736 - . - INFO - remove edge polys
2024-01-31 09:30:00,146 - . - INFO - raw segmentation result contains 0 rows
2024-01-31 09:30:00,146 - . - INFO - fuze across z
2024-01-31 09:30:00,150 - . - INFO - remove edge polys
2024-01-31 09:30:00,152 - . - INFO - fuse_task_polygons
2024-01-31 09:30:00,159 - . - INFO - Found 0 overlaps
2024-01-31 09:30:00,161 - . - INFO - After union of large overlaps, found 0 overlaps
2024-01-31 09:30:00,162 - . - INFO - After both resolution steps, found 0 uncaught overlaps
2024-01-31 09:30:00,216 - . - INFO - Run segmentation on tile 0 finished
2024-01-31 09:30:00,216 - . - INFO - Run segmentation on tile 1 started
2024-01-31 09:30:00,218 - . - INFO - Tile 1 [1153, 0, 2800, 2800]
2024-01-31 09:30:02,967 - . - INFO - raw segmentation result contains 0 rows
2024-01-31 09:30:02,967 - . - INFO - fuze across z
2024-01-31 09:30:02,971 - . - INFO - remove edge polys
2024-01-31 09:30:04,135 - . - INFO - raw segmentation result contains 0 rows

[JessicaChevallier]

Hi,

@wangbo00129 I also think that's most likely the issue. It seems that the segmentation step is not running properly but no error appears at the end. So far I've been unable to figure out where the issue is coming from. My experimental data matches the segmentation algorithm file and I've checked multiple times that VPT is installed correctly. Are you also using the cellpose_default_1_ZLevel.json provided by the authors in the example_analysis_algorithm folder?

@timwiggin or @connor-yu4322 it would be really helpful for us if you could take a look at this issue when you get the chance. Have you encountered something like this in the past?

Thank you!

[pakiessling]

Have you checked that cellpose_micron_space.parquet, cell_by_gene.csv, cell_metadata.csv contain valid entries?

I have been using vpt for a long time now and never got this error

[JessicaChevallier]

Hi,

@pakiessling Are you referring to the files after running vpt? If so, the contents of the cell_by_gene.csv file and the cellpose_micron_space.parquet are incorrect. At least that's what I think when I compared them to the files received directly from Vizgen. I also do not have a cell_metadata.csv output. It seems like the issue comes from running cellpose, but because no error is provided at that step of the pipeline I'm not sure how to fix this.

Have you also been using the json files suggested by the authors for your segmentation analysis?

[pakiessling]

I am talking about the output of the vpt pipeline, yes.
What exactly do you mean "incorrect"? They are obviousy supposed to be different than the input files you received from your machine.
Can you open them / do they contain a realistic amount of cells?

Have you also been using the json files suggested by the authors for your segmentation analysis?

Yes I also have used cellpose_default_1_ZLevel.json succesfully before.

[JessicaChevallier]

What I mean by "incorrect" is that the cellpose_micron_space.parquet is blank. It contains just the column headings and no cells because as I suggested the cellpose step of the pipeline did not work properly.

+------+------------+----------+------------+--------------+------------+
| ID   | EntityID   | ZIndex   | Geometry   | ParentType   | ParentID   |
|------+------------+----------+------------+--------------+------------|
+------+------------+----------+------------+--------------+------------+

Consequently, the cell_by_gene.csv file is blank and no cell_metadata.csv file was created.

[pakiessling]

Ok, that means the segmentation completely failed.

I would run the VPT like this: vpt --verbose --log-level 1 run-segmentation ... and check the log file.

Also try segmenting a part of your images with your Cellpose installation outside of the VPT pipeline (same parameters as the json file)

[JessicaChevallier]

Hi,

I stopped encountering the previous issues when I installed vpt using the docker image provided by the authors. I had originally installed vpt in a conda environment and although no errors appeared during the installation process there seem to be some issues running the vpt commands inside this environment (maybe due to package dependencies). If anyone is encountering the issues mentioned by @katielong3768, @wangbo00129 and myself I recommend installing vpt using the docker image and trying the tutorial again to make sure everything is working fine.