Problem with Uniform spike picking

[jakecroft]

Hello, I am trying to run uniform spike picking on manually picked virions. When I run with no spike picking, it works fine and it detects the virions based on my manual positions, but when I add uniform spike detection it gets an error and no auto virions or auto spikes are output.

Here is the output error:

2024-11-27 10:09:08 [INFO] /opt/pyp/bin/run/pyp:3232 | Job 8413156144770791730_1 (v0.6.4) launching on eggg.medchem.washington.edu using 8 task(s) 

2 2024-11-27 10:09:08 [INFO] /opt/pyp/bin/run/pyp:3178 | Filesystem Size Used Avail Use% Mounted on
3 2024-11-27 10:09:08 [INFO] /opt/pyp/bin/run/pyp:3178 | /dev/nvme0n1p3 1.8T 822G 905G 48% /tmp/pyp
4 2024-11-27 10:09:08 [INFO] /opt/pyp/bin/run/pyp:1347 | Working path: /tmp/pyp/nextpyp/8413156144770791730_1/MMTV-MLVs-24jun04a-TS-58
5 2024-11-27 10:09:08 [INFO] pyp/inout/utils/core.py:17 | Retrieving .pyp_config.toml
6 2024-11-27 10:09:08 [INFO] pyp/inout/utils/core.py:17 | Retrieving tomo-preprocessing-fYHvdxz3b5w42nGF.films
7 2024-11-27 10:09:08 [INFO] pyp/inout/utils/core.py:17 | Retrieving tomo-preprocessing-fYHvdxz3b5w42nGF.micrographs
8 2024-11-27 10:09:08 [INFO] pyp/inout/utils/core.py:17 | Retrieving MMTV-MLVs-24jun04a-TS-58_sides.webp
9 2024-11-27 10:09:08 [INFO] pyp/inout/utils/core.py:17 | Retrieving MMTV-MLVs-24jun04a-TS-58_raw.webp
10 2024-11-27 10:09:08 [INFO] pyp/inout/utils/core.py:17 | Retrieving MMTV-MLVs-24jun04a-TS-58.mrc
11 2024-11-27 10:09:08 [INFO] pyp/inout/utils/core.py:17 | Retrieving MMTV-MLVs-24jun04a-TS-58.rec
12 2024-11-27 10:09:08 [INFO] pyp/inout/utils/core.py:17 | Retrieving MMTV-MLVs-24jun04a-TS-58.next
13 2024-11-27 10:09:08 [INFO] pyp/inout/utils/core.py:17 | Retrieving MMTV-MLVs-24jun04a-TS-58.pkl
14 2024-11-27 10:09:08 [INFO] pyp/inout/utils/core.py:17 | Retrieving MMTV-MLVs-24jun04a-TS-58_rec.webp
15 2024-11-27 10:09:08 [INFO] pyp/inout/utils/core.py:17 | Retrieving MMTV-MLVs-24jun04a-TS-58.webp
16 2024-11-27 10:09:08 [INFO] pyp/inout/utils/core.py:17 | Retrieving MMTV-MLVs-24jun04a-TS-58_ali.webp
17 2024-11-27 10:09:11 [INFO] /opt/pyp/bin/run/pyp:57 | Loading results took: 00h 00m 03s
18 2024-11-27 10:09:18 [INFO] pyp/system/mpi.py:97 | Running 41 command(s)
19 2024-11-27 10:09:18100%00:04<00:009.81it/s
20 2024-11-27 10:09:22 [INFO] pyp/system/mpi.py:111 | 41 command(s) finished
21 2024-11-27 10:09:22 [INFO] pyp/preprocess/core.py:624 | Original dimensions: [ X ,Y, Z ] = [ 5760, 4092, 41 ], pixel = 1.3864, kV = 300.0, mag = 10000, axis = -174.400000
22 2024-11-27 10:09:22 [INFO] pyp/system/mpi.py:97 | Running 41 command(s)
23 2024-11-27 10:09:22100%00:04<00:009.36it/s
24 2024-11-27 10:09:27 [INFO] pyp/system/mpi.py:111 | 41 command(s) finished
25 2024-11-27 10:09:34 [INFO] pyp/preprocess/core.py:57 | Convert tilt-series into squares took: 00h 00m 11s
26 2024-11-27 10:09:34 [INFO] pyp/preprocess/core.py:674 | After resizing/binning: [ X ,Y, Z ] = [ 5760, 4092, 41 ], pixel = 1.3864, kV = 300.0, mag = 10000, axis = -174.400000
27 2024-11-27 10:09:34 [INFO] /opt/pyp/bin/run/pyp:1510 | Using existing tilt-series alignments
28 2024-11-27 10:09:34 [INFO] pyp/inout/image/core.py:1456 | Detected TILT AXIS ANGLE = 79.29
29 2024-11-27 10:09:34 [INFO] pyp/system/mpi.py:97 | Running 41 command(s)
30 2024-11-27 10:09:34100%00:07<00:005.84it/s
31 2024-11-27 10:09:41 [INFO] pyp/system/mpi.py:111 | 41 command(s) finished
32 2024-11-27 10:09:48 [INFO] /opt/pyp/bin/run/pyp:57 | Apply alignment to tiltseries took: 00h 00m 14s
33 2024-11-27 10:09:48 [INFO] /opt/pyp/bin/run/pyp:1568 | Using existing CTF estimation
34 2024-11-27 10:09:48 [INFO] pyp/detect/tomo/core.py:1152 | Using virion manual picking
35 2024-11-27 10:09:48 [INFO] pyp/detect/tomo/core.py:1159 | Using manually picked coordinates
36 2024-11-27 10:09:49 [INFO] pyp/detect/tomo/core.py:669 | Found 1 virions
37 2024-11-27 10:09:49 [INFO] pyp/detect/tomo/core.py:740 | Detecting spikes using mesh vertices
38 2024-11-27 10:09:49 [INFO] pyp/detect/tomo/core.py:744 | Using 0.1 as segmentation threshold (column 1) for virion 0000
39 2024-11-27 10:09:53 [INFO] pyp/system/mpi.py:178 | Running 1 function(s)
40 2024-11-27 10:09:53100%00:24<00:0024.51s/it
41 2024-11-27 10:10:17 [ERROR] pyp/system/local_run.py:72 | ERROR: In /nfs/bartesaghilab/hl325/programs/ettk/ettk/external/VTK5.10.1/Graphics/vtkSmoothPolyDataFilter.cxx, line 211
42 vtkSmoothPolyDataFilter (0x3a118d0): No data to smooth!
43
44 ERROR: In /nfs/bartesaghilab/hl325/programs/ettk/ettk/external/VTK5.10.1/Graphics/vtkTransformPolyDataFilter.cxx, line 84
45 vtkTransformPolyDataFilter (0x3a175c0): No input data
46
47 ERROR: In /nfs/bartesaghilab/hl325/programs/ettk/ettk/external/VTK5.10.1/Graphics/vtkSmoothPolyDataFilter.cxx, line 211
48 vtkSmoothPolyDataFilter (0x3a12820): No data to smooth!
49
50 ERROR: In /nfs/bartesaghilab/hl325/programs/ettk/ettk/external/VTK5.10.1/Graphics/vtkTransformPolyDataFilter.cxx, line 84
51 vtkTransformPolyDataFilter (0x3a176a0): No input data
52
53 ERROR: In /nfs/bartesaghilab/hl325/programs/ettk/ettk/external/VTK5.10.1/Graphics/vtkSmoothPolyDataFilter.cxx, line 211
54 vtkSmoothPolyDataFilter (0x3a12940): No data to smooth!
55
56 ERROR: In /nfs/bartesaghilab/hl325/programs/ettk/ettk/external/VTK5.10.1/Graphics/vtkTransformPolyDataFilter.cxx, line 84
57 vtkTransformPolyDataFilter (0x3a176a0): No input data
58
59 ERROR: In /nfs/bartesaghilab/hl325/programs/ettk/ettk/external/VTK5.10.1/Graphics/vtkSmoothPolyDataFilter.cxx, line 211
60 vtkSmoothPolyDataFilter (0x3a12870): No data to smooth!
61
62 ERROR: In /nfs/bartesaghilab/hl325/programs/ettk/ettk/external/VTK5.10.1/Graphics/vtkTransformPolyDataFilter.cxx, line 84
63 vtkTransformPolyDataFilter (0x3a17650): No input data
64
65 ERROR: In /nfs/bartesaghilab/hl325/programs/ettk/ettk/external/VTK5.10.1/Graphics/vtkSmoothPolyDataFilter.cxx, line 211
66 vtkSmoothPolyDataFilter (0x3a12870): No data to smooth!
67
68 ERROR: In /nfs/bartesaghilab/hl325/programs/ettk/ettk/external/VTK5.10.1/Graphics/vtkTransformPolyDataFilter.cxx, line 84
69 vtkTransformPolyDataFilter (0x3a176b0): No input data
70
71 ERROR: In /nfs/bartesaghilab/hl325/programs/ettk/ettk/external/VTK5.10.1/Graphics/vtkSmoothPolyDataFilter.cxx, line 211
72 vtkSmoothPolyDataFilter (0x3a12850): No data to smooth!
73
74 ERROR: In /nfs/bartesaghilab/hl325/programs/ettk/ettk/external/VTK5.10.1/Graphics/vtkTransformPolyDataFilter.cxx, line 84
75 vtkTransformPolyDataFilter (0x3a17670): No input data
76
77 ERROR: In /nfs/bartesaghilab/hl325/programs/ettk/ettk/external/VTK5.10.1/Graphics/vtkSmoothPolyDataFilter.cxx, line 211
78 vtkSmoothPolyDataFilter (0x3a128c0): No data to smooth!
79
80 ERROR: In /nfs/bartesaghilab/hl325/programs/ettk/ettk/external/VTK5.10.1/Graphics/vtkTransformPolyDataFilter.cxx, line 84
81 vtkTransformPolyDataFilter (0x3a17680): No input data
82
83 ERROR: In /nfs/bartesaghilab/hl325/programs/ettk/ettk/external/VTK5.10.1/Graphics/vtkSmoothPolyDataFilter.cxx, line 211
84 vtkSmoothPolyDataFilter (0x3a12950): No data to smooth!
85
86 ERROR: In /nfs/bartesaghilab/hl325/programs/ettk/ettk/external/VTK5.10.1/Graphics/vtkTransformPolyDataFilter.cxx, line 84
87 vtkTransformPolyDataFilter (0x3a176a0): No input data
88
89
90 2024-11-27 10:10:18 [INFO] /opt/pyp/bin/run/pyp:57 | Total time in tomo_swarm 00h 01m 10s seconds
91 2024-11-27 10:10:18 [INFO] /opt/pyp/bin/run/pyp:3178 | Filesystem Size Used Avail Use% Mounted on
92 2024-11-27 10:10:18 [INFO] /opt/pyp/bin/run/pyp:3178 | /dev/nvme0n1p3 1.8T 876G 851G 51% /tmp/pyp
93 2024-11-27 10:10:18 [ERROR] /opt/pyp/bin/run/pyp:3453 | PYP (tomoswarm) failed

[abartesaghi]

Do you see yellow segmentation curves that makes sense when using 0.1 as segmentation threshold (column 1)?

[jakecroft]

I do not see any yellow curves for the segmentations, just images of the tomogram that don't seem to correspond with where the "Auto virions" are on the normal view of the tomogram, so I had selected "-" (no segmentation) for them

[abartesaghi]

Uniform spike picking requires the execution of three steps: 1) virion center selection, 2) virion segmentation, and 3) spike picking. For 1), it looks like you are using manually selected positions which is fine. According to the logs, step 2) ran successfully and a threshold value of 0.1 (first column, which is the default selection) was used as the segmentation threshold. However, this is inconsistent with your selection of the "-" column, which essentially means that the virion will be ignored for the purpose of spike picking.

In the columns of the Segmentation tab, you should see copies of your manually selected virion, which does not seem to be the case for you. If you first did auto-virion detection and then switched to manual virion detection, you may need to re-run the block to calculate the segmentation of the manually selected virion.

An important factor that determines the success of the segmentation step is the virion radius. Did you specify the correct virion radius in A? The size of the virion marker in the Reconstruction tab should allow you to verify this.

[jakecroft]

Thanks. Got it, I misunderstood and thought selecting "-" meant it would just place the spikes based on the radius alone. It does appear correct for about half the virions, although a number of them are more heterogenous in size and up to 2X as big.

That was useful though, seems like segmentation is the main issue. I re-ran without spike picking to get the auto virions back from the manual positions and the radius looks approximately right for many of them, but when I look at the segmentations for those there are no yellow lines drawn on them and the image of the virion is cropped to only include half of it rather than centered on the virion like the images in the tutorial.

[abartesaghi]

That should not happen. The images in the Segmentation tab should be centered at the locations where virions were picked. Can you share the Command from the launch job?

[jakecroft]

Here it is:

"/opt/pyp/bin/run/pyp" -scope_pixel 1.3864 -scope_voltage 300.0 -scope_tilt_axis -84.4 -data_path "/archive/qnap/Jake/Active_projects/24jun04a_MMTV-MLVs_krios/nextpyp/symlinks/*.tif" -data_mode tomo -gain_reference "/archive/qnap/Jake/Active_projects/24jun04a_MMTV-MLVs_krios/gainref/CountCDSRef_MMTV_VLPs_24jun04a_56_0000_-0.0.mrc" -gain_rotation 180 -gain_fliph -tomo_ali_method aretomo -tomo_ali_patches_x 0 -tomo_ali_patches_y 0 -tomo_ali_aretomo_dark_tol 0.1 -no-tomo_rec_square -tomo_vir_rad 800.0 -tomo_vir_method manual -tomo_vir_number 3 -tomo_vir_det_tol 200.0 -tomo_vir_seg_tol 100.0 -tomo_vir_detect_method none -tomo_vir_detect_dist 5 -slurm_tasks 8 -slurm_memory 60 -slurm_merge_tasks 41

[abartesaghi]

Nothing unusual there other than the fact that your virions seem to be rather large. If you start a fresh pre-processing block and do auto virion selection, do the reconstructions in the Segmentation tab match the location of the virion centers? If you are willing to share an example tilt-series, email us and we'll be happy to take a look.

[jakecroft]

This is what it looks like, if that is helpful for diagnosis. I've attached a photo of the auto virion centers, and of what the segmentation tab looks like (I reduced the virion radius to 600 for this run, it seems to do a better job for most of them, I was just trying to be more inclusive of the larger ones before hoping it would find them both).

I will email you, I would greatly appreciate any time you could devote to troubleshooting one of my tilt series.