Missing files (?) and error when using sMRIPrep +fMRIPrep

[araikes]

What happened?

I'm using smriprep 0.16.1 coupled with fmriprep 24.0.1. I was attempting to break the workflow down into steps for greater control on when things were ready. My data are multi-session (pre-treatment/post-treatment) with a 3 month(ish) interval. I have T1w and T2w images as well as resting-state BOLD. I ran sMRIPrep first without any obvious errors:

Everything appears to be present and runs without error. In the next step, I'm just running the anatomical stage of fMRIPrep and using --derivatives smriprep=$SMRIPREP. I know I could do everything all at once, but I have need to apply some filtering for specific subjects on specific sessions because of missing fieldmaps, so I'm doing things piecemeal. When I do this, I run into two issues:

1. There's no native space preprocessed T1w image in the fmriprep/sub-xxx/anat folder, only the T2w image. It's present in the sMRIPrep output.
   

2. I get an error (crash logs below).

240829-11:36:51,106 nipype.workflow ERROR:
	 could not run node: fmriprep_24_0_wf.sub_002_wf.hcp_morphometrics_wf.dilate_curv.a0
240829-11:36:51,110 nipype.workflow ERROR:
	 could not run node: fmriprep_24_0_wf.sub_002_wf.hcp_morphometrics_wf.dilate_curv.a1

What command did you use?

export APPTAINERENV_TEMPLATEFLOW_HOME=/opt/templateflow
export APPTAINERENV_TEMPLATEFLOW_USE_DATALAD

# sMRIPrep
apptainer run --containall -B /xdisk/adamraikes/PD_allo/nifti:/nifti:ro \
-B /xdisk/adamraikes/PD_allo/derivatives/smriprep_0.16.1:/xdisk/adamraikes/PD_allo/derivatives/smriprep_0.16.1 \
-B /groups/adamraikes/templateflow:/opt/templateflow \
-B /groups/adamraikes/license.txt:/license.txt \
-B /tmp:/tmp /groups/adamraikes/singularity_images/smriprep_0.16.1.sif \
/nifti /xdisk/adamraikes/PD_allo/derivatives/smriprep_0.16.1 participant \
--participant-label 002 \
--nthreads 12 \
--omp-nthreads 10 \
--longitudinal \
--output-spaces MNI152NLin2009cAsym \
--cifti-output \
--fs-license-file \
/license.txt -w /tmp

# fMRIPrep anatomical
apptainer run --containall \
-B /xdisk/adamraikes/PD_allo/nifti:/nifti:ro \
-B /xdisk/adamraikes/PD_allo/derivatives/fmriprep_24.0.1:/xdisk/adamraikes/PD_allo/derivatives/fmriprep_24.0.1 \
-B /xdisk/adamraikes/PD_allo/derivatives/smriprep_0.16.1:/xdisk/adamraikes/PD_allo/derivatives/smriprep_0.16.1 \
-B /groups/adamraikes/templateflow:/opt/templateflow \
-B /groups/adamraikes/license.txt:/license.txt \
-B /tmp:/tmp \
/groups/adamraikes/singularity_images/fmriprep_24.0.1.sif \
/nifti /xdisk/adamraikes/PD_allo/derivatives/fmriprep_24.0.1 participant \
--participant-label 002 \
--nthreads 12 \
--omp-nthreads 10 \
--cifti-output 91k \
--fs-license-file /license.txt \
--longitudinal \
--derivatives smriprep=/xdisk/adamraikes/PD_allo/derivatives/smriprep_0.16.1 \
--skip-bids-validation \
-w /xdisk/adamraikes/PD_allo/derivatives/fmriprep_24.0.1/scratch \
--anat-only

What version of fMRIPrep are you running?

24.0.1

How are you running fMRIPrep?

Singularity

Is your data BIDS valid?

Yes

Are you reusing any previously computed results?

Anatomical derivatives

Please copy and paste any relevant log output.

Node: fmriprep_24_0_wf.sub_002_wf.hcp_morphometrics_wf.dilate_curv
Working directory: /xdisk/adamraikes/PD_allo/derivatives/fmriprep_24.0.1/scratch/fmriprep_24_0_wf/sub_002_wf/hcp_morphometrics_wf/_hemi_L/dilate_curv

Node inputs:

args = <undefined>
bad_vertex_roi_file = <undefined>
column = <undefined>
corrected_areas = <undefined>
data_roi_file = <undefined>
distance = 10.0
environ = {'OMP_NUM_THREADS': '10'}
exponent = <undefined>
in_file = /xdisk/adamraikes/PD_allo/derivatives/fmriprep_24.0.1/scratch/fmriprep_24_0_wf/sub_002_wf/hcp_morphometrics_wf/_hemi_L/invert_curv/sub-002.L.curv.native.shape.gii
legacy_cutoff = <undefined>
linear = <undefined>
nearest = True
num_threads = 10
out_file = <undefined>
surf_file = /xdisk/adamraikes/PD_allo/derivatives/smriprep_0.16.1/smriprep/sub-002/anat/sub-002_run-001_hemi-L_midthickness.surf.gii

Traceback (most recent call last):
  File "/opt/conda/envs/fmriprep/lib/python3.11/site-packages/nipype/pipeline/plugins/multiproc.py", line 67, in run_node
    result["result"] = node.run(updatehash=updatehash)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/fmriprep/lib/python3.11/site-packages/nipype/pipeline/engine/nodes.py", line 527, in run
    result = self._run_interface(execute=True)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/fmriprep/lib/python3.11/site-packages/nipype/pipeline/engine/nodes.py", line 645, in _run_interface
    return self._run_command(execute)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/fmriprep/lib/python3.11/site-packages/nipype/pipeline/engine/nodes.py", line 771, in _run_command
    raise NodeExecutionError(msg)
nipype.pipeline.engine.nodes.NodeExecutionError: Exception raised while executing Node dilate_curv.

Cmdline:
	wb_command -metric-dilate  /xdisk/adamraikes/PD_allo/derivatives/fmriprep_24.0.1/scratch/fmriprep_24_0_wf/sub_002_wf/hcp_morphometrics_wf/_hemi_L/invert_curv/sub-002.L.curv.native.shape.gii  /xdisk/adamraikes/PD_allo/derivatives/smriprep_0.16.1/smriprep/sub-002/anat/sub-002_run-001_hemi-L_midthickness.surf.gii  10.000000  sub-002.L.curv.native.shape.func.gii  -nearest 
Stdout:

Stderr:

	While running:
	/opt/workbench/bin_linux64/../exe_linux64/wb_command -metric-dilate /xdisk/adamraikes/PD_allo/derivatives/fmriprep_24.0.1/scratch/fmriprep_24_0_wf/sub_002_wf/hcp_morphometrics_wf/_hemi_L/invert_curv/sub-002.L.curv.native.shape.gii /xdisk/adamraikes/PD_allo/derivatives/smriprep_0.16.1/smriprep/sub-002/anat/sub-002_run-001_hemi-L_midthickness.surf.gii 10.000000 sub-002.L.curv.native.shape.func.gii -nearest

	ERROR: surface and metric number of vertices do not match

Traceback:
	Traceback (most recent call last):
	  File "/opt/conda/envs/fmriprep/lib/python3.11/site-packages/nipype/interfaces/base/core.py", line 453, in aggregate_outputs
	    setattr(outputs, key, val)
	  File "/opt/conda/envs/fmriprep/lib/python3.11/site-packages/nipype/interfaces/base/traits_extension.py", line 330, in validate
	    value = super(File, self).validate(objekt, name, value, return_pathlike=True)
	            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	  File "/opt/conda/envs/fmriprep/lib/python3.11/site-packages/nipype/interfaces/base/traits_extension.py", line 135, in validate
	    self.error(objekt, name, str(value))
	  File "/opt/conda/envs/fmriprep/lib/python3.11/site-packages/traits/base_trait_handler.py", line 74, in error
	    raise TraitError(
	traits.trait_errors.TraitError: The 'out_file' trait of a MetricDilateOutputSpec instance must be a pathlike object or string representing an existing file, but a value of '/xdisk/adamraikes/PD_allo/derivatives/fmriprep_24.0.1/scratch/fmriprep_24_0_wf/sub_002_wf/hcp_morphometrics_wf/_hemi_L/dilate_curv/sub-002.L.curv.native.shape.func.gii' <class 'str'> was specified.

	During handling of the above exception, another exception occurred:

	Traceback (most recent call last):
	  File "/opt/conda/envs/fmriprep/lib/python3.11/site-packages/nipype/interfaces/base/core.py", line 400, in run
	    outputs = self.aggregate_outputs(runtime)
	              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	  File "/opt/conda/envs/fmriprep/lib/python3.11/site-packages/nipype/interfaces/base/core.py", line 460, in aggregate_outputs
	    raise FileNotFoundError(msg)
	FileNotFoundError: No such file or directory '/xdisk/adamraikes/PD_allo/derivatives/fmriprep_24.0.1/scratch/fmriprep_24_0_wf/sub_002_wf/hcp_morphometrics_wf/_hemi_L/dilate_curv/sub-002.L.curv.native.shape.func.gii' for output 'out_file' of a MetricDilate interface

Additional information / screenshots

No response

[effigies]

As to issue 1, it was not a specific goal of mine to copy precomputed files that did not need modification. It's not obvious that that is generally desirable, as it would make unnecessary copies.

For the curv problem, is this reliably reproducible? I can try to look into it. (Note, I'll have very little time to work on this for at least two weeks.)

[araikes]

@effigies

1. Understood on point number 1. I guess my follow-up to that would be what modifications are being made to those files from the sMRIPrep processing such that the T2w image gets pulled over as well as the standard space registration files? Or, is it just that those get used for the functional processing stream and therefore written to the derivatives even without modification? Given that other tools, like XCP-D take fMRIPrep outputs as inputs, is it "safe" to either copy the "missing" anat components from sMRIPrep or is there going to be missing processing that sMRIprep does?

2. I've tested it on several individuals in this dataset. I even tested dropping out the longitudinal flag from both sMRIPrep and fMRIPrep as a sanity check and it's still gives the curv problem. I'll test it in another dataset as well but haven't had a chance yet.

[effigies]

In general, fMRIPrep should do nothing on the structural side that sMRIPrep doesn't do, so it should be safe to copy from sMRIPrep to fMRIPrep. The most likely cause for making new copies of some things and not others is that something is wrong in our logic to not recompute them.