fMRIPrep failed using --use-syn-sdc

[m9080212]

What happened?

Hi.
I'm preprocessing 5 subjects' data using fieldmap-less distortion correction. However, some errors show up for 3 of the subjects:
240814-19:59:58,544 nipype.workflow ERROR:
could not run node: fmriprep_24_0_wf.sub_1_wf.fmap_preproc_wf.wf_auto_00009.syn
240814-19:59:58,564 nipype.workflow ERROR:
could not run node: fmriprep_24_0_wf.sub_2_wf.fmap_preproc_wf.wf_auto_00016.syn
240814-19:59:58,575 nipype.workflow ERROR:
could not run node: fmriprep_24_0_wf.sub_2_wf.fmap_preproc_wf.wf_auto_00012.syn
240814-19:59:58,583 nipype.workflow ERROR:
could not run node: fmriprep_24_0_wf.sub_2_wf.fmap_preproc_wf.wf_auto_00018.syn
240814-19:59:58,589 nipype.workflow ERROR:
could not run node: fmriprep_24_0_wf.sub_3_wf.fmap_preproc_wf.wf_auto_00027.syn
240814-19:59:58,596 nipype.workflow ERROR:
could not run node: fmriprep_24_0_wf.sub_3_wf.fmap_preproc_wf.wf_auto_00026.syn
240814-19:59:58,603 nipype.workflow ERROR:
could not run node: fmriprep_24_0_wf.sub_3_wf.fmap_preproc_wf.wf_auto_00029.syn
240814-19:59:59,240 nipype.workflow CRITICAL:
fMRIPrep failed: 7 raised. Re-raising first.

And one of the ERROR logs is attached below.

What command did you use?

subj_start=1
subj_end=5

participant_labels=$(seq -s " " $subj_start $subj_end)  

fmriprep-docker '/media/menglab/Work2/hy/face_imagery/slow_event_related_design/BIDS' \
 '/media/menglab/Work2/hy/face_imagery/slow_event_related_design/BIDS/derivatives/fMRIprep' \
 -w '/media/menglab/Work2/hy/face_imagery/slow_event_related_design/fMRIprep_working_directory'\
  --fs-license-file '/home/menglab/freesurfer/license.txt' \
  --participant-label $participant_labels \
  --ignore slicetiming --output-spaces anat MNI152NLin2009cAsym --skull-strip-t1w force\
  --use-syn-sdc --force-syn

What version of fMRIPrep are you running?

24.0.0

How are you running fMRIPrep?

Docker

Is your data BIDS valid?

Yes

Are you reusing any previously computed results?

FreeSurfer, Anatomical derivatives, Work directory

Please copy and paste any relevant log output.

Node: fmriprep_24_0_wf.sub_1_wf.fmap_preproc_wf.wf_auto_00009.syn
Working directory: /scratch/fmriprep_24_0_wf/sub_1_wf/fmap_preproc_wf/wf_auto_00009/syn

Node inputs:

args = <undefined>
collapse_output_transforms = True
convergence_threshold = [1e-06, 1e-08]
convergence_window_size = [5, 2]
dimension = 3
environ = {'NSLOTS': '8'}
fixed_image = ['/scratch/fmriprep_24_0_wf/sub_1_wf/syn_preprocessing_auto_00009/anat2epi/clipped_noise_corrected_trans.nii.gz', '/scratch/fmriprep_24_0_wf/sub_1_wf/fmap_preproc_wf/wf_auto_00009/lap_anat_norm/clipped_noise_corrected_trans_maths_norm.nii.gz']
fixed_image_mask = <undefined>
fixed_image_masks = ['/scratch/fmriprep_24_0_wf/sub_1_wf/syn_preprocessing_auto_00009/mask2epi/sub-1_desc-brain_mask_trans_uint8.nii.gz', '/scratch/fmriprep_24_0_wf/sub_1_wf/syn_preprocessing_auto_00009/mask2epi/sub-1_desc-brain_mask_trans_uint8.nii.gz', '/scratch/fmriprep_24_0_wf/sub_1_wf/fmap_preproc_wf/wf_auto_00009/atlas_msk/fmap_atlas_trans_mask.nii.gz']
float = <undefined>
initial_moving_transform = <undefined>
initial_moving_transform_com = <undefined>
initialize_transforms_per_stage = False
interpolation = Linear
interpolation_parameters = <undefined>
invert_initial_moving_transform = <undefined>
metric = [['Mattes', 'Mattes'], ['Mattes', 'Mattes']]
metric_item_trait = <undefined>
metric_stage_trait = <undefined>
metric_weight = [[0.5, 0.5], [0.5, 0.5]]
metric_weight_item_trait = 1.0
metric_weight_stage_trait = <undefined>
moving_image = ['/scratch/fmriprep_24_0_wf/sub_1_wf/fmap_preproc_wf/wf_auto_00009/clip_epi/clipped.nii.gz', '/scratch/fmriprep_24_0_wf/sub_1_wf/fmap_preproc_wf/wf_auto_00009/lap_epi_norm/clipped_maths_norm.nii.gz']
moving_image_mask = <undefined>
moving_image_masks = ['/scratch/fmriprep_24_0_wf/sub_1_wf/fmap_preproc_wf/wf_auto_00009/epi_umask/clipped_mask_union.nii.gz', '/scratch/fmriprep_24_0_wf/sub_1_wf/fmap_preproc_wf/wf_auto_00009/epi_umask/clipped_mask_union.nii.gz', '/scratch/fmriprep_24_0_wf/sub_1_wf/fmap_preproc_wf/wf_auto_00009/epi_umask/clipped_mask_union.nii.gz']
num_threads = 8
number_of_iterations = [[200, 100], [10]]
output_inverse_warped_image = False
output_transform_prefix = fmap_syn
output_warped_image = False
radius_bins_item_trait = 5
radius_bins_stage_trait = <undefined>
radius_or_number_of_bins = [[48, 48], [48, 48]]
random_seed = <undefined>
restore_state = <undefined>
restrict_deformation = [[0.1, 1.0, 0.1], [0.1, 1.0, 0.1]]
sampling_percentage = [[0.8, 0.8], [1.0, 1.0]]
sampling_percentage_item_trait = <undefined>
sampling_percentage_stage_trait = <undefined>
sampling_strategy = [['Random', 'Random'], [None, None]]
sampling_strategy_item_trait = <undefined>
sampling_strategy_stage_trait = <undefined>
save_state = <undefined>
shrink_factors = [[1, 1], [1]]
sigma_units = ['vox', 'vox']
smoothing_sigmas = [[2.0, 0.0], [0.0]]
transform_parameters = [(0.8, 6.0, 3.0), (0.8, 2.0, 1.0)]
transforms = ['SyN', 'SyN']
use_estimate_learning_rate_once = <undefined>
use_histogram_matching = [True, True]
verbose = False
winsorize_lower_quantile = 0.001
winsorize_upper_quantile = 0.998
write_composite_transform = False

Traceback (most recent call last):
  File "/opt/conda/envs/fmriprep/lib/python3.11/site-packages/nipype/pipeline/plugins/multiproc.py", line 67, in run_node
    result["result"] = node.run(updatehash=updatehash)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/fmriprep/lib/python3.11/site-packages/nipype/pipeline/engine/nodes.py", line 527, in run
    result = self._run_interface(execute=True)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/fmriprep/lib/python3.11/site-packages/nipype/pipeline/engine/nodes.py", line 645, in _run_interface
    return self._run_command(execute)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/fmriprep/lib/python3.11/site-packages/nipype/pipeline/engine/nodes.py", line 771, in _run_command
    raise NodeExecutionError(msg)
nipype.pipeline.engine.nodes.NodeExecutionError: Exception raised while executing Node syn.

Cmdline:
	antsRegistration --collapse-output-transforms 1 --dimensionality 3 --initialize-transforms-per-stage 0 --interpolation Linear --output fmap_syn --transform SyN[ 0.8, 6.0, 3.0 ] --metric Mattes[ /scratch/fmriprep_24_0_wf/sub_1_wf/syn_preprocessing_auto_00009/anat2epi/clipped_noise_corrected_trans.nii.gz, /scratch/fmriprep_24_0_wf/sub_1_wf/fmap_preproc_wf/wf_auto_00009/clip_epi/clipped.nii.gz, 0.5, 48, Random, 0.8 ] --metric Mattes[ /scratch/fmriprep_24_0_wf/sub_1_wf/fmap_preproc_wf/wf_auto_00009/lap_anat_norm/clipped_noise_corrected_trans_maths_norm.nii.gz, /scratch/fmriprep_24_0_wf/sub_1_wf/fmap_preproc_wf/wf_auto_00009/lap_epi_norm/clipped_maths_norm.nii.gz, 0.5, 48, Random, 0.8 ] --convergence [ 200x100, 1e-06, 5 ] --smoothing-sigmas 2.0x0.0vox --shrink-factors 1x1 --use-histogram-matching 1 --restrict-deformation 0.1x1.0x0.1 --masks [ /scratch/fmriprep_24_0_wf/sub_1_wf/syn_preprocessing_auto_00009/mask2epi/sub-1_desc-brain_mask_trans_uint8.nii.gz, /scratch/fmriprep_24_0_wf/sub_1_wf/fmap_preproc_wf/wf_auto_00009/epi_umask/clipped_mask_union.nii.gz ] --transform SyN[ 0.8, 2.0, 1.0 ] --metric Mattes[ /scratch/fmriprep_24_0_wf/sub_1_wf/syn_preprocessing_auto_00009/anat2epi/clipped_noise_corrected_trans.nii.gz, /scratch/fmriprep_24_0_wf/sub_1_wf/fmap_preproc_wf/wf_auto_00009/clip_epi/clipped.nii.gz, 0.5, 48 ] --metric Mattes[ /scratch/fmriprep_24_0_wf/sub_1_wf/fmap_preproc_wf/wf_auto_00009/lap_anat_norm/clipped_noise_corrected_trans_maths_norm.nii.gz, /scratch/fmriprep_24_0_wf/sub_1_wf/fmap_preproc_wf/wf_auto_00009/lap_epi_norm/clipped_maths_norm.nii.gz, 0.5, 48 ] --convergence [ 10, 1e-08, 2 ] --smoothing-sigmas 0.0vox --shrink-factors 1 --use-histogram-matching 1 --restrict-deformation 0.1x1.0x0.1 --masks [ /scratch/fmriprep_24_0_wf/sub_1_wf/syn_preprocessing_auto_00009/mask2epi/sub-1_desc-brain_mask_trans_uint8.nii.gz, /scratch/fmriprep_24_0_wf/sub_1_wf/fmap_preproc_wf/wf_auto_00009/epi_umask/clipped_mask_union.nii.gz ] --winsorize-image-intensities [ 0.001, 0.998 ]  --write-composite-transform 0
Stdout:

Stderr:
	Killed
Traceback:
	Traceback (most recent call last):
	  File "/opt/conda/envs/fmriprep/lib/python3.11/site-packages/nipype/interfaces/base/core.py", line 453, in aggregate_outputs
	    setattr(outputs, key, val)
	  File "/opt/conda/envs/fmriprep/lib/python3.11/site-packages/traits/trait_types.py", line 2699, in validate
	    return TraitListObject(self, object, name, value)
	           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	  File "/opt/conda/envs/fmriprep/lib/python3.11/site-packages/traits/trait_list_object.py", line 582, in __init__
	    super().__init__(
	  File "/opt/conda/envs/fmriprep/lib/python3.11/site-packages/traits/trait_list_object.py", line 213, in __init__
	    super().__init__(self.item_validator(item) for item in iterable)
	  File "/opt/conda/envs/fmriprep/lib/python3.11/site-packages/traits/trait_list_object.py", line 213, in <genexpr>
	    super().__init__(self.item_validator(item) for item in iterable)
	                     ^^^^^^^^^^^^^^^^^^^^^^^^^
	  File "/opt/conda/envs/fmriprep/lib/python3.11/site-packages/traits/trait_list_object.py", line 865, in _item_validator
	    return trait_validator(object, self.name, value)
	           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	  File "/opt/conda/envs/fmriprep/lib/python3.11/site-packages/nipype/interfaces/base/traits_extension.py", line 330, in validate
	    value = super(File, self).validate(objekt, name, value, return_pathlike=True)
	            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	  File "/opt/conda/envs/fmriprep/lib/python3.11/site-packages/nipype/interfaces/base/traits_extension.py", line 135, in validate
	    self.error(objekt, name, str(value))
	  File "/opt/conda/envs/fmriprep/lib/python3.11/site-packages/traits/base_trait_handler.py", line 74, in error
	    raise TraitError(
	traits.trait_errors.TraitError: Each element of the 'forward_transforms' trait of a RegistrationOutputSpec instance must be a pathlike object or string representing an existing file, but a value of '/scratch/fmriprep_24_0_wf/sub_1_wf/fmap_preproc_wf/wf_auto_00009/syn/fmap_syn0Warp.nii.gz' <class 'str'> was specified.

	During handling of the above exception, another exception occurred:

	Traceback (most recent call last):
	  File "/opt/conda/envs/fmriprep/lib/python3.11/site-packages/nipype/interfaces/base/core.py", line 400, in run
	    outputs = self.aggregate_outputs(runtime)
	              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	  File "/opt/conda/envs/fmriprep/lib/python3.11/site-packages/nipype/interfaces/base/core.py", line 460, in aggregate_outputs
	    raise FileNotFoundError(msg)
	FileNotFoundError: No such file or directory '['/scratch/fmriprep_24_0_wf/sub_1_wf/fmap_preproc_wf/wf_auto_00009/syn/fmap_syn0Warp.nii.gz']' for output 'forward_transforms' of a FixHeaderRegistration interface

Additional information / screenshots

BTW, I ran the same command with 20 subjects before hand. And the process could never finished, showing the same FileNotFoundError for fmap_syn0Warp.nii.gz. So this time I chose only 5 subjects to test the code.

[mmockford]

I have also received the exact same errors:

240826-23:31:38,936 nipype.workflow ERROR:
	 could not run node: fmriprep_24_0_wf.sub_CRPS01_wf.bold_task_illusion_run_3_wf.bold_native_wf.bold_stc_wf.slice_timing_correction
240826-23:31:38,948 nipype.workflow ERROR:
	 could not run node: fmriprep_24_0_wf.sub_CRPS01_wf.bold_task_illusion_run_1_wf.bold_fit_wf.bold_hmc_wf.mcflirt
240826-23:31:38,955 nipype.workflow ERROR:
	 could not run node: fmriprep_24_0_wf.sub_CRPS01_wf.bold_task_illusion_run_2_wf.bold_fit_wf.bold_hmc_wf.mcflirt
240826-23:31:38,961 nipype.workflow ERROR:
	 could not run node: fmriprep_24_0_wf.sub_CRPS01_wf.bold_task_illusion_run_3_wf.bold_fit_wf.bold_hmc_wf.mcflirt
240826-23:31:38,966 nipype.workflow ERROR:
	 could not run node: fmriprep_24_0_wf.sub_CRPS01_wf.bold_task_illusion_run_4_wf.bold_fit_wf.bold_hmc_wf.mcflirt
240826-23:31:38,971 nipype.workflow ERROR:
	 could not run node: fmriprep_24_0_wf.sub_CRPS01_wf.bold_task_rest_wf.bold_fit_wf.bold_hmc_wf.mcflirt
240826-23:31:38,975 nipype.workflow ERROR:
	 could not run node: fmriprep_24_0_wf.sub_CRPS01_wf.bold_task_rest_wf.bold_native_wf.bold_stc_wf.slice_timing_correction
240826-23:31:39,242 nipype.workflow CRITICAL:
	 fMRIPrep failed: 7 raised. Re-raising first.

The command I ran was:

fmriprep-docker /Volumes/LaCie/Lewis_ARUK_data/CRPS/HILLUS_CRPS01/BIDS --skip-bids-validation --fs-license-file /Applications/freesurfer/license.txt --mem-mb 7500

[m9080212]

It seems that the problem is related to parallel processing or something. If I run 2 subjects at a time, fMRIPrep will finish without errors.