Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Analyzer workitems failing (ndiop_1993 and chonyi_1999) #63

Open
krosenfeld-IDM opened this issue Jul 20, 2023 · 1 comment
Open

Analyzer workitems failing (ndiop_1993 and chonyi_1999) #63

krosenfeld-IDM opened this issue Jul 20, 2023 · 1 comment

Comments

@krosenfeld-IDM
Copy link

I installed the package and ran the simulations using the snakemake workflow

pip install -e . -i https://packages.idmod.org/api/pypi/pypi-production/simple
cd simulations
snakemake -j

The simulations seem to have run successfully, but two analyzers, ndiop_1993 and chonyi_1999 are failing with the similar error messages:

Error message:
Errors encountered during file download, directory creation, and/or symlink processing. The most common cause of this error is the use of long file paths and/or names. See logs for more details.
CorrelationId: 08dd7c56-9ad5-4a03-9e3a-13dade66b63e

I am not sure which log would contain more details. There is no output associated with the workitem. My COMPS_log.log contains:

07/20 09:29:52 [DEBUG] [140120075540288] [COMPS.Data.SerializableEntity] OutputFileMetadata --> {'Id': '_internal_id', 'Length': 'length', 'FriendlyName': 'friendly_name', 'PathFromRoot': 'path_from_root', 'Url': 'url', 'MimeType': 'mime_type'}
07/20 09:29:52 [DEBUG] [140120075540288] [COMPS.Data.SerializableEntity] AssetFile --> {'FileName': 'file_name', 'MD5Checksum': 'md5_checksum', 'Length': 'length', 'Uri': 'uri'}
07/20 09:29:52 [DEBUG] [140120075540288] [COMPS.Data.SerializableEntity] AssetCollectionFile --> {'RelativePath': 'relative_path', 'Tags': 'tags', 'FileName': 'file_name', 'MD5Checksum': 'md5_checksum', 'Length': 'length', 'Uri': 'uri'}
07/20 09:29:52 [DEBUG] [140120075540288] [COMPS.Data.SerializableEntity] AssetCollection --> {'Id': 'id', 'DateCreated': 'date_created', 'Tags': 'tags', 'Assets': 'assets'}
07/20 09:29:52 [DEBUG] [140120075540288] [COMPS.Data.SerializableEntity] Configuration --> {'EnvironmentName': 'environment_name', 'SimulationInputArgs': 'simulation_input_args', 'WorkingDirectoryRoot': 'working_directory_root', 'ExecutablePath': 'executable_path', 'NodeGroupName': 'node_group_name', 'MaximumNumberOfRetries': 'maximum_number_of_retries', 'Priority': 'priority', 'MinCores': 'min_cores', 'MaxCores': 'max_cores', 'Exclusive': 'exclusive', 'AssetCollectionId': 'asset_collection_id'}
07/20 09:29:52 [DEBUG] [140120075540288] [COMPS.Data.SerializableEntity] SimulationFile --> {'FileType': 'file_type', 'Description': 'description', 'FileName': 'file_name', 'MD5Checksum': 'md5_checksum', 'Length': 'length', 'Uri': 'uri'}
07/20 09:29:52 [DEBUG] [140120075540288] [COMPS.Data.SerializableEntity] WorkItemFile --> {'FileType': 'file_type', 'Description': 'description', 'FileName': 'file_name', 'MD5Checksum': 'md5_checksum', 'Length': 'length', 'Uri': 'uri'}
07/20 09:29:52 [DEBUG] [140120075540288] [COMPS.Data.SerializableEntity] HpcJob --> {'Id': '_internal_id', 'JobId': 'job_id', 'JobState': 'job_state', 'Priority': 'priority', 'WorkingDirectory': 'working_directory', 'OutputDirectorySize': 'output_directory_size', 'SubmitTime': 'submit_time', 'StartTime': 'start_time', 'EndTime': 'end_time', 'ErrorMessage': 'error_message', 'Configuration': 'configuration'}
07/20 09:29:52 [DEBUG] [140120075540288] [COMPS.Data.SerializableEntity] Simulation --> {'Id': 'id', 'ExperimentId': 'experiment_id', 'Name': 'name', 'Description': 'description', 'Owner': 'owner', 'DateCreated': 'date_created', 'LastModified': 'last_modified', 'SimulationState': 'state', 'ErrorMessage': 'error_message', 'Tags': 'tags', 'Configuration': 'configuration', 'Files': 'files', 'HPCJobs': 'hpc_jobs'}
07/20 09:29:52 [DEBUG] [140120075540288] [COMPS.Data.SerializableEntity] Experiment --> {'Id': 'id', 'SuiteId': 'suite_id', 'Name': 'name', 'Description': 'description', 'Owner': 'owner', 'DateCreated': 'date_created', 'LastModified': 'last_modified', 'Tags': 'tags', 'Configuration': 'configuration'}
07/20 09:29:52 [DEBUG] [140120075540288] [COMPS.Data.SerializableEntity] Suite --> {'Id': 'id', 'Name': 'name', 'Description': 'description', 'Owner': 'owner', 'DateCreated': 'date_created', 'LastModified': 'last_modified', 'Tags': 'tags', 'Configuration': 'configuration'}
07/20 09:29:52 [DEBUG] [140120075540288] [COMPS.Data.SerializableEntity] WorkItem --> {'Id': 'id', 'Name': 'name', 'Worker': 'worker', 'EnvironmentName': 'environment_name', 'Description': 'description', 'Owner': 'owner', 'DateCreated': 'date_created', 'LastModified': 'last_modified', 'State': 'state', 'ErrorMessage': 'error_message', 'HostName': 'host_name', 'WorkerInstanceId': 'worker_instance_id', 'Priority': 'priority', 'WorkingDirectory': 'working_directory', 'WorkingDirectorySize': 'working_directory_size', 'AssetCollectionId': 'asset_collection_id', 'Tags': 'tags', 'Files': 'files', 'Plugins': 'plugins'}
07/20 09:29:53 [DEBUG] [140120075540288] [COMPS.Client] Parsed login host: ('https', 'comps.idmod.org', None)
07/20 09:29:53 [DEBUG] [140120075540288] [COMPS.Client] REQUEST -> GET https://comps.idmod.org/api/Experiments/35911bda-5526-ee11-aa08-b88303911bc1 {'fields': 'Id,Name,SuiteId', 'children': 'Tags,Configuration'}
07/20 09:29:53 [DEBUG] [140120075540288] [COMPS.Client] REQUEST -> GET https://comps.idmod.org/api/Suites/b805d021-5626-ee11-aa08-b88303911bc1 {'fields': 'Id,Name', 'children': 'Tags,Configuration'}
07/20 09:29:53 [DEBUG] [140120075540288] [COMPS.Client] REQUEST -> GET https://comps.idmod.org/api/Experiments {'filters': 'SuiteId=b805d021-5626-ee11-aa08-b88303911bc1'}
07/20 09:29:53 [DEBUG] [140120075540288] [COMPS.Client] REQUEST -> GET https://comps.idmod.org/api/Simulations {'fields': 'Id,Name,ExperimentId,SimulationState', 'children': 'Tags,Files,Configuration', 'filters': 'ExperimentId=26f723cc-5526-ee11-aa08-b88303911bc1', 'count': 100000}
07/20 09:29:53 [DEBUG] [140120075540288] [COMPS.Client] REQUEST -> GET https://comps.idmod.org/api/Experiments/26f723cc-5526-ee11-aa08-b88303911bc1 {'children': 'Configuration'}
07/20 09:29:53 [DEBUG] [140120075540288] [COMPS.Client] REQUEST -> GET https://comps.idmod.org/api/Simulations {'fields': 'Id,Name,ExperimentId,SimulationState', 'children': 'Tags,Files,Configuration', 'filters': 'ExperimentId=27f723cc-5526-ee11-aa08-b88303911bc1', 'count': 100000}
07/20 09:29:53 [DEBUG] [140120075540288] [COMPS.Client] REQUEST -> GET https://comps.idmod.org/api/Experiments/27f723cc-5526-ee11-aa08-b88303911bc1 {'children': 'Configuration'}
07/20 09:29:53 [DEBUG] [140120075540288] [COMPS.Client] REQUEST -> GET https://comps.idmod.org/api/Simulations {'fields': 'Id,Name,ExperimentId,SimulationState', 'children': 'Tags,Files,Configuration', 'filters': 'ExperimentId=28f723cc-5526-ee11-aa08-b88303911bc1', 'count': 100000}
07/20 09:29:53 [DEBUG] [140120075540288] [COMPS.Client] REQUEST -> GET https://comps.idmod.org/api/Experiments/28f723cc-5526-ee11-aa08-b88303911bc1 {'children': 'Configuration'}
07/20 09:29:53 [DEBUG] [140120075540288] [COMPS.Client] REQUEST -> GET https://comps.idmod.org/api/Simulations {'fields': 'Id,Name,ExperimentId,SimulationState', 'children': 'Tags,Files,Configuration', 'filters': 'ExperimentId=29f723cc-5526-ee11-aa08-b88303911bc1', 'count': 100000}
07/20 09:29:53 [DEBUG] [140120075540288] [COMPS.Client] REQUEST -> GET https://comps.idmod.org/api/Experiments/29f723cc-5526-ee11-aa08-b88303911bc1 {'children': 'Configuration'}
07/20 09:29:53 [DEBUG] [140120075540288] [COMPS.Client] REQUEST -> GET https://comps.idmod.org/api/Simulations {'fields': 'Id,Name,ExperimentId,SimulationState', 'children': 'Tags,Files,Configuration', 'filters': 'ExperimentId=d38824d3-5526-ee11-aa08-b88303911bc1', 'count': 100000}
07/20 09:29:53 [DEBUG] [140120075540288] [COMPS.Client] REQUEST -> GET https://comps.idmod.org/api/Experiments/d38824d3-5526-ee11-aa08-b88303911bc1 {'children': 'Configuration'}

I am working on a RHEL8 machine with conda environment:

# packages in environment at /home/krosenfeld/playpen/malaria-model_validation/env:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       2_gnu    conda-forge
appdirs                   1.4.4                    pypi_0    pypi
astor                     0.8.1                    pypi_0    pypi
astunparse                1.6.3                    pypi_0    pypi
attrs                     23.1.0                   pypi_0    pypi
backoff                   2.2.1                    pypi_0    pypi
beautifulsoup4            4.12.2                   pypi_0    pypi
bs4                       0.0.1                    pypi_0    pypi
bzip2                     1.0.8                h7f98852_4    conda-forge
ca-certificates           2023.5.7             hbcca054_0    conda-forge
certifi                   2023.5.7                 pypi_0    pypi
cffi                      1.15.1                   pypi_0    pypi
charset-normalizer        3.2.0                    pypi_0    pypi
click                     7.1.2                    pypi_0    pypi
coloredlogs               15.0.1                   pypi_0    pypi
configargparse            1.5.5                    pypi_0    pypi
connection-pool           0.0.3                    pypi_0    pypi
contourpy                 1.1.0                    pypi_0    pypi
cryptography              41.0.2                   pypi_0    pypi
cycler                    0.11.0                   pypi_0    pypi
datar                     0.12.2                   pypi_0    pypi
datrie                    0.8.2                    pypi_0    pypi
defusedxml                0.7.1                    pypi_0    pypi
diot                      0.2.2                    pypi_0    pypi
diskcache                 5.4.0                    pypi_0    pypi
docopt                    0.6.2                    pypi_0    pypi
docutils                  0.20.1                   pypi_0    pypi
dpath                     2.1.6                    pypi_0    pypi
emod-api                  1.29.7                   pypi_0    pypi
emod-malaria              0.0.58                   pypi_0    pypi
emodpy                    1.21.1                   pypi_0    pypi
emodpy-malaria            2.4.3                    pypi_0    pypi
executing                 1.2.0                    pypi_0    pypi
fastjsonschema            2.17.1                   pypi_0    pypi
filelock                  3.12.2                   pypi_0    pypi
fonttools                 4.41.0                   pypi_0    pypi
fpdf2                     2.7.4                    pypi_0    pypi
future                    0.18.3                   pypi_0    pypi
geographiclib             2.0                      pypi_0    pypi
gitdb                     4.0.10                   pypi_0    pypi
gitpython                 3.1.32                   pypi_0    pypi
humanfriendly             10.0                     pypi_0    pypi
idmtools                  1.7.8                    pypi_0    pypi
idmtools-models           1.7.8                    pypi_0    pypi
idmtools-platform-comps   1.7.8                    pypi_0    pypi
idna                      3.4                      pypi_0    pypi
importlib-metadata        6.8.0                    pypi_0    pypi
importlib-resources       6.0.0                    pypi_0    pypi
inflection                0.5.1                    pypi_0    pypi
jaraco-classes            3.3.0                    pypi_0    pypi
jeepney                   0.8.0                    pypi_0    pypi
jinja2                    3.1.2                    pypi_0    pypi
joblib                    1.3.1                    pypi_0    pypi
jsonschema                4.18.4                   pypi_0    pypi
jsonschema-specifications 2023.7.1                 pypi_0    pypi
jupyter-core              5.3.1                    pypi_0    pypi
keyring                   24.2.0                   pypi_0    pypi
kiwisolver                1.4.4                    pypi_0    pypi
ld_impl_linux-64          2.40                 h41732ed_0    conda-forge
libffi                    3.4.2                h7f98852_5    conda-forge
libgcc-ng                 13.1.0               he5830b7_0    conda-forge
libgomp                   13.1.0               he5830b7_0    conda-forge
libnsl                    2.0.0                h7f98852_0    conda-forge
libsqlite                 3.42.0               h2797004_0    conda-forge
libuuid                   2.38.1               h0b41bf4_0    conda-forge
libzlib                   1.2.13               hd590300_5    conda-forge
lz4                       4.3.2                    pypi_0    pypi
malaria-model-validation  1.2.0                    pypi_0    pypi
markupsafe                2.1.3                    pypi_0    pypi
matplotlib                3.7.2                    pypi_0    pypi
mizani                    0.9.2                    pypi_0    pypi
more-itertools            9.0.0                    pypi_0    pypi
natsort                   8.2.0                    pypi_0    pypi
nbformat                  5.9.1                    pypi_0    pypi
ncurses                   6.4                  hcb278e6_0    conda-forge
numpy                     1.25.1                   pypi_0    pypi
openssl                   3.1.1                hd590300_1    conda-forge
packaging                 21.3                     pypi_0    pypi
pandas                    2.0.3                    pypi_0    pypi
parse                     1.19.1                   pypi_0    pypi
patsy                     0.5.3                    pypi_0    pypi
pillow                    10.0.0                   pypi_0    pypi
pip                       23.2               pyhd8ed1ab_0    conda-forge
pipda                     0.12.0                   pypi_0    pypi
pipreqs                   0.4.13                   pypi_0    pypi
plac                      1.3.5                    pypi_0    pypi
platformdirs              3.9.1                    pypi_0    pypi
plotnine                  0.12.1                   pypi_0    pypi
pluggy                    1.0.0                    pypi_0    pypi
prodict                   0.8.18                   pypi_0    pypi
psutil                    5.9.5                    pypi_0    pypi
pulp                      2.7.0                    pypi_0    pypi
pycomps                   2.10.0                   pypi_0    pypi
pycparser                 2.21                     pypi_0    pypi
pygit2                    1.10.1                   pypi_0    pypi
pyparsing                 3.0.9                    pypi_0    pypi
pyproj                    3.6.0                    pypi_0    pypi
python                    3.9.16          h2782a2a_0_cpython    conda-forge
@krosenfeld-IDM
Copy link
Author

Came back a few days later and tried rerunning:

snakemake -j

The chonyi_1999 work item and the ndiop_1993 work item ran this time. I didn't make any changes to files. I guess there is something buggy about how some temporary files or linking is going on?

@krosenfeld-IDM krosenfeld-IDM mentioned this issue Jul 25, 2023
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@krosenfeld-IDM