Failures when using tar to pack archives during experiment submission

[karlmutch]

Got the following error:
2018-06-15 20:18:28 INFO S3Provider - git location for experiment /home/user/.studioml/experiments/completion_service_user-xxx-Jun11-test-dir-ada68a6c-c0f1-4a34-83b9-e9f486f356d2_1093/workspace
2018-06-15 20:18:28 DEBUG S3ArtifactStore - Tarring artifact. tar_filename = /tmp/9207ea16-da90-4fbb-bbd6-28556a8fdad4, local_path = /mnt/Documents/Sentient/omniglot-2/omniglot-runner/results/user-xxx-Jun11_test_dir/snapshot.tar.gz, key = None
2018-06-15 20:18:28 DEBUG S3ArtifactStore - Tar cmd = tar -cf /tmp/9207ea16-da90-4fbb-bbd6-28556a8fdad4 -C /mnt/Documents/Sentient/omniglot-2/omniglot-runner/results/user-xxx-Jun11_test_dir snapshot.tar.gz
2018-06-15 20:18:28 INFO S3ArtifactStore - tar had a non-zero return code ! (-11)
2018-06-15 20:18:28 INFO S3ArtifactStore - tar output:

2018-06-15 20:18:28 INFO S3ArtifactStore - tar stderr output:
None
2018-06-15 20:18:28 INFO S3ArtifactStore - tar finished in 0.173414945602s
Traceback (most recent call last):
File "/home/user/Python_virtual_envs3/local/lib/python2.7/site-packages/studio/runner.py", line 443, in add_experiment
db.add_experiment(e)
File "/home/user/Python_virtual_envs3/local/lib/python2.7/site-packages/studio/keyvalue_provider.py", line 95, in add_experiment
art['key'] = self.store.put_artifact(art)
File "/home/user/Python_virtual_envs3/local/lib/python2.7/site-packages/studio/tartifact_store.py", line 120, in put_artifact

• '.tar' + compression_to_extension(self.compression)
  File "/home/user/Python_virtual_envs3/local/lib/python2.7/site-packages/studio/util.py", line 43, in sha256_checksum
  return filehash(filename, block_size, hashobj=hashlib.sha256())
  File "/home/user/Python_virtual_envs3/local/lib/python2.7/site-packages/studio/util.py", line 47, in filehash
  with open(filename, 'rb') as f:
  IOError: [Errno 2] No such file or directory: '/tmp/9207ea16-da90-4fbb-bbd6-28556a8fdad4'
  2018-06-15 20:18:28 INFO CompletionService - Studioml completion service shutting down
  Shutdown completion service with following signal: None
  Traceback (most recent call last):
  File "session_server.py", line 977, in run
  save_best=True, checkpoint_generation=1)
  File "session_server.py", line 1019, in epoch
  session_dictionary = self.process_one_epoch(session_dictionary)
  File "session_server.py", line 1041, in process_one_epoch
  population_results = self.evaluate(population)
  File "session_server.py", line 476, in evaluate
  job_id=candidate_id
  File "/mnt/Documents/Sentient/omniglot-2/omniglot-runner/experimenthost/completion_service.py", line 163, in submitTaskWithFiles
  job_id=job_id
  File "/home/user/Python_virtual_envs3/local/lib/python2.7/site-packages/studio/completion_service/completion_service.py", line 208, in submitTaskWithFiles
  queue_name=self.queue_name)
  File "/home/user/Python_virtual_envs3/local/lib/python2.7/site-packages/studio/runner.py", line 609, in submit_experiments
  experiments)]
  File "/home/user/Python_virtual_envs3/local/lib/python2.7/site-packages/studio/runner.py", line 443, in add_experiment
  db.add_experiment(e)
  File "/home/user/Python_virtual_envs3/local/lib/python2.7/site-packages/studio/keyvalue_provider.py", line 95, in add_experiment
  art['key'] = self.store.put_artifact(art)
  File "/home/user/Python_virtual_envs3/local/lib/python2.7/site-packages/studio/tartifact_store.py", line 120, in put_artifact
• '.tar' + compression_to_extension(self.compression)
  File "/home/user/Python_virtual_envs3/local/lib/python2.7/site-packages/studio/util.py", line 43, in sha256_checksum
  return filehash(filename, block_size, hashobj=hashlib.sha256())
  File "/home/user/Python_virtual_envs3/local/lib/python2.7/site-packages/studio/util.py", line 47, in filehash
  with open(filename, 'rb') as f:
  IOError: [Errno 2] No such file or directory: '/tmp/9207ea16-da90-4fbb-bbd6-28556a8fdad4'

[karlmutch]

Suspected application issue rather than a StudioML issue