Tensorflow/ dataset.cache Error

[santiaci325]

Hi! I am trying to train a dataset on user labeled frames across four videos, and for some reason I get this error that stops training:

    [[{{node EagerPyFunc}}]] [Op:IteratorGetNext]

2022-12-21 22:20:31.192184: W tensorflow/core/kernels/data/cache_dataset_ops.cc:768] The calling iterator did not fully read the dataset being cached. In order to avoid unexpected truncation of the dataset, the partially cached contents of the dataset will be discarded. This can happen if you have an input pipeline similar to dataset.cache().take(k).repeat(). You should use dataset.take(k).cache().repeat() instead.
INFO:sleap.nn.callbacks:Closing the reporter controller/context.
INFO:sleap.nn.callbacks:Closing the training controller socket/context.
Run Path: Z:/Chiaki/Data/SLEAP/tracktake3\models\221221_222015.single_instance.n=120

Since I am using the GUI, I am not entirely sure how to fix this problem, could someone help me with this? :) Thanks!

[roomrys]

Hi @santiaci325,

Do you mind posting the full error message (starting a bit before the Traceback(most recent call last)). The bit of information I need is just above the message you sent, but it would be helpful to have the full message.

Usually this occurs when there is a mismatch in sizing between what the model expects and what is given (which could be a color/grayscale mismatch).

Thanks,
Liezl

[santiaci325]

Hi!

Here is the full message!:

   "filename": "C:\\Users\\Chiaki\\AppData\\Local\\Temp\\tmp49ppfxxe\\221221_222015_training_job.json"
}
INFO:sleap.nn.training:
2022-12-21 22:20:22.848778: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'nvcuda.dll'; dlerror: nvcuda.dll not found
2022-12-21 22:20:22.848926: W tensorflow/stream_executor/cuda/cuda_driver.cc:269] failed call to cuInit: UNKNOWN ERROR (303)
2022-12-21 22:20:22.852496: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:169] retrieving CUDA diagnostic information for host: Chiaki-PC
2022-12-21 22:20:22.852746: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:176] hostname: Chiaki-PC
INFO:sleap.nn.training:Running in CPU-only mode.
INFO:sleap.nn.training:System:
GPUs: None detected.
INFO:sleap.nn.training:
INFO:sleap.nn.training:Initializing trainer...
INFO:sleap.nn.training:Loading training labels from: Z:/Chiaki/Data/SLEAP/tracktake3/labels.slp
INFO:sleap.nn.training:Creating training and validation splits from validation fraction: 0.1
INFO:sleap.nn.training:  Splits: Training = 108 / Validation = 12.
INFO:sleap.nn.training:Setting up for training...
INFO:sleap.nn.training:Setting up pipeline builders...
INFO:sleap.nn.training:Setting up model...
INFO:sleap.nn.training:Building test pipeline...
2022-12-21 22:20:24.833577: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX AVX2
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2022-12-21 22:20:25.494111: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:185] None of the MLIR Optimization Passes are enabled (registered 2)
INFO:sleap.nn.training:Loaded test example. [2.434s]
INFO:sleap.nn.training:  Input shape: (1088, 1440, 1)
INFO:sleap.nn.training:Created Keras model.
INFO:sleap.nn.training:  Backbone: UNet(stacks=1, filters=16, filters_rate=2.0, kernel_size=3, stem_kernel_size=7, convs_per_block=2, stem_blocks=0, down_blocks=4, middle_block=True, up_blocks=3, up_interpolate=True, block_contraction=False)
INFO:sleap.nn.training:  Max stride: 16
INFO:sleap.nn.training:  Parameters: 1,953,237
INFO:sleap.nn.training:  Heads:
INFO:sleap.nn.training:    [0] = SingleInstanceConfmapsHead(part_names=['left_ear', 'right_ear', 'nose', 'hind', 'middle'], sigma=2.5, output_stride=2, loss_weight=1.0)
INFO:sleap.nn.training:  Outputs:
INFO:sleap.nn.training:    [0] = KerasTensor(type_spec=TensorSpec(shape=(None, 544, 720, 5), dtype=tf.float32, name=None), name='SingleInstanceConfmapsHead/BiasAdd:0', description="created by layer 'SingleInstanceConfmapsHead'")
INFO:sleap.nn.training:Setting up data pipelines...
INFO:sleap.nn.training:Training set: n = 108
INFO:sleap.nn.training:Validation set: n = 11
INFO:sleap.nn.training:Setting up optimization...
INFO:sleap.nn.training:  Learning rate schedule: LearningRateScheduleConfig(reduce_on_plateau=True, reduction_factor=0.5, plateau_min_delta=1e-06, plateau_patience=5, plateau_cooldown=3, min_learning_rate=1e-08)
INFO:sleap.nn.training:  Early stopping: EarlyStoppingConfig(stop_training_on_plateau=True, plateau_min_delta=1e-08, plateau_patience=10)
INFO:sleap.nn.training:Setting up outputs...
INFO:sleap.nn.callbacks:Training controller subscribed to: tcp://127.0.0.1:9000 (topic: )
INFO:sleap.nn.training:  ZMQ controller subcribed to: tcp://127.0.0.1:9000
INFO:sleap.nn.callbacks:Progress reporter publishing on: tcp://127.0.0.1:9001 for: not_set
INFO:sleap.nn.training:  ZMQ progress reporter publish on: tcp://127.0.0.1:9001
INFO:sleap.nn.training:Created run path: Z:/Chiaki/Data/SLEAP/tracktake3\models\221221_222015.single_instance.n=120
INFO:sleap.nn.training:Setting up visualization...
INFO:sleap.nn.training:Finished trainer set up. [3.4s]
INFO:sleap.nn.training:Creating tf.data.Datasets for training data generation...
2022-12-21 22:20:31.046310: W tensorflow/core/framework/op_kernel.cc:1680] Unknown: KeyError: "Unable to load frame 50374 from MediaVideo(filename='Z:/Chiaki/Data/SLEAP/tracktake3/03992.mp4', grayscale=True, bgr=True, dataset='', input_format='')."
Traceback (most recent call last):

  File "C:\Users\Chiaki\miniconda3\envs\sleap\lib\site-packages\tensorflow\python\ops\script_ops.py", line 247, in __call__
    return func(device, token, args)

  File "C:\Users\Chiaki\miniconda3\envs\sleap\lib\site-packages\tensorflow\python\ops\script_ops.py", line 135, in __call__
    ret = self._func(*args)

  File "C:\Users\Chiaki\miniconda3\envs\sleap\lib\site-packages\tensorflow\python\autograph\impl\api.py", line 645, in wrapper
    return func(*args, **kwargs)

  File "C:\Users\Chiaki\miniconda3\envs\sleap\lib\site-packages\sleap\nn\data\providers.py", line 198, in py_fetch_lf
    raw_image = lf.image

  File "C:\Users\Chiaki\miniconda3\envs\sleap\lib\site-packages\sleap\instance.py", line 1765, in image
    return self.video.get_frame(self.frame_idx)

  File "C:\Users\Chiaki\miniconda3\envs\sleap\lib\site-packages\sleap\io\video.py", line 1087, in get_frame
    return self.backend.get_frame(idx)

  File "C:\Users\Chiaki\miniconda3\envs\sleap\lib\site-packages\sleap\io\video.py", line 492, in get_frame
    raise KeyError(f"Unable to load frame {idx} from {self}.")

KeyError: "Unable to load frame 50374 from MediaVideo(filename='Z:/Chiaki/Data/SLEAP/tracktake3/03992.mp4', grayscale=True, bgr=True, dataset='', input_format='')."


Traceback (most recent call last):
  File "C:\Users\Chiaki\miniconda3\envs\sleap\Scripts\sleap-train-script.py", line 33, in <module>
    sys.exit(load_entry_point('sleap==1.2.9', 'console_scripts', 'sleap-train')())
  File "C:\Users\Chiaki\miniconda3\envs\sleap\lib\site-packages\sleap\nn\training.py", line 1981, in main
    trainer.train()
  File "C:\Users\Chiaki\miniconda3\envs\sleap\lib\site-packages\sleap\nn\training.py", line 914, in train
    training_ds = self.training_pipeline.make_dataset()
  File "C:\Users\Chiaki\miniconda3\envs\sleap\lib\site-packages\sleap\nn\data\pipelines.py", line 287, in make_dataset
    ds = transformer.transform_dataset(ds)
  File "C:\Users\Chiaki\miniconda3\envs\sleap\lib\site-packages\sleap\nn\data\dataset_ops.py", line 318, in transform_dataset
    self.examples = list(iter(ds))
  File "C:\Users\Chiaki\miniconda3\envs\sleap\lib\site-packages\tensorflow\python\data\ops\iterator_ops.py", line 761, in __next__
    return self._next_internal()
  File "C:\Users\Chiaki\miniconda3\envs\sleap\lib\site-packages\tensorflow\python\data\ops\iterator_ops.py", line 747, in _next_internal
    output_shapes=self._flat_output_shapes)
  File "C:\Users\Chiaki\miniconda3\envs\sleap\lib\site-packages\tensorflow\python\ops\gen_dataset_ops.py", line 2727, in iterator_get_next
    _ops.raise_from_not_ok_status(e, name)
  File "C:\Users\Chiaki\miniconda3\envs\sleap\lib\site-packages\tensorflow\python\framework\ops.py", line 6941, in raise_from_not_ok_status
    six.raise_from(core._status_to_exception(e.code, message), None)
  File "<string>", line 3, in raise_from
tensorflow.python.framework.errors_impl.UnknownError: KeyError: "Unable to load frame 50374 from MediaVideo(filename='Z:/Chiaki/Data/SLEAP/tracktake3/03992.mp4', grayscale=True, bgr=True, dataset='', input_format='')."
Traceback (most recent call last):

  File "C:\Users\Chiaki\miniconda3\envs\sleap\lib\site-packages\tensorflow\python\ops\script_ops.py", line 247, in __call__
    return func(device, token, args)

  File "C:\Users\Chiaki\miniconda3\envs\sleap\lib\site-packages\tensorflow\python\ops\script_ops.py", line 135, in __call__
    ret = self._func(*args)

  File "C:\Users\Chiaki\miniconda3\envs\sleap\lib\site-packages\tensorflow\python\autograph\impl\api.py", line 645, in wrapper
    return func(*args, **kwargs)

  File "C:\Users\Chiaki\miniconda3\envs\sleap\lib\site-packages\sleap\nn\data\providers.py", line 198, in py_fetch_lf
    raw_image = lf.image

  File "C:\Users\Chiaki\miniconda3\envs\sleap\lib\site-packages\sleap\instance.py", line 1765, in image
    return self.video.get_frame(self.frame_idx)

  File "C:\Users\Chiaki\miniconda3\envs\sleap\lib\site-packages\sleap\io\video.py", line 1087, in get_frame
    return self.backend.get_frame(idx)

  File "C:\Users\Chiaki\miniconda3\envs\sleap\lib\site-packages\sleap\io\video.py", line 492, in get_frame
    raise KeyError(f"Unable to load frame {idx} from {self}.")

KeyError: "Unable to load frame 50374 from MediaVideo(filename='Z:/Chiaki/Data/SLEAP/tracktake3/03992.mp4', grayscale=True, bgr=True, dataset='', input_format='')."

[roomrys]

Hi @santiaci325,

Hmm this seems to be a different error message which occurs when the video is not reliably seekable. Try reencoding your video and let us know how that works.

Thanks,
Liezl

P.S. Also linking similar problems here for reference (namely Unable to load frame #366, KeyError when trying to run inference #630, Encoding Multiple Videos using Batch File #710, and KeyError "Unable to load frame" causing SLEAP to crash when predicting #765 ).