Using Training Models on Other Videos

[Jaspermoray]

Hi all,
I have a question about using training models on different videos: If a new video is added to a project that has a fully predicted video in it (with accompanying training model), can you simply use the inference feature to run that model on the new video? Or must the new vide be trained again from scratch?

I tried inferring frames on a new video using an existing model in the same project, but it didn't work. Maybe I missed something though?

Best,
Jasper Coles Hood

---

Continued from / duplicate of #1166

Happy SLEAPing! :)
Using already trained model for centroid: /Volumes/JasperCray/models/230208_151438.centroid.n=55/initial_config.json
Using already trained model for centered_instance: /Volumes/JasperCray/models/230208_150601.centered_instance.n=55/initial_config.json
Command line call:
sleap-track /Volumes/JasperCray/33instances.v002.slp --video.index 1 --frames 129,323,261,390,9,151,343,88,90,27,160,166,167,298,107,109,366,304,182,315 -m /Volumes/JasperCray/models/230208_151438.centroid.n=55/initial_config.json -m /Volumes/JasperCray/models/230208_150601.centered_instance.n=55/initial_config.json --tracking.tracker none -o /Volumes/JasperCray/predictions/33instances.v002.slp.230208_164537.predictions.slp --verbosity json --no-empty-frames

Started inference at: 2023-02-08 16:45:41.369169
Args:
{
│   'data_path': '/Volumes/JasperCray/33instances.v002.slp',
│   'models': [
│   │   '/Volumes/JasperCray/models/230208_151438.centroid.n=55/initial_config.json',
│   │   '/Volumes/JasperCray/models/230208_150601.centered_instance.n=55/initial_config.json'
│   ],
│   'frames': '129,323,261,390,9,151,343,88,90,27,160,166,167,298,107,109,366,304,182,315',
│   'only_labeled_frames': False,
│   'only_suggested_frames': False,
2023-02-08 16:45:41.926891: I tensorflow/core/common_runtime/pluggable_device/pluggable_device_factory.cc:305] Could not identify NUMA node of platform GPU ID 0, defaulting to 0. Your kernel may not have been built with NUMA support.
2023-02-08 16:45:41.927032: I tensorflow/core/common_runtime/pluggable_device/pluggable_device_factory.cc:271] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 0 MB memory) -> physical PluggableDevice (device: 0, name: METAL, pci bus id: <undefined>)
│   'output': '/Volumes/JasperCray/predictions/33instances.v002.slp.230208_164537.predictions.slp',
│   'no_empty_frames': True,
│   'verbosity': 'json',
│   'video.dataset': None,
│   'video.input_format': 'channels_last',
│   'video.index': '1',
│   'cpu': False,
│   'first_gpu': False,
│   'last_gpu': False,
│   'gpu': 'auto',
│   'max_edge_length_ratio': 0.25,
│   'dist_penalty_weight': 1.0,
│   'batch_size': 4,
│   'open_in_gui': False,
│   'peak_threshold': 0.2,
│   'tracking.tracker': 'none',
│   'tracking.target_instance_count': None,
│   'tracking.pre_cull_to_target': None,
│   'tracking.pre_cull_iou_threshold': None,
2023-02-08 16:45:42.907327: W tensorflow/core/platform/profile_utils/cpu_utils.cc:128] Failed to get CPU frequency: 0 Hz
│   'tracking.post_connect_single_breaks': None,
│   'tracking.clean_instance_count': None,
│   'tracking.clean_iou_threshold': None,
│   'tracking.similarity': None,
│   'tracking.match': None,
│   'tracking.robust': None,
│   'tracking.track_window': None,
│   'tracking.min_new_track_points': None,
│   'tracking.min_match_points': None,
│   'tracking.img_scale': None,
│   'tracking.of_window_size': None,
│   'tracking.of_max_levels': None,
│   'tracking.save_shifted_instances': None,
│   'tracking.kf_node_indices': None,
│   'tracking.kf_init_frame_count': None
}

INFO:sleap.nn.inference:Failed to query GPU memory from nvidia-smi. Defaulting to first GPU.
Metal device set to: Apple M1 Pro
2023-02-08 16:45:45.375250: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:113] Plugin optimizer for device_type GPU is enabled.
2023-02-08 16:45:45.449978: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: -45 } dim { size: -46 } dim { size: -47 } dim { size: 1 } } } inputs { dtype: DT_FLOAT shape { dim { size: -15 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -15 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" model: "0" num_cores: 10 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -15 } dim { size: -48 } dim { size: -49 } dim { size: 1 } } }
2023-02-08 16:45:45.450212: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_UINT8 } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_UINT8 shape { dim { size: 4 } dim { size: 1080 } dim { size: 1920 } dim { size: 3 } } } inputs { dtype: DT_FLOAT shape { dim { size: -15 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -15 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" model: "0" num_cores: 10 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -15 } dim { size: -56 } dim { size: -57 } dim { size: 3 } } }
2023-02-08 16:45:45.453203: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: -91 } dim { size: -92 } dim { size: -93 } dim { size: 1 } } } inputs { dtype: DT_FLOAT shape { dim { size: -20 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -20 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" model: "0" num_cores: 10 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -20 } dim { size: -95 } dim { size: -96 } dim { size: 1 } } }
Traceback (most recent call last):
  File "/Users/jasperhood/miniconda3/envs/sleap/bin/sleap-track", line 33, in <module>
Versions:
    sys.exit(load_entry_point('sleap', 'console_scripts', 'sleap-track')())
  File "/Users/jasperhood/sleap/sleap/nn/inference.py", line 5229, in main
    labels_pr = predictor.predict(provider)
  File "/Users/jasperhood/sleap/sleap/nn/inference.py", line 515, in predict
    self._make_labeled_frames_from_generator(generator, data)
  File "/Users/jasperhood/sleap/sleap/nn/inference.py", line 2519, in _make_labeled_frames_from_generator
    for ex in generator:
  File "/Users/jasperhood/sleap/sleap/nn/inference.py", line 446, in _predict_generator
    ex = process_batch(ex)
  File "/Users/jasperhood/sleap/sleap/nn/inference.py", line 388, in process_batch
    preds = self.inference_model.predict_on_batch(ex, numpy=True)
  File "/Users/jasperhood/sleap/sleap/nn/inference.py", line 1059, in predict_on_batch
    outs = super().predict_on_batch(data, **kwargs)
  File "/Users/jasperhood/miniconda3/envs/sleap/lib/python3.9/site-packages/keras/engine/training.py", line 2230, in predict_on_batch
    outputs = self.predict_function(iterator)
  File "/Users/jasperhood/miniconda3/envs/sleap/lib/python3.9/site-packages/tensorflow/python/util/traceback_utils.py", line 153, in error_handler
    raise e.with_traceback(filtered_tb) from None
  File "/Users/jasperhood/miniconda3/envs/sleap/lib/python3.9/site-packages/tensorflow/python/eager/execute.py", line 54, in quick_execute
    tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
tensorflow.python.framework.errors_impl.InternalError: Graph execution error:

Detected at node 'top_down_inference_model/find_instance_peaks/model/stack0_dec0_s16_to_s8_interp_bilinear/resize/ResizeBilinear' defined at (most recent call last):
    File "/Users/jasperhood/miniconda3/envs/sleap/bin/sleap-track", line 33, in <module>
      sys.exit(load_entry_point('sleap', 'console_scripts', 'sleap-track')())
    File "/Users/jasperhood/sleap/sleap/nn/inference.py", line 5229, in main
      labels_pr = predictor.predict(provider)
    File "/Users/jasperhood/sleap/sleap/nn/inference.py", line 515, in predict
      self._make_labeled_frames_from_generator(generator, data)
    File "/Users/jasperhood/sleap/sleap/nn/inference.py", line 2519, in _make_labeled_frames_from_generator
      for ex in generator:
    File "/Users/jasperhood/sleap/sleap/nn/inference.py", line 446, in _predict_generator
      ex = process_batch(ex)
    File "/Users/jasperhood/sleap/sleap/nn/inference.py", line 388, in process_batch
      preds = self.inference_model.predict_on_batch(ex, numpy=True)
    File "/Users/jasperhood/sleap/sleap/nn/inference.py", line 1059, in predict_on_batch
      outs = super().predict_on_batch(data, **kwargs)
    File "/Users/jasperhood/miniconda3/envs/sleap/lib/python3.9/site-packages/keras/engine/training.py", line 2230, in predict_on_batch
      outputs = self.predict_function(iterator)
    File "/Users/jasperhood/miniconda3/envs/sleap/lib/python3.9/site-packages/keras/engine/training.py", line 1845, in predict_function
      return step_function(self, iterator)
    File "/Users/jasperhood/miniconda3/envs/sleap/lib/python3.9/site-packages/keras/engine/training.py", line 1834, in step_function
      outputs = model.distribute_strategy.run(run_step, args=(data,))
    File "/Users/jasperhood/miniconda3/envs/sleap/lib/python3.9/site-packages/keras/engine/training.py", line 1823, in run_step
      outputs = model.predict_step(data)
    File "/Users/jasperhood/miniconda3/envs/sleap/lib/python3.9/site-packages/keras/engine/training.py", line 1791, in predict_step
      return self(x, training=False)
    File "/Users/jasperhood/miniconda3/envs/sleap/lib/python3.9/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
      return fn(*args, **kwargs)
    File "/Users/jasperhood/miniconda3/envs/sleap/lib/python3.9/site-packages/keras/engine/training.py", line 490, in __call__
      return super().__call__(*args, **kwargs)
    File "/Users/jasperhood/miniconda3/envs/sleap/lib/python3.9/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
      return fn(*args, **kwargs)
    File "/Users/jasperhood/miniconda3/envs/sleap/lib/python3.9/site-packages/keras/engine/base_layer.py", line 1014, in __call__
      outputs = call_fn(inputs, *args, **kwargs)
    File "/Users/jasperhood/miniconda3/envs/sleap/lib/python3.9/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
      return fn(*args, **kwargs)
    File "/Users/jasperhood/sleap/sleap/nn/inference.py", line 2220, in call
      if isinstance(self.instance_peaks, FindInstancePeaksGroundTruth):
    File "/Users/jasperhood/sleap/sleap/nn/inference.py", line 2230, in call
      peaks_output = self.instance_peaks(crop_output)
    File "/Users/jasperhood/miniconda3/envs/sleap/lib/python3.9/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
      return fn(*args, **kwargs)
    File "/Users/jasperhood/miniconda3/envs/sleap/lib/python3.9/site-packages/keras/engine/base_layer.py", line 1014, in __call__
      outputs = call_fn(inputs, *args, **kwargs)
    File "/Users/jasperhood/miniconda3/envs/sleap/lib/python3.9/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
      return fn(*args, **kwargs)
    File "/Users/jasperhood/sleap/sleap/nn/inference.py", line 2052, in call
      out = self.keras_model(crops)
    File "/Users/jasperhood/miniconda3/envs/sleap/lib/python3.9/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
      return fn(*args, **kwargs)
    File "/Users/jasperhood/miniconda3/envs/sleap/lib/python3.9/site-packages/keras/engine/training.py", line 490, in __call__
      return super().__call__(*args, **kwargs)
    File "/Users/jasperhood/miniconda3/envs/sleap/lib/python3.9/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
      return fn(*args, **kwargs)
    File "/Users/jasperhood/miniconda3/envs/sleap/lib/python3.9/site-packages/keras/engine/base_layer.py", line 1014, in __call__
      outputs = call_fn(inputs, *args, **kwargs)
    File "/Users/jasperhood/miniconda3/envs/sleap/lib/python3.9/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
      return fn(*args, **kwargs)
    File "/Users/jasperhood/miniconda3/envs/sleap/lib/python3.9/site-packages/keras/engine/functional.py", line 458, in call
      return self._run_internal_graph(
    File "/Users/jasperhood/miniconda3/envs/sleap/lib/python3.9/site-packages/keras/engine/functional.py", line 596, in _run_internal_graph
      outputs = node.layer(*args, **kwargs)
    File "/Users/jasperhood/miniconda3/envs/sleap/lib/python3.9/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
      return fn(*args, **kwargs)
    File "/Users/jasperhood/miniconda3/envs/sleap/lib/python3.9/site-packages/keras/engine/base_layer.py", line 1014, in __call__
      outputs = call_fn(inputs, *args, **kwargs)
    File "/Users/jasperhood/miniconda3/envs/sleap/lib/python3.9/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
      return fn(*args, **kwargs)
    File "/Users/jasperhood/miniconda3/envs/sleap/lib/python3.9/site-packages/keras/layers/reshaping/up_sampling2d.py", line 129, in call
      return backend.resize_images(
    File "/Users/jasperhood/miniconda3/envs/sleap/lib/python3.9/site-packages/keras/backend.py", line 3432, in resize_images
      x = tf.image.resize(x, new_shape, method=interpolations[interpolation])
Node: 'top_down_inference_model/find_instance_peaks/model/stack0_dec0_s16_to_s8_interp_bilinear/resize/ResizeBilinear'
Detected at node 'top_down_inference_model/find_instance_peaks/model/stack0_dec0_s16_to_s8_interp_bilinear/resize/ResizeBilinear' defined at (most recent call last):
    File "/Users/jasperhood/miniconda3/envs/sleap/bin/sleap-track", line 33, in <module>
      sys.exit(load_entry_point('sleap', 'console_scripts', 'sleap-track')())
    File "/Users/jasperhood/sleap/sleap/nn/inference.py", line 5229, in main
      labels_pr = predictor.predict(provider)
    File "/Users/jasperhood/sleap/sleap/nn/inference.py", line 515, in predict
      self._make_labeled_frames_from_generator(generator, data)
    File "/Users/jasperhood/sleap/sleap/nn/inference.py", line 2519, in _make_labeled_frames_from_generator
      for ex in generator:
    File "/Users/jasperhood/sleap/sleap/nn/inference.py", line 446, in _predict_generator
      ex = process_batch(ex)
    File "/Users/jasperhood/sleap/sleap/nn/inference.py", line 388, in process_batch
      preds = self.inference_model.predict_on_batch(ex, numpy=True)
    File "/Users/jasperhood/sleap/sleap/nn/inference.py", line 1059, in predict_on_batch
      outs = super().predict_on_batch(data, **kwargs)
    File "/Users/jasperhood/miniconda3/envs/sleap/lib/python3.9/site-packages/keras/engine/training.py", line 2230, in predict_on_batch
      outputs = self.predict_function(iterator)
    File "/Users/jasperhood/miniconda3/envs/sleap/lib/python3.9/site-packages/keras/engine/training.py", line 1845, in predict_function
      return step_function(self, iterator)
    File "/Users/jasperhood/miniconda3/envs/sleap/lib/python3.9/site-packages/keras/engine/training.py", line 1834, in step_function
      outputs = model.distribute_strategy.run(run_step, args=(data,))
    File "/Users/jasperhood/miniconda3/envs/sleap/lib/python3.9/site-packages/keras/engine/training.py", line 1823, in run_step
      outputs = model.predict_step(data)
    File "/Users/jasperhood/miniconda3/envs/sleap/lib/python3.9/site-packages/keras/engine/training.py", line 1791, in predict_step
      return self(x, training=False)
    File "/Users/jasperhood/miniconda3/envs/sleap/lib/python3.9/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
      return fn(*args, **kwargs)
    File "/Users/jasperhood/miniconda3/envs/sleap/lib/python3.9/site-packages/keras/engine/training.py", line 490, in __call__
      return super().__call__(*args, **kwargs)
    File "/Users/jasperhood/miniconda3/envs/sleap/lib/python3.9/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
      return fn(*args, **kwargs)
    File "/Users/jasperhood/miniconda3/envs/sleap/lib/python3.9/site-packages/keras/engine/base_layer.py", line 1014, in __call__
      outputs = call_fn(inputs, *args, **kwargs)
    File "/Users/jasperhood/miniconda3/envs/sleap/lib/python3.9/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
      return fn(*args, **kwargs)
    File "/Users/jasperhood/sleap/sleap/nn/inference.py", line 2220, in call
      if isinstance(self.instance_peaks, FindInstancePeaksGroundTruth):
    File "/Users/jasperhood/sleap/sleap/nn/inference.py", line 2230, in call
      peaks_output = self.instance_peaks(crop_output)
    File "/Users/jasperhood/miniconda3/envs/sleap/lib/python3.9/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
      return fn(*args, **kwargs)
    File "/Users/jasperhood/miniconda3/envs/sleap/lib/python3.9/site-packages/keras/engine/base_layer.py", line 1014, in __call__
      outputs = call_fn(inputs, *args, **kwargs)
    File "/Users/jasperhood/miniconda3/envs/sleap/lib/python3.9/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
      return fn(*args, **kwargs)
    File "/Users/jasperhood/sleap/sleap/nn/inference.py", line 2052, in call
      out = self.keras_model(crops)
    File "/Users/jasperhood/miniconda3/envs/sleap/lib/python3.9/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
      return fn(*args, **kwargs)
    File "/Users/jasperhood/miniconda3/envs/sleap/lib/python3.9/site-packages/keras/engine/training.py", line 490, in __call__
      return super().__call__(*args, **kwargs)
    File "/Users/jasperhood/miniconda3/envs/sleap/lib/python3.9/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
      return fn(*args, **kwargs)
    File "/Users/jasperhood/miniconda3/envs/sleap/lib/python3.9/site-packages/keras/engine/base_layer.py", line 1014, in __call__
      outputs = call_fn(inputs, *args, **kwargs)
    File "/Users/jasperhood/miniconda3/envs/sleap/lib/python3.9/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
      return fn(*args, **kwargs)
    File "/Users/jasperhood/miniconda3/envs/sleap/lib/python3.9/site-packages/keras/engine/functional.py", line 458, in call
      return self._run_internal_graph(
    File "/Users/jasperhood/miniconda3/envs/sleap/lib/python3.9/site-packages/keras/engine/functional.py", line 596, in _run_internal_graph
      outputs = node.layer(*args, **kwargs)
    File "/Users/jasperhood/miniconda3/envs/sleap/lib/python3.9/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
      return fn(*args, **kwargs)
    File "/Users/jasperhood/miniconda3/envs/sleap/lib/python3.9/site-packages/keras/engine/base_layer.py", line 1014, in __call__
      outputs = call_fn(inputs, *args, **kwargs)
    File "/Users/jasperhood/miniconda3/envs/sleap/lib/python3.9/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
      return fn(*args, **kwargs)
    File "/Users/jasperhood/miniconda3/envs/sleap/lib/python3.9/site-packages/keras/layers/reshaping/up_sampling2d.py", line 129, in call
      return backend.resize_images(
    File "/Users/jasperhood/miniconda3/envs/sleap/lib/python3.9/site-packages/keras/backend.py", line 3432, in resize_images
      x = tf.image.resize(x, new_shape, method=interpolations[interpolation])
Node: 'top_down_inference_model/find_instance_peaks/model/stack0_dec0_s16_to_s8_interp_bilinear/resize/ResizeBilinear'
2 root error(s) found.
  (0) INTERNAL:  Missing 0-th output from {{node top_down_inference_model/find_instance_peaks/model/stack0_dec0_s16_to_s8_interp_bilinear/resize/ResizeBilinear}}
	 [[top_down_inference_model/find_instance_peaks/RaggedFromValueRowIds_1/RowPartitionFromValueRowIds/assert_non_negative/assert_less_equal/LessEqual/_216]]
  (1) INTERNAL:  Missing 0-th output from {{node top_down_inference_model/find_instance_peaks/model/stack0_dec0_s16_to_s8_interp_bilinear/resize/ResizeBilinear}}
0 successful operations.
0 derived errors ignored. [Op:__inference_predict_function_4345]
SLEAP: 1.2.9
TensorFlow: 2.9.2
Numpy: 1.22.3
Python: 3.9.16
OS: macOS-12.5-arm64-arm-64bit

System:
GPUs: 1/1 available
  Device: /physical_device:GPU:0
         Available: True
Process return code: 1

<\details>

[roomrys]

Hi @Jaspermoray,

Is the new video a different size from the videos you already trained on? Flexible resizing of the input layer (which handles prediction on new/unseen video sizes) is in the develop branch and will be available in the next SLEAP version. If you are already using the develop branch, let us know, then something is amiss.

Thanks,
Liezl

[Jaspermoray]

No, they are the same size...

[Jaspermoray]

Are there other factors that could be causing this problem? I assume there is supposed to be functionality in this case.

[roomrys]

For future users:

This is the same issue as #1151

Thanks for the report! This is happening because there are some chunks of frame during which no centroids are detected, which causes an error downstream in the centered instance model. This works in Windows/Linux, but due to the implementation of certain CUDA/TensorFlow operations on Apple Silicon, the behavior is different with 0-length tensors and it throws this error. We're trying to find a fix for it, but it's a bit tricky since Apple's tensorflow-metal is pretty experimental and under heavy development.
We'll post updates on that issue as we make progress on it.