Replies: 1 comment 2 replies
-
|
Hi @dusk0703, It looks like you're running out of memory on your GPU to run this model. This typically would happen during the training step since that requires more GPU memory than inference, so I presume you trained this on Colab or another machine? In any case, the problem is that the GPU you have (NVIDIA GeForce RTX 3050 Ti Laptop GPU) only has 1657 MB of memory, which is a bit too small for most deep neural network models unfortunately. Things you can try:
Give those a go and let us know if you have any questions! Cheers, Talmo |
Beta Was this translation helpful? Give feedback.
-
|
Thank you very much, was kind of the expected answer but good to get confirmation! We are managing to run training and inference on a different machine but running into issues with inference again, using the command line terminal. I've used this code and not had this error before if you wouldn't mind having a look? Thank you very much in advance, (base) C:\Users\Ben07>mamba activate sleap (sleap) C:\Users\Ben07>sleap-label Software versions: Happy SLEAPing! :) (sleap) C:\Users\Ben07>sleap-track -m "C:\Users\Ben07\OneDrive - The University of Nottingham\General - Calving MRes\sleap\New folder\final_model_development.slp.training_job_2\models\231218_150401.centroid" "C:\Users\Ben07\OneDrive - The University of Nottingham\General - Calving MRes\sleap\New folder\final_model_development.slp.training_job_2\models\231218_150401.centered_instance" --only-suggested-frames -o "labels_with_predictions.slp" "C:\Users\Ben07\OneDrive - The University of Nottingham\General - Calving MRes\sleap\New folder\final_model_development.slp" INFO:sleap.nn.inference:Auto-selected GPU 0 with 3990 MiB of free memory. System: Video: C:\Users\Ben07\OneDrive - The University of Nottingham\General - Calving MRes\sleap\New folder\final_model_development.slp.training_job_2\models\231218_150401.centered_instance Traceback (most recent call last): |
Beta Was this translation helpful? Give feedback.
-
|
Hi @dusk0703, Glad you got it working! The new issue is a pretty simple one: just add a missing The Let us know if that works for you! Cheers, Talmo |
Beta Was this translation helpful? Give feedback.
-
Hi, I'm trying to run an inference on suggested frames through the gui but keep getting errors. Fairly new to all of this so hoping you might be able to point me in the right direction? All help very much appreciated!
Command line call:
sleap-track C:/Users/Ben07/OneDrive - The University of Nottingham/General - Calving MRes/sleap/New folder/final_model_development.slp --only-suggested-frames -m C:/Users/Ben07/OneDrive - The University of Nottingham/General - Calving MRes/sleap/New folder\models\231213_100935.centroid.n=84\training_config.json -m C:/Users/Ben07/OneDrive - The University of Nottingham/General - Calving MRes/sleap/New folder\models\231213_111317.centered_instance.n=84\training_config.json --tracking.tracker none -o C:/Users/Ben07/OneDrive - The University of Nottingham/General - Calving MRes/sleap/New folder\predictions\final_model_development.slp.231215_115653.predictions.slp --verbosity json --no-empty-frames
Started inference at: 2023-12-15 11:56:57.816946
Args:
{
'data_path': 'C:/Users/Ben07/OneDrive - The University of Nottingham/General - Calving MRes/sleap/New folder/final_model_development.slp',
'models': [
'C:/Users/Ben07/OneDrive - The University of Nottingham/General - Calving MRes/sleap/New folder\models\231213_100935.centroid.n=84\training_config.json',
'C:/Users/Ben07/OneDrive - The University of Nottingham/General - Calving MRes/sleap/New folder\models\231213_111317.centered_instance.n=84\training_config.json'
],
'frames': '',
'only_labeled_frames': False,
'only_suggested_frames': True,
'output': 'C:/Users/Ben07/OneDrive - The University of Nottingham/General - Calving MRes/sleap/New folder\predictions\final_model_development.slp.231215_115653.predictions.slp',
'no_empty_frames': True,
'verbosity': 'json',
'video.dataset': None,
'video.input_format': 'channels_last',
'video.index': '',
'cpu': False,
'first_gpu': False,
'last_gpu': False,
'gpu': 'auto',
'max_edge_length_ratio': 0.25,
'dist_penalty_weight': 1.0,
'batch_size': 4,
'open_in_gui': False,
'peak_threshold': 0.2,
2023-12-15 11:56:59.212059: I tensorflow/core/platform/cpu_feature_guard.cc:151] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX AVX2
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
'max_instances': None,
'tracking.tracker': 'none',
'tracking.max_tracking': None,
'tracking.max_tracks': None,
'tracking.target_instance_count': None,
'tracking.pre_cull_to_target': None,
'tracking.pre_cull_iou_threshold': None,
'tracking.post_connect_single_breaks': None,
'tracking.clean_instance_count': None,
2023-12-15 11:56:59.669441: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1525] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 1657 MB memory: -> device: 0, name: NVIDIA GeForce RTX 3050 Ti Laptop GPU, pci bus id: 0000:01:00.0, compute capability: 8.6
'tracking.clean_iou_threshold': None,
'tracking.similarity': None,
'tracking.match': None,
'tracking.robust': None,
'tracking.track_window': None,
'tracking.min_new_track_points': None,
'tracking.min_match_points': None,
'tracking.img_scale': None,
'tracking.of_window_size': None,
'tracking.of_max_levels': None,
'tracking.save_shifted_instances': None,
'tracking.kf_node_indices': None,
'tracking.kf_init_frame_count': None
}
INFO:sleap.nn.inference:Auto-selected GPU 0 with 3475 MiB of free memory.
2023-12-15 11:57:28.027186: I tensorflow/stream_executor/cuda/cuda_dnn.cc:366] Loaded cuDNN version 8201
2023-12-15 11:57:35.809774: W tensorflow/core/common_runtime/bfc_allocator.cc:275] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.61GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2023-12-15 11:57:35.810164: W tensorflow/core/common_runtime/bfc_allocator.cc:275] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.61GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2023-12-15 11:57:35.908552: W tensorflow/core/common_runtime/bfc_allocator.cc:275] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.63GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2023-12-15 11:57:35.908870: W tensorflow/core/common_runtime/bfc_allocator.cc:275] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.63GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2023-12-15 11:57:36.029954: W tensorflow/core/common_runtime/bfc_allocator.cc:275] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.72GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2023-12-15 11:57:36.030291: W tensorflow/core/common_runtime/bfc_allocator.cc:275] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.72GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2023-12-15 11:57:36.166764: W tensorflow/core/common_runtime/bfc_allocator.cc:343] Garbage collection: deallocate free memory regions (i.e., allocations) so that we can re-allocate a larger region to avoid OOM due to memory fragmentation. If you see this message frequently, you are running near the threshold of the available device memory and re-allocation may incur great performance overhead. You may try smaller batch sizes to observe the performance impact. Set TF_ENABLE_GPU_GARBAGE_COLLECTION=false if you'd like to disable this feature.
2023-12-15 11:57:36.201295: W tensorflow/core/common_runtime/bfc_allocator.cc:275] Allocator (GPU_0_bfc) ran out of memory trying to allocate 766.00MiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2023-12-15 11:57:36.201494: W tensorflow/core/kernels/gpu_utils.cc:49] Failed to allocate memory for convolution redzone checking; skipping this check. This is benign and only means that we won't check cudnn for out-of-bounds reads and writes. This message will only be printed once.
2023-12-15 11:57:36.260895: W tensorflow/core/common_runtime/bfc_allocator.cc:275] Allocator (GPU_0_bfc) ran out of memory trying to allocate 859.75MiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2023-12-15 11:57:36.261380: W tensorflow/core/common_runtime/bfc_allocator.cc:275] Allocator (GPU_0_bfc) ran out of memory trying to allocate 859.75MiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2023-12-15 11:57:36.363563: W tensorflow/core/common_runtime/bfc_allocator.cc:275] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.98GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2023-12-15 11:57:46.376269: W tensorflow/core/common_runtime/bfc_allocator.cc:462] Allocator (GPU_0_bfc) ran out of memory trying to allocate 750.00MiB (rounded to 786432000)requested by op top_down_inference_model/find_instance_peaks/model/stack0_enc0_act1_relu/Relu
If the cause is memory fragmentation maybe the environment variable 'TF_GPU_ALLOCATOR=cuda_malloc_async' will improve the situation.
Current allocation summary follows.
Current allocation summary follows.
2023-12-15 11:57:46.376508: I tensorflow/core/common_runtime/bfc_allocator.cc:1010] BFCAllocator dump for GPU_0_bfc
2023-12-15 11:57:46.376652: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (256): Total Chunks: 93, Chunks in use: 93. 23.2KiB allocated for chunks. 23.2KiB in use in bin. 3.3KiB client-requested in use in bin.
2023-12-15 11:57:46.378421: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (512): Total Chunks: 7, Chunks in use: 7. 4.0KiB allocated for chunks. 4.0KiB in use in bin. 3.5KiB client-requested in use in bin.
2023-12-15 11:57:46.378463: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (1024): Total Chunks: 11, Chunks in use: 9. 16.0KiB allocated for chunks. 12.8KiB in use in bin. 11.4KiB client-requested in use in bin.
2023-12-15 11:57:46.378520: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (2048): Total Chunks: 4, Chunks in use: 3. 11.8KiB allocated for chunks. 8.8KiB in use in bin. 8.5KiB client-requested in use in bin.
2023-12-15 11:57:46.378548: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (4096): Total Chunks: 1, Chunks in use: 0. 4.5KiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2023-12-15 11:57:46.378584: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (8192): Total Chunks: 2, Chunks in use: 1. 21.8KiB allocated for chunks. 13.0KiB in use in bin. 9.0KiB client-requested in use in bin.
2023-12-15 11:57:46.378612: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (16384): Total Chunks: 1, Chunks in use: 1. 18.0KiB allocated for chunks. 18.0KiB in use in bin. 18.0KiB client-requested in use in bin.
2023-12-15 11:57:46.378641: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (32768): Total Chunks: 10, Chunks in use: 5. 419.5KiB allocated for chunks. 194.2KiB in use in bin. 174.8KiB client-requested in use in bin.
2023-12-15 11:57:46.378669: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (65536): Total Chunks: 4, Chunks in use: 3. 397.5KiB allocated for chunks. 316.5KiB in use in bin. 261.0KiB client-requested in use in bin.
2023-12-15 11:57:46.378696: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (131072): Total Chunks: 5, Chunks in use: 3. 777.0KiB allocated for chunks. 453.0KiB in use in bin. 450.0KiB client-requested in use in bin.
2023-12-15 11:57:46.378761: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (262144): Total Chunks: 4, Chunks in use: 3. 1.46MiB allocated for chunks. 1.17MiB in use in bin. 1.02MiB client-requested in use in bin.
2023-12-15 11:57:46.378789: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (524288): Total Chunks: 5, Chunks in use: 3. 3.38MiB allocated for chunks. 1.76MiB in use in bin. 1.76MiB client-requested in use in bin.
2023-12-15 11:57:46.378816: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (1048576): Total Chunks: 2, Chunks in use: 1. 2.62MiB allocated for chunks. 1.12MiB in use in bin. 1.12MiB client-requested in use in bin.
2023-12-15 11:57:46.378842: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (2097152): Total Chunks: 5, Chunks in use: 4. 11.66MiB allocated for chunks. 8.82MiB in use in bin. 7.73MiB client-requested in use in bin.
2023-12-15 11:57:46.378871: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (4194304): Total Chunks: 5, Chunks in use: 2. 26.95MiB allocated for chunks. 10.12MiB in use in bin. 10.12MiB client-requested in use in bin.
2023-12-15 11:57:46.378919: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (8388608): Total Chunks: 3, Chunks in use: 3. 37.00MiB allocated for chunks. 37.00MiB in use in bin. 37.00MiB client-requested in use in bin.
2023-12-15 11:57:46.378947: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (16777216): Total Chunks: 3, Chunks in use: 1. 66.53MiB allocated for chunks. 20.25MiB in use in bin. 20.25MiB client-requested in use in bin.
2023-12-15 11:57:46.378973: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (33554432): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2023-12-15 11:57:46.379000: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (67108864): Total Chunks: 1, Chunks in use: 0. 70.75MiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2023-12-15 11:57:46.379027: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (134217728): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2023-12-15 11:57:46.379054: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (268435456): Total Chunks: 3, Chunks in use: 1. 1.40GiB allocated for chunks. 750.00MiB in use in bin. 750.00MiB client-requested in use in bin.
2023-12-15 11:57:46.379082: I tensorflow/core/common_runtime/bfc_allocator.cc:1033] Bin for 750.00MiB was 256.00MiB, Chunk State:
2023-12-15 11:57:46.379109: I tensorflow/core/common_runtime/bfc_allocator.cc:1039] Size: 256.00MiB | Requested Size: 93.75MiB | in_use: 0 | bin_num: 20
2023-12-15 11:57:46.379135: I tensorflow/core/common_runtime/bfc_allocator.cc:1039] Size: 429.88MiB | Requested Size: 455.17MiB | in_use: 0 | bin_num: 20, prev: Size: 750.00MiB | Requested Size: 750.00MiB | in_use: 1 | bin_num: -1
2023-12-15 11:57:46.379161: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] Next region of size 2097152
2023-12-15 11:57:46.379567: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0c800000 of size 1280 next 1
2023-12-15 11:57:46.379737: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0c800500 of size 256 next 2
2023-12-15 11:57:46.379769: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0c800600 of size 256 next 3
2023-12-15 11:57:46.379793: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0c800700 of size 256 next 4
2023-12-15 11:57:46.379818: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0c800800 of size 256 next 5
2023-12-15 11:57:46.379842: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0c800900 of size 256 next 8
2023-12-15 11:57:46.379864: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0c800a00 of size 256 next 9
2023-12-15 11:57:46.379888: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0c800b00 of size 256 next 10
2023-12-15 11:57:46.379911: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0c800c00 of size 256 next 13
2023-12-15 11:57:46.379934: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0c800d00 of size 256 next 14
2023-12-15 11:57:46.379956: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0c800e00 of size 256 next 17
2023-12-15 11:57:46.379979: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0c800f00 of size 256 next 18
2023-12-15 11:57:46.380001: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0c801000 of size 256 next 19
2023-12-15 11:57:46.380025: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0c801100 of size 256 next 20
2023-12-15 11:57:46.380047: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0c801200 of size 256 next 23
2023-12-15 11:57:46.380071: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0c801300 of size 256 next 24
2023-12-15 11:57:46.380093: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0c801400 of size 256 next 27
2023-12-15 11:57:46.380116: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0c801500 of size 256 next 6
2023-12-15 11:57:46.380139: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] Free at b0c801600 of size 1792 next 7
2023-12-15 11:57:46.380169: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0c801d00 of size 256 next 28
2023-12-15 11:57:46.380192: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0c801e00 of size 256 next 29
2023-12-15 11:57:46.380216: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0c801f00 of size 256 next 32
2023-12-15 11:57:46.380259: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0c802000 of size 256 next 33
2023-12-15 11:57:46.380282: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0c802100 of size 256 next 147
2023-12-15 11:57:46.380305: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0c802200 of size 256 next 36
2023-12-15 11:57:46.380330: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0c802300 of size 256 next 37
2023-12-15 11:57:46.380352: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0c802400 of size 256 next 38
2023-12-15 11:57:46.380375: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0c802500 of size 256 next 107
2023-12-15 11:57:46.380502: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0c802600 of size 256 next 42
2023-12-15 11:57:46.380604: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0c802700 of size 256 next 43
2023-12-15 11:57:46.380657: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0c802800 of size 256 next 44
2023-12-15 11:57:46.380692: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0c802900 of size 256 next 63
2023-12-15 11:57:46.380724: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0c802a00 of size 256 next 152
2023-12-15 11:57:46.380766: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0c802b00 of size 256 next 125
2023-12-15 11:57:46.380794: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0c802c00 of size 256 next 45
2023-12-15 11:57:46.380821: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0c802d00 of size 256 next 48
2023-12-15 11:57:46.380848: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0c802e00 of size 256 next 49
2023-12-15 11:57:46.380874: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0c802f00 of size 256 next 157
2023-12-15 11:57:46.380901: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0c803000 of size 256 next 117
2023-12-15 11:57:46.380928: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0c803100 of size 256 next 124
2023-12-15 11:57:46.380955: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0c803200 of size 256 next 50
2023-12-15 11:57:46.380982: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0c803300 of size 256 next 118
2023-12-15 11:57:46.381009: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0c803400 of size 256 next 60
2023-12-15 11:57:46.381036: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0c803500 of size 256 next 22
2023-12-15 11:57:46.381071: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0c803600 of size 256 next 73
2023-12-15 11:57:46.381114: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0c803700 of size 256 next 74
2023-12-15 11:57:46.381141: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0c803800 of size 256 next 77
2023-12-15 11:57:46.381194: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0c803900 of size 256 next 79
2023-12-15 11:57:46.381220: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0c803a00 of size 256 next 62
2023-12-15 11:57:46.381247: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0c803b00 of size 256 next 64
2023-12-15 11:57:46.381274: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0c803c00 of size 256 next 65
2023-12-15 11:57:46.381301: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0c803d00 of size 256 next 66
2023-12-15 11:57:46.381328: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0c803e00 of size 256 next 67
2023-12-15 11:57:46.381354: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0c803f00 of size 256 next 68
2023-12-15 11:57:46.381381: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0c804000 of size 256 next 30
2023-12-15 11:57:46.381408: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] Free at b0c804100 of size 1536 next 69
2023-12-15 11:57:46.381436: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0c804700 of size 512 next 83
2023-12-15 11:57:46.381462: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0c804900 of size 1536 next 89
2023-12-15 11:57:46.381496: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0c804f00 of size 512 next 90
2023-12-15 11:57:46.381525: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0c805100 of size 256 next 108
2023-12-15 11:57:46.381558: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0c805200 of size 256 next 109
2023-12-15 11:57:46.381608: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0c805300 of size 256 next 110
2023-12-15 11:57:46.381634: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0c805400 of size 256 next 39
2023-12-15 11:57:46.381661: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0c805500 of size 13312 next 11
2023-12-15 11:57:46.381689: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] Free at b0c808900 of size 147456 next 21
2023-12-15 11:57:46.381717: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0c82c900 of size 512 next 93
2023-12-15 11:57:46.381743: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0c82cb00 of size 256 next 94
2023-12-15 11:57:46.381769: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0c82cc00 of size 256 next 95
2023-12-15 11:57:46.381797: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0c82cd00 of size 256 next 99
2023-12-15 11:57:46.381823: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0c82ce00 of size 256 next 101
2023-12-15 11:57:46.381850: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0c82cf00 of size 256 next 88
2023-12-15 11:57:46.381876: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0c82d000 of size 256 next 103
2023-12-15 11:57:46.381903: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0c82d100 of size 256 next 104
2023-12-15 11:57:46.381930: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0c82d200 of size 1792 next 105
2023-12-15 11:57:46.381957: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0c82d900 of size 256 next 41
2023-12-15 11:57:46.381983: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0c82da00 of size 512 next 15
2023-12-15 11:57:46.382011: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0c82dc00 of size 1024 next 86
2023-12-15 11:57:46.382037: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0c82e000 of size 256 next 113
2023-12-15 11:57:46.382064: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0c82e100 of size 256 next 115
2023-12-15 11:57:46.382110: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0c82e200 of size 256 next 116
2023-12-15 11:57:46.382143: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0c82e300 of size 256 next 131
2023-12-15 11:57:46.382170: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0c82e400 of size 256 next 119
2023-12-15 11:57:46.382197: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0c82e500 of size 256 next 112
2023-12-15 11:57:46.382228: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0c82e600 of size 256 next 120
2023-12-15 11:57:46.382259: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0c82e700 of size 256 next 121
2023-12-15 11:57:46.382286: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0c82e800 of size 256 next 123
2023-12-15 11:57:46.382312: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0c82e900 of size 256 next 148
2023-12-15 11:57:46.382339: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0c82ea00 of size 256 next 132
2023-12-15 11:57:46.382365: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0c82eb00 of size 256 next 126
2023-12-15 11:57:46.382393: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0c82ec00 of size 768 next 127
2023-12-15 11:57:46.382420: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0c82ef00 of size 256 next 128
2023-12-15 11:57:46.382446: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0c82f000 of size 256 next 130
2023-12-15 11:57:46.382473: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0c82f100 of size 256 next 133
2023-12-15 11:57:46.382500: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0c82f200 of size 256 next 134
2023-12-15 11:57:46.382526: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0c82f300 of size 256 next 139
2023-12-15 11:57:46.382553: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0c82f400 of size 256 next 140
2023-12-15 11:57:46.382588: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0c82f500 of size 256 next 106
2023-12-15 11:57:46.382630: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] Free at b0c82f600 of size 8960 next 143
2023-12-15 11:57:46.382658: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0c831900 of size 256 next 144
2023-12-15 11:57:46.382685: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] Free at b0c831a00 of size 3072 next 142
2023-12-15 11:57:46.382711: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0c832600 of size 1536 next 174
2023-12-15 11:57:46.382738: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0c832c00 of size 1536 next 151
2023-12-15 11:57:46.382765: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0c833200 of size 256 next 153
2023-12-15 11:57:46.382793: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0c833300 of size 256 next 154
2023-12-15 11:57:46.382819: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0c833400 of size 256 next 156
2023-12-15 11:57:46.382845: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] Free at b0c833500 of size 64512 next 58
2023-12-15 11:57:46.382873: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0c843100 of size 18432 next 72
2023-12-15 11:57:46.382900: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0c847900 of size 129024 next 75
2023-12-15 11:57:46.382927: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] Free at b0c867100 of size 36864 next 102
2023-12-15 11:57:46.382961: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0c870100 of size 36864 next 78
2023-12-15 11:57:46.382989: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0c879100 of size 36864 next 96
2023-12-15 11:57:46.383017: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] Free at b0c882100 of size 1564416 next 18446744073709551615
2023-12-15 11:57:46.383055: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] Next region of size 4194304
2023-12-15 11:57:46.383074: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] Free at b0ca00000 of size 294912 next 26
2023-12-15 11:57:46.383104: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0ca48000 of size 589824 next 25
2023-12-15 11:57:46.383170: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0cad8000 of size 1179648 next 85
2023-12-15 11:57:46.383200: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0cbf8000 of size 2129920 next 18446744073709551615
2023-12-15 11:57:46.383231: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] Next region of size 8388608
2023-12-15 11:57:46.383249: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] Free at b0ce00000 of size 1032192 next 53
2023-12-15 11:57:46.383276: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0cefc000 of size 110592 next 100
2023-12-15 11:57:46.383304: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] Free at b0cf17000 of size 184320 next 81
2023-12-15 11:57:46.383331: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0cf44000 of size 147456 next 80
2023-12-15 11:57:46.383357: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] Free at b0cf68000 of size 41472 next 111
2023-12-15 11:57:46.383384: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0cf72200 of size 3072 next 177
2023-12-15 11:57:46.383413: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0cf72e00 of size 1536 next 182
2023-12-15 11:57:46.383440: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] Free at b0cf73400 of size 4608 next 185
2023-12-15 11:57:46.383466: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0cf74600 of size 2816 next 189
2023-12-15 11:57:46.383493: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0cf75100 of size 256 next 150
2023-12-15 11:57:46.383520: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0cf75200 of size 512 next 164
2023-12-15 11:57:46.383547: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0cf75400 of size 768 next 169
2023-12-15 11:57:46.383573: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0cf75700 of size 1536 next 171
2023-12-15 11:57:46.383604: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0cf75d00 of size 3072 next 179
2023-12-15 11:57:46.383654: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0cf76900 of size 43008 next 184
2023-12-15 11:57:46.383681: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] Free at b0cf81100 of size 44800 next 98
2023-12-15 11:57:46.383714: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0cf8c000 of size 147456 next 82
2023-12-15 11:57:46.383742: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0cfb0000 of size 442368 next 56
2023-12-15 11:57:46.383769: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] Free at b0d01c000 of size 6176768 next 18446744073709551615
2023-12-15 11:57:46.383797: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] Next region of size 16777216
2023-12-15 11:57:46.383818: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0d600000 of size 589824 next 91
2023-12-15 11:57:46.383847: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] Free at b0d690000 of size 82944 next 114
2023-12-15 11:57:46.383873: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0d6a4400 of size 1280 next 168
2023-12-15 11:57:46.383899: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0d6a4900 of size 256 next 180
2023-12-15 11:57:46.383925: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0d6a4a00 of size 256 next 183
2023-12-15 11:57:46.383951: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0d6a4b00 of size 256 next 165
2023-12-15 11:57:46.383978: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0d6a4c00 of size 256 next 188
2023-12-15 11:57:46.384004: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0d6a4d00 of size 40704 next 155
2023-12-15 11:57:46.384031: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] Free at b0d6aec00 of size 663552 next 122
2023-12-15 11:57:46.384057: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0d750c00 of size 41472 next 54
2023-12-15 11:57:46.384083: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0d75ae00 of size 84480 next 187
2023-12-15 11:57:46.384109: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] Free at b0d76f800 of size 43008 next 186
2023-12-15 11:57:46.384151: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0d77a000 of size 494592 next 163
2023-12-15 11:57:46.384185: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0d7f2c00 of size 168960 next 97
2023-12-15 11:57:46.384211: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0d81c000 of size 294912 next 76
2023-12-15 11:57:46.384237: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] Free at b0d864000 of size 2985984 next 129
2023-12-15 11:57:46.384263: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0db3d000 of size 663552 next 167
2023-12-15 11:57:46.384290: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0dbdf000 of size 2101248 next 92
2023-12-15 11:57:46.384316: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0dde0000 of size 2359296 next 84
2023-12-15 11:57:46.384342: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] Free at b0e020000 of size 6160384 next 18446744073709551615
2023-12-15 11:57:46.384369: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] Next region of size 67108864
2023-12-15 11:57:46.384388: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] Free at b10600000 of size 5308416 next 145
2023-12-15 11:57:46.384420: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b10b10000 of size 2654208 next 173
2023-12-15 11:57:46.384447: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] Free at b10d98000 of size 18579456 next 149
2023-12-15 11:57:46.384473: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b11f50000 of size 5308416 next 176
2023-12-15 11:57:46.384500: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b12460000 of size 5308416 next 175
2023-12-15 11:57:46.384527: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] Free at b12970000 of size 29949952 next 18446744073709551615
2023-12-15 11:57:46.384554: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] Next region of size 134217728
2023-12-15 11:57:46.384573: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b14600000 of size 10616832 next 172
2023-12-15 11:57:46.384599: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b15020000 of size 21233664 next 178
2023-12-15 11:57:46.384635: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b16460000 of size 15925248 next 181
2023-12-15 11:57:46.384681: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b17390000 of size 12257280 next 170
2023-12-15 11:57:46.384707: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] Free at b17f40800 of size 74184704 next 18446744073709551615
2023-12-15 11:57:46.384734: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] Next region of size 268435456
2023-12-15 11:57:46.384752: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] Free at b1c600000 of size 268435456 next 18446744073709551615
2023-12-15 11:57:46.384780: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] Next region of size 1237188608
2023-12-15 11:57:46.384798: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b89800000 of size 786432000 next 57
2023-12-15 11:57:46.384824: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] Free at bb8600000 of size 450756608 next 18446744073709551615
2023-12-15 11:57:46.384850: I tensorflow/core/common_runtime/bfc_allocator.cc:1071] Summary of in-use Chunks by size:
2023-12-15 11:57:46.384880: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 93 Chunks of size 256 totalling 23.2KiB
2023-12-15 11:57:46.384908: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 5 Chunks of size 512 totalling 2.5KiB
2023-12-15 11:57:46.384935: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 2 Chunks of size 768 totalling 1.5KiB
2023-12-15 11:57:46.384962: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 1 Chunks of size 1024 totalling 1.0KiB
2023-12-15 11:57:46.384989: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 2 Chunks of size 1280 totalling 2.5KiB
2023-12-15 11:57:46.385017: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 5 Chunks of size 1536 totalling 7.5KiB
2023-12-15 11:57:46.385043: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 1 Chunks of size 1792 totalling 1.8KiB
2023-12-15 11:57:46.385070: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 1 Chunks of size 2816 totalling 2.8KiB
2023-12-15 11:57:46.385097: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 2 Chunks of size 3072 totalling 6.0KiB
2023-12-15 11:57:46.385151: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 1 Chunks of size 13312 totalling 13.0KiB
2023-12-15 11:57:46.385188: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 1 Chunks of size 18432 totalling 18.0KiB
2023-12-15 11:57:46.385220: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 2 Chunks of size 36864 totalling 72.0KiB
2023-12-15 11:57:46.385247: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 1 Chunks of size 40704 totalling 39.8KiB
2023-12-15 11:57:46.385274: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 1 Chunks of size 41472 totalling 40.5KiB
2023-12-15 11:57:46.385301: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 1 Chunks of size 43008 totalling 42.0KiB
2023-12-15 11:57:46.385328: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 1 Chunks of size 84480 totalling 82.5KiB
2023-12-15 11:57:46.385355: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 1 Chunks of size 110592 totalling 108.0KiB
2023-12-15 11:57:46.385382: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 1 Chunks of size 129024 totalling 126.0KiB
2023-12-15 11:57:46.385410: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 2 Chunks of size 147456 totalling 288.0KiB
2023-12-15 11:57:46.385437: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 1 Chunks of size 168960 totalling 165.0KiB
2023-12-15 11:57:46.385464: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 1 Chunks of size 294912 totalling 288.0KiB
2023-12-15 11:57:46.385490: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 1 Chunks of size 442368 totalling 432.0KiB
2023-12-15 11:57:46.385518: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 1 Chunks of size 494592 totalling 483.0KiB
2023-12-15 11:57:46.385545: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 2 Chunks of size 589824 totalling 1.12MiB
2023-12-15 11:57:46.385571: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 1 Chunks of size 663552 totalling 648.0KiB
2023-12-15 11:57:46.385598: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 1 Chunks of size 1179648 totalling 1.12MiB
2023-12-15 11:57:46.385640: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 1 Chunks of size 2101248 totalling 2.00MiB
2023-12-15 11:57:46.385669: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 1 Chunks of size 2129920 totalling 2.03MiB
2023-12-15 11:57:46.385713: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 1 Chunks of size 2359296 totalling 2.25MiB
2023-12-15 11:57:46.385739: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 1 Chunks of size 2654208 totalling 2.53MiB
2023-12-15 11:57:46.385767: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 2 Chunks of size 5308416 totalling 10.12MiB
2023-12-15 11:57:46.385794: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 1 Chunks of size 10616832 totalling 10.12MiB
2023-12-15 11:57:46.385821: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 1 Chunks of size 12257280 totalling 11.69MiB
2023-12-15 11:57:46.385855: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 1 Chunks of size 15925248 totalling 15.19MiB
2023-12-15 11:57:46.385882: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 1 Chunks of size 21233664 totalling 20.25MiB
2023-12-15 11:57:46.385910: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 1 Chunks of size 786432000 totalling 750.00MiB
2023-12-15 11:57:46.385936: I tensorflow/core/common_runtime/bfc_allocator.cc:1078] Sum Total of in-use chunks: 831.27MiB
2023-12-15 11:57:46.385964: I tensorflow/core/common_runtime/bfc_allocator.cc:1080] total_region_allocated_bytes_: 1738407936 memory_limit_: 1738407936 available bytes: 0 curr_region_allocation_bytes_: 4294967296
2023-12-15 11:57:46.385997: I tensorflow/core/common_runtime/bfc_allocator.cc:1086] Stats:
Limit: 1738407936
InUse: 871649792
MaxInUse: 1090313728
NumAllocs: 963
MaxAllocSize: 786432000
Reserved: 0
PeakReserved: 0
LargestFreeBlock: 0
2023-12-15 11:57:46.386095: W tensorflow/core/common_runtime/bfc_allocator.cc:474] ********************************************************________
2023-12-15 11:57:46.386281: W tensorflow/core/framework/op_kernel.cc:1745] OP_REQUIRES failed at conv_ops_fused_impl.h:663 : RESOURCE_EXHAUSTED: OOM when allocating tensor with shape[20,24,640,640] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
Versions:
Traceback (most recent call last):
File "C:\Users\Ben07\mambaforge\New_folder\envs\sleap\Scripts\sleap-track-script.py", line 33, in
sys.exit(load_entry_point('sleap==1.3.3', 'console_scripts', 'sleap-track')())
File "C:\Users\Ben07\mambaforge\New_folder\envs\sleap\lib\site-packages\sleap\nn\inference.py", line 5424, in main
labels_pr = predictor.predict(provider)
File "C:\Users\Ben07\mambaforge\New_folder\envs\sleap\lib\site-packages\sleap\nn\inference.py", line 526, in predict
self._make_labeled_frames_from_generator(generator, data)
File "C:\Users\Ben07\mambaforge\New_folder\envs\sleap\lib\site-packages\sleap\nn\inference.py", line 2633, in _make_labeled_frames_from_generator
for ex in generator:
File "C:\Users\Ben07\mambaforge\New_folder\envs\sleap\lib\site-packages\sleap\nn\inference.py", line 457, in _predict_generator
ex = process_batch(ex)
File "C:\Users\Ben07\mambaforge\New_folder\envs\sleap\lib\site-packages\sleap\nn\inference.py", line 399, in process_batch
preds = self.inference_model.predict_on_batch(ex, numpy=True)
File "C:\Users\Ben07\mambaforge\New_folder\envs\sleap\lib\site-packages\sleap\nn\inference.py", line 1069, in predict_on_batch
outs = super().predict_on_batch(data, **kwargs)
File "C:\Users\Ben07\mambaforge\New_folder\envs\sleap\lib\site-packages\keras\engine\training.py", line 1986, in predict_on_batch
outputs = self.predict_function(iterator)
File "C:\Users\Ben07\mambaforge\New_folder\envs\sleap\lib\site-packages\tensorflow\python\util\traceback_utils.py", line 153, in error_handler
raise e.with_traceback(filtered_tb) from None
File "C:\Users\Ben07\mambaforge\New_folder\envs\sleap\lib\site-packages\tensorflow\python\eager\execute.py", line 59, in quick_execute
inputs, attrs, num_outputs)
tensorflow.python.framework.errors_impl.ResourceExhaustedError: 2 root error(s) found.
(0) RESOURCE_EXHAUSTED: OOM when allocating tensor with shape[20,24,640,640] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
[[node top_down_inference_model/find_instance_peaks/model/stack0_enc0_act1_relu/Relu
(defined at C:\Users\Ben07\mambaforge\New_folder\envs\sleap\lib\site-packages\keras\backend.py:4867)
]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. This isn't available when running in Eager mode.
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. This isn't available when running in Eager mode.
(1) RESOURCE_EXHAUSTED: OOM when allocating tensor with shape[20,24,640,640] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
[[node top_down_inference_model/find_instance_peaks/model/stack0_enc0_act1_relu/Relu
(defined at C:\Users\Ben07\mambaforge\New_folder\envs\sleap\lib\site-packages\keras\backend.py:4867)
]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. This isn't available when running in Eager mode.
0 successful operations.
0 derived errors ignored. [Op:__inference_predict_function_4598]
Errors may have originated from an input operation.
Input Source operations connected to node top_down_inference_model/find_instance_peaks/model/stack0_enc0_act1_relu/Relu:
In[0] top_down_inference_model/find_instance_peaks/model/stack0_enc0_conv1/BiasAdd (defined at C:\Users\Ben07\mambaforge\New_folder\envs\sleap\lib\site-packages\keras\layers\convolutional.py:265)
Operation defined at: (most recent call last)
Input Source operations connected to node top_down_inference_model/find_instance_peaks/model/stack0_enc0_act1_relu/Relu:
In[0] top_down_inference_model/find_instance_peaks/model/stack0_enc0_conv1/BiasAdd (defined at C:\Users\Ben07\mambaforge\New_folder\envs\sleap\lib\site-packages\keras\layers\convolutional.py:265)
Operation defined at: (most recent call last)
Function call stack:
predict_function -> predict_function
SLEAP: 1.3.3
TensorFlow: 2.7.0
Numpy: 1.21.6
Python: 3.7.12
OS: Windows-10-10.0.22621-SP0
System:
GPUs: 1/1 available
Device: /physical_device:GPU:0
Available: True
Initalized: False
Memory growth: True
Process return code: 1
Beta Was this translation helpful? Give feedback.
All reactions