Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Not able to optimize tensorflow (tf 1.14) object detection model graph #289

Open
purvang3 opened this issue Apr 7, 2022 · 0 comments
Open

Not able to optimize tensorflow (tf 1.14) object detection model graph #289

purvang3 opened this issue Apr 7, 2022 · 0 comments

Comments

@purvang3
Copy link

purvang3 commented Apr 7, 2022

I followed guide from "https://on-demand.gputechconf.com/gtc-cn/2019/pdf/CN9456/presentation.pdf". I am able to produce
tensorrt based graph def but when I load and run on to image using converted graph, due to key error, getting following error.

2022-04-05 20:54:55.842903: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcuda.so.1
2022-04-05 20:54:55.871025: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there m
ust be at least one NUMA node, so returning NUMA node zero
2022-04-05 20:54:55.871433: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties:
name: GeForce RTX 2080 major: 7 minor: 5 memoryClockRate(GHz): 1.8
pciBusID: 0000:01:00.0
2022-04-05 20:54:55.871725: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.0
2022-04-05 20:54:55.872640: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10.0
2022-04-05 20:54:55.873482: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10.0
2022-04-05 20:54:55.873775: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10.0
2022-04-05 20:54:55.874721: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10.0
2022-04-05 20:54:55.875525: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10.0
2022-04-05 20:54:55.877523: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2022-04-05 20:54:55.877615: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there m
ust be at least one NUMA node, so returning NUMA node zero
2022-04-05 20:54:55.878174: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there m
ust be at least one NUMA node, so returning NUMA node zero
2022-04-05 20:54:55.878593: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0
2022-04-05 20:54:55.879006: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to us
e: AVX2 FMA
2022-04-05 20:54:55.949906: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there m
ust be at least one NUMA node, so returning NUMA node zero
2022-04-05 20:54:55.950377: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x54582e0 executing computations on platform CUDA. Devices:
2022-04-05 20:54:55.950405: I tensorflow/compiler/xla/service/service.cc:175] StreamExecutor device (0): GeForce RTX 2080, Compute Capability 7.5
2022-04-05 20:54:55.952053: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 3600000000 Hz
2022-04-05 20:54:55.952579: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x5ac41f0 executing computations on platform Host. Devices:
2022-04-05 20:54:55.952591: I tensorflow/compiler/xla/service/service.cc:175] StreamExecutor device (0): ,
2022-04-05 20:54:55.952725: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there m
ust be at least one NUMA node, so returning NUMA node zero
2022-04-05 20:54:55.953043: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties:
name: GeForce RTX 2080 major: 7 minor: 5 memoryClockRate(GHz): 1.8
pciBusID: 0000:01:00.0
2022-04-05 20:54:55.953091: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.0
2022-04-05 20:54:55.953102: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10.0
2022-04-05 20:54:55.953113: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10.0
2022-04-05 20:54:55.953123: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10.0
2022-04-05 20:54:55.953162: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10.0

2022-04-05 20:54:55.953171: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10.0 [35/1991]
2022-04-05 20:54:55.953183: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2022-04-05 20:54:55.953221: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there m
ust be at least one NUMA node, so returning NUMA node zero
2022-04-05 20:54:55.953471: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there m
ust be at least one NUMA node, so returning NUMA node zero
2022-04-05 20:54:55.953703: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0
2022-04-05 20:54:55.953727: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.0
2022-04-05 20:54:55.954313: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
2022-04-05 20:54:55.954322: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187] 0
2022-04-05 20:54:55.954326: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0: N
2022-04-05 20:54:55.954402: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there m
ust be at least one NUMA node, so returning NUMA node zero
2022-04-05 20:54:55.954757: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there m
ust be at least one NUMA node, so returning NUMA node zero
2022-04-05 20:54:55.955005: W tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:40] Overriding allow_growth setting because the TF_FORCE_GPU_ALLOW_GROWTH en
vironment variable is set. Original config value was 0.
2022-04-05 20:54:55.955020: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with
7214 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2080, pci bus id: 0000:01:00.0, compute capability: 7.5)
WARNING:tensorflow:From trr.py:13: The name tf.train.import_meta_graph is deprecated. Please use tf.compat.v1.train.import_meta_graph instead.

WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/training/saver.py:1276: checkpoint_exists (from tensorflow.python.training.checkp
oint_management) is deprecated and will be removed in a future version.
Instructions for updating:
Use standard file APIs to check for files with this prefix.
WARNING:tensorflow:From trr.py:17: convert_variables_to_constants (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future
version.
Instructions for updating:
Use tf.compat.v1.graph_util.convert_variables_to_constants
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/graph_util_impl.py:270: extract_sub_graph (from tensorflow.python.frame
work.graph_util_impl) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.compat.v1.graph_util.extract_sub_graph
2022-04-05 20:54:58.248773: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there m
ust be at least one NUMA node, so returning NUMA node zero
2022-04-05 20:54:58.249250: I tensorflow/core/grappler/devices.cc:55] Number of eligible GPUs (core count >= 8, compute capability >= 0.0): 1
2022-04-05 20:54:58.249793: I tensorflow/core/grappler/clusters/single_machine.cc:359] Starting new session
2022-04-05 20:54:58.250844: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there $ust be at least one NUMA node, so returning NUMA node zero
2022-04-05 20:54:58.251104: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties:
name: GeForce RTX 2080 major: 7 minor: 5 memoryClockRate(GHz): 1.8
pciBusID: 0000:01:00.0
2022-04-05 20:54:58.2511

2022-04-05 20:54:58.249250: I tensorflow/core/grappler/devices.cc:55] Number of eligible GPUs (core count >= 8, compute capability >= 0.0): 1 [0/1991]
2022-04-05 20:54:58.249793: I tensorflow/core/grappler/clusters/single_machine.cc:359] Starting new session
2022-04-05 20:54:58.250844: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-04-05 20:54:58.251104: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties:
name: GeForce RTX 2080 major: 7 minor: 5 memoryClockRate(GHz): 1.8
pciBusID: 0000:01:00.0
2022-04-05 20:54:58.251144: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.0
2022-04-05 20:54:58.251175: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10.0
2022-04-05 20:54:58.251225: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10.0
2022-04-05 20:54:58.251239: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10.0
2022-04-05 20:54:58.251250: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10.0
2022-04-05 20:54:58.251263: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10.0
2022-04-05 20:54:58.251274: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2022-04-05 20:54:58.251311: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-04-05 20:54:58.251570: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-04-05 20:54:58.251786: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0
2022-04-05 20:54:58.251804: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
2022-04-05 20:54:58.251808: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187] 0
2022-04-05 20:54:58.251812: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0: N
2022-04-05 20:54:58.251898: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-04-05 20:54:58.252173: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-04-05 20:54:58.252404: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with
7214 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2080, pci bus id: 0000:01:00.0, compute capability: 7.5)
2022-04-05 20:54:58.928171: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:716] Optimization results for grappler item: tf_graph
2022-04-05 20:54:58.928197: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:718] constant folding: Graph size after: 2817 nodes (-768), 3934 edges (-797), time = 278.71ms.
2022-04-05 20:54:58.928202: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:718] layout: Graph size after: 2866 nodes (49), 3984 edges (50), time = 84.02ms.
2022-04-05 20:54:58.928205: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:718] constant folding: Graph size after: 2856 nodes (-10), 3984 edges (0), time = 177.412ms.
WARNING:tensorflow:From trr.py:23: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead.

2022-04-05 20:54:59.733245: F tensorflow/core/grappler/utils.cc:120] Check failed: ret.second Pair (ConstantFolding/Postprocessor/Decode/truediv_4_recip,0x37657e90) is not inserted because the same key already exists.
Aborted (core dumped)

Environment
TensorRT Version: compiled with tensorflow 1.14 (tensorflow.contrib.tensorrt)
NVIDIA GPU: GeForce RTX 2080
NVIDIA Driver Version: 450.102.04
CUDA Version: 10.0.130
CUDNN Version: 7.5.0.56
Operating System: Ubuntu 18.04
Python Version (if applicable): 3.6.7
Tensorflow Version (if applicable): 1.14
PyTorch Version (if applicable):
Baremetal or Container (if so, version): container

Description
I followed guide from "https://on-demand.gputechconf.com/gtc-cn/2019/pdf/CN9456/presentation.pdf". I am able to produce
tensorrt based graph def but when I load and run on to image using converted graph, due to key error, getting following error.

2022-04-05 20:54:55.842903: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcuda.so.1
2022-04-05 20:54:55.871025: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there m
ust be at least one NUMA node, so returning NUMA node zero
2022-04-05 20:54:55.871433: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties:
name: GeForce RTX 2080 major: 7 minor: 5 memoryClockRate(GHz): 1.8
pciBusID: 0000:01:00.0
2022-04-05 20:54:55.871725: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.0
2022-04-05 20:54:55.872640: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10.0
2022-04-05 20:54:55.873482: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10.0
2022-04-05 20:54:55.873775: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10.0
2022-04-05 20:54:55.874721: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10.0
2022-04-05 20:54:55.875525: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10.0
2022-04-05 20:54:55.877523: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2022-04-05 20:54:55.877615: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there m
ust be at least one NUMA node, so returning NUMA node zero
2022-04-05 20:54:55.878174: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there m
ust be at least one NUMA node, so returning NUMA node zero
2022-04-05 20:54:55.878593: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0
2022-04-05 20:54:55.879006: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to us
e: AVX2 FMA
2022-04-05 20:54:55.949906: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there m
ust be at least one NUMA node, so returning NUMA node zero
2022-04-05 20:54:55.950377: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x54582e0 executing computations on platform CUDA. Devices:
2022-04-05 20:54:55.950405: I tensorflow/compiler/xla/service/service.cc:175] StreamExecutor device (0): GeForce RTX 2080, Compute Capability 7.5
2022-04-05 20:54:55.952053: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 3600000000 Hz
2022-04-05 20:54:55.952579: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x5ac41f0 executing computations on platform Host. Devices:
2022-04-05 20:54:55.952591: I tensorflow/compiler/xla/service/service.cc:175] StreamExecutor device (0): ,
2022-04-05 20:54:55.952725: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there m
ust be at least one NUMA node, so returning NUMA node zero
2022-04-05 20:54:55.953043: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties:
name: GeForce RTX 2080 major: 7 minor: 5 memoryClockRate(GHz): 1.8
pciBusID: 0000:01:00.0
2022-04-05 20:54:55.953091: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.0
2022-04-05 20:54:55.953102: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10.0
2022-04-05 20:54:55.953113: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10.0
2022-04-05 20:54:55.953123: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10.0
2022-04-05 20:54:55.953162: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10.0

2022-04-05 20:54:55.953171: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10.0 [35/1991]
2022-04-05 20:54:55.953183: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2022-04-05 20:54:55.953221: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there m
ust be at least one NUMA node, so returning NUMA node zero
2022-04-05 20:54:55.953471: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there m
ust be at least one NUMA node, so returning NUMA node zero
2022-04-05 20:54:55.953703: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0
2022-04-05 20:54:55.953727: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.0
2022-04-05 20:54:55.954313: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
2022-04-05 20:54:55.954322: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187] 0
2022-04-05 20:54:55.954326: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0: N
2022-04-05 20:54:55.954402: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there m
ust be at least one NUMA node, so returning NUMA node zero
2022-04-05 20:54:55.954757: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there m
ust be at least one NUMA node, so returning NUMA node zero
2022-04-05 20:54:55.955005: W tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:40] Overriding allow_growth setting because the TF_FORCE_GPU_ALLOW_GROWTH en
vironment variable is set. Original config value was 0.
2022-04-05 20:54:55.955020: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with
7214 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2080, pci bus id: 0000:01:00.0, compute capability: 7.5)
WARNING:tensorflow:From trr.py:13: The name tf.train.import_meta_graph is deprecated. Please use tf.compat.v1.train.import_meta_graph instead.

WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/training/saver.py:1276: checkpoint_exists (from tensorflow.python.training.checkp
oint_management) is deprecated and will be removed in a future version.
Instructions for updating:
Use standard file APIs to check for files with this prefix.
WARNING:tensorflow:From trr.py:17: convert_variables_to_constants (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future
version.
Instructions for updating:
Use tf.compat.v1.graph_util.convert_variables_to_constants
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/graph_util_impl.py:270: extract_sub_graph (from tensorflow.python.frame
work.graph_util_impl) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.compat.v1.graph_util.extract_sub_graph
2022-04-05 20:54:58.248773: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there m
ust be at least one NUMA node, so returning NUMA node zero
2022-04-05 20:54:58.249250: I tensorflow/core/grappler/devices.cc:55] Number of eligible GPUs (core count >= 8, compute capability >= 0.0): 1
2022-04-05 20:54:58.249793: I tensorflow/core/grappler/clusters/single_machine.cc:359] Starting new session
2022-04-05 20:54:58.250844: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there $ust be at least one NUMA node, so returning NUMA node zero
2022-04-05 20:54:58.251104: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties:
name: GeForce RTX 2080 major: 7 minor: 5 memoryClockRate(GHz): 1.8
pciBusID: 0000:01:00.0
2022-04-05 20:54:58.2511

2022-04-05 20:54:58.249250: I tensorflow/core/grappler/devices.cc:55] Number of eligible GPUs (core count >= 8, compute capability >= 0.0): 1 [0/1991]
2022-04-05 20:54:58.249793: I tensorflow/core/grappler/clusters/single_machine.cc:359] Starting new session
2022-04-05 20:54:58.250844: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-04-05 20:54:58.251104: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties:
name: GeForce RTX 2080 major: 7 minor: 5 memoryClockRate(GHz): 1.8
pciBusID: 0000:01:00.0
2022-04-05 20:54:58.251144: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.0
2022-04-05 20:54:58.251175: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10.0
2022-04-05 20:54:58.251225: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10.0
2022-04-05 20:54:58.251239: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10.0
2022-04-05 20:54:58.251250: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10.0
2022-04-05 20:54:58.251263: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10.0
2022-04-05 20:54:58.251274: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2022-04-05 20:54:58.251311: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-04-05 20:54:58.251570: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-04-05 20:54:58.251786: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0
2022-04-05 20:54:58.251804: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
2022-04-05 20:54:58.251808: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187] 0
2022-04-05 20:54:58.251812: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0: N
2022-04-05 20:54:58.251898: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-04-05 20:54:58.252173: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-04-05 20:54:58.252404: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with
7214 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2080, pci bus id: 0000:01:00.0, compute capability: 7.5)
2022-04-05 20:54:58.928171: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:716] Optimization results for grappler item: tf_graph
2022-04-05 20:54:58.928197: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:718] constant folding: Graph size after: 2817 nodes (-768), 3934 edges (-797), time = 278.71ms.
2022-04-05 20:54:58.928202: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:718] layout: Graph size after: 2866 nodes (49), 3984 edges (50), time = 84.02ms.
2022-04-05 20:54:58.928205: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:718] constant folding: Graph size after: 2856 nodes (-10), 3984 edges (0), time = 177.412ms.
WARNING:tensorflow:From trr.py:23: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead.

2022-04-05 20:54:59.733245: F tensorflow/core/grappler/utils.cc:120] Check failed: ret.second Pair (ConstantFolding/Postprocessor/Decode/truediv_4_recip,0x37657e90) is not inserted because the same key already exists.
Aborted (core dumped)

Environment
TensorRT Version: compiled with tensorflow 1.14 (tensorflow.contrib.tensorrt)
NVIDIA GPU: GeForce RTX 2080
NVIDIA Driver Version: 450.102.04
CUDA Version: 10.0.130
CUDNN Version: 7.5.0.56
Operating System: Ubuntu 18.04
Python Version (if applicable): 3.6.7
Tensorflow Version (if applicable): 1.14
PyTorch Version (if applicable):
Baremetal or Container (if so, version): container

please let me know if more information needed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@purvang3