Improving performance of Tensorflow with AMD Radeon card on WSL

[ff137]

Good day,

I am running Tensorflow code through WSL (Ubuntu). I have an AMD Radeon card, and wish to utilise Tensorflow DirectML to train a model using the GPU. I have used Nvidia / CUDA, on pure Ubuntu, in the past, but have no experience running Tensorflow with AMD cards, let alone through WSL.

I have configured everything as detailed here, and my GPU is now available for running with Tensorflow. The code runs, but poorly.

There is a significant performance downgrade when going from CPU to GPU.
For context, the hardware specs are:

• CPU: AMD Ryzen 5 3600X (6 core 4 GHz)
• GPU: AMD Radeon RX 5700 XT (8 GB VRAM)

Additionally, it seems to require significantly more RAM for the same amount of input data being trained on (it's blue screened twice so far during testing, but I can get it to run on smaller datasets).

I would expect the GPU performance to be at least on par with CPU. I don't know how reasonable this assumption is with DirectML and the above hardware specs.

I can see in Task Manager that only 5% of GPU is being utilized when running a fairly large model/dataset. This reporting appears to be accurate (showing WSL usage, not just Windows), as GPU memory usage goes up when Tensorflow model sets up, while the utilization remains low.

My best guess for this discrepancy is probably that the GPU resources are being shared between Windows and WSL, and Windows is reserving most of it. I don't know how likely that is, because I won't expect rendering to keep 95% of the card on standby. Who knows. Windows apps could be using the GPU engine.

My instinct is to unplug my monitor's display port cable from the GPU and swap to using the onboard GPU instead, with the belief that the GPU would then not be utilised by Windows, and become fully available to WSL.

For some reason, my onboard HDMI port is not working, which I can probably fix by updating all the drivers.
But, before I go through that, I figured I should first ask the developers or community: what would be the best practice for getting WSL/Tensorflow to fully utilise the system GPU? Is it possible?

Does anyone have experience with running Tensorflow ... on WSL ... with an AMD Radeon card ... and optimizing it?
Any insight will be appreciated! Thank you.

Further details:

• Windows 10: Version 10.0.19045 Build 19045
• WSL Kernel: Linux version 5.10.102.1-microsoft-standard-WSL2
• Driver Version: 22.20.29.10-221130a-386458C-AMD-Software-Adrenalin-Edition
• AMD Windows Driver Version: 31.0.12029.10015

[PatriceVignola]

The most likely culprit here is that the model may have have some operations that DirectML doesn't support yet, and therefore needs to frequently fall back to the CPU. RX 5700 XT should definitely be faster than the CPU. It would help if you could direct us to the model you've been using so we can try and repro the issue!

[ff137]

Thanks for the feedback!

So, in general, following the steps I described should allow Tensorflow on WSL to have full access to the GPU?

The model I am testing is fairly simple. It's a Sequential model consisting of 2 Dense layers, with Batch Normalisation in the mix. So it doesn't feel like anything fancy, or any operations that shouldn't be supported. To give an idea:

model = Sequential()
model.add(Dense(neurons1, input_dim=num_inputs))

if batchnorm:
    model.add(BatchNormalization())
model.add(Activation(activation1))
model.add(Dropout(dropout1))

model.add(Dense(neurons2))
if batchnorm2:
    model.add(BatchNormalization())
model.add(Activation(activation2))

model.add(Dense(num_outputs, activation=activation_out))

opt = Adam(learning_rate=learning_rate, beta_1=0.99987,
           beta_2=0.9999985, epsilon=1e-12, amsgrad=True)

model.compile(loss="mean_squared_error", optimizer=opt,
              metrics=[RootMeanSquaredError()])`

Activations1 and 2 are relu, with tanh out.

Could it be the optimizer settings? Please let me know if there's anything I can test to narrow down the cause.

[PatriceVignola]

This model looks fine to me, but it's also a fairly small and simple model so it's possible that the CPU is faster since the constant GPU overhead may be bigger than the GPU performance. But it's hard to investigate deeper without having a standalone repro.

[ff137]

The dataset is fairly large (1M+ rows, 120 inputs, 20 outputs).

On CPU, 1 epoch takes about 3 minutes with a small batch size. When I tested with GPU, it gave an ETA of 5+ minutes. The memory usage was also significantly greater / model took longer to initialize and start fitting with GPU, so I figured there must be something wrong with resource availability.

I can't replicate this model + dataset for you very easily. I'll have to experiment some more. With the confirmation that it should be working as is, I'll do some more testing and report back what I can figure out.

[ff137]

@PatriceVignola As it turns out ... tensorflow 2.11 was installed, together with tensorflow-cpu 2.10. So I had not been running the plugin correctly. I'm using keras wrappers, so it'll take some reconfiguring to get it running with tensorflow-cpu.

[ff137]

Here is a printout of logs when running with GPU.
This takes longer to start running than with CPU, and uses more RAM.
GPU utilization is actually non-existent at 3%, but it does load onto GPU memory.

2022-12-14 12:11:05.292428: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2022-12-14 12:11:06.193805: I tensorflow/c/logging.cc:34] Successfully opened dynamic library libdirectml.d6f03b303ac3c4f2eeb8ca631688c9757b361310.so
2022-12-14 12:11:06.193886: I tensorflow/c/logging.cc:34] Successfully opened dynamic library libdxcore.so
2022-12-14 12:11:06.205491: I tensorflow/c/logging.cc:34] Successfully opened dynamic library libd3d12.so
2022-12-14 12:11:09.453040: I tensorflow/c/logging.cc:34] DirectML device enumeration: found 1 compatible adapters.

GPU AVAILABLE !
[PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')]

[INFO] initializing model...

2022-12-14 12:11:19.407781: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2022-12-14 12:11:19.409137: I tensorflow/c/logging.cc:34] DirectML: creating device on adapter 0 (AMD Radeon RX 5700 XT)
2022-12-14 12:11:20.570090: I tensorflow/core/common_runtime/pluggable_device/pluggable_device_factory.cc:306] Could not identify NUMA node of platform GPU ID 0, defaulting to 0. Your kernel may not have been built with NUMA support.
2022-12-14 12:11:20.570161: W tensorflow/core/common_runtime/pluggable_device/pluggable_device_bfc_allocator.cc:28] Overriding allow_growth setting because force_memory_growth was requested by the device.
2022-12-14 12:11:21.814074: I tensorflow/core/common_runtime/pluggable_device/pluggable_device_factory.cc:272] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 21596 MB memory) -> physical PluggableDevice (device: 0, name: DML, pci bus id: <undefined>)

[INFO] training model... 
2022-12-14 12:11:23.561599: W tensorflow/core/framework/cpu_allocator_impl.cc:82] Allocation of 1365382656 exceeds 10% of free system memory.
2022-12-14 12:11:29.236492: W tensorflow/core/framework/cpu_allocator_impl.cc:82] Allocation of 1365382656 exceeds 10% of free system memory.

Epoch 1/18
2022-12-14 12:11:39.262755: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:114] Plugin optimizer for device_type GPU is enabled.
 6088/43362 [===>..........................] - ETA: 4:14 - loss: 0.0175 - root_mean_squared_error: 0.1322

Versus when disabling the GPU and running with CPU:

2022-12-14 12:15:47.551740: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2022-12-14 12:15:48.679375: I tensorflow/c/logging.cc:34] Successfully opened dynamic library libdirectml.d6f03b303ac3c4f2eeb8ca631688c9757b361310.so
2022-12-14 12:15:48.679552: I tensorflow/c/logging.cc:34] Successfully opened dynamic library libdxcore.so
2022-12-14 12:15:48.694328: I tensorflow/c/logging.cc:34] Successfully opened dynamic library libd3d12.so
2022-12-14 12:15:51.599152: I tensorflow/c/logging.cc:34] DirectML device enumeration: found 1 compatible adapters.

GPU AVAILABLE !
[PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')]
Disabling GPU

[INFO] initializing model...

2022-12-14 12:16:04.797672: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.

[INFO] training model...
2022-12-14 12:16:06.727254: W tensorflow/core/framework/cpu_allocator_impl.cc:82] Allocation of 1365382656 exceeds 10% of free system memory.
Epoch 1/18
 1555/43362 [>.............................] - ETA: 2:06 - loss: 0.0177 - root_mean_squared_error: 0.1331   

[ff137]

@PatriceVignola Does anything stand out for you from these logs?