Running Dockerized Deepsparse on MacOS

[djacobs7]

Hi,

I am trying to run Deepsparse on MacOS. I created a docker with ubuntu:20.04. I have a machine with the x86 instruction set and the avx2 support.

I was able to succesfully pip install deepsparse

I tried to run the following code from one of the tutorials

from deepsparse import compile_model
import numpy

batch_size = 64
sample_inputs = [numpy.random.randn(batch_size, 3, 224, 224).astype(numpy.float32)]

# run baseline benchmarking
engine_base = compile_model(
    model="zoo:cv/classification/resnet_v1-50/pytorch/sparseml/imagenet/base-none", 
    batch_size=batch_size,
)

I got the following error.

DeepSparse Engine, Copyright 2021-present / Neuralmagic, Inc. version: 0.2.0 (8a9ec179) (release) (optimized) (system=avx2, binary=avx2)
Traceback (most recent call last):
  File "test.py", line 10, in <module>
    engine_base = compile_model(
  File "/home/nm_user/.local/lib/python3.8/site-packages/deepsparse/engine.py", line 515, in compile_model
    return Engine(model, batch_size, num_cores, num_sockets)
  File "/home/nm_user/.local/lib/python3.8/site-packages/deepsparse/engine.py", line 154, in __init__
    self._eng_net = LIB.deepsparse_engine(
RuntimeError: NM: error: error setting preferred memory policy for engine (0) on socket (0), errno=1, msg=Operation not permitted

I dont think I can trace the error further back than that, because it seems to be coming from deepsparse_engine.so ( I think?).

Do you have any idea what might be going on here?

[bnellnm]

There are a couple possible workarounds for this issue.

1. You can try running the docker as root which should allow the call to set_mempolicy to succeed.
2. Make sure you compile the model with num_sockets=1.
3. set NM_DISABLE_BATCH_SPLITTING=1 in your environment (this should have the same effect as compiling with num_sockets=1)

Note that options 2&3 will prevent the engine from running on more than one socket

[bnellnm]

I've got an update to my previous answer. It appears that docker running on MacOSX, does not report the number of cores/sockets on a system properly. We verified this on some of our Mac laptops and discovered that cores were being reported as individual sockets, e.g. a 1 socket/8 core machine would be reported as 8 sockets with 1 core each.

So, even with the above workarounds, it is likely that the engine will run very poorly. We will be working on a better solution to this problem soon.

[djacobs7]

Thanks. I had not had a chance to try yet, but I will still give it a shot. Is that idea that with 1 socket this would work basically the same as if it was running in a non-optimized way on a CPU?

In any case, I'll can try it out and send you the benchmarks so you have a reference ( if you haven't checked already ).

[djacobs7]

1. Thanks! The things you suggested ( using sudo and setting num_sockets=1) solved the problem and I was able to run some of the models. See section (3) of this note for benchmark results.

2. Some models did not work at all. The model we are using is based on yolo, so I wanted to try out the yolo model from Neural Magic.

I tried

batch_size = 64
sample_inputs = [numpy.random.randn(batch_size, 3, 640, 640).astype(numpy.float32)]
print("zoo:cv/detection/yolo_v3-spp/pytorch/ultralytics/coco/base-none")
# run baseline benchmarking
engine_base = compile_model(
    model="zoo:cv/detection/yolo_v3-spp/pytorch/ultralytics/coco/base-none", 
    batch_size=batch_size,
    num_sockets = 1
)
benchmarks_base = engine_base.benchmark(sample_inputs)
print(benchmarks_base)

it successfully downloads the model, and then , I think sometime during the benchmarking, it just prints

Killed

I assume this is some sort of memory issue or something? Do you have any advice on how I might debug that?

3. If it's helpful, here are the benchmark results when compiled with num_sockets = 1:

sample_inputs = [numpy.random.randn(batch_size, 3, 224, 224).astype(numpy.float32)]
num_sockets = 1

**** model="zoo:cv/classification/resnet_v1-50/pytorch/sparseml/imagenet/base-none",

102472864/102472864 [00:58<00:00, 1752622.06it/s]
DeepSparse Engine, Copyright 2021-present / Neuralmagic, Inc. version: 0.2.0 (8a9ec179) (release) (optimized) (system=avx2, binary=avx2)
BenchmarkResults:
items_per_second: 8.068620504793575
ms_per_batch: 7931.963086128235
batch_times_mean: 7.931963086128235
batch_times_median: 7.943235158920288
batch_times_std: 0.05381609335646593

---

zoo:cv/classification/resnet_v1-50/pytorch/sparseml/imagenet/pruned-conservative
downloading...: 100%|█████████████████████████████████████████████| 102472864/102472864 [00:39<00:00, 2567872.67it/s]
DeepSparse Engine, Copyright 2021-present / Neuralmagic, Inc. version: 0.2.0 (8a9ec179) (release) (optimized) (system=avx2, binary=avx2)
WARNING: VNNI instructions not detected, quantization speedup not well supported

BenchmarkResults:
items_per_second: 20.40812764780723
ms_per_batch: 3136.005473136902
batch_times_mean: 3.136005473136902
batch_times_median: 3.134085178375244
batch_times_std: 0.03332687198690534

zoo:cv/classification/resnet_v1-50/pytorch/sparseml/imagenet/pruned-moderate

downloading...: 100%|█████████████████████████████████████████████| 102472864/102472864 [00:42<00:00, 2388516.37it/s]
DeepSparse Engine, Copyright 2021-present / Neuralmagic, Inc. version: 0.2.0 (8a9ec179) (release) (optimized) (system=avx2, binary=avx2)
WARNING: VNNI instructions not detected, quantization speedup not well supported

BenchmarkResults:
items_per_second: 28.091248163923726
ms_per_batch: 2278.2896518707275
batch_times_mean: 2.2782896518707276
batch_times_median: 2.2740538120269775
batch_times_std: 0.0169216612542975

4. THANKS!

[bnellnm]

My guess about the yolo model is that you are running out of memory. Yolo is pretty memory hungry and batch size 64 will use a lot of resources. I'd suggest trying with a smaller batch size.

We have a fix for the number of sockets/cores + mempolicy issues that will be in the next release (if not sooner in a nightly release). You'll get really poor performance numbers until that fix is in since you'll essentially be running on a single core. We are also working on reducing our memory footprint but that is a longer term project.