Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

neuron runtime error #2

Open
littlemex opened this issue Mar 29, 2022 · 0 comments
Open

neuron runtime error #2

littlemex opened this issue Mar 29, 2022 · 0 comments

Comments

@littlemex
Copy link

littlemex commented Mar 29, 2022

I executed ./trace.sh according to README, but neuron runtime error has occered.

Part of error log.

Question: What does the little engine say?
2022-Mar-29 07:07:29.0082    11:11    ERROR   NRT:nrt_init                                Unable to determine Neuron Driver version. Please check aws-neuron-dkms package is installed.
Traceback (most recent call last):
  File "model-tracer.py", line 101, in <module>
    answer_logits = model_traced(*example_inputs)
  File "/usr/local/lib64/python3.7/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
RuntimeError: The following operation failed in the TorchScript interpreter.
Traceback of TorchScript (most recent call last):
/usr/local/lib64/python3.7/site-packages/torch_neuron/decorators.py(373): forward
/usr/local/lib64/python3.7/site-packages/torch/nn/modules/module.py(1090): _slow_forward
/usr/local/lib64/python3.7/site-packages/torch/nn/modules/module.py(1102): _call_impl
/usr/local/lib64/python3.7/site-packages/torch_neuron/graph.py(546): __call__
/usr/local/lib64/python3.7/site-packages/torch_neuron/graph.py(205): run_op
/usr/local/lib64/python3.7/site-packages/torch_neuron/graph.py(194): __call__
/usr/local/lib64/python3.7/site-packages/torch_neuron/convert.py(217): forward
/usr/local/lib64/python3.7/site-packages/torch/nn/modules/module.py(1090): _slow_forward
/usr/local/lib64/python3.7/site-packages/torch/nn/modules/module.py(1102): _call_impl
/usr/local/lib64/python3.7/site-packages/torch/jit/_trace.py(965): trace_module
/usr/local/lib64/python3.7/site-packages/torch/jit/_trace.py(750): trace
/usr/local/lib64/python3.7/site-packages/torch_neuron/convert.py(183): trace
model-tracer.py(92): <module>
RuntimeError: The PyTorch Neuron Runtime could not be initialized. Neuron Driver issues are logged
to your system logs. See the Neuron Runtime's troubleshooting guide for help on this
topic: https://awsdocs-neuron.readthedocs-hosted.com/en/latest/

I changed 1-build/Dockerfile-base-inf#L16

After changed, worked normally in my environment.

RUN yum update -y && \
    yum install -y python3 python3-devel gcc-c++ && \
    yum install -y tar gzip ca-certificates procps net-tools which vim wget libgomp htop jq bind-utils bc && \
    yum install -y kernel-devel-$(uname -r) kernel-headers-$(uname -r) aws-neuron-dkms aws-neuron-tools # I changed here.

I' AWS employee in Japan. alias akazawt

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant