00-anomaly-detection-tutorial: Container fails to start on ACI

[rolandkru]

Tutorial: 00-anomaly-detection-tutorial
Step: 5: Test model on Azure Container Instance

Issue: Container failed to start on ACI. Error log:

2020-06-28T11:55:11,852765769+00:00 - rsyslog/run
2020-06-28T11:55:11,852562969+00:00 - iot-server/run
2020-06-28T11:55:11,864671678+00:00 - nginx/run
2020-06-28T11:55:11,865188479+00:00 - gunicorn/run
EdgeHubConnectionString and IOTEDGE_IOTHUBHOSTNAME are not set. Exiting...
2020-06-28T11:55:12,130239579+00:00 - iot-server/finish 1 0
2020-06-28T11:55:12,135812284+00:00 - Exit code 1 is normal. Not restarting iot-server.
Starting gunicorn 19.9.0
Listening at: http://127.0.0.1:31311 (11)
Using worker: sync
worker timeout is set to 300
Booting worker with pid: 43
Exception in worker process
Traceback (most recent call last):
File "/opt/miniconda/lib/python3.6/site-packages/gunicorn/arbiter.py", line 583, in spawn_worker
worker.init_process()
File "/opt/miniconda/lib/python3.6/site-packages/gunicorn/workers/base.py", line 129, in init_process
self.load_wsgi()
File "/opt/miniconda/lib/python3.6/site-packages/gunicorn/workers/base.py", line 138, in load_wsgi
self.wsgi = self.app.wsgi()
File "/opt/miniconda/lib/python3.6/site-packages/gunicorn/app/base.py", line 67, in wsgi
self.callable = self.load()
File "/opt/miniconda/lib/python3.6/site-packages/gunicorn/app/wsgiapp.py", line 52, in load
return self.load_wsgiapp()
File "/opt/miniconda/lib/python3.6/site-packages/gunicorn/app/wsgiapp.py", line 41, in load_wsgiapp
return util.import_app(self.app_uri)
File "/opt/miniconda/lib/python3.6/site-packages/gunicorn/util.py", line 350, in import_app
import(module)
File "/var/azureml-server/wsgi.py", line 1, in
import create_app
File "/var/azureml-server/create_app.py", line 3, in
from app import main
File "/var/azureml-server/app.py", line 31, in
import main as user_main
File "/var/azureml-app/main.py", line 18, in
driver_module_spec.loader.exec_module(driver_module)
File "iot_score.py", line 8, in
from sklearn.externals import joblib
ImportError: cannot import name 'joblib'
Worker exiting (pid: 43)
Shutting down: Master
Reason: Worker failed to boot.
2020-06-28T11:55:15,652639446+00:00 - gunicorn/finish 3 0
2020-06-28T11:55:15,653865147+00:00 - Exit code 3 is not normal. Killing image.

[rolandkru]

Referencing joblib from top level helped me to solve this issue

[kiranpradeep]

I am also facing this issue. https://stackoverflow.com/a/61895560/1180117

[leestott]

Hi so using print(ws.webservices['mysvc'].get_logs())

I get the same error

2020-09-01T13:06:56,682182228+00:00 - iot-server/run
2020-09-01T13:06:56,683466724+00:00 - rsyslog/run
2020-09-01T13:06:56,713681736+00:00 - gunicorn/run
2020-09-01T13:06:56,721711113+00:00 - nginx/run

EdgeHubConnectionString and IOTEDGE_IOTHUBHOSTNAME are not set. Exiting...

2020-09-01T13:06:56,996583013+00:00 - iot-server/finish 1 0
2020-09-01T13:06:57,004199091+00:00 - Exit code 1 is normal. Not restarting iot-server. Starting gunicorn 19.9.0

Listening at: http://127.0.0.1:31311 (13)
Using worker: sync
worker timeout is set to 300
Booting worker with pid: 45
Exception in worker process
Traceback (most recent call last): File "/opt/miniconda/lib/python3.6/site-packages/gunicorn/arbiter.py", line 583, in spawn_worker worker.init_process() File "/opt/miniconda/lib/python3.6/site-packages/gunicorn/workers/base.py", line 129, in init_process self.load_wsgi() File "/opt/miniconda/lib/python3.6/site-packages/gunicorn/workers/base.py", line 138, in load_wsgi self.wsgi = self.app.wsgi() File "/opt/miniconda/lib/python3.6/site-packages/gunicorn/app/base.py", line 67, in wsgi self.callable = self.load() File "/opt/miniconda/lib/python3.6/site-packages/gunicorn/app/wsgiapp.py", line 52, in load return self.load_wsgiapp() File "/opt/miniconda/lib/python3.6/site-packages/gunicorn/app/wsgiapp.py", line 41, in load_wsgiapp return util.import_app(self.app_uri) File "/opt/miniconda/lib/python3.6/site-packages/gunicorn/util.py", line 350, in import_app import(module) File "/var/azureml-server/wsgi.py", line 1, in import create_app File "/var/azureml-server/create_app.py", line 3, in from app import main File "/var/azureml-server/app.py", line 31, in import main as user_main File "/var/azureml-app/main.py", line 12, in driver_module_spec.loader.exec_module(driver_module) File "/var/azureml-app/iot_score.py", line 10, in from azureml.core.model import Model File "/opt/miniconda/lib/python3.6/site-packages/azureml/core/init.py", line 13, in from .workspace import Workspace File "/opt/miniconda/lib/python3.6/site-packages/azureml/core/workspace.py", line 34, in from azureml.core.image import Image File "/opt/miniconda/lib/python3.6/site-packages/azureml/core/image/init.py", line 28, in from .image import Image File "/opt/miniconda/lib/python3.6/site-packages/azureml/core/image/image.py", line 19, in from azureml.core.model import Model File "/opt/miniconda/lib/python3.6/site-packages/azureml/core/model.py", line 39, in from azureml.core.environment import Environment File "/opt/miniconda/lib/python3.6/site-packages/azureml/core/environment.py", line 31, in from azureml.core.conda_dependencies import CondaDependencies, PYTHON_DEFAULT_VERSION File "/opt/miniconda/lib/python3.6/site-packages/azureml/core/conda_dependencies.py", line 11, in import ruamel.yaml ModuleNotFoundError: No module named 'ruamel' Worker exiting (pid: 45) Shutting down: Master Reason: Worker failed to boot. 2020-09-01T13:07:01,788170279+00:00 - gunicorn/finish 3 0 2020-09-01T13:07:01,789377376+00:00 - Exit code 3 is not normal. Killing image.

[kawofong]

I am also getting the same error ModuleNotFoundError: No module named 'ruamel' after deploying the ACI webservice. From a quick internet search, I found that this is a known issue (https://docs.microsoft.com/en-us/azure/machine-learning/resource-known-issues#installation-and-import):

However, it is not clear how we can apply the suggested fix to the anomaly-detection-tutorial notebook.

[leestott]

@kawo123 yes same issue

Output log shows -

import ruamel.yaml ModuleNotFoundError: No module named 'ruamel' Worker exiting (pid: 45) Shutting down: Master Reason: Worker failed to boot. 2020-09-10T06:55:03,932157139+00:00 - gunicorn/finish 3 0 2020-09-10T06:55:03,933713250+00:00 - Exit code 3 is not normal. Killing image.

So pip install --upgrade azureml-sdk[notebooks,automl] --ignore-installed PyYAML needs to be added as requirement to the docker build step.

Create docker Image

Create docker Image This specifies the dependencies to include in the environment

from azureml.core.conda_dependencies import CondaDependencies

myenv = CondaDependencies.create(conda_packages=['pandas', 'scikit-learn', 'numpy'])

with open("myenv.yml","w") as f:
f.write(myenv.serialize_to_string())

[Karthika-Arumugam]

Included ruamel & ruamel.yaml as CondaDependencies while creating the docker image like below:

This specifies the dependencies to include in the environment

from azureml.core.conda_dependencies import CondaDependencies

myenv = CondaDependencies.create(conda_packages=['pandas', 'scikit-learn', 'numpy','ruamel','ruamel.yaml'])

with open("myenv.yml","w") as f:
f.write(myenv.serialize_to_string())

This fixed the issue for me related to deploying the ACI web service with the created image.