Azure ML extension throws warnings even after muting the warnings by only_show_errors

[eddmik]

Describe the bug

In the Azure Devops Pipeline I do not want to receive any warnings as the pipeline fails. Therefore my idea was to mute the warning s by only-show-errors. Still I am recevieng the following messages that make my pipeline to file however the job is submitted. When running on compute instance it does not throw an error but the message about classes are there.

##[error]Class AutoDeleteSettingSchema: This is an experimental class, and may change at any time. Please see https://aka.ms/azuremlexperimental for more information.
Class AutoDeleteConditionSchema: This is an experimental class, and may change at any time. Please see https://aka.ms/azuremlexperimental for more information.
Class BaseAutoDeleteSettingSchema: This is an experimental class, and may change at any time. Please see https://aka.ms/azuremlexperimental for more information.
Class IntellectualPropertySchema: This is an experimental class, and may change at any time. Please see https://aka.ms/azuremlexperimental for more information.
Class ProtectionLevelSchema: This is an experimental class, and may change at any time. Please see https://aka.ms/azuremlexperimental for more information.
Class BaseIntellectualPropertySchema: This is an experimental class, and may change at any time. Please see https://aka.ms/azuremlexperimental for more information.

Related command

az config set core.only_show_errors=true --only-show-errors
az extension add --name ml --version 2.25.0
az --version
az ml job create --file $yaml_path

Errors

  "type": "pipeline"
}

##[error]Class AutoDeleteSettingSchema: This is an experimental class, and may change at any time. Please see https://aka.ms/azuremlexperimental for more information.
Class AutoDeleteConditionSchema: This is an experimental class, and may change at any time. Please see https://aka.ms/azuremlexperimental for more information.
Class BaseAutoDeleteSettingSchema: This is an experimental class, and may change at any time. Please see https://aka.ms/azuremlexperimental for more information.
Class IntellectualPropertySchema: This is an experimental class, and may change at any time. Please see https://aka.ms/azuremlexperimental for more information.
Class ProtectionLevelSchema: This is an experimental class, and may change at any time. Please see https://aka.ms/azuremlexperimental for more information.
Class BaseIntellectualPropertySchema: This is an experimental class, and may change at any time. Please see https://aka.ms/azuremlexperimental for more information.
##[error]Script has output to stderr. Failing as failOnStdErr is set to true.
/usr/bin/az account clear
Finishing: AzureCLI

Issue script & Debug output

Environment: data-env with Version: 4 exists!
$schema: https://azuremlschemas.azureedge.net/latest/pipelineJob.schema.json
name: pipeline-data-2126
type: pipeline
experiment_name: 00-pipeline-data
display_name: Data_Pipeline
jobs:
  data_job:
    code: /home/vsts/work/1/s/self
    command: pip install -e . && python pppsg/dataset.py --output_path ${{inputs.output_folder_name}}
    inputs:
      output_folder_name: "./outputs"
    compute: azureml:cpu-cluster
    environment: azureml:data-env:4DEBUG: cli.knack.cli: Command arguments: ['ml', 'job', 'create', '--file', './ml_pipelines/pipeline_data.yml', '--debug']
DEBUG: cli.knack.cli: __init__ debug log:

DEBUG: cli.knack.cli: Event: CommandInvoker.OnPreParseArgs []
DEBUG: cli.knack.cli: Event: CommandInvoker.OnPostParseArgs [<function OutputProducer.handle_output_argument at 0x7fcd3aba6200>, <function CLIQuery.handle_query_parameter at 0x7fcd3ace0>, <function register_ids_argument.<locals>.parse_ids_arguments at 0x7fcd390c0>]
INFO: az_command_data_logger: extension name: ml
INFO: az_command_data_logger: extension version: 2.25.0
DEBUG: cli.azure.cli.core.commands.client_factory: Getting management service client client_type=AzureMachineLearningWorkspaces
DEBUG: cli.azure.cli.core.auth.persistence: build_persistence: location='/home/vsts/work/_temp/.azclitask/service_principal_entries.json', encrypt=False
DEBUG: cli.azure.cli.core.auth.persistence: build_persistence: location='/home/vsts/work/_temp/.azclitask/msal_token_cache.json', encrypt=False
DEBUG: cli.azure.cli.core.auth.binary_cache: load: /home/vsts/work/_temp/.azclitask/msal_http_cache.bin
DEBUG: urllib3.util.retry: Converted retries value: 1 -> Retry(total=1, connect=None, read=None, redirect=None, status=None)
DEBUG: msal.authority: openid_config = {'token_endpoint': 'https://login.microsoftonline.com/****/oauth2/v2.0/token', 'token_endpoint_auth_methods_supported': ['client_secret_post', 'private_key_jwt', 'client_secret_basic'], 'jwks_uri': 'https://login.microsoftonline.com/****/discovery/v2.0/keys', 'response_modes_supported': ['query', 'fragment', 'form_post'], 'subject_types_supported': ['pairwise'], 'id_token_signing_alg_values_supported': ['RS256'], 'response_types_supported': ['code', 'id_token', 'code id_token', 'id_token token'], 'scopes_supported': ['openid', 'profile', 'email', 'offline_access'], 'issuer': 'https://login.microsoftonline.com/****/v2.0', 'request_uri_parameter_supported': False, 'userinfo_endpoint': 'https://graph.microsoft.com/oidc/userinfo', 'authorization_endpoint': 'https://login.microsoftonline.com/****/oauth2/v2.0/authorize', 'device_authorization_endpoint': 'https://login.microsoftonline.com/****/oauth2/v2.0/devicecode', 'http_logout_supported': True, 'frontchannel_logout_supported': True, 'end_session_endpoint': 'https://login.microsoftonline.com/****/oauth2/v2.0/logout', 'claims_supported': ['sub', 'iss', 'cloud_instance_name', 'cloud_instance_host_name', 'cloud_graph_host_name', 'msgraph_host', 'aud', 'exp', 'iat', 'auth_time', 'acr', 'nonce', 'preferred_username', 'name', 'tid', 'ver', 'at_hash', 'c_hash', 'email'], 'kerberos_endpoint': 'https://login.microsoftonline.com/****/kerberos', 'tenant_region_scope': 'EU', 'cloud_instance_name': 'microsoftonline.com', 'cloud_graph_host_name': 'graph.windows.net', 'msgraph_host': 'graph.microsoft.com', 'rbac_url': 'https://pas.windows.net'}
DEBUG: msal.application: Broker enabled? None
DEBUG: cli.azure.cli.core.commands.client_factory: Getting management service client client_type=_ml_client_cli
DEBUG: urllib3.util.retry: Converted retries value: 1 -> Retry(total=1, connect=None, read=None, redirect=None, status=None)
DEBUG: msal.authority: openid_config = {'token_endpoint': 'https://login.microsoftonline.com/****/oauth2/v2.0/token', 'token_endpoint_auth_methods_supported': ['client_secret_post', 'private_key_jwt', 'client_secret_basic'], 'jwks_uri': 'https://login.microsoftonline.com/****/discovery/v2.0/keys', 'response_modes_supported': ['query', 'fragment', 'form_post'], 'subject_types_supported': ['pairwise'], 'id_token_signing_alg_values_supported': ['RS256'], 'response_types_supported': ['code', 'id_token', 'code id_token', 'id_token token'], 'scopes_supported': ['openid', 'profile', 'email', 'offline_access'], 'issuer': 'https://login.microsoftonline.com/****/v2.0', 'request_uri_parameter_supported': False, 'userinfo_endpoint': 'https://graph.microsoft.com/oidc/userinfo', 'authorization_endpoint': 'https://login.microsoftonline.com/****/oauth2/v2.0/authorize', 'device_authorization_endpoint': 'https://login.microsoftonline.com/****/oauth2/v2.0/devicecode', 'http_logout_supported': True, 'frontchannel_logout_supported': True, 'end_session_endpoint': 'https://login.microsoftonline.com/****/oauth2/v2.0/logout', 'claims_supported': ['sub', 'iss', 'cloud_instance_name', 'cloud_instance_host_name', 'cloud_graph_host_name', 'msgraph_host', 'aud', 'exp', 'iat', 'auth_time', 'acr', 'nonce', 'preferred_username', 'name', 'tid', 'ver', 'at_hash', 'c_hash', 'email'], 'kerberos_endpoint': 'https://login.microsoftonline.com/****/kerberos', 'tenant_region_scope': 'EU', 'cloud_instance_name': 'microsoftonline.com', 'cloud_graph_host_name': 'graph.windows.net', 'msgraph_host': 'graph.microsoft.com', 'rbac_url': 'https://pas.windows.net'}
DEBUG: msal.application: Broker enabled? None
Class AutoDeleteSettingSchema: This is an experimental class, and may change at any time. Please see https://aka.ms/azuremlexperimental for more information.
Class AutoDeleteConditionSchema: This is an experimental class, and may change at any time. Please see https://aka.ms/azuremlexperimental for more information.
Class BaseAutoDeleteSettingSchema: This is an experimental class, and may change at any time. Please see https://aka.ms/azuremlexperimental for more information.
Class IntellectualPropertySchema: This is an experimental class, and may change at any time. Please see https://aka.ms/azuremlexperimental for more information.
Class ProtectionLevelSchema: This is an experimental class, and may change at any time. Please see https://aka.ms/azuremlexperimental for more information.
Class BaseIntellectualPropertySchema: This is an experimental class, and may change at any time. Please see https://aka.ms/azuremlexperimental for more information.
DEBUG: attr_dict: getting schema_ignored
DEBUG: attr_dict: setting schema_ignored to {}
DEBUG: attr_dict: getting schema_ignored
DEBUG: attr_dict: setting schema_ignored to {}
DEBUG: cli.azure.cli.core.auth.credential_adaptor: CredentialAdaptor.get_token: scopes=('https://management.azure.com/.default',), kwargs={}
DEBUG: cli.azure.cli.core.auth.msal_authentication: ServicePrincipalCredential.get_token: scopes=('https://management.azure.com/.default',), kwargs={}
DEBUG: msal.telemetry: Generate or reuse correlation_id: ****
DEBUG: urllib3.connectionpool: Starting new HTTPS connection (1): login.microsoftonline.com:443

...


  "type": "pipeline"
}

##[error]DEBUG: cli.knack.cli: Command arguments: ['ml', 'job', 'create', '--file', './ml_pipelines/pipeline_data.yml', '--debug']
DEBUG: cli.knack.cli: __init__ debug log:
Cannot enable color.
DEBUG: cli.knack.cli: Event: Cli.PreExecute []
DEBUG: cli.knack.cli: Event: CommandParser.OnGlobalArgumentsCreate [<function CLILogging.on_global_arguments at 0x7fcd3abf8040>, <function OutputProducer.on_global_arguments at 0x7fcd3aba6160>, <function CLIQuery.on_global_arguments at 0x7fcd3abdfc40>]
DEBUG: cli.knack.cli: Event: CommandInvoker.OnPreCommandTableCreate []
DEBUG: cli.azure.cli.core: Modules found from index for 'ml': ['azext_mlv2']
DEBUG: cli.azure.cli.core: Loading command modules:
DEBUG: cli.azure.cli.core: Name                  Load Time    Groups  Commands
DEBUG: cli.azure.cli.core: Total (0)                 0.000         0         0
##[error]Script has output to stderr. Failing as failOnStdErr is set to true.
/usr/bin/az account clear
Finishing: AzureCLI

Expected behavior

Expected behavior is not to throw any warnings not to break the pipeline, or if it is an error to understand what error is it. When running on compute instance it does not throw an error but the message about classes are there. So please give me a solution to run az ml commands in Azure Devops Pipeline and not fail for nothing.

Environment Summary

azure-cli                         2.59.0

core                              2.59.0
telemetry                          1.1.0

Extensions:
azure-devops                       1.0.0
ml                                2.25.0

Dependencies:
msal                              1.27.0
azure-mgmt-resource             23.1.0b2

Python location '/opt/az/bin/python3'
Extensions directory '/opt/az/azcliextensions'

Python (Linux) 3.11.8 (main, Mar 27 2024, 04:03:04) [GCC 11.4.0]

Legal docs and information: aka.ms/AzureCliLegal


Your CLI is up-to-date.

Additional context

No response

[yonzhan]

Thank you for opening this issue, we will look into it.

[diondrapeck]

@eddmik

Thanks for bringing this to our attention. Those class messages are not errors, they're log warnings. All of our experimental classes are marked with a decorator to let users know that they are experimental, so that they don't use them in prod builds, or at the very least are cautious about doing so. This handled by a call to Python's logger and Python sends all logs to stderr; however, the --only-show-errors flag is implemented by Azure CLI (not our ml extension) and Azure CLI's own documentation explicitly says here that experimental warnings should be swallowed by that flag, so I'll assign this back to them to look at.

@yonzhan I've removed the ml labels

[bebound]

@diondrapeck I think from knack.log import get_logger's logger works with --only-show-errors.

@eddmik It is not recommended to use failOnStderr to determine the pipeline failure. See Azure/azure-cli#18372

[eddmik]

@bebound thank you for your comment, I have read the comments, however it is only in the case where --debug is used it does not make sense to use the failOnStderr: true, in our case my understanding is, it makes sense to use failOnStderr: true in combination with --only-show-errors (See. Azure/azure-cli/issues/18372#issuecomment) please correct me if I am wrong. Otherwise how would you see that the job is failed?

[diondrapeck]

@bebound Gotcha. I just checked and knack isn't GA yet, so due to Azure SDK for Python guidelines, we can't depend on it in our GA ML SDK (which the ml CLI extension is built on top of).

[bebound]

@bebound thank you for your comment, I have read the comments, however it is only in the case where --debug is used it does not make sense to use the failOnStderr: true, in our case my understanding is, it makes sense to use failOnStderr: true in combination with --only-show-errors (See. Azure/azure-cli/issues/18372#issuecomment) please correct me if I am wrong. Otherwise how would you see that the job is failed?

Use $LASTEXITCODE instead. The --only=show-errors option may not be reliable as any CLI dependency could write to stderr and interrupt your pipeline.