Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Error when running az ml data import #6997

Closed
matsuobasho opened this issue Nov 15, 2023 · 2 comments
Closed

Error when running az ml data import #6997

matsuobasho opened this issue Nov 15, 2023 · 2 comments
Labels
Auto-Assign Auto assign by bot bug This issue requires a change to an existing behavior in the product in order to be resolved. customer-reported Issues that are reported by GitHub users external to the Azure organization. extension/ml Machine Learning question The issue doesn't require a change to the product in order to be resolved. Most issues start as that Service Attention This issue is responsible by Azure service team.

Comments

@matsuobasho
Copy link

matsuobasho commented Nov 15, 2023 •
edited
Loading

Describe the bug

Get an error when trying to upload a dataset to a blobstorage.

Related command

  1. Create local_file.yaml file:
name: test-data
description: Data asset created from local file.
type: uri_file
path: ../test_data/file_name.pkl
  1. Run
az ml data import -f absolute_path/cloud/local_file.yaml --workspace-name <ws_name> --resource-group <rg> --datastore <ds_name>

Note that the file for upload is in a directory that's parallel to the cloud subdirectory, hence I specify a relative path.

Errors

Class WorkspaceHubOperations: This is an experimental class, and may change at any time. Please see https://aka.ms/azuremlexperimental for more information.
Method import_data: This is an experimental method, and may change at any time. Please see https://aka.ms/azuremlexperimental for more information.
←[91mMet error <class 'AttributeError'>:'Data' object has no attribute 'source'
Please check log by running the command with '--debug' for more details.←[0m

Issue script & Debug output

Enable color in terminal.←[0m
←[36mcli.knack.cli: Event: Cli.PreExecute []←[0m
←[36mcli.knack.cli: Event: CommandParser.OnGlobalArgumentsCreate [<function CLILogging.on_global_arguments at 0x00000221049153F0>, <function OutputProducer.on_global_arguments at 0x0000022104A900D0>, <function CLIQuery.on_global_arguments at 
0x0000022104CAD3F0>]←[0m
←[36mcli.knack.cli: Event: CommandInvoker.OnPreCommandTableCreate []←[0m
←[36mcli.azure.cli.core: Modules found from index for 'ml': ['azext_mlv2']←[0m
←[36mcli.azure.cli.core: Loading command modules:←[0m
←[36mcli.azure.cli.core: Name                  Load Time    Groups  Commands←[0m
←[36mcli.azure.cli.core: Total (0)                 0.000         0         0←[0m
←[36mcli.azure.cli.core: These extensions are not installed and will be skipped: ['azext_ai_examples', 'azext_next']←[0m
←[36mcli.azure.cli.core: Loading extensions:←[0m
←[36mcli.azure.cli.core: Name                  Load Time    Groups  Commands  Directory←[0m
←[36mazext_mlv2.manual.vendored_curated_sdk.azure.ai.ml._azure_environments: Using the default cloud configuration: 'AzureCloud'.←[0m
←[36mazext_mlv2.manual.vendored_curated_sdk.azure.ai.ml._azure_environments: Using the default cloud configuration: 'AzureCloud'.←[0m
←[36mazext_mlv2.manual.vendored_curated_sdk.azure.ai.ml._azure_environments: Using the default cloud configuration: 'AzureCloud'.←[0m
←[36mcli.azure.cli.core: ml                        1.659        21       145  C:\Users\bob\.azure\cliextensions\ml←[0m
←[36mcli.azure.cli.core: Total (1)                 1.659        21       145  ←[0m
←[36mcli.azure.cli.core: Loaded 21 groups, 145 commands.←[0m
←[36mcli.azure.cli.core: Found a match in the command table.←[0m
←[36mcli.azure.cli.core: Raw command  : ml data import←[0m
←[36mcli.azure.cli.core: Command table: ml data import←[0m
←[36mcli.knack.cli: Event: CommandInvoker.OnPreCommandTableTruncate [<function AzCliLogging.init_command_file_logging at 0x00000221073F0550>]←[0m
←[36mcli.azure.cli.core.azlogging: metadata file logging enabled - writing logs to 'C:\Users\bob\.azure\commands\2023-11-14.19-11-27.ml_data_import.15740.log'.←[0m
←[32maz_command_data_logger: command args: ml data import -f {} --workspace-name {} --resource-group {} --datastore {} --debug←[0m
←[36mcli.knack.cli: Event: CommandInvoker.OnPreArgumentLoad [<function register_global_subscription_argument.<locals>.add_subscription_parameter at 0x00000221073F3130>]←[0m
←[32mcli.knack.commands: Configured default 'bob-rg' for arg resource_group_name←[0m
←[32mcli.knack.commands: Configured default 'aus-east' for arg workspace_name←[0m
←[36mcli.knack.cli: Event: CommandInvoker.OnPostArgumentLoad []←[0m
←[36mcli.knack.cli: Event: CommandInvoker.OnPostCommandTableCreate [<function register_ids_argument.<locals>.add_ids_arguments at 0x000002210743D1B0>, <function register_cache_arguments.<locals>.add_cache_arguments at 0x000002210743D2D0>]←[0m
←[36mcli.knack.cli: Event: CommandInvoker.OnCommandTableLoaded []←[0m
←[36mcli.knack.cli: Event: CommandInvoker.OnPreParseArgs []←[0m
←[36mcli.knack.cli: Event: CommandInvoker.OnPostParseArgs [<function OutputProducer.handle_output_argument at 0x0000022104A90160>, <function CLIQuery.handle_query_parameter at 0x0000022104CAD480>, <function register_ids_argument.<locals>.parse_ids_arguments at 0x000002210743D240>]←[0m
←[32maz_command_data_logger: extension name: ml←[0m
←[32maz_command_data_logger: extension version: 2.20.0←[0m
This command is in preview and under development. Reference and support levels: https://aka.ms/CLI_refstatus
←[36mcli.azure.cli.core.commands.client_factory: Getting management service client client_type=AzureMachineLearningWorkspaces←[0m
←[36mcli.azure.cli.core.auth.persistence: build_persistence: location='C:\\Users\\bob\\.azure\\msal_token_cache.bin', encrypt=True←[0m
←[36mcli.azure.cli.core.auth.binary_cache: load: C:\Users\bob\.azure\msal_http_cache.bin←[0m
←[36murllib3.util.retry: Converted retries value: 1 -> Retry(total=1, connect=None, read=None, redirect=None, status=None)←[0m
←[36mmsal.authority: openid_config = {'token_endpoint': 'https://login.microsoftonline.com/9b0533c8-9417-4751-928b-c8331e471849/oauth2/v2.0/token', 'token_endpoint_auth_methods_supported': ['client_secret_post', 'private_key_jwt', 'client_secret_basic'], 'jwks_uri': 'https://login.microsoftonline.com/9b0533c8-9417-4751-928b-c8331e471849/discovery/v2.0/keys', 'response_modes_supported': ['query', 'fragment', 'form_post'], 'subject_types_supported': ['pairwise'], 'id_token_signing_alg_values_supported': ['RS256'], 'response_types_supported': ['code', 'id_token', 'code id_token', 'id_token token'], 'scopes_supported': ['openid', 'profile', 'email', 'offline_access'], 'issuer': 'https://login.microsoftonline.com/9b0533c8-9417-4751-928b-c8331e471849/v2.0', 'request_uri_parameter_supported': False, 'userinfo_endpoint': 'https://graph.microsoft.com/oidc/userinfo', 'authorization_endpoint': 'https://login.microsoftonline.com/9b0533c8-9417-4751-928b-c8331e471849/oauth2/v2.0/authorize', 'device_authorization_endpoint': 'https://login.microsoftonline.com/9b0533c8-9417-4751-928b-c8331e471849/oauth2/v2.0/devicecode', 'http_logout_supported': True, 'frontchannel_logout_supported': True, 'end_session_endpoint': 'https://login.microsoftonline.com/9b0533c8-9417-4751-928b-c8331e471849/oauth2/v2.0/logout', 'claims_supported': ['sub', 'iss', 'cloud_instance_name', 'cloud_instance_host_name', 'cloud_graph_host_name', 'msgraph_host', 'aud', 'exp', 'iat', 'auth_time', 'acr', 'nonce', 'preferred_username', 'name', 'tid', 'ver', 'at_hash', 'c_hash', 'email'], 'kerberos_endpoint': 'https://login.microsoftonline.com/9b0533c8-9417-4751-928b-c8331e471849/kerberos', 'tenant_region_scope': 'NA', 
'cloud_instance_name': 'microsoftonline.com', 'cloud_graph_host_name': 'graph.windows.net', 'msgraph_host': 'graph.microsoft.com', 'rbac_url': 'https://pas.windows.net'}←[0m
←[36mmsal.application: Broker enabled? False←[0m
←[36mcli.azure.cli.core.commands.client_factory: Getting management service client client_type=_ml_client_cli←[0m
←[36murllib3.util.retry: Converted retries value: 1 -> Retry(total=1, connect=None, read=None, redirect=None, status=None)←[0m
←[36mmsal.authority: openid_config = {'token_endpoint': 'https://login.microsoftonline.com/9b0533c8-9417-4751-928b-c8331e471849/oauth2/v2.0/token', 'token_endpoint_auth_methods_supported': ['client_secret_post', 'private_key_jwt', 'client_secret_basic'], 'jwks_uri': 'https://login.microsoftonline.com/9b0533c8-9417-4751-928b-c8331e471849/discovery/v2.0/keys', 'response_modes_supported': ['query', 'fragment', 'form_post'], 'subject_types_supported': ['pairwise'], 'id_token_signing_alg_values_supported': ['RS256'], 'response_types_supported': ['code', 'id_token', 'code id_token', 'id_token token'], 'scopes_supported': ['openid', 'profile', 'email', 'offline_access'], 'issuer': 'https://login.microsoftonline.com/9b0533c8-9417-4751-928b-c8331e471849/v2.0', 'request_uri_parameter_supported': False, 'userinfo_endpoint': 'https://graph.microsoft.com/oidc/userinfo', 'authorization_endpoint': 'https://login.microsoftonline.com/9b0533c8-9417-4751-928b-c8331e471849/oauth2/v2.0/authorize', 'device_authorization_endpoint': 'https://login.microsoftonline.com/9b0533c8-9417-4751-928b-c8331e471849/oauth2/v2.0/devicecode', 'http_logout_supported': True, 'frontchannel_logout_supported': True, 'end_session_endpoint': 'https://login.microsoftonline.com/9b0533c8-9417-4751-928b-c8331e471849/oauth2/v2.0/logout', 'claims_supported': ['sub', 'iss', 'cloud_instance_name', 'cloud_instance_host_name', 'cloud_graph_host_name', 'msgraph_host', 'aud', 'exp', 'iat', 'auth_time', 'acr', 'nonce', 'preferred_username', 'name', 'tid', 'ver', 'at_hash', 'c_hash', 'email'], 'kerberos_endpoint': 'https://login.microsoftonline.com/9b0533c8-9417-4751-928b-c8331e471849/kerberos', 'tenant_region_scope': 'NA', 
'cloud_instance_name': 'microsoftonline.com', 'cloud_graph_host_name': 'graph.windows.net', 'msgraph_host': 'graph.microsoft.com', 'rbac_url': 'https://pas.windows.net'}←[0m
←[36mmsal.application: Broker enabled? False←[0m
Class WorkspaceHubOperations: This is an experimental class, and may change at any time. Please see https://aka.ms/azuremlexperimental for more information.
Method import_data: This is an experimental method, and may change at any time. Please see https://aka.ms/azuremlexperimental for more information.
Traceback (most recent call last):
  File "C:\Users\bob\.azure\cliextensions\ml\azext_mlv2\manual\custom\data.py", line 152, in ml_data_import
    job = ml_client.data.import_data(data_import, **kwargs)
  File "C:\Users\bob\.azure\cliextensions\ml\azext_mlv2\manual\vendored_curated_sdk\azure\ai\ml\_telemetry\activity.py", line 275, in wrapper
    return f(*args, **kwargs)
  File "C:\Users\bob\.azure\cliextensions\ml\azext_mlv2\manual\vendored_curated_sdk\azure\ai\ml\_utils\_experimental.py", line 100, in wrapped
    return func(*args, **kwargs)
  File "C:\Users\bob\.azure\cliextensions\ml\azext_mlv2\manual\vendored_curated_sdk\azure\ai\ml\operations\_data_operations.py", line 388, in import_data
    data_import.type = AssetTypes.MLTABLE if isinstance(data_import.source, Database) else AssetTypes.URI_FOLDER
AttributeError: 'Data' object has no attribute 'source'
←[91mcli: None←[0m
←[36mcli: Traceback (most recent call last):
  File "C:\Users\bob\.azure\cliextensions\ml\azext_mlv2\manual\custom\data.py", line 152, in ml_data_import
    job = ml_client.data.import_data(data_import, **kwargs)
  File "C:\Users\bob\.azure\cliextensions\ml\azext_mlv2\manual\vendored_curated_sdk\azure\ai\ml\_telemetry\activity.py", line 275, in wrapper
    return f(*args, **kwargs)
  File "C:\Users\bob\.azure\cliextensions\ml\azext_mlv2\manual\vendored_curated_sdk\azure\ai\ml\_utils\_experimental.py", line 100, in wrapped
    return func(*args, **kwargs)
  File "C:\Users\bob\.azure\cliextensions\ml\azext_mlv2\manual\vendored_curated_sdk\azure\ai\ml\operations\_data_operations.py", line 388, in import_data
    data_import.type = AssetTypes.MLTABLE if isinstance(data_import.source, Database) else AssetTypes.URI_FOLDER
AttributeError: 'Data' object has no attribute 'source'
←[0m
←[36mcli.azure.cli.core.azclierror: Traceback (most recent call last):
  File "C:\Users\bob\.azure\cliextensions\ml\azext_mlv2\manual\custom\data.py", line 152, in ml_data_import
    job = ml_client.data.import_data(data_import, **kwargs)
  File "C:\Users\bob\.azure\cliextensions\ml\azext_mlv2\manual\vendored_curated_sdk\azure\ai\ml\_telemetry\activity.py", line 275, in wrapper
    return f(*args, **kwargs)
  File "C:\Users\bob\.azure\cliextensions\ml\azext_mlv2\manual\vendored_curated_sdk\azure\ai\ml\_utils\_experimental.py", line 100, in wrapped
    return func(*args, **kwargs)
  File "C:\Users\bob\.azure\cliextensions\ml\azext_mlv2\manual\vendored_curated_sdk\azure\ai\ml\operations\_data_operations.py", line 388, in import_data
    data_import.type = AssetTypes.MLTABLE if isinstance(data_import.source, Database) else AssetTypes.URI_FOLDER
AttributeError: 'Data' object has no attribute 'source'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "D:\a\_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\knack/cli.py", line 233, in invoke
  File "D:\a\_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/__init__.py", line 663, in execute
  File "D:\a\_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/__init__.py", line 726, in _run_jobs_serially
  File "D:\a\_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/__init__.py", line 697, in _run_job
  File "D:\a\_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/__init__.py", line 333, in __call__
  File "D:\a\_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/command_operation.py", line 121, in handler
  File "C:\Users\bob\.azure\cliextensions\ml\azext_mlv2\manual\custom\data.py", line 156, in ml_data_import
    log_and_raise_error(err, debug, yaml_operation=yaml_operation)
  File "C:\Users\bob\.azure\cliextensions\ml\azext_mlv2\manual\custom\raise_error.py", line 181, in log_and_raise_error
    raise cli_error
knack.util.CLIError: Met error <class 'AttributeError'>:'Data' object has no attribute 'source'
Please check log by running the command with '--debug' for more details.
←[0m
←[91mcli.azure.cli.core.azclierror: Met error <class 'AttributeError'>:'Data' object has no attribute 'source'
Please check log by running the command with '--debug' for more details.←[0m
←[91maz_command_data_logger: Met error <class 'AttributeError'>:'Data' object has no attribute 'source'
Please check log by running the command with '--debug' for more details.←[0m
←[36mcli.knack.cli: Event: Cli.PostExecute [<function AzCliLogging.deinit_cmd_metadata_logging at 0x00000221073F0790>]←[0m
←[32maz_command_data_logger: exit code: 1←[0m
←[32mcli.__main__: Command ran in 2.344 seconds (init: 0.245, invoke: 2.099)←[0m
←[32mtelemetry.main: Begin splitting cli events and extra events, total events: 1←[0m
←[32mtelemetry.client: Accumulated 0 events. Flush the clients.←[0m
←[32mtelemetry.main: Finish splitting cli events and extra events, cli events: 1←[0m
←[32mtelemetry.save: Save telemetry record of length 3868 in cache←[0m
←[32mtelemetry.main: Begin creating telemetry upload process.←[0m
←[32mtelemetry.process: Creating upload process: "C:\Program Files\Microsoft SDKs\Azure\CLI2\python.exe C:\Program Files\Microsoft SDKs\Azure\CLI2\Lib\site-packages\azure\cli\telemetry\__init__.pyc C:\Users\bob\.azure"←[0m
←[32mtelemetry.process: Return from creating process←[0m
←[32mtelemetry.main: Finish creating telemetry upl

Expected behavior

Dataset is added to the blobstore.

Environment Summary

azure-cli: 2.53.0
azure-cli-core: 2.53.0
azure-cli-telemetry: 1.1.0
extensions:
  ml: 2.20.0

Additional context

Issue perhaps related to the blob storage that I created. When I try to add the data manually to it in the GUI, I see this message:
image

I created this using the following CLI command:
az ml datastore create -f path_to_file/blob.yaml --workspace-name <ws_name> --resource-group <rg>
@matsuobasho matsuobasho added the bug This issue requires a change to an existing behavior in the product in order to be resolved. label Nov 15, 2023
@yonzhan
Copy link
Collaborator

yonzhan commented Nov 15, 2023

Thank you for opening this issue, we will look into it.

@microsoft-github-policy-service microsoft-github-policy-service bot added question The issue doesn't require a change to the product in order to be resolved. Most issues start as that customer-reported Issues that are reported by GitHub users external to the Azure organization. Auto-Assign Auto assign by bot Service Attention This issue is responsible by Azure service team. Machine Learning extension/ml labels Nov 15, 2023
@matsuobasho
Copy link
Author

I realized my mistake, I was running az ml data import instead of az ml data create. The former is intended for import from database or filesystem, while I'm importing from my local machine.
https://learn.microsoft.com/en-us/cli/azure/ml/data?view=azure-cli-latest#az-ml-data-create

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Auto-Assign Auto assign by bot bug This issue requires a change to an existing behavior in the product in order to be resolved. customer-reported Issues that are reported by GitHub users external to the Azure organization. extension/ml Machine Learning question The issue doesn't require a change to the product in order to be resolved. Most issues start as that Service Attention This issue is responsible by Azure service team.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@matsuobasho @yonzhan