How to disable check version update on environments without internet connection

[norbertorodas]

Which version of the AzCopy was used?

Note: The version is visible when running AzCopy without any argument

User-Agent: AzCopy/10.24.0 azsdk-go-azblob/v1.3.1 (go1.19.12; linux)

Which platform are you using? (ex: Windows, Mac, Linux)

Linux hosted in Azure

What command did you run?

Note: Please remove the SAS to avoid exposing your credentials. If you cannot remember the exact command, please retrieve it from the beginning of the log file.

azcopy sync "https://${AZURE_STORAGE_ACCOUNT_NAME}${container_source_name}?${SAS}" "$container_target_dir" \ --log-level=NONE \ --skip-version-check \ --recursive=true \ --delete-destination=true \ --output-level=quiet

*we are syncing million of files, distributed in more than 4000 containers, in total around 4.5TB

What problem was encountered?

azcopy tries to check the latest version from internet but there is no internet on the server causing the sync process failure

2024/09/17 15:13:58 ==> REQUEST/RESPONSE (Try=1/6.244145ms, OpTime=6.24609ms) -- REQUEST ERROR
                                                           GET https://azcopyvnextrelease.blob.core.windows.net/releasemetadata/latest_version.txt
                                                           Accept: application/xml
                                                           User-Agent: AzCopy/10.24.0 azsdk-go-azblob/v1.3.1 (go1.19.12; linux)
                                                           X-Ms-Client-Request-Id: 23574e9f-b2a0-4089-6ea6-fb59838c0f08
                                                           x-ms-version: 2023-11-03
                                                           --------------------------------------------------------------------------------
                                                           ERROR:
                                                        Get "https://azcopyvnextrelease.blob.core.windows.net/releasemetadata/latest_version.txt": EOF
                                                        goroutine 53 [running]:
                                                        github.com/Azure/azure-storage-azcopy/v10/ste.stack()
                                                                /home/vsts/work/1/s/ste/xferLogPolicy.go:100 +0x65
                                                        github.com/Azure/azure-storage-azcopy/v10/ste.logPolicy.Do({{{0xb2d05e00, 0x0}, 0x1152fa0, 0x1152f98}, 0xc000683920, 0xc000683950, 0xc000683980}, 0xc000698ac0)
                                                                /home/vsts/work/1/s/ste/xferLogPolicy.go:232 +0xb5d
                                                        github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/exported.(*Request).Next(0xc000698a80)
                                                                /home/vsts/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/[email protected]/internal/exported/request.go:107 +0xf8
                                                        github.com/Azure/azure-storage-azcopy/v10/ste.(*retryNotificationPolicy).Do(0x0?, 0xc000698a80)
                                                                /home/vsts/work/1/s/ste/xferRetryNotificationPolicy.go:64 +0x36
                                                        github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/exported.(*Request).Next(0xc00057a790)
                                                                /home/vsts/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/[email protected]/internal/exported/request.go:107 +0xf8
                                                        github.com/Azure/azure-sdk-for-go/sdk/azcore/runtime.(*retryPolicy).Do(0x40f31f?, 0xc0006989c0)
                                                                /home/vsts/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/[email protected]/runtime/policy_retry.go:126 +0x5b4
                                                        github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/exported.(*Request).Next(0xc000698980)
                                                                /home/vsts/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/[email protected]/internal/exported/request.go:107 +0xf8
                                                        github.com/Azure/azure-storage-azcopy/v10/ste.(*fileUploadRangeFromURLFixPolicy).Do(0x16?, 0xc000698980)
                                                                /home/vsts/work/1/s/ste/sender-azureFileFromURL.go:118 +0x15f
                                                        github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/exported.(*Request).Next(0xc000698940)
                                                                /home/vsts/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/[email protected]/internal/exported/request.go:107 +0xf8
                                                        github.com/Azure/azure-storage-azcopy/v10/ste.(*versionPolicy).Do(0xe9f940?, 0xc000698940)
                                                                /home/vsts/work/1/s/ste/xferVersionPolicy.go:49 +0x134
                                                        github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/exported.(*Request).Next(0xc000698900)
                                                                /home/vsts/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/[email protected]/internal/exported/request.go:107 +0xf8
                                                        github.com/Azure/azure-sdk-for-go/sdk/azcore/runtime.(*requestIDPolicy).Do(0xc000570a98?, 0xc000698900)
                                                                /home/vsts/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/[email protected]/runtime/policy_request_id.go:33 +0x146
                                                        github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/exported.(*Request).Next(0xc0006988c0)
                                                                /home/vsts/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/[email protected]/internal/exported/request.go:107 +0xf8
                                                        github.com/Azure/azure-sdk-for-go/sdk/azcore/runtime.telemetryPolicy.Do({{0xc00069e600?, 0xfe6000?}}, 0xc0006988c0)
                                                                /home/vsts/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/[email protected]/runtime/policy_telemetry.go:70 +0x1c5
                                                        github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/exported.(*Request).Next(0xc000698880)
                                                                /home/vsts/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/[email protected]/internal/exported/request.go:107 +0xf8
                                                        github.com/Azure/azure-sdk-for-go/sdk/azcore/runtime.includeResponsePolicy(0xc000698880)
                                                                /home/vsts/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/[email protected]/runtime/policy_include_response.go:19 +0x25
                                                        github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/exported.PolicyFunc.Do(0xf4ea20?, 0x19163c8?)
                                                                /home/vsts/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/[email protected]/internal/exported/request.go:177 +0x1f
                                                        github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/exported.(*Request).Next(0xc000698840)
                                                                /home/vsts/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/[email protected]/internal/exported/request.go:107 +0xf8
                                                        github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/exported.Pipeline.Do({{0xc000694c80?, 0x12a8748?, 0xc00003c090?}}, 0x0?)
                                                                /home/vsts/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/[email protected]/internal/exported/pipeline.go:76 +0x56
                                                        github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/internal/generated.(*BlobClient).Download(_, {_, _}, _, _, _, _)
                                                                /home/vsts/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/storage/[email protected]/internal/generated/zz_blob_client.go:901 +0xaa
                                                        github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/blob.(*Client).DownloadStream(_, {_, _}, _)
                                                                /home/vsts/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/storage/[email protected]/blob/client.go:408 +0x148
                                                        github.com/Azure/azure-storage-azcopy/v10/cmd.beginDetectNewVersion.func1()
                                                                /home/vsts/work/1/s/cmd/root.go:308 +0x1ed
                                                        created by github.com/Azure/azure-storage-azcopy/v10/cmd.beginDetectNewVersion
                                                                /home/vsts/work/1/s/cmd/root.go:279 +0x6a

How can we reproduce the problem in the simplest way?

Run multiple azcopy sync commands (for example multiple folders or containers) from a host without internet connection.

Have you found a mitigation/solution?

No

We have tried the following without success:

1. flag --skip-version-check or --skip-version-check=true

2. env var export AZCOPY_DISABLE_AUTO_UPDATE=true

3. env var export AZCOPY_AUTO_UPDATE=false

4. Since release 10.22.0 a cache feature was introduced

   Added support to cache latest AzCopy version and check the remote version every 24 hours instead of every run.

   We have created the file latest_version.txt in ~/.azcopy including the azcopy version (10.24.0) and a datetime less than 1 day (e.g. 10.24.0,2024-09-18T15:01:08Z ) but the azcopy sync command is still trying to reach internet to check the latest version.

Best Regards,
Norberto

[dphulkar-msft]

hi @norbertorodas,

We tried to repro the scenario on our environment, but we do not see any REST call happening when the version check is disabled with --skip-version-check flag or if any of the env variable is set AZCOPY_DISABLE_AUTO_UPDATE=true or AZCOPY_AUTO_UPDATE=false