upload a vmdk to vsan datastore leaves disk unusable

[Klaas-]

Describe the bug

Hi,
uploading a vmdk with pyvmomi to a vsan datastore seem to leave the resulting file in an unusable state.

This pops up for me because I am using vsphere_copy in ansible (which is using the sdk in background).

For simplicity I reproduced the issue with the sample datastore upload python:
https://github.com/vmware/pyvmomi-community-samples/blob/master/samples/upload_file_to_datastore.py

Version 8.0.2.0.1
Python 3.11 on a RHEL8.9

vmdk is a stream optimized v6 image, using the same file and uploading it via the webui through firefox results in a usable vmdk.

Uploading the vmdk via api results in an error when trying to use in vsphere:
"Unsupported or invalid disk type 22 for 'scsi0:0'. Ensure that the disk has been imported."

vCenter: 7.0.3 Build: 22357613 (VxRail 7.0.482)

Reproduction steps

1. have a stream optimized vmdk + a vsphere + vsan
2. python3.11 upload_file_to_datastore.py -s vsphere_server -o 443 -u [email protected] -p "password" --disable-ssl-verification --datastore-name VxRail-Virtual-SAN-Datastore-id --local-file-path ~/diskimages/file.vmdk --remote-file-path import/file.vmdk
3. create a VM that uses that VMDK as disk
   ...

Expected behavior

Works, no error

Additional context

Actual behavior:
VM does not start; error "Unsupported or invalid disk type 22 for 'scsi0:0'. Ensure that the disk has been imported."

[Klaas-]

I figured it out, so not sure if this belongs here or somewhere else :)

While comparing what the sample/sdk does to what the WebUI does I noticed that the webui sets an additional argument that is not documented as far as I could find

So if you set the option like this:

        params = {"dsName": datastore.info.name,
                  "dcPath": datacenter.name,
                  "diskFormat": "StreamVmdk"
                 }

It works fine. Not sure if this is considered to be a problem in the sdk/example but it for sure is missing from the api documentation https://vdc-download.vmware.com/vmwb-repository/dcr-public/c369f363-3f34-4acb-8620-e861a6c24013/1c6e721f-2d3b-4740-bf94-3409efb5f08b/WebServicesSDKprogramming802.pdf

[mariolenz]

I didn't find any documentation about this diskFormat parameter. I didn't really find any documentation about uploading files at all.

It would be really great to see this officially documented somewhere ;-)

[Klaas-]

And a small follow up, there seems to be a second issue with python 3.10+ which is creating errors during large put requests ( happens often like 9/10 times - python/cpython#110467 ), this also seems to be happening on 3.9 but not as often :) so now I just use python 3.9 and I do automatic retries if upload fails :)

[Klaas-]

and to document my workaround for the python file upload issue: I now just execute curl, it seems to have a 100% success rate :)

curl -u "user:pass" "https://vsphere.tld/folder/DIRNAME/FILENAME.vmdk?dsName=VxRail-Virtual-SAN-Datastore-UUID&dcPath=DATACENTERNAME&diskFormat=StreamVmdk" -H 'Content-type: application/octet-stream' -T /path/to/local.vmdk