How is the Azure storage Data Plane module supposed to get used?

[Julian-Marco]

I am currently facing the situation where I want to copy data from an S3 storage to an Azure Blob storage, so I would have to use both the S3 data plane as well as the Azure storage data plane. I am facing a couple of issues here.

1. Does the provider need both of these data planes in order for this to work? I would've assumed that the provider only needs the S3 data plane (respectively the S3 data source) and that the consumer only needs the Azure storage data plane (respectively the Azure storage data sink), but apparently the provider needs both of these data planes, at least when working with a provider push. I deduced this from the fact that the provider connector is telling me that it's missing a data plane that can handle the transfer when only using the S3 data plane. So would it be like this?

Provider -> S3 data plane + Azure storage data plane
Consumer -> Azure storage data plane

2. Another difficulty is that the Azure storage data sink wants the sharedAccessSignature delivered out of an Azure Key Vault which can be seen in this line: https://github.com/eclipse-edc/Technology-Azure/blob/main/extensions/data-plane/data-plane-azure-storage/src/main/java/org/eclipse/edc/connector/dataplane/azure/storage/pipeline/AzureStorageDataSinkFactory.java#L97

This raises a couple of questions:

2.1. In general, is it even possible to realize this use case without using the Azure vault extension? https://github.com/eclipse-edc/Technology-Azure/tree/main/extensions/common/vault/vault-azure If so, shouldn't the vault extension be a direct dependency of the Azure storage data plane? https://github.com/eclipse-edc/Technology-Azure/blob/main/extensions/data-plane/data-plane-azure-storage/build.gradle.kts Or is there a proper way to use the data plane without using the vault extension?

2.2. Assuming that the vault extension is necessary to use in this case and assuming that the provider needs to use the Azure data sink, wouldn't this also mean that the provider needs to do the whole setup for himself? Or is it only the consumer that needs to do the vault setup and it automatically grants access to the vault via its connector?

[justinanthonyvtt]

Perhaps through simpler REST API if the subscription and authentication is done
https://learn.microsoft.com/en-us/rest/api/datashare/2020-09-01/data-sets/get?tabs=HTTP