Is there support for Data Apps?

[epanagou-ubitech]

Hi all,

After going through the EDC documentation, various READMEs and issues, I could not determine whether or not Data Apps are currently supported in the EDC.

In discussion eclipse-edc/Connector#2578 it is mentioned that the IDS Dataspace Protocol specifications do not include the concept of an AppStore. However, Data Apps and the App Store are mentioned in the IDS RAM.

Are they currently supported or are there plans to do so in the future?

Thank you in advance!

[juliapampus]

Data apps can be of different kinds, putting their focus on data provisioning, data processing, or similar. A data app itself can be a sharable asset. The EDC support data apps, but not in a way the IDS RAM originally defined it.

An EDC connector itself can be deployed as single service or a set of multiple services, dependent on your use cases and requirements. As the connector is only one application in your IT landscape connecting existing systems to the dataspace, it is not to be seen as an orchestrator of your deployment, rather an orchestrator of data/information flows. Thus, the deployment of a data app would look like every deployment of a service, e.g., pulling from a container registry, using Kubernetes or similar and deploying the service. One possibility would be the IDS App Store as a centralized provider of certified data apps. However, as you mentioned above, following the Dataspace Protocol specifications, the application sharing or execution are not part of a/the dataspace protocol anymore. (Unless the app is considered an asset, e.g., in code-to-data scenarios.)

I could envision the following scenarios:

1. Data apps as decoupled services that interact with an EDC connector as data providing or consuming services: The connector does not need to "be aware" of these data apps. A data app can use interfaces of the EDC, e.g., the management API or eventing system, if needed, and process data, e.g., as soon as it was pushed to the data source via an EDC data plane. Or before it is pushed/pulled, depending on the kind of data app.
2. Data apps as part of the policy enforcement. In a classic example where a data app would anonymize data, the need to do so and to use one specific data app would be a negotiated policy that ends up in an agreement that is referenced for a data transfer.
   a) The policy could define that the data app should be called during the transfer: A data plane that must support this, needs to implement a policy scope that calls a policy function that can handle the enforcement, and, in the context of this, calls the specific data app.
   b) The policy could define that the data app should be called after the transfer: "external" (not the connector) systems on the consumer side will need to enforce the policy by calling the specific data app.
   c) The policy could define that the data app should be called before the transfer (preparing data): "external" (not the connector) systems on the provider side will need to enforce the policy by calling the specific data app.

I am sure there a multiple more. All cases are supported by the EDC, e.g., with the extensible data plane framework and the policy engine.

[epanagou-ubitech]

Thank you, that was extremely helpful!

From my understanding, using any of these 2 scenarios, we have a lot of freedom regarding the implementation of a data app. I am mainly referring to the languages/frameworks used, since you have already covered the deployment part.

For instance, regarding a data processing app:

Is it correct to assume that as long as it provides an API endpoint through which it can acquire data as its input, process it internally, and produce the expected output, it can be considered a data app, no matter how data processing is achieved?