Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix CI and DAPS interaction issues #642

Merged
merged 13 commits into from
Nov 30, 2023
Merged
1 change: 1 addition & 0 deletions .github/ISSUE_TEMPLATE/release.md
Original file line number Diff line number Diff line change
Expand Up @@ -53,6 +53,7 @@ Feel free to edit this release checklist in-progress depending on what tasks nee
- [ ] Merge the `release-prep` PR.
- [ ] Wait for the main branch to be green.
- [ ] Test the release `docker-compose.yaml` with `RELEASE_EDC_IMAGE=ghcr.io/sovity/edc-dev:latest`.
- [ ] Ensure with a `docker ps -a` that all containers are healthy, and not `healthy: starting` or `healthy: unhealthy`.
- [ ] Test the postman collection against that running docker-compose.
- [ ] Create a release and re-use the changelog section as release description, and the version as title.
- [ ] Check if the pipeline built the release versions in the Actions-Section (or you won't see it).
Expand Down
6 changes: 3 additions & 3 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -76,11 +76,11 @@ jobs:
GRADLE_ARGS="-PsovityEdcExtensionsVersion=${GITHUB_REF#refs/tags/v}"
echo "GRADLE_ARGS=$GRADLE_ARGS" >> $GITHUB_ENV
- name: "Gradle: Build"
uses: gradle/gradle-build-action@a4cf152f482c7ca97ef56ead29bf08bcd953284c
uses: gradle/gradle-build-action@v2.10.0
with:
arguments: build ${{ env.GRADLE_ARGS }}
- name: "Gradle: Publish (Main & Release Only)"
uses: gradle/gradle-build-action@a4cf152f482c7ca97ef56ead29bf08bcd953284c
uses: gradle/gradle-build-action@v2.10.0
if: ${{ github.ref == 'refs/heads/main' || startsWith(github.ref, 'refs/tags/v') }}
with:
arguments: publish ${{ env.GRADLE_ARGS }}
Expand Down Expand Up @@ -152,7 +152,7 @@ jobs:
- name: "Gradle: Validate Gradle Wrapper"
uses: gradle/wrapper-validation-action@ccb4328a959376b642e027874838f60f8e596de3
- name: "Gradle: Generate TS Code"
uses: gradle/gradle-build-action@a4cf152f482c7ca97ef56ead29bf08bcd953284c
uses: gradle/gradle-build-action@v2.10.0
with:
arguments: :extensions:wrapper:wrapper:build -x test
- name: "NPM: Dist Tag & Version"
Expand Down
6 changes: 6 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,12 +17,18 @@ please see [changelog_updates.md](docs/dev/changelog_updates.md).

#### Patch Changes

- Marked `MY_EDC_NAME_KEBAB_CASE` as deprecated in favor of `MY_EDC_PARTICIPANT_ID`.
- Fixed an issue preventing Contract Negotiations between MDS Connectors.
- Improved `:extensions:wrapper:wrapper-common-mappers` for broker: `AssetJsonLdUtils`, made some methods public.
- Added example for using the API Wrapper to offer and consume data.
- Improved development documentation documenting our CHANGELOG.

### Deployment Migration Notes

- Renamed ~~`MY_EDC_NAME_KEBAB_CASE`~~ to `MY_EDC_PARTICIPANT_ID`. ~~`MY_EDC_NAME_KEBAB_CASE`~~ continues working, but prints a warning on startup.
- The value of `MY_EDC_PARTICIPANT_ID` must coincide with the claim value `referringConnector`
as configured for this Connector in the DAPS.

#### Compatible Versions

## [6.0.0] - 2023-11-17
Expand Down
2 changes: 1 addition & 1 deletion build.gradle.kts
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,7 @@ allprojects {
tasks.getByName<Test>("test") {
useJUnitPlatform()
testLogging {
events = setOf(TestLogEvent.FAILED)
events = setOf(TestLogEvent.SKIPPED, TestLogEvent.FAILED)
exceptionFormat = TestExceptionFormat.FULL
showExceptions = true
showCauses = true
Expand Down
4 changes: 2 additions & 2 deletions docker-compose-dev.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ services:
depends_on:
- postgresql
environment:
MY_EDC_NAME_KEBAB_CASE: "my-edc"
MY_EDC_PARTICIPANT_ID: "my-edc"
MY_EDC_TITLE: "EDC Connector"
MY_EDC_DESCRIPTION: "sovity Community Edition EDC Connector"
MY_EDC_CURATOR_URL: "https://example.com"
Expand Down Expand Up @@ -60,7 +60,7 @@ services:
depends_on:
- postgresql2
environment:
MY_EDC_NAME_KEBAB_CASE: "my-edc2"
MY_EDC_PARTICIPANT_ID: "my-edc2"
MY_EDC_TITLE: "EDC Connector 2"
MY_EDC_DESCRIPTION: "sovity Community Edition EDC Connector"
MY_EDC_CURATOR_URL: "https://example.com"
Expand Down
55 changes: 27 additions & 28 deletions docs/deployment-guide/goals/production/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,25 +7,34 @@ Productive Deployment Guide

This is a productive deployment guide for self-hosting a functional sovity CE EDC Connector or MDS CE EDC Connector.

## Requirements
## Prerequisites

A productive EDC Connector deployment has strict requirements, with slight errors in configuration already causing
contract negotiations / data transfer to fail.
### Technical Skills

In general a productive EDC Connector requires a DAPS Server, DAPS Credentials, a reverse proxy configured in detail due
to technical reasons, reachability via the internet and well-defined URLs across all configurations.
- Ability to deploy, run and expose containered applications to the internet.
- Ability to configure ingress routes or a reverse proxy of your choice to merge multiple services under a single
domain.
- Know-how on how to secure an otherwise unprotected application with an auth proxy or other solutions fitting
your situation.

### Dataspace

- Must have a running DAPS that follows the subset of OAuth2 as described in the DSP Specification.
- You must have a valid Connector Certificate in the form of [a generated SKI/AKI pair and .jks file](#faq).
- You must have a valid Participant ID / Connector ID, which is configured in the claim "referringConnector" in the
DAPS.

## Deployment Units

To deploy an EDC multiple deployment units must be deployed and configured.

| Deployment Unit | Version / Details |
|----------------------------------------------------------------|---------------------------------------------------------------------------------------------|
| An Auth Proxy / Auth solution of your choice. | (deployment specific, required to secure UI and management API) |
| Reverse Proxy that merges the UI+Backend and removes the ports | (deployment specific) |
| Postgresql | 13 or compatible version |
| EDC Backend | edc-ce or edc-ce-mds, see [CHANGELOG.md](../../../../CHANGELOG.md) for compatible versions. |
| EDC UI | edc-ui, see [CHANGELOG.md](../../../../CHANGELOG.md) for compatible versions. |
| Deployment Unit | Version / Details |
|-------------------------------------------------------------------|---------------------------------------------------------------------------------------------|
| An Auth Proxy / Auth solution of your choice. | (deployment specific, required to secure UI and management API) |
| Reverse Proxy that merges multiple services and removes the ports | (deployment specific) |
| Postgresql | 13 or compatible version |
| EDC Backend | edc-ce or edc-ce-mds, see [CHANGELOG.md](../../../../CHANGELOG.md) for compatible versions. |
| EDC UI | edc-ui, see [CHANGELOG.md](../../../../CHANGELOG.md) for compatible versions. |

## Configuration

Expand Down Expand Up @@ -79,13 +88,7 @@ EDC_UI_CONFIG_URL: "edc-ui-config"

## EDC Backend Configuration

A sovity EDC CE or MDS EDC CE Backend deployment requires:

- A running DAPS
- (MDS Only) A running Clearing House
- DAPS Access
and [a generated SKI/AKI pair and .jks file](#faq)
- The following configuration properties
A sovity EDC CE or MDS EDC CE Backend deployment requires the following environment variables:

> [!WARNING]
> Please be careful with overriding any of the ENV Vars set in our [launchers/.env](../../../../launchers/.env). Our
Expand All @@ -97,8 +100,9 @@ A sovity EDC CE or MDS EDC CE Backend deployment requires:
# Connector Host Name
MY_EDC_FQDN: "my-edc-deployment1.example.com"

# Connector Technical Name
MY_EDC_NAME_KEBAB_CASE: "example-connector"
# Participant ID / Connector ID
# Must be configured as the value of the "referringConnector" claim in the DAPS for this connector
MY_EDC_PARTICIPANT_ID: "MDSL1234XX.C1234XX"

# Connector Localized Name / Title
MY_EDC_TITLE: "EDC Connector"
Expand Down Expand Up @@ -165,10 +169,5 @@ configuration and be accesible from the data provider via REST calls.

### Can I still use the deprecated Omejdn DAPS?

For Omejdn one needs the following overrides in the backend:

```yaml
EDC_OAUTH_PROVIDER_AUDIENCE: idsc:IDS_CONNECTORS_ALL
EDC_OAUTH_ENDPOINT_AUDIENCE: idsc:IDS_CONNECTORS_ALL
EDC_AGENT_IDENTITY_KEY: client_id
```
In the current version of the sovity EDC CE Connector the Omejdn DAPS is not supported due to the Omejdn DAPS requiring
a special OAuth2 extension and custom messages that exceed the default DSP Oauth2 Specification.
2 changes: 1 addition & 1 deletion docs/getting-started/documentation/api_wrapper.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,4 +42,4 @@ Using The Client
Feel free to use the endpoints of the aforementioned API groups.

A full example providing and consuming a data offer using the API Wrapper Client Library can be found
in [ApiWrapperDemoTest.java](../../../launchers/connectors/sovity-dev/src/test/java/de/sovity/edc/e2e/ApiWrapperDemoTest.java).
in [ApiWrapperDemoTest.java](../../../tests/src/test/java/de/sovity/edc/e2e/ApiWrapperDemoTest.java).
167 changes: 5 additions & 162 deletions docs/sovity-edc-api-wrapper.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ externalDocs:
description: EDC API Wrapper Project in sovity/edc-extensions
url: https://github.com/sovity/edc-extensions/tree/main/extensions/wrapper
servers:
- url: "https://[MY_EDC_FQDN]/api/management"
- url: https://my-connector/api/management
tags:
- name: Enterprise Edition
description: sovity Enterprise Edition EDC API Endpoints. Requires our sovity Enterprise
Expand Down Expand Up @@ -384,23 +384,6 @@ paths:
application/json:
schema:
$ref: '#/components/schemas/IdResponseDto'
/wrapper/use-case-api/data-offer:
post:
tags:
- Use Case
description: "Creates a new data offer, consisting of an asset, a policy definition\
\ and a contract definition."
operationId: createOffer
requestBody:
content:
application/json:
schema:
$ref: '#/components/schemas/CreateOfferingDto'
responses:
default:
description: default response
content:
'*/*': {}
/wrapper/use-case-api/kpis:
get:
tags:
Expand Down Expand Up @@ -1215,6 +1198,7 @@ components:
- assetName
- contractAgreementId
- counterPartyConnectorEndpoint
- counterPartyParticipantId
- createdDate
- direction
- lastUpdatedDate
Expand Down Expand Up @@ -1243,6 +1227,9 @@ components:
counterPartyConnectorEndpoint:
type: string
description: Other Connector's Endpoint
counterPartyParticipantId:
type: string
description: Other Connector's Participant ID
assetName:
type: string
description: Asset Name
Expand Down Expand Up @@ -1328,150 +1315,6 @@ components:
description: Additional transfer process properties. These are not passed
to the consumer EDC
description: "For type PARAMS_ONLY: Required data for starting a Transfer Process"
AssetEntryDto:
required:
- assetRequestId
- dataAddressProperties
type: object
properties:
assetRequestId:
type: string
assetRequestProperties:
type: object
additionalProperties:
type: object
dataAddressProperties:
type: object
additionalProperties:
type: string
description: At least a property 'type' must be set
description: At least a property 'type' must be set
AtomicConstraintDto:
required:
- leftExpression
- operator
- rightExpression
type: object
properties:
leftExpression:
type: string
description: Left part of the constraint.
operator:
$ref: '#/components/schemas/OperatorDto'
rightExpression:
type: string
description: Right part of the constraint.
description: Type-Safe OpenAPI generator friendly Constraint DTO that supports
an opinionated subset of the original EDC Constraint Entity.
ContractDefinitionRequestDto:
required:
- accessPolicyId
- assetsSelector
- contractPolicyId
type: object
properties:
id:
type: string
accessPolicyId:
type: string
contractPolicyId:
type: string
assetsSelector:
type: array
items:
$ref: '#/components/schemas/CriterionDto'
CreateOfferingDto:
type: object
properties:
assetEntry:
$ref: '#/components/schemas/AssetEntryDto'
policyDefinitionRequest:
$ref: '#/components/schemas/PolicyDefinitionRequestDto'
contractDefinitionRequest:
$ref: '#/components/schemas/ContractDefinitionRequestDto'
description: Necessary data for creating an offer
CriterionDto:
required:
- operandLeft
- operator
type: object
properties:
operandLeft:
type: object
operator:
type: string
operandRight:
type: object
ExpressionDto:
type: object
properties:
type:
$ref: '#/components/schemas/ExpressionType'
atomicConstraint:
$ref: '#/components/schemas/AtomicConstraintDto'
and:
type: array
items:
$ref: '#/components/schemas/ExpressionDto'
or:
type: array
items:
$ref: '#/components/schemas/ExpressionDto'
xor:
type: array
items:
$ref: '#/components/schemas/ExpressionDto'
description: Possible constraints for the permission
ExpressionType:
type: string
description: |
Expression types:
* `EMPTY` - No constraints for the policy
* `ATOMIC_CONSTRAINT` - A single constraint for the policy
* `AND` - Several constraints, all of which must be respected
* `OR` - Several constraints, of which at least one must be respected
* `XOR` - Several constraints, of which exactly one must be respected
enum:
- EMPTY
- ATOMIC_CONSTRAINT
- AND
- OR
- XOR
PermissionDto:
required:
- constraints
type: object
properties:
constraints:
$ref: '#/components/schemas/ExpressionDto'
description: Permission for this policy
PolicyDefinitionRequestDto:
required:
- id
- policy
type: object
properties:
id:
type: string
description: ID chosen by the user
policy:
$ref: '#/components/schemas/PolicyDto'
description: Data for creating a policy definition request
PolicyDto:
required:
- legacyPolicy
- permission
type: object
properties:
legacyPolicy:
type: string
description: Legacy JSON as built by the Management API. Will be replaced
in the future by a type-safe variant without polymorphisms that can be
used for our generated clients.
permission:
$ref: '#/components/schemas/PermissionDto'
description: Type-Safe OpenAPI generator friendly Policy DTO that supports an
opinionated subset of the original EDC Policy Entity.
KpiResult:
required:
- assetsCount
Expand Down
Loading
Loading