[New Feature]: Improve dependency management for MI Connectors

[ShammiL]

Description

The current implementations in MI for adding connectors to an integration project and resolving connector dependencies involve several complex and manual steps from both the connector developer and the user sides. To mitigate the problems in the existing implementation, the current mechanisms need to be reformed.

Describe your problem(s)

1. The VSCode extension uses a static JSON file containing all connector metadata. Developers are required to manually update this list every time a connector is updated or a new one is developed.
2. When a user adds a connector to an integration project, the user is required to manually add all the dependency jars.
3. The connector is downloaded and stored inside the project long before build time, when ideally connector and its dependencies should be resolved at build time.

Describe your solution(s)

1. Come up with a proper API spec for the connector store that allows
   • Listing connectors based on a priority algorithm
   • Searching connectors and operations
   • fetching connector metadata including operations and versions released
2. Implement workflows to automate releasing connectors to Maven and treat connectors as Maven dependencies of a project
3. Implement a dependency resolution mechanism to handle connector dependencies and release the users from the burden of having to manually add jars.

Related issue(s) (optional)

No response

Suggested label(s) (optional)

No response

Suggested assignee(s) (optional)

No response

[ShammiL]

Following initial discussions with the teams overseeing the MI VSCode extension and the Connector Store, an API specification has been drafted outlining the required functionalities from the Connector Store side. The APIs will provide access to the following features:

• Retrieve a list of all connectors along with their metadata.
• Search for specific connectors.
• Publish new connectors or release updates to existing connectors in the store.
• Fetch metadata for previous versions of a connector.

The specification can be found here

A final specification will be prepared after reviewing the above draft

[ShammiL]

One goal of this feature is to automate the connector release process, which is currently done manually. At present, the procedure involves creating the connector ZIP artifacts locally and uploading them to manually created Git release tags. To streamline this process, the original plan was to implement Jenkins workflows to:

• Build and package the connector into a ZIP artifact,
• Release the ZIP artifact to the WSO2 public Maven repository,
• Call the connector store's publish API,
• Create a release in the corresponding Git repository.

As a preliminary step, before setting these workflows up in the main connector repositories and deploying artifacts to the public Maven repository, the process was tested using a local setup and a fork of a connector repository. This involved:

• Installing Jenkins locally to create ZIP artifacts, push them to a locally hosted Nexus repository, and generate release tags in the forked Git repository.
• Setting up a Git Webhook in the forked repository to trigger the Jenkins build process.

However, a decision was taken to transition from Jenkins to GitHub workflows for this task, aligning with the ongoing effort to migrate all MI-related automation tasks from Jenkins to GitHub workflows.

[ShammiL]

After finalizing the API spec for the connector store[1], as the proposed solution would require collaboration and consensus across multiple teams, and considering the time constraints, a milestone plan has also been outlined in this design document.[2]

[1] https://gist.github.com/ShammiL/606f1bc87af66a33b391864ff6fb12e8
[2] https://docs.google.com/document/d/1XlWv_71iE2VKQtpuWPIzKNpWY66RZcb8qx39Te2R96k/edit?usp=sharing

As our next step, we will be initiating discussions with the teams we believe will be impacted by the proposed changes to ensure alignment and address any concerns.

Simultaneously, other areas highlighted in the design will also be addressed.

[ShammiL]

After a discussion with members of the IS team looking over IS connectors, it was concluded that the changes proposed for the connector store in Milestone 1 will have no impact on the IS connectors or their current release workflows. Along with that, we also came to the following conclusions:

1. The new GET API will be implemented aligning with the proposed API spec and the new optional fields in the meta files will be handled from the connector store. This will have no impact on the IS connectors and their release workflows.
2. Decisions taken regarding the query parameters are as below:

• The parameter “product” will continue to be used to filter connectors by the product (IS and MI)
• The "category" parameter will be used exclusively to filter connectors by their type, rather than being utilized to categorize connectors based on various metrics as it is now.In the case of MI connectors there will be two types: Inbound and Outbound connectors
• The MI and IS teams will define an ENUM of accepted values for the "category" field, which will be used by the connector store to validate and enforce the categories specified in connector metadata files.
• In the Connector Store UI, filtering by the category will only be visible to the user once a product is selected.

However, a concern was raised about allocating resources for the implementations proposed for the connector store, due to the limited availability of members on the internal apps team.

[ShammiL]

A release workflow has been implemented in the Redis Connector repository, marking the initial step in integrating release workflows across all MI connector repositories. The workflow makes use of the Maven release plugin to perform the release.
PR - wso2-extensions/mi-connector-redis#60

This workflow will:

• Create a ZIP release artifact through a Maven build
• Publish the artifact to the WSO2 Nexus repository using the correct release versions
• Generate release tags in the Git repository and attach the release artifacts
• Prepare the connector for the next release.

The individual steps and the overall workflow were initially tested in a local environment and then on a forked repository of the Redis Connector. A locally configured Nexus server was used to simulate the WSO2 release repository.

Once the workflow has been tested on this repository, the same changes will be applied to all MI connector repositories

[ShammiL]

Due to the lack of resources to be allocated from the digital transformation team, the MI team will implement the new GET endpoint to retrieve all MI connector details to be used by the MI VS Code extension. [1].

This implementation will cover the following:

1. Add two new tables to the connector store database mi_metadata and mi_operations.
2. Add a new GET endpoint to retrieve connector metadata including the mi-specific metadata and operations
3. Update the webhooks and the daily sync jobs to read the new metadata fields in the MI connector repositories and update the database accordingly.
   Tentative structures for the tables are as below:

mi_metadata

mi_operations

[1] https://github.com/wso2-enterprise/integration-product-management/issues/248