RFC: New API for import

Summary

Introduces a new process for importing an app into the Flogo Server that addresses existing limitations.

Motivation

The current import process only allows to tap into the processing of a single handler or a single resource at a time. This approach limits the flexibility of the import and makes it complicated to incorporate processing actions defined at the root app.json level and to process handlers that are linked to more than on action (triggers[].handler.actions[] array). It also makes it complicated to incorporate concepts that span beyond the scope of a single handler or resource, such as channels and subflows, because there's no access to the rest of the application being imported.

Support for shared actions

In a flogo.json a handler can be linked to a resource through an inline action or through a shared action. At the same time, a handler can declare a single action or multiple actions. With these concepts combined we have four variants to link a handler to a resource. The current import process only has support for one of those variants: a handler linked to a single resource through a single inlined action.

Examples of the possible options:

{
  "triggers": [
    {
      "id": "someId",
      "handlers": [
        {
          // single shared action
          "action": {
            "id": "sharedActionId"
          }
        },
        {
          // multiple shared actions
          "actions": [
            {
              "id": "sharedActionId"
            }
          ]
        },
        {
          // single inline action
          "action": {
            "ref": "github.com/some/repo/cool-resource",
            // the property that links to the resource is defined by the resource itself, we don't have a generic way to know
            // the name of this property, it has to be processed by the plugin that corresponds to this type of resource
            "linkToTheResource": "res://cool-resource:myResourceId"
          }
        },
        {
          // multiple shared actions
          "actions": [
            {
              "ref": "github.com/some/repo/cool-resource",
              "linkToTheResource": "res://cool-resource:myResourceId"
            }
          ]
        }
      ]
    }
  ],
  "actions": [
    {
      "id": "sharedActionId",
      "ref": "github.com/some/repo/cool-resource",
      // the property that links to the resource is defined by the resource itself, we don't have a generic way to know
      // the name of this property, it has to be processed by the plugin that corresponds to this type of resource
      "linkToTheResource": "res://cool-resource:myResourceId"
    }
  ],
  "resources": [
    {
      "id": "cool-resource:someResourceId",
      "name": "my cool resource",
      "data": {}
    }
  ]
}

Basic example

The plugin exposes a single function that when called will import all its related objects (handlers, actions, resources).

The general idea is that we provide both the raw app and an application "draft", the application draft represents the work in progress. If the plugin wants to import something it needs to register it into the app draft. Whatever changes made to the app draft are the changes that are going to be persisted.

/** 
Function exposed by the plugin. This function will be called during the import process.
@param {RawAppWrapper} rawAppWrapper - original app being imported plus some utility functions to make it easier to query the app 
@param {AppDraft} appDraft - the in progress work
*/
export function customPluginImporter(rawAppWrapper: RawAppWrapper, appDraft: AppDraft) {
  // use utility to query for handlers 
  const originalHandlers = rawAppWrapper.getHandlersByType('github.com/some/resource/type');
 
  originalHandlers.forEach(
    originalHandler => {
      const normalizedHandler = runSomeResourceSpecificLogic(originalHandler);
      // plugin knows how to navigate its type of handler
      const resourceId = origHandler.action.linkToResource;
      let resource = appDraft.resources.get(resourceId);
      if (!resource) {
         // resource hasn't been registered yet
         const rawResource = rawAppWrapper.resources.get(resourceId);
         resource = normalizeResource(resource);
      }
      const trigger = appDraft.triggers.get(origHandler.triggerId);
      // resource is a direct reference, this way we can remove the burden from resource plugin of reconciling the
      // resources/triggers/handlers with the normalized ids.
      trigger.addHandler({
        ...origHandler,
        resource,
      });
    }
  );

}

Detailed design

Glossary:

id normalization: process to translate the user defined IDs (trigger id, action id, etc.) in an app.json to the internal format of the ids.
id reconciliation: process to reconcile the initial IDs to the internal IDs. It's necessary to fix the references between entities (ex. handler to resource, subflows) after the ID normalization has been applied.

Requirements

It should allow a way to validate the data and report import errors back to the user.
--- ??

Handler to resource links will need to be normalized so internally we only deal with one type of link. We will have the actions always to be "shared" actions, this will allow to support all possible combinations of handler to resource links. For this all inlined actions will need to be converted to shared actions during import.

Normalized model:

+---------+       +----------+      +--------+         +----------+
| trigger |------>| handler |------>| action |-------->| resource |
+---------+       +---+-----+       +--------+         +----------+

High level view of the process from Flogo Server perspective

The import process relies in two principal components:

The RawAppWrapper: which contains the initial data being imported (the flogo.json data) and additional utilities to query for data, for example a getHandlersByRef(). This is the data to be imported and should be considered read-only, changes to this structure won't be persisted.
The `AppDraft: represents an app in-progress of being imported. The plugins can modify the draft as they deem convenient. At the end the draft will be converted into the app that will be stored in the server.

Import process steps:

Create the app wrapper out of the raw import data
Create the app draft out of the app wrapper. At this point we process the entities from which we know their structure: triggers, handlers and some of the handler's inline action data. Also normalize handler action references.
Delegate control to the plugins and pass the app wrapper and app draft to allow them to make modifications to the app draft.
Apply final modifications to the draft such as id normalization.

Pseudocode:

// interface that the plugin needs to implement
type ImporterFn = (rawAppWrapper: RawAppWrapper, appDraft: AppDraft) => void;

// sample of import process in the flogo server app
function importApp(rawApp: object, plugins: ImporterFn[]) {
  // minimum normalization. no ids normalization
  const appWrapper: RawAppWrapper = createAppWrapper(rawApp);
  const appDraft: AppDraft = createAppDraft(appWrapper);
  plugins.forEach(plugin => plugin(appWrapper, appDraft));
  const importedApp = finalizeDraft(appDraft);
  appService.save(importedApp);
}

Exposed to the plugin

RawAppWrapper

RawAppWrapper provides access to the app being imported and expands it with utilities to make it easier to navigate said app.

interface RawAppWrapper {
  rawApp: FlogoCore.App;
  getResourcesByRef(ref: string): FlogoCore.Resource[];
  getHandlersByRef(ref: string): FlogoCore.Handler[];
  getResourceById(id: string): FlogoCore.Resource;
}

App Drafts

App in progress of being imported. Plugin will update the draft with its changes and the draft will be processed and persisted as the actual UI app being saved.

interface AppDraft {
  name: string;
  // ... other properties like description
  triggers: Map<string, TriggerDraft>;
  actions: Map<string, Action>;
  resources: Map<string, ResourceDraft>;
}

interface TriggerDraft {
  id: string;
  ref: string;
  getHandlers(): HandlerDraft[];
  addHandler(handler: HandlerDraft);
}

interface ActionDraft {
  id: string;
  ref: string;
  getHandlers(): HandlerDraft[];
  addHandler(handler: HandlerDraft);
}


interface HandlerDraft {
  // direct access to its parent
  trigger: TriggerDraft;
  // expect object references and not ids, this way we can normalize the ids afterwards
  // and processing should feel more natural
  resource: ResourceDraft;
}

interface ResourceDraft {
  id: string;
  ref: string;
  name: string;
  data: unknown;
}

Drawbacks

todo

Alternatives