PROVISIONING.md

Provisioning

Lightning offers the ability to configure projects via the HTTP API.

By providing a JSON document with the desired configuration, the project can be configured to your liking.

Using the API

The API is available at /api/provision, and expects an application/json Content-Type.

Authentication

The API requires a valid auth token to be provided in the Authorization header.

Example Request

 curl -X POST \
   -d @project.json \
   -H "Authorization: Bearer $TOKEN" \
   -H "Content-Type: application/json" \
   $ENDPOINT/api/provision

Document Structure

The provisioning document is a JSON document with the project at the root.

All entities must have an id field, which is a UUIDv4 string. In the case of new entities, this must be generated by the client.

The API is idempotent, and the distinction between creating and updating is determined by the presence of the id field.

{
  "id": "<<project-id>>",
  "name": "<<project-name>>",
  "workflows": [
    {
      "id": "<<workflow-id>>",
      "name": "<<workflow-name>>",
      "jobs": [
        {
          "id": "<<job-id>>",
          "name": "<<job-name>>",
          "body": "<<job-body>>",
          "adaptor": "<<adaptor-name>>",
          "enabled": true
        }
        // ... more jobs
      ],
      "triggers": [
        {
          "id": "<<trigger-id>>",
          "name": "<<trigger-name>>",
          "type": "webhook"
        }
        // ... more triggers
      ],
      "edges": [
        {
          "id": "<<edge-id>>",
          "source_trigger_id": "<<trigger-id>>",
          "target_job_id": "<<job-id>>"
        }
        // ... more edges
      ]
    }
    // ... more workflows
  ]
}

API Behaviour

The API expects all existing entities to be provided in the provisioning document.

If the document provided is out of date (e.g. a new job was added on the server), a new reference document should be fetched and the changes applied to it.

Deleting Entities

Entities can be deleted by setting the disabled key to true.

Example:

{
  "id": "<<project-id>>",
  "workflows": [
    {
      "id": "<<workflow-id>>",
      "jobs": [
        {
          "id": "<<job-id>>",
          "delete": true // <== delete this job
        }
      ]
    }
  ]
}

Relationship with Projects as Code

The Projects as Code spec is a superset of the provisioning API.

Projects as Code allows for the user to specify a key for each entity, which makes it easier to manage the project in the future.

For example:

name: my-project
workflows:
  workflow-one:
    jobs:
      job-one:
        body: |
          console.log("Hello World");
        adaptor: '@openfn/language-common'
        enabled: true
    triggers:
      trigger-one:
        type: webhook
    edges:
      - source_trigger: trigger-one
        target_job: job-one

The above YAML document illustrates the use of keys being used to identify entities. Allowing the user to provision the same project to multiple environments.

The API is unaware of 'keys', and expects IDs to be provided by the client.

In order to convert the above YAML document to a provisioning document, the CLI uses a local state file (if available) to map the keys to UUIDs.

Using the example above a state file might look like this:

{
  "id": "f6ba9a8c-b687-473a-908e-e250686f1eed",
  "workflows": {
    "workflow-one": {
      "id": "f206aa85-4fce-492e-94eb-ffd32c75d178",
      "jobs": {},
      "triggers": {}
    }
  }
}

The state file shows that the project and workflow already exist, but the job, trigger and edge do not. In order to create these new entities, IDs will be applied them.

On a successful application of the provisioning document, the state file will be updated to reflect the new IDs and entities.

{
  "id": "f6ba9a8c-b687-473a-908e-e250686f1eed",
  "workflows": {
    "workflow-one": {
      "id": "f206aa85-4fce-492e-94eb-ffd32c75d178",
      "jobs": {
        "job-one": { "id": "18ed71de-caf8-4822-aefc-5b19351f4016" }
      },
      "triggers": {
        "trigger-one": { "id": "e0b9f357-9cf9-4206-9924-4d5674aad830" }
      },
      "edges": [
        {
          "id": "c239d994-6662-4637-90f8-0293c924b461",
          "source_trigger_id": "e0b9f357-9cf9-4206-9924-4d5674aad830",
          "target_job_id": "18ed71de-caf8-4822-aefc-5b19351f4016"
        }
      ]
    }
  }
}