add feature to export and import BPMN state, incl. resume capability

[nitram509]

it would be awesome for use cases where long-running processes and are implemented in short living microservices,
if lib-bpmn-engine would be able to export and import+resume it's internal state.

• must: the data must be represented as ASCII-save string, to allow maximum compatibility and avoid codepage/unicode issues
• could: the data format should be human readable, e.g. JSON
• optional: data compression
• optional: base64 encoding
• must: the feature must be agnostic to library upgrades and for example allowing schema changes (optional: split this into another feature)
  • e.g. version info in the serialized file
• documentation is up to date

[cq-z]

Suggestion:
add ”dump“ and ” regain“ api 。So that compatible on the storage implementation
The storage implementation can be create tables (events 、tasks...) to save state and data like：https://github.com/antlinker/flow/blob/develop/schema/s_flow.go

[nitram509]

Hi @cq-z thank you for your suggestion.
I will try my best to make it somewhat compatible, but will not make it a 1:1 fit.
I rather think of optimizing for this lib but allowing an optional "adapter function", which could map the schemas.

@cq-z May I ask: what use case do you have in mind, for making this suggestion?

PS:
See also https://github.com/nitram509/lib-bpmn-engine/blob/feature/export_and_import_BPMN_state/pkg/bpmn_engine/engine_marshal.go
for my latest work in progress.

[cq-z]

example :
Marshal to model like https://gorm.io/docs/has_many.html, after that model to save。
model Select， after that model Unmarshal to BpmnEngineState

Marshal support [] byte or struct , or like https://github.com/jinzhu/copier implementation BpmnEngineState to model
Whether the scalability is better

[cq-z]

support grom like demo

package main

import (
	"fmt"
	"github.com/jinzhu/copier"
	"github.com/nitram509/lib-bpmn-engine/pkg/bpmn_engine"
	"github.com/stretchr/testify/assert"
	"gorm.io/driver/mysql"
	"gorm.io/gorm"
	"testing"
)

type serializedBpmnEngine struct {
	Id                   int64                  `gorm:"foreignKey:autoIncrement"`
	Version              int                    `json:"version"`
	Name                 string                 `json:"name"`
	MessageSubscriptions []*MessageSubscription `json:"MessageSubscriptions,omitempty" gorm:"foreignKey:BpmnId"  copier:"GetMessageSubscriptions"`
	Processes            []ProcessInfo          `json:"Processes,omitempty" gorm:"foreignKey:BpmnId" copier:"GetProcessInstances"`
}
type MessageSubscription struct {
	Id                 int64 `gorm:"foreignKey:autoIncrement"`
	BpmnId             int64
	ElementId          string `json:"ElementId"`
	ElementInstanceKey int64  `json:"ElementInstanceKey"`
	ProcessInstanceKey int64  `json:"ProcessInstanceKey"`
	Name               string `json:"Name"`
	State              string `json:"State"`
	CreatedAt          int64  `json:"CreatedAt"`
}
type ProcessInfo struct {
	Id            int64 `gorm:"foreignKey:autoIncrement"`
	BpmnId        int64
	BpmnProcessId string `json:"BpmnProcessId"` // The ID as defined in the BPMN file
	Version       int32  `json:"Version"`       // A version of the process, default=1, incremented, when another process with the same ID is loaded
	ProcessKey    int64  `json:"ProcessKey"`    // The engines key for this given process with version

	// TODO: make them private again?
	Definitions   string `json:"definitions"`   // parsed file content
	ChecksumBytes string `json:"checksumBytes"` // internal checksum to identify different versions
}

var bpmnByte = `
<?xml version="1.0" encoding="UTF-8"?>
<bpmn:definitions xmlns:bpmn="http://www.omg.org/spec/BPMN/20100524/MODEL" xmlns:bpmndi="http://www.omg.org/spec/BPMN/20100524/DI" xmlns:dc="http://www.omg.org/spec/DD/20100524/DC" xmlns:zeebe="http://camunda.org/schema/zeebe/1.0" xmlns:di="http://www.omg.org/spec/DD/20100524/DI" xmlns:modeler="http://camunda.org/schema/modeler/1.0" id="Definitions_1u3x2yl" targetNamespace="http://bpmn.io/schema/bpmn" exporter="Camunda Modeler" exporterVersion="5.0.0" modeler:executionPlatform="Camunda Cloud" modeler:executionPlatformVersion="1.0.0">
  <bpmn:process id="simple-user-task" name="simple-user-task" isExecutable="true">
    <bpmn:startEvent id="StartEvent_1">
      <bpmn:outgoing>Flow_0xt1d7q</bpmn:outgoing>
    </bpmn:startEvent>
    <bpmn:sequenceFlow id="Flow_0xt1d7q" sourceRef="StartEvent_1" targetRef="user-task" />
    <bpmn:endEvent id="Event_1j4mcqg">
      <bpmn:incoming>Flow_1vz4oo2</bpmn:incoming>
    </bpmn:endEvent>
    <bpmn:sequenceFlow id="Flow_1vz4oo2" sourceRef="user-task" targetRef="Event_1j4mcqg" />
    <bpmn:userTask id="user-task" name="user-task">
      <bpmn:extensionElements>
        <zeebe:assignmentDefinition assignee="assignee" candidateGroups="candicate-groups" />
        <zeebe:formDefinition formKey="form-key" />
      </bpmn:extensionElements>
      <bpmn:incoming>Flow_0xt1d7q</bpmn:incoming>
      <bpmn:outgoing>Flow_1vz4oo2</bpmn:outgoing>
    </bpmn:userTask>
  </bpmn:process>
  <bpmndi:BPMNDiagram id="BPMNDiagram_1">
    <bpmndi:BPMNPlane id="BPMNPlane_1" bpmnElement="simple-user-task">
      <bpmndi:BPMNEdge id="Flow_1vz4oo2_di" bpmnElement="Flow_1vz4oo2">
        <di:waypoint x="370" y="117" />
        <di:waypoint x="432" y="117" />
      </bpmndi:BPMNEdge>
      <bpmndi:BPMNEdge id="Flow_0xt1d7q_di" bpmnElement="Flow_0xt1d7q">
        <di:waypoint x="215" y="117" />
        <di:waypoint x="270" y="117" />
      </bpmndi:BPMNEdge>
      <bpmndi:BPMNShape id="_BPMNShape_StartEvent_2" bpmnElement="StartEvent_1">
        <dc:Bounds x="179" y="99" width="36" height="36" />
      </bpmndi:BPMNShape>
      <bpmndi:BPMNShape id="Event_1j4mcqg_di" bpmnElement="Event_1j4mcqg">
        <dc:Bounds x="432" y="99" width="36" height="36" />
      </bpmndi:BPMNShape>
      <bpmndi:BPMNShape id="Activity_102w2rp_di" bpmnElement="user-task">
        <dc:Bounds x="270" y="77" width="100" height="80" />
        <bpmndi:BPMNLabel />
      </bpmndi:BPMNShape>
    </bpmndi:BPMNPlane>
  </bpmndi:BPMNDiagram>
</bpmn:definitions>
`

func TestGrom(t *testing.T) {

	db, err := gorm.Open(mysql.Open("root:123456@tcp(127.0.0.1:3306)/test"), &gorm.Config{})
	if err != nil {
		panic("failed to connect database")
	}

	db.AutoMigrate(serializedBpmnEngine{}, ProcessInfo{}, MessageSubscription{})

	// setup
	bpmnEngine := bpmn_engine.New("name")
	process, _ := bpmnEngine.LoadFromBytes([]byte(bpmnByte))

	variables := map[string]interface{}{
		"price": -50,
	}

	// when
	bpmnEngine.CreateAndRunInstance(process.ProcessKey, variables)
	processes := bpmnEngine.GetProcessInstances()
	var gormProcesses []ProcessInfo
	err = copier.Copy(&gormProcesses, processes)
	fmt.Println(err)
	messageSubscriptions := bpmnEngine.GetMessageSubscriptions()
	var gormMessageSubscriptions []*MessageSubscription
	err = copier.Copy(&gormMessageSubscriptions, messageSubscriptions)
	fmt.Println(err)
	s := &serializedBpmnEngine{
		Name:                 bpmnEngine.GetName(),
		Processes:            gormProcesses,
		MessageSubscriptions: gormMessageSubscriptions,
	}
	//err = copier.CopyWithOption(s, bpmnEngine, copier.Option{
	//	IgnoreEmpty: true,
	//	DeepCopy:    true,
	//	Converters: []copier.TypeConverter{
	//		{
	//			SrcType: []ProcessInfo{},
	//			DstType: []bpmn_engine.ProcessInfo{},
	//			Fn: func(src interface{}) (interface{}, error) {
	//				s, ok := src.(string)
	//				if !ok {
	//					return nil, fmt.Errorf("src type not matching")
	//				}
	//				return s, nil
	//			},
	//		},
	//	}})
	fmt.Println(err)
	// then
	db.Create(s)
	s1 := &serializedBpmnEngine{}
	db.Model(&serializedBpmnEngine{}).First(&s1)
	var bpmnEngine1 bpmn_engine.BpmnEngineState
	err = copier.CopyWithOption(&bpmnEngine1, s1, copier.Option{})
	assert.Equal(t, bpmnEngine1, bpmnEngine)
}

[cq-z]

Other solutions：
bpmnEngine to json, json to grom struct.
grom struct to json , json to bpmnEngine.

[eriknyk]

@nitram509 any idea when feature/export_and_import_BPMN_state will be merged to main branch, and release new version?
In terms of generic way to persist the state looks enough.

Regards.

[nitram509]

FYI: I'm working on this and have pushed first working version (main branch).

[cq-z]

Hello @nitram509 , feature/export_and_import_BPMN_state is ready?

[nitram509]

@cq-z unfortunately, still in progress.

I found little spare time during the last weeks/months and plan to continue work the next weeks.

[cq-z]

I can do something for you

[nitram509]

Hi @cq-z , @eriknyk ,

I did some work to implement the marshalling and un-marschalling support.
If you're interested, have a look at the recent https://github.com/nitram509/lib-bpmn-engine/releases/tag/v0.3.0-rc1
release.
Documentation is missing, but have a look at tests/marshalling_test.go which shows how the feature can be used.

I will continue improving documentation and likely do some more changes towards Getter&Setter API rework as the other open issues for a proper v0.3.0 release indicate.

I would be glad to hear your feedback, and if the marshall & un-marshall works for you.