PROCEED Engine ‐ Process Instance, Tokens, Datastructure and REST Endpoint - PROCEED-Labs/proceed GitHub Wiki

Legend:

  • end state: refers to a state value of FORWARDED, ENDED, ABORTED, FAILED, TERMINATED, ERROR-SEMANTIC, ERROR-TECHNICAL, ERROR-CONSTRAINT-UNFULFILLED, ERROR-UNKNOWN, or STOPPED

General

This page explains the execution of a process by looking at the internal instance state. The exact instance data, explanation and REST endpoints are explained later. The following diagram depicts the state changes and what is done in the background:

PROCEED-BPMN-Engine_Execution_and_Deployment_States_2022_01_17

If a process was started, the instance data is never deleted again. Even not if the instance ended. The PROCEED engine then just stores the last state of the instance data in the internal database (it does that if the instanceState changes to an end state, see Legend). This persistent storage is useful and necessary to monitor the process execution (pull monitoring), even if it already ended on one Machine.

The Instance

An Instance in a PROCEED Engine has the following data structure:

{
  "processId": "<process-id>",
  "processVersion": <number: date-time in ms since 1.1.1970>,
  "processInstanceId": "<random uuid>",
  "globalStartTime": <number: date-time in ms since 1.1.1970>,
  "instanceState": [
    "PAUSING||PAUSED||STOPPED", // variant 1: single status
    "RUNNING", // variant 2: mixed status if tokens have different state
    "READY", // variant 2
    "DEPLOYMENT-WAITING", // variant 2
    "FORWARDED", // variant 2
    "ENDED", // variant 2
    "ABORTED", // variant 2
    "FAILED", // variant 2
    "TERMINATED", // variant 2
    "ERROR-SEMANTIC", // variant 2
    "ERROR-TECHNICAL", // variant 2
    "ERROR-CONSTRAINT-UNFULFILLED", // variant 2
    "ERROR-UNKNOWN" // variant 2
  ],
  "tokens": [{
      "tokenId": "<random 7 char ID with numbers and letters>",
      "state": "RUNNING|READY|PAUSED|DEPLOYMENT-WAITING|FORWARDED|ENDED|ABORTED|FAILED|TERMINATED|ERROR-SEMANTIC|ERROR-TECHNICAL|ERROR-CONSTRAINT-UNFULFILLED|ERROR-UNKNOWN|SKIPPED|STOPPED",
      "currentFlowElementId": "<flow-element-id>",
      "previousFlowElementId": "<flow-element-id>",
      "currentFlowNodeIsExternal": <boolean>,
      "currentFlowNodeState": "READY|ACTIVE|EXTERNAL|COMPLETED|FAILED|TERMINATED",  //  <-- cleared after FlowNode
      "currentFlowElementStartTime": <number: date-time in ms since 1.1.1970>,
      "localStartTime": <number: date-time in ms since 1.1.1970>,
      "localExecutionTime": <number: time in ms>,
      "currentFlowNodeProgress": <number: 0-100>,  //  <-- cleared after FlowNode
      "actualOwner": ["c0923d63-753f-4c12-a41a-7f6a179ccc83", "e6915b75-ac0e-454a-9ad3-3fae08d5576e"],  //  <-- user ids who work on a usertask
      "intermediateVariablesState": {  // cleared after flow node
        "a": 20
      },
      "milestones": {     //  <-- optional, cleared after FlowNode
            "<milestone-id>": <number: 0-100>,
            "<milestone-id>": <number: 0-100>,
            ...
      },
      "costsRealSetByOwner": "<string: costs with specified currency>",
      "deciderStorageRounds": 0,
      "deciderStorageTime": <number: time in ms>,
      "machineHops": 0,
      "nextMachine": { "id": "<machine-id>", "ip": "<ip>", "port": "<port>","name": "<machine-name>"}
    },{
      "tokenId": "<concatenated tokenIds, e.g., 4r90id6|2-3-dfui83e_65efxmn|1-2-pkle9q1#w3a5azs>",
      "state": "RUNNING|READY|PAUSED|DEPLOYMENT-WAITING|FORWARDED|ENDED|ABORTED|FAILED|TERMINATED|ERROR-SEMANTIC|ERROR-TECHNICAL|ERROR-CONSTRAINT-UNFULFILLED|ERROR-UNKNOWN|SKIPPED|STOPPED",
      "currentFlowElementId": "flow-element-id",
      "previousFlowElementId": "<flow-element-id>",
      "currentFlowNodeIsExternal": <boolean>,
      "currentFlowNodeState": "READY|ACTIVE|EXTERNAL|COMPLETED|FAILED|TERMINATED",  //  <-- cleared after FlowNode
      "currentFlowElementStartTime": <number: date-time in ms since 1.1.1970>,
      "localStartTime": <number: date-time in ms since 1.1.1970>,
      "localExecutionTime": <number: time in ms>,
      "currentFlowNodeProgress": <number: 0-100>,  //  <-- cleared after FlowNode
      "actualOwner": ["c0923d63-753f-4c12-a41a-7f6a179ccc83", "e6915b75-ac0e-454a-9ad3-3fae08d5576e"],  //  <-- user ids who work on a usertask
      "intermediateVariablesState": {  // cleared after flow node
        "b": 35
      },
      "milestones": {     //  <-- optional, cleared after FlowNode
            "<milestone-id>": <number: 0-100>,
            "<milestone-id>": <number: 0-100>,
            ...
      },
      "costsRealSetByOwner": "<string: costs with specified currency>",
      "deciderStorageRounds": 0,
      "deciderStorageTime": <number: time in ms>,
      "machineHops": 0,
      "nextMachine": { "id": "<machine-id>", "ip": "<ip>", "port": "<port>", "name": "<machine-name>"}
    }
  ],
  "variables": {
    "a": {
      "value": 10,
      "log": [
        { "changedTime": <number: date-time in ms since 1.1.1970>, "changedBy": "<flow-node-id>", "oldValue": <old-process-value> },
        { "changedTime": <number: date-time in ms since 1.1.1970>, "changedBy": "<flow-node-id>", "oldValue": <old-process-value> },
        ...
      ]
    },
    "name": {
      "value": "max",
      "log": [
        { "changedTime": <number: date-time in ms since 1.1.1970>, "changedBy": "<flow-node-id>", "oldValue": <old-process-value> },
        { "changedTime": <number: date-time in ms since 1.1.1970>, "changedBy": "<flow-node-id>", "oldValue": <old-process-value> },
        ...
      ]
  },
  "log": [
      {
        "executionState": "COMPLETED|ABORTED|FAILED|TERMINATED|ERROR-SEMANTIC|ERROR-TECHNICAL|ERROR-CONSTRAINT-UNFULFILLED|ERROR-UNKNOWN|SKIPPED|STOPPED",
        "tokenId": "_safdss...",
        "flowElementId": "<flow-node-id>",
        "machine": { "id": "<machine-id>", "ip": "<ip>", "name": "<machine-name>"},
        "startTime": <number: date-time in ms since 1.1.1970>,
        "endTime": <number: date-time in ms since 1.1.1970>,
        "milestones": {     //  <-- optional
            "<milestone-id>": <number: 0-100>,
            "<milestone-id>": <number: 0-100>,
            ...
        },
        "costsRealSetByOwner": "<string: costs with specified currency>",
        "external": boolean,
        "stopped": boolean,
      },
      {
        "executionState": "ERROR-SEMANTIC|ERROR-TECHNICAL|ERROR-CONSTRAINT-UNFULFILLED|ERROR-UNKNOWN",
        "tokenId": "_safdss...",
        "flowElementId": "<flow-node-id>",
        "machine": { "id": "<machine-id>", "ip": "<ip>", "name": "<machine-name>"},
        "startTime": <number: date-time in ms since 1.1.1970>,
        "endTime": <number: date-time in ms since 1.1.1970>,
        "milestones": {     //  <-- optional
            "<milestone-id>": <number: 0-100>,
            "<milestone-id>": <number: 0-100>,
            ...
        },
        "errorMessage": "<error msg in case of a fault>",
      },
      {
        "executionState": "FORWARDED",
        "tokenId": "_safdss...",
        "flowElementId": "<flow-element-id>",
        "startTime": <number: date-time in ms since 1.1.1970>,
        "endTime": <number: date-time in ms since 1.1.1970>,
        "nextMachine": { "id": "<machine-id>", "ip": "<ip>", "port": "<port>", "name": "<machine-name>"}
      },
      ...
  ],
  "adaptationLog": [
      {
        "type": "MIGRATION|TOKEN-ADD|TOKEN-REMOVE|TOKEN-MOVE|VARIABLE-ADAPTATION",
        "time": <number: date-time in ms since 1.1.1970>,
        "sourceVersion": <number: version that was used prior to a migration>,
        "targetVersion": <number: version that is used after a migration>,
        "currentFlowElementId": <string: id of the element a token was added to in case of TOKEN-ADD or TOKEN-MOVE>,
        "targetFlowElementId": <string: id of the element a token was removed from in case of TOKEN-REMOVE or TOKEN-MOVE>,
      }
  ],
  "isCurrentlyExecutedInBpmnEngine": false
}
Var Name Explanation Set by PROCEED Engine Set by BPMN Engine Given to the BPMN Engine at process start
processId id of the main <process> of the BPMN process description X
processVersion process version of the BPMN diagram (if a process is edited multiple times, there are multiple different versions of one process; unique number) X
instanceId globally unique identifier for an instance X X
instanceState indicates the current State of a Process Instance. Usually, this Array is calculated by the state of every token - we call this the mixed state. If all tokens have the same state, it contains only this one entry. If the tokens have different states, the Array contains every unique state only one time.
There is also a single state with only one entry in the array. This state is set via the API, overriding and indicating the intended state which is valid for all tokens.

RUNNING = instance is executing
PAUSING = instance will be paused in the future and is waiting for every token to be paused. This can need some time because every activity (especially Script Tasks) needs to finish its execution first to not be in an inconsistent state. PAUSING is a single state and will override all other states to indicate that the instance is going to pause.
PAUSED = indicates that the instance is completely paused. After an instance is in PAUSING mode, it needs to wait for every active token to go into the PAUSED state. This can take a while since the running activity is first fully executed. If all tokens are in PAUSED state or in an end state, the instanceState also goes into PAUSED. PAUSED is a single state and will override all other states to indicate that the instance is paused.
READY = waiting at a GW for further tokens or at an (intermediate) catching event for the event to occur
DEPLOYMENT-WAITING = waiting to find an appropriate next Machine for deployment
FORWARDED = all instance tokens are forwarded to other engines, instance ended on this machine
ENDED = all tokens have ended
ABORTED = a Terminate end event ended all open tokens (not named "terminated" because ambiguity with the BPMN Spec for interrupting boundary events)
FAILED = an Interrupting Error Event had occured (on a boundary or in an event subprocess)
TERMINATED = an Interrupting Non-Error Event had occured, e.g. Timer, Condition, etc. (on a boundary or in an event subprocess)
ERROR-SEMANTIC = for functional error within the BPMN diagram. For example, if an activity throws an error, but there is no error handler available (e.g. no error boundary event)
ERROR-TECHNICAL = occurs if the token notices critical technical problems, e.g. a syntax error in a script
ERROR-CONSTRAINT-UNFULFILLED = the token/instance is stopped because an execution constraint was not fulfilled (e.g. maxStorageTime)
ERROR-UNKNOWN = is the catch-all error. It occurs if there was an error thrown that we (the devs) didn't foresee and that we could not catch by another error handler. By catching this error, we prevent that the Engine crashes and manual recovery should be possible.
STOPPED = instance was stopped by an external application via the API, e.g. by an MS (not in mixed state)
SKIPPED = is a very short status of a token. It occurs if a token is set to another FlowElement by an external application via the API, e.g. by an MS. The status is set, the log is written and then the token is put to the new FlowElement, resulting in overriding the token with another state like RUNNING.

The start of a process, PAUSING/PAUSED and STOPPED state is set by an external program via the API for the whole instance (see /instanceState endpoint).

If an error happens during token execution (state ERROR-SEMANTIC or ERROR-TECHNICAL) only this one token is stopped not the whole instance (Reason: to don't interrupt parallel processes and allow manual recovery).		 X X X
globalStartTime time and date of the instance start X X
variables the process variables X X X
log contains entries about the execution of a FlowNode by the tokens (not FlowElement, i.e. no Sequence Flow). It is usually filled after a FlowNode was executed. But there are also some more situations when an entry is written: if the token is forwarded to another Engine, if a splitting Gateway is reached (the incoming token is logged and deleted, new outgoing tokens are then generated), and if a boundary event occurs.
There are different types of log entries: normal execution entry, error entry, and forwarding entry

Error Entry: errorMessage is optional and only filled in case of the state ERROR-SEMANTIC, ERROR-TECHNICAL, ERROR-CONSTRAINT-UNFULFILLED, ERROR-UNKNOWN.

Forwarding Entry: If the token gets into the state FORWARDED, the PROCEED Engine logs the tokenId, flowNodeId (which is a SequenceFlow), the startTime (time before the Decider started), the endTime (time when the Token is sent to the next Machine) and nextMachine info.		 X X
adaptationLog contains entries about changes to the instance state that were triggered by a user and are not a result of regular execution X
isCurrentlyExecuted InBpmnEngine if the local execution of the instance is still running inside the BPMN Engine (not the surrounding PROCEED Engine) X
token. tokenId a uniquely identifying string for a token. It usually has 7 characters with numbers and letters. Because during process runtime there are often "new" tokens generated, there is a convention how these tokenIds are generated depending on the trigger:

inclusive or parallel Split or Boundary Event: every outgoing sequence flow contains a new, cloned token based on the incoming token (which is deleted afterwards). The id has the form <random-7-char-ID of incoming token>|<sequenceFlow-Number>-<quantity-of-all-outgoing-sequence-flows>-<random-7-char-ID> with a | as separator, e.g., for the token on sequence flow 2 out of 3 outgoing sequence flows 4r90id6|2-3-dfui83e

inclusive or parallel Merge: the outgoing sequence flow(s) contains a new, cloned token based on the incoming tokens (which are deleted afterwards). The id has the form <incoming tokenId>_<incoming tokenId>_<incoming tokenId>_... with a _ as separator, e.g., for two incoming tokes on a parallel GW the outgoing sequence flow has 4r90id6|2-3-dfui83e_4r90id6|3-3-kjdf79y. Only in special cases (structured diagrams) the beginning of a token is merged and the trailing part is deleted.

Embedded Subprocesses: a new token based on the parent token is generated. The id has the form <parent tokenId>#<childTokenId random-7-char-ID> with a # as separator, e.g., 4r90id6#ieu7io1. The subprocesses tokens stay ended inside the token array after the token has been consumed. The parent token is continued in the parent process.		 X X
token.state the current state of a token, see instanceState for explanations; does not contain PAUSING and STOPPED;
DEPLOYMENT-WAITING, FORWARDED, ERROR-UNKNOWN, ERROR-CONSTRAINT-UNFULFILLED is set by the PROCEED Engine; READY, RUNNING, PAUSED, FAILED, TERMINATED, ERROR-SEMANTIC, ERROR-TECHNICAL, ENDED is set by the BPMN engine; ABORTED is set by both		 X X X
token. previousFlowElementId the id of the element the token came from when it reached the current flow element. This can be a FlowNode (Activity, GW, Event) or a SequenceFlow. X X
token. currentFlowElementId the id of the element where the token current is. This can be a FlowNode (Activity, GW, Event) or a SequenceFlow. X X
token. currentFlowNodeIsExternal At design time, an activity can be configured in the BPMN Editor to be executed externally, i.e. it contains no execution instructions. Instead it is executed by an external program which know how to execute the activity. This is down via the REST API (see below). The token is not automatically moved by the process engine, only by calls to the REST API. X
token. currentFlowNodeState Indicates the state of the currently executed FlowNode, on which the token currently resides. Attention: this is about a FlowNode (Activity, Gateway, or Event) not about a FlowElement (e.g. not a Sequence Flow).
Status: Currently, only a simple state diagram for Activities is implemented with the following states: READY, ACTIVE, EXTERNAL, COMPLETED, FAILED, TERMINATED. See the BPMN Spec. 13.3.2 for most state explanations.
"READY": an activity is waiting in state READY if it has the external attribute set to true in the process model.
"EXTERNAL" means that the activity is executed by an external program (see API).		 X X
token. currentFlowElementStartTime the time when the currently executed FlowElement was started X X
token. localStartTime time and date of the token start (on the local Engine). Is not always the same with globalStartTime because a token can be transferred during execution to other Machines. So, multiple parallel tokens of a process can have different localStartTimes. If multiple tokens are merged at a GW, the earliest start time of all merged tokens is taken. X X
token. localExecutionTime relative time in milli-seconds a token has already needed for executing the BPMN elements on the local Machine. Every finished FlowNode adds the needed seconds to this value (That means, no Sequence Flow or Decider time is added). If multiple tokens are merged at a GW, the longest time of all merged tokens is taken. X X
token. currentFlowNodeProgress Indicates the current progress of a FlowNode (currently only Script Task Activities are supported). X
token. milestones Indicates the current progress of multiple milestones. A UserTask and ScriptTask can set Milestones with a <milestone-id> and the progress between 0 and 100. E.g. "M1": 87. The milestones object is written to the log and deleted from the token after the FlowNode has completed and the token moves on. X
token. actualOwner Contains the ids of the users who worked on a user task X
token. intermediateVariablesState Contains the changes of process variables made during the execution of the current FlowNode. They are usually not immediately written to the global variables (unless you do it on purpose within the FlowNode). When the FlowNode is successfully completed, the variables will be written to the instances variables. X
token. costsRealSetByOwner The actual costs of the current activity, which are set during execution X
token. deciderStorageRounds The number of Decider rounds a Token already waits for deployment X
token. deciderStorageTime The milliseconds (number) of Storage Time inside the re-evaluation storage a Token already waits for deployment. This time is not completely correct, since it gets only updated once out of the re-evaluation storage and, again, in the decider for another round. X
token.machineHops The number of Machines a token was already on (hopping) X X
token.nextMachine Infos where the token was transmitted to X X

Rest Endpoints: Interacting with processes instances

Process Artefact REST Method REST URL Body/Return Value Notes
Get ALL Process Instances1 GET /process/{definitions-Id}

---> /instance [?state=<processState>] 		Returns: a list of all instances also includes "old" instances (ENDED, FORWARDED or STOPPED)
New Process Instance POST /process/{definitions-Id}/versions/{version-id}

---> /instance 		Return: sends an Instance-ID (and URL with 201) back POST starts a new process instance of a specific process version, the request body can contain all variable values
Continue Instance on another Engine1 PUT /process/{definitions-Id}

---> /instance/{instanceId} 		Request Body: contains the complete instance info, but only with the forwarded token Forwards and starts an instance on another machine (the token is not deleted on the machine it is coming from, just put into the FORWARDED state)
Get Instance / Tokens1 GET /process/{definitions-Id}

---> /instance/{instanceId} 		Gets the current process status and all tokens
Set Instance Status1 PUT /process/{definitions-Id}

---> /instance/{instanceId}/ instanceState 		Body: { instanceState: "<newState>" }, where "<newState>" can be "paused", "resume", "stopped" or "aborted" Sets the state of the whole process instance. It can pause (with paused), resume (with resume), stop (with stopped) or terminate (with aborted) the instance execution. It changes the instanceState to PAUSING/PAUSED, RUNNING or STOPPED and the tokens state to ABORTED
Create new Token1 POST /process/{definitions-Id} /instance/{instanceId} /tokens Body: { currentFlowElementId } Creates a new Token on the specified Instance at the specified currentFlowElementId
Update Token: position on specific FlowElement1 PUT /process/{definitions-Id} /instance/{instanceId} /tokens/{tokenId} Body: { currentFlowElementId: "<flowElementId>" }

returns an Error if "<flowElementId>" does not exist in the process		 This modifies the place of a token. The token can be placed 1. on a Sequence Flow (that results in giving it to the decider) or 2. on any FlowNode of the process. In either case, it interrupts the execution of the currently executed FlowElement.
If the token execution is continued on another FlowElement, then the state SKIPPED is written to the log for the previous FlowNode.
Delete a Token1 DELETE /process/{definitions-Id} /instance/{instanceId} /tokens/{tokenId} Deletes the specified Token from the Instance. This functionality is indicated in the flowNode log with the stopped: true property.
Update instance variables1 POST /process/{definitions-Id} /instance/{instanceId} /variables Body: { <variable-name>: <variable-value>, ... } Changes the variables of an instance according to the changes proposed in the request body
Migrate instances from one process version to another POST /process/{definitions-Id} /versions/{source-version-id} /instance/migration Body:
{
"targetVersion": <number>,
"instanceIds": <uuid>[],
"tokenMapping": {
"add": [{ "targetFlowElementId": <element-id>, ...}, ...],
"move": [{ "tokenId": <token-id>, "targetFlowElementId": <element-id>}, ...],
"remove": [<token-id>, ...]
},
"flowElementMapping": { <source-element-id>: [<target-elemet-id>, ...], ...}
} 		Will migrate the given instances from the one version of the process to another

tokenMapping: (Only when migrating a single instance) Defines the token changes that should be made while migrating the instance

flowElementMapping: Defines which elements should be activated after a migration if a specific element is active before the migration
Set FlowNode/ Activity Status for external FlowNodes1 PUT /process/{definitions-Id} /instance/{instanceId} /tokens/{tokenId} /currentFlowNodeState Body: { currentFlowNodeState: "<cFNState>", [variables], [boundaryEventReference] } where "<cFNState>" is either 'EXTERNAL', 'EXTERNAL-COMPLETED', 'EXTERNAL-TERMINATED', or 'EXTERNAL-FAILED'


returns an Error if try to set to state EXTERNAL but external attribute is not enabled for the current FlowNode

returns an Error if try to set other states but currentFlowNodeState is != EXTERNAL 		Sets the currentFlowNodeState of the currently executed FlowNode to EXTERNAL, meaning the activity is executed somewhere else. This can only be set for Flow Nodes (currently only Activities are supported) waiting in state READY (i.e., the external attribute is set to true in the process model).

Afterwards, the external application can set the state a second time to "COMPLETED" (normal end, with 'EXTERNAL-COMPLETED'), "FAILED" (interrupting error event, with 'EXTERNAL-FAILED'), or "TERMINATED" (other interrupting event, with 'EXTERNAL-TERMINATED') which usually ends the flowNode execution to go on. In this case, the property external with true is written to the log.

Process data in form of variables can be added from the external application.

The "FAILED" and "TERMINATED" state triggers a new token on boundary events. It is possible that there are multiple boundary events on one activity. Therefore, boundaryEventReference can select which one to use, see here.

Exception: "EXTERNAL-TERMINATED" can also trigger non-interrupting boundary events. In this case a new token is spawn and the currentFlowNodeState stays in "EXTERNAL" state.

1: Usually a specific process (referenced as {definitions-Id} in the endpoint path) can have multiple versions. So, many REST endpoints have .../versions/{version-id}/... in their path. The instance endpoints don't have that. Instead the request/response-body usually contains a reference to the specific process-version, which can be used for getting the correct process version. Reason for not having the version-id in the instance endpoint: Migrating one running process instance to a new process version would change the REST endpoint for monitoring the execution (e.g. for monitoring one instance you would get /process/{p-id}/version/{v-id}/instance/{i-id} -> after migration the process version (v-id) changes, so you would need to request another path)

⚠️ **GitHub.com Fallback** ⚠️