Queues - reanahub/reana GitHub Wiki

Contents

1. About queues
2. jobs-status queues

About queues

REANA is using a queue system to pass messages between components.

Overview of the queues in REANA:

Name	Description
jobs-status	contains messages that update status and logs of running workflow
workflow-submission	WIP

jobs-status queue

• publishers: workflow engines like Yadage, CWL, Serial or Snakemake
• consumers: single jobs status consumer

JSON schema of the message (WIP):

{
  "$id": "reana/jobs-status-message.schema.json",
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "title": "jobs-status message",
  "description": "Describes jobs-status queue message",
  "type": "object",
  "properties": {
    "workflow_uuid": {
      "description": "The unique UUID identifier for a workflow",
      "type": "string"
    },
    "status": {
      "description": "Status of a workflow (1 - running, 2 - finished, 3 - failed)",
      "type": "integer",
      "enum": [1, 2, 3]
    },
    "logs": {
      "description": "Workflow logs",
      "type": "string"
    },
    "message": {
      "type": "object",
      "properties": {
        "progress": {
          "engine_specific": {
            "type": "string"
          },
          "failed": {
            "$ref": "reana/jobs-status-message-progress-status.schema.json"
          },
          "total": {
            "$ref": "reana/jobs-status-message-progress-status.schema.json"
          },
          "running": {
            "$ref": "reana/jobs-status-message-progress-status.schema.json"
          },
          "finished": {
            "$ref": "reana/jobs-status-message-progress-status.schema.json"
          }
        }
      }
    }
  },
  "required": ["workflow_uuid", "status"]
}

{
  "$id": "reana/jobs-status-message-progress-status.schema.json",
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "title": "WIP",
  "description": "WIP",
  "required": ["total", "job_ids"],
  "type": "object",
  "properties": {
    "total": {
      "type": "integer"
    },
    "job_ids": {
      "type": "array",
      "items": {
        "type": "string"
      }
    }
  }
}