Config proposal.md - noetl/noetl GitHub Wiki

  1. The configuration file must have a root key "workflow". One configuration file is equal to one Workflow. Only one workflow must be defined in one configuration file. As an analogy, we can refer to Java classes where each class has to be defined in the single separate file.
  2. Any variables might be defined inside a named context in the workflow level of the configuration file.
  3. The workflow contains a list of tasks. Tasks are executed in parallel.
  4. Each task has a list of steps to be executed. Steps are executed in sequential order. A step is a reference to the actual module has a set of parameters and may have reference to the context variables.
  5. Both task and step can have a loop or range. The values extracted from the loop list or generated list of range can be applied to the body of the task or module and executed in parallel by default.
  6. Etcd distributed key-value storage is used as a database for configuration files.

Config example: sample-demo.yaml

workflow:
  version: "1"
  id: "sample-demo" 
  description: "Retrieves BTC/USD exchange rate from 3 different exchanges, calculates an average and puts the latter into db."
  context:
    aws:
      access-key: "sdFASDFA"
      secret-key: "ASDASDFASDF"
  tasks: 
    retrive-btc-usd:
      description: "Retrieves BTC/USD exchange rate from 3 different exchanges"
      steps: 
        - s3:
            action: "directory"
            path: "s3a://datalake/{{ loop.value | strip(\"https://\") | \"/\" }}"
            validate: "empty"
          context: "aws"
        - rest:
            action: "download" 
            url: "{{ .value }}"
            path: "s3a://datalake/{{ loop.value | strip(\"https://\") | \"/\" }}"
          context: "aws"
    loop: 
      - "https://cex.io/api/last_price/BTC/USD"
      - "https://api.bitfinex.com/v1/pubticker/btcusd"
      - "https://api.hitbtc.com/api/2/public/ticker/BTCUSD"

    aggragate-btc-usd:
      description: "Calculate BTC/USD average and display result"
      steps: 
        - aggregate:
            action: "average"
            sources: 
              - "s3a://datalake/cex.io/api/last_price/BTC/USD/"
              - "s3a://datalake/api.bitfinex.com/v1/pubticker/btcusd/"
              - "s3a://datalake/api.hitbtc.com/api/2/public/ticker/BTCUSD/"
          context: "aws"
      require: 
        - "retrive-btc-usd"