Misc - nicolasramy/tada-engine GitHub Wiki

  • Use a job queue?
    • Beanstalk
    • Redis
    • In Memory?
  • Internal compression for data exchange (snappy)
  • use LevelDB
  • use snappy
  • use SQLite
  • web interface
    • dashboard
    • monitoring
    • create / edit workflow
  • workflow
    • linear
    • unordered
    • graph
  • input formats
    • JSONLines .jsonl
    • JSON .json
    • CSV .csv
    • TSV .tsv
    • Text file .txt
  • input / output sources
    • file
    • http
    • stream
    • db
    • aws s3
    • HDFS, etc ...
    • ...
  • open an endpoint for monitoring
  • open an endpoint for task submit