Appendix04.DataPipeline copy - haymant/trading GitHub Wiki
Data pipelines serves the purposes of data enrichment, transformation, analysis and input/output very well. Typical data pipeline solution such as Spring Cloud Data Flow and Flink leverage underlying database and message queues for data storage and communciation. For monolithic architecture, we may rely on Guava EventBus and implement simple pipeline APIs.
βββββββββββββββββββββββββ¬βββββββββββββββ¬βββββββββββββ
β β β β
β β β Dashboard β
β Web UI β Shell β Grafana β
β β β β
βββββββββββββββββββββββββΌβββββββββββββββΌββββ¬βββββββββ€
β Dataflow Srv β Skipper β β β
β β βββββΊ β
βββββββββββββββββββββββββ΄βββββββββββββββ€ β β
β SCDF A stream β βtime se-β
β βββββββ βββββββ βββββββ β βries DB β
β β Src β βProc β ... βSink β β β β
β β β β β βββββββββ β β β β β
β βββββββ€ βββββββ€ βββββββ€ βββββΊInfluxDBβ
βββ΄βββββΌβ΄βββ΄βββ²ββΌβ΄βββββββββββββ΄ββ²ββββ΄βββ βPromet..β
β β β β MetricsβWavefro.β
βββββ ββΌβββββββ΄ββΌββββββββββββββββ΄ββββ β β
βDB β β Message Queue (rabbit/kafka)β β β
βββββ βββββββββββββββββββββββββββββββ ββββββββββRefer to Appendix0 for more details.
$ bin/kafka-topics.sh --describe --topic quickstart-events --bootstrap-server 192.168.1.100:9092Refer to Appendix0 for more details.
$ psql -h 192.168.1.100 -U postgres
postgres=# CREATE DATABASE dataflow;