Local dev setup

The technical stack includes some shared services:

1. Relational DBMS: Postgres
2. Java for Kafka
3. Message queue: Kafka
4. Redis
5. SCDF server
6. Docker/docker-compose: can be used to launch kafka, SCDF, Skipper, Prometheus, Grafna and out of box SCDF apps.

This document give rough steps to setup them on a standalone MAC at 192.168.1.100.

Postgres

$ brew install postgres
$ vim data/pg_hba.conf
# allow remote access, add:
host all all 192.168.0.0/16 md5
# verify port is open
$ telnet 192.168.1.100 5432
# create database for SCDF
$ psql -h 192.168.1.100 -U postgres
postgres=# CREATE DATABASE dataflow;
postgres=# create user scdf with encrypted password 'scdf';
postgres=# grant all privileges on database dataflow to scdf;

Java

$ brew install java
$ export PATH="/usr/loca/opt/openjdk/bin:$PATH" >> ~/.zshrc

Kafka

Please refer to Docker if you want to use docker-compose.

$ brew install kafka
$ zookeeper-server-start /usr/local/etc/kafka/zookeeper.properties
$ vim /usr/local/etc/kafka/server.properties
# add listeners=PLAINTEXT://192.168.1.100:9092
$ kafka-server-start /usr/local/etc/kafka/server.properties
# test from Ubuntu
$ bin/kafka-topics.sh --create --topic quickstart-events --bootstrap-server 192.168.1.100:9092
$ bin/kafka-topics.sh --describe --topic quickstart-events --bootstrap-server 192.168.1.100:9092
Topic: quickstart-events	TopicId: 3ZHe2LVpTRuKekqSUY0oYA	PartitionCount: 1	ReplicationFactor: 1	Configs: segment.bytes=1073741824
	Topic: quickstart-events	Partition: 0	Leader: 0	Replicas: 0	Isr: 0
$ bin/kafka-console-producer.sh --topic quickstart-events --bootstrap-server 192.168.1.100:9092
$ bin/kafka-console-consumer.sh --topic quickstart-events --from-beginning --bootstrap-server 192.168.1.100:9092

Redis

$ brew install redis
$ redis-server --protected-mode no
$ redis-cli -h 192.168.1.100 -p 6379
192.168.1.100:6379> lpush test hello
192.168.1.100:6379> rpop test

Spring Cloud Data Flow

Please refer to Docker if you want to use docker-compose.

$ wget https://repo.spring.io/release/org/springframework/cloud/spring-cloud-dataflow-server/2.9.2/spring-cloud-dataflow-server-2.9.2.jar
$ wget https://repo.spring.io/release/org/springframework/cloud/spring-cloud-dataflow-shell/2.9.2/spring-cloud-dataflow-shell-2.9.2.jar
$ wget https://repo.spring.io/release/org/springframework/cloud/spring-cloud-skipper-server/2.8.2/spring-cloud-skipper

Docker

Follow https://docs.docker.com/desktop/mac/install/ to install docker and docker-compose is included in docker desktop on Mac.

wget -O docker-compose.yml https://raw.githubusercontent.com/spring-cloud/spring-cloud-dataflow/v2.9.1/src/docker-compose/docker-compose.yml; \
DATAFLOW_VERSION=2.9.2 SKIPPER_VERSION=2.8.2 \
docker-compose up