Minio Kafka Bucket Notifications (no TLS) - allanrogerr/public GitHub Wiki
https://kafka.apache.org/documentation/#quickstart
Run on kafka-demo vm-broker instance
ssh -p 20058 [email protected] -o "ServerAliveInterval=5" -o "ServerAliveCountMax=100000" -o "StrictHostKeyChecking=off"
Get Kafka
wget https://dlcdn.apache.org/kafka/3.8.1/kafka_2.12-3.8.1.tgz
tar -xzf kafka_2.12-3.8.1.tgz
cd kafka_2.12-3.8.1
Get Java
sudo apt update
sudo apt install default-jre
java -version
Output
openjdk version "21.0.4" 2024-07-16
OpenJDK Runtime Environment (build 21.0.4+7-Ubuntu-1ubuntu224.04)
OpenJDK 64-Bit Server VM (build 21.0.4+7-Ubuntu-1ubuntu224.04, mixed mode, sharing)
Start the Kafka Environment
Kafka with ZooKeeper
bin/zookeeper-server-start.sh config/zookeeper.properties
New terminal
ssh -p 20058 [email protected] -o "ServerAliveInterval=5" -o "ServerAliveCountMax=100000" -o "StrictHostKeyChecking=off"
cd kafka_2.12-3.8.1
bin/kafka-server-start.sh config/server.properties
Create a topic to store your events
New terminal. Note that port 9092 was part of the output of the previous command
ssh -p 20058 [email protected] -o "ServerAliveInterval=5" -o "ServerAliveCountMax=100000" -o "StrictHostKeyChecking=off"
cd kafka_2.12-3.8.1
bin/kafka-topics.sh --create --topic quickstart-events --bootstrap-server localhost:9092
Output
Created topic quickstart-events.
Describe topic
bin/kafka-topics.sh --describe --topic quickstart-events --bootstrap-server localhost:9092
Write Some Events Into The Topic
bin/kafka-console-producer.sh --topic quickstart-events --bootstrap-server localhost:9092
Input the following examples after the > prompt:
First event
Second event
Final event
Hit Ctrl-C
Read The Events
bin/kafka-console-consumer.sh --topic quickstart-events --from-beginning --bootstrap-server localhost:9092
Output
First event
Second event
Final event
Hit Ctrl-C
Import/export Your Data As Streams Of Events With Kafka Connect
echo "plugin.path=libs/connect-file-3.8.1.jar" >> config/connect-standalone.properties
Create seed data
echo -e "foo\nbar" > test.txt
Start two connectors
bin/connect-standalone.sh config/connect-standalone.properties \
config/connect-file-source.properties \
config/connect-file-sink.properties
Verify data delivered New terminal
ssh -p 20058 [email protected] -o "ServerAliveInterval=5" -o "ServerAliveCountMax=100000" -o "StrictHostKeyChecking=off"
cd kafka_2.12-3.8.1
more test.sink.txt
Output
foo
bar
Use consumer to check data in topic
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic connect-test --from-beginning
Output
{"schema":{"type":"string","optional":false},"payload":"foo"}
{"schema":{"type":"string","optional":false},"payload":"bar"}
Processed a total of 2 messages
Observe delivery
New terminal
ssh -p 20058 [email protected] -o "ServerAliveInterval=5" -o "ServerAliveCountMax=100000" -o "StrictHostKeyChecking=off"
cd kafka_2.12-3.8.1
echo Another line>> test.txt
Observe output in consumer (previous terminal)
{"schema":{"type":"string","optional":false},"payload":"foo"}
{"schema":{"type":"string","optional":false},"payload":"bar"}
{"schema":{"type":"string","optional":false},"payload":"Another line"}
Stop Kafka processes in all terminals with Hit Ctrl-C
Minio integration
Start kafka with ZooKeeper
Terminal 1
ssh -p 20058 [email protected] -o "ServerAliveInterval=5" -o "ServerAliveCountMax=100000" -o "StrictHostKeyChecking=off"
cd ~/kafka_2.12-3.8.1 && bin/zookeeper-server-start.sh config/zookeeper.properties
Terminal 2
ssh -p 20058 [email protected] -o "ServerAliveInterval=5" -o "ServerAliveCountMax=100000" -o "StrictHostKeyChecking=off"
cd ~/kafka_2.12-3.8.1 && bin/kafka-server-start.sh config/server.properties
Create a topic Terminal 3
ssh -p 20058 [email protected] -o "ServerAliveInterval=5" -o "ServerAliveCountMax=100000" -o "StrictHostKeyChecking=off"
cd ~/kafka_2.12-3.8.1 && bin/kafka-topics.sh --create --topic minio-events --bootstrap-server localhost:9092
Output
Created topic minio-events.
ssh -p 20058 [email protected] -o "ServerAliveInterval=5" -o "ServerAliveCountMax=100000" -o "StrictHostKeyChecking=off"
cd ~/kafka_2.12-3.8.1 && bin/kafka-topics.sh --create --topic minio-audit --bootstrap-server localhost:9092
Output
Created topic minio-events.
Start consuming events
bin/kafka-console-consumer.sh --topic minio-events --from-beginning --bootstrap-server localhost:9092
Minio Integration
Terminal 3
ssh -p 20058 [email protected] -o "ServerAliveInterval=5" -o "ServerAliveCountMax=100000" -o "StrictHostKeyChecking=off"
Install and run minio
wget https://dl.min.io/server/minio/release/linux-amd64/minio
chmod +x minio
CI=on ./minio server /tmp/data --address :10000 --console-address :10090
Install mc and configure kafka
Terminal 4
ssh -p 20058 [email protected] -o "ServerAliveInterval=5" -o "ServerAliveCountMax=100000" -o "StrictHostKeyChecking=off"
mkdir ~/mc && cd ~/mc && wget https://dl.min.io/client/mc/release/linux-amd64/mc
chmod +x mc
./mc alias set minio-kafka http://127.0.0.1:10000 minioadmin minioadmin
Add the Kafka Endpoint to MinIO
./mc admin config set minio-kafka/ notify_kafka:PRIMARY \
brokers="127.0.0.1:9092" \
topic="minio-events" \
queue_dir="/tmp/minio/events" \
queue_limit="10" \
comment="initial"
Obtain ARN from minio startup or from ./mc admin info --json minio-kafka
./mc admin info --json minio-kafka | grep sqsARN
"sqsARN":["arn:minio:sqs::PRIMARY:kafka"]
Create bucket
./mc mb minio-kafka/kafka-demo-no-tls
Configure bucket notifications
./mc event add minio-kafka/kafka-demo-no-tls arn:minio:sqs::PRIMARY:kafka --event "put,delete,get,replica,ilm,scanner"
Output
Successfully added arn:minio:sqs::PRIMARY:kafka
OR in UI
Configure audit
./mc admin config set minio-kafka audit_kafka:KAFKA1 \
enable=on \
brokers=localhost:9092 \
topic=minio-audit \
queue_size=1000 \
queue_dir=/tmp/audit_queue
./mc admin config get minio-kafka audit_kafka
Modify bucket and observe bucket events e.g.
ubuntu@kafka-no-tls:~/kafka_2.12-3.8.1$ bin/kafka-console-consumer.sh --topic minio-events --from-beginning --bootstrap-server localhost:9092
{"EventName":"s3:ObjectAccessed:Head","Key":"kafka-demo-no-tls/text1","Records":[{"eventVersion":"2.0","eventSource":"minio:s3","awsRegion":"","eventTime":"2024-10-30T11:12:04.441Z","eventName":"s3:ObjectAccessed:Head","userIdentity":{"principalId":"minioadmin"},"requestParameters":{"principalId":"minioadmin","region":"","sourceIPAddress":"98.45.6.62"},"responseElements":{"content-length":"4","x-amz-id-2":"4583eb6562a9e7815e67b696a120c09f09901d3770caef92bcabadf9f989ca5c","x-amz-request-id":"1803369BC1D07053","x-minio-deployment-id":"21d6dc19-49f6-45d3-b9e1-66e7a5f15b3f","x-minio-origin-endpoint":"http://10.214.226.245:10000"},"s3":{"s3SchemaVersion":"1.0","configurationId":"Config","bucket":{"name":"kafka-demo-no-tls","ownerIdentity":{"principalId":"minioadmin"},"arn":"arn:aws:s3:::kafka-demo-no-tls"},"object":{"key":"text1","size":4,"eTag":"1cb251ec0d568de6a929b520c4aed8d1","contentType":"application/octet-stream","userMetadata":{"content-type":"application/octet-stream"},"sequencer":"18033660213701B1"}},"source":{"host":"98.45.6.62","port":"","userAgent":"MinIO (linux; amd64) minio-go/v7.0.77 MinIO Console/(dev)"}}]}
{"EventName":"s3:ObjectRemoved:Delete","Key":"kafka-demo-no-tls/text1","Records":[{"eventVersion":"2.0","eventSource":"minio:s3","awsRegion":"","eventTime":"2024-10-30T11:12:08.377Z","eventName":"s3:ObjectRemoved:Delete","userIdentity":{"principalId":"minioadmin"},"requestParameters":{"principalId":"minioadmin","region":"","sourceIPAddress":"98.45.6.62"},"responseElements":{"content-length":"151","x-amz-id-2":"4583eb6562a9e7815e67b696a120c09f09901d3770caef92bcabadf9f989ca5c","x-amz-request-id":"1803369CAC4E281D","x-minio-deployment-id":"21d6dc19-49f6-45d3-b9e1-66e7a5f15b3f","x-minio-origin-endpoint":"http://10.214.226.245:10000"},"s3":{"s3SchemaVersion":"1.0","configurationId":"Config","bucket":{"name":"kafka-demo-no-tls","ownerIdentity":{"principalId":"minioadmin"},"arn":"arn:aws:s3:::kafka-demo-no-tls"},"object":{"key":"text1","sequencer":"1803369CAC82E135"}},"source":{"host":"98.45.6.62","port":"","userAgent":"MinIO (linux; amd64) minio-go/v7.0.77 MinIO Console/(dev)"}}]}
Modify bucket and observe audit events e.g.
ubuntu@kafka-no-tls:~/kafka_2.12-3.8.1$ bin/kafka-console-consumer.sh --topic minio-audit --from-beginning --bootstrap-server localhost:9092
{"EventName":"s3:ObjectCreated:Put","Key":"kafka-demo-no-tls/text1","Records":[{"eventVersion":"2.0","eventSource":"minio:s3","awsRegion":"","eventTime":"2024-10-30T11:07:48.347Z","eventName":"s3:ObjectCreated:Put","userIdentity":{"principalId":"minioadmin"},"requestParameters":{"principalId":"minioadmin","region":"","sourceIPAddress":"98.45.6.62"},"responseElements":{"x-amz-id-2":"4583eb6562a9e7815e67b696a120c09f09901d3770caef92bcabadf9f989ca5c","x-amz-request-id":"18033660211A55DE","x-minio-deployment-id":"21d6dc19-49f6-45d3-b9e1-66e7a5f15b3f","x-minio-origin-endpoint":"http://10.214.226.245:10000"},"s3":{"s3SchemaVersion":"1.0","configurationId":"Config","bucket":{"name":"kafka-demo-no-tls","ownerIdentity":{"principalId":"minioadmin"},"arn":"arn:aws:s3:::kafka-demo-no-tls"},"object":{"key":"text1","size":4,"eTag":"1cb251ec0d568de6a929b520c4aed8d1","contentType":"application/octet-stream","userMetadata":{"content-type":"application/octet-stream"},"sequencer":"18033660213701B1"}},"source":{"host":"98.45.6.62","port":"","userAgent":"MinIO (linux; amd64) minio-go/v7.0.77 MinIO Console/(dev)"}}]}
{"accessKey":"minioadmin","api":{"name":"SetConfigKV","rx":169,"status":"OK","statusCode":200,"timeToResponse":"114262852ns","timeToResponseInNS":"114262852","tx":0},"deploymentid":"21d6dc19-49f6-45d3-b9e1-66e7a5f15b3f","event":"","remotehost":"127.0.0.1","requestHeader":{"Accept-Encoding":"zstd,gzip","Authorization":"AWS4-HMAC-SHA256 Credential=minioadmin/20241030//s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=a1cb38b1e610dbb7f785be639cdbc083a2f35c0f513b5dc7df3bb128c3cb8929","Content-Length":"169","User-Agent":"MinIO (linux; amd64) madmin-go/3.0.70 mc/RELEASE.2024-10-08T09-37-26Z","X-Amz-Content-Sha256":"69c89c7f99a05e22fab7c1c61e5141595fb836c9b8b07d4d1278e3bbb0b4f98e","X-Amz-Date":"20241030T111118Z"},"requestHost":"127.0.0.1:10000","requestID":"18033691160CE252","requestPath":"/minio/admin/v3/set-config-kv","responseHeader":{"Accept-Ranges":"bytes","Content-Length":"0","Server":"MinIO","Strict-Transport-Security":"max-age=31536000; includeSubDomains","Vary":"Origin,Accept-Encoding","X-Amz-Id-2":"4583eb6562a9e7815e67b696a120c09f09901d3770caef92bcabadf9f989ca5c","X-Amz-Request-Id":"18033691160CE252","X-Content-Type-Options":"nosniff","X-Minio-Config-Applied":"true","X-Xss-Protection":"1; mode=block"},"tags":{"GetObject":"name=config/config.json,pool=1,set=1","PutObject":"name=config/history/bfdb8730-3d14-4245-8a9c-08298c03b862.kv,pool=1,set=1"},"time":"2024-10-30T11:11:18.722790228Z","trigger":"incoming","userAgent":"MinIO (linux; amd64) madmin-go/3.0.70 mc/RELEASE.2024-10-08T09-37-26Z","version":"1"}