Using Kafka CLI Helper Scripts - theryanwatson/confluent-kafka-scripts GitHub Wiki

After following Setup-Kafka-CLI-Helper-Scripts, you can use the scripts from any directory to access any Confluent Platform Kafka/Schema Registry setup.

Commonly-used Confluent CLI Scripts have been wrapped to make them easier to use, by allowing you to specify an environment and have all of the commonly-used settings included for you. For instance, the *-consumer and *-producer scripts assume that the topic key is a string and is configured to accept them for producers and print them for consumers. For the *-avro-* scripts, the Schema Registry path is parsed from the ~/.ccloud/{{ env }}.config file and the latest schema id is retrieved.

Common Properties Already Configured

Instead of manually looking up the schema latest id, and then calling:

$ kafka-avro-console-producer --broker-list abc-1xxxx.us-east-1.aws.confluent.cloud:9092 --producer.config ~/.ccloud/qa.config --property schema.registry.url=http://schema-registry.my-company.com --property value.schema.id=1 --property key.serializer=org.apache.kafka.common.serialization.StringSerializer --property key.schema= --property parse.key=true --topic my-avro-topic

You could call:

$ k-avro-console-producer qa --topic my-avro-topic

Usage

The wrapper scripts are used by calling with the environment variable and then using the normal configuration properties normally used with the kafka-* scripts. k-topics local --list, k-topics prod --list, or k-consumer-groups qa --describe --all-groups

Some use-case examples:

Console Consumer Use-Cases

  1. View all messages coming from a local topic or Local Dev Host topic:
    • k-console-consumer local --topic event-my-new-topic
    • k-console-consumer my-host.my-company.com --topic event-my-new-topic
    • k-avro-console-consumer local --topic my-new-avro-topic
    • k-avro-console-consumer dev-host.my-company.com --topic my-new-avro-topic
  2. PPV a new message value field from a QA topic:
    • k-console-consumer qa --topic event-call-detail-update | grep -Eo '"source_customer_id":"[^"]+"'
    • k-avro-console-consumer qa --topic event-call-data-adapter | grep -Eo '"caller_info_email":"[^"]+"'
  3. Get a sampling of messages from a Production Topic:
    • k-console-consumer prod --topic event-call-end --from-beginning --max-messages 10
    • k-avro-console-consumer prod --topic event-call-transcript --from-beginning --max-messages 2
  4. Print the value a bad message from a production topic:
    • k-avro-console-consumer prod --topic event-call-data-adapter --partition 0 --offset 20377 --max-messages 1
    • Or, in the case of an Avro encoding issue, use the standard consumer the same way: k-console-consumer prod --topic event-call-data-adapter --partition 0 --offset 20377 --max-messages 1

Console Producer Use-Cases

  1. Add new messages to a local topic or Dev Host topic:
    • k-console-producer local --topic event-my-new-topic
    • k-console-consumer dev-host.my-company.com --topic event-my-new-topic
  2. Copy a schema from QA to local schema registry and add messages to a local topic:
    • k-copy-schema --topic my-avro-topic && k-avro-console-producer local --topic my-avro-topic
    • k-copy-schema --destination dev-host.my-company.com --topic my-avro-topic && k-avro-console-producer dev-host.my-company.com --topic my-avro-topic