Context

Pepper-box's README provides the context for this wiki page i.e. please follow the instructions there to build and add pepper-box to jmeter.

Notes & Advice

I built the code for jmeter version 3.2 (which uses log4j) and kafka 0.10.2.1 (as I experienced runtime issues when I tried building for 0.11). The key command is simply: mvn clean package as this uses the default values specified in the pom.xml file that match these versions. It also runs the tests which should all pass. If not, it's worth investigating why not and fixing the causes.

I hope we'll be able to build, deploy and run pepper-box for recent and current releases of both jmeter and kafka, for now if you use the same settings I did you may avoid some problems. Once you have a working environment then it's easier and more fruitful to explore using newer releases. Check the kafka Apache release notes for and (in-)compatibility issues.

Summary of steps

• Download and unpack apache jmeter 3.2 (if necessary) tar xvf apache-jmeter3.2.tar
• Build pepper-box using the instructions in the README and copy pepper-box-1.0.jar to ./lib/ext/ of where you've unpacked jmeter e.g. ~/apache-jmeter3.2/
• assuming you're using a terminal window (as you're likely to do to run a full test), add jmeter to the path e.g. export PATH=``pwd``/bin:$PATH (replace the double back-ticks around pwd with single back-ticks, they're a workaround so the command is displayed here.)
• Start jmeter and create a test script that incorporates a Pepper Box Config and a Java Sampler that uses the pepper-box Sampler (using the instructions in the README)
• Configure both. Details of what parameters to include for SSL_SASL follow this section
• Save the configuration file.
• Tip: start a consumer that's correctly configured to use SSL_SASL (for now that's outside the scope of this wiki page). This will enable you to see the messages produced by pepper-box as they arrive.
• Run the test in jmeter. Tip: consider setting the log level to DEBUG in jmeter especially if things don't seem to be working. The log can be viewed in jmeter, or by reading the log file that's being created, typically called jmeter.log in the same folder that jmeter was started in.

Key settings to use SASL_SSL

Set security.protocol to SASL_SSL this is essential both for Kafka and to cause the Sampler to then add the rest of the parameters to the properties that will be passed to Kafka.

These properties are:

• sasl.jaas.config which needs to have the correct username and password that your Kafka Brokers are configured to accept. The format of the value is org.apache.kafka.security.plain.PlainLoginModule required username=\"alice\" password=\"secret\"; Replace alice with the correct user name and secret with the password for that user/account.
• sasl.mechanism needs to be set to PLAIN (at least that's the only way I've tested the new capability, other settings may be appropriate for other security configurations for Kafka).

The rest of the parameters are to configure SSL:

• ssl.enabled.protocols needs to match the protocols enabled on the Kafka Brokers. Here we set it to TLSv1.2
• ssl.truststore.location needs to specify the file that contains additional security details. You may well have this file already if you use the command-line kafka utilities such as kafka-console-producer, if so you can probably reference that file here.
• ssl.truststore.password should be set to the password for the truststore if you're using one. Ours does. I've not tested with a truststore that isn't password protected - it's not a recommended practice anyway.
• ssl.truststore.type needs to be configured appropriately. Here we set it to JKS which represents a Java KeyStore.

Notes: if you're not using SASL_SSL

The parameters and default values are included in the PepperBox Sampler, however they're only used if security.protocol is set to SASL_SSL. AFAIK There is limited scope to optionally add parameters automatically depending on what else is configured, I think is a feature or limitation of the current way the Sampler is implemented. Perhaps it'd be worth developing a better GUI or creating a set of Samplers? (similar to the way pepper-box has two Configuration objects).

Deploying jmeter, the test and pepper-box to load generators

We're likely to create and test our test plan locally. However this may not be the best place to run the actual tests. Often these are run from a dedicated node (or set of nodes) that are proximate (near) to kafka. These notes will help you deploy the software and test script.

Key ingredients

Here's what's needed, as a minimum, on each node for jmeter to run:

• jmeter: either download jmeter directly to each node or distribute it wget is your friend
• pepper-box-1.0.jar: distribute it to each node. scp or rsync are useful to use
• the text file with additional security settings, the same one referred to by ssl.truststore.location. Note: the location is likely to change on the destination node, if so, edit your test script accordingly. A text-based editor such as vi, vim, etc. and knowing how to search for the current value in that editor will help you get the job done.
• the test script you created. This has a file extension of .jmx

On the target machine (using ssh or a direct console, etc)

hostname      # Check we're on the machine we intend to use
cd            # go to the user's home folder, pick another location if you prefer
wget https://archive.apache.org/dist/jmeter/binaries/apache-jmeter-3.2.tgz
tar xvf apache-jmeter-3.2.tgz
cd apache-jmeter-3.2
ls -l lib/ext # there should be various jar files, not pepper-box yet

In a separate session, transfer the pepper-box jar and the test script, then:

cd ~/apache-jmeter-3.2  # key if you've reconnected at this point
ls -l lib/ext # there should be various jar files, INCLUDING pepper-box
ls -lart      # there should be the jmx test script file (it's a text file and editable)
export PATH=`pwd`/bin:$PATH   # Add the jmeter executable's folder to the PATH
jmeter --nongui --testfile Kafka_DoLittle32.jmx    # NB: Replace the last item with the test script's name