Multiple Kafka Clusters (CORS Method) - linkedin/cruise-control-ui GitHub Wiki

WARNING WARNING : Do not deploy in this mode without understanding security consequences. WARNING WARNING

Pre-Read: https://github.com/linkedin/cruise-control-ui/wiki/CORS-Method

This tutorial is helpful for users who are running multiple kafka clusters and would like to use Cruise Control UI (CCFE) in CORS Method.

Goal

By the end of this exercise you will have single CCFE managing all your kafka clusters. Requests to CC are made directly by the web-browser thats running CCFE using CORS.

This is how the data flow looks like in this setup

                                                                     +-----------------------+
                                +-------------+                      |                       |
     BROWSER                    |             |       +------------->+ cc-1 (10.0.0.1:9090)  |
+----------------+  GET /       |             |       |              |                       |
|                |              |  nginx (or) |       |              +-----------------------+
|   User / CCFE  +<------------>+  Apache     |       |
|                |   CCFE FILES |             |       |              +-----------------------+
+-------+--------+              |  10.0.0.4   |       |              |cc-2 (10.0.0.2:9090)   |
        ^                       |  port 80    |       +------------->+                       |
        |                       +-------------+       |              +-----------------------+
        |                                             |
        +-------------->------------>---------------->+              +-----------------------+
      {GET,POST}                                      |              |                       |
      http://$ip:$port/kafkacruisecontrol             +------------->+ cc-3 (10.0.0.3:9090)  |
      CC REST API                                                    |                       |
                                                                     +-----------------------+

Assumptions

  1. Zookeeper(s) is/are up and running.
  2. Three Kafka Clusters (1 or more) are setup and running.
  3. Three Cruise Control (1 or more) instances are setup and running with CORS Method.
    1. CC-1 Running on ip 10.0.0.1 & port 9090 with hostname cc-1.example.com
    2. CC-2 Running on ip 10.0.0.2 & port 9090 with hostname cc-2.example.com
    3. CC-3 Running on ip 10.0.0.3 & port 9090 with hostname cc-3.example.com
  4. Cruise Control Servers are exposing their REST API on URL Path /kafkacruisecontrol/*

Requirements

You need a webserver like Apache, Nginx which supports serving static content.

  1. Lets say, this webserver is running on ip 10.0.0.4 & port 80 with hostname ccfe.example.com
  2. Document root for the webserver is configured at /var/www/html/
  3. Webserver (10.0.0.4:80) is accessible from selective users (or ip address, subnets) from web-browser.
  4. CC-1,CC-2,CC-3 port 9090 are accessible for selective users (or ip address, subnets) from web-browser.

In general, please deny everything and allow only the traffic thats accepted.

Configure Webserver

All the common steps are shown below. Wherever there are dedicated steps required for the targeted webserver, they are highlighted as needed.

CCFE with Nginx (or) Apache

  1. Download the latest compiled artifacts from https://github.com/linkedin/cruise-control-ui/releases page

This will have the following files once extracted.

/home/user/Downloads $ tar zxvf cruise-control-ui.tar.gz 
cruise-control-ui/
cruise-control-ui/dist/
cruise-control-ui/README.txt
cruise-control-ui/dist/index.html
cruise-control-ui/dist/static/
cruise-control-ui/dist/static/cc-logo.png
cruise-control-ui/dist/static/css/
cruise-control-ui/dist/static/js/
cruise-control-ui/dist/static/config.csv
cruise-control-ui/dist/static/js/manifest.js
cruise-control-ui/dist/static/js/vendor.js
cruise-control-ui/dist/static/js/app.js
cruise-control-ui/dist/static/css/app.css
cruise-control-ui/dist/static/css/app.css.map
  1. Copy the cruise-control-ui.tar.gz to the webserver.
scp cruise-control-ui.tar.gz [email protected]:/tmp/
  1. Extract the cruise-control-ui.tar.gz inside the server root folder.
ssh [email protected]
cd /var/www/html/
sudo tar zxvf /tmp/cruise-control-ui.tar.gz
  1. Update the config.csv so that webserver can reach the Cruise Control Server(s)
ssh [email protected]
cd /var/www/html/
mv /var/www/html/cruise-control-ui/dist/* /var/www/html/
rmdir /var/www/html/cruise-control-ui/dist
rmdir /var/www/html/cruise-control-ui

Give Absolute URLs of CC REST API end points here so that browser (CCFE) will directly make requests to CC.

cat /var/www/html/
cat config.csv
region-1,cc-one,http://cc-1.example.com:9090/kafkacruisecontrol
region-1,cc-two,http://cc-2.example.com:9090/kafkacruisecontrol
region-1,cc-three,http://cc-3.example.com:9090/kafkacruisecontrol
  1. Webserver specific configuration

Nginx: no special url routing configuration required here. Apache: no special url routing configuration required here.

  1. Hit the Cruise Control host & port in the browser to access CCFE

http://cc-host.example.com/

This will show UI with 3 drop down links in the top navigation bar. Select the cluster that you want to operate next.

Security

Its very important to follow this setup only on environments where Access to Cruise Control is given to authorized clients.