Explore Services - TheLadders/pipeline GitHub Wiki

Test from Inside the Docker Container

Kafka Native

root@docker$ kafka-topics --zookeeper 127.0.0.1:2181 --list
_schemas
ratings

kafka-console-consumer --topic ratings --zookeeper localhost

Spark Submit

root@docker$ spark-submit --class org.apache.spark.examples.SparkPi --master spark://127.0.0.1:7077 $SPARK_EXAMPLES_JAR 10

Cassandra

root@docker$ cqlsh
cqlsh> use pipeline;

cqlsh:pipeline> select fromuserid, touserid, rating, batchtime from real_time_ratings;

 fromuserid | touserid | batchtime | ratings
------------+----------+-----------+---------

(0 rows)

cqlsh> describe pipeline;
...

cqlsh:pipeline> exit;

ZooKeeper

root@docker$ zookeeper-shell 127.0.0.1:2181

Connecting to 127.0.0.1:2181
Welcome to ZooKeeper!
JLine support is disabled

WATCHER::

WatchedEvent state:SyncConnected type:None path:null

MySQL

root@docker$ mysql -u root -p 
Enter password:  password

Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 47
Server version: 5.5.44-0ubuntu0.14.04.1 (Ubuntu)

Copyright (c) 2000, 2015, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql>

JDBC ODBC Hive ThriftServer

Run the following to test with Beeline

root@docker$ beeline -u jdbc:hive2://127.0.0.1:10000 -n hiveuser -p ''
0: jdbc:hive2://127.0.0.1:10000> SELECT id, gender FROM gender_json_file LIMIT 10;
+-----+---------+
| id  | gender  |
+-----+---------+
| 1   | F       |
| 2   | F       |
| 3   | U       |
| 4   | F       |
| 5   | F       |
| 6   | F       |
| 7   | F       |
| 8   | M       |
| 9   | M       |
| 10  | M       |
+-----+---------+

Test from Outside boot2docker and the Docker Container

Launch a new local macosx or windows terminal
Run the commands below - in the new terminal - to verify your setup
The IP of your Docker Container can be found here:

macosx-laptop$ docker-machine ip pipelinebythebay

On Linux just use 'localhost', no need to install and use docker-machine.

Kafka REST API Proxy

open opens a browser on a Mac

macosx-laptop$ open http://<ip-from-above>:34042/topics

Apache Spark Master Admin Web UI

macosx-laptop$ open http://<ip-from-above>:36060

Apache Spark Worker Admin Web UI

macosx-laptop$ open http://<ip-from-above>:36061

Spark Notebook

macosx-laptop$ open http://<ip-from-above>:39000