Job Server - TheLadders/pipeline GitHub Wiki

Start the Job Server

Spark Job Server provides a REST service for sharing and managing Spark jobs.

cd ~/pipeline && ./bythebay-jobserver.sh

This uploads two jars for you to test the job server with.

To validate the script worked:

• From within the Docker container: curl localhost:8099/jars and you should see two JSON elements, one for "test" and one for "streaming".
• From outside Docker: use port 38099 instead, and 'localhost' from the Ubuntu VM or the VM ip address from OSX/Windows etc.

Run a job

curl -d "input.string = a b c a b see" 'localhost:8099/jobs?appName=test&classPath=spark.jobserver.WordCountExample&sync=true'

You should see output like this:

{
  "status": "OK",
  "result": {
    "b": 2,
    "a": 2,
    "see": 1,
    "c": 1
  }
}⏎