elastic job cloud - TuPengXiong/TuPengXiong.github.io GitHub Wiki

http://mesos.apache.org/documentation/latest/building/

  • Install a recent kernel for full support of process isolation.
$ sudo rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org
$ sudo rpm -Uvh http://www.elrepo.org/elrepo-release-6-6.el6.elrepo.noarch.rpm
$ sudo yum --enablerepo=elrepo-kernel install -y kernel-lt
  • Make the just installed kernel the one booted by default, and reboot.
$ sudo sed -i 's/default=1/default=0/g' /boot/grub/grub.conf
$ sudo reboot
  • Install a few utility tools. This also forces an update of nss,
  • which is necessary for the Java bindings to build properly.
$ sudo yum install -y tar wget git which nss
  • 'Mesos > 0.21.0' requires a C++ compiler with full C++11 support,
  • (e.g. GCC > 4.8) which is available via 'devtoolset-2'.
  • Fetch the Scientific Linux CERN devtoolset repo file.
$ sudo wget -O /etc/yum.repos.d/slc6-devtoolset.repo http://linuxsoft.cern.ch/cern/devtoolset/slc6-devtoolset.repo
  • Import the CERN GPG key.
$ sudo rpm --import http://linuxsoft.cern.ch/cern/centos/7/os/x86_64/RPM-GPG-KEY-cern
  • Fetch the Apache Maven repo file.
$ sudo wget http://repos.fedorapeople.org/repos/dchen/apache-maven/epel-apache-maven.repo -O /etc/yum.repos.d/epel-apache-maven.repo
  • 'Mesos > 0.21.0' requires 'subversion > 1.8' devel package, which is
  • not available in the default repositories.
  • Create a WANdisco SVN repo file to install the correct version:
$ sudo bash -c 'cat > /etc/yum.repos.d/wandisco-svn.repo <<EOF
[WANdiscoSVN]
name=WANdisco SVN Repo 1.8
enabled=1
baseurl=http://opensource.wandisco.com/centos/6/svn-1.8/RPMS/$basearch/
gpgcheck=1
gpgkey=http://opensource.wandisco.com/RPM-GPG-KEY-WANdisco
EOF'
  • Install essential development tools.
$ sudo yum groupinstall -y "Development Tools"
  • Install 'devtoolset-2-toolchain' which includes GCC 4.8.2 and related packages.
$ sudo yum install -y devtoolset-2-toolchain
  • Install other Mesos dependencies.
$ sudo yum install -y apache-maven python-devel python-six python-virtualenv java-1.7.0-openjdk-devel zlib-devel libcurl-devel openssl-devel cyrus-sasl-devel cyrus-sasl-md5 apr-devel subversion-devel apr-util-devel
  • Enter a shell with 'devtoolset-2' enabled.
$ scl enable devtoolset-2 bash
$ g++ --version  # Make sure you've got GCC > 4.8!
  • Process isolation is using cgroups that are managed by 'cgconfig'.
  • The 'cgconfig' service is not started by default on CentOS 6.6.
  • Also the default configuration does not attach the 'perf_event' subsystem.
  • To do this, add 'perf_event = /cgroup/perf_event;' to the entries in '/etc/cgconfig.conf'.
$ sudo yum install -y libcgroup
$ sudo service cgconfig start

Building Mesos

  • Change working directory.
$ cd mesos
  • Bootstrap (Only required if building from git repository).
$ ./bootstrap
  • Configure and build.
$ mkdir build
$ cd build
$ ../configure
$ make
$ make install 

cat mesos-master-env.sh.template> mesos-master-env.sh  
cat mesos-slave-env.sh.template> mesos-slave-env.sh  
cat mesos-deploy-env.sh.template> mesos-deploy-env.sh  
in mesos-slave-env.sh mesos-agent-env.sh

  • start zookeeper

  • start mesos master and agent

./mesos-master.sh --ip=192.168.0.123 \
--work_dir=/var/lib/mesos \
--zk='zk://192.168.0.123:2181/mesos' \
--quorum=1 \
 --log_dir=/var/lib/mesos/master/master.log


./mesos-agent.sh \
--master='zk://192.168.0.123:2181/mesos' \
 --work_dir=/var/lib/mesos/agent
  • start elastic-job-cloud-scheduler
git clone https://github.com/elasticjob/elastic-job-cloud.git
/root/elastic-job-cloud-scheduler-3.0.0.M1-SNAPSHOT/bin/start.sh
  • example
git clone https://github.com/elasticjob/elastic-job-example.git

mvn clean install -Dmaven.test.skip=true
  • 注册app
curl -l -H "Content-type: application/json" -X POST -d '{"appName":"test_app_tupx","appURL":"http://192.168.0.123/local/test-job-1.0-SNAPSHOT.tar.gz","cpuCount":0.1,"memoryMB":64.0,"bootstrapScript":"bin/start.sh","appCacheEnable":true}' http://127.0.0.1:8899/api/app

注册任务

curl -l -H "Content-type: application/json" -X POST -d '{"jobName":"test_simple_job_tupx","jobClass":"com.aidai.job.simple.TaskSimpleJob","jobType":"SIMPLE","jobExecutionType":"TRANSIENT","cron":"0/20 * * * * ?","shardingTotalCount":5,"cpuCount":0.1,"memoryMB":64.0,"appName":"test_app_tupx","failover":true,"misfire":true}' http://127.0.0.1:8899/api/job/register

-- 注册app

curl -l -H "Content-type: application/json" -X POST -d '{"appName":"job-web","appURL":"http://192.168.0.123/local/elastic-job-example-cloud-2.1.5.tar.gz","cpuCount":0.1,"memoryMB":64.0,"bootstrapScript":"bin/start.sh","appCacheEnable":true}' http://127.0.0.1:8899/api/app
curl -l -H "Content-type: application/json" -X POST -d '{"appName":"exampleApp","appURL":"http://192.168.0.123/local/elastic-job-example-cloud-2.1.5.tar.gz","cpuCount":0.1,"memoryMB":64.0,"bootstrapScript":"bin/start.sh","appCacheEnable":true}' http://localhost:8899/api/app

-- 注册任务

curl -l -H "Content-type: application/json" -X POST -d '{"jobName":"job-web-test","jobClass":"com.aidai.job.simple.TaskSimpleJob","jobType":"SIMPLE","jobExecutionType":"TRANSIENT","cron":"0/10 * * * * ?","shardingTotalCount":5,"cpuCount":0.1,"memoryMB":64.0,"appName":"job-web","failover":true,"misfire":true}' http://127.0.0.1:8899/api/job/register

curl -l -H "Content-type: application/json" -X POST -d '{"jobName":"test_job_dataflow","appName":"job-web","jobType":"DATAFLOW","jobExecutionType":"DAEMON","jobClass":"com.aidai.job.dataflow.TaskDataFlowJob","cron":"0/10 * * * * ?","shardingTotalCount":3,"cpuCount":0.1,"memoryMB":64.0}' http://localhost:8899/api/job/register

时间同步

ntpdate cn.pool.ntp.org
hwclock --systohc
date

##防火墙

service iptables status
service iptables start
service iptables stop
  1. 注册APP:
curl -l -H "Content-type: application/json" -X POST -d '{"appName":"exampleApp","appURL":"http://localhost:8080/elastic-job-example-cloud-2.1.4.tar.gz","cpuCount":0.1,"memoryMB":64.0,"bootstrapScript":"bin/start.sh","appCacheEnable":true}' http://localhost:8899/api/app
  1. Java启动方式作业注册:
curl -l -H "Content-type: application/json" -X POST -d '{"jobName":"test_job_simple","appName":"exampleApp","jobType":"SIMPLE","jobExecutionType":"TRANSIENT","jobClass":"com.dangdang.ddframe.job.example.job.simple.JavaSimpleJob","cron":"0/10 * * * * ?","shardingTotalCount":1,"cpuCount":0.1,"memoryMB":64.0}' http://localhost:8899/api/job/register

curl -l -H "Content-type: application/json" -X POST -d '{"jobName":"test_job_dataflow","appName":"exampleApp","jobType":"DATAFLOW","jobExecutionType":"DAEMON","jobClass":"com.dangdang.ddframe.job.example.job.dataflow.JavaDataflowJob","cron":"0/10 * * * * ?","shardingTotalCount":3,"cpuCount":0.1,"memoryMB":64.0}' http://localhost:8899/api/job/register

curl -l -H "Content-type: application/json" -X POST -d '{"jobName":"test_job_script","appName":"exampleApp","jobType":"SCRIPT","jobExecutionType":"TRANSIENT","cron":"0/10 * * * * ?","shardingTotalCount":3,"cpuCount":0.1,"memoryMB":64.0, scriptCommandLine="script/demo.sh"}' http://localhost:8899/api/job/register
  1. Spring启动方式作业注册:
curl -l -H "Content-type: application/json" -X POST -d '{"jobName":"test_job_simple_spring","appName":"exampleApp","jobType":"SIMPLE","jobExecutionType":"TRANSIENT","jobClass":"com.dangdang.ddframe.job.example.job.simple.SpringSimpleJob","beanName":"springSimpleJob","applicationContext":"classpath:META-INF/applicationContext.xml","cron":"0/10 * * * * ?","shardingTotalCount":1,"cpuCount":0.1,"memoryMB":64.0}' http://localhost:8899/api/job/register

curl -l -H "Content-type: application/json" -X POST -d '{"jobName":"test_job_dataflow_spring","appName":"exampleApp","jobType":"DATAFLOW","jobExecutionType":"DAEMON","jobClass":"com.dangdang.ddframe.job.example.job.dataflow.SpringDataflowJob","beanName":"springDataflowJob","applicationContext":"classpath:META-INF/applicationContext.xml","cron":"0/10 * * * * ?","shardi