Skyhook Development on CloudLab - uccross/skyhookdm-ceph GitHub Wiki

Spin up CloudLab instance (try image skyhookceph-ubuntu14-1osds)
Install packages:

sudo apt-get update ;
sudo apt-get install tmux ;
sudo apt-get install vim ;

Change hostnames on all nodes to avoid networking confusion. On client0:

sudo hostname client0 ;
HOSTNAME=client0 ;

On osd0:

sudo hostname osd0 ;
HOSTNAME=osd0 ;

Edit /etc/hosts on each node to clean up addresses sudo vi /etc/hosts e.g.

127.0.0.1       localhost
10.10.1.2       client0
10.10.1.1       osd0

On client0, copy these files from the shared directory.

cp /proj/skyhook-PG0/projscripts/format-sda4.sh . ;
cp /proj/skyhook-PG0/projscripts/cluster_setup_copy_ssh_keys.sh  . ;

Copy your PRIVATE Cloudlab ssh key from your login machine (e.g. your laptop) to a new ~/.ssh/id_rsa on each node of your Cloudlab instance and change permissions to 0600.

chmod 0600 .ssh/id_rsa ;

On client0, configure the cluster ssh.

sh cluster_setup_copy_ssh_keys.sh ;

On client0 ONLY, make sure sda4 is the 500GB SSD (check with lsblk) and run format-sda4.sh. If sda4 is not the 500GB SSD, replace the string 'sda4' in the format-sda4.sh script with the appropriate identifier.

sh format-sda4.sh ; # pass in username e.g. 'sh format-sda4.sh kat ;'

Clone the Skyhook repo into /mnt/sda4.

cd /mnt/sda4 ;
git clone https://github.com/uccross/skyhook-ceph.git

Install and build Skyhook.

sudo ./install-deps.sh ;
./do_cmake.sh ;
cd build/ ;
tmux new -s mybuild
make -j34
detach from the build session CONTROL-B D

Spin up the virtual cluster.

../src/stop.sh ;
make vstart ;
../src/stop.sh;
../src/vstart.sh -d -n -x;
bin/ceph -s ;
bin/rados mkpool tpchflatbuf ;
bin/ceph osd lspools ;
wget https://users.soe.ucsc.edu/~jlefevre/skyhookdb/testdata/skyhook.test2objs.flatbufv2.lineitem.0 ;
wget https://users.soe.ucsc.edu/~jlefevre/skyhookdb/testdata/skyhook.test2objs.flatbufv2.lineitem.1 ;
yes | PATH=$PATH:bin ../src/progly/rados-store-glob.sh tpchflatbuf  skyhook.test2objs.flatbufv2.lineitem.* ;
bin/rados df  # should see tpchflatbuf pool with the 2 above objects ;
bin/run-query --num-objs 2 --pool tpchflatbuf --wthreads 1 --qdepth 10 --query flatbuf --select "*" ;
bin/ceph -s ;

Check if it worked.

kat@client0:/mnt/sda4/skyhook-ceph/build$ bin/rados df
2019-05-30 19:00:43.605199 7faf60894e40 -1 WARNING: all dangerous and experimental features are enabled.
2019-05-30 19:00:43.605505 7faf60894e40 -1 WARNING: all dangerous and experimental features are enabled.
2019-05-30 19:00:43.607205 7faf60894e40 -1 WARNING: all dangerous and experimental features are enabled.
POOL_NAME         USED OBJECTS CLONES COPIES MISSING_ON_PRIMARY UNFOUND DEGRADED RD_OPS RD   WR_OPS WR
cephfs_data_a        0       0      0      0                  0       0        0      0    0      0    0
cephfs_metadata_a 2246      21      0     63                  0       0        0      0    0     44 8192
tpchflatbuf       5632       2      0      6                  0       0        0      2 6144      2 6144

total_objects    23
total_used       152G
total_avail      1102G
total_space      1254G
kat@client0:/mnt/sda4/skyhook-ceph/build$
kat@client0:/mnt/sda4/skyhook-ceph/build$ bin/ceph -s
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
2019-05-30 18:59:18.000220 7ff4c384e700 -1 WARNING: all dangerous and experimental features are enabled.
2019-05-30 18:59:18.005158 7ff4c384e700 -1 WARNING: all dangerous and experimental features are enabled.
  cluster:
    id:     72c80fe7-ad5d-483c-95cc-11cd18dab655
    health: HEALTH_WARN
            application not enabled on 1 pool(s)

  services:
    mon: 3 daemons, quorum a,b,c
    mgr: x(active)
    mds: cephfs_a-1/1/1 up  {0=b=up:active}, 2 up:standby
    osd: 3 osds: 3 up, 3 in

  data:
    pools:   3 pools, 24 pgs
    objects: 23 objects, 7878 bytes
    usage:   152 GB used, 1102 GB / 1254 GB avail
    pgs:     24 active+clean

kat@client0:/mnt/sda4/skyhook-ceph/build