Paper Experiments - uccross/skyhookdm-ceph GitHub Wiki

======================================================
SET UP CEPH ON INSTANCE NODES
======================================================

  1. Spin up an instance on CloudLab. This example uses a 2-node client0-osd0 configuration.
  2. Install packages:
sudo apt-get update ;
sudo apt-get install tmux ;
sudo apt-get install vim ;
sudo apt-get install sysstat ;
sudo apt-get install systemd ;
  1. Change hostnames on all nodes to avoid networking confusion.
sudo hostname client0 ;
HOSTNAME=client0 ;
  1. On client0, copy these files from the shared and get a nodes.txt file with the node labels in a list.
cp /proj/skyhook-PG0/projscripts/format-sd* . ;
cp /proj/skyhook-PG0/projscripts/cluster_setup_copy_ssh_keys.sh  . ;
cp /proj/skyhook-PG0/projscripts/zap-sd* . ;
echo client0 >> nodes.txt ;
echo osd0 >> nodes.txt ;
  1. On each osd, copy only the format scripts.
cp /proj/skyhook-PG0/projscripts/format-sd* . ;
  1. Copy your PRIVATE Cloudlab ssh key from your login machine (e.g. your laptop) to a new ~/.ssh/id_rsa on each node of your Cloudlab instance and change permissions to 0600.
chmod 0600 .ssh/id_rsa ;
  1. On client0, configure the cluster ssh.
sh cluster_setup_copy_ssh_keys.sh ;
  1. On client0 ONLY, make sure sda4 is the 500GB SSD (check with lsblk) and run format-sda4.sh. If sda4 is not the 500GB SSD, replace the string 'sda4' in the format-sda4.sh script with the appropriate identifier.
sh format-sda4.sh ; # pass in username e.g. 'sh format-sda4.sh kat ;'
  1. On all nodes, also reformat the 1TB HDD, which is sdb in the case of c220g5 nodes.
sh format-sdb.sh ;  # pass in username e.g. 'sh format-sdb.sh kat ;'

Also, on ALL OSD NODES reformat the SSD as the device owned by the OSD code (DO NOT MOUNT!!!!).

sudo mkfs -t ext4 /dev/sda4 ;
  1. Install some version of Ceph on each node in the instance. Here's the line for installing from deb files saved in the shared dir. Change the path to something appropriate. This will take awhile--may want to use tmux (but would need to install it first). Be ROOT during this process only. RECOMMENDATION: do this in a tmux session.
sudo su - ;
apt-get update ; apt-get -f install ;  apt-get update; sudo dpkg -i /proj/skyhook-PG0/cephbits/kat_skyhook1227_ub14_g5/*.deb; sudo apt-get install -f -y; sudo dpkg -i /proj/skyhook-PG0/cephbits/kat_skyhook1227_ub14_g5/*.deb ;
exit ; # exit root back to normal user
  1. Install ceph-deploy (VERSION 1.5.37!!!!) on client0 only and run initial setup commands.
sudo apt-get -f install ;
sudo apt-get install -y python-virtualenv ;
mkdir cluster ;
cd cluster ;
virtualenv env ;
env/bin/pip install ceph-deploy==1.5.37 ;
env/bin/ceph-deploy new client0 ;
env/bin/ceph-deploy mon create-initial ;
  1. On client0, copy nodes.txt and zap-sda4.sh into ~/cluster.
  2. On client0 in ~/cluster, remove 'client0' from nodes.txt.
  3. On client0 in ~/cluster, run the zap-sdb.sh script to reformat the 1TB HDDs on the osd nodes.
sh zap-sdb.sh ;
  1. More setup stuff.
env/bin/ceph-deploy admin client0 ;
sudo chmod a+r /etc/ceph/ceph.client.admin.keyring ;
ceph osd set noscrub ;
ceph osd set nodeep-scrub ;

======================================================
SET UP CEPH FILE SYSTEM ON INSTANCE NODES
======================================================
16. Create MDS for filesystem.

env/bin/ceph-deploy mds create client0 ;
  1. Create metadata and data pools for the filesystem.
ceph osd pool create katfs_meta 64 64 replicated ;
ceph osd pool create katfs_data 64 64 replicated ;
ceph osd pool set katfs_meta size 1 ;
ceph osd pool set katfs_data size 1 ;
  1. Create the filesystem.
ceph fs new katfs katfs_meta katfs_data
  1. Make a directory to mount with cephfs.
mkdir /mnt/katfs ;
  1. Check status to make sure MDS is up and active.
  2. Mount cephfs in the newly made directory.
ceph-fuse -k /etc/ceph/ceph.client.admin.keyring -c /etc/ceph/ceph.conf /mnt/katfs ;
  1. In /mnt/sda4 on client0, clone the repo, checkout the appropriate branch, and compile.
cd /mnt/sda4 ;
git clone https://github.com/uccross/skyhook-ceph.git ;
cd skyhook-ceph ;
git checkout skyhook-transforms ;
./install-deps.sh ;
./do_cmake.sh ;
cd build ;
make -j 34 ;

======================================================
SET UP STORED DATA FOR PAPER EXPERIMENTS : OBJ V FILE READS
======================================================
23. Make the pool and store the data.

ceph osd pool create paper_experiments_pool 64 64 replicated ;
ceph osd pool set paper_experiments_pool size 1 ;
rados -p paper_experiments_pool put objvfile_1mb /proj/skyhook-PG0/kat_stuff/dataset_obj_v_file_1mb.txt ;
rados -p paper_experiments_pool put objvfile_10mb /proj/skyhook-PG0/kat_stuff/dataset_obj_v_file_10mb.txt ;
rados -p paper_experiments_pool put objvfile_100mb /proj/skyhook-PG0/kat_stuff/dataset_obj_v_file_100mb.txt ;
  1. Also copy the data files to /mnt/katfs.
cp /proj/skyhook-PG0/kat_stuff/dataset_obj_v_file_1mb.txt /mnt/katfs ;
cp /proj/skyhook-PG0/kat_stuff/dataset_obj_v_file_10mb.txt /mnt/katfs ;
cp /proj/skyhook-PG0/kat_stuff/dataset_obj_v_file_100mb.txt /mnt/katfs ;
⚠️ **GitHub.com Fallback** ⚠️