Giulia Meeting Notes rolling log - EdgeVU/group-notes GitHub Wiki

2021-05-28

We discussed about the early benchmarks for cassandra. Testing in localhost with a 5 node cluster, the difference between different consistency protocols is not that evident

ping to localhost is in the order of microseconds, but the total cost of the get put operations is in the order of 2-3 ms for 16-byte kv pair

todo:

  • match number of replicas with number of nodes, makes more sense
  • plot the distribution of the latency for all the iterations
  • try to distribute the cassandra nodes on all the different machines
  • tc on one machine to see if the latency adds up and how it affects multiple quorum levels
  • e.g. tc with 5 ms latency on localhost
  • compare python and c++ driver
    • python can be justify for ml workloads and whatnot
    • c++ for efficiency
  • clone and build cassandra on my own, will have to do that soon for benchmarking purposes
  • it makes sense to think about the project as an evaluation of the footprint of cassandra

read paper link

next steps

  • basic traffic shaping single machine
  • traffic shaping on multiple physical nodes so we dont have interference in the NIC
  • try to understand performance breakdown of the get-put operation. with basically null RTT cost it takes 2-3 ms for the get-put of 16 bytes even in single-copy consistency -> goal is to have performance breakdown
  • extend python script for benchmarking, support multithreading, different kv sizes, different load distributions for requests to the cluster...
  • look into cassandra consistency algorithm -> paxos
  • medium-term goal is to scope down the phd project to decide the angle, mobility vs programmabilty vs dynamic monitoring/replica placement

20-05-2021

14-5-2021

7-5-2021

todo:

  • deploy dockerized cassandra
  • deploy native cassandra

target: deploy 5/7 nodes in cassandra → then add tc rules with different latencies also look at pathstore, necessary to benchmark as well as cassandra

30-04-2021

action points

  • from last week: I had to evaluate if fogify is good to experiment with cassandra

results

  • ran ping and iperf in fogify topology to validate the latency and bandwidth
  • results seem reasonable within the constraints specified by the topology model

discussion:

  • go on with fogify and go ahead with cassandra and pathstore deployment
  • goal for SEC is an initial evaluation of Cassandra and Pathstore
  • we need to find reasonable topologies

todo (for next meeting)

  • understand a good topology for cassandra/pathstore experiments
  • deploy cassandra on docker/fogify

23-04-2021

action points

  • discussion about individual project for Tom, validate fogify and potentially port to kubernetes

what I'm doing now:

  • stuck on cassandra multinode docker deployment
  • stuck on fogify deployment of infra

todo for next meeting

  • native cassandra benchmark
  • fogify benchmark for cassandra
  • docker benchmark on fogify

16-04-2021

action points

  • update on fogify demo and network capabilities
  • presentation on fogify capabilities. Looks promising to emulate all the characteristics that are needed for now
  • potential master thesis ideas: support for mobility of end nodes, migration from docker to cgroups
  • Q: how many containers does our cluster support?
  • decision on rejected paper draft -> go over the reviews, need to substantiate the programmability aspect.

todo for next meeting

  • docker deployment to test latency, bandwidth and drop rate compliance, does fogify match the expected behavior?
  • make revision plan for hotstorage for position paper, address the comments from the reviewers

9-04-2021

action points

  • update on fogify state of things
  • solved missing image so now I can deploy barebone fogify controllers, but still not deploy an application scenario
  • problem at the moment is that they treat and deploy the application and infrastructure in a coupled way, whereas I would find it more convenient to deploy them separately

roadmap:

  • finish taxi demo
  • identify network capabilities in fogify, can it support mobility scenarios, network trace emulations, degradation of quality of signal as user moves, nodes joining and leaving
  • deploy empty docker images on fogify, decouple infrastructure from application
  • cassandra on docker (off fogify)
  • YCSB experiment on the cluster, plot the results and understand how the latency behaves and if it matches with the cluster specs
  • cassandra on docker (on fogify)
  • evaluate cassandra at the edge

todo for next meeting

  • Create a couple of slides with fogify infrastructure overview and functionality
  • run demo example for the taxi app
  • (hopefully) run cassandra example on docker

26-03-2021

With Lin:

  • talked about joining the HPDC reading group Sys4ML (FYI)
  • discussed BSc thesis topic on in-network computing

todo for next meeting

5-3-2021

action points

  • missing edge platform, need to build a system to deploy nodes/sensors and emulate different network conditions
  • need to do paper reading for background and foundational papers for storage and networking

todo for next meeting

  • create wiki page for papers to dump

26-2-2021

  • final review and work to submit the edgesys paper

19-2-2021

  • skipped meeting due to ACE seminar