Measurement - shuwens/NetBricks GitHub Wiki

• Instrumentation and measurement
• CPU and memory
  • for measuring latency
    • m-better-w-groupby
    • m-verify-order-1
    • m-verify-order-1-w-pc
    • m-verify-order-2
    • m-verify-order-excessive
• Using eBPF to measure disk io and network latency
  • Disk
  • Network IO
• Measuring elapsed time

CPU/mem measure again

1

https://stackoverflow.com/questions/1221555/retrieve-cpu-usage-and-memory-usage-of-a-single-process-on-linux

pidstat -u 1 10 | grep ^Average | sort -r -n -b -k 8,8

top -b -n 2 -d 0.2 -p 6962 | tail -1 | awk '{print $9}'

2

https://unix.stackexchange.com/questions/288589/get-chromes-total-memory-usage

detailed output, in kB apparently

smem -t -P chrom

just the total PSS, with automatic unit:

smem -t -k -c pss -P chrom | tail -n 1

3

https://stackoverflow.com/questions/1221555/retrieve-cpu-usage-and-memory-usage-of-a-single-process-on-linux

4

https://askubuntu.com/questions/56266/how-to-log-memory-and-cpu-usage-of-an-application

top -b -d 10 -n 3 >> top-file

5

https://stackoverflow.com/questions/16726779/how-do-i-get-the-total-cpu-usage-of-an-application-from-proc-pid-stat

6: sum ps mem

https://unix.stackexchange.com/questions/209689/sum-the-memory-usages-of-all-the-processes-of-a-program

7

https://unix.stackexchange.com/questions/13968/show-top-five-cpu-consuming-processes-with-ps

8

https://stackoverflow.com/questions/1221555/retrieve-cpu-usage-and-memory-usage-of-a-single-process-on-linux

top -b -n 2 -d 0.2 -p 6962 | tail -1 | awk '{print $9}'

9

https://phoenixnap.com/kb/check-cpu-usage-load-linux

10

https://www.cyberciti.biz/tips/how-do-i-find-out-linux-cpu-utilization.html

11

https://linoxide.com/monitoring-2/10-tools-monitor-cpu-performance-usage-linux-command-line/

Instrumentation and measurement

ps 2: figure out how to calculate child process together

https://stackoverflow.com/questions/38341366/how-to-specify-commandline-arguments-in-pgrep-in-bash

https://stackoverflow.com/questions/29879057/pgrep-command-not-returning-pid

https://linuxize.com/post/pgrep-command-in-linux/

https://unix.stackexchange.com/questions/74185/how-can-i-prevent-grep-from-showing-up-in-ps-results

https://unix.stackexchange.com/questions/428035/linux-use-pgrep-command

https://unix.stackexchange.com/questions/285147/do-not-produce-output-when-process-does-not-exist

https://unix.stackexchange.com/questions/507290/why-cant-i-get-pgrep-output-right-to-variable-on-bash-script

https://unix.stackexchange.com/questions/267007/pgrep-full-match-not-work-only-part-why

https://serverfault.com/questions/77162/how-to-get-pgrep-to-display-full-process-info/210823

https://www.howtogeek.com/448271/how-to-use-the-ps-command-to-monitor-linux-processes/

https://www.tecmint.com/ps-command-examples-for-linux-process-monitoring/

https://www.linuxtechi.com/ps-command-examples-monitor-linux-processes/

https://superuser.com/questions/102005/how-can-i-display-the-memory-usage-of-each-process-if-i-do-a-ps-ef

https://superuser.com/questions/363169/ps-how-can-i-recursively-get-all-child-process-for-a-given-pid

https://superuser.com/questions/1416328/check-total-usage-of-physical-memory-and-cpu-for-a-program-on-linux

https://stackoverflow.com/questions/938733/total-memory-used-by-python-process

https://stackoverflow.com/questions/38751772/how-do-i-find-the-total-memory-and-cpu-usage-of-a-process-and-all-of-its-childre

https://unix.stackexchange.com/questions/20483/how-to-find-which-process-is-causing-high-cpu-usage

https://unix.stackexchange.com/questions/554/how-to-monitor-cpu-memory-usage-of-a-single-process

https://unix.stackexchange.com/questions/147857/how-to-find-the-cpu-and-memory-usage-of-child-processes

top

https://askubuntu.com/questions/56266/how-to-log-memory-and-cpu-usage-of-an-application

https://askubuntu.com/questions/923922/how-to-save-cpu-logs-or-gpu-usage-values

https://stackoverflow.com/questions/29845711/limit-top-command-to-only-display-top-x-processes-on-command-line

https://stackoverflow.com/questions/46129404/linux-terminal-take-screenshot-of-top-command-and-email-it

htop

https://stackoverflow.com/questions/57252196/i-am-copying-htop-output-to-a-txt-file-but-it-does-not-copy-the-full-command

else

https://askubuntu.com/questions/726333/how-to-save-htop-output-to-file

https://askubuntu.com/questions/726333/how-to-save-htop-output-to-file

https://coderwall.com/p/ej7bqq/log-memory-usage-per-second-or-minute-on-linux

CPU and memory

https://linuxhint.com/check_memory_usage_process_linux/

https://alvinalexander.com/linux/unix-linux-process-memory-sort-ps-command-cpu

https://stackoverflow.com/questions/131303/how-to-measure-actual-memory-usage-of-an-application-or-process

https://virtualthreads.blogspot.com/2006/02/understanding-memory-usage-on-linux.html

https://stackoverflow.com/questions/22261452/finding-memory-usage-of-a-process-in-linux

https://stackoverflow.com/questions/1221555/retrieve-cpu-usage-and-memory-usage-of-a-single-process-on-linux

https://stackoverflow.com/questions/4802481/how-to-see-top-processes-sorted-by-actual-memory-usage

https://stackoverflow.com/questions/774556/peak-memory-usage-of-a-linux-unix-process

https://www.geeksforgeeks.org/tracing-memory-usage-linux/

https://www.binarytides.com/linux-command-check-memory-usage/

https://unix.stackexchange.com/questions/164653/actual-memory-usage-of-a-process

https://unix.stackexchange.com/questions/554/how-to-monitor-cpu-memory-usage-of-a-single-process

https://unix.stackexchange.com/questions/166558/how-to-check-which-process-is-using-most-memory

https://unix.stackexchange.com/questions/334232/memory-consumption-of-process

https://www.golinuxcloud.com/check-memory-usage-per-process-linux/

https://www.linux.com/tutorials/5-commands-checking-memory-usage-linux/

https://www.tecmint.com/find-linux-processes-memory-ram-cpu-usage/

http://www.brendangregg.com/FlameGraphs/memoryflamegraphs.html

https://www.circonus.com/2018/05/linux-system-monitoring-with-ebpf/

https://www.collabora.com/news-and-blog/blog/2019/05/14/an-ebpf-overview-part-5-tracing-user-processes/

https://jvns.ca/blog/2018/01/31/spying-on-a-ruby-process-s-memory-allocations/

https://sematext.com/blog/linux-kernel-observability-ebpf/

https://www.joyfulbikeshedding.com/blog/2019-01-31-full-system-dynamic-tracing-on-linux-using-ebpf-and-bpftrace.html

https://www.nccgroup.trust/us/about-us/newsroom-and-events/blog/2019/march/ebpf-adventures-fiddling-with-the-linux-kernel-and-unix-domain-sockets/

for measuring latency

m-better-w-groupby

src/nf.rs

pkt count 1100
# of start ts: 12791, # of stop ts: 1000
start to reset: avg processing time is 3.562315ms

Pkt counters

1
1
pkt counter 1: 12791
pkt counter 2: 0
pkt counter 6: 1000
30.01 OVERALL RX 3413135.87 TX 3412570.66

m-verify-order-1

src/nf.rs

m-verify-order-1-w-pc

src/nf.rs

After meta: Measurement started
NUM_TO_IGNORE: 10000000, pkt_count: 10000000

After tcph Measurement started
NUM_TO_IGNORE: 10000000, pkt_count: 10000000

Actual processing Measurement started
NUM_TO_IGNORE: 10000000, pkt_count: 10000000
pkt count 11000000
# of start ts: 1000019, # of stop ts: 999999
start to transform after meta: avg processing time is 1.461µs
pkt count 11000000
# of start ts: 1000019, # of stop ts: 999999
start t0 after tcph: avg processing time is 2.79µs
pkt count 11000000
# of start ts: 1000019, # of stop ts: 999999
start to after actural processing: avg processing time is 7.48µs

Pkt counters

pkt counter 1: 1001824
pkt counter 2: 1001824
pkt counter 3: 1000019
pkt counter 4: 1000019
pkt counter 5: 1000019
pkt counter 6: 1000000

m-verify-order-2

src/nf.rs

m-verify-order-excessive

src/nf.rs

Using eBPF to measure disk io and network latency

Disk

tools/biotop: Top for disks: Summarize block device I/O by process. Examples.

Network IO

tools/tcptop: Summarize TCP send/recv throughput by host. Top for TCP. Examples.

tools/tcpdrop: Trace kernel-based TCP packet drops with details. Examples.

tools/tcplife: Trace TCP sessions and summarize lifespan. Examples.

tools/tcpstates: Trace TCP session state changes with durations. Examples.

tools/tcpsubnet: Summarize and aggregate TCP send by subnet. Examples.

tools/tcptracer: Trace TCP established connections (connect(), accept(), close()). Examples.

new (udp)

https://www.unixmen.com/iptraf-tcpudp-network-monitoring-utility/

https://www.tecmint.com/watch-tcp-and-udp-ports-in-linux/

https://www.tecmint.com/20-netstat-commands-for-linux-network-management/

https://askubuntu.com/questions/257263/how-to-display-network-traffic-in-the-terminal

https://askubuntu.com/questions/333921/list-all-internet-connections

https://blog.packagecloud.io/eng/2017/02/06/monitoring-tuning-linux-networking-stack-sending-data/

https://www.cyberciti.biz/tips/linux-investigate-sockets-network-connections.html

https://linuxhint.com/send_receive_udp_packets_linux_cli/

net io

iftop

nethogs

iptraf-ng

https://www.slashroot.in/linux-iptraf-and-iftop-monitor-and-analyse-network-traffic-and-bandwidth

sar: https://askubuntu.com/questions/257263/how-to-display-network-traffic-in-the-terminal

ifstat

iftop

glances

slurm

nettop: https://askubuntu.com/questions/333921/list-all-internet-connections

qperf: https://www.opsdash.com/blog/network-performance-linux.html

https://opensourceforu.com/2016/10/network-performance-monitoring/

https://support.cumulusnetworks.com/hc/en-us/articles/216509388-Throughput-Testing-and-Troubleshooting

Measuring elapsed time

std::time

https://docs.rs/chrono/0.4.9/chrono/

https://doc.rust-lang.org/beta/std/time/struct.Instant.html

https://docs.rs/time/0.1.42/time/

iptraf-ng

https://github.com/sandreae/jacktrip_tests/blob/21c84b706e22c8fc1de184307de5f8f966343630/loopback_recording/_loopback_recording.sh

https://github.com/JeremyMonitor/SupervisionScripts/blob/3554daa09ff5c5568e705662f8b92a08f3806a8a/check_TOP_IP.sh

https://github.com/ramlohith/NOCManager/blob/b2fef82812fe54b3b7c88c69c9f3e229b2055d16/testing.sh

https://github.com/sandreae/jacktrip_tests/blob/21c84b706e22c8fc1de184307de5f8f966343630/loopback_recording/_loopback_recording.sh

https://github.com/CentOS/sig-core-t_functional/blob/58528f85c77d6e8cbcdd13981def0b0e585af84e/tests/p_iptraf/5-test_iptraf.sh

https://github.com/dekuRockShooter/home/blob/c269d92c51ca5edd415ec082247fda018026e265/scripts/deku_iptraf.sh

https://github.com/autotest/autotest-client-tests/blob/994485ce4d4a1bd3926b1e2b846d2c23d427e917/linux-tools/iptraf_ng/iptraf.sh

https://github.com/subhshetty/iptraf-ng/blob/2aebc41b53e87edf1ec5704dedd1e3606a00ee8d/iptraf-ng.sh

https://github.com/aewnfin/helperDoc/blob/75152d799597eaa53830993e0183c88e06384312/%E5%B7%A5%E7%A8%8B%E5%B8%88/%E8%BF%90%E7%BB%B4/tools/%E7%BD%91%E7%BB%9C%E6%B5%81%E9%87%8F.sh

https://github.com/usui-tk/amazon-ec2-userdata

https://github.com/maravento/gateproxy/blob/8ec1f839517b651c11a63c325c0ac572be81fa83/gateproxy.sh

https://github.com/sandreae/jacktrip_tests/blob/21c84b706e22c8fc1de184307de5f8f966343630/loopback_recording/_loopback_recording.sh