cgroup io weight test - dublio/iotrack GitHub Wiki
Welcome to the iotrack wiki!
cgroup io weight test for bfq, blk-iocost, nvme-wrr
- kernel code: https://github.com/dublio/linux/tree/wrr-v5-iotrack-v2
- test script: https://github.com/dublio/tools/blob/master/shell/fio_script/test.sh
- iotrack: https://github.com/dublio/iotrack
0. SDD spec P4510 2TB
Sequential Read (up to) 3200 MB/s
Sequential Write (up to) 2000 MB/s
Random Read (100% Span) 637000 IOPS
Random Write (100% Span) 81500 IOPS
1. cgroup weight
mkdir /sys/fs/cgroup/test1
mkdir /sys/fs/cgroup/test2
test1.weight : test2.weight = 8 : 1
1.1 blk-iocost weight
# disable io scheduler and bfq weight
echo none > /sys/block/nvme1n1/queue/scheduler
# configure qos model, which came from the result of iocost_coef_gen.py
echo "`cat /sys/block/nvme1n1/dev` ctrl=user model=linear rbps=3286476297 rseqiops=547837 rrandiops=793881 wbps=2001272356 wseqiops=482243 wrandiops=483037" > /sys/fs/cgroup/io.cost.model
# Actually, from ssd specification, we can see there are little difference from these parameters
# configure weight
echo 800 > /sys/fs/cgroup/test1/io.weight
echo 100 > /sys/fs/cgroup/test2/io.weight
# clear nvme-wrr weight if set
echo "`cat /sys/block/nvme1n1/dev` none" > /sys/fs/cgroup/test1/io.wrr
echo "`cat /sys/block/nvme1n1/dev` none" > /sys/fs/cgroup/test2/io.wrr
1.2 bfq weight
# enable ioscheduler and bfq weight
echo bfq > /sys/block/nvme1n1/queue/scheduler
# configure weight
echo 800 > /sys/fs/cgroup/test1/io.bfq.weight
echo 100 > /sys/fs/cgroup/test2/io.bfq.weight
# clear blk-iocost if set
echo 100 > /sys/fs/cgroup/test1/io.weight
# clear nvme-wrr weight if set
echo "`cat /sys/block/nvme1n1/dev` none" > /sys/fs/cgroup/test1/io.wrr
echo "`cat /sys/block/nvme1n1/dev` none" > /sys/fs/cgroup/test2/io.wrr
1.3 nvme-wrr weight
# enable high and low queue
# If you have port nvme-wrr patch
modprobe -r nvme
modprobe nvme wrr_high_queues=20 wrr_low_queues=20
# disable io scheduler
echo none > /sys/block/nvme1n1/queue/scheduler
# clear blk-iocost and bfq weight
echo 100 > /sys/fs/cgroup/test1/io.bfq.weight
echo 100 > /sys/fs/cgroup/test1/io.weight
# configure nvme weight
h=64;m=32;l=8;ab=0;
nvme set-feature /dev/nvme1n1 -f 1 -v $(printf "0x%x\n" $(($ab<<0|$l<<8|$m<<16|$h<<24)))
echo "`cat /sys/block/nvme1n1/dev` high" > /sys/fs/cgroup/test1/io.wrr
echo "`cat /sys/block/nvme1n1/dev` low" > /sys/fs/cgroup/test2/io.wrr
2. fio test result(fio & iotrack) for cgroup
Now we test randread 4KB @numjobs @iodepth
Test script: https://github.com/dublio/tools/blob/master/shell/fio_script/test.sh
- iotrack result
2.1 randread vs randread
test1.randread.weight : test2.randread.weight = 8 : 1
test1.randread: bs=4K, numjobs=8 , iodepth=32
test2.randread: bs=4K, numjobs=8 , iodepth=32
2.1.1 blk-iocost
sh test.sh iocost rr 8 32
Time Device rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrqkb avgqu-sz await r_await w_await svctm %util conc
2020-03-27 14:18:07.714 nvme1n1 0 0 753021 0 2941.49 0.00 4.00 102.92 0.68 0.68 0.00 0.00 100.10 102.81
Time Device io/s MB/s %io %byte %tm %dtm %d2c ad2c aq2c %hit0 %hit1 %hit2 %hit3 %hit4 %hit5 %hit6 %hit7 cgroup
2020-03-27 14:18:07.714 nvme1n1 753019 2941.48 100.00 100.00 100.00 100.00 99.91 0.68 0.68 0.00 0.00 0.06 6.67 86.78 99.60 100.00 100.00 /
2020-03-27 14:18:07.714 nvme1n1 376926 1472.37 50.06 50.06 50.02 50.02 99.91 0.67 0.68 0.00 0.00 0.06 6.77 86.69 99.63 100.00 100.00 /test1
2020-03-27 14:18:07.714 nvme1n1 376079 1469.06 49.94 49.94 49.99 49.99 99.91 0.68 0.68 0.00 0.00 0.05 6.58 86.86 99.58 100.00 100.00 /test2
2.1.2 bfq
sh test.sh bfq rr 8 32
Time Device rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrqkb avgqu-sz await r_await w_await svctm %util conc
2020-03-27 14:19:34.867 nvme1n1 0 0 210583 0 822.59 0.00 4.00 418.52 2.08 2.08 0.00 0.00 100.10 418.11
Time Device io/s MB/s %io %byte %tm %dtm %d2c ad2c aq2c %hit0 %hit1 %hit2 %hit3 %hit4 %hit5 %hit6 %hit7 cgroup
2020-03-27 14:19:34.867 nvme1n1 210580 822.58 100.00 100.00 100.00 100.00 4.49 0.09 2.08 25.11 63.33 89.33 97.94 99.86 99.86 99.86 99.86 /
2020-03-27 14:19:34.867 nvme1n1 193686 756.59 91.98 91.98 52.37 84.79 7.27 0.09 1.19 27.28 65.72 91.29 98.54 99.87 99.87 99.87 99.87 /test1
2020-03-27 14:19:34.867 nvme1n1 16894 65.99 8.02 8.02 47.63 15.20 1.43 0.18 12.37 0.28 35.88 66.86 91.10 99.81 99.81 99.81 99.81 /test2
2.1.3 nvme-wrr
sh test.sh wrr rr 8 32
Time Device rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrqkb avgqu-sz await r_await w_await svctm %util conc
2020-03-27 14:21:25.137 nvme1n1 0 0 760193 0 2969.50 0.00 4.00 234.77 0.67 0.67 0.00 0.00 100.20 234.30
Time Device io/s MB/s %io %byte %tm %dtm %d2c ad2c aq2c %hit0 %hit1 %hit2 %hit3 %hit4 %hit5 %hit6 %hit7 cgroup
2020-03-27 14:21:25.137 nvme1n1 760185 2969.47 100.00 100.00 100.00 100.00 99.91 0.67 0.67 0.15 3.38 24.57 57.84 84.83 91.95 99.66 100.00 /
2020-03-27 14:21:25.137 nvme1n1 667725 2608.30 87.84 87.84 49.46 49.43 99.85 0.37 0.38 0.17 3.86 27.97 65.84 96.58 99.92 100.00 100.00 /test1
2020-03-27 14:21:25.137 nvme1n1 92460 361.17 12.16 12.16 50.54 50.58 99.98 2.77 2.77 0.00 0.00 0.00 0.00 0.00 34.42 97.18 100.00 /test2
2.1.4 none
Time: 2020-03-27 18:45:24
Device rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrqkb avgqu-sz await r_await w_await svctm %util conc
nvme1n1 0 0 753169 0 2942.05 0.00 4.00 70.95 0.68 0.68 0.00 0.00 100.40 70.66
Device io/s MB/s %io %byte %tm %dtm %d2c ad2c aq2c %hit0 %hit1 %hit2 %hit3 %hit4 %hit5 %hit6 %hit7 cgroup
nvme1n1 753180 2942.11 100.00 100.00 100.00 100.00 99.92 0.68 0.68 0.00 0.00 0.02 3.21 90.83 99.81 100.00 100.00 /
nvme1n1 376605 1471.11 50.00 50.00 49.98 49.98 99.91 0.67 0.68 0.00 0.00 0.02 3.19 90.82 99.81 100.00 100.00 /test1
nvme1n1 376585 1471.04 50.00 50.00 50.02 50.02 99.92 0.68 0.68 0.00 0.00 0.02 3.22 90.85 99.80 100.00 100.00 /test2
2.1.5 summary (fio output)
test case bw iops rd_avg_lat wr_avg_lat rd_p99_lat wr_p99_lat
=======================================================================================
bfq_test1 767226 191806 1333.30 0.00 536.00 0.00
bfq_test2 94607 23651 10816.06 0.00 610.00 0.00
iocost_test1 1457718 364429 701.76 0.00 1630.00 0.00
iocost_test2 1466337 366584 697.62 0.00 1613.00 0.00
none_test1 1456585 364146 702.22 0.00 1646.00 0.00
none_test2 1463090 365772 699.12 0.00 1613.00 0.00
wrr_test1 2635391 658847 387.94 0.00 1236.00 0.00
wrr_test2 365428 91357 2801.00 0.00 5537.00 0.00
2.2 readrand vs seq write
test1.randread.weight : test2.write.weight = 8 : 1
test1.randread: bs=4K, numjobs=8 , iodepth=32
test2.write: bs=256K, numjobs=1 , iodepth=32
2.2.1 blk-iocost
Time: 2020-03-27 18:26:51
Device rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrqkb avgqu-sz await r_await w_await svctm %util conc
nvme1n1 0 0 69990 12272 273.40 1534.00 22.50 277.29 3.85 3.65 4.97 0.00 100.30 276.46
Device io/s MB/s %io %byte %tm %dtm %d2c ad2c aq2c %hit0 %hit1 %hit2 %hit3 %hit4 %hit5 %hit6 %hit7 cgroup
nvme1n1 82265 1807.53 100.00 100.00 100.00 100.00 99.96 3.85 3.85 0.00 0.12 2.49 16.91 36.58 40.47 43.29 97.56 /
nvme1n1 70022 273.52 85.12 15.13 80.75 80.76 99.96 3.65 3.65 0.00 0.14 2.93 19.90 43.00 47.56 50.76 97.15 /test1
nvme1n1 12280 1535.00 14.93 84.92 19.29 19.29 99.92 4.97 4.98 0.00 0.00 0.00 0.00 0.00 0.00 0.70 99.97 /test2
2.2.2 bfq
Time: 2020-03-27 18:25:07
Device rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrqkb avgqu-sz await r_await w_await svctm %util conc
nvme1n1 0 0 184600 1082 721.10 135.25 4.72 309.45 1.80 1.32 83.56 0.00 100.20 308.83
Device io/s MB/s %io %byte %tm %dtm %d2c ad2c aq2c %hit0 %hit1 %hit2 %hit3 %hit4 %hit5 %hit6 %hit7 cgroup
nvme1n1 185669 856.29 100.00 100.00 100.00 100.00 8.37 0.15 1.80 0.70 43.90 84.25 97.32 99.13 99.22 99.48 99.74 /
nvme1n1 184352 720.12 99.29 84.10 72.84 85.38 9.82 0.13 1.32 0.70 44.14 84.72 97.88 99.71 99.81 99.81 99.81 /test1
nvme1n1 1082 135.25 0.58 15.79 27.15 14.55 4.49 3.77 83.93 0.00 0.00 0.00 0.00 0.00 0.00 42.88 88.26 /test2
2.2.3 nvme-wrr
Time: 2020-03-27 18:24:00
Device rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrqkb avgqu-sz await r_await w_await svctm %util conc
nvme1n1 0 0 161819 9294 632.11 1161.75 10.74 241.37 1.86 1.58 6.72 0.00 100.20 240.89
Device io/s MB/s %io %byte %tm %dtm %d2c ad2c aq2c %hit0 %hit1 %hit2 %hit3 %hit4 %hit5 %hit6 %hit7 cgroup
nvme1n1 171113 1793.86 100.00 100.00 100.00 100.00 99.94 1.85 1.85 0.00 0.85 7.78 25.45 44.08 59.79 91.28 99.72 /
nvme1n1 161741 631.80 94.52 35.22 80.25 80.24 99.93 1.57 1.57 0.00 0.90 8.23 26.92 46.64 63.23 96.52 100.00 /test1
nvme1n1 9285 1160.62 5.43 64.70 19.70 19.71 99.95 6.73 6.74 0.00 0.00 0.00 0.00 0.00 0.00 0.00 94.91 /test2
2.2.4 none
Time: 2020-03-27 18:20:49
Device rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrqkb avgqu-sz await r_await w_await svctm %util conc
nvme1n1 0 0 68290 12042 266.76 1505.25 22.59 278.40 3.94 3.75 5.05 0.00 100.20 277.84
Device io/s MB/s %io %byte %tm %dtm %d2c ad2c aq2c %hit0 %hit1 %hit2 %hit3 %hit4 %hit5 %hit6 %hit7 cgroup
nvme1n1 80332 1772.25 100.00 100.00 100.00 100.00 99.96 3.95 3.96 0.00 0.14 2.79 16.31 35.59 40.04 43.87 91.26 /
nvme1n1 68312 266.84 85.04 15.06 80.62 80.63 99.97 3.75 3.75 0.00 0.17 3.28 19.14 41.80 47.05 51.34 89.73 /test1
nvme1n1 12049 1506.12 15.00 84.98 19.49 19.48 99.92 5.14 5.14 0.00 0.00 0.00 0.00 0.00 0.00 1.25 100.00 /test2
2.2.5 summary (fio output)
test case bw iops rd_avg_lat wr_avg_lat rd_p99_lat wr_p99_lat
=======================================================================================
bfq_test1 814327 203581 1256.19 0.00 593.00 0.00
bfq_test2 104758 409 0.00 78196.32 0.00 1052770.00
iocost_test1 270467 67616 3784.02 0.00 9371.00 0.00
iocost_test2 1541575 6021 0.00 5313.02 0.00 6848.00
none_test1 271708 67927 3767.01 0.00 9502.00 0.00
none_test2 1541951 6023 0.00 5311.50 0.00 6848.00
wrr_test1 775005 193751 1320.17 0.00 4112.00 0.00
wrr_test2 1198319 4680 0.00 6835.30 0.00 8847.00