ublk - animeshtrivedi/notes GitHub Wiki

Code location

Talks

ublk command

atr@u24clean:~$ ublk
ublk add -t {null|loop|qcow2|nbd}
	-n DEV_ID -q NR_HW_QUEUES -d QUEUE_DEPTH
	-u URING_COMP -g NEED_GET_DATA -r USER_RECOVERY
	-i USER_RECOVERY_REISSUE --debug_mask=0x{DBG_MASK}
	--unprivileged

	target specific command line:
           loop: -f backing_file [--buffered_io]
           	default is direct IO to backing file
           qcow2: -f backing_file
           nbd: --host=$HOST [--port=$PORT] | --unix=$UNIX_PATH
ublk del -n DEV_ID [-a | --all]
ublk list [-n DEV_ID]
ublk recover [-n DEV_ID]
ublk features
ublk -v [--version]
ublk -h [--help]

Enable debugging

Change the build_with_uring_src file to enable passing all bash params from $1 to $@

PARAM=$@
echo $PARAM

./build_with_liburing_src --prefix=/home/$USER/local --enable-debug

Then during adding

sudo ./ublk add -t loop -f /tmp/ramdisk1/1.img -q 8 -d 4 --debug_mask=0xFF

/* 32bit debug mask, high 16 bits are for target code, and low 16 bits for lib */
#define	UBLK_DBG_DEV		(1U << 0)  -- 0x01
#define	UBLK_DBG_QUEUE		(1U << 1)  -- 0x02
#define	UBLK_DBG_IO_CMD		(1U << 2)  -- 0x04
#define	UBLK_DBG_IO		(1U << 3)  -- 0x08
#define	UBLK_DBG_CTRL_CMD	(1U << 4)  -- 0x10

2024-12 performance fluctuation

FIXME: non-privileged access to the device

atr@u24clean:~/src/ublksrv$ sudo ./ublk add  -t null -q 16 -d 512 --unprivileged
dev id 0: nr_hw_queues 8 queue_depth 512 block size 512 dev_capacity 524288000
	max rq size 524288 daemon pid 13473 flags 0x42 state LIVE
	ublkc: 237:0 ublkb: 259:1 owner: 0:0
	queue 0: tid 13474 affinity(0 )
	queue 1: tid 13475 affinity(1 )
	queue 2: tid 13476 affinity(2 )
	queue 3: tid 13477 affinity(3 )
	queue 4: tid 13478 affinity(4 )
	queue 5: tid 13479 affinity(5 )
	queue 6: tid 13480 affinity(6 )
	queue 7: tid 13481 affinity(7 )
	target {"dev_size":268435456000,"name":"null","type":0}

I copied the ./utils/ublk_dev.rules to /etc/udev/rules.d/ folder and then updated (the path to my home sbin)

$ cat /etc/udev/rules.d/ublk_dev.rules 
# SPDX-License-Identifier: MIT or GPL-2.0-only
# Udev rules for ublk devices.

KERNEL=="ublk-control", MODE="0666", OPTIONS+="static_node=ublk-control"
ACTION=="add",KERNEL=="ublk[bc]*",RUN+="/home/atr/local/sbin/ublk_chown.sh %k 'add' '%M' '%m'"
ACTION=="remove",KERNEL=="ublk[bc]*",RUN+="/home/atr/local/sbin/ublk_chown.sh %k 'remove' '%M' '%m'"

With that I have this error: sudo journalctl -f

Aug 05 11:25:26 u24clean ublksrvd-0[13060]: tid 13066: ublk dev 0 queue 5 started
Aug 05 11:25:26 u24clean sudo[13042]: pam_unix(sudo:session): session closed for user root
Aug 05 11:25:26 u24clean (udev-worker)[13075]: ublkb0: Process '/usr/bin/unshare -m /usr/bin/snap auto-import --mount=/dev/ublkb0' failed with exit code 1.

Access without sudo does not work

What are the different configuration options? And what is the life cycle of an I/O request?

Code exploration and configuration options

where the logs: journalctl | grep ublksrvd

Linux doc: https://docs.kernel.org/block/ublk.html

Goal for today

  • Setup runs with python scripts to do a simple latency sweep QD={1,...512} in Python
    • plot CDF, and loaded latencies
  • Setup runs with python scripts to do a simple bandwidth sweep QD={1,...512} in Python

Compilation and setup

$ git clone https://github.com/ming1/ubdsrv.git
$ autoreconf -i
$ ./configure --prefix=/home/atr/local/
$ make -j 
$ make install

some issues (with misleading error messages like):

atr@u24clean:~/src/ublksrv$ autoreconf -i 
libtoolize: putting auxiliary files in '.'.
libtoolize: copying file './ltmain.sh'
libtoolize: putting macros in AC_CONFIG_MACRO_DIRS, 'm4'.
libtoolize: copying file 'm4/libtool.m4'
libtoolize: copying file 'm4/ltoptions.m4'
libtoolize: copying file 'm4/ltsugar.m4'
libtoolize: copying file 'm4/ltversion.m4'
libtoolize: copying file 'm4/lt~obsolete.m4'
configure.ac:32: error: possibly undefined macro: AC_MSG_ERROR
      If this token and others are legitimate, please use m4_pattern_allow.
      See the Autoconf documentation.
configure.ac:71: error: possibly undefined macro: AC_DEFINE
autoreconf: error: /usr/bin/autoconf failed with exit status: 1
atr@u24clean:~/src/ublksrv$ autoreconf -i 
autoheader: warning: missing template: HAVE_GNUTLS
autoheader: warning: Use AC_DEFINE([HAVE_GNUTLS], [], [Description])
autoreconf: error: /usr/bin/autoheader failed with exit status: 1

I installed:

sudo apt-get install libgnutls28-dev gnutls-bin  libgnutls30t64

Still the problem persisted. Then I cleaned the build files with rm -rf * and then git checkout .. Fresh try:

configure.ac:32: error: possibly undefined macro: AC_MSG_ERROR
      If this token and others are legitimate, please use m4_pattern_allow.
      See the Autoconf documentation.
configure.ac:71: error: possibly undefined macro: AC_DEFINE
autoreconf: error: /usr/bin/autoconf failed with exit status: 1

Reason: pkg-config was missing. https://stackoverflow.com/questions/8811381/possibly-undefined-macro-ac-msg-error

sudo apt install pkgconf

Then liburing was missing, that I installed too see here

After a successful compilation

source: https://dev.to/amarjargal/testing-ublk-on-ubuntu-2204-9pe

The kernel compiled must support it, luckily what I compiled it supported it: image

Neat track to disable the security key signing (I have not tested it yet):

scripts/config --disable SYSTEM_TRUSTED_KEYS
scripts/config --disable SYSTEM_REVOCATION_KEYS

Insert the kernel module (should already be there from the source code built):

sudo modprobe ublk_drv

Then we are good to go. I setup a loopback device on NVMe from QEMU:

atr@u24clean:~/src/ublksrv$ sudo ./ublk add -t loop -f /dev/nvme1n1 
dev id 0: nr_hw_queues 1 queue_depth 128 block size 512 dev_capacity 7837906
	max rq size 524288 daemon pid 16241 flags 0x42 state LIVE
	ublkc: 237:0 ublkb: 259:2 owner: 0:0
	queue 0: tid 16242 affinity(0 1 2 3 4 5 6 7 )
	target {"backing_file":"/dev/nvme1n1","dev_size":4013007872,"direct_io":1,"name":"loop","type":1}
atr@u24clean:~/src/ublksrv$ echo $?
0

ublk is not installed in the local/bin hence I put the build in the $PATH.