Running an application from a FreeBSD jail like a host application

When we install an application on our FreeBSD system, it probably needs dependencies, configuration, and we expect it to coexist well with other applications without causing conflicts. We also expect that this or that other application will not misbehave and will not damage our system or introduce security problems. However, we do not live in an ideal world and these things are likely to happen.

Fortunately FreeBSD provides lightweight virtualization systems also known as jails. Jails solve the problems described above by isolating the applications we need to use. However, not all people feel comfortable creating jails using vanilla jails as it wastes a lot of time, especially configuring other things we may need, such as firewalls, resource limits, network management, and so on, and we will probably need to reproduce the jails on other systems, perhaps with a different configuration. Luckily AppJail exists and provides the automation we need.

In this howto we use a feature of AppJail named Makejail, a simple text file to automate the building of a jail, but the idea is to run an application installed on a jail like an application installed on the host. We will use the Badwolf browser as it is simple and lightweight.

Requirements

Privileges

AppJail needs root access to do its job, but throughout this howto we will try to limit root access to only run the application.

Firewall

Setting up the firewall only takes a moment. Read the AppJail documentation for more details.

Virtual network

Although we can use other networking options that AppJail supports, for simplicity we will use a virtual network. If you don't have a virtual network, don't worry:

# appjail network add -d "Virtual network for testing purposes" testing 10.42.0.0/24
# appjail network list
NAME        NETWORK    CIDR  BROADCAST       GATEWAY    MINADDR    MAXADDR         ADDRESSES  DESCRIPTION
testing     10.42.0.0  24    10.42.0.255     10.42.0.1  10.42.0.1  10.42.0.254     254        Virtual network for testing purposes

testing is the name of the virtual network, 10.42.0.0/24 its address and Virtual network for testing purposes is just a description or a comment, useful for when we are on vacation and do not want a call from a co-worker who needs to know what is the purpose of this virtual network... Feel free to provide anything you need, these values are for simplicity.

Creating the jail

Instead of creating the jail with Badwolf (or another application, remember that Badwolf is just an example to give an idea) from scratch we use a Makejail from the Centralized repository: AppJail-makejails/badwolf. However we need, at least in FreeBSD, to set clear_tmp_X to NO to avoid the removal of x11 sockets in a custom rc.conf(5) file.

mkdir -p /tmp/files/etc
sysrc -f /tmp/files/etc/rc.conf clear_tmp_X=NO

/tmp/files is a directory that mimics the root directory that AppJail will use to copy one or more files. In this case, we create only the /etc directory and put on it our rc.conf(5) file. When we are ready we can create the jail with just one command, but if you don't have the components to create jails yet, don't worry:

appjail fetch

The above will download the components, so you can make a coffee to wait. Once the components have been retrieved and extracted, proceed with the command itself:

appjail makejail -f gh+AppJail-makejails/badwolf -j badwolf \
    -o virtualnet="testing:badwolf default" \
    -o nat \
    -o copydir=/tmp/files \
    -o file=/etc/rc.conf \
    -o x11

With the -f gh+AppJail-makejails/badwolf we indicate to AppJail that we want the Makejail for Badwolf in the Centralized repository. With -j badwolf we indicate the name of the jail. With -o virtualnet="testing:badwolf" default we indicate that we want to use testing as the virtual network for this jail, badwolf (it is just a convention, you can use whatever you want) as the interface name and we make this virtual network the default router using the default option. With the -o nat option we indicate that we want to apply NAT for this jail. With the -o copydir=/tmp/files option we indicate that the /tmp/files directory will be used as copy directory which should mimic the root directory of the file system. With -o file=/etc/rc.conf we indicate that we want to copy /etc/rc.conf which is in the /tmp/files directory to the jail as is. And finally, with -o x11 we indicate that we want to mount the x11 sockets directory in our jail since Badwolf is an x11 application.

# appjail jail list
STATUS  NAME     TYPE  VERSION       PORTS  NETWORK_IP4
DOWN    badwolf  thin  13.1-RELEASE  -      10.42.0.3

The Makejail for Badwolf stops the jail so we need to start it if we want to run Badwolf.

# appjail start badwolf
[00:00:00] [ info  ] [badwolf] Starting badwolf...
access control disabled, clients can connect from any host
ea_badwolf
eb_badwolf
badwolf: created
add net default: gateway 10.42.0.1
defaultrouter: 10.42.0.1 -> 10.42.0.1
# appjail jail list
STATUS  NAME     TYPE  VERSION       PORTS  NETWORK_IP4
UP      badwolf  thin  13.1-RELEASE  -      10.42.0.3

To run Badwolf we can use the badwolf_open custom stage and provide the url argument to open a specific web site.

appjail run -s badwolf_open -p url=http://example.org badwolf

Makejails vs. Command-line

Makejail options can be set either from the command-line or using another Makejail file. A Makejail has advantages over the command-line: it is much easier to transport, it can be included in other Makejails or other Makejails can be included. We can also use Makejail instructions to make our work much easier.

So, for example, let's take the command used in the previous sections:

appjail makejail -f gh+AppJail-makejails/badwolf -j badwolf \
    -o virtualnet="testing:badwolf default" \
    -o nat \
    -o copydir=/tmp/files \
    -o file=/etc/rc.conf \
    -o x11

And convert it to a Makejail:

Makejail:

INCLUDE gh+AppJail-makejails/badwolf

OPTION virtualnet=testing:badwolf default
OPTION nat
OPTION copydir=/tmp/files
OPTION file=/etc/rc.conf
OPTION x11

It may be more useful to separate the networking options into a separate Makejail.

Makejail (v2):

INCLUDE gh+AppJail-makejails/badwolf
INCLUDE options/network.makejail

OPTION copydir=/tmp/files
OPTION file=/etc/rc.conf
OPTION x11

options/network.makejail:

OPTION virtualnet=testing:badwolf default
OPTION nat

To provide another level of modularization, we can use, in other Makejail, miscellaneous options.

Makejail (v3):

INCLUDE gh+AppJail-makejails/badwolf
INCLUDE options/network.makejail
INCLUDE options/misc.makejail

options/misc.makejail:

OPTION copydir=/tmp/files
OPTION file=/etc/rc.conf
OPTION x11

To execute this Makejail we will use the same command but without options:

appjail makejail -j badwolf

Sharing our profile

Simply running an application is not useful when we do not adapt it to our needs. AppJail provides simple ways to configure the application in the jail. We have two options for this application, Badwolf: first, we can copy all the data into a specific directory or completely emulate the behavior of the application and copy the directories using the same path that the application uses. The second, we can mount the directories on the jail. Both offer advantages and disadvantages. If we only copy the directories the application needs, any changes to the files are not reflected on the host, but this method is much simpler. The other method, mounting the directories, provides tracking of which directories are used and provides the other advantage that any changes to the files are reflected both on the jail and on the host.

copy

The Makejail for Badwolf automatically creates a user named badwolf whose home directory is /home/badwolf. We need to create /home/badwolf/.config/badwolf and /home/badwolf/.local/share/badwolf and copy the files from the host to the jail. We also need to change the owner and group of those files after the copy process to match the badwolf user.

INCLUDE gh+AppJail-makejails/badwolf
INCLUDE options/network.makejail
INCLUDE options/misc.makejail

COPY home
CMD --local-jaildir chown -fR 1001:1001 home/badwolf

As you can see, only two instructions have been added to the Makejail used in previous sections. COPY will copy a file or directory from the host to the jail using the exact path tree (see below). CMD with the --local-jaildir parameter will execute a command but on the host inside the jail directory. As mentioned, we need to change the owner and group to match the badwolf user, but remember that UIDs/GIDs in a jail are not the same as in the host, so we must explicitly set them using 1001 as uid and 1001 as gid, all of which correspond to badwolf and badwolf. The tree in the home directory is as follows:

tree -apug home
[drwxr-xr-x root     wheel   ]  home
└── [drwxr-xr-x root     wheel   ]  badwolf
    ├── [drwxr-x--- root     wheel   ]  .config
    │   └── [drwxr-xr-x root     wheel   ]  badwolf
    │       └── [-rw-r----- root     wheel   ]  content-filters.json
    └── [drwxr-xr-x root     wheel   ]  .local
        └── [drwxr-xr-x root     wheel   ]  share
            └── [drwxr-x--- root     wheel   ]  badwolf
                ├── [-rw-r----- root     wheel   ]  bookmarks-test.xbel
                ├── [-rw-r----- root     wheel   ]  bookmarks.xbel
                ├── [-rw-r----- root     wheel   ]  interface.css
                └── [drwxr-x--- root     wheel   ]  webkit-web-extension

8 directories, 4 files

There is no difference between the old command used to build, start and run the jail, just use the same commands:

appjail makejail -j badwolf
appjail start badwolf
appjail run -s badwolf_open -p url=http://example.org badwolf

As you can see, the files load correctly just as they do when the application is running on the host.

mount

Although this option is a bit more complex, when you understand it, you feel comfortable with it.

INCLUDE gh+AppJail-makejails/badwolf
INCLUDE options/network.makejail
INCLUDE options/misc.makejail

CMD --local-jaildir mkdir -p home/badwolf/.config/badwolf
CMD --local-jaildir mkdir -p home/badwolf/.local/share/badwolf
# Unnecessary except when we need the correct permissions when the directories we
# are unmounted.
CMD --local-jaildir chown -f 1001:1001 home/badwolf/.config
CMD --local-jaildir chown -f 1001:1001 home/badwolf/.local/share/badwolf

MOUNT --nomount /var/appjails/badwolf/config /home/badwolf/.config/badwolf
MOUNT --nomount /var/appjails/badwolf/datadir /home/badwolf/.local/share/badwolf
# The current owner and group when the directories are mounted are not the same
# as when they are unmounted, we need to run chown(8) again.
CMD --local chown -fR 1001:1001 /var/appjails/badwolf/config
CMD --local chown -fR 1001:1001 /var/appjails/badwolf/datadir

First, we create the structure used by Badwolf in the home directory of the badwolf user. Although we change the owner and group of the directories before mounting them, it is not necessary in most cases except when we need to use those directories without mounting other directories on them. MOUNT is used to create an entry in the badwolf jail's fstab. With the --nomount we indicate to the MOUNT instruction that we do not want to compile the entries and mount the devices, this is necessary since the badwolf jail has not been started. When the jail starts, those directories will be mounted automatically. Finally, we need to change the owner and group of the /var/appjails/badwolf/config and /var/appjails/badwolf/datadir directories to match the badwolf user and group. The tree structure of the /var/appjails/badwolf directory is as follows:

tree -pug /var/appjails/badwolf
[drwxr-xr-x root     wheel   ]  /var/appjails/badwolf
├── [drwxr-xr-x root     wheel   ]  config
│   └── [-rw-r----- root     wheel   ]  content-filters.json
└── [drwxr-x--- root     wheel   ]  datadir
    ├── [-rw-r----- root     wheel   ]  bookmarks-test.xbel
    ├── [-rw-r----- root     wheel   ]  bookmarks.xbel
    ├── [-rw-r----- root     wheel   ]  interface.css
    └── [drwxr-x--- root     wheel   ]  webkit-web-extension

4 directories, 4 files

Repeat the process again: build, start and run:

appjail makejail -j badwolf
appjail start badwolf
appjail run -s badwolf_open -p url=http://example.org badwolf

No privileges

Up to this point we only run the application with root access but for some users or some environments it will probably be necessary to reduce privileges. We don't want to complicate things, so we need four things: first, security/doas, second, /usr/local/bin/badwolf.appjail a script to run, third, /usr/local/bin/scripts/badwolf-jail.sh and fourth a rule for doas.conf(5). security/doas is used to execute commands as another user. /usr/local/bin/badwolf.appjail has the sole responsability to execute /usr/local/bin/scripts/badwolf-jail.sh as root. /usr/local/bin/scripts/badwolf-jail.sh is used to execute the custom badwolf_open stage of the badwolf jail. Our doas.conf(5) rule allows /usr/local/bin/scripts/badwolf-jail.sh to run as root.

/usr/local/bin/badwolf.appjail:

#!/bin/sh

doas /usr/local/bin/scripts/badwolf-jail.sh "$1"

/usr/local/bin/scripts/badwolf-jail.sh:

#!/bin/sh

appjail run -s badwolf_open \
        -p display="${DISPLAY}" \
        -p url="$1" \
        badwolf

/usr/local/etc/doas.conf:

permit nopass dtxdf-fbsd cmd /usr/local/bin/scripts/badwolf-jail.sh

Now we can run badwolf.appjail as a non-root user and there are no problems:

badwolf.appjail

Multi-user environment

Throughout this howto we have used a single user to run the application. For some users this is sufficient, but for others, especially in a multi-user environment, it is probably not. To make things more complicated we can mimic exactly the behavior of the application, copying or mounting exactly the path the application needs just for the user running the application. We also create the user in the jail if it does not exist with the exact name as in the host. In our doas.conf(5) rules, instead of specifying a single user, we can use a group named appjail.

copy

Makejail:

INCLUDE gh+AppJail-makejails/badwolf
INCLUDE options/network.makejail
INCLUDE options/misc.makejail

CMD --local echo "======> Installing dependencies (host) <======"
PKG --local doas

CMD --local echo "======> Installing scripts <======"

CMD --local mkdir -p /usr/local/bin
CMD --local cp scripts/badwolf.appjail /usr/local/bin
CMD --local mkdir -p /usr/local/bin/scripts
CMD --local cp scripts/rjail.sh /usr/local/bin/scripts
CMD --local cp scripts/ajcopy.sh /usr/local/bin/scripts

CMD --local echo "======> Creating group 'appjail' <======"
CMD --local pw groupadd -n appjail 2> /dev/null || :
CMD --local chown -R root:appjail /usr/local/bin/badwolf.appjail

CMD --local echo "======> Done <======"

CMD --local echo
CMD --local echo "===> Note #1: Add the following rules to your doas.conf(5): <==="
CMD --local echo
CMD --local echo "# badwolf.appjail"
CMD --local echo "permit nopass :appjail cmd /usr/local/bin/scripts/rjail.sh"
CMD --local echo "permit nopass :appjail cmd /usr/local/bin/scripts/ajcopy.sh"

CMD --local echo

CMD --local echo "===> Note #2: Add your user to the 'appjail' group <==="
CMD --local echo

scripts/ajcopy.sh:

#!/bin/sh

# see sysexits(3)
EX_USAGE=64

main()
{
	local _o
	local dst jail src user

	while getopts ":d:j:s:u:" _o; do
		case "${_o}" in
			d)
				dst="${OPTARG}"
				;;
			j)
				jail="${OPTARG}"
				;;
			s)
				src="${OPTARG}"
				;;
			u)
				user="${OPTARG}"
				;;
			*)
				usage
				;;
		esac
	done

	if [ -z "${dst}" -o -z "${jail}" -o -z "${src}" -o -z "${user}" ]; then
		usage
	fi

	# copy
	appjail cmd local "${jail}" \
		cp -a "${src}" "${dst}"
	# owner and group
	appjail cmd jexec "${jail}" \
		chown -R "${user}:${user}" "/${dst}"
}

usage()
{
	echo "usage: ajcopy.sh -d dst -j jail -s src -u user" >&2
	exit ${EX_USAGE}
}

main "$@"

scripts/badwolf.appjail:

#!/bin/sh

# scripts
RJAIL="/usr/local/bin/scripts/rjail.sh"
AJCOPY="/usr/local/bin/scripts/ajcopy.sh"

# config
JAIL="badwolf"
RUNAS="${RUNAS:-doas}"

case "${RUNAS}" in
	sudo|doas|qsudo) ;;
	*) echo "Only \"sudo, doas or qsudo\" are allowed." >&2; exit 1
esac

if ! which -s "${RUNAS}"; then
	echo "Command \"${RUNAS}\" not found." >&2
	exit 1
fi

# user information
USER=`id -un`
REAL_HOMEDIR=`getent passwd "${USER}" 2> /dev/null | cut -d: -f6`
CONFIGDIR="${REAL_HOMEDIR}/.config/badwolf"
DATADIR="${REAL_HOMEDIR}/.local/share/badwolf"

if [ -d "${CONFIGDIR}" ]; then
	${RUNAS} "${RJAIL}" -j "${JAIL}" -u "${USER}" \
		mkdir -p "/home/${USER}/.local/share/badwolf"
	${RUNAS} "${AJCOPY}" -j "${JAIL}" -u "${USER}" \
		-s "${DATADIR}/" -d "home/${USER}/.local/share/badwolf"
fi

if [ -d "${DATADIR}" ]; then
	${RUNAS} "${RJAIL}" -j "${JAIL}" -u "${USER}" \
		mkdir -p "/home/${USER}/.config/badwolf"
	${RUNAS} "${AJCOPY}" -j "${JAIL}" -u "${USER}" \
		-s "${CONFIGDIR}/" -d "home/${USER}/.config/badwolf"
fi

${RUNAS} "${RJAIL}" -j "${JAIL}" -u "${USER}" \
	env DISPLAY="${DISPLAY}" badwolf "$@"

scripts/rjail.sh:

#!/bin/sh

# see sysexits(3)
EX_USAGE=64
EX_NOUSER=67

main()
{
	local _o
	local jail= user=

	while getopts ":j:u:" _o; do
		case "${_o}" in
			j)
				jail="${OPTARG}"
				;;
			u)
				user="${OPTARG}"
				;;
			*)
				usage
				;;
		esac
	done
	shift $((OPTIND-1))

	if [ -z "${jail}" -o -z "${user}" -o $# -eq 0 ]; then
		usage
	fi

	local errcode

	check_user "${jail}" "${user}"

	errcode=$?

	if [ ${errcode} -eq 1 ]; then
		create_user "${jail}" "${user}"
	elif [ ${errcode} -eq 2 ]; then
		echo "An unknown error has been occurred: ${ERRNO}"
		return ${ERRNO}
	fi

	run_cmd "${jail}" "${user}" "$@"
}

check_user()
{
	local jail="$1" user="$2"

	if [ -z "${jail}" -o -z "${user}" ]; then
		echo "usage: check_user jail user" >&2
		return ${EX_USAGE}
	fi

	local errcode

	appjail cmd jexec "${jail}" \
		pw usershow -n "${user}" > /dev/null 2>&1

	errcode=$?

	if [ ${errcode} -eq 0 ]; then
		return 0
	elif [ ${errcode} -eq ${EX_NOUSER} ]; then
		return 1
	else
		ERRNO=${errcode}
		return 2
	fi
}

create_user()
{
	local jail="$1" user="$2"

	if [ -z "${jail}" -o -z "${user}" ]; then
		echo "usage: create_user jail user" >&2
		return ${EX_USAGE}
	fi

	appjail cmd jexec "${jail}" \
		pw useradd -n "${user}" -md "/home/${user}" -s /usr/sbin/nologin || return $?

	return 0
}

run_cmd()
{
	local jail="$1" user="$2" cmd="$3"

	if [ -z "${jail}" -o -z "${user}" -o -z "${cmd}" ]; then
		echo "usage: run_cmd jail user cmd args..." >&2
		return ${EX_USAGE}
	fi

	shift 3 # > jail > user > cmd

	appjail cmd jexec "${jail}" -U "${user}" -- \
		"${cmd}" "$@"
}

usage()
{
	echo "usage: rjail.sh -j jail -u user cmd args..." >&2
	exit ${EX_USAGE}
}

main "$@"

Repeat the process again: build, start but instead of using AppJail explicitly, we can run badwolf.appjail with a non-root user.

mount

Makejail:

INCLUDE gh+AppJail-makejails/badwolf
INCLUDE options/network.makejail
INCLUDE options/misc.makejail

CMD --local echo "======> Installing dependencies (host) <======"
PKG --local doas

CMD --local echo "======> Installing scripts <======"

CMD --local mkdir -p /usr/local/bin
CMD --local cp scripts/badwolf.appjail /usr/local/bin
CMD --local mkdir -p /usr/local/bin/scripts
CMD --local cp scripts/rjail.sh /usr/local/bin/scripts
CMD --local cp scripts/ajnullfs_badwolf.sh /usr/local/bin/scripts

CMD --local echo "======> Creating group 'appjail' <======"
CMD --local pw groupadd -n appjail 2> /dev/null || :
CMD --local chown -R root:appjail /usr/local/bin/badwolf.appjail

CMD --local echo "======> Done <======"

CMD --local echo
CMD --local echo "===> Note #1: Add the following rules to your doas.conf(5): <==="
CMD --local echo
CMD --local echo "# badwolf.appjail"
CMD --local echo "permit nopass :appjail cmd /usr/local/bin/scripts/rjail.sh"
CMD --local echo "permit nopass :appjail cmd /usr/local/bin/scripts/ajnullfs_badwolf.sh"

CMD --local echo

CMD --local echo "===> Note #2: Add your user to the 'appjail' group <==="
CMD --local echo

scripts/badwolf.appjail:

#!/bin/sh

# scripts
RJAIL="/usr/local/bin/scripts/rjail.sh"; export RJAIL
AJNULLFS="/usr/local/bin/scripts/ajnullfs_badwolf.sh"

# config
JAIL="badwolf"
RUNAS="${RUNAS:-doas}"

case "${RUNAS}" in
	sudo|doas|qsudo) ;;
	*) echo "Only \"sudo, doas or qsudo\" are allowed." >&2; exit 1
esac

if ! which -s "${RUNAS}"; then
	echo "Command \"${RUNAS}\" not found." >&2
	exit 1
fi

# user information
USER=`id -un`

${RUNAS} "${AJNULLFS}" -j "${JAIL}" -u "${USER}"

${RUNAS} "${RJAIL}" -j "${JAIL}" -u "${USER}" \
	env DISPLAY="${DISPLAY}" badwolf "$@"

scripts/ajnullfs_badwolf.sh:

#!/bin/sh

# see sysexits(3)
EX_USAGE=64

# scripts
RJAIL="${RJAIL:-/usr/local/bin/scripts/rjail.sh}"

main()
{
	local _o
	local jail user

	while getopts ":j:u:" _o; do
		case "${_o}" in
			j)
				jail="${OPTARG}"
				;;
			u)
				user="${OPTARG}"
				;;
			*)
				usage
				;;
		esac
	done

	if [ -z "${jail}" -o -z "${user}" ]; then
		usage
	fi

	local nro nrofile="/home/${user}/.nro"

	# nro
	if appjail cmd jexec "${jail}" test -f "${nrofile}"; then
		nro=`appjail cmd jexec "${jail}" head -1 "${nrofile}"`
	else
		nro=`jot -r 1 1000 1000000000`
		"${RJAIL}" -j "${jail}" -u "${user}" \
			sh -c "echo ${nro} > ${nrofile}"
	fi

	# user information
	local real_homedir=`getent passwd "${user}" 2> /dev/null | cut -d: -f6`
	local configdir="${real_homedir}/.config/badwolf"
	local datadir="${real_homedir}/.local/share/badwolf"

	# config
	if [ -d "${configdir}" ]; then
		"${RJAIL}" -j "${jail}" -u "${user}" \
			mkdir -p "/home/${user}/.config/badwolf"
		appjail fstab jail "${jail}" \
			set -n ${nro} -d "${configdir}" -m "/home/${user}/.config/badwolf"
	fi
	
	# data
	if [ -d "${datadir}" ]; then
		"${RJAIL}" -j "${jail}" -u "${user}" \
			mkdir -p "/home/${user}/.local/share/badwolf"
		appjail fstab jail "${jail}" \
			set -n $((nro+1)) -d "${datadir}" -m "/home/${user}/.local/share/badwolf"
	fi

	# compile and mount
	appjail fstab jail "${jail}" compile
	appjail fstab jail "${jail}" mount -a

	#
	# We will have to change the owner and group of the newly mounted directories to
	# match the user and group of the jail.
	#
	
	if appjail cmd jexec "${jail}" test -d "${configdir}"; then
		appjail cmd jexec "${jail}" \
			chown -R "${user}:${user}" "/home/${user}/.config/badwolf"
	fi

	if appjail cmd jexec "${jail}" test -d "${datadir}"; then
		appjail cmd jexec "${jail}" \
			chown -R "${user}:${user}" "/home/${user}/.local/share/badwolf"
	fi
}

usage()
{
	echo "usage: ajnullfs_badwolf.sh -j jail -u user" >&2
	exit ${EX_USAGE}
}

main "$@"

scripts/rjail.sh: it is the same as using the copy option.

Repeat the process again: build, start but instead of using AppJail explicitly, we can run badwolf.appjail with a non-root user.

After running the script we will have another user and we will also have more entries in the fstab.

Afterword

Create a Makejail, automate everything.