Environment Loading - blitterated/docker-dev-env-s6 GitHub Wiki

Why does this article exist? I was having issues with s6 and docker exec. Exec'ing a shell against a daemonized container was leaving off /command from the path. I didn't know enough about Linux initialization, so I was asking wrong questions like "why doesn't docker exec load environment values from .bashrc or from profile.d?

I needed to understand the following:

Shell types
- What's an interactive shell?
- What's a login shell?
- Are permutations of the above possible?
How does docker exec work?
Evironment Loading
- What are the files used to load environment variables?
- When are they and their vars/values loaded?
- Is there a difference with the above two between Linux and Docker?

Shell Types

There are a couple of important shell qualities to learn about. Shells can be either Login or Nonlogin. And shells can either be Interactive or Noninteractive.

Login/Nonlogin

Login Shells

Login shells are created when you log on to a system from a terminal. In other words, you're at a login prompt, you enter your credentials, and now your at a shell prompt.

These can be created by:

logging in to an actual serial terminal connected to a port
logging into a virtual terminal running a /bin/getty process
logging in via ssh or telnet

Nonlogin Shells

Nonlogin shells are created by starting a new subshell from a login shell, e.g. running bash or sh from a prompt. They can also be created by opening a terminal session in a windowing system where you're already logged in.

Interactive/Noninteractive

Interactive Shells

An Interactive shell is meant to process commands from the command line as they're typed in. They also have STDOUT and STDERR bound to the shell's terminal by default.

Noninteractive Shells

A Noninteractive shell is typically a shell that is processing commands in a script or directly passed to the shell. It's a shell that the user won't be interacting with directly.

These can be created by:

passing a shell script as an argument on the shell's invocation
- bash somescript.sh
passing a command as an argument on the shell's invocation
- bash -c 'echo foo'

Four Permutations

The answer to this StackExchange question sums these up quite well:

interactive login shell: You log into a remote computer via, for example ssh. Alternatively, you drop to a tty on your local machine (Ctrl+Alt+F1) and log in there.

interactive non-login shell: Open a new terminal.

non-interactive non-login shell: Run a script. All scripts run in their own subshell and this shell is not interactive. It only opens to execute the script and closes immediately once the script is finished.

non-interactive login shell: This is extremely rare, and you're unlikey to encounter it. One way of launching one is echo command | ssh server. When ssh is launched without a command (so ssh instead of ssh command which will run command on the remote shell) it starts a login shell. If the stdin of the ssh is not a tty, it starts a non-interactive shell. This is why echo command | ssh server will launch a non-interactive login shell. You can also start one with bash -l -c command.

Detecting shell type

The same StackExchange question shows how we can detect the shell's type from the command line.

You can test for the various types of shell as follows:

Is this shell interactive?

Check the contents of the $- variable. For interactive shells, it will include i:

## Normal shell, just running a command in a terminal: interactive
$ echo $-
himBHs

## Non interactive shell
$ bash -c 'echo $-'
hBc

Is this a login shell?

There is no portable way of checking this but, for bash, you can check if the login_shell option is set:

## Normal shell, just running a command in a terminal: interacive
shopt login_shell
login_shell     off

# Login shell;
ssh localhost
shopt login_shell
login_shell     on

Putting all this together, here's one of each possible type of shell:

  ## Interactive, non-login shell. Regular terminal
  $ echo $-; shopt login_shell
  himBHs
  login_shell     off

  ## Interactive login shell
  $ bash -l
  $ echo $-; shopt login_shell
  himBHs
  login_shell     on

   ## Non-interactive, non-login shell
  $ bash -c 'echo $-; shopt login_shell'
  hBc
  login_shell     off

   ## Non-interactive login shell
  $ echo 'echo $-; shopt login_shell' | ssh localhost
  Pseudo-terminal will not be allocated because stdin is not a terminal.
  hBs
  login_shell     on

The last example, Non-interactive login shell, could also be demonstrated with:

bash -lc 'echo $-; shopt login_shell'

References

How docker exec works

Image for experiments

build an image

docker build -t dckr_exec -<<"EOB"
FROM ubuntu

RUN apt update && apt --yes upgrade && \
    apt install --yes htop psmisc

EOB

examine process tree in a container

start up a daemon container

We'll fire up a container running /bin/sh. -d will run it in daemon mode, and -t will attach a tty to keep the process from exiting immediately.

CNT_ID=$(docker run -td --rm dckr_exec /bin/sh)

Alternatively, we can run a container with the additional --init flag to have the tini based docker-init running as PID 1 instead.

CNT_ID=$(docker run -td --rm --init dckr_exec /bin/sh)

How to keep Docker container running after starting services?

Open a bash shell on the daemon container with `docker exec`.

docker exec -it $CNT_ID /bin/bash

Let's try different ways to view the process tree within the container.

htop

Run htop and switch to process tree view (F5). Note that /bin/bash is a top level process next to the original /bin/sh.

  PID USER      PRI  NI  VIRT   RES   SHR S CPU% MEM%   TIME+  Command
    1 root       20   0  2616   608   540 S  0.0  0.0  0:00.02 /bin/sh
    8 root       20   0  4116  3412  2960 S  0.0  0.2  0:00.02 /bin/bash
   16 root       20   0  5512  4080  2888 R  0.0  0.2  0:00.05 `- htop

Here's htop again, but with the --init flag passed to docker run. Note that /bin/bash is again a top level process, and doesn't fall under /sbin/docker-init.

  PID USER      PRI  NI  VIRT   RES   SHR S CPU% MEM%   TIME+  Command
    1 root       20   0  1020     4     0 S  0.0  0.0  0:00.03 /sbin/docker-init -- /bin/sh
    7 root       20   0  2616   612   540 S  0.0  0.0  0:00.00 `- /bin/sh
    8 root       20   0  4116  3488  3032 S  0.0  0.2  0:00.02 /bin/bash
   16 root       20   0  5512  4052  2868 R  0.0  0.2  0:00.05 `- htop

pstree

pstree is part of the psmisc package. It shows a view similar to htop's tree view, but it makes the assumption that all processes descend from PID 1. That means that we won't see any processes in the container started by docker exec...

root@e3369ab4bc21:/# pstree
docker-init---sh

... unless we employ a trick. We'll give pstree the PID of 0.

root@e3369ab4bc21:/# pstree 0
?-+-bash---pstree
  `-docker-init---sh

ps

Here's some interesting flags for ps.

-a        Write information for all processes associated with terminals.
-e        Write information for all processes.
-f        Generate a full listing (PID & PPID)
--forest  Show listing as a tree.

The results aren't quite as nice as the above two options though. But with PID and PPID, we get what we need.

root@e3369ab4bc21:/# ps -aef --forest
UID        PID  PPID  C STIME TTY          TIME CMD
root         8     0  0 04:13 pts/1    00:00:00 /bin/bash
root        31     8  0 22:47 pts/1    00:00:00  \_ ps -aef --forest
root         1     0  0 04:13 pts/0    00:00:09 /sbin/docker-init -- /bin/sh
root         7     1  0 04:13 pts/0    00:00:00 /bin/sh

References

examine process tree from outside a container

Since I'm on a Mac, I'll need nsenter1 to poke around inside the LinuxKit instance hosting the container.

Take the Red Pill

Open a different terminal and run the following:

docker run -it --rm --privileged --pid=host justincormack/nsenter1

Run pstree with -p for PIDs. It's all containerd related process unless you run pstree 0 to see the kernel threads as well.

pstree -p

init(1)-+-containerd(870)
        |-containerd-shim(890)
        |-containerd-shim(940)---allowlist(960)
        |-containerd-shim(985)
        |-containerd-shim(1032)---devenv-server(1064)
        |-containerd-shim(1098)---dhcpcd(1120)
        |-containerd-shim(1147)---diagnosticsd(1169)---sh(1238)
        |-containerd-shim(1193)---dns-forwarder(1219)
        |-containerd-shim(1248)-+-containerd-shim(25383)---sh(25408)---pstree(25780)
        |                       |-containerd-shim(25697)-+-bash(25765)
        |                       |                        `-docker-init(25723)---sh(25754)
        |                       |-docker-init(1271)---entrypoint.sh(1284)---logwrite(1311)---lifecycle-serve(1316)-+-logwrit+
        |                       |                                                                                  `-logwrit+
        |                       |-rpc.statd(1430)
        |                       `-rpcbind(1403)
        |-containerd-shim(1293)---http-proxy(1326)
        |-containerd-shim(1374)
        |-containerd-shim(1493)---start(1524)---sntpc(1543)
        |-containerd-shim(1551)
        |-containerd-shim(1600)---trim-after-dele(1628)
        |-containerd-shim(1656)---volume-contents(1677)
        |-containerd-shim(1708)---vpnkit-forwarde(1727)
        |-logwrite(361)---vpnkit-bridge(370)
        |-logwrite(364)---procd(376)
        |-memlogd(352)
        |-rungetty.sh(334)---login(336)---sh(340)
        `-rungetty.sh(343)---login(344)---sh(363)

Let's look closer at the containerd-shim running the processes we expect to see.

pstree 25697

containerd-shim-+-bash
                `-docker-init---sh

And there's the sh process we started with docker run --init and the bash process we started with docker exec, both visible from the hosting linux instance the container and docker is running under.

Let's try one more time but with PIDs

pstree -p 25697

containerd-shim(25697)-+-bash(25765)
                       `-docker-init(25723)---sh(25754)

References

How to know a process is running under docker?

So what kind of shell does using `docker exec` get us?

Back in our bash shell run via docker exec, run the following:

echo $-; shopt login_shell

Turns out it's non-login, interactive.

himBHs
login_shell    	off

Here we run an addition docker exec and pass in the commands to a bash process with -c.

docker exec $CNT_ID /bin/bash -c 'echo $-; shopt login_shell'

Now it's non-login, non-interactive.

hBc
login_shell    	off

Conclusions

An important takeaway is that the process running as PID 1 inside the container, docker-init above since we used the --init flag, is actually just another process, 25723, in the process tree in the hosting linux kernel.

Then there's this little tidbit from a docker blog post regarding containerd:

Since there is no such thing as Linux containers in the kernelspace, containers are various kernel features tied together, when you are building a large platform or distributed system you want an abstraction layer between your management code and the syscalls and duct tape of features to run a container. That is where containerd lives. It provides a client layer of types that platforms can build on top of without ever having to drop down to the kernel level. It’s so much nicer towork with Container, Task, and Snapshot types than it is to manage calls to clone() or mount().

So it would appear the docker daemon simply adds a process to a paired cgroup and namespace that make up a container when using docker exec via containerd. Now we can see why both docker-init and bash were top level processes when viewed from within the container.

How the Environment Loads

Let's take a look at how an Ubuntu based distribution loads and configures PATH and the environment. Let's also see how this all behaves while loading Ubuntu in a Docker container.

login program

Found in /etc/login.defs on Ubuntu:

# *REQUIRED*  The default PATH settings, for superuser and normal users.
#
# (they are minimal, add the rest in the shell startup files)
ENV_SUPATH      PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
ENV_PATH        PATH=/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games

Looks like it matches ENV_SUPATH.

root@d2d500bcea44:/# echo $PATH
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
root@d2d500bcea44:/# whoami
root

Add /root to ENV_SUPATH

Startup a container.

docker run -it --name login_defs ubuntu /bin/bash

Make note of the value for $PATH.

root@46419531bf79:/# echo $PATH
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

Add an editor.

apt update
apt --yes install vim

Add /root to ENV_SUPATH in /etc/login.defs.

ENV_SUPATH      PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/root

Exit and restart the container.

exit
docker start -ai login_defs

Examine $PATH.

root@46419531bf79:/# echo $PATH
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

There's no /root

Detach with Ctrl-p, Ctrl-q and try running a new shell with docker exec.

docker exec -it login_defs /bin/bash

Examine $PATH.

root@46419531bf79:/# echo $PATH
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

Hmm, still no /root.

Exit out of, stop, and remove the container.

exit

docker stop login_defs
docker rm login_defs

Looks like Docker doesn't use login.

References

PAM

PAM authentication uses pam_env.so to load the environment each time a user logs in.

/etc/environment - read during any login.
/etc/security/pam_env.conf - read during any login. can override /etc/environment.
~/.pam_environment - read during a specific user's login.

On Ubuntu in Docker

In /etc/environment there is a path specified, but it doesn't match what we saw above with login.

root@05af7653f98f:/# cat /etc/environment
PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin"

In /etc/security/pam_env.conf the PATH line is disabled.

root@05af7653f98f:/# cat /etc/security/pam_env.conf | grep PATH
# be useful to be set: NNTPSERVER, LESS, PATH, PAGER, MANPAGER .....
#PATH		DEFAULT=${HOME}/bin:/usr/local/bin:/bin\

There is no ~/.pam_environment.

It does not seem like anything uses PAM in Ubuntu's minimal image.

References

bash startup files

Login shells
- First, read and execute commands from /etc/profile if it exists
  - On a few distributions, /etc/profile contains code to also scan /etc/profile.d for *.sh and run the scripts that match.
  - On a few distributions, /etc/profile contains code to also source /etc/bash.bashrc.
- Next, read and execute commands from the first found in the following:
  - ~/.bash_profile
  - ~/.bash_login
  - ~/.profile
- Finally, evaluate $BASH_ENV to a fully qualified filename, and source that file.
Nonlogin shells
- interactive
  - read and execute commands from /etc/bash.bashrc if it exists
  - read and execute commands from ~/.bashrc if it exists
- non-interactive
  - Evaluate $BASH_ENV to a fully qualified filename, and source that file.

Other considerations:

bash can be configured to detect if it's being run by sshd or rshd and consequently source both /etc/bash.bashrc and ~/.bashrc. source
bash behaves differently if it detects it was run as sh.
bash behaves differently if run in posix mode.

Testing out how `bash` loads startup files

Fire up a basic Ubuntu container.

docker run -it --rm ubuntu /bin/bash

Setup a few files to log their access to the console.

echo "echo BASH_ENV" > /root/benv.sh
export BASH_ENV=/root/benv.sh
echo "echo .bashrc" > /root/.bashrc
echo "echo .bash_profile" > /root/.bash_profile
echo "echo /etc/profile" >> /etc/profile
echo "echo /etc/bash.bashrc" >> /etc/bash.bashrc

Run each of the following in order, one by one. Be sure to exit the interactive shells.

bash # non-login / interactive

bash -c "echo non-login / non-interactive"

bash -l # login / interactive

bash -lc "echo login / non-interactive"

Here's the output to show the results.

root@95e9aa7c230a:~# bash # non-login / interactive
/etc/bash.bashrc
.bashrc
root@95e9aa7c230a:~# exit
exit

root@95e9aa7c230a:~# bash -c "echo non-login / non-interactive"
BASH_ENV
non-login / non-interactive

root@95e9aa7c230a:~# bash -l # login / interactive
/etc/bash.bashrc
/etc/profile
.bash_profile
root@95e9aa7c230a:~# exit
logout

root@95e9aa7c230a:~# bash -lc "echo login / non-interactive"
/etc/profile
.bash_profile
BASH_ENV
login / non-interactive

NOTE: on Ubuntu /etc/profile sources /etc/bash.bashrc. We only appended an echo statement to the end of each script. That's why /etc/bash.bashrc appears under "login / interactive".

Interesting bits from bash's source code

shell.c line 1056

/* Source the bash startup files.  If POSIXLY_CORRECT is non-zero, we obey
   the Posix.2 startup file rules:  $ENV is expanded, and if the file it
   names exists, that file is sourced.  The Posix.2 rules are in effect
   for interactive shells only. (section 4.56.5.3) */

shell.c line 1061

/* Execute ~/.bashrc for most shells.  Never execute it if
   ACT_LIKE_SH is set, or if NO_RC is set.

   If the executable file "/usr/gnu/src/bash/foo" contains:

   #!/usr/gnu/bin/bash
   echo hello

   then:

	 COMMAND	    EXECUTE BASHRC
	 --------------------------------
	 bash -c foo		NO
	 bash foo		NO
	 foo			NO
	 rsh machine ls		YES (for rsh, which calls `bash -c')
	 rsh machine foo	YES (for shell started by rsh) NO (for foo!)
	 echo ls | bash		NO
	 login			NO
	 bash			YES
*/

shell.c line 1140

  /* A shell begun with the --login (or -l) flag that is not in posix mode
     runs the login shell startup files, no matter whether or not it is
     interactive.  If NON_INTERACTIVE_LOGIN_SHELLS is defined, run the
     startup files if argv[0][0] == '-' as well. */
#if defined (NON_INTERACTIVE_LOGIN_SHELLS)
  if (login_shell && posixly_correct == 0)
#else
  if (login_shell < 0 && posixly_correct == 0)
#endif
    {
      /* We don't execute .bashrc for login shells. */
      no_rc++;

      /* Execute /etc/profile and one of the personal login shell
	 initialization files. */
      if (no_profile == 0)
	{
	  maybe_execute_file (SYS_PROFILE, 1);

	  if (act_like_sh)	/* sh */
	    maybe_execute_file ("~/.profile", 1);
	  else if ((maybe_execute_file ("~/.bash_profile", 1) == 0) &&
		   (maybe_execute_file ("~/.bash_login", 1) == 0))	/* bash */
	    maybe_execute_file ("~/.profile", 1);
	}

      sourced_login = 1;
    }

  /* A non-interactive shell not named `sh' and not in posix mode reads and
     executes commands from $BASH_ENV.  If `su' starts a shell with `-c cmd'
     and `-su' as the name of the shell, we want to read the startup files.
     No other non-interactive shells read any startup files. */
  if (interactive_shell == 0 && !(su_shell && login_shell))
    {
      if (posixly_correct == 0 && act_like_sh == 0 && privileged_mode == 0 &&
	    sourced_env++ == 0)
	execute_env_file (get_string_value ("BASH_ENV"));
      return;
    }

  /* Interactive shell or `-su' shell. */
  if (posixly_correct == 0)		  /* bash, sh */
    {
      if (login_shell && sourced_login++ == 0)
	{
	  /* We don't execute .bashrc for login shells. */
	  no_rc++;

	  /* Execute /etc/profile and one of the personal login shell
	     initialization files. */
	  if (no_profile == 0)
	    {
	      maybe_execute_file (SYS_PROFILE, 1);

	      if (act_like_sh)	/* sh */
		maybe_execute_file ("~/.profile", 1);
	      else if ((maybe_execute_file ("~/.bash_profile", 1) == 0) &&
		       (maybe_execute_file ("~/.bash_login", 1) == 0))	/* bash */
		maybe_execute_file ("~/.profile", 1);
	    }
	}

      /* bash */
      if (act_like_sh == 0 && no_rc == 0)
	{
#ifdef SYS_BASHRC
#  if defined (__OPENNT)
	  maybe_execute_file (_prefixInstallPath(SYS_BASHRC, NULL, 0), 1);
#  else
	  maybe_execute_file (SYS_BASHRC, 1);
#  endif
#endif
	  maybe_execute_file (bashrc_file, 1);
	}
      /* sh */
      else if (act_like_sh && privileged_mode == 0 && sourced_env++ == 0)
	execute_env_file (get_string_value ("ENV"));
    }
  else		/* bash --posix, sh --posix */
    {
      /* bash and sh */
      if (interactive_shell && privileged_mode == 0 && sourced_env++ == 0)
	execute_env_file (get_string_value ("ENV"));
    }

References

dash startup files

Ubuntu defaults to dash as the system's shell. dash is really ash with better POSIX conformance and better performance than bash. A few distributions, like Debian and Ubuntu, use it in place of sh.

Login shells
- First, read and execute commands from /etc/profile if it exists
- Next, read and execute commands from ~/.profile if it exists
- Finally, evaluate $BASH_ENV to a fully qualified filename, and source that file.
Nonlogin shells
- interactive
  - read and execute commands from /etc/bash.bashrc if it exists
  - read and execute commands from ~/.bashrc if it exists
- non-interactive
  - Evaluate $BASH_ENV to a fully qualified filename, and source that file.

Testing out how `bash` loads startup files

Fire up a basic Ubuntu container.

docker run -it --rm ubuntu /bin/dash

Setup a few files to log their access to the console.

echo "echo ENV file" > /root/denv.sh
export ENV=/root/denv.sh
echo "echo .profile" > /root/.profile
echo "echo /etc/profile" >> /etc/profile

Run each of the following in order, one by one. Be sure to exit the interactive shells.

dash # non-login / interactive

dash -c "echo non-login / non-interactive"

dash -l # login / interactive

dash -lc "echo login / non-interactive"

Here's the output to show the results.

# dash # non-login / interactive
ENV file
# exit

# dash -c "echo non-login / non-interactive"
non-login / non-interactive

# dash -l # login / interactive
/etc/profile
.profile
ENV file
# exit

# dash -lc "echo login / non-interactive"
/etc/profile
.profile
login / non-interactive

Wikipedia: Dash dash(1) man page dash source: initialization dash source: read_profile

The Big Conclusion

docker exec with no flags simply adds a process to the container (paired cgroup and namespace). Unless the process specified is a shell, nothing will be loaded into the process' environment that isn't already a default. [VERIFY]

Environment Loading - blitterated/docker-dev-env-s6 GitHub Wiki

Shell Types

Login/Nonlogin

Login Shells

Nonlogin Shells

Interactive/Noninteractive

Interactive Shells

Noninteractive Shells

Four Permutations

Detecting shell type

References

How docker exec works

Image for experiments

build an image

examine process tree in a container

start up a daemon container

Open a bash shell on the daemon container with docker exec.

htop

pstree

ps

References

examine process tree from outside a container

Take the Red Pill

References

So what kind of shell does using docker exec get us?

Conclusions

How the Environment Loads

login program

Add /root to ENV_SUPATH

References

PAM

On Ubuntu in Docker

References

bash startup files

Testing out how bash loads startup files

Interesting bits from bash's source code

References

dash startup files

Testing out how bash loads startup files

The Big Conclusion

Open a bash shell on the daemon container with `docker exec`.

So what kind of shell does using `docker exec` get us?

Testing out how `bash` loads startup files

Testing out how `bash` loads startup files