We work with Docker containers quite a bit in the lab.

Here is a very non-exhaustive list of some of the Docker issues we've run into.

Running out of space/inodes

I've noticed this under Vagrant, it's possible for a long-running Docker container to eventually fill up the inode limit of a filesystem, specifically the /run filesystem. This will show up as a space error when running/starting a new container:

docker: Error response from daemon: mkdir /var/run/docker/libcontainerd/9377e826df97a729a5746f69f0de635f8c40ea075fc628306d27429fde147e62: no space left on device.

If you then do a:

du -ih

you may see the /run filesystem taking 100% of the inodes, this is the problem.

The solution is to first attempt to stop/kill the offending container, then clean up the run directory.

If you are not able to remove the container using docker rm or docker kill, please see the "Manually removing a running container" below.

Once the container is terminated, check the number of inodes again as above. If /run is still at 100%, you can manually remove the offending files by: sudo rm -rf /var/run/docker/libcontainerd/<long_string_here>

You can determine what <long_string_here> but checking the number of files underneath any candidate subdirectories of /var/run/docker/libcontainerd.

Manually removing a running container

If for some reason a container won't die/be killed/rm'd, you can manually remove it:

rm -rf /var/lib/docker/<underlying_filesystem/<container_id>*

Example: rm -rf /var/lib/docker/overlay2/4d1df8b6e31b5e4*

Then do a: docker rm -f <container_id>

And then make sure it's not in a network (e.g. bridge): docker network disconnect -f bridge <container_name>