How building works - accetto/ubuntu-vnc-xfce-g3 GitHub Wiki
Version: G3v8
Updated: 2025-04-26
This page describes the actual implementation of the seventh version (G3v7) of the building pipeline.
Before reading this page it can be helpful to read the pages Concepts of Dockerfiles, Concepts of building, Concepts of README files and Building stages first.
The building pipeline originally began as a mirror of the standard Docker Hub auto-building pipeline as it's described on the page Advanced automated builds. However, it has evolved since then.
Later the Docker Hub has removed the auto-building feature from the free plan and therefore the building on the Docker Hub stage has been abandoned.
The local stage is the default place for building the images now. Also CI/CD building in GitLab has been successfully tested, e.g. using a local GitLab installation in Docker container.
The version G3v2 of the building pipeline brought a significantly higher performance by building sets of images or by repeated builds.
It has been achieved by utilizing the local g3-cache, which is described separately.
The G3v7 pipeline builds the images even faster, because a temporary helper image is used as an external cache.
The complete building pipeline consists of
- hook scripts
- source files
- helper scripts
They are all stored in the folder docker/hooks/.
Also the Dockerfiles stored in the docker/ folder are part of the pipeline.
The docker/ folder itself is also referred as a docker building context or simply a building context.
The hook scripts pre_build, build, push and post_push make the actual building pipeline.
The source files are the helper files that contain additional source code, that is sourced by the hook scripts.
The helper scripts are used as external utilities.
The pipeline file structure looks like this:
-
docker/-
hooks/buildcacheenv.rcpost_pushpre_buildpushrelease_ofutil.rc
Dockerfile.xfce
-
It should be noted that some of the hook scripts use, or even require, some specific environment variables to be set.
The building pipeline can be executed completely or per parts.
All hook scripts require the following two inputs arguments
branchblend
These are explained on the page Concepts of building (section Other building concepts).
The hook scripts pre_build and build can take also additional Docker CLI arguments (e.g --no-cache), that will be passed to the docker build command used internally.
The common usage pattern:
<hook-script> <branch> <blend> [[--no-cache] [<other-docker-cli-options>]]
Additionally to the input arguments, also some environment variables play a role in the building process. Those are described on the page Building stages.
The source files contain additional source code, that is sourced by the hook scripts.
This file prepares the building environment and is conceptually close to a configuration file. This file is source by all hook scripts.
The code in env.rc processes the input arguments branch and blend and sets all the variables, that are necessary for building a particular image.
The best description is probably an example of a possible source code. Note that this is not the actual up-to-date code, but an illustration of the concept.
case "${_branch}" in
# default (master), developer (dev, dev-*) and release (v*) builds
master | dev | dev-* | v* )
# feature blends in this building branch
case "${_blend}" in
vnc )
DOCKERFILE_PATH="${_build_context}/Dockerfile.xfce"
_deploy_repo="${_owner}/ubuntu-vnc-xfce-g3"
_deploy_tags=( "vnc" )
_readme_context="docker/xfce"
;;
latest | vnc-novnc )
FEATURES_NOVNC=1
DOCKERFILE_PATH="${_build_context}/Dockerfile.xfce"
_deploy_repo="${_owner}/ubuntu-vnc-xfce-g3"
_deploy_tags=( "latest" "vnc-novnc" )
_readme_context="docker/xfce"
;;
vnc-fugo )
FEATURES_USER_GROUP_OVERRIDE=1
DOCKERFILE_PATH="${_build_context}/Dockerfile.xfce"
_deploy_repo="${_owner}/ubuntu-vnc-xfce-g3"
_deploy_tags=( "vnc-fugo" )
_readme_context="docker/xfce"
;;
vnc-chromium )
DOCKERFILE_PATH="${_build_context}/Dockerfile.xfce.chromium"
_deploy_repo="${_owner}/ubuntu-vnc-xfce-chromium-g3"
_deploy_tags=( "vnc" )
_readme_context="docker/xfce-chromium"
CHROMIUM_VERSION=$( "${_mydir}"/release_of 'chromium-browser' )
;;
*)
echo "Unsupported blend '${_blend}'"
exit 1
;;
esac
;;
# experimental builds
experimental | exp-* )
# feature blends in the experimental branch
case "${_blend}" in
rdp )
# set some variables here
;;
nomachine )
# set some variables here
;;
*)
echo "Unsupported blend '${_blend}'"
exit 1
;;
esac
;;
*)
echo "Unsupported branch '${_branch}'"
exit 1
;;
esac
This file contains some utility procedures and is also sourced by all hook scripts.
Examples of the utility procedures:
get_gist_file()update_gist()
This file is used for setting the environment variables.
If the source file env.rc finds the file secrets.rc in the folder docker/hooks/, then it sources it automatically, so you don't need to do it manually.
You can use the provided file example-secrets.rc as a template.
Be careful to exclude the file from your public commits, because it probably contains the real secrets. Adjust the .gitignore file correctly.
The helper scripts are used as external utilities.
This helper script returns the current release versions of the products, that are installed explicitly by their versions (e.g. Chromium Browser).
This helper script refreshes the local g3-cache, which must be always located inside the docker building context.
Note that this helper script is often referenced also as a hook script.
It is always executed by the pre_build and build hook scripts. However, it can be used also stand-alone. For example for checking, if there are newer versions of the packages that are normally downloaded from the external sources.
The g3-cache and the rules for its refreshing are described separately.
SInce the pipeline version G3v8 (Release 25.05) the script checks, if the shared g3-cache folder, which is defined by the environment variable SHARED_G3_CACHE_PATH is reachable and writable.
The hook scripts would override the default auto-building handlers on the Docker Hub stage, as it is described on the page Advanced automated builds.
However, since the version G3v2 the local stage took the role of the default building stage. Nevertheless, the original naming of the hook scripts has not been changed.
This script builds a temporary helper image, gets the current verbose version sticker value of it and stores the value into the temporary helper file scrap-version_sticker-verbose_current.tmp.
Then it gets the verbose version sticker value of the previous image release from the GitHub Gist belonging to the builder repository and stores the value into the temporary helper file scrap-version_sticker-verbose_previous.tmp.
The contents of the both helper files are then compared and if they are the same, then an another temporary helper file named scrap-demand-stop-building is created.
The file is actually empty, but its sheer presence has the meaning of a demand building stop command.
The comparison itself can be overridden and the image re-building can be forced by setting the environment variable FORCE_BUILDING=1.
The environment variable PROHIBIT_BUILDING=1 has the opposite effect.
If the building process should continue, then the short version sticker value is got using the helper image. The value is stored into the temporary helper file scrap-version_sticker_current.tmp.
Till the pipeline version G3v7, the temporary helper image was not needed any more and it was deleted.
It was required only if the up-to-date version sticker values were needed. This was usually the case when the images had to be published to the Docker Hub.
However, since the pipeline G3v7, the temporary helper image is also used as an external cache by building the final image.
Strictly speaking, executing the pre_build hook script is still not necessary in all scenarios, but using an external cache significantly speeds up the image building.
As it can be seen from the actual source code, the pre_build hook script can process also additional Docker CLI arguments, that will be passed to the internally used docker build command (notice the part docker build $@). Probably the most usual case is for refreshing the Docker builder cache:
./docker/hooks/pre_build dev latest --no-cacheThe Docker builder cache should not be confused with the local g3-cache, which is described separately.
The pre_build hook script always refreshes the local g3-cache before it starts building the temporary helper image. It uses the helper script cache for that.
It should be noticed, that the temporary helper image name has the name and the tag of the builder repository.
Additionally, its name gets the suffix ${helper_suffix} (-helper by default) and it gets no explicit tag.
The variable ${helper_suffix} is declared and set in the env.rc hook script.
Note that since the G3v7 (Release 25.03) pipeline the temporary helper image is not deleted by the pre_build hook script.
However, since the version G3v8 (Release 25.05) it's still the case if there will be no call of the build hook script.
It's then, when the helper temporary file scrap-demand-stop-building is present.
TIP: The pre_build hook script is the only script you need to execute, if you only want to check, if the image needs a refresh.
This script builds the persistent final image and it also sets its created and version-sticker labels to the up-to-date values.
The docker build command gets a number of arguments that are partially dependent on the feature variables set by the env.rc source file.
Before building the image, the build hook script refreshes the local g3-cache by executing the cache helper script and checks if the temporary helper file scrap-demand-stop-building with the meaning of a demand building stop command is present. If the file is found, then the script does not continue and it exits gracefully.
Similar to the previously described pre_build hook script, the build hook script can also get additional Docker CLI arguments (e.g. --no-cache), that will be passed to the internally used docker build command.
It should be noticed, that the final image has the name and the tag of the builder repository.
Note that since the G3v7 pipeline the temporary helper image is used as an external cache during the image building and deleted afterwards.
TIP: The build hook script is the only script you have to execute, if you want to build an image only for a local use and you are not interested in the up-to-date values of the image's labels.
Note that if you build an image without building the helper image, then there will be an error message about trying to remove the non-existing helper image. You can safely ignore the message.
For example:
### The next line would build the helper image, but it was not executed.
#./docker/hooks/pre_build dev latest
./docker/hooks/build dev latest
### then somewhere near the end of the log
Removing helper image
Error response from daemon: No such image: accetto/ubuntu-g3_latest-helper:latestThis script pushes the persistent final image into its target deployment repository on the Docker Hub.
The image gets the name and the tag of the deployment repository.
The same final image can be pushed several times into the same deployment repository, each time with a different deployment tag.
Before building the image, the script checks, if the temporary helper file scrap-demand-stop-building with the meaning of a demand building stop command is present. If it is found, the script does not continue and it exits gracefully.
You don't need to execute this script in all scenarios. Only if you want to push the image to its deployment repository on the Docker Hub.
It should be noticed, that if the target repository has the reserved name void, then the push will be skipped.
This script first checks, if the temporary helper file scrap-demand-stop-building with the meaning of a demand building stop command is present. If it is found, then the script does nothing and it exits gracefully.
In its normal course the post_push hook script begins with getting the values of the labels created and version-sticker (the short version sticker) from the persistent final image. The current value of the verbose version sticker is got from the temporary helper file created by the pre_build hook script.
All three values are then published to the GitHub Gist belonging to the builder repository.
Then the same three values are published also to the GitHub Gist belonging to the deployment repository. This is done for each deployment tag.
The other published data include the JSON endpoints for the badges that are embedded in the README files.
At the end this hook script removes the temporary helper files, unless the environment variable KEEP_HELPER_FILES is not set to 1.
You don't need to execute this script in all scenarios. Only if you want to update the GitHub Gists.
It should be noticed, that since the version G3v2 this hook script does not publish the README file to the Docker Hub. It has to be done separately as it is described bellow.
Note that since the pipeline version G3v8 (Release 25.05) the badge values for updating the deployment gists are extracted or generated ad-hoc using the locally available deployment images. Those do not need to be just built locally, but they can be also pulled from the Docker Hub.
The values for the badges created and version sticker will be extracted from the locally available images and the verbose version sticker badge values generated by them just before updating the gists.
This change allows refreshing the gists using the "historical" values from the previously published images.
If a gist update fails, it will be retried up to 3 times with the delay of 5 seconds between the retries.
Just a reminder: Deployment gists are publicly accessible files on the GitHub, that contain values used for generating the badges for the README files that are published on the Docker Hub.
If you want to execute the complete building pipeline, then you don't need to execute the hook scripts individually.
Since the version G3v2 you can use the provided helper scripts builder.sh and ci-builder.sh that can build the individual images or also sets of images. They are described on the page How CI works.
The scripts also create the log files scrap_builder.log and scrap_ci-builder.log in the project's root directory.
The automatic publishing of the README files to the Docker Hub had to be removed, because it was not working properly any more.
However, the README files for the Docker Hub can still be prepared with the provided utility util-readme.sh.
The content of the generated file scrap-readme.md can then be copy-and-pasted to the Docker Hub manually.
The Wiki page "Utility util-readme.sh" describes how to use the utility.