We can run a Log Processor/Forwarder to collect logs/events from the application and connected infrastructure.

Fluent-bit is one such tool that can be run as 'sidecar', along-side our Common Service API's. The steps required for deploying a Fluent-bit sidecar are shown below. Note: These steps are to deploy the sidecar.

Logs from the CDOGS application are written to a file that Fluent-bit can then sort through, filter, and forward to different locations. We currently forward all logs (except readiness probe events) to a separate Log aggregator (Fluentd) that runs in a more centralized, 'Observability' namespace. From this location, we can trigger error notifications and use other tools to store/index logs and visualize the data.

You can read about our Log Aggregation wtih Fluentd.

For this example, we deploy a Fluent-bit sidecar for our CDOGS node application. The deployment uses gets Fluent-bit configuration in a config map template.

Our NodeJS apps output logs to a configurable file path (for example app/app.log). This is done using using a logger script. For example see our COMS app logger

In the same OpenShift template that deploys the NodeJS application, we can deploy a sidecar Fluent-bit container from a docker image. Note: Each release of Fluent-bit also comes with a debug version (for example: fluent-bit:1.X-debug) that includes some other Linux tools such as busybox, bash, etc. and make testing the installation easier.

Our Fluent-bit container will mount the directory /var/log from our node application and read the log file app.log

Fluent-bit has its own configuration file /fluent-bit/etc/fluent-bit.conf that we can create using an OpenShift configMap. We can configure parsers (define our log formats) by providing a Path in the SERVICE section of this configuration file that links to a separate file parser.conf (Parsers_File parsers.conf)

To add this configuration to an OpenShift configMap we can run these commands from our CDOGS Openshift directory:

# define our environment variables for the commands
export APP_NAME=<app name>                                        # eg: cdogs
export REPO_NAME=<your repo name>                                 # URI of the github repo. eg: common-document-generation-service
export JOB_NAME=<job name>                                        # eg: master
export NAMESPACE=<namespce>                                       # eg: idcqvl-dev
export LOGGING_HOST_NAME=<hostname for fluentd service>           # eg: fluentd-csst.apps.silver.devops.gov.bc.ca

# create the config map using the sidecar configMap template /openshift/fluent-bit.cm.yaml
oc process -n $NAMESPACE -f fluent-bit.cm.yaml \
  -p NAMESPACE=$NAMESPACE \
  -p APP_NAME=$APP_NAME \
  -p REPO_NAME=$REPO_NAME \
  -p JOB_NAME=$JOB_NAME \
  -p LOGGING_HOST_NAME=$LOGGING_HOST_NAME \
  -o yaml | oc -n $NAMESPACE apply -f -

# define additional environment variables for deployment
export APP_NAME=<app name>                                        # eg: cdogs
export REPO_NAME=<your repo name>                                 # URI of the github repo. eg: common-document-generation-service
export JOB_NAME=<job name>                                        # eg: master
export NAMESPACE=<namespce>                                       # eg: idcqvl-dev
export HOST_ROUTE=<route given to the CDOGS node application>     # eg: cdogs-dev.apps.silver.devops.gov.bc.ca

oc process -n $NAMESPACE -f app.dc.yaml \
  -p REPO_NAME=$REPO_NAME \
  -p JOB_NAME=$JOB_NAME \
  -p NAMESPACE=$NAMESPACE \
  -p APP_NAME=$APP_NAME \
  -p HOST_ROUTE=$HOST_ROUTE \
  -o yaml | oc -n $NAMESPACE apply -f -

We are using Fluent-bit's HTTP output plugin to communicate with Fluentd. We have the option to enable security for this connection. See: enable TLS on HTTP output

This configuration is currently disabled (commented out) in the deploymentConfig template

# create secrets from SSL cert files
# see here for a useful guide: https://banzaicloud.com/blog/k8s-logging-tls/
oc create -n $NAMESPACE secret generic fluent-bit-tls --from-file=ca.crt.pem=./certs/ca.crt.pem --from-file=client.crt.pem=./certs/client.crt.pem --from-file=client.key.pem=./private/client.key.pem

Our Jenkins pipeline (that handles the promotion of code changes through dev > test > prod environments), uses this same OpenShift deployment template. We need to create the fluent-bit configuration (configMap) in each of our project namespaces (dev, test and prod) that require logging. If we decide we only want to have logging on production environments and a configuration file does not exist, the sidecar container should fail gracefully.