Fluent Bit logs from OpenShift to OpenSearch

August 2024 notice

The New SPAR application has now logs being sent to Kibana, instead of OpenSearch, due to OCIO SysDig integration. Please take a look at this page on Confluence for more information (logs location, retention period): https://apps.nrs.gov.bc.ca/int/confluence/display/FSADT2/Application+Logs

But if you're looking to learn the steps to setup your services with OpenSearch, please keep going.

End of August 2024 notice

Observability is related the extent to which you can understand the internal state or condition of a complex system based only on knowledge of its external outputs.(1) With that, we're working to add observability features to SPAR, and send logs to OpenSearch.

In order to have a fully working system, you'll need to:

Have your application logging to a file in the ECS Format (Elastic Common Schema), to make it easier to be parsed;
Have your application logs accessible on OpenShift. (In this page you'll find how to deploy on a PVC);
A running instance of FluentBit. (In this page you'll find how to deploy with an existing DeploymentConfig);
Create a FluentBit configuration file. (For that, you may need to learn more FluentBit itself);

OpenSearch SPAR Dashboard can be found here: SPAR OpenSearch Dashboard

Below you can find all required steps to get your Spring Boot application ready. Note that these steps cover how we did for SPAR, so that's one way of getting it done.

Dependencies

No extra dependency is required. Spring Boot dependencies already includes all logging systems and Logback. But in case your project is different, you can add it to your dependency tree.

Logback configurations

You need to create a file called logback-spring.xml in the resources folder. The content can be different due to project needs.

Note that:

Some environment variables are being created in the top of the file, like serviceEnv, applicationName, and others;
There's a springProfile tag that enables logging to file only for production;
Spring Boot provides logging options and pattern settings out of the box right in the applications properties (or yaml) file. Feel free to go with a separate file or modify the properties one;
Pay extra attention to the pattern of your JSON log file;

logback-spring.xml file for SPAR backend

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
 <property name="LOGS" value="/logs" />
 <springProperty scope="context" name="serviceEnv" source="nr-spar-env" />
 <springProperty scope="context" name="applicationName" source="spring.application.name" />
 <springProperty scope="context" name="sparTeamEmail" source="nr-spar-team-email-address" />
 <springProperty scope="context" name="ecsVersion" source="nr-spar-ecs-version" />

 <appender name="Console" class="ch.qos.logback.core.ConsoleAppender">
   <!-- https://logback.qos.ch/manual/layouts.html -->
   <!-- %d = 2006-10-20 14:06:49,812 or we can use like %date{HH:mm:ss.SSS}, or %d{ISO8601} -->
   <!-- %p = level -->
   <!-- %C{length} = fully qualified class name of the caller -->
   <!-- %t = thread name -->
   <!-- %m = message -->
   <!-- %n = new line -->
   <layout class="ch.qos.logback.classic.PatternLayout">
     <Pattern>
       %date{HH:mm:ss.SSS} %highlight(%-5level) [%blue(%t)] %yellow(%C): %msg%n%throwable
     </Pattern>
   </layout>
 </appender>

 <appender name="RollingFile" class="ch.qos.logback.core.rolling.RollingFileAppender">
   <file>${LOGS}/postgres-api.log</file>
   <encoder class="ch.qos.logback.classic.encoder.PatternLayoutEncoder">
     <Pattern>{"labels.project":"${applicationName}","service.environment":"${serviceEnv}","@timestamp":"%date{yyyy-MM-dd HH:mm:ss.SSS}","log.level":"%p","log.logger":"%logger{36}","message":"%m","ecs.version":"${ecsVersion}","event.category":"web","event.dataset":"application.log.utc","event.ingested":"diagnostic","event.kind":"event","organization.id":"${sparTeamEmail}","organization.name":"TeamSPAR"}%n</Pattern>
   </encoder>
   <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
     <!-- rollover daily and when the file reaches 10 MegaBytes -->
     <fileNamePattern>${LOGS}/archived/postgres-api-%d{yyyy-MM-dd}.%i.log</fileNamePattern>
     <timeBasedFileNamingAndTriggeringPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedFNATP">
       <maxFileSize>10MB</maxFileSize>
     </timeBasedFileNamingAndTriggeringPolicy>
   </rollingPolicy>
 </appender>

 <springProfile name="!prod">
   <!-- LOG everything at INFO level -->
   <root level="info">
     <appender-ref ref="Console" />
   </root>
   <!-- LOG "ca.bc.gov.backendstartapi*" at TRACE level -->
   <logger name="ca.bc.gov.backendstartapi" level="trace" additivity="false">
     <appender-ref ref="Console" />
   </logger>
 </springProfile>

 <springProfile name="prod">
   <!-- LOG everything at INFO level -->
   <root level="info">
     <appender-ref ref="RollingFile" />
     <appender-ref ref="Console" />
   </root>
   <!-- LOG "ca.bc.gov.backendstartapi*" at TRACE level -->
   <logger name="ca.bc.gov.backendstartapi" level="trace" additivity="false">
     <appender-ref ref="RollingFile" />
     <appender-ref ref="Console" />
   </logger> </springProfile>
 </configuration>

ECS configurations - Elastic Common Schema

The logging pattern should be in place accordingly to ECS. And also should attend some especific configurations, those are:

labels.project: your can use the service name here. For SPAR we have: spar-postgres-api
service.environment: do not use contractions, like dev. It should be development.
thread: not included in the application generic index (according to One Team). This is a default option for Java logs, but you can remove, because it will not be parsed.
ecs.version: 8.8
event.category: web
event.dataset: application.log.utc
organization.id: team mail
organization.name: team name

If you want to understand more about ECS, take a look on these links:

Fluent Bit Configurations and Funbucks

Your applications needs to have its own FluentBit configuration. This is done using Funbucks. Funbucks enables you to configure your applications's input, parser, filter and outputs. Through Funbucks you can put your configuration in place, and using its command line instructions you can generate final files that can be used, including files for OpenShift.

At the time of this writing, Funbucks doesn't provide a template, so you need to copy an existing application and modify according to your application needs.

Key notes for Funbucks:

You need to log your application's message using single quote, otherwise parsing issues will be faced;
In case you're sending the timestamp in UTC, update your event.dataset to application.log.utc;
The typeTag variable comes from config/server/app_spar.json (where app_spar.json is the file created for SPAR). In this file look the key apps.id;
You'll need an input file path, where the log will come from. This should be defined in the file config/templates/app_spar_postgres/app_spar_postgres.json (where app_spar_postgres is the name created for SPAR). In this file look for the key context.inputFile;
- Note that the variable name inside context may change. You can name it as you wish, but make sure to reference the same name in the input/inputs.conf.njk file.
Make sure to double check the filters.conf.njk file. Here is where some tags will be added or handled. There are required items:
- Add service.name spar_postgres_api
- Add service.type postgres_api
Also on this same file (filters), feel free to remove tags that are being sent by your application logs, like these:
- Add event.category web
- Add event.kind event

Running Funbucks commands to generate your config files

You can generate all the output files with (from Funbucks repo root dir): ./bin/dev gen -s app_spar
- In case you're generating for local testing, add the -l flag: ./bin/dev gen -l -s app_spar
Once you finished, you can generate the OpenShift template yml with: ./bin/dev oc -s app_spar

Getting started with Fluent Bit

What it is: https://docs.fluentbit.io/manual/about/what-is-fluent-bit
Key concepts: https://docs.fluentbit.io/manual/concepts/key-concepts
Inputs: https://docs.fluentbit.io/manual/pipeline/inputs
Parsers: https://docs.fluentbit.io/manual/pipeline/parsers
Filters: https://docs.fluentbit.io/manual/pipeline/filters
Outputs: https://docs.fluentbit.io/manual/pipeline/outputs

You can learn more about Funbuck on its repository, here: https://github.com/bcgov-nr/nr-funbucks

FluentBit Config files for SPAR

File config/server/app_spar.json (1/5)

{
  "address": "",
  "proxy": "",
  "logsProxyDisabled": "true",
  "os": "openshift",
  "apps": [
    {
      "id": "spar",
      "type": "app_spar_postgres",
      "context": {
        "environment": "production"
      }
    }
  ],
  "context": {
    "fluentLogFileEnabled": false
  },
  "oc": {
    "configmap": {
      "metadata": {
        "name": "spar-config",
        "labels": {
          "app": "nr-spar-backend"
        }
      },
      "prefix": "{{- if .Values.fluentbit.enable -}}\n",
      "suffix": "\n{{- end -}}"
    },
    "volume": {
      "indent": 6
    }
  }
}

File config/templates/app_spar_postgres/filter/filters.conf.njk (2/5)

{% import "../../macros/lua.njk" as lua %}
{{ lua.renderAppendTimestamp(typeTag + '.*') }}

[FILTER]
    Name modify
    Match {{typeTag}}.*
    Add service.name spar_postgres_api
    Add service.type postgres_api{% if organization_id %}
    Add organization.id {{ organization_id }}{% endif %}{% if organization_name %}
    Add organization.name {{ organization_name }}{% endif %}

File config/templates/app_spar_postgres/input/inputs.conf.njk (3/5)

[INPUT]
    Name tail
    Tag {{typeTag}}.log
    Buffer_Max_Size 1024k
    Parser {{typeTag}}.json
    Path {{inputLogPath}}
    Path_Key log_file_path
    Offset_Key event_sequence
    DB {{inputDb}}
    Read_from_Head True
    Refresh_Interval 15

File config/templates/app_spar_postgres/parser/parsers.conf.njk (4/5)

[PARSER]
    Name {{typeTag}}.json
    Match *
    Format json
    Time_Key @timestamp
    Time_Format %Y-%m-%dT%H:%M:%S.%LZ

File File config/templates/app_spar_postgres/app_spar_postgres.json (5/5)

{
  "measurementType": "historic",
  "context": {
    "!inputDb": "{{ '/tmp/fluent-bit-logs.db' if outputLocalLambdaEnabled else '/logs/fluent-bit-logs.db' }}",
    "!inputLogPath": "{{ localLogsPathPrefix | default('/logs/postgres-api.log') }}"
  },
  "files": [
    { "tmpl": "filter/filters.conf.njk", "type": "filter"},
    { "tmpl": "input/inputs.conf.njk", "type": "input"},
    { "tmpl": "parser/parsers.conf.njk", "type": "parser"}
  ]
}

Deploying to OpenShift with GitHub Actions

Logs will be stored into a PVC (Persistent Volume Claim). You'll need to update your OpenShift deployment file to include new lines containing variables, the fluent-bit image, deployment config, and the pvc. Take a look on below file, for reference:

This is the file for Backend API, a Java REST API:

File backend/openshift.deploy.yml

apiVersion: template.openshift.io/v1
...
objects:
  - apiVersion: v1
    kind: DeploymentConfig
    ...
    spec:
      template:
        spec:
          containers:
            - name: ...
              image: ...
              volumeMounts:
                - name: log-storage-${ZONE}
                  mountPath: /logs

And for a better understanding and organisation, we decided to create a new file only for getting Fluent Bit deployed.

File backend/openshift.fluentbit.yml

apiVersion: template.openshift.io/v1
kind: Template
labels:
  app: ${NAME}-${ZONE}
parameters:
  - name: NAME
    description: Product name
    value: nr-spar
  - name: COMPONENT
    description: Component name
    value: fluentbit
  - name: ZONE
    description: Deployment zone, e.g. pr-### or prod
    required: true
  - name: NAMESPACE
    description: Target namespace reference (i.e. '9f0fbe-dev')
    displayName: Target Namespace
    required: true
  - name: AWS_KINESIS_STREAM
    description: AWS Kinesis Stream identifier
    required: false
  - name: AWS_KINESIS_ROLE_ARN
    description: AWS OpenSearch/Kinesis Resource Name
    required: false
  - name: FLUENT_CONF_HOME
    description: FluentBit configuration home
    value: "/fluent-bit/etc"
  - name: FLUENT_VERSION
    description: FluentBit version
    value: "2.1"
  # Parameters for logging sidecar
  - name: LOGGING_CPU_LIMIT
    description: Limit Peak CPU per pod (in millicores ex. 1000m)
    displayName: CPU Limit
    value: 100m
  - name: LOGGING_CPU_REQUEST
    description: Requested CPU per pod (in millicores ex. 500m)
    displayName: CPU Request
    value: 10m
  - name: LOGGING_MEMORY_LIMIT
    description: Limit Peak Memory per pod (in gigabytes Gi or megabytes Mi ex. 2Gi)
    displayName: Memory Limit
    value: 64Mi
  - name: LOGGING_MEMORY_REQUEST
    description: Requested Memory per pod (in gigabytes Gi or megabytes Mi ex. 500Mi)
    displayName: Memory Request
    value: 16Mi
objects:
  - kind: DeploymentConfig
    apiVersion: v1
    metadata:
      labels:
        app: ${NAME}-${ZONE}
      name: ${NAME}-${ZONE}-${COMPONENT}
    spec:
      replicas: 1
      selector:
        deploymentconfig: ${NAME}-${ZONE}-${COMPONENT}
      strategy:
        type: Rolling
      template:
        metadata:
          labels:
            app: ${NAME}-${ZONE}
            deploymentconfig: ${NAME}-${ZONE}-${COMPONENT}
        spec:
          containers:
          - name: ${NAME}
            image: docker.io/fluent/fluent-bit:${FLUENT_VERSION}
            imagePullPolicy: Always
            livenessProbe:
              httpGet:
                path: /
                port: 2020
                scheme: HTTP
              initialDelaySeconds: 10
              timeoutSeconds: 1
              failureThreshold: 3
            ports:
            - containerPort: 2020
              name: metrics
              protocol: TCP
            - containerPort: 80
              name: http-plugin
              protocol: TCP
            readinessProbe:
              httpGet:
                path: /
                port: 2020
                scheme: HTTP
              initialDelaySeconds: 10
              timeoutSeconds: 1
              failureThreshold: 3
            resources:
              requests:
                cpu: "${LOGGING_CPU_REQUEST}"
                memory: "${LOGGING_MEMORY_REQUEST}"
              limits:
                cpu: "${LOGGING_CPU_LIMIT}"
                memory: "${LOGGING_MEMORY_LIMIT}"
            env:
              - name: AWS_ACCESS_KEY_ID
                valueFrom:
                  secretKeyRef:
                    name: ${NAME}-${ZONE}-fluentbit
                    key: aws-kinesis-secret-username
              - name: AWS_SECRET_ACCESS_KEY
                valueFrom:
                  secretKeyRef:
                    name: ${NAME}-${ZONE}-fluentbit
                    key: aws-kinesis-secret-password
              - name: STREAM_NAME
                valueFrom:
                  secretKeyRef:
                    name: ${NAME}-${ZONE}-fluentbit
                    key: aws-kinesis-stream
              - name: ROLE_ARN
                valueFrom:
                  secretKeyRef:
                    name: ${NAME}-${ZONE}-fluentbit
                    key: aws-kinesis-role-arn
              - name: FLUENT_CONF_HOME
                value: ${FLUENT_CONF_HOME}
              - name: FLUENT_VERSION
                value: ${FLUENT_VERSION}
              - name: AGENT_NAME
                value: ${NAME}-${ZONE}
            volumeMounts:
              - name: ${NAME}-${ZONE}-${COMPONENT}-logs
              mountPath: /logs
              - name: ${NAME}-${ZONE}-${COMPONENT}-configs
              mountPath: ${FLUENT_CONF_HOME}
        volumes:
          - name: ${NAME}-${ZONE}-${COMPONENT}-logs
            persistentVolumeClaim:
              claimName: ${NAME}-${ZONE}-${COMPONENT}-logs
          - name: ${NAME}-${ZONE}-${COMPONENT}-configs
            configMap:
              name: ${NAME}-${ZONE}-${COMPONENT}-configs
              items:
                - key: filters.conf
                  path: filters.conf
                - key: fluent-bit.conf
                  path: fluent-bit.conf
                - key: generic_json_parsers.conf
                  path: generic_json_parsers.conf
                - key: host_metadata.lua
                  path: host_metadata.lua
                - key: outputs.conf
                  path: outputs.conf
                - key: parsers.conf
                  path: parsers.conf
                - key: spar_filter_filters.conf
                  path: spar/filter/filters.conf
                - key: spar_input_inputs.conf
                  path: spar/input/inputs.conf
                - key: spar_parser_parsers.conf
                  path: spar/parser/parsers.conf
                - key: timestamp.lua
                  path: timestamp.lua
              defaultMode: 0644
  - kind: PersistentVolumeClaim
    apiVersion: v1
    metadata:
      labels:
        app: ${NAME}-${ZONE}
      name: ${NAME}-${ZONE}-${COMPONENT}-logs
    spec:
      accessModes:
        - ReadWriteMany
      resources:
        requests:
          storage: "20Mi"
      storageClassName: netapp-file-standard
  - kind: ConfigMap
    apiVersion: v1
    metadata:
      name: ${NAME}-${ZONE}-${COMPONENT}-configs
      namespace: ${NAMESPACE}
    data:
      filters.conf: >

        [FILTER]
          Name modify
          Match *
          Add agent.type fluentbit
          Add agent.version ${FLUENT_VERSION}
          Add agent.name ${AGENT_NAME}
          Add ecs.version 8.4
          Rename event_sequence event.sequence
          Rename log_file_path log.file.path

        [FILTER]
          Name lua
          Match spar
          script ${FLUENT_CONF_HOME}/timestamp.lua
          time_as_table True
          call append_event_created

        # There is a bug when resolving environment variables with spaces in the value :(
        # So, we have to use Lua script for now
        # Reference: https://github.com/fluent/fluent-bit/issues/1225

        [FILTER]
          Name lua
          Match *
          script ${FLUENT_CONF_HOME}/host_metadata.lua
          time_as_table True
          call add_host_metadata

      fluent-bit.conf: |
        [SERVICE]
          Log_Level info
          Parsers_File parsers.conf

        @INCLUDE spar/input/inputs.conf
        @INCLUDE spar/filter/filters.conf
        @INCLUDE filters.conf
        @INCLUDE outputs.conf

      generic_json_parsers.conf: |
        [PARSER]
          Name generic_json
          Format json

      host_metadata.lua: |
        -- Space delimited values to array
        function sdv2array(s)
            delimiter = "%S+"
            result = {};
            for match in string.gmatch(s, delimiter) do
                table.insert(result, match);
            end
            return result;
        end

        function isempty(s)
            return s == nil or s == ''
        end

        function copy(obj)
            if type(obj) ~= 'table' then return obj end
            local res = {}
            for k, v in pairs(obj) do res[copy(k)] = copy(v) end
            return res
        end

        function remove_nil_fields(tag, timestamp, record)
            return 2, timestamp, record
        end

        function add_host_metadata(tag, timestamp, record)
            new_record = record
            if isempty(new_record["host"]) then
                new_record["host"] = {}
            end
            local host = new_record["host"]
            if isempty(host["os"]) then
                host["os"] = {}
            end
            host["os"]["name"] = os.getenv("HOST_OS_NAME")
            host["os"]["type"] = os.getenv("HOST_OS_TYPE")
            host["os"]["family"] = os.getenv("HOST_OS_FAMILY")
            host["os"]["kernel"] = os.getenv("HOST_OS_KERNEL")
            host["os"]["full"] = os.getenv("HOST_OS_FULL")
            host["os"]["version"] = os.getenv("HOST_OS_VERSION")
            host["ip"] = os.getenv("HOST_IP")
            host["mac"] = os.getenv("HOST_MAC")
            if os.getenv("HOST_NAME") ~= nil then
                host["name"] = string.lower(os.getenv("HOST_NAME"))
            end
            if os.getenv("HOSTNAME") ~= nil then
                host["hostname"] = string.lower(os.getenv("HOSTNAME"))
            end
            host["domain"] = os.getenv("HOST_DOMAIN")
            host["architecture"] = os.getenv("HOST_ARCH")
            if not(isempty(host["ip"])) then
                host["ip"] = sdv2array(host["ip"])
            else
                host["ip"] = nil
            end

            if not(isempty(host["mac"])) then
                host["mac"] = sdv2array(host["mac"])
            else
                host["mac"] = nil
            end

            if not(isempty(host["name"])) then
                host["name"] = sdv2array(host["name"])
            else
                host["name"] = nil
            end

            if not(isempty(host["domain"])) then
                host["domain"] = sdv2array(host["domain"])
            else
                host["domain"] = nil
            end

            return 2, timestamp, new_record
        end
      outputs.conf: >
        [OUTPUT]
          Name kinesis_streams
          Match *
          region ca-central-1
          stream ${STREAM_NAME}
          role_arn ${ROLE_ARN}
          Retry_Limit 3

      parsers.conf: |+
        @INCLUDE generic_json_parsers.conf
        @INCLUDE spar/parser/parsers.conf

      spar_filter_filters.conf: |
        [FILTER]
          Name lua
          Match spar.*
          script ${FLUENT_CONF_HOME}/timestamp.lua
          time_as_table True
          call append_timestamp

        [FILTER]
          Name modify
          Match spar.*
          Add event.kind event
          Add event.category web
          Add @metadata.keyAsPath true

      spar_input_inputs.conf: |
        [INPUT]
          Name tail
          Tag spar.log
          Buffer_Max_Size 1024k
          Parser spar.json
          Path /logs/postgres-api.log
          Path_Key log_file_path
          Offset_Key event_sequence
          DB /logs/fluent-bit-logs.db
          Read_from_Head True
          Refresh_Interval 15

      spar_parser_parsers.conf: |
        [PARSER]
          Name spar.json
          Match *
          Format json
          Time_Key @timestamp
          Time_Format %Y-%m-%dT%H:%M:%S.%LZ

      timestamp.lua: >
        function append_event_created(tag, timestamp, record)
            new_record = record
            new_record["event.created"] = (os.date("!%Y-%m-%dT%H:%M:%S", timestamp["sec"]) .. '.' .. math.floor(timestamp["nsec"] / 1000000) .. 'Z')
            return 2, timestamp, new_record
        end

        function append_timestamp(tag, timestamp, record)
            new_record = record
            new_record["@timestamp"] = (os.date("!%Y-%m-%dT%H:%M:%S", timestamp["sec"]) .. '.' .. math.floor(timestamp["nsec"] / 1000000) .. 'Z')
            return 2, timestamp, new_record
 end

Workflow changes

First you need to update your init yaml file to include Fluent Bit secrets

File common/openshift.init.yml (1/2)

apiVersion: template.openshift.io/v1
...
parameters:
  - name: AWS_KINESIS_USER
    description: AWS Kinesis Secret Key ID
    required: true
  - name: AWS_KINESIS_PASS
    description: AWS Kinesis Secret Key password
    required: true
  - name: AWS_KINESIS_STREAM
    description: AWS Kinesis stream name
    required: true
  - name: AWS_KINESIS_ROLE_ARN
    description: AWS Kinesis Role ARN
    required: true
objects:
  - apiVersion: v1
    kind: Secret
    metadata:
      name: ${NAME}-${ZONE}-fluentbit
      labels:
        app: ${NAME}-${ZONE}
    stringData:
      aws-kinesis-secret-username: ${AWS_KINESIS_USER}
      aws-kinesis-secret-password: ${AWS_KINESIS_PASS}
      aws-kinesis-stream: ${AWS_KINESIS_STREAM}
      aws-kinesis-role-arn: ${AWS_KINESIS_ROLE_ARN}

Then you need to include all secrets to the init job, and the new job to deploy Fluent Bit

File .github/workflows/pr-open.yml (2/2)

jobs:
  init:
    name: Initialize
    ...
    steps:
      - name: OpenShift Init
        uses: bcgov-nr/[email protected]
        with:
          oc_namespace: ${{ vars.OC_NAMESPACE }}
          ...
          parameters:
            -p AWS_KINESIS_USER='${{ secrets.AWS_KINESIS_USER }}'
            -p AWS_KINESIS_PASS='${{ secrets.AWS_KINESIS_PASS }}'
            -p AWS_KINESIS_STREAM='${{ secrets.AWS_KINESIS_STREAM }}'
            -p AWS_KINESIS_ROLE_ARN='${{ secrets.AWS_KINESIS_ROLE_ARN }}'
      - name: FluentBit Init
        uses: bcgov-nr/[email protected]
        with:
          oc_namespace: ${{ vars.OC_NAMESPACE }}
          oc_server: ${{ vars.OC_SERVER }}
          oc_token: ${{ secrets.OC_TOKEN }}
          file: backend/openshift.fluentbit.yml
          overwrite: true
          parameters:
            -p NAMESPACE=${{ vars.OC_NAMESPACE }}
            -p ZONE=${{ github.event.number }}

⚠️ Don't forget to update both TEST and PROD workflows.

Lessons learned

Here are the items that caused headaches.

You'll need a fair understanding of FluentBit;
Deploy FluentBit first. If you're using our Deployer GH Action, you can create a new step in the init job.
When copying the files generated by Funbucks, watch out before pasting them into existing files. Yaml files have a strict syntax.
You may face issues when parsing log message with empty space. We had been trying to send organization.name with "SPAR Team", but that can be a problem. Change it to "TeamSPAR" or just remove the space;
Watch out for environment variables names;
Funbucks try to get the host name from the HOST_HOSTNAME envvar. You need to send this value, but another option is to change the envvar name from the lua script, just replace it by HOSTNAME;
Pay attention to the pvc size. The minimum is 20Mi, but you should be able to store at least one week of logs;
If you have questions, look for One Team or someone from SPAR Team;

Troubleshooting

You can run Fluent Bit locally to make sure all log files are correct. To do so, follow these steps:

Create a temporary folder. E.g.: mkdir $HOME/fluentbit
Create all files from the configMap: fluent-bit.conf, filter.conf, generic_json_parsers.conf, host_metadata.lua, outputs.conf, parsers.conf, timestamp.lua, and all files for your applications. In this case: spar/filter/filters.conf, spar/input/inputs.conf and spar/parser/parsers.conf
Copy your log sample to this folder. Ours is called postgres-api.log
Update the outputs.conf file to

[OUTPUT]
    Name  stdout
    Match *
    Retry_Limit 3

Update the fluent-bit.conf file to Log_Level debug;
Create a Dockerfile with this content:

FROM fluent/fluent-bit:2.1-debug
COPY postgres-api.log /logs/postgres-api.log
ADD . /fluent-bit/etc/

Create your docker image with: docker build -t fluentbit-local .
Run your image with:

docker run -ti --rm \
 -e FLUENT_VERSION=2.1 \
 -e AGENT_NAME=nr-spar-202 \
 -e FLUENT_CONF_HOME=/fluent-bit/etc/ \
 fluentbit-local

Now you can see the logs being parsed and look for error messages.
Good luck!

(1) From IBM article: https://www.ibm.com/topics/observability (2) How to test with FunBucks: https://bcdevops.github.io/nr-apm-stack/#/testing

Fluent Bit logs from OpenShift to OpenSearch - bcgov/nr-spar GitHub Wiki

August 2024 notice

End of August 2024 notice

Dependencies

Logback configurations

logback-spring.xml file for SPAR backend

ECS configurations - Elastic Common Schema

Fluent Bit Configurations and Funbucks

Running Funbucks commands to generate your config files

Getting started with Fluent Bit

FluentBit Config files for SPAR

Deploying to OpenShift with GitHub Actions

Workflow changes

Lessons learned

Troubleshooting

⚠️ GitHub.com Fallback ⚠️

Fluent Bit logs from OpenShift to OpenSearch - bcgov/nr-spar GitHub Wiki

August 2024 notice

End of August 2024 notice

Dependencies

Logback configurations

logback-spring.xml file for SPAR backend

ECS configurations - Elastic Common Schema

Fluent Bit Configurations and Funbucks

Running Funbucks commands to generate your config files

Getting started with Fluent Bit

FluentBit Config files for SPAR

Deploying to OpenShift with GitHub Actions

Workflow changes

Lessons learned

Troubleshooting

⚠️ **GitHub.com Fallback** ⚠️

⚠️ GitHub.com Fallback ⚠️