General introduction

We store our logs inside Elastic Search and get any information from there afterwards. The logs on each Docker server are only kept for a certain time until the filebeat comes by and takes them to the Elastic infrastructure.

Therefore, if external developers need to investigate older issues, they need old logs and must access the Kibana dashboard or API.

Kibana Settings

• Go to https://logs.opendatahub.bz.it
• Login with Elasticsearch
• Default space
• "Manage" (right hand side of "Home")

Then, create a role that corresponds to your use case. See odh-mobility-external-developers as an example

• Indices = filebeat-*
• Privileges = read and view_index_metadata
• Activate Grant read privileges to specific documents; example filter could be like this: \

  {
    "term" : { 
      "container.labels.com_docker_compose_project" : "odh-mobility-dc-traffic-a22-elaborations" 
    }
  }
  

• Add custom Kibana privileges below:
  • Analytics - Discover: ALL
  • Analytics - Dashboard: ALL
  • Management - Dev Tools: READ
  • Management - Saved Objects Management: READ
  • ...all others are NONE

Finally, create users and give them the role above. In addition, if you want that those users can create and download CSV reports, give them also the reporting_user privilege, which is unfortunately not assignable to roles.

Access to logs via Kibana

Go to Analytics > Discover and use the filter and search functions there.

Also CSV exports can be done with "Save" + "Share > CSV Reports > Generate CSV". This is limited to a not too high number of logs...

Access to logs via Elasticsearch API

To get you started here is an example shell-script, that takes certain logs for a mobility data collector called odh-mobility-dc-traffic-a22-elaborations between a specified interval. After that it gets only some fields out of the JSON response with jq and creates a CSV output.

To use this script, copy/paste it into a file and make that file executable. Call that file with optional parameters:

1. offset
2. limit
3. start_incl
4. end_excl
5. csv (0 or 1)

#!/bin/bash

set -euo pipefail

HOST='https://16587ef4ae8f4a23b477025805374935.eu-west-1.aws.found.io:9243/filebeat-*/_search'

ELASTIC_USR=${ELASTIC_USR:?"Specify your elastic user's name"}
ELASTIC_PWD=${ELASTIC_PWD:?"Specify your elastic user's password, quote it eventually like 'your-secret'"}

OFFSET=${1:-0}
LIMIT=${2:-1000}
START_INCL=${3:-'2022-06-23T23'}
END_EXCL=${4:-'2022-06-24'}

# Set this to 0 to retrieve the original result of elastic search calls
# You need to install "jq" if you want to have a CSV output
CSV=${5:-1}

RESULT=$(
  curl -s --user "$ELASTIC_USR:$ELASTIC_PWD" "$HOST" -XPOST -H 'Content-Type: application/json' --data-binary @- <<EOF
  {
    "fields": [
      "json.message",
      "json.level",
      "@timestamp"
    ],
    "_source": false,
    "from": $OFFSET,
    "size": $LIMIT,
    "query": {
      "bool": {
        "must": {
          "term": {
            "container.labels.com_docker_compose_project": "odh-mobility-dc-traffic-a22-elaborations"
          }
        },
        "filter": {
          "range" : {
            "@timestamp": {
              "gte" : "$START_INCL",
              "lt" : "$END_EXCL"
            }
          }
        }
      }
    },
    "sort": [
      {
        "@timestamp" : { 
          "order": "asc"
        }
      }
    ]
  } 
EOF
)


if [ "$CSV" = "1" ]; then
  echo '"timestamp","log_level","message"'
  echo "$RESULT" | jq -r '.hits.hits | .[] | .fields | ."@timestamp" + ."json.level" + ."json.message" | @csv'
else
  echo "$RESULT"
fi
 
exit 0

For more details on all Elasticsearch capabilities, please refer to their documentation.