Logging - psu-libraries/scholarsphere GitHub Wiki

ScholarSphere logs to splunk, below are some useful queries to find what you're looking for

index=libraries  | spath "k8s-cluster" | search "k8s-cluster"="psul-dev" | spath "kubernetes.namespace_name" | search "kubernetes.namespace_name"="scholarsphere-qa"

• where "scholarsphere-qa" is the release name. if you pushed up "preview/new-font" the name would be "scholarsphere-new-font"

index=libraries  | spath "k8s-cluster" | search "k8s-cluster"="psul-prod" | spath "kubernetes.namespace_name" | search "kubernetes.namespace_name"="scholarsphere"

You can drill down into any service, or object you want.

<base-query> | search "kubernetes.container_name"="scholarsphere"

where "kubernetes.containername" is one of

scholarsphere # rails app
scholarsphere-sidekiq # sidekiq
redis
solr

To see all available service containers in the time period you selected:

<base-query> | stats count by kubernetes.container_name

A user submits a ticket and says a work is giving them a 404 error, and they believe it should be.

1. start with the base query

2. filter down to only the rails pods

3. Choose status 404

In this example we see the user tried to get to the 1eb62dbd-9c15-49e8-bbdd-cd7535de8dcs0 resource. we look in the rails console and don't find that resource, but we do find a 1eb62dbd-9c15-49e8-bbdd-cd7535de8dcs resource, so this must be a typo on the users part, we reply to the message with the proper URL and the guest is super happy

1. filtering IN is more performant than filtering OUT.
2. cast a wide net, then filter down. i.e if you were looking for status code of 404, you would start the search by finding the rails container, and then searching for status == 404, and not the other way around.
3. select a small time period, and select it first