Logging
ScholarSphere logs to splunk, below are some useful queries to find what you're looking for
index=libraries | spath "k8s-cluster" | search "k8s-cluster"="psul-dev" | spath "kubernetes.namespace_name" | search "kubernetes.namespace_name"="scholarsphere-qa"
- where "scholarsphere-qa" is the release name. if you pushed up "preview/new-font" the name would be "scholarsphere-new-font"
index=libraries | spath "k8s-cluster" | search "k8s-cluster"="psul-prod" | spath "kubernetes.namespace_name" | search "kubernetes.namespace_name"="scholarsphere"
You can drill down into any service, or object you want.
<base-query> | search "kubernetes.container_name"="scholarsphere"
where "kubernetes.containername" is one of
scholarsphere # rails app
scholarsphere-sidekiq # sidekiq
redis
solr
To see all available service containers in the time period you selected:
<base-query> | stats count by kubernetes.container_name
A user submits a ticket and says a work is giving them a 404 error, and they believe it should be.
- start with the base query
- filter down to only the rails pods
- Choose status 404
In this example we see the user tried to get to the 1eb62dbd-9c15-49e8-bbdd-cd7535de8dcs0 resource. we look in the rails console and don't find that resource, but we do find a 1eb62dbd-9c15-49e8-bbdd-cd7535de8dcs resource, so this must be a typo on the users part, we reply to the message with the proper URL and the guest is super happy
- filtering IN is more performant than filtering OUT.
- cast a wide net, then filter down. i.e if you were looking for status code of 404, you would start the search by finding the rails container, and then searching for status == 404, and not the other way around.
- select a small time period, and select it first