Grafana series (XII): How to use Loki to create a Grafana dashboard for searching logs

overview

Create a simple Grafana dashboard to enable quick search of logs.

Experienced directly use Grafana’s Explore feature on it.

But for the inexperienced people, how can they have a dashboard that is already preset with a simple tag search to help some teams quickly find what they are looking for when troubleshooting. While Explore is great for this use case, it can also be a bit intimidating for first-time users, or for people who receive alerts in the middle of the night.

So, try using some template variables to create a Grafana dashboard that displays logs.

steps

Implementation overview

Create a Logs Panel dashboard, then add a few variables that the Logs Panel’s LogQL expression introduces.

Create variables

Because Loki and Prometheus share a set of labels, they can be used as variables for Loki by Prometheus’ Label. As shown in the following figure:

Here one can be created by: namespace, pod, 日志过滤条件 Dashboard to filter logs:

1. namespace variable, through which all are obtained namespace As a variable filter:
   1. Type: Query
   2. DataSource: Prometheus
   3. Query: label_values(kube_pod_info, namespace)
2. pod variable, by which the correspondence is obtained namespace lower pod Filter criteria as variables to narrow the log search space:
   1. Type: Query
   2. DataSource: Prometheus
   3. Query: label_values(container_network_receive_bytes_total{namespace=~"$namespace"},pod)
   4. Multi-value: checked;
   5. Include All option: Checked
   6. Custom all value: .*
   7. 🐾 Note, usecontainer_network_receive_bytes_totalUse the metric name to look for available pods, but you can use any metric to represent all pods in your environment.
3. search variable, through which the variable acts as a pipe filter condition for LogQL, which is used to actually perform the search:
   1. Type: Text Box
   2. Default value: rror (Hit.) Error or error)

Perform LogQL

Finally, concatenate the above variables, add the Logs Panel, use Loki as the data source, and use{namespace="$namespace", instance=~"$pod"} |~ "$search" As LogQL. As follows:

All of this adds up to a nice and simple interface for searching logs – unfamiliar people don’t even need to write a LogQL query!

If you want to check it out, here’s the dashboard json

Do another one to view the Journal log

Again, do another look at the Journal log:

Create variables

Use Loki’s Label directly here.

Here one can be created by: hostname, unit, 日志过滤条件 Dashboard to filter logs:

1. hostname variable, through which all are obtained hostname As a variable filter:
   1. Type: Query
   2. DataSource: Loki
   3. Query: label_values(hostname)
   4. Multi-value: Checked
   5. Include All option: Checked
   6. Custom all value: .+
2. unit variable, by which the correspondence is obtained hostname lower unit Filter criteria as variables to narrow the log search space:
   1. Type: Query
   2. DataSource: Loki
   3. Query: label_values({hostname=~"$hostname"}, unit)
   4. Multi-value: checked;
   5. Include All option: Checked
   6. Custom all value: .+
   7. 🐾 Note, usecontainer_network_receive_bytes_totalUse the metric name to look for available pods, but you can use any metric to represent all pods in your environment.
3. search variable, through which the variable acts as a pipe filter condition for LogQL, which is used to actually perform the search:
   1. Type: Text Box
   2. Default value: rror (Hit.) Error or error)

Perform LogQL

Finally, concatenate the above variables, add the Logs Panel, use Loki as the data source, and use{hostname=~"$hostname", unit=~"$unit"} |~ "$search" As LogQL. As follows:

The final effect

The end effect is as follows:

and as follows:

🎉🎉🎉

Grafana series of articles

Grafana series of articles