Grafana Series (VI): Logs in Grafana Explore
This article was last updated on: July 24, 2024 am
👉️URL: https://grafana.com/docs/grafana/latest/explore/logs-integration/#labels-and-detected-fields
📝Description:
Logs in Explore
In addition to metrics, Explore allows you to investigate your logs in the following data sources.
During infrastructure monitoring and incident response, you can drill down into metrics and logs to find causes. Explore also allows you to correlate by viewing metrics and logs side-by-side. This creates a new debugging workflow.
- An alert was received.
- Dig deeper and examine the metrics.
- Dig deeper again, searching for logs related to metrics and time intervals (and distributed tracing in the future).
Log visualization
The results of the log query are displayed in a histogram in a graph, and individual logs are explained in the following sections.
If the data source supports a full-range log volume histogram, the log distribution graph for all entered log queries is automatically displayed. This feature is currently supported by Elasticsearch and Loki data sources.
If the data source does not support loading a full-range log volume histogram, the log model calculates a time series based on the number of log lines for the automatically calculated time interval, and then anchors the timestamp of the first log line at the beginning of the histogram of the results. The end of the time series is anchored to the time selectorTorange.
Log level
For logs that have a level label assigned, we use the value of the tag to determine the level of the log and update the color accordingly. If the log doesn’t specify a level label, we try to find out if its contents match any of the supported expressions (more on this below). The log level is always determined by the first match. If Grafana cannot determine a log level, it visualizes it with an unknown log level.
Tip: If you use a Loki data source, and the “level” is in your log line, use the parser (
JSON、logfmt、regex… Extract the level information as a level label that determines the log level. This will cause the histogram to show different log levels in different bar charts.
Supported log levels and mapping of log level abbreviations and expressions:
| Supported expressions | Log level | Color |
|---|---|---|
| emerg | critical | Purple |
| fatal | critical | Purple |
| alert | critical | Purple |
| crit | critical | Purple |
| critical | critical | Purple |
| err | error | Red |
| eror | error | Red |
| error | error | Red |
| warn | warning | Yellow |
| warning | warning | Yellow |
| info | info | Green |
| information | info | Green |
| notice | info | Green |
| dbug | debug | Blue |
| debug | debug | Blue |
| trace | trace | Light blue |
| * | unknown | Grey |
Log navigation
The log navigation next to the log line can be used to request more logs. You can do this by clicking the Older logs button at the bottom of the navigation. This is especially useful when you run into row limits and you want to see more logs. Each request run from the navigation appears in the navigation as a separate page. Each page displays the start and end timestamps of incoming log lines. You can view previous results by clicking on the page. Explore caches the last five requests that run from the log navigation, so you don’t rerun the same query when you click on those pages.
Visualization options
You can customize how the logs are displayed and choose which columns to display.
Time
Show or hide the time column. This is the timestamp associated with the log line, reported by the data source.
Unique labels
Show or hide unique tabs that include only unusual tabs. All common labels are displayed on it.
Line break
If you want the display to use line breaks, set this to true; Set toFalse, which will cause horizontal scrolling.
Prettify JSON
Set this to trueto print all JSON logs beautifully. This setting does not affect logs in any format other than JSON.
Deduping
Log data can be very duplicated, and Explore can help by hiding duplicate log lines. You can use several different deduplication algorithms.
- precise - Exact matches are made throughout the row, except for the date field.
- digit - Match on the line after the split number, such as duration, IP address, etc.
- signature - The most aggressive culling, which will strip away all letters and numbers and match on the remaining white space and punctuation.
Flip results order
You can change the order of the received logs from the default descending (newest first) to ascending (oldest first).
Labels and detected fields
Each log line has an expandable area with its labels and detection fields for more robust interaction. For all labels, we have added the ability to filter (forward filter) and filter (reverse filter) selected labels. Each field or label also has a stat icon to display special statistics related to all displayed logs.
Derived field links
By using derived fields, you can turn any part of the log message into an internal or external link. The created link appears as a button next to the Detected field in the log details view.
Toggle detected fields
Note: Available in Grafana 7.2 and later.
If your logs are based onjsonorlogfmtis constructed, then you can show or hide the detected fields. Expand a log line and tap the eye icon to show or hide the field.
Loki-specific features
As mentioned earlier, one of the log integrations is the new open source log aggregation system for Grafana Labs-Loki。 Loki is designed to be very cost-effective because it does not index the contents of the logs, but instead provides a set of labels for each log stream. Loki’s log query is similar to the query used in Prometheus using the tag selector. It groups log streams using tags that can be matched to your Prometheus tags. For more information about Grafana Loki, please refer to Grafana Loki Or a managed version of Grafana Labs:Grafana Cloud Logs。
For more information, please refer to Data source documentation for Loki Information about how to query log data.
Convert from metrics to logs
If you switch from a Prometheus query to a log query (you can do a split first to put your metrics and logs side by side), then it will retain the tags in your query that are present in the logs and use those tags to query the log stream. For example, the following Prometheus query.
grafana_alerting_active_alerts{job="grafana"}
After switching to the Logs data source, the query results become:
{job="grafana"}
This returns a large block of logs within the selected time range, which can be searched for grepped/text.
Live Tailing
Use the live scrolling feature to view real-time logs for supported data sources.
Click Explore on the toolbarreal timebutton to switch to live scroll view.
In the live scroll view, new logs appear from the bottom of the screen and have a gradient contrast background so you can keep track of what’s new. clickTime outbutton or scroll the log view to pause the live trace and explore previous logs without interruption. clickrecoverbutton to resume real-time tracking, or clickStop itbutton to exit live tracking and return to the standard Explore view.
