How to query logs in Grafana Loki

In this article:

• Introduction
• Querying your first logs
• Easily filter logs in Grafana
• How to parse Loki logs in Grafana
• Using context in Grafana Loki

Introduction

I’ve previously wrote an article on how to centralize and easily query logs with Promtail + Loki + Grafana.

In this article I will focus on the querying part. How we’re doing it for Emilio and how to properly query logs in Grafana and how to parse your log format to easily filter the lines you want to see.

Querying your first logs

While you can create fancy dashboards, querying logs really shines in the Explore view in Grafana.

You can find that view in the left panel:

If you’ve followed my previous tutorial you will be able to select either a job or a filename in label filters:

Now at the top right, hit run query:

Then, you will see something like this:

The UI consists on a bar graph indicating the amount of logs over time and a query of 1000 lines of the latest logs.

In the top panel you can adjust the timeframe for your query:

And in the query panel you can change the limit. Just be aware that increasing the limit too much might crash your Loki instance:

Easily filter logs in Grafana

It’s quite easy to start filtering logs.

Start by hitting: Operations

These are the current operations available at the time of writing this post:

The most useful one when you’re just starting is Line filters

You can then choose the filter you’d like to search specific lines. Here’s an example to query lines containing “ERROR”:

You can add as many operations as you want.

The bar graph is then a great way to see how many times the lines with your query happened over time. Cool to track errors, warnings, specific errors…

How to parse Loki logs in Grafana

This is a really cool thing that Grafana allows you to do and helps filtering a LOT.

I’ll use our logs format as an example:

2024-03-19 09:48:57,608 - 244843ce48f8 - INFO - views - History ID: 123782

The log pattern is as follows:

<datetime> - <docker_container_id> - <log_level> - <module> - <log_message>

For you to have more reference, you could also do fancy stuff like this:

0.191.12.2 - - [10/Jun/2021:09:14:29 +0000] "GET /api/plugins/versioncheck HTTP/1.1" 200 2 "-" "Go-http-client/2.0" "13.76.247.102, 34.120.177.193" "TLSv1.2" "US" ""

<ip> - - <_> "<method> <uri> <_>" <status> <size> <_> "<agent>" <_>

To parse the lines with a specific pattern like ours, go to + Operations, Formats and then Patterns

Then you just insert the pattern with the same format as above:

If you query again the logs you will see that nothing special happens, but with this we can now query on specific labels!

Hit + Operations and select Label Filters and then Label Filter Expressions

With these I can now filter the logs really selectively.

Let’s say I want to query ERROR logs for the user “daniel”:

Now you can combine labels and filters to really find what you’re looking for.

Using context in Grafana Loki

Grafana released this last year and it’s AMAZING.

Say you found instances of an error. Great, but what information does that give you? You want to see what happened before and after that log line to see the cause and consequence of that error.

For this, it’s really easy:

1. Hover above the line you want to see context for
2. Hit the show context icon at the top right of the line

3. A new window will open were you will see the context (logs before and after) of that line:

4. You can now use labels to filter different filenames, jobs or log labels!