05 Nov 22:27

slim-bean

3987933

v2.4.0

🎊 🌮 Loki 2.4.0 🌮 🎊

With Loki 2.4 we’ve focused on 2 main things:

Making it easier than ever to get your logs into Loki.
Making it easier to run and operate, anyone not using Kubernetes or allergic to microservices should pay close attention, this release has a lot for you.

Features and enhancements

Loki no longer requires logs to be sent in perfect chronological order. Support for out of order logs is one of the most highly requested features for Loki and in 2.4 the strict ordering constraint has been removed.
Scaling Loki is now much easier with a hybrid mode between a single binary and microservices we are calling the Simple Scalable Deployment. It’s now possible to scale Loki with new read and write targets. Where previously you would have needed Kubernetes and the microservices approach to start tapping into Loki’s potential, it’s now possible to do this in a much simpler way.
We introduced a common config section and updated and updated a lot of defaults. This results in a 70% smaller Loki config and comes out of the box with more appropriate defaults and limits. Check out the example local config as the new reference for running Loki.
Recording rules are no longer marked experimental and have a much more resilient implementation which leverages the existing write ahead log code in Prometheus.
Do you use Kafka for your logs? You should check out the new Promtail Kafka Consumer which can easily get your logs out of Kafka and into Loki.
LogQL has received some nice enhancements thanks to the amazing Loki community in the form of group_left and group_right as well as the label_format and line_formatfunctions now support working with dates and times
Another great community contribution to Promtail allows it to accept ndjson and plaintext log files over HTTP.

All in all about 260 PR’s went into Loki 2.4 and we thank everyone for helping us make the best Loki yet.

For a full list of all changes please look at the CHANGELOG

Upgrade Considerations

Please read the upgrade guide before updating Loki.

We made a lot of changes to Loki’s configuration as part of this release, we have tried our best to make sure changes are compatible with existing configurations however some changes to default limits may impact users who didn't have values explicitly set for these limits in their config files.

Here is a list of limits changed:

config	new default	old default
ingestion_rate_strategy	"global"	"local"
max_global_streams_per_user	5000	0 (no limit)
max_query_length	"721h"	"0h" (no limit)
max_query_parallelism	32	14
max_streams_per_user	0 (no limit)	10000
reject_old_samples	true	false
reject_old_samples_max_age	"168h"	"336h"
per_stream_rate_limit	3MB	-
per_stream_rate_limit_burst	15MB	-

Also we have enabled the in-memory FIFO cache by default when no existing external cache is configured. This helps improve Loki's performance however it will also allow Loki to consume more memory.

Bug fixes

Lists of bug fixes for 2.4.x.

2.4.0 bug fixes

Here is a list of some of the more important fixes in 2.4.0:

4598 kavirajk: Fix ip matcher lexer to differentiate filter from identifier
4563 cyriltovena: Fixes the Series function to handle properly sharding.
4518 slim-bean: Loki: Fix bug where items are returned to a sync.Pool incorrectly
4411 slim-bean: Loki: Bug: frontend waiting on results which would never come
4238 liguozhong: [fix] distributor: fix goroutine leak

Security fixes

List of security fixes for 2.4.x

2.4.0 security fixes

2.4.0 contains one fix which is security related

4627 Loki: Explicitly define allowed HTTP methods on HTTP endpoints

A community user noticed that all Loki endpoints would respond to HTTP OPTIONS requests, and the proxy they placed in front of Loki responsible for HTTP authentication would pass OPTIONS requests unauthenticated to Loki allowing users to make unauthenticated/unauthorized queries to Loki. This PR restricts the type of HTTP methods allowed on each endpoint and disallows OPTIONS requests.

Installation:

The components of Loki are currently distributed in plain binary form and as Docker container images. Choose what fits your use-case best.

Docker container:

$ docker pull "grafana/loki:2.4.0"
$ docker pull "grafana/promtail:2.4.0"

Binary

We provide pre-compiled binary executables for the most common operating systems and architectures.
Choose from the assets below for the application and architecture matching your system.
Example for Loki on the linux operating system and amd64 architecture:

$ curl -O -L "https://github.com/grafana/loki/releases/download/v2.4.0/loki-linux-amd64.zip"
# extract the binary
$ unzip "loki-linux-amd64.zip"
# make sure it is executable
$ chmod a+x "loki-linux-amd64"

Assets 32

06 Aug 18:29

slim-bean

v2.3.0

f5fd029

v2.3.0

🥳🎉 Loki 2.3.0 🎉🥳

The Loki team is excited to announce the release of Loki 2.3!

It's been nearly 6 months since 2.2 was released and we have made good use of that time to bring forward several significant improvements and requested features.

2.3 is also the first version of Loki released under the AGPLv3 license. You can read more about our licensing here.

Some parts of the Loki repo will remain Apache-2.0 licensed (mainly clients and some tooling), for more details please read LICENSING.md.

Features and enhancements

Loki now has the ability to apply custom retention based on stream selectors! This will allow much finer control over log retention all of which is now handled by Loki, no longer requiring the use of object store configs for retention.
Coming along hand in hand with storing logs for longer durations is the ability to delete log streams. The initial implementation lets you submit delete request jobs which will be processed after 24 hours.
A very exciting new LogQL parser has been introduced: the pattern parser. Much simpler and faster than regexp for log lines that have a little bit of structure to them such as the Common Log Format. This is now Loki's fastest parser so try it out on any of your log lines!
Extending on the work of Alerting Rules, Loki now accepts recording rules. This lets you turn your logs into metrics and push them to Prometheus or any Prometheus compatible remote_write endpoint.
LogQL can understand IP addresses! This enables filtering on IP addresses and subnet ranges.

For those of you running Loki as microservices, the following features will improve performance operations significantly for many operations.

We created an index gateway which takes on the task of downloading the boltdb-shipper index files allowing you to run your queriers without any local disk requirements, this is really helpful in Kubernetes environments where you can return your queriers from Statefulsets back to Deployments and save a lot of PVC costs and operational headaches.
Ingester queriers are now shardable, this is a significant performance boost for high volume log streams when querying recent data.
Instant queries can now be split and sharded making them just as fast as range queries.

A very common feature requested has also been included in 2.3:

3919 github-vincent-miszczak: Add ingester.autoforget-unhealthy-timeout opt-in feature

Without revisiting the decisions and discussions around the somewhat controversial behavior of unhealthy ingesters, you can now decided how you would like them to be handled: manually or automatically.

Lastly several useful additions to the LogQL query language have been included:

More text/template functions are included for label_format and line_format with PR 3515, please check out the documentation for template functions.
Also support for math functions withing label_format and line_format was included with 3434.
Two additional metric functions with some interesting use cases first_over_time and last_over_time were added in PR 3050. These can be useful for some down sampling approaches where instead of taking an average, max, or min of samples over a range in a metrics query, you can select the first or last log line to use from that range.

For the full list of changes (almost 300!!) check out the CHANGELOG

Upgrade considerations

The path from 2.2.1 to 2.3.0 should be smooth, as always, please check out the Upgrade Guide for important upgrade guidance.

One change we consider noteworthy however is:

3216 sandeepsukhani: check for stream selectors to have at least one equality matcher.

This change now rejects any query which does not contain at least one equality matcher, an example may better illustrate:

{namespace=~".*"}

This query will now be rejected, however there are several ways to modify it for it to succeed:

Add at least one equals label matcher:

{cluster="us-east-1",namespace=~".*"}

Use .+ instead of .*

{namespace=~".+"}

This difference may seem subtle but if we break it down . matches any character, * matches zero or more of the preceding character and + matches one or more of the preceding character. The .* case will match empty values where .+ will not, this is the important difference. {namespace=""} is an invalid request (unless you add another equals label matcher like the example above).

The reasoning for this change has to do with how index lookups work in Loki, if you don't have at least one equality matcher Loki has to perform a complete index table scan which is an expensive and slow operation.

Security fixes

List of security fixes for 2.3.x.

2.3.0 security fixes

2.3.0 contains an important security fix:

4020 simonswine: Restrict path segments in TenantIDs (CVE-2021-36156 CVE-2021-36157).

Note Exploitation of this vulnerability requires the ability for an attacker to craft and send directly to Loki an X-Scope-OrgID header, end users should not have the ability to create and send this header directly to Loki as it controls access to tenants and is important to control setting of this header for proper tenant isolation and security. We always recommend having a proxy or gateway be responsible for setting the X-Scope-OrgID.

Bug fixes

Lists of bug fixes for 2.3.x.

2.3.0 bug fixes

An important fix for leaking resources was patched with 3733, when queries were canceled a goroutine was left running which would hold memory resources creating a memory leak.
3686 fixes a panic with the frontend when use with downstream URL. Note we recommend using the GRPC Pull Model, better performance and fair scheduling between tenants can be obtained with the GPRC Pull Model.

Installation:

The components of Loki are currently distributed in plain binary form and as Docker container images. Choose what fits your use-case best.

Docker container:

$ docker pull "grafana/loki:2.3.0"
$ docker pull "grafana/promtail:2.3.0"

Binary

$ curl -O -L "https://github.com/grafana/loki/releases/download/v2.3.0/loki-linux-amd64.zip"
# extract the binary
$ unzip "loki-linux-amd64.zip"
# make sure it is executable
$ chmod a+x "loki-linux-amd64"

Assets 28

06 Apr 01:11

slim-bean

v2.2.1

babea82

v2.2.1

2.2.1 fixes several important bugs, it is recommended everyone running 2.2.0 upgrade to 2.2.1

2.2.1 also adds the labelallow pipeline stage in Promtail which lets an allowlist be created for what labels will be sent by Promtail to Loki.

3468 adityacs: Support labelallow stage in Promtail
3502 cyriltovena: Fixes a bug where unpack would mutate log line.
3540 cyriltovena: Support for single step metric query.
3550 cyriltovena: Fixes a bug in MatrixStepper when sharding queries.
3566 cyriltovena: Properly release the ticker in Loki client.
3573 cyriltovena: Fixes a race when using specific tenant and multi-client.

Installation:

The components of Loki are currently distributed in plain binary form and as Docker container images. Choose what fits your use-case best.

Docker container:

$ docker pull "grafana/loki:2.2.1"
$ docker pull "grafana/promtail:2.2.1"

Binary

$ curl -O -L "https://github.com/grafana/loki/releases/download/v2.2.1/loki-linux-amd64.zip"
# extract the binary
$ unzip "loki-linux-amd64.zip"
# make sure it is executable
$ chmod a+x "loki-linux-amd64"

Assets 28

11 Mar 16:58

slim-bean

v2.2.0

a27c799

v2.2.0

🎉🎉 Loki 2.2.0 is released!! 🎉🎉

Please read the Upgrade Guide before upgrading for a smooth experience.

Upgrade Guide TL;DR Loki 2.2 changes the internal chunk format which limits what versions you can downgrade to, a change was made in how many queries were allowed to be scheduled per tenant which might affect your max_query_parallelism and max_outstanding_per_tenant settings, and we fixed a mistake related scrape_configs which do not have a pipeline_stages defined. If you have any Promtail scrape_configs which do not specify pipeline_stages you should go read the upgrade notes!

With over 200 PR's 2.2 includes significant features, performance improvements, and bug fixes!

The most upvoted issue for Loki was closed in this release! Issue 74 requesting support for handling multi-line logs in Promtail was implemented in PR 3024. Thanks @jeschkies!

Other exciting news for Promtail, PR 3246 by @cyriltovena introduces support for reading Windows Events!

Switching to Loki, @owen-d has added a write ahead log to Loki! PR 2981 was the first of many as we have spent the last several months using and abusing our write ahead logs to flush out any bugs!

A significant number of the PR's in this release have gone to improving the features introduced in Loki 2.0. @cyriltovena overhauled the JSON parser in PR 3163 (and a few other PR's), to provide both a faster and smarter parsing to only extract JSON content which is used in the query output. The newest Loki squad member @dannykopping fine tuned the JSON parser options in PR 3280 allowing you to specific individual JSON elements, including support now for accessing elements in an array. Many, many other additional improvements have been made, as well as several fixes to the new LogQL features added some months ago, this upgrade should have everyone seeing improvements in their queries.

@cyriltovena also set his PPROF skills loose on the Loki write path which resulted in about 8x less memory usage on our distributors and a much more stable memory usage when ingesters are flushing a lot of chunks at the same time.

There are many other noteworthy additions and fixes, too many to list, but we should call out one more feature all you Google Cloud Platform users might be excited about: in PR 3083 @kavirajk added support to Promtail for listening on Google Pub/Sub topics, letting you setup log sinks for your GCP logs to be ingested by Promtail and sent to Loki!

For a full list of changes, please checkout the CHANGELOG

Thanks to everyone for another exciting Loki release!!

Installation:

The components of Loki are currently distributed in plain binary form and as Docker container images. Choose what fits your use-case best.

Docker container:

$ docker pull "grafana/loki:2.2.0"
$ docker pull "grafana/promtail:2.2.0"

Binary

$ curl -O -L "https://github.com/grafana/loki/releases/download/v2.2.0/loki-linux-amd64.zip"
# extract the binary
$ unzip "loki-linux-amd64.zip"
# make sure it is executable
$ chmod a+x "loki-linux-amd64"

Assets 28

24 Dec 15:15

slim-bean

v2.1.0

1b79df3

v2.1.0

🎆🎁Happy Holidays from the Loki team!🎁🎆

Please enjoy Loki 2.1.0 to welcome in the New Year!

This release contains a number of fixes, performance improvements and enhancements to the 2.0.0 release!

Notable changes

Helm users read this!

The Helm charts have moved!

2720 torstenwalter: Deprecate Charts as they have been moved

This was done to consolidate Grafana's helm charts for all Grafana projects in one place: https://github.com/grafana/helm-charts/

From now moving forward, please use the new Helm repo url: https://grafana.github.io/helm-charts

The charts in the Loki repo will soon be removed so please update your Helm repo to the new URL and submit your PR's over there as well

Special thanks to @torstenwalter, @unguiculus, and @scottrigby for their initiative and amazing work to make this happen!

Also go check out the microservices helm chart contributed by @unguiculus in the new repo!

Fluent bit plugin users read this!

Fluent bit officially supports Loki as an output plugin now! WoooHOOO!

However this created a naming conflict with our existing output plugin (the new native output uses the name loki) so we have renamed our plugin.

2974 hedss: fluent-bit: Rename Fluent Bit plugin output name.

In time our plan is to deprecate and eliminate our output plugin in favor of the native Loki support. However until then you can continue using the plugin with the following change:

Old:

[Output]
    Name loki

New:

[Output]
    Name grafana-loki

Fixes

A lot of work went into 2.0 with a lot of new code and rewrites to existing, this introduced and uncovered some bugs which are fixed in 2.1:

2807 cyriltovena: Fix error swallowed in the frontend.
2805 cyriltovena: Improve pipeline stages ast errors.
2824 owen-d: Fix/validate compactor config
2830 sandeepsukhani: fix panic in ingester when not running with boltdb shipper while queriers does
2850 owen-d: Only applies entry limits to non-SampleExprs.
2855 sandeepsukhani: fix query intervals when running boltdb-shipper in single binary
2895 shokada: Fix error 'Unexpected: ("$", "$") while parsing field definition'
2902 cyriltovena: Fixes metric query issue with no grouping.
2901 cyriltovena: Fixes a panic with the logql.NoopPipeline.
2913 cyriltovena: Fixes logql.QueryType.
2917 cyriltovena: Fixes race condition in tailer since logql v2.
2960 sandeepsukhani: fix table deletion in table client for boltdb-shipper

Enhancements

A number of performance and resource improvements have been made as well!

2911 sandeepsukhani: Boltdb shipper query readiness
2875 cyriltovena: Labels computation LogQLv2
2927 cyriltovena: Improve logql parser allocations.
2926 cyriltovena: Cache label strings in ingester to improve memory usage.
2931 cyriltovena: Only append tailed entries if needed.
2973 cyriltovena: Avoid parsing labels when tailer is sending from a stream.
2959 cyriltovena: Improve tailer matcher function.
2876 jkellerer: LogQL: Add unwrap bytes() conversion function

Notable mentions

Thanks to @timbyr for adding an often requested feature, the ability to support environment variable expansion in config files!

2837 timbyr: Configuration: Support environment expansion in configuration

Thanks to @huikang for adding a new docker-compose file for running Loki as microservices!

2740 huikang: Deploy: add docker-compose cluster deployment file

All Changes

For a full list of changes, please checkout the CHANGELOG

Installation:

The components of Loki are currently distributed in plain binary form and as Docker container images. Choose what fits your use-case best.

Docker container:

$ docker pull "grafana/loki:2.1.0"
$ docker pull "grafana/promtail:2.1.0"

Binary

$ curl -O -L "https://github.com/grafana/loki/releases/download/v2.1.0/loki-linux-amd64.zip"
# extract the binary
$ unzip "loki-linux-amd64.zip"
# make sure it is executable
$ chmod a+x "loki-linux-amd64"

Assets 27

23 Dec 20:32

slim-bean

v2.0.1

d5141c0

v2.0.1

2.0.1 is a special release, it only exists to add the v3 support to Loki's chunk format.

There is no reason to upgrade from 2.0.0 to 2.0.1

This chunk version is internal to Loki and not configurable, and in a future version v3 will become the default (Likely 2.2.0).

We are creating this to enable users to roll back from a future release which was writing v3 chunks, back as far as 2.0.0 and still be able to read chunks.

This is mostly a safety measure to help if someone upgrades from 2.0.0 and skips versions to a future version which is writing v3 chunks and they encounter an issue which they would like to roll back. They would be able to then roll back to 2.0.1 and still read v3 chunks.

It should be noted this does not help anyone upgrading from a version older than 2.0.0, that is you should at least upgrade to 2.0.0 before going to a newer version if you are on a version older than 2.0.0.

Installation:

The components of Loki are currently distributed in plain binary form and as Docker container images. Choose what fits your use-case best.

Docker container:

$ docker pull "grafana/loki:2.0.1"
$ docker pull "grafana/promtail:2.0.1"

Binary

$ curl -O -L "https://github.com/grafana/loki/releases/download/v2.0.1/loki-linux-amd64.zip"
# extract the binary
$ unzip "loki-linux-amd64.zip"
# make sure it is executable
$ chmod a+x "loki-linux-amd64"

Assets 27

26 Oct 16:36

slim-bean

v2.0.0

6978ee5

v2.0.0

🎉🎉 Loki 2.0.0 is here!! 🎉🎉

We are extremely excited about the new features in 2.0.0, unlocking a whole new world of observability of our logs.

Thanks again for the many incredible contributions and improvements from the wonderful Loki community, we are very excited for the future!

Important Notes

Please Note There are several changes in this release which require your attention!

Anyone using a docker image please go read the upgrade guide!! There is one important consideration around a potentially breaking schema change depending on your configuration.
MAJOR changes have been made to the boltdb-shipper index, breaking changes are not expected but extra precautions are highly recommended, more details in the upgrade guide.
The long deprecated entry_parser config in Promtail has been removed, use pipeline_stages instead.

Check the upgrade guide for detailed information on all these changes.

2.0!!!!

There are too many PR's to list individually for the major improvements which we thought justified a 2.0 but here is the high level:

Significant enhancements to the LogQL query language!
- Parse your logs to extract labels at query time.
- Filter on query time extracted labels.
- Format your log lines any way you please!
- Graph the contents of your log lines as metrics, including support for many more of your favorite PromQL functions.
Generate prometheus alerts directly from your logs!
- Create alerts using the same prometheus alert rule syntax and let Loki send alerts directly to your Prometheus Alertmanager!
boltdb-shipper is now production ready!
- This is it! Now Loki only needs a single object store (S3,GCS,Filesystem...) to store all the data, no more Cassandra, DynamoDB or Bigtable!

We are extremely excited about these new features, expect some talks, webinars, and blogs where we explain all this new functionality in detail.

Notable mention

This is a small change but very helpful!

2737 dlemel8: cmd/loki: add "verify-config" flag

Thank you @dlemel8 for this PR! Now you can start Loki with -verify-config to make sure your config is valid and Loki will exit with a status code 0 if it is!

All Changes

For a full list of changes, please checkout the CHANGELOG

Installation:

The components of Loki are currently distributed in plain binary form and as Docker container images. Choose what fits your use-case best.

Docker container:

$ docker pull "grafana/loki:2.0.0"
$ docker pull "grafana/promtail:2.0.0"

Binary

$ curl -O -L "https://github.com/grafana/loki/releases/download/v2.0.0/loki-linux-amd64.zip"
# extract the binary
$ unzip "loki-linux-amd64.zip"
# make sure it is executable
$ chmod a+x "loki-linux-amd64"

Assets 27

24 Aug 19:29

slim-bean

v1.6.1

623858d

v1.6.1

This is a small release and only contains two fixes for Promtail:

2542 slim-bean: Promtail: implement shutdown for the no-op server
2532 slim-bean: Promtail: Restart the tailer if we fail to read and upate current position

The first only applies if you are running Promtail with both --stdin and --server.disabled=true flags.

The second is a minor rework to how Promtail handles a very specific error when attempting to read the size of a file and failing to do so.

Upgrading Promtail from 1.6.0 to 1.6.1 is only necessary if you have logs full of msg="error getting tail position and/or size",
the code changed in this release has been unchanged for a long time and we suspect very few people are seeing this issue.

No changes to any other components (Loki, Logcli, etc) are included in this release.

Installation:

The components of Loki are currently distributed in plain binary form and as Docker container images. Choose what fits your use-case best.

Docker container:

$ docker pull "grafana/loki:v1.6.1"
$ docker pull "grafana/promtail:v1.6.1"

Binary

$ curl -O -L "https://github.com/grafana/loki/releases/download/v1.6.1/loki-linux-amd64.zip"
# extract the binary
$ unzip "loki-linux-amd64.zip"
# make sure it is executable
$ chmod a+x "loki-linux-amd64"

Assets 27

13 Aug 18:17

slim-bean

v1.6.0

6e3e2fa

v1.6.0

It's the second thursday of the eighth month of the year which means it's time for another Loki Release!!

Before we highlight important features and changes, congratulations to @adityacs, who is the newest member of the Loki team!
Aditya has been regularly contributing to the Loki project for the past year, with each contribution better than the last.
Many of the items on the following list were thanks to his hard work. Thank you, Aditya, and welcome to the team!

I think we might have set a new record with 189 PR's in this release!

Important Notes

Please Note There are several changes in this release which might require your attention!

The NET_BIND_SERVICE capability was removed from the Loki process in the docker image, it's no longer possible to run Loki with the supplied image on a port less than 1024
If you run microservices, there is an important rollout sequence to prevent query errors.
Scrape configs have changed for Promtail in both Helm and Ksonnet affecting two labels: instance -> pod and container_name -> container.
Almost all of the Loki Canary metrics were renamed.
A few command line flags where changed (although they are likely not commonly used)
If you use ksonnet and run on GCS and Bigtable you may see an error in your config as a default value was removed.
If you are using boltdb-shipper, you will likekly need to add a new schema_config entry.

Check the upgrade guide for detailed information on all these changes.

Notable Features and Fixes

Query language enhancements

2150 introduces bytes_rate, which calculates the per second byte rate of a log stream, and bytes_over_time, which returns the byte size of a log stream.
2182 introduces a long list of comparison operators, which will let you write queries like count_over_time({foo="bar"}[1m]) > 10. Check out the PR for a more detailed description.

Loki performance improvements

2216, 2218, and 2219 all improve how memory is allocated and reused for queries.
2239 is a huge improvement for certain cases in which a query covers a large number of streams that all overlap in time. Overlapping data is now internally cached while Loki works to sort all the streams into the proper time order.
2293 was a big refactor to how Loki internally processes log queries vs. metric queries, creating separate code paths to further optimize metric queries. Metric query performance is now 2 to 10 times faster.

If you are using the query-frontend:

2441 improves how label queries can be split and queried in parallel
2123 allows queries to the series API to be split by time and parallelized; and last but most significant
1927 allows for a much larger range of queries to be sharded and performed in parallel. Query sharding is a topic in itself, but as a rough summary, this type of sharding is not time dependent and leverages how data is already stored by Loki to be able to split queries up into 16 separate pieces to be queried at the same time.

Promtail

2296 allows Promtail to expose the Loki Push API. With this, you can push from any client to Promtail as if it were Loki, and Promtail can then forward those logs to another Promtail or to Loki. There are some good use cases for this with the Loki Docker Logging Driver; if you want an easier way to configure pipelines or expose metrics collection, point your Docker drivers at a Promtail instance.
2282 contains an example Amazon Lambda where you can use a fan-in approach and ingestion timestamping in Promtail to work around out of order issues with multiple Lambdas processing the same log stream. This is one way to get logs from a high-cardinality source without adding a high-cardinality label.
2060 introduces the Replace stage, which lets you find and replace or remove text inside a log line. Combined with 2422 and 2480, you can now find and replace sensitive data in a log line like a password or email address and replace it with ****, or hash the value to prevent readability, while still being able to trace the value through your logs. Last on the list of pipeline additions,
2496 adds a Drop pipeline stage, which lets you drop log lines based on several criteria options including regex matching content, line length, or the age of the log line. The last two are useful to prevent sending to Loki logs that you know would be rejected based on configured limits in the Loki server.

Logstash output plugin

1822 added a Logstash output plugin for Loki. If you have an existing Logstash install, you can now use this plugin to send your logs to Loki to make it easier to try out, or use Loki alongside an existing logging installation.

Loki Canary

2344 improved the canaries capabilities for checking for data integrity, including spot checking for logs over a longer time window and running metric queries to verify count_over_time accuracy.

Logcli

2470 allows you to color code your log lines based on their stream labels for a nice visual indicator of streams.
2497 expands on the series API query to Loki with the--analyze-labels flag, which can show you a detailed breakdown of your label key and value combinations. This is very useful for finding improper label usage in Loki or labels with high cardinality.
2482, in which LogCLI will automatically batch requests to Loki to allow making queries with a --limit= far larger than the server side limit defined in Loki. LogCLI will dispatch the request in a series of queries configured by the --batch= parameter (which defaults to 1000) until the requested limit is reached!

Misc

2453 improves the error messages when a query times out, as Context Deadline Exceeded wasn’t the most intuitive.
2336 provides two new flags that will print the entire Loki config object at startup. Be warned there are a lot of config options, and many won’t apply to your setup (such as storage configs you aren’t using), but this can be a really useful tool when troubleshooting. Sticking with the theme of best for last,
2224 and 2288 improve support for running Loki with a shared Ring using memberlist while not requiring Consul or Etcd. We need to follow up soon with some better documentation or a blog post on this!

All Changes

For a full list of changes, please checkout the CHANGELOG

Dependencies

Go Version: 1.14.2
Cortex Version: 7014ff11ed70d9d59ad29d0a95e73999c436c47c

Installation:

The components of Loki are currently distributed in plain binary form and as Docker container images. Choose what fits your use-case best.

Docker container:

$ docker pull "grafana/loki:1.6.0"
$ docker pull "grafana/promtail:1.6.0"

Binary

$ curl -O -L "https://github.com/grafana/loki/releases/download/v1.6.0/loki-linux-amd64.zip"
# extract the binary
$ unzip "loki-linux-amd64.zip"
# make sure it is executable
$ chmod a+x "loki-linux-amd64"

Assets 27

20 May 16:56

slim-bean

v1.5.0

12c7eab

v1.5.0

It's been a busy month and a half since 1.4.0 was released, and a lot of new improvements have been added to Loki since!

Be prepared for some configuration changes that may cause some bumps when upgrading,
we apologize for this but are always striving to reach the right compromise of code simplicity and user/operating experience.

In this case we opted to keep a simplified configuration inline with Cortex rather than a more complicated and error prone internal config mapping or difficult to implement support for multiple config names for the same feature.

This does result in breaking config changes for some configurations, however, these will fail fast and with the list of diffs from the Cortex project should be quick to fix.

Important Notes

Be prepared for breaking config changes. Loki 1.5.0 vendors cortex v1.0.1-0.20200430170006-3462eb63f324,
there were substantial breaking config changes in Cortex 1.0 which standardized config options, and fixed typos.

The Loki docker image user has changed to no longer be root

Check the upgrade guide for more detailed information on these changes.

Notable Features and Fixes

There are quite a few we want to mention listed in order they were merged (mostly)

1837 sandeepsukhani: flush boltdb to object store

This is perhaps the most exciting feature of 1.5.0, the first steps in removing a dependency on a separate index store! This feature is still very new and experimental, however, we want this to be the future for Loki. Only requiring just an object store.

If you want to test this new feature, and help us find any bugs, check out the docs to learn more and get started.

2073 slim-bean: Loki: Allow configuring query_store_max_look_back_period when running a filesystem store and boltdb-shipper

This is even more experimental than the previous feature mentioned however also pretty exciting for Loki users who use the filesystem storage. We can leverage changes made in 1837 to now allow Loki to run in a clustered mode with individual filesystem stores!

Please check out the last section in the filesystem docs for more details on how this works and how to use it!

2095 cyriltovena: Adds backtick for the quoted string token lexer.

This will come as a big win to anyone who is writing complicated reqular expressions in either their Label matchers or Filter Expressions. Starting now you can use the backtick to encapsulate your regex and not have to do any escaping of special characters!!

Examples:

{name="cassandra"} |~ `error=\w+`
{name!~`mysql-\d+`}

2055 aknuds1: Chore: Fix spelling of per second in code

This is technically a breaking change for anyone who wrote code to processes the new statistics output in the query result added in 1.4.0, we apologize to anyone in this situation but if we don't fix this kind of error now it will be there forever.
And at the same time we didn't feel it was appropriate to make any major api revision changes for such a new feature and simple change. We are always trying to use our best judgement in cases like this.

2031 cyriltovena: Improve protobuf serialization

Thanks @cyriltovena for another big performance improvement in Loki, this time around protbuf's!

2021 slim-bean: Loki: refactor validation and improve error messages
2012 slim-bean: Loki: Improve logging and add metrics to streams dropped by stream limit

These two changes standardize the metrics used to report when a tenant hits a limit, now all discarded samples should be reported under loki_discarded_samples_total and you no longer need to also reference cortex_discarded_samples_total.
Additionally error messages were improved to help clients take better action when hitting limits.

1970 cyriltovena: Allow to aggregate binary operations.

Another nice improvement to the query language which allows queries like this to work now:

sum by (job) (count_over_time({namespace="tns"}[5m] |= "level=error") / count_over_time({namespace="tns"}[5m]))

1713 adityacs: Log error message for invalid checksum

In the event something went wrong with a stored chunk, rather than fail the query we ignore the chunk and return the rest.

2066 slim-bean: Promtail: metrics stage can also count line bytes

This is a nice extension to a previous feature which let you add a metric to count log lines per stream, you can now count log bytes per stream.

Check out this example to configure this in your promtail pipelines.

1935 cyriltovena: Support stdin target via flag instead of automatic detection.

Third times a charm! With 1.4.0 we allowed sending logs directly to promtail via stdin, with 1.4.1 we released a patch for this feature which wasn't detecting stdin correctly on some operating systems.
Unfortunately after a few more bug reports it seems this change caused some more undesired side effects so we decided to not try to autodetect stdin at all, instead now you must pass the --stdin flag if you want Promtail to listen for logs on stdin.

2076 cyriltovena: Allows to pass inlined pipeline stages to the docker driver.
1906 cyriltovena: Add no-file and keep-file log option for docker driver.

The docker logging driver received a couple very nice updates, it's always been challenging to configure pipeline stages for the docker driver, with the first PR there are now a few easier ways to do this!
In the second PR we added config options to control keeping any log files on the host when using the docker logging driver, allowing you to run with no disk access if you would like, as well as allowing you to control keeping log files available after container restarts.

1864 cyriltovena: Sign helm package with GPG.

We now GPG sign helm packages!

All Changes

Too many to list here, see the CHANGELOG for a full list of changes!

Installation:

The components of Loki are currently distributed in plain binary form and as Docker container images. Choose what fits your use-case best.

Docker container:

$ docker pull "grafana/loki:1.5.0"
$ docker pull "grafana/promtail:1.5.0"

Binary

$ curl -O -L "https://github.com/grafana/loki/releases/download/v1.5.0/loki-linux-amd64.zip"
# extract the binary
$ unzip "loki-linux-amd64.zip"
# make sure it is executable
$ chmod a+x "loki-linux-amd64"

Assets 27

Releases: grafana/loki

v2.4.0

🎊 🌮 Loki 2.4.0 🌮 🎊

Features and enhancements

Upgrade Considerations

Bug fixes

2.4.0 bug fixes

Security fixes

2.4.0 security fixes

Installation:

Docker container:

Binary

v2.3.0

🥳🎉 Loki 2.3.0 🎉🥳

Features and enhancements

Upgrade considerations

Security fixes

2.3.0 security fixes

Bug fixes

2.3.0 bug fixes

Installation:

Docker container:

Binary

v2.2.1

Installation:

Docker container:

Binary

v2.2.0

🎉🎉 Loki 2.2.0 is released!! 🎉🎉

Installation:

Docker container:

Binary

v2.1.0

🎆🎁Happy Holidays from the Loki team!🎁🎆

Notable changes

Helm users read this!

Fluent bit plugin users read this!

Fixes

Enhancements

Notable mentions

All Changes

Installation:

Docker container:

Binary

v2.0.1

Installation:

Docker container:

Binary

v2.0.0

🎉🎉 Loki 2.0.0 is here!! 🎉🎉

Important Notes

2.0!!!!

Notable mention

All Changes

Installation:

Docker container:

Binary

v1.6.1

Installation:

Docker container:

Binary

v1.6.0

Important Notes

Notable Features and Fixes

Query language enhancements

Loki performance improvements

Promtail

Logstash output plugin

Loki Canary

Logcli

Misc

All Changes

Dependencies

Installation:

Docker container:

Binary

v1.5.0

Important Notes

Notable Features and Fixes

All Changes