Like Prometheus, but for logs.

Loki Logo

Drone CI CircleCI Go Report Card Slack

Loki: like Prometheus, but for logs.

Loki is a horizontally-scalable, highly-available, multi-tenant log aggregation system inspired by Prometheus. It is designed to be very cost effective and easy to operate. It does not index the contents of the logs, but rather a set of labels for each log stream.

Compared to other log aggregation systems, Loki:

  • does not do full text indexing on logs. By storing compressed, unstructured logs and only indexing metadata, Loki is simpler to operate and cheaper to run.
  • indexes and groups log streams using the same labels you’re already using with Prometheus, enabling you to seamlessly switch between metrics and logs using the same labels that you’re already using with Prometheus.
  • is an especially good fit for storing Kubernetes Pod logs. Metadata such as Pod labels is automatically scraped and indexed.
  • has native support in Grafana (needs Grafana v6.0).

A Loki-based logging stack consists of 3 components:

  • promtail is the agent, responsible for gathering logs and sending them to Loki.
  • loki is the main server, responsible for storing logs and processing queries.
  • Grafana for querying and displaying the logs.

Loki is like Prometheus, but for logs: we prefer a multidimensional label-based approach to indexing, and want a single-binary, easy to operate system with no dependencies. Loki differs from Prometheus by focusing on logs instead of metrics, and delivering logs via push, instead of pull.

Getting started

Upgrading

Documentation

Commonly used sections:

  • API documentation for alternative ways of getting logs into Loki.
  • Labels
  • Operations for important aspects of running Loki.
  • Promtail is an agent which can tail your log files and push them to Loki.
  • Pipelines for detailed log processing pipeline documentation
  • Docker Logging Driver is a docker plugin to send logs directly to Loki from Docker containers.
  • LogCLI on how to query your logs without Grafana.
  • Loki Canary for monitoring your Loki installation for missing logs.
  • Troubleshooting for help around frequent error messages.
  • Loki in Grafana for how to set up a Loki datasource in Grafana and query your logs.

Getting Help

If you have any questions or feedback regarding Loki:

Your feedback is always welcome.

Further Reading

Contributing

Refer to CONTRIBUTING.md

Building from source

Loki can be run in a single host, no-dependencies mode using the following commands.

You need go, we recommend using the version found in our build Dockerfile

$ go get github.com/grafana/loki
$ cd $GOPATH/src/github.com/grafana/loki # GOPATH is $HOME/go by default.

$ go build ./cmd/loki
$ ./loki -config.file=./cmd/loki/loki-local-config.yaml
...

To build Promtail on non-Linux platforms, use the following command:

$ go build ./cmd/promtail

On Linux, Promtail requires the systemd headers to be installed for Journal support.

With Journal support on Ubuntu, run with the following commands:

$ sudo apt install -y libsystemd-dev
$ go build ./cmd/promtail

With Journal support on CentOS, run with the following commands:

$ sudo yum install -y systemd-devel
$ go build ./cmd/promtail

Otherwise, to build Promtail without Journal support, run go build with CGO disabled:

$ CGO_ENABLED=0 go build ./cmd/promtail

License

Apache License 2.0, see LICENSE.

Owner
Grafana Labs
Grafana Labs is behind leading open source projects Grafana and Loki, and the creator of the first open & composable observability platform.
Grafana Labs
Comments
  • Feature: multi-line logs

    Feature: multi-line logs

    Currently when an multi-line event is written to a logfile promtail will take this as each row is its own entry and send the separately to loki.

    It would be great if Loki could handle multi-line events due to stacktraces.

    This might also be a bug if promtail already should handle multi-lines.

  • Allow out of order log submission

    Allow out of order log submission

    I'd love to use Loki in a distributed system easier and without being forced relatively high cardinality labels based on something like process ID. This goes double for systems like AWS Lambda.

    This main obstacle to this for me is being unable to submit 'out of order' log lines, it would be great if loki could have a feature that would enable this.

    At one point I found an old issue relating to this request but it was closed with "not something we need before releasing". Perhaps it is time to revisit this?

    Cheers

  • OOMKIlled on every query

    OOMKIlled on every query

    Describe the bug Everytime I make a query against loki running the helm chart with 8G requests and limits, the server goes into OOMKilled and restarts.

    To Reproduce Steps to reproduce the behavior:

    1. Started loki (SHA or version): master as of today
    2. Started promtail (SHA or version) to tail '...'
    3. Query: {app="my-app"} exception

    Expected behavior I should get results. Instead I got crashes.

    Environment:

    • Infrastructure: Kubernetes
    • Deployment tool: helm

    Related to https://github.com/grafana/loki/issues/191

  • operator: Add support for built-in-cert-rotation for all internal lokistack encryption

    operator: Add support for built-in-cert-rotation for all internal lokistack encryption

    What this PR does / why we need it: The following PR adds support for built-in cert-rotation for all internal LokiStack data-in-transit encryption. This ensures that users enabling the feature gate HTTPEncryption and GRPCEncryption none of the running components except the lokistack-gateway can be access with a TLS client certificate signed by the operator-managed CA.

    In particular the lokistack-gateway requires from users enabling the HTTPEncryption feature gate to provide:

    • A custom CA bundle that consists of the list of singning certificates as a ConfigMap with a single key service-ca.crt
    • A custom Cert/Key pair that represents the serving certificate/key pair as a Secret with keys tls.crt and `tls.key

    This distinction is in place because the lokistack-gateway represents the public available API of the LokiStack. In general public APIs might be secured by a different PKI with custom validity, etc. The internal communication though between the lokistack-gateway and the upstream Loki services distributor and query-frontend is maintained by the built-in cert rotation provide PKI.

    Which issue(s) this PR fixes: Fixes #

    Special notes for your reviewer: Requires:

    • https://github.com/observatorium/api/pull/370
    • https://github.com/grafana/loki/pull/7069
    • https://github.com/grafana/loki/pull/7085
    • https://github.com/grafana/loki/pull/7328
    • https://github.com/grafana/loki/pull/7423

    Checklist

    • [ ] Documentation added
    • [x] Tests updated
    • [ ] Is this an important fix or new feature? Add an entry in the CHANGELOG.md.
    • [ ] Changes that require user attention or interaction to upgrade are documented in docs/sources/upgrading/_index.md
  • promtail: ratelimiting by label

    promtail: ratelimiting by label

    What this PR does / why we need it:

    Which issue(s) this PR fixes: Fixes #

    Special notes for your reviewer:

    Limit stage can now takes optional by_label_name param, which tracks distinct values of that label and ratelimit entries independently.

    Log lines without expected label are not considered for ratelimiting and passed to the next pipeline stage.

    To avoid stalling whole stage by just few labels it requires 'drop', to be set to true.

    Checklist

    • [x] Reviewed the CONTRIBUTING.md guide
    • [x] Documentation added
    • [x] Tests updated
    • [x] CHANGELOG.md updated
    • [ ] Changes that require user attention or interaction to upgrade are documented in docs/sources/upgrading/_index.md
  • Datasource proxy returning

    Datasource proxy returning "too many outstanding requests"

    What happened: We were trying out loki and the integration into grafana. When adding a dashboard with not only logs but also time series visualization, we encountered "too many outstanding requests" (HTTP 429) responses from grafana in the network monitor. The exclamation mark symbol was shown in the panels on the top left side with the same text "too many outstanding requests". However a quick google search for exactly that term in combination with loki OR grafana reveiled nothing that seemed to be the same issue.

    What you expected to happen: I would expect to be able to configure how many requests can be processed simultaneously and also to find possible response codes in the documentation. There should be a possibility to troubleshoot the issue without diving into the code.

    How to reproduce it (as minimally and precisely as possible):

    • Setup Prometheus stack with grafana
    • Setup Loki distributed
    • Ingest lots of events for ~24h
    • Add a dashboard with logs and time series visualization
    • Set timeframe to 24h or more

    Anything else we need to know?:

    Environment:

    • Grafana version: v8.1.5
    • Data source type & version: loki 2.3.0
    • OS Grafana is installed on: docker image grafana/grafana:8.1.5 on k8s
    • User OS & Browser: Linux / Chrome
    • Grafana plugins: Just core plugins
    • Others: -
  • Docker driver doesn't work with warning

    Docker driver doesn't work with warning "no logs are available with the 'loki' log driver"

    Describe the bug Docker driver configured as written in docs but after running docker-compose up docker shows warning. And there's also no logs in Loki.

    To Reproduce Steps to reproduce the behavior:

    1. Started Loki (master)
    2. Installed and enabled Docker-driver via docker plugin install grafana/loki-docker-driver:latest --alias loki --grant-all-permissions command
    3. Added logging section to docker-compose file as said in docs
    4. Run docker-compose up
    5. See the warning: app_1 | WARNING: no logs are available with the 'loki' log driver

    Expected behavior Logs should fall into Loki but they don't.

    Environment:

    • Infrastructure: Docker Desktop 2.1.0.5 (Docker Engine 19.03.5 and Compose 1.24.1) on MacOS Catalina
    • Deployment tool: docker-compose

    Screenshots, Promtail config, or terminal output docker-compose.yml

    version: '3.7'
    
    services:
      loki:
        image: grafana/loki:master
        container_name: loki
        command: -config.file=/etc/loki/local-config.yaml
      app:
        build:
          context: .
          dockerfile: Dockerfile
        logging:
          driver: loki
          options:
            loki-url: "http://loki:3100/loki/api/v1/push"
    
  • Add support for ARM64 to lambda-promtail drone build job

    Add support for ARM64 to lambda-promtail drone build job

    What this PR does / why we need it:

    Using ARM64 architecture for Lambda functions can lead to significant price/performance gain. For reference:

    • https://aws.amazon.com/about-aws/whats-new/2021/09/better-price-performance-aws-lambda-functions-aws-graviton2-processor/
    • https://aws.amazon.com/blogs/aws/aws-lambda-functions-powered-by-aws-graviton2-processor-run-your-functions-on-arm-and-get-up-to-34-better-price-performance/

    Up until now, the solution was to custom-build an ARM64 image for Promtail Lambda and push it to the private ECR registry. However, since the release of pull through cache repositories (https://aws.amazon.com/about-aws/whats-new/2021/11/amazon-ecr-cache-repositories/) it's now much easier to keep the images from public registries in private ECR registry up to date. This is especially true when using IaC (e.g. Pulumi).

    Which issue(s) this PR fixes:

    Special notes for your reviewer:

    Discussed on Slack with @cstyan (https://grafana.slack.com/archives/CEPJRLQNL/p1644258844007079).

    I tried to build the ARM64 image and use it in production with success.

    Checklist

    • [ ] Documentation added
    • [ ] Tests updated
    • [x] Add an entry in the CHANGELOG.md about the changes.
  • [bug] metrics: New version of loki s3 metrics becomes very unstable

    [bug] metrics: New version of loki s3 metrics becomes very unstable

    Presumably related to this PR. <<Remove instance level streams lock #5179>> https://github.com/grafana/loki/pull/5179

    sum(cortex_ingester_flush_queue_length)

    This metrics is currently our most important alarm metrics, P0 level. Our oncall developers are most concerned about this metrics, because the s3 in our region is not very stable.

    This PR can indeed effectively reduce the push duration, but if it affects the metrics of the sla alarm, we cannot use it in the production environment

    image

    image

  • Windows EventLog support

    Windows EventLog support

    Is your feature request related to a problem? Please describe. Windows logs are stored in Event Log (.evtx files), which currently not possible to scrape it via currently available promtail methods.

    Describe the solution you'd like Since we do have systemd journal support for Linux, it would be nice to have support for Event Log on Windows in a similar matter.

    Describe alternatives you've considered Key part of the solution is actually being able to parse the logs. If I haven't missed anything, it seems that there are currently two Golang modules that can do that: github.com/0xrawsec/golang-evtx and github.com/elastic/beats/winlogbeat/eventlog.

  • Support for JSON log lines

    Support for JSON log lines

    Loki looks very promising! 🏆

    Are there any plans to support ingestion of JSON log lines?

    It seems to be a pretty common structure for logs these days. Here are some examples (can add more):

    • https://www.loggly.com/blog/why-json-is-the-best-application-log-format-and-how-to-switch/
    • https://docs.fluentd.org/v0.12/articles/parser_json
    • https://www.envoyproxy.io/docs/envoy/latest/configuration/access_log#format-dictionaries
    • https://docs.docker.com/config/containers/logging/json-file/
    • https://jahed.io/2018/07/05/always-log-to-json/
  • make query-frontend service grpc lb friendly

    make query-frontend service grpc lb friendly

    What this PR does / why we need it: In PR #7966, I made changes to Query Frontend service. Part of it was creating a new service. To make the service properly load balance grpc requests, we need to use grpclbServiceFor, but since we do not use grpc in QF, I went with the usual method, i.e. I used serviceFor. Thinking more about it, I think there is no harm in keeping it grpc LB friendly, and it would save us from any possible bugs if we make QF accept queries via grpc.

  • [loki-canary] Support passing loki address as environment variable

    [loki-canary] Support passing loki address as environment variable

    What this PR does / why we need it:

    • Replace flag with kingpin which has more capabilities to override defaults
    • Override default for addr, user and pass from environment the variables LOKI_ADDRESS, LOKI_USERNAME and LOKI_PASSWORD respectively.

    Which issue(s) this PR fixes: Fixes #8021

    Special notes for your reviewer:

    Checklist

    • [x] Reviewed the CONTRIBUTING.md guide
    • [x] Documentation added
    • [ ] Tests updated
    • [x] CHANGELOG.md updated
    • [ ] Changes that require user attention or interaction to upgrade are documented in docs/sources/upgrading/_index.md
  • grafana-loki via Helm Chart, logs older than 2 hours disappear

    grafana-loki via Helm Chart, logs older than 2 hours disappear

    Describe the bug I have installed grafana-loki from the bitnami helm chart repository and it works so far. My only issue is, logs older than 2 hours disappear. Been looking for the past 3 days and I can't find the issue. The settings are pretty much default,

    The values are pretty much default, I only disabled promtail since it was getting logs from all pods and I was only interested in containers sending logs via Serilog Sink to the loki instance.

    To Reproduce Install grafana loki from bitnami via

    helm install grafana-loki bitnami/grafana-loki -n monitoring --values grafana-loki.yaml
    

    and add it as data source to grafana.

    Expected behavior Logs to retain for 7 days (default settings)

    Environment:

    • Infrastructure: Self-Hosted Kubernetes Cluster (no could)
    • Deployment tool: helm

    Screenshots, Promtail config, or terminal output values.yaml

    promtail:
      enable: false
    
  • Use gotestsum to run unit tests

    Use gotestsum to run unit tests

    What this PR does / why we need it:

    Add gotestsum to the build image. Then use it to run the unit tests. The output format is easier to parse so we can get statistics about the builds.

    Passing test cases look like this:

    PASS pkg/storage/stores/shipper/index/compactor.TestTable_RecreateCompactedDB/compacted_db_old_enough (2.64s)
    PASS pkg/storage/stores/shipper/index/compactor.TestTable_RecreateCompactedDB (10.26s)
    PASS pkg/storage/stores/shipper/index/compactor.TestExtractIntervalFromTableName/0 (0.00s)
    

    Failing test cases are shown at the end and are easier to parse:

    === FAIL: pkg/storage/stores/indexshipper/compactor/deletion TestGRPCGetCacheGenNumbers (unknown)
    panic: Fail in goroutine after TestGRPCGetCacheGenNumbers/error_getting_from_store has completed
    
    goroutine 293 [running]:
    testing.(*common).Fail(0xc0005aa680)
            /usr/local/Cellar/go/1.19.4/libexec/src/testing/testing.go:824 +0xe5
    ...
    

    Checklist

    • [ ] Reviewed the CONTRIBUTING.md guide
    • [ ] Documentation added
    • [X] Tests updated
    • [ ] CHANGELOG.md updated
    • [ ] Changes that require user attention or interaction to upgrade are documented in docs/sources/upgrading/_index.md

    Signed-off-by: Michel Hollands [email protected]

  • [loki-canary] Support passing loki address as environment variable

    [loki-canary] Support passing loki address as environment variable

    Is your feature request related to a problem? Please describe. Loki canary takes all its configuration via command line arguments. This is an issue especially when passing configurations which contain secrets/tokens. It would be nice if the Loki write address which can have tokens to be read from the runtime environment.

    Describe the solution you'd like Support for passing configuration via environment variables as a fallback. The precedence can be command line arguments > environment.

    Describe alternatives you've considered NA

    Additional context NIL

  • Promtail hostPath mount

    Promtail hostPath mount

    Hi :) I’ve been trying to collect pods logs in openshift with Promtail, but it wouldn’t work for me because of permissions problems with the hostPath in the config file. Is it possible to replace the hostPath mounts to one I don’t need high permissions to?

Leveled execution logs for Go

glog ==== Leveled execution logs for Go. This is an efficient pure Go implementation of leveled logs in the manner of the open source C++ package h

Dec 24, 2022
Library and program to parse and forward HAProxy logs

haminer Library and program to parse and forward HAProxy logs. Supported forwarder, Influxdb Requirements Go for building from source code git for dow

Aug 17, 2022
raft variant with topology order logs

Safe: A log that is safe if it has been replicated to a quorum, no matter whether or not the committed flag is set on any replica.

May 28, 2022
Very powerful server agent for collecting & sending logs & metrics with an easy-to-use web console.
Very powerful server agent for collecting & sending logs & metrics with an easy-to-use web console.

logkit-community 中文版 Introduce Very powerful server agent for collecting & sending logs & metrics with an easy-to-use web console. logkit-community De

Dec 29, 2022
Pixie gives you instant visibility by giving access to metrics, events, traces and logs without changing code.
Pixie gives you instant visibility by giving access to metrics, events, traces and logs without changing code.

Pixie gives you instant visibility by giving access to metrics, events, traces and logs without changing code.

Jan 4, 2023
Search and analysis tooling for structured logs

Zed The Zed system provides an open-source, cloud-native, and searchable data lake for semi-structured and structured data. Zed lakes utilize a supers

Jan 5, 2023
gtl - Gemini Tiny Logs - A simple TUI for the tinylog format on gemini
gtl - Gemini Tiny Logs - A simple TUI for the tinylog format on gemini

GTL: Gemini Tiny Logs Goal: A TUI for the tinylogs format on the gemini space. See screenshots Installation gtl requires go ≥ 1.16 From Source git clo

Dec 1, 2022
mtail - extract internal monitoring data from application logs for collection into a timeseries database
 mtail - extract internal monitoring data from application logs for collection into a timeseries database

mtail - extract internal monitoring data from application logs for collection into a timeseries database mtail is a tool for extracting metrics from a

Dec 29, 2022
A customized GORM logger that implements the appropriate interface and uses Logrus to output logs

CryptoMath GORM Logger A customized GORM logger that implements the appropriate interface and uses Logrus to output logs. Install go get github.com/ma

Nov 6, 2021
Lumberjack is a Go package for writing logs to rolling files.

Lumberjack is a Go package for writing logs to rolling files.

Feb 24, 2022
This POC is built with the goal to collect events/logs from the host systems such as Kubernetes, Docker, VMs, etc. A buffering layer is added to buffer events from the collector
This POC is built with the goal to collect events/logs from the host systems such as Kubernetes, Docker, VMs, etc. A buffering layer is added to buffer events from the collector

What is does This POC is build with the goal to collect events/logs from the host systems such as Kubernetes, docker, VMs etc. A buffering layer is ad

Nov 11, 2022
Leveled execution logs for Go.

glog Leveled execution logs for Go. This is an efficient pure Go implementation of leveled logs in the manner of the open source C++ package glog. By

Nov 29, 2021
WIP Go Thing to download HCP Vault Logs

Example Go Script to pull HCP Vault Audit Logs WARNING: This makes use of unstable preview APIs which could change at any time! USE AT YOUR OWN PERIL

Feb 6, 2022
Stream logs through websockets, written in Go

Stream logs through websockets, written in Go

Jan 8, 2022
Request-logging-tool - A tool logs the md5 codes of the responses of the given domains in parameter

request-logging-tool Application to send http requests and log the md5 responses

Jan 7, 2022
Log-generator - A simple CLI tool that generates near real logs for testing

Log-generator - A simple CLI tool that generates near real logs for testing

Jan 22, 2022
A logrus.Hook that logs with a zap.Logger

zaprus Ever had a 3rd-party dependency requiring a logrus, but you're using zap? zaprus provides a logrus.Hook that makes a logrus.(Entry|Logger) repl

Feb 27, 2022
Logger - Some implementations for Logs written in Go

Logger will be a tool for sending logs to different places in your software. Rig

Feb 16, 2022
A simple to use log system, minimalist but with features for debugging and differentiation of messages
A simple to use log system, minimalist but with features for debugging and differentiation of messages

A simple to use log system, minimalist but with features for debugging and differentiation of messages

Sep 26, 2022