A minimal and extensible structured logger

⚠️   PRE-RELEASE   ⚠️

DO NOT IMPORT THIS MODULE

YOUR PROJECT WILL BREAK

package log

package log provides a minimal interface for structured logging in services. It may be wrapped to encode conventions, enforce type-safety, provide leveled logging, and so on. It can be used for both typical application log events, and log-structured data streams.

Structured logging

Structured logging is, basically, conceding to the reality that logs are data, and warrant some level of schematic rigor. Using a stricter, key/value-oriented message format for our logs, containing contextual and semantic information, makes it much easier to get insight into the operational activity of the systems we build. Consequently, package log is of the strong belief that "the benefits of structured logging outweigh the minimal effort involved".

Migrating from unstructured to structured logging is probably a lot easier than you'd expect.

// Unstructured
log.Printf("HTTP server listening on %s", addr)

// Structured
logger.Log("transport", "HTTP", "addr", addr, "msg", "listening")

Usage

Typical application logging

w := log.NewSyncWriter(os.Stderr)
logger := log.NewLogfmtLogger(w)
logger.Log("question", "what is the meaning of life?", "answer", 42)

// Output:
// question="what is the meaning of life?" answer=42

Contextual Loggers

func main() {
	var logger log.Logger
	logger = log.NewLogfmtLogger(log.NewSyncWriter(os.Stderr))
	logger = log.With(logger, "instance_id", 123)

	logger.Log("msg", "starting")
	NewWorker(log.With(logger, "component", "worker")).Run()
	NewSlacker(log.With(logger, "component", "slacker")).Run()
}

// Output:
// instance_id=123 msg=starting
// instance_id=123 component=worker msg=running
// instance_id=123 component=slacker msg=running

Interact with stdlib logger

Redirect stdlib logger to Go kit logger.

import (
	"os"
	stdlog "log"
	kitlog "github.com/go-kit/log"
)

func main() {
	logger := kitlog.NewJSONLogger(kitlog.NewSyncWriter(os.Stdout))
	stdlog.SetOutput(kitlog.NewStdlibAdapter(logger))
	stdlog.Print("I sure like pie")
}

// Output:
// {"msg":"I sure like pie","ts":"2016/01/01 12:34:56"}

Or, if, for legacy reasons, you need to pipe all of your logging through the stdlib log package, you can redirect Go kit logger to the stdlib logger.

logger := kitlog.NewLogfmtLogger(kitlog.StdlibWriter{})
logger.Log("legacy", true, "msg", "at least it's something")

// Output:
// 2016/01/01 12:34:56 legacy=true msg="at least it's something"

Timestamps and callers

var logger log.Logger
logger = log.NewLogfmtLogger(log.NewSyncWriter(os.Stderr))
logger = log.With(logger, "ts", log.DefaultTimestampUTC, "caller", log.DefaultCaller)

logger.Log("msg", "hello")

// Output:
// ts=2016-01-01T12:34:56Z caller=main.go:15 msg=hello

Levels

Log levels are supported via the level package.

Supported output formats

Enhancements

package log is centered on the one-method Logger interface.

type Logger interface {
	Log(keyvals ...interface{}) error
}

This interface, and its supporting code like is the product of much iteration and evaluation. For more details on the evolution of the Logger interface, see The Hunt for a Logger Interface, a talk by Chris Hines. Also, please see #63, #76, #131, #157, #164, and #252 to review historical conversations about package log and the Logger interface.

Value-add packages and suggestions, like improvements to the leveled logger, are of course welcome. Good proposals should

  • Be composable with contextual loggers,
  • Not break the behavior of log.Caller in any wrapped contextual loggers, and
  • Be friendly to packages that accept only an unadorned log.Logger.

Benchmarks & comparisons

There are a few Go logging benchmarks and comparisons that include Go kit's package log.

Owner
Go kit
A Go toolkit for microservices.
Go kit
Comments
  • Allow to configure allowed levels by string value

    Allow to configure allowed levels by string value

    This PR allows to configure the allowed log levels from a string.

    Closes: #18

    Use Case:

    Read the allowed log levels reading environment variable

    logger = level.NewFilter(logger, level.AllowByString(os.Getenv("LOG_LEVEL"), level.AllowInfo)) // <--
    
  • Updates for Go 1.17

    Updates for Go 1.17

    I want to update this package and tag a patch release before updating go-kit/kit in https://github.com/go-kit/kit/pull/1105. I've already updated both of this package's dependencies along the same lines. Just working my way up the tree to minimize the number of new releases and maximize the benefit of the Go 1.17 module pruning feature for consumers.

  • add NewTestingLogger for usage in tests

    add NewTestingLogger for usage in tests

    Having access to the log can be quite useful in tests. However it is even nicer if the log is displayed only for failing tests, as well as the location where the call was made.

    This PR add a NewTestingLogger function which supports such usecase, by calling t.Helper and t.Log with the logfmt-formatted string.

    It uses logfmt.MarshalKeyvals directly, since I don't think that performance is so critical in tests.

  • Replacing

    Replacing "level" field

    Hi,

    logger := log.NewLogfmtLogger(log.NewSyncWriter(os.Stdout))
    logger.Log("msg", "foo")
    
    logger = level.Error(logger)
    logger.Log("msg", "foo")
    
    logger = level.Info(logger)
    logger.Log("msg", "foo")
    

    produces the following output playground:

    msg=foo
    level=error msg=foo
    level=info level=error msg=foo
    

    so level appears twice in last log.

    What do you think about adding a new function in log:

    func Replace(logger Logger, k, v interface{})
    

    which works as log.With but replace key k if it already exists, in logger, in the new returned Logger instance?

    So level.{Debug,Error,Info,Warn} can use it instead of log.WithPrefix as now.

    Replace could be variadic as With* functions, but is there a real need?

    If you are interested (variadic or not), I can provide a PR.

  • consider providing a method to compute the log level based on a String

    consider providing a method to compute the log level based on a String

    For most of our apps we setup the log level based on environment variables. However go-kit/log currently has hardcoded internal levels which makes mapping from the env var setting to the real level more cumbersome.

    I checked the code since you already have a Value interface for the log levels which should be usable through this function, unfortunately it seems unused as here it is casted to levelValue instead of Value.

    Hope it helps

  • Update CI and add badges to README

    Update CI and add badges to README

    CI updates:

    • Update Go versions (1.17.x and 1.18.x)
    • Update actions versions
    • Use actions for staticcheck
    • Drop golint (deprecated)
    • Report test coverage
  • Filtered-out log.Level lines still cost

    Filtered-out log.Level lines still cost

    I noticed a portion of our CPU was going into level.Debug(...).Log() lines, although that level wasn't enabled.

    I think what is happening is the Log() call goes into context.Log() first, which calls bindValues which is the expensive part, calling runtime.Caller, formatting a timestamp, etc., then after that it goes to level.Log() which discards everything.

    I looked around but didn't find any previous debate on this point. Am I holding it wrong? Is this expected?

    Since we are using a wrapper, something like https://github.com/go-kit/kit/pull/322 would let us work around the problem by checking before doing the work, but that was rejected.

  • broken link for slides

    broken link for slides "The Hunt for a Logger Interface"

    Hello, from the README, following the link to the presentation slides "The Hunt for a Logger Interface" at http://go-talks.appspot.com/github.com/ChrisHines/talks/structured-logging/structured-logging.slide#1

    gives

    Error accessing api.github.com.
    403: (https://api.github.com/repos/ChrisHines/talks?client_id=XXX&client_secret=YYY)
    
  • Leveled Logger: Get current log-level from logger

    Leveled Logger: Get current log-level from logger

    I'm using the leveled logger and would like to check the log level of the logger instance. Given an instance, is there any way to get the current log level it is set to?

    Context: I want to see which level is used before running a specific compute action that can be omitted if the logger is not on a debug level.

  • Logging error type that panic on Error() will break json format

    Logging error type that panic on Error() will break json format

    Hi, i found that logging error that cause panic when Error() func called, will cause json logger implementation to break. in contrast, logfmt format implementation safely output the log even if its failed to log the error description. i also put a sample code to replicate the issue on the bottom.

    i think the safeError() & safeString on json format implementation should always recover from panic and instead return the panic description that happen when its encode the value into string.

    i can help to provide a pr for it if you agree with my suggestion. thanks in advance!

    package main
    
    import (
    	kitlog "github.com/go-kit/log"
    	"os"
    )
    
    type myerror struct {
    }
    
    func main() {
    	// will safely output log even its failed
    	log := kitlog.NewLogfmtLogger(os.Stdout)
    	log.Log("err", myerror{})
    
    	// will panic
    	log = kitlog.NewJSONLogger(os.Stdout)
    	log.Log("err", myerror{})
    }
    
    func (e myerror) Error() string {
    	panic("fail retrurn error value")
    	return "test"
    }
    
    

    output:

    go run main.go
    err="PANIC:fail retrurn error value"
    panic: fail retrurn error value [recovered]
        panic: fail retrurn error value
    
    goroutine 1 [running]:
    github.com/go-kit/log.safeError.func1()
        /home/runner/go/pkg/mod/github.com/go-kit/[email protected]/json_logger.go:85 +0x134
    ...
    
  • feat: add a new zap logger adapter witch support github.com/go-kit/log/level

    feat: add a new zap logger adapter witch support github.com/go-kit/log/level

    i have edited the https://github.com/go-kit/kit/tree/master/log/zap adapter, and added a new feature to support github.com/go-kit/log/level, the new logger will try to parse kit-log's level into zap logger's level, if fail, it will use default log level instead

  • Line-buffered logger

    Line-buffered logger

    The current logger implementation writes a new line to the underlying writer every time a line is logged. If this writer is a file handle, this is mechanically unsympathetic to the Go scheduler because the goroutine and the underlying M (OS thread) are blocked while this synchronous operation occurs.

    This is generally not a big deal, but when you have a program writing thousands of lines a second (we can debate the merits of that separately), this can cause a fair amount of scheduler churn. Each file write incurs both a goroutine and an OS thread context-switch, which adds up fast in high performance software.

    A buffered logger protects against this by deamplifying the writes to the underlying writer (only useful if it's a file handle). We could use bufio.NewWriter, but then we would have incomplete lines being flushed. A line-buffered logger is the solution here.

    The logger is configured to wait until a number of lines has been written, and then a flush will be triggered. I've added a couple extra features such as a preallocated buffer and a flush timeout.

    In the benchmark below, I've chosen a buffer size of 256 lines, and comparing it against a normal logger writing each line to the file. Both benchmarks measure the time to write 1 million lines.

    $ go test -test.bench '\QLineBuffered\E$' -test.run '^$' -benchtime=1000000x  
    goos: linux
    goarch: amd64
    pkg: github.com/go-kit/log
    cpu: Intel(R) Core(TM) i7-10510U CPU @ 1.80GHz
    BenchmarkLineBuffered/capacity:256-8             1000000               399.5 ns/op            96 B/op          2 allocs/op
    PASS
    ok      github.com/go-kit/log   0.462s
    
    $ go test -test.bench '\QLineUnbuffered\E$' -test.run '^$' -benchtime=1000000x
    goos: linux
    goarch: amd64
    pkg: github.com/go-kit/log
    cpu: Intel(R) Core(TM) i7-10510U CPU @ 1.80GHz
    BenchmarkLineUnbuffered-8        1000000              1337 ns/op              96 B/op          2 allocs/op
    PASS
    

    This change improves log writing performance by roughly 70% when the writer is a file handle, and performs roughly the same if the writer is io.Discard:

    $ go test -test.bench '\QLineDiscard\E$' -test.run '^$' -benchtime=1000000x
    goos: linux
    goarch: amd64
    pkg: github.com/go-kit/log
    cpu: Intel(R) Core(TM) i7-10510U CPU @ 1.80GHz
    BenchmarkLineDiscard-8           1000000               369.4 ns/op            96 B/op          2 allocs/op
    PASS
    ok      github.com/go-kit/log   0.372s
    

    PS: the above benchmarks have been tested with an SSD (HP SSD EX900 Pro 1TB).

    PPS: I'm a maintainer for Grafana Loki and we implemented this in https://github.com/grafana/loki/pull/6954.

  • Idea: 'Helper' method for Caller

    Idea: 'Helper' method for Caller

    Currently, it is necessary to figure out how each log method will be called in order to specify a count of stack frames to Caller. https://github.com/go-kit/log/blob/5739c2646c7293c773ed50d2086e843e3330781c/value.go#L82-L84

    This makes functions that wrap or forward to go-kit/log hard to use.

    Suggestion: have a Helper method like in testing, which tells Caller to skip that function from the count.

    This would be relatively expensive – a stack walk, lock and map lookup each time the helper is entered, and more map lookups when actually logging – but much more pleasant to use.

  • Documentation link broken

    Documentation link broken

    https://www.thoughtworks.com/radar/techniques/structured-logging in the README is no longer maintained:

    image

    We should either use a web archive link or link to a different article.

Minimal structured logging library for Go
Minimal structured logging library for Go

slog slog is a minimal structured logging library for Go. Install go get cdr.dev/slog Features Minimal API First class context.Context support First c

Dec 29, 2022
Convenient Logger interface and std logger wrapper

Convenient logger interface and wrapper around std logger Interface type Logger interface { Error(err error) Debugf(format string, args ...interface

Nov 28, 2021
Logger - Simple logger without written with std pkg

Go-Logger Simple usage is: package main

Jan 2, 2022
Logger - A thin wrapper of uber-go/zap logger for personal project

a thin wraper of uber-go/zap logger for personal project 0. thanks uber-go/zap B

Sep 17, 2022
Simple, configurable and scalable Structured Logging for Go.

log Log is a simple, highly configurable, Structured Logging library Why another logging library? There's allot of great stuff out there, but also tho

Sep 26, 2022
Hierarchical, leveled, and structured logging library for Go

spacelog Please see http://godoc.org/github.com/spacemonkeygo/spacelog for info License Copyright (C) 2014 Space Monkey, Inc. Licensed under the Apach

Apr 27, 2021
Search and analysis tooling for structured logs

Zed The Zed system provides an open-source, cloud-native, and searchable data lake for semi-structured and structured data. Zed lakes utilize a supers

Jan 5, 2023
Simple and extensible monitoring agent / library for Kubernetes: https://gravitational.com/blog/monitoring_kubernetes_satellite/

Satellite Satellite is an agent written in Go for collecting health information in a kubernetes cluster. It is both a library and an application. As a

Nov 10, 2022
Structured logging package for Go.
Structured logging package for Go.

Package log implements a simple structured logging API inspired by Logrus, designed with centralization in mind. Read more on Medium. Handlers apexlog

Dec 24, 2022
Structured log interface

Structured log interface Package log provides the separation of the logging interface from its implementation and decouples the logger backend from yo

Sep 26, 2022
Structured, composable logging for Go
Structured, composable logging for Go

log15 Package log15 provides an opinionated, simple toolkit for best-practice logging in Go (golang) that is both human and machine readable. It is mo

Dec 18, 2022
Structured, pluggable logging for Go.
Structured, pluggable logging for Go.

Logrus Logrus is a structured logger for Go (golang), completely API compatible with the standard library logger. Logrus is in maintenance-mode. We wi

Jan 9, 2023
Structured Logging Made Easy
Structured Logging Made Easy

Structured Logging Made Easy Features Dependency Free Simple and Clean Interface Consistent Writer IOWriter, io.Writer wrapper FileWriter, rotating &

Jan 3, 2023
Blazing fast, structured, leveled logging in Go.

⚡ zap Blazing fast, structured, leveled logging in Go. Installation go get -u go.uber.org/zap Note that zap only supports the two most recent minor ve

Jan 7, 2023
Logrus is a structured, pluggable logging for Go.
Logrus is a structured, pluggable logging for Go.

Logrus is a structured logger for Go (golang), completely API compatible with the standard library logger.

May 25, 2021
Fully asynchronous, structured, pluggable logging for Go.

logr Logr is a fully asynchronous, contextual logger for Go. It is very much inspired by Logrus but addresses two issues: Logr is fully asynchronous,

Dec 28, 2022
structured logging helper

Logart Logart is a structured logging tool that aims to simplify logging to a database It is not yet in stable state, but is used in production and ac

Apr 24, 2021
Log-structured virtual disk in Ceph
Log-structured virtual disk in Ceph

lsd_ceph Log-structured virtual disk in Ceph 1. Vision and Goals of the Project Implement the basic librbd API to work with the research block device

Dec 13, 2021
Go-metalog - Standard API for structured logging

Metalog is a standard API for structured logging and adapters for its implementa

Jan 20, 2022