Port of perl5 File::RotateLogs to Go

file-rotatelogs

Provide an io.Writer that periodically rotates log files from within the application. Port of File::RotateLogs from Perl to Go.

Build Status

GoDoc

SYNOPSIS

import (
  "log"
  "net/http"

  apachelog "github.com/lestrrat-go/apache-logformat"
  rotatelogs "github.com/lestrrat-go/file-rotatelogs"
)

func main() {
  mux := http.NewServeMux()
  mux.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) { ... })

  logf, err := rotatelogs.New(
    "/path/to/access_log.%Y%m%d%H%M",
    rotatelogs.WithLinkName("/path/to/access_log"),
    rotatelogs.WithMaxAge(24 * time.Hour),
    rotatelogs.WithRotationTime(time.Hour),
  )
  if err != nil {
    log.Printf("failed to create rotatelogs: %s", err)
    return
  }

  // Now you must write to logf. apache-logformat library can create
  // a http.Handler that only writes the approriate logs for the request
  // to the given handle
  http.ListenAndServe(":8080", apachelog.CombinedLog.Wrap(mux, logf))
}

DESCRIPTION

When you integrate this to into your app, it automatically write to logs that are rotated from within the app: No more disk-full alerts because you forgot to setup logrotate!

To install, simply issue a go get:

go get github.com/lestrrat-go/file-rotatelogs

It's normally expected that this library is used with some other logging service, such as the built-in log library, or loggers such as github.com/lestrrat-go/apache-logformat.

import(
  "log"
  "github.com/lestrrat-go/file-rotatelogs"
)

func main() {
  rl, _ := rotatelogs.New("/path/to/access_log.%Y%m%d%H%M")

  log.SetOutput(rl)

  /* elsewhere ... */
  log.Printf("Hello, World!")
}

OPTIONS

Pattern (Required)

The pattern used to generate actual log file names. You should use patterns using the strftime (3) format. For example:

  rotatelogs.New("/var/log/myapp/log.%Y%m%d")

Clock (default: rotatelogs.Local)

You may specify an object that implements the roatatelogs.Clock interface. When this option is supplied, it's used to determine the current time to base all of the calculations on. For example, if you want to base your calculations in UTC, you may specify rotatelogs.UTC

  rotatelogs.New(
    "/var/log/myapp/log.%Y%m%d",
    rotatelogs.WithClock(rotatelogs.UTC),
  )

Location

This is an alternative to the WithClock option. Instead of providing an explicit clock, you can provide a location for you times. We will create a Clock object that produces times in your specified location, and configure the rotatelog to respect it.

LinkName (default: "")

Path where a symlink for the actual log file is placed. This allows you to always check at the same location for log files even if the logs were rotated

  rotatelogs.New(
    "/var/log/myapp/log.%Y%m%d",
    rotatelogs.WithLinkName("/var/log/myapp/current"),
  )
  // Else where
  $ tail -f /var/log/myapp/current

Links that share the same parent directory with the main log path will get a special treatment: namely, linked paths will be RELATIVE to the main log file.

Main log file name Link name Linked path
/path/to/log.%Y%m%d /path/to/log log.YYYYMMDD
/path/to/log.%Y%m%d /path/to/nested/log ../log.YYYYMMDD
/path/to/log.%Y%m%d /foo/bar/baz/log /path/to/log.YYYYMMDD

If not provided, no link will be written.

RotationTime (default: 86400 sec)

Interval between file rotation. By default logs are rotated every 86400 seconds. Note: Remember to use time.Duration values.

  // Rotate every hour
  rotatelogs.New(
    "/var/log/myapp/log.%Y%m%d",
    rotatelogs.WithRotationTime(time.Hour),
  )

MaxAge (default: 7 days)

Time to wait until old logs are purged. By default no logs are purged, which certainly isn't what you want. Note: Remember to use time.Duration values.

  // Purge logs older than 1 hour
  rotatelogs.New(
    "/var/log/myapp/log.%Y%m%d",
    rotatelogs.WithMaxAge(time.Hour),
  )

RotationCount (default: -1)

The number of files should be kept. By default, this option is disabled.

Note: MaxAge should be disabled by specifing WithMaxAge(-1) explicitly.

  // Purge logs except latest 7 files
  rotatelogs.New(
    "/var/log/myapp/log.%Y%m%d",
    rotatelogs.WithMaxAge(-1),
    rotatelogs.WithRotationCount(7),
  )

Handler (default: nil)

Sets the event handler to receive event notifications from the RotateLogs object. Currently only supported event type is FiledRotated

  rotatelogs.New(
    "/var/log/myapp/log.%Y%m%d",
    rotatelogs.WithHandler(rotatelogs.HandlerFunc(func(e rotatelogs.Event) {
      if e.Type() != rotatelogs.FileRotatedEventType {
        return
      }

      // Do what you want with the data. This is just an idea:
      storeLogFileToRemoteStorage(e.(*rotatelogs.FileRotatedEvent).PreviousFile())
    })),
  )

ForceNewFile

Ensure a new file is created every time New() is called. If the base file name already exists, an implicit rotation is performed.

  rotatelogs.New(
    "/var/log/myapp/log.%Y%m%d",
    rotatelogs.ForceNewFile(),
  )

Rotating files forcefully

If you want to rotate files forcefully before the actual rotation time has reached, you may use the Rotate() method. This method forcefully rotates the logs, but if the generated file name clashes, then a numeric suffix is added so that the new file will forcefully appear on disk.

For example, suppose you had a pattern of '%Y.log' with a rotation time of 86400 so that it only gets rotated every year, but for whatever reason you wanted to rotate the logs now, you could install a signal handler to trigger this rotation:

rl := rotatelogs.New(...)

signal.Notify(ch, syscall.SIGHUP)

go func(ch chan os.Signal) {
  <-ch
  rl.Rotate()
}()

And you will get a log file name in like 2018.log.1, 2018.log.2, etc.

Owner
Comments
  • set link to relative path file

    set link to relative path file

    logf := rotatelogs.New(
    		"/var/log/xxx/runtime.%Y%m%d",
    		rotatelogs.WithLinkName("/var/log/xxx/runtime"), 
    		rotatelogs.WithRotationTime(24*time.Hour),
    		rotatelogs.WithMaxAge(24*time.Hour*7),
    	)
    

    if my go app run in docker,with volume mount like this:

    volumes:
     - hostPath:
          path: /var/lib/k8s/log/xxx
          type: File
        name: log
    volumeMounts:
    - mountPath: /var/log/xxx
      name: log
    

    and when i go to the container's hypervisor, going to the path /var/lib/k8s/log/xxx, link file runtime cannot open(error: No such file or directory.) since the link file linked /var/log/xxx/runtime.%Y%m%d which is in the container path not the hypervisor path.

    So we need make the link file link to relative path. And I see the k8s log link file also link to relative path.

    kube-apiserver.INFO -> kube-apiserver.MASTER1.root.log.INFO.20200814-101251.2053
    
    
  • Support rotationCount option

    Support rotationCount option

    It is useful if count of rotation option is also supported. To keep compatibility, maxAge is still required option. rotationCount overrides maxAge option if it is specified.

  • Fix handling of MaxAge/RotationCount

    Fix handling of MaxAge/RotationCount

    fixes #16

    • New method, "Rotate" is introduced.
    • MaxAge default is now "time.Duration(0)", which is the same as disabled
    • RotationCount default is now "0", which is the same as disabled
    • RotationCount is now uint, instead of int
    • options are now name/value pairs, instead of doing silly method calls to Configure
  • Create directory if it does not already exist

    Create directory if it does not already exist

    Case:

            logf, err := rotatelogs.New(
                    "./foo/bar.log.%Y%m%d%H",
                    rotatelogs.WithLinkName("./foo/bar.log"),
                    rotatelogs.WithMaxAge(24*time.Hour),
                    rotatelogs.WithRotationTime(time.Hour),
            )
    

    and when the ./foo directory isn't existed, error occur!

    open ./foo/bar.log.2018101616: no such file or directory
    

    We must ensure that the dir is existed by doing so can we use os.OpenFile to create or append the filename

  • Increase the presence of a detection file when writing a log

    Increase the presence of a detection file when writing a log

    After the program is started, the log files are deleted manually, but no files are written. So, Increase the presence of a detection file when writing a log

  • After a forced generational rotate, ensure new writes go to the new file.

    After a forced generational rotate, ensure new writes go to the new file.

    Keep track of the base filename for log files, so when a forced generational rotate is performed, the existing log file becomes a new version and subsequent writes go to the new file.

  • fix: set default maxAge by 7 days

    fix: set default maxAge by 7 days

    In the current version, default maxAge as 0 will failed when rotate:

    Soure:

    rl, _ := rotatelogs.NewRotateLogs("/path/to/access_log.%Y%m%d%H%M")
    

    Output:

    failed to rotate: maxAge not set, not rotating
    

    Breaking line:

    https://github.com/lestrrat/go-file-rotatelogs/blob/master/rotatelogs.go#L237

    Thie PR will fix this by add a default value in New method.

  • tebeka/strftime from Github instead of Bitbucket

    tebeka/strftime from Github instead of Bitbucket

    @lestrrat

    I find that Miki Tebeka had already imported his projects into Github

    and the strftime for golang is here: https://github.com/tebeka/strftime

    it would be easier for people to install rotatelogs without install the HG

  • README: Fix the code example to use the v2 API.

    README: Fix the code example to use the v2 API.

    What is this patch?

    For now, the code example in README.md does not work because it relies on the old API (which has been deprecated since v2.0.0).

    This small patch fixes this issue by migrating the code to use the newer interface. With this patch applied, the code example will work fine.

  • Use lestrrat/go-strftime

    Use lestrrat/go-strftime

    BACKWARDS INCOMPATIBLE CHANGE! New() now returns an object AND an error.

    Otherwise, the file generation is 3~4x faster, and you no longer have to install hg to install this library. Your welcome :D

Port information web scraper written in Go.

Whatport is an open source tool that scrapes port information from SpeedGuide's Port Database Usage whatport [port(s)] (Seperate ports with a space)

Aug 18, 2022
Time based rotating file writer

cronowriter This is a simple file writer that it writes message to the specified format path. The file path is constructed based on current date and t

Dec 29, 2022
A simple logging module for go, with a rotating file feature and console logging.

A simple logging module for go, with a rotating file feature and console logging. Installation go get github.com/jbrodriguez/mlog Usage Sample usage W

Dec 14, 2022
Rotating File Writer

Rotating File writer An io.writer & io.Closer compliant file writer that will always write to the path that you give it, even if somebody deletes/rena

Jul 12, 2021
A hack example to servce file as a webserver with a Google Cloud Functions

Overview This project is a hack example to servce file as a webserver with a Google Cloud Functions (normally single purpose, only one entry point). I

Nov 26, 2021
Goimportcycle - a tool to visualize Go imports resolved to the file level
Goimportcycle - a tool to visualize Go imports resolved to the file level

Go Import Cycle goimportcycle is a tool to visualize Go imports resolved to the

Dec 8, 2022
Filez - A tiny package showing you File info

filez A tiny package showing you File info Install go get -v github.com/Cne3Rd/f

Feb 4, 2022
The full power of the Go Compiler directly in your browser, including a virtual file system implementation. Deployable as a static website.
The full power of the Go Compiler directly in your browser, including a virtual file system implementation. Deployable as a static website.

Static Go Playground Features Full Go Compiler running on the browser. Supports using custom build tags. Incremental builds (build cache). Supports mu

Jun 16, 2022
This is a simple file storage server. User can upload file, delete file and list file on the server.
This is a simple file storage server.  User can upload file,  delete file and list file on the server.

Simple File Storage Server This is a simple file storage server. User can upload file, delete file and list file on the server. If you want to build a

Jan 19, 2022
A little websocket TCP proxy to let browsers talk to a fixed port on arbitrary hosts. Built for Gemini (gemini://, port 1965)

Kepler A little websocket TCP proxy built to let Amfora talk to Gemini servers when running in a browser. Usage $ git clone https://github.com/awfulco

May 27, 2022
Port-proxy - Temporary expose port for remote connections

Port proxy util Temporary expose port for remote connections. E.g. database/wind

Jan 27, 2022
A feature flag solution, with only a YAML file in the backend (S3, GitHub, HTTP, local file ...), no server to install, just add a file in a central system and refer to it. 🎛️
A feature flag solution, with only a YAML file in the backend (S3, GitHub, HTTP, local file ...), no server to install, just add a file in a central system and refer to it. 🎛️

??️ go-feature-flag A feature flag solution, with YAML file in the backend (S3, GitHub, HTTP, local file ...). No server to install, just add a file i

Dec 29, 2022
Lima launches Linux virtual machines on macOS, with automatic file sharing, port forwarding, and containerd.

Lima: Linux-on-Mac ("macOS subsystem for Linux", "containerd for Mac")

Jan 8, 2023
Convert scanned image PDF file to text annotated PDF file
Convert scanned image PDF file to text annotated PDF file

Jisui (自炊) This tool is PoC (Proof of Concept). Jisui is a helper tool to create e-book. Ordinary the scanned book have not text information, so you c

Dec 11, 2022
go-fastdfs 是一个简单的分布式文件系统(私有云存储),具有无中心、高性能,高可靠,免维护等优点,支持断点续传,分块上传,小文件合并,自动同步,自动修复。Go-fastdfs is a simple distributed file system (private cloud storage), with no center, high performance, high reliability, maintenance free and other advantages, support breakpoint continuation, block upload, small file merge, automatic synchronization, automatic repair.(similar fastdfs).
go-fastdfs 是一个简单的分布式文件系统(私有云存储),具有无中心、高性能,高可靠,免维护等优点,支持断点续传,分块上传,小文件合并,自动同步,自动修复。Go-fastdfs is a simple distributed file system (private cloud storage), with no center, high performance, high reliability, maintenance free and other advantages, support breakpoint continuation, block upload, small file merge, automatic synchronization, automatic repair.(similar fastdfs).

中文 English 愿景:为用户提供最简单、可靠、高效的分布式文件系统。 go-fastdfs是一个基于http协议的分布式文件系统,它基于大道至简的设计理念,一切从简设计,使得它的运维及扩展变得更加简单,它具有高性能、高可靠、无中心、免维护等优点。 大家担心的是这么简单的文件系统,靠不靠谱,可不

Jan 8, 2023
Parse awesome-go README file and generate a new README file with repo info.

Awesome Go Extra All data are from awesome-go and GitHub API. Audio and Music Libraries for manipulating audio. Name Description Star Open Issues Crea

Aug 11, 2022
Simple Golang API to demonstrate file upload to fireabase storage and retrieving url of uploaded file.

go-firebase-storage -Work in progress ??️ Simple Golang API that uses Firebase as its backend to demonstrate various firebase services using Go such a

Oct 4, 2021
This is a Go Cli app that receives an string path to a log file, and based on it generates and prints in console an encoded polyline with the locations found in the log file.
This is a Go Cli app that receives an string path to a log file, and based on it generates  and prints in console an encoded polyline with the locations found in the log file.

GEOENCODE GO CLI APP DESCRIPTION This is a Go Cli app that receives an string path to a log file, and based on it generates and prints in console an e

Oct 1, 2021
This command line converts .webarchive file to resources embed .html file

webarchive-to-singlefile This command line converts Safari's .webarchive file to complete .html. Notice Only tested on MacOS. Google Chrome required.

Dec 30, 2022
Distributed File Store Application Consist of API Server to handle file operations and command line tool to do operations

Filestore Distributed File Store Application Consist of API Server to handle file operations and command line tool to do operations (store named binar

Nov 7, 2021