Native LZO implementation in Go

go-lzo

Build status Coverage Status

Native LZO1X implementation in Golang

This code has been written using the original LZO1X source code as a reference, to study and understand the algorithms. Both the LZO1X-1 and LZO1X-999 algorithms are implemented. These are the most popular of the whole LZO suite of algorithms.

Being a straightforward port of the original source code, it shares the same license (GPLv2) as I can't possibly claim any copyright on it.

I plan to eventually reimplement LZO1X-1 from scratch. At that point, I will be also changing license.

Benchmarks

These are the benchmarks obtained running the testsuite over the Canterbury corpus for the available compressor levels:

Compressor Level Original Compressed Factor Time Speed
LZO1X-1 - 18521760 8957481 51.6% 0.16s 109MiB/s
LZO1X-999 1 18521760 8217347 55.6% 1.38s 13MiB/s
LZO1X-999 2 18521760 7724879 58.3% 1.50s 12MiB/s
LZO1X-999 3 18521760 7384377 60.1% 1.68s 10MiB/s
LZO1X-999 4 18521760 7266674 60.8% 1.69s 10MiB/s
LZO1X-999 5 18521760 6979879 62.3% 2.75s 6.4MiB/s
LZO1X-999 6 18521760 6938593 62.5% 4.53s 3.9MiB/s
LZO1X-999 7 18521760 6905362 62.7% 6.94s 2.5MiB/s
LZO1X-999 8 18521760 6713477 63.8% 20.96s 863KiB/s
LZO1X-999 9 18521760 6712069 63.8% 22.82s 792KiB/s
Owner
Giovanni Bajo
Developer Relation @italia / @teamdigitale. CTO of @develersrl and @greenapes; Python, Go, C++. Interested in algorithms & security.
Giovanni Bajo
Comments
  • Update Decompress1X handling of runtime error for Go1.13.

    Update Decompress1X handling of runtime error for Go1.13.

    Upcoming Go 1.13 change https://golang.org/cl/161477 updates the runtime error strings for index out of range errors.

    This change simply makes the comparison a bit more loose to handle both pre-1.13 and 1.13.

  • Go1.13 golang.org/cl/161477 will break Decompress1X runtime error handling

    Go1.13 golang.org/cl/161477 will break Decompress1X runtime error handling

    Upcoming Go 1.13 change https://golang.org/cl/161477 updates the runtime error strings for index out of range errors, which breaks the assumption that Decompress1X func has on it at https://github.com/rasky/go-lzo/blob/master/decompress.go#L160.

    $ go version
    $ go version devel +431b5c69ca Tue Apr 16 21:54:01 2019 +0000 darwin/amd64
    $ go test
    --- FAIL: TestDecompInlen (0.00s)
    panic: runtime error: index out of range [0] with length 0 [recovered]
            panic: runtime error: index out of range [0] with length 0 [recovered]
            panic: runtime error: index out of range [0] with length 0
    
    goroutine 20 [running]:
    testing.tRunner.func1(0xc0000e8100)
            /Users/herbie/work/go113/go/src/testing/testing.go:830 +0x3a3
    panic(0x115b440, 0xc00010a020)
            /Users/herbie/work/go113/go/src/runtime/panic.go:619 +0x1b2
    github.com/rasky/go-lzo.Decompress1X.func1(0xc000106e70)
            /Users/herbie/work/google3-go/lzo/decompress.go:165 +0xfa
    panic(0x115b440, 0xc00010a020)
            /Users/herbie/work/go113/go/src/runtime/panic.go:619 +0x1b2
    github.com/rasky/go-lzo.(*reader).ReadU16(...)
            /Users/herbie/work/google3-go/lzo/decompress.go:94
    github.com/rasky/go-lzo.Decompress1X(0x1199c40, 0xc0000c2040, 0x31, 0x0, 0xc0000d6090, 0xf, 0x10, 0x0, 0x0)
            /Users/herbie/work/google3-go/lzo/decompress.go:234 +0x944
    github.com/rasky/go-lzo.TestDecompInlen(0xc0000e8100)
            /Users/herbie/work/google3-go/lzo/compress_test.go:126 +0x20c
    testing.tRunner(0xc0000e8100, 0x1177800)
            /Users/herbie/work/go113/go/src/testing/testing.go:865 +0xbf
    created by testing.(*T).Run
            /Users/herbie/work/go113/go/src/testing/testing.go:916 +0x350
    exit status 2
    FAIL    github.com/rasky/go-lzo 0.024s
    

    I'll send a PR to update the check.

  • Proposal to dual license the project

    Proposal to dual license the project

    Can this project be dual licensed? perhaps in Apache 2.0 or MIT such that Apache 2.0 projects can link against this library?

    Current LICENSE.gpl is too restrictive and incompatible with Apache 2.0 released projects.

  • Can't decompress LZO compressed in Python

    Can't decompress LZO compressed in Python

    Hi Giovanni,

    I have a Python application that compresses data using python-lzo and then decompresses when needed in another part of application.

    I'm currently porting decompressing part of the application to golang and for some reason when I'm trying to use your library it always throws LookBehindUnderrun error at me. Here is the code I'm using:

    import (
        "bytes"
        "fmt"
    
        "github.com/rasky/go-lzo"
    )
    
    outData, err = lzo.Decompress1X(bytes.NewReader(inData), 0, 0)
    if err != nil {
        fmt.Println(err)
    }
    

    Any ideas ?

    Thanks in advance.

  • Possible infinite loop with single byte input

    Possible infinite loop with single byte input

    Hello,

    I just ran go-fuzz on the decompressor and it found this possibly triggering an infinite loop:

    https://github.com/pmezard/go-lzo/blob/fuzz-it/crashers/5ba93c9db0cff93f52b521d7420e43f6eda2784f

    with the fuzzer here:

    https://github.com/pmezard/go-lzo/blob/fuzz-it/fuzz.go

    (I committed the corpus/ to the branch as well if you are interested).

    I do not know if you implemented it just as an experiment or if you care about such reports.

  • compression of files

    compression of files

    Hi, I am using go language. And i want to compress files which are retrieved by ls command. I want to use rasky/go-lzo package. please guide me how can i do this?

  • Add compress/decompress tests for lzop(1) compatibility

    Add compress/decompress tests for lzop(1) compatibility

    Make existing decompress tests a bit more idiomatic by having them check returned error values.

    Add two tests which compare uncompressed and compressed output from lzop(1).

    Currently, we can now uncompress data compressed with lzop -1.

    Make debug prints conditioned on a variable (Verbose), not commented out, and expose both Verbose and the print function (Debug) so users of this package can set them. Debug defaults to an empty function.

    For the loops where we do not want to continually call an empty function, Verbose is used to control whether we call Debug at all.

    Still kind of a WIP, comments appreciated. I actually have a need for this package in u-root.

    Signed-off-by: Ronald G Minnich [email protected]

Related tags
A simple zip compactor app written in golang to help you life. Usage with native GUI and CLI.
A simple zip compactor app written in golang to help you life. Usage with native GUI and CLI.

Usage Install go install github.com/gustavonobreza/zip-compactor Run in GUI (Can select many files) zip-compactor Run in CLI (Can select just one file

Nov 12, 2021
Parallel implementation of Gzip for modern multi-core machines written in Go

gzip Parallel implementation of gzip for modern multi-core machines written in Go Usage: gzip [OPTION]... [FILE] Compress or uncompress FILE (by defau

Nov 16, 2021
A simple zip implementation in Go

A simple zip implementation in Go

Dec 16, 2022
Go wrapper for LZO compression library

This is a cgo wrapper around the LZO real-time compression library. LZO is available at http://www.oberhumer.com/opensource/lzo/ lzo.go is the go pack

Mar 4, 2022
An open-source, distributed, cloud-native CD (Continuous Delivery) product designed for developersAn open-source, distributed, cloud-native CD (Continuous Delivery) product designed for developers
An open-source, distributed, cloud-native CD (Continuous Delivery) product designed for developersAn open-source, distributed, cloud-native CD (Continuous Delivery) product designed for developers

Developer-oriented Continuous Delivery Product ⁣ English | 简体中文 Table of Contents Zadig Table of Contents What is Zadig Quick start How to use? How to

Oct 19, 2021
Go language implementation of a blockchain based on the BDLS BFT protocol. The implementation was adapted from Ethereum and Sperax implementation

BDLS protocol based PoS Blockchain Most functionalities of this client is similar to the Ethereum golang implementation. If you do not find your quest

Oct 14, 2022
Go implementation of systemd Journal's native API for logging

journald Package journald offers Go implementation of systemd Journal's native API for logging. Key features are: based on a connection-less socket wo

Dec 23, 2022
A native Go clean room implementation of the Porter Stemming algorithm.

Go Porter Stemmer A native Go clean room implementation of the Porter Stemming Algorithm. This algorithm is of interest to people doing Machine Learni

Jan 3, 2023
Full-native go implementation of Telegram API
Full-native go implementation of Telegram API

MTProto Full-native implementation of MTProto protocol on Golang! english русский 简体中文 Features Full native implementation All code, from sending requ

Jan 1, 2023
Constant Database native golang implementation

CDB golang implementation cdb is a fast, reliable, simple package for creating and reading constant databases see docs for more details Advantages Ite

Jul 15, 2022
go implementation of fissions web-native file system

wnfs-go go language implementation of the fission web-native file system, using the typescript implementation as a reference. Development Status: Work

Oct 15, 2022
Package rsync contains a native Go rsync implementation.

gokrazy rsync Package rsync contains a native Go rsync implementation. ⚠ Beware: very fresh. Might eat your data. You have been warned! ⚠ The only com

Jan 2, 2023
CVE-2021-4034 - A Golang implementation of clubby789's implementation of CVE-2021-4034

CVE-2021-4034 January 25, 2022 | An00bRektn This is a golang implementation of C

Feb 3, 2022
Sequence-based Go-native audio mixer for music apps

Mix https://github.com/go-mix/mix Sequence-based Go-native audio mixer for music apps See demo/demo.go: package main import ( "fmt" "os" "time"

Dec 1, 2022
A "native" ogg vorbis decoder for Go (uses inline stb_vorbis)

vorbis This Go package provides a "native" ogg vorbis decoder, but still requires cgo, as it uses inline code from stb_vorbis. Someday, it won't. The

Oct 24, 2022
An opinionated configuration loading framework for Containerized and Cloud-Native applications.
An opinionated configuration loading framework for Containerized and Cloud-Native applications.

Opinionated configuration loading framework for Containerized and 12-Factor compliant applications. Read configurations from Environment Variables, an

Dec 16, 2022
Drone is a Container-Native, Continuous Delivery Platform
Drone is a Container-Native, Continuous Delivery Platform

Drone is a Continuous Delivery system built on container technology. Drone uses a simple YAML configuration file, a superset of docker-compose, to def

Dec 28, 2022
Go native library for fast point tracking and K-Nearest queries

Geo Index Geo Index library Overview Splits the earth surface in a grid. At each cell we can store data, such as list of points, count of points, etc.

Dec 3, 2022
CockroachDB - the open source, cloud-native distributed SQL database.
CockroachDB - the open source, cloud-native distributed SQL database.

CockroachDB is a cloud-native SQL database for building global, scalable cloud services that survive disasters. What is CockroachDB? Docs Quickstart C

Jan 2, 2023
Native GraphQL Database with graph backend
Native GraphQL Database with graph backend

The Only Native GraphQL Database With A Graph Backend. Dgraph is a horizontally scalable and distributed GraphQL database with a graph backend. It pro

Jan 4, 2023