Setup:

• backend which uses go-lz4
• client which uses official LZ4 C libraries (no way to Go unfortunately)

The problem I have is that sometimes data from backend couldn't be decoded on client. The keyword is "sometimes". I've got one sample (55K compressed, 114K uncompressed).

I'm not sure yet which side contains bug but let me know if you want to check it out.

AFAICT, cp() needs to append byte by byte because initially there may not be enough in the dst to cover the length requested. But once there's enough in dst, you can just append a whole slice.

This is the size of the internal buffer, but there appear to be no checks to make sure we don't overrun it when we load data in in writer.go encoder.cache(). I haven't checked the reader, but it seems likely the same problem will occur in flush() if we try to write out a larger section that what we've read it.

The typical signature for NewWriter is func NewWriter(wr io.Writer) *Writer whereas the LZ4 code has NewWriter(r io.Reader) io.ReadCloser. This makes it unusable within normal Go programs

By preallocating our destination buffer, we can eliminate all the calls to append(). This plus some inlining two hot routines give a considerable speedup to Decode().

Below are benchmarks ported from snappy-go:

benchmark                  old ns/op    new ns/op    delta
BenchmarkLZ4Decode           4480128      3150442  -29.68%
BenchmarkWordsDecode1e3         6071         3506  -42.25%
BenchmarkWordsDecode1e4        69195        45798  -33.81%
BenchmarkWordsDecode1e5       744347       539174  -27.56%
BenchmarkWordsDecode1e6      6616125      4841891  -26.82%

benchmark                   old MB/s     new MB/s  speedup
BenchmarkWordsDecode1e3       164.71       285.18    1.73x
BenchmarkWordsDecode1e4       144.52       218.35    1.51x
BenchmarkWordsDecode1e5       134.35       185.47    1.38x
BenchmarkWordsDecode1e6       151.15       206.53    1.37x

These are some sanity tests to make sure we actually encode/decode things.

I'm using /usr/share/dict/words as a source of input. If you're prefer (because the line-endings suggest to me you're on windows) we can import a novel from Project Gutenberg and use that instead so the data is actually commited to the repo.

The snappy tests actually download their larger test data at benchmark time, if requested, so that's an option too. I wouldn't want to do that for the basic tests though.

This allows people to more easily use the library from within their code without having to pull it down to a different location and manipulate their GOPATH.

$ go tool pprof populate /tmp/profile764028996/mem.pprof
Entering interactive mode (type "help" for commands)
(pprof) list lz4.Encode
Total: 3.90GB
ROUTINE ======================== github.com/bkaradzic/go-lz4.Encode in /home/ubuntu/go/src/github.com/bkaradzic/go-lz4/writer.go
    3.70GB     3.70GB (flat, cum) 94.89% of Total
         .          .    107:   if len(src) >= MaxInputSize {
         .          .    108:           return nil, ErrTooLarge
         .          .    109:   }
         .          .    110:
         .          .    111:   if n := CompressBound(len(src)); len(dst) < n {
    8.52MB     8.52MB    112:           dst = make([]byte, n)
         .          .    113:   }
         .          .    114:
    3.69GB     3.69GB    115:   e := encoder{src: src, dst: dst, hashTable: make([]uint32, hashTableSize)}
         .          .    116:
         .          .    117:   binary.LittleEndian.PutUint32(dst, uint32(len(src)))
         .          .    118:   e.dpos = 4
         .          .    119:
         .          .    120:   var (

This line in the code is causing Badger to OOM when loading data really fast. Ideally, you want to reuse the same hashTable. Can be done via sync.Pool. Happy to send a PR if that'd help.

Also, modify the benchmarks to reuse the allocated slice. This improves the ns/op for both encoding and decoding by 0.4s/op (2.4s -> 2.0s for Decode, and similar for encode).

I'm using two github versions of lz4: yours for Golang https://github.com/jpountz/lz4-java for java.

I used your library in the code below:

package main

import (
	lz4 "github.com/bkaradzic/go-lz4"
	"fmt"
	"os"
	"encoding/base64"
)

func main() {
	var data []byte
	var err error

	to_compress:="xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"

	if data,err=lz4.Encode(nil,[]byte(to_compress)); err!=nil {
		fmt.Fprintf(os.Stderr,"Failed to compress: '%s'",err)
		return
	}

	fmt.Fprintf(os.Stderr,"Success! Length=%d\n",len(to_compress))

	fmt.Fprintf(os.Stdout,"%s\n",base64.StdEncoding.EncodeToString(data))
}

and obtain following output:

raffi@iot-micro-raffi ~/tmp > go run ./test_lz4.go 
Success! Length=100
ZAAAAB94AQBLUHh4eHh4
raffi@iot-micro-raffi ~/tmp > 

Then I inject this output into a Java version to decode the result:

import java.util.Base64;
import net.jpountz.lz4.LZ4Factory;
import net.jpountz.lz4.LZ4FastDecompressor;

public class test_lz4_decomp {

    public static void main(String[] args) {
        String compressed_base64= "ZAAAAB94AQBLUHh4eHh4";
        int original_size = 100;
        byte [] compressed = Base64.getDecoder().decode(compressed_base64);

        LZ4Factory _factory= LZ4Factory.fastestInstance();
        LZ4FastDecompressor decompressor = _factory.fastDecompressor();
        byte []restored = new byte[original_size];
        decompressor.decompress(compressed, 0, restored, 0,original_size);

        String decompressed_str=new String(restored);

        System.out.println(decompressed_str); 
    }

}

but obtain the following output:

raffi@iot-micro-raffi ~/tmp > java -cp ".:lz4-1.3.0.jar" test_lz4_decompException in thread "main" net.jpountz.lz4.LZ4Exception: Error decoding offset 58 of input buffer
	at net.jpountz.lz4.LZ4JNIFastDecompressor.decompress(LZ4JNIFastDecompressor.java:39)
	at test_lz4_decomp.main(test_lz4_decomp.java:16)
raffi@iot-micro-raffi ~/tmp > 

The expected result would be that the Java version should be able to decompress the output of the Golang version.

I'd like to see LZ4 framing supported.

The frame-descriptor includes an optional content-length, which may help to eliminate go-lz4's unique means of storing such:

go-lz4 saves a uint32 with the original uncompressed length at the beginning of the encoded buffer

I'd love to use this code, but my project involves compressing files up to gigabytes in size.

For that use case I really need a streaming version of the algorithm (i.e. using the io.Writer/io.Reader interfaces). I see there's one as part of the reference C implementation, but I really have no idea where to start with porting it to Go.