LevelDB key/value database in Go.

Last update: Jan 1, 2023

Comments: 16

This is an implementation of the LevelDB key/value database in the Go programming language.

Installation

go get github.com/syndtr/goleveldb/leveldb

Requirements

Need at least go1.5 or newer.

Usage

Create or open a database:

// The returned DB instance is safe for concurrent use. Which mean that all
// DB's methods may be called concurrently from multiple goroutine.
db, err := leveldb.OpenFile("path/to/db", nil)
...
defer db.Close()
...

Read or modify the database content:

// Remember that the contents of the returned slice should not be modified.
data, err := db.Get([]byte("key"), nil)
...
err = db.Put([]byte("key"), []byte("value"), nil)
...
err = db.Delete([]byte("key"), nil)
...

Iterate over database content:

iter := db.NewIterator(nil, nil)
for iter.Next() {
	// Remember that the contents of the returned slice should not be modified, and
	// only valid until the next call to Next.
	key := iter.Key()
	value := iter.Value()
	...
}
iter.Release()
err = iter.Error()
...

Seek-then-Iterate:

iter := db.NewIterator(nil, nil)
for ok := iter.Seek(key); ok; ok = iter.Next() {
	// Use key/value.
	...
}
iter.Release()
err = iter.Error()
...

Iterate over subset of database content:

iter := db.NewIterator(&util.Range{Start: []byte("foo"), Limit: []byte("xoo")}, nil)
for iter.Next() {
	// Use key/value.
	...
}
iter.Release()
err = iter.Error()
...

Iterate over subset of database content with a particular prefix:

iter := db.NewIterator(util.BytesPrefix([]byte("foo-")), nil)
for iter.Next() {
	// Use key/value.
	...
}
iter.Release()
err = iter.Error()
...

Batch writes:

batch := new(leveldb.Batch)
batch.Put([]byte("foo"), []byte("value"))
batch.Put([]byte("bar"), []byte("another value"))
batch.Delete([]byte("baz"))
err = db.Write(batch, nil)
...

Use bloom filter:

o := &opt.Options{
	Filter: filter.NewBloomFilter(10),
}
db, err := leveldb.OpenFile("path/to/db", o)
...
defer db.Close()
...

Documentation

You can read package documentation here.

Owner

Suryandaru Triandana

https://github.com/syndtr/goleveldb

Comments

Require explicit general database closing.

To be idiomatic and play nicely as a library, control of the database should exist solely in the hands of the library user and not in that of the memory manager.

Major Caveat: A number of internal types around iterators and versions retain their finalizers for now. I need to take some time to diagram their lifecycle to propose a solution for this. This domain is where, I believe, the nondeterminism I found was, but I want to start fixing the easy part first.

@syndtr, if you have some recommendations for how to approach this, I would love to talk with you. If you look at other LevelDB implementations, they explicitly require the user to call Iterator.Close to free associated resources, including the an official but incomplete implementation from Google in Go at https://code.google.com/p/leveldb-go/source/browse/leveldb/db/db.go#72.
Slice bounds out of range error during compaction

Here's the stackdump:

panic: runtime error: slice bounds out of range [recovered] panic: runtime error: slice bounds out of range

goroutine 50 [running]: github.com/syndtr/goleveldb/leveldb.func·015() /Users/sascha/Development/Go/projects/src/github.com/syndtr/goleveldb/leveldb/db_compaction.go:392 +0x172 github.com/syndtr/goleveldb/leveldb.iKey.ukey(0xc2006180df, 0x0, 0xa7, 0x0, 0xa7, ...) /Users/sascha/Development/Go/projects/src/github.com/syndtr/goleveldb/leveldb/key.go:73 +0x75 github.com/syndtr/goleveldb/leveldb.(_iComparer).Compare(0xc20016a660, 0xc2006180df, 0x0, 0xa7, 0xc201207720, ...) /Users/sascha/Development/Go/projects/src/github.com/syndtr/goleveldb/leveldb/comparer.go:21 +0x85 github.com/syndtr/goleveldb/leveldb/iterator.(_MergedIterator).smallest(0xc200afca50) /Users/sascha/Development/Go/projects/src/github.com/syndtr/goleveldb/leveldb/iterator/merged_iter.go:178 +0x158 github.com/syndtr/goleveldb/leveldb/iterator.(_MergedIterator).Next(0xc200afca50, 0xc20061802d) /Users/sascha/Development/Go/projects/src/github.com/syndtr/goleveldb/leveldb/iterator/merged_iter.go:113 +0x185 github.com/syndtr/goleveldb/leveldb.func·013(0x0, 0x0) /Users/sascha/Development/Go/projects/src/github.com/syndtr/goleveldb/leveldb/db_compaction.go:247 +0x1cc github.com/syndtr/goleveldb/leveldb.(_DB).transact(0xc20012d180, 0x6c7a50) /Users/sascha/Development/Go/projects/src/github.com/syndtr/goleveldb/leveldb/db_compaction.go:136 +0xaa github.com/syndtr/goleveldb/leveldb.(_DB).doCompaction(0xc20012d180, 0xc200172540, 0x6c7f00) /Users/sascha/Development/Go/projects/src/github.com/syndtr/goleveldb/leveldb/db_compaction.go:365 +0x7e0 github.com/syndtr/goleveldb/leveldb.(_DB).compaction(0xc20012d180) /Users/sascha/Development/Go/projects/src/github.com/syndtr/goleveldb/leveldb/db_compaction.go:463 +0x1cb created by github.com/syndtr/goleveldb/leveldb.open /Users/sascha/Development/Go/projects/src/github.com/syndtr/goleveldb/leveldb/db.go:70 +0x1ec
Possible memory leak

I'm using goleveldb to keep an index from certain kinds metadata to entries in my logfiles. I have four leveldbs for each day which are rotated at midnight. After the process is running several days the memory consumption keeps growing and growing. I'm using a write buffer of 16 Mbytes.

Here's the output from lsof:

process 30008 user 68uW REG 253,2 0 15310869 ./20140527-indexdb.ldb/servicenames/LOCK process 30008 user 70w REG 253,2 165 15310870 ./20140527-indexdb.ldb/servicenames/LOG process 30008 user 74w REG 253,2 54 15310871 ./20140527-indexdb.ldb/servicenames/MANIFEST-000000 process 30008 user 82w REG 253,2 8740127 15310873 ./20140527-indexdb.ldb/servicenames/000001.log process 30008 user 129uW REG 253,2 0 15310875 ./20140527-indexdb.ldb/servicemetadata/LOCK process 30008 user 130w REG 253,2 165 15310876 ./20140527-indexdb.ldb/servicemetadata/LOG process 30008 user 131w REG 253,2 54 15310877 ./20140527-indexdb.ldb/servicemetadata/MANIFEST-000000 process 30008 user 134w REG 253,2 361647 15310879 ./20140527-indexdb.ldb/servicemetadata/000001.log process 30008 user 146uW REG 253,2 0 15310851 ./20140527-indexdb.ldb/base/LOCK process 30008 user 153w REG 253,2 24820721 15310912 ./20140527-indexdb.ldb/transactions/000002.log process 30008 user 155w REG 253,2 165 15310852 ./20140527-indexdb.ldb/base/LOG process 30008 user 159w REG 253,2 54 15310853 ./20140527-indexdb.ldb/base/MANIFEST-000000 process 30008 user 161w REG 253,2 59070 15310855 ./20140527-indexdb.ldb/base/000001.log process 30008 user 164uW REG 253,2 0 15310857 ./20140527-indexdb.ldb/transactions/LOCK process 30008 user 171w REG 253,2 365 15310858 ./20140527-indexdb.ldb/transactions/LOG process 30008 user 175w REG 253,2 128 15310859 ./20140527-indexdb.ldb/transactions/MANIFEST-000000 process 30008 user 192uW REG 253,2 0 15310863 ./20140527-indexdb.ldb/message/LOCK process 30008 user 193w REG 253,2 166 15310864 ./20140527-indexdb.ldb/message/LOG process 30008 user 194w REG 253,2 54 15310865 ./20140527-indexdb.ldb/message/MANIFEST-000000 process 30008 user 196w REG 253,2 26185266 15310867 ./20140527-indexdb.ldb/message/000001.log

Here's the output from pprof:
Effort to reduce heap allocations

Hi!

I use goleveldb in syncthing (https://github.com/syncthing/syncthing) and have been doing a fair bit of profiling and hacking lately to reduce syncthing's memory footprint. I'm now at the point where goleveldb represents ~70-80 MB of a 100 MB footprint in my benchmarks. To a certain extent this is fair, with caches etc. But a large part of it is temporary heap allocations, in fact pretty much all of them made by snappy.Decode() from readRawBlock(). I made a naive attempt to reuse buffers with a sync.Pool in https://github.com/calmh/goleveldb/tree/reduce-heap, and it makes a dent but it's not huge. Probably the reason it's not very effective is that the pool isn't filled very often, and I'm not sure about the code paths that result in the data buffers becoming unused to hook into.

This isn't a bug report as such, more a humble request to give it some work at some point, or pass me some hints and I'm happy to root around in it. :)
Compaction time can be quite big

Hey! I am investigation the performance decline in our software https://github.com/tendermint/tendermint/issues/1835. We're using goleveldb as a primary DB for storing blocks plus some other data.

Looks like saving a batch of data (~1MB) can result in ~ 15 sec. due to compaction process or something:

https://github.com/tendermint/tendermint/issues/1835#issuecomment-402054099 (the time seems to be increasing with every spike)

The write speed is approximately 1MB per second.

Is this expected and we need to fine tune compaction options or this looks like a bug to you?

Thank you!
Possible corruption on the block data

This could be caused by bug in the table writer or block buffer.

Corruption on the block buffer could be caused by same buffer acquired more than once from buffer pool by two different block instance (or probably added to buffer pool more than once?).

https://gist.github.com/syndtr/a97bef48b12df8181d6b

Recovering the DB doesn't indicate any corruption but the corrupted table may already been deleted.

Could be related to syncthing/syncthing#987.
db corruption on 'out of disk space'

Using btcd, which currently uses goleveldb on commit 92de35cffa094e330ee854d44ccbcb954e344d08, if disk space runs out (write /tmp/mainnet/blocks_leveldb/000054.log: no space left on device), btcd can be restarted without problem.

After updating goleveldb to commit 62cc4d4e8b1ba35c2d1597e54892c022c376995f, goleveldb corrupts the database on the same error. When btcd starts back up, it says leveldb: bad record in batch

Significant memory usage when trying to insert data

I am trying to profile the memory usage of my go program, and found that the majority of memory seems to be consumed within leveldb functions (github.com/syndtr/goleveldb/leveldb/memdb.(*DB).Put).

Below is the top profile output:

(pprof) top
6815712.39kB of 6881450.49kB total (99.04%)
Dropped 372 nodes (cum <= 34407.25kB)
Showing top 10 nodes out of 28 (cum >= 242528.01kB)
      flat  flat%   sum%        cum   cum%
 6471904kB 94.05% 94.05%  6471904kB 94.05%  github.com/syndtr/goleveldb/leveldb/memdb.(*DB).Put
242528.01kB  3.52% 97.57% 242528.01kB  3.52%  bytes.makeSlice
   55040kB   0.8% 98.37%    55040kB   0.8%  lib.FP.GenerateHashMap
46004.76kB  0.67% 99.04% 54040.69kB  0.79%  main.main
  149.16kB 0.0022% 99.04% 6483852.34kB 94.22%  github.com/syndtr/goleveldb/leveldb.(*DB).Write
   68.67kB 0.001% 99.04% 319400.85kB  4.64%  main.Worker
   13.77kB 0.0002% 99.04% 6507694.47kB 94.57%  main.Worker1
    4.03kB 5.9e-05% 99.04% 6495736.37kB 94.39%  github.com/syndtr/goleveldb/leveldb.(*DB).Put
         0     0% 99.04%   196616kB  2.86%  bytes.(*Buffer).WriteString
         0     0% 99.04% 242528.01kB  3.52%  bytes.(*Buffer).grow

disable auto compaction

Is there a way to disable the auto db compaction goleveldb does? My db keeps being corrupted, and if it does panic, it is always in db_compaction.go

I want to test filling my DB without goleveldb running compaction at all to see if it still corrupts.

Compression change behind the scenes

btcd (github.com/conformal/btcd) uses goleveldb to store data.

The database is opened with:

        myCache := cache.NewEmptyCache()
        opts := &opt.Options{Flag: flag,
                BlockCache:      myCache,
                MaxOpenFiles:    256,
                CompressionType: opt.NoCompression,
        }

After a while, the database throws an error about a missing data.

I added two fmt.Printfs to reader.go in readRawBlock()'s select case to print the type of Compression being used. I also added spew.Dump(bh) We have a tool to dump data.

(table.blockHandle) {
 offset: (uint64) 1937402,
 length: (uint64) 4827
}
(uint8) 0
No Compression

On a different block:

(table.blockHandle) {
 offset: (uint64) 0,
 length: (uint64) 22302
}
(uint8) 1
Snappy Compression
1381438195854721313 [Warn] unable to locate block sha 00000000000033640c6b34c40b8b95ba4fae759ec7970e546660a3abd9f9314c err snappy: unsupported COPY_4 tag
1381438195854866024 [Info] Invalid block 00000000000033640c6b34c40b8b95ba4fae759ec7970e546660a3abd9f9314c


And another:
(table.blockHandle) {
 offset: (uint64) 1237318,
 length: (uint64) 3572
}
(uint8) 1
Snappy Compression

Yet another:
(table.blockHandle) {
 offset: (uint64) 1119730,
 length: (uint64) 4331
}
(uint8) 0
No Compression

Any idea what is causing the compression to be changed?  Multiple people are seeing this on different machines.  I am the only one getting the COPY_4 tag issue.

go test fail on Windows/386

Hello,

I saw you fixed #14 so I wanted to test your package and here what I got:

go version
go version devel +d29da2ced72b Mon May 13 10:28:12 2013 +0400 windows/386

Windows 7 x32

Error:

D:\Golibs\os\src\github.com\syndtr\goleveldb\leveldb>go test
fatal error: runtime: cannot map pages in arena address space

goroutine 328 [running]:
[fp=0x16c787c] runtime.throw(0x665341)
        c:/Go/src/pkg/runtime/panic.c:473 +0x65
[fp=0x16c789c] runtime.SysMap(0x3ffa0000, 0x100000)
        c:/Go/src/pkg/runtime/mem_windows.c:70 +0x78
[fp=0x16c78bc] runtime.MHeap_SysAlloc(0x1730000, 0x100000)
        c:/Go/src/pkg/runtime/zmalloc_windows_386.c:360 +0x103
[fp=0x16c78e4] MHeap_Grow(0x1730000, 0x10)
        c:/Go/src/pkg/runtime/mheap.c:224 +0x4f
[fp=0x16c7904] MHeap_AllocLocked(0x1730000, 0x4, 0x2b)
        c:/Go/src/pkg/runtime/mheap.c:108 +0x286
[fp=0x16c7918] runtime.MHeap_Alloc(0x1730000, 0x4, 0x2b, 0x0, 0x1, ...)
        c:/Go/src/pkg/runtime/mheap.c:77 +0x5c
[fp=0x16c7950] MCentral_Grow(0x1b35d4c)
        c:/Go/src/pkg/runtime/mcentral.c:203 +0x81
[fp=0x16c7974] runtime.MCentral_AllocList(0x1b35d4c, 0xc, 0x16c7988)
        c:/Go/src/pkg/runtime/mcentral.c:47 +0x4c
[fp=0x16c7994] runtime.MCache_Alloc(0x15d0000, 0x2b, 0x1400, 0x1)
        c:/Go/src/pkg/runtime/mcache.c:24 +0x72
[fp=0x16c79c4] runtime.mallocgc(0x1400, 0x1, 0x1, 0x1)
        c:/Go/src/pkg/runtime/zmalloc_windows_386.c:47 +0xdf
[fp=0x16c79e4] makeslice1(0x5021e0, 0x13e0, 0x13e0, 0x16c7a18)
        c:/Go/src/pkg/runtime/slice.c:61 +0x7f
[fp=0x16c7a04] runtime.makeslice(0x5021e0, 0x13e0, 0x0, 0x13e0, 0x0, ...)
        c:/Go/src/pkg/runtime/slice.c:34 +0x9c
[fp=0x16c7a58] github.com/syndtr/goleveldb/leveldb/table.(*bInfo).readAll(0x3f9feb80, 0x201818c0, 0x3f9b4a98, 0x20181800, 0x0, ...)
        D:/Golibs/os/src/github.com/syndtr/goleveldb/leveldb/table/block.go:66 +0x74
[fp=0x16c7a90] github.com/syndtr/goleveldb/leveldb/table.(*Reader).getBlock(0x3f9bd0f0, 0x3f9feb80, 0x20181440, 0x3f9b4988, 0x0, ...)
        D:/Golibs/os/src/github.com/syndtr/goleveldb/leveldb/table/reader.go:171 +0x8e
[fp=0x16c7ac8] github.com/syndtr/goleveldb/leveldb/table.(*Reader).getDataIter(0x3f9bd0f0, 0x3f9feb80, 0x20181440, 0x3f9b4988, 0x0, ...)
        D:/Golibs/os/src/github.com/syndtr/goleveldb/leveldb/table/reader.go:202 +0x174
[fp=0x16c7afc] github.com/syndtr/goleveldb/leveldb/table.(*indexIter).Get(0x3f9b19c0, 0x0, 0x0, 0x0, 0x0, ...)
        D:/Golibs/os/src/github.com/syndtr/goleveldb/leveldb/table/reader.go:234 +0xd0
[fp=0x16c7b14] github.com/syndtr/goleveldb/leveldb/iterator.(*IndexedIterator).setData(0x3f9b19e0, 0x1)
        D:/Golibs/os/src/github.com/syndtr/goleveldb/leveldb/iterator/indexed_iter.go:138
+0x2d
[fp=0x16c7b20] github.com/syndtr/goleveldb/leveldb/iterator.(*IndexedIterator).Next(0x3f9b19e0, 0x3ff9d800)
        D:/Golibs/os/src/github.com/syndtr/goleveldb/leveldb/iterator/indexed_iter.go:85 +0x7f
[fp=0x16c7b2c] github.com/syndtr/goleveldb/leveldb/iterator.(*IndexedIterator).Next(0x3f9b17e0, 0x4ba7c0)
        D:/Golibs/os/src/github.com/syndtr/goleveldb/leveldb/iterator/indexed_iter.go:84 +0x45
[fp=0x16c7b70] github.com/syndtr/goleveldb/leveldb/iterator.(*MergedIterator).Next(0x3ea83ea0, 0x3fdb03a0)
        D:/Golibs/os/src/github.com/syndtr/goleveldb/leveldb/iterator/merged_iter.go:112 +0x124
[fp=0x16c7c1c] github.com/syndtr/goleveldb/leveldb.func┬Ě015(0x0, 0x0)
        D:/Golibs/os/src/github.com/syndtr/goleveldb/leveldb/db_compaction.go:247 +0x172
[fp=0x16c7c6c] github.com/syndtr/goleveldb/leveldb.(*DB).transact(0x200bd500, 0x16c7cac)
        D:/Golibs/os/src/github.com/syndtr/goleveldb/leveldb/db_compaction.go:136 +0x74
[fp=0x16c7f2c] github.com/syndtr/goleveldb/leveldb.(*DB).doCompaction(0x200bd500, 0x3e997480, 0x16c7f00)
        D:/Golibs/os/src/github.com/syndtr/goleveldb/leveldb/db_compaction.go:365 +0x75d
[fp=0x16c7fd4] github.com/syndtr/goleveldb/leveldb.(*DB).compaction(0x200bd500)
        D:/Golibs/os/src/github.com/syndtr/goleveldb/leveldb/db_compaction.go:463 +0x17b
[fp=0x16c7fd8] runtime.goexit()
        c:/Go/src/pkg/runtime/proc.c:1223
created by github.com/syndtr/goleveldb/leveldb.openDB
        D:/Golibs/os/src/github.com/syndtr/goleveldb/leveldb/db.go:72 +0x1e6

goroutine 1 [chan receive]:
testing.RunTests(0x58aa7c, 0x666dc0, 0x3c, 0x3c, 0x1, ...)
        c:/Go/src/pkg/testing/testing.go:434 +0x69e
testing.Main(0x58aa7c, 0x666dc0, 0x3c, 0x3c, 0x665a60, ...)
        c:/Go/src/pkg/testing/testing.go:365 +0x68
main.main()
        C:/Users/Dobek/AppData/Local/Temp/go-build124087311/github.com/syndtr/goleveldb/leveldb/_test/_testmain.go:199 +0x80

goroutine 327 [chan send]:
github.com/syndtr/goleveldb/leveldb.(*DB).flush(0x200bd500, 0x0, 0x0, 0x0)
        D:/Golibs/os/src/github.com/syndtr/goleveldb/leveldb/db_write.go:62 +0x176
github.com/syndtr/goleveldb/leveldb.(*DB).Write(0x200bd500, 0x3ea85f20, 0x204cd010, 0x0, 0x0, ...)
        D:/Golibs/os/src/github.com/syndtr/goleveldb/leveldb/db_write.go:109 +0x181
github.com/syndtr/goleveldb/leveldb.(*DB).Put(0x200bd500, 0x3ea6ecb0, 0xb, 0xb, 0x3eaa6400, ...)
        D:/Golibs/os/src/github.com/syndtr/goleveldb/leveldb/db_write.go:162 +0x84
github.com/syndtr/goleveldb/leveldb.(*dbHarness).put(0x204f5480, 0x3ea6eca0, 0xb, 0x21ba8800, 0x3e8, ...)
        D:/Golibs/os/src/github.com/syndtr/goleveldb/leveldb/db_test.go:122 +0xb5
github.com/syndtr/goleveldb/leveldb.TestDb_SparseMerge(0x2012b540)
        D:/Golibs/os/src/github.com/syndtr/goleveldb/leveldb/db_test.go:928 +0x165
testing.tRunner(0x2012b540, 0x666f4c)
        c:/Go/src/pkg/testing/testing.go:353 +0x86
created by testing.RunTests
        c:/Go/src/pkg/testing/testing.go:433 +0x683

goroutine 329 [chan receive]:
github.com/syndtr/goleveldb/leveldb.(*DB).writeJournal(0x200bd500)
        D:/Golibs/os/src/github.com/syndtr/goleveldb/leveldb/db_write.go:25 +0x3b
created by github.com/syndtr/goleveldb/leveldb.openDB
        D:/Golibs/os/src/github.com/syndtr/goleveldb/leveldb/db.go:73 +0x1fb

goroutine 7 [runnable]:
exit status 2
FAIL    github.com/syndtr/goleveldb/leveldb     15.997s

Best regards, Dobrosław Żybort

migrate ci to github actions, test against go1.19
migrate away from travis CI in favor of github actions:

patch also does the following

explicitly test against go1.19

upgrade golangci-lint to v1.50

slice bounds out of range [:18014398510996847] with capacity 4194304

I got a weird panic, what's going on?

panic: runtime error: slice bounds out of range [:18014398510996847] with capacity 4194304 [recovered]
        panic: runtime error: slice bounds out of range [:18014398510996847] with capacity 4194304 [recovered]
        panic: runtime error: slice bounds out of range [:18014398510996847] with capacity 4194304
goroutine 74586376 [running]:
github.com/syndtr/goleveldb/leveldb.(*DB).mCompaction.func1()
        /root/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/db_compaction.go:752 +0xc5
panic({0xb58fc0, 0xc00e3dbda0})
        /usr/lib/go-1.18/src/runtime/panic.go:838 +0x207
github.com/syndtr/goleveldb/leveldb.(*DB).compactionTransact.func1()
        /root/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/db_compaction.go:158 +0x9b
panic({0xb58fc0, 0xc00e3dbda0})
        /usr/lib/go-1.18/src/runtime/panic.go:838 +0x207
github.com/syndtr/goleveldb/leveldb/memdb.(*dbIter).fill(0x1e?, 0x91?, 0xe2?)
        /root/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/memdb/memdb.go:42 +0x291
github.com/syndtr/goleveldb/leveldb/memdb.(*dbIter).Next(0xc0166770e0?)
        /root/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/memdb/memdb.go:131 +0x12e
github.com/syndtr/goleveldb/leveldb.(*tOps).createFrom(0xc0031825a0?, {0xcab698, 0xc00c8a6080})
        /root/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/table.go:329 +0x106
github.com/syndtr/goleveldb/leveldb.(*session).flushMemdb(0xc00807e4d0, 0xc01edb5900, 0xc012fb8000, 0x7faab3f5a498?)
        /root/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/session_compaction.go:27 +0x105
github.com/syndtr/goleveldb/leveldb.(*DB).memCompaction.func1(0xc016677001?)
        /root/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/db_compaction.go:304 +0x9c
github.com/syndtr/goleveldb/leveldb.(*compactionTransactFunc).run(0x3337333836300000?, 0x0?)
        /root/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/db_compaction.go:241 +0x1f
github.com/syndtr/goleveldb/leveldb.(*DB).compactionTransact(0xc00b0ec000, {0xb9f8b2, 0xb}, {0xca7fa0, 0xc00c896730})
        /root/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/db_compaction.go:185 +0x217
github.com/syndtr/goleveldb/leveldb.(*DB).compactionTransactFunc(...)
        /root/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/db_compaction.go:252
github.com/syndtr/goleveldb/leveldb.(*DB).memCompaction(0xc00b0ec000)
        /root/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/db_compaction.go:302 +0x3ee
github.com/syndtr/goleveldb/leveldb.(*DB).mCompaction(0xc00b0ec000)
        /root/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/db_compaction.go:766 +0x93
created by github.com/syndtr/goleveldb/leveldb.openDB
        /root/go/pkg/mod/github.com/syndtr/[email protected]/leveldb/db.go:150 +0x5d6

some node will pass to f() function many times, is this a bug or a feature?
https://github.com/syndtr/goleveldb/blob/126854af5e6d8295ef8e8bee3040dd8380ae72e8/leveldb/cache/cache.go#L327

var nodes []*Node for x := range h.buckets { b := h.initBucket(uint32(x)) b.mu.Lock() nodes = append(nodes, b.nodes...) b.mu.Unlock() f(nodes) }

data race

env GORACE=halt_on_error=1 go test -race ./...

==================
WARNING: DATA RACE
Read at 0x00c0001afd50 by goroutine 2253:
  runtime.chansend()
      /opt/hostedtoolcache/go/1.19.0/x64/src/runtime/chan.go:160 +0x0
  github.com/syndtr/goleveldb/leveldb.cAuto.ack()
      /home/runner/work/goleveldb/goleveldb/leveldb/db_compaction.go:695 +0x87
  github.com/syndtr/goleveldb/leveldb.(*DB).mCompaction()
      /home/runner/work/goleveldb/goleveldb/leveldb/db_compaction.go:787 +0xea
  github.com/syndtr/goleveldb/leveldb.openDB.func4()
      /home/runner/work/goleveldb/goleveldb/leveldb/db.go:155 +0x39
Previous write at 0x00c0001afd50 by goroutine 2248:
  runtime.closechan()
      /opt/hostedtoolcache/go/1.19.0/x64/src/runtime/chan.go:357 +0x0
  github.com/syndtr/goleveldb/leveldb.(*DB).compTriggerWait.func1()
      /home/runner/work/goleveldb/goleveldb/leveldb/db_compaction.go:725 +0x39
  runtime.deferreturn()
      /opt/hostedtoolcache/go/1.19.0/x64/src/runtime/panic.go:476 +0x32
  github.com/syndtr/goleveldb/leveldb.(*DB).rotateMem()
      /home/runner/work/goleveldb/goleveldb/leveldb/db_write.go:39 +0x75
  github.com/syndtr/goleveldb/leveldb.(*DB).flush.func1()
      /home/runner/work/goleveldb/goleveldb/leveldb/db_write.go:106 +0x46a
  github.com/syndtr/goleveldb/leveldb.(*DB).flush()
      /home/runner/work/goleveldb/goleveldb/leveldb/db_write.go:118 +0x2c1
  github.com/syndtr/goleveldb/leveldb.(*DB).writeLocked()
      /home/runner/work/goleveldb/goleveldb/leveldb/db_write.go:157 +0xa6
  github.com/syndtr/goleveldb/leveldb.(*DB).Write()
      /home/runner/work/goleveldb/goleveldb/leveldb/db_write.go:320 +0x92a
  github.com/syndtr/goleveldb/leveldb.TestDB_TransientError()
      /home/runner/work/goleveldb/goleveldb/leveldb/db_test.go:2389 +0xe04
  fmt.Sscanf()
      /opt/hostedtoolcache/go/1.19.0/x64/src/fmt/scan.go:114 +0x164
  github.com/syndtr/goleveldb/leveldb/storage.fsParseName()
      /home/runner/work/goleveldb/goleveldb/leveldb/storage/file_storage.go:651 +0x26
  github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).List()
      /home/runner/work/goleveldb/goleveldb/leveldb/storage/file_storage.go:465 +0x3e6
  fmt.Fscanf()
      /opt/hostedtoolcache/go/1.19.0/x64/src/fmt/scan.go:143 +0xe9
  fmt.Sscanf()
      /opt/hostedtoolcache/go/1.19.0/x64/src/fmt/scan.go:114 +0x164
  github.com/syndtr/goleveldb/leveldb/storage.fsParseName()
      /home/runner/work/goleveldb/goleveldb/leveldb/storage/file_storage.go:651 +0x26
  github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).List()
      /home/runner/work/goleveldb/goleveldb/leveldb/storage/file_storage.go:465 +0x3e6
  fmt.Fscanf()
      /opt/hostedtoolcache/go/1.19.0/x64/src/fmt/scan.go:143 +0xe9
  fmt.Sscanf()
      /opt/hostedtoolcache/go/1.19.0/x64/src/fmt/scan.go:114 +0x164
  github.com/syndtr/goleveldb/leveldb/storage.fsParseName()
      /home/runner/work/goleveldb/goleveldb/leveldb/storage/file_storage.go:651 +0x26
  github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).List()
      /home/runner/work/goleveldb/goleveldb/leveldb/storage/file_storage.go:465 +0x3e6
  github.com/syndtr/goleveldb/leveldb/testutil.(*Storage).List()
      /home/runner/work/goleveldb/goleveldb/leveldb/testutil/storage.go:349 +0x6d
  github.com/syndtr/goleveldb/leveldb.(*DB).checkAndCleanFiles()
      /home/runner/work/goleveldb/goleveldb/leveldb/db_util.go:52 +0x1fe
  github.com/syndtr/goleveldb/leveldb.openDB()
      /home/runner/work/goleveldb/goleveldb/leveldb/db.go:134 +0x7bd
  fmt.Fscanf()
      /opt/hostedtoolcache/go/1.19.0/x64/src/fmt/scan.go:143 +0xe9
  fmt.Sscanf()
      /opt/hostedtoolcache/go/1.19.0/x64/src/fmt/scan.go:114 +0x164
  github.com/syndtr/goleveldb/leveldb/storage.fsParseName()
      /home/runner/work/goleveldb/goleveldb/leveldb/storage/file_storage.go:651 +0x26
  github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).List()
      /home/runner/work/goleveldb/goleveldb/leveldb/storage/file_storage.go:465 +0x3e6
  fmt.Fscanf()
      /opt/hostedtoolcache/go/1.19.0/x64/src/fmt/scan.go:143 +0xe9
  fmt.Sscanf()
      /opt/hostedtoolcache/go/1.19.0/x64/src/fmt/scan.go:114 +0x164
  github.com/syndtr/goleveldb/leveldb/storage.fsParseName()
      /home/runner/work/goleveldb/goleveldb/leveldb/storage/file_storage.go:651 +0x26
  github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).List()
      /home/runner/work/goleveldb/goleveldb/leveldb/storage/file_storage.go:465 +0x3e6
  fmt.Fscanf()
      /opt/hostedtoolcache/go/1.19.0/x64/src/fmt/scan.go:143 +0xe9
  fmt.Sscanf()
      /opt/hostedtoolcache/go/1.19.0/x64/src/fmt/scan.go:114 +0x164
  github.com/syndtr/goleveldb/leveldb/storage.fsParseName()
      /home/runner/work/goleveldb/goleveldb/leveldb/storage/file_storage.go:651 +0x26
  github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).List()
      /home/runner/work/goleveldb/goleveldb/leveldb/storage/file_storage.go:465 +0x3e6
  fmt.Fscanf()
      /opt/hostedtoolcache/go/1.19.0/x64/src/fmt/scan.go:143 +0xe9
  fmt.Sscanf()
      /opt/hostedtoolcache/go/1.19.0/x64/src/fmt/scan.go:114 +0x164
  github.com/syndtr/goleveldb/leveldb/storage.fsParseName()
      /home/runner/work/goleveldb/goleveldb/leveldb/storage/file_storage.go:651 +0x26
  github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).List()
      /home/runner/work/goleveldb/goleveldb/leveldb/storage/file_storage.go:465 +0x3e6
  github.com/syndtr/goleveldb/leveldb/testutil.(*Storage).List()
      /home/runner/work/goleveldb/goleveldb/leveldb/testutil/storage.go:349 +0x6d
  github.com/syndtr/goleveldb/leveldb.(*DB).recoverJournal()
      /home/runner/work/goleveldb/goleveldb/leveldb/db.go:489 +0xb2
  github.com/syndtr/goleveldb/leveldb.openDB()
      /home/runner/work/goleveldb/goleveldb/leveldb/db.go:129 +0x7a7
  github.com/syndtr/goleveldb/leveldb.Open()
      /home/runner/work/goleveldb/goleveldb/leveldb/db.go:202 +0x35c
  fmt.Sscanf()
      /opt/hostedtoolcache/go/1.19.0/x64/src/fmt/scan.go:114 +0x164
  github.com/syndtr/goleveldb/leveldb/storage.fsParseName()
      /home/runner/work/goleveldb/goleveldb/leveldb/storage/file_storage.go:651 +0x26
  github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).List()
      /home/runner/work/goleveldb/goleveldb/leveldb/storage/file_storage.go:465 +0x3e6
  github.com/syndtr/goleveldb/leveldb/testutil.(*Storage).List()
      /home/runner/work/goleveldb/goleveldb/leveldb/testutil/storage.go:349 +0x6d
  github.com/syndtr/goleveldb/leveldb.(*session).recover.func1()
      /home/runner/work/goleveldb/goleveldb/leveldb/session.go:134 +0xd0
  runtime.deferreturn()
      /opt/hostedtoolcache/go/1.19.0/x64/src/runtime/panic.go:476 +0x32
  github.com/syndtr/goleveldb/leveldb.Open()
      /home/runner/work/goleveldb/goleveldb/leveldb/db.go:188 +0x104
  github.com/syndtr/goleveldb/leveldb.(*dbHarness).openDB0()
      /home/runner/work/goleveldb/goleveldb/leveldb/db_test.go:98 +0xb4
  github.com/syndtr/goleveldb/leveldb.(*dbHarness).init()
      /home/runner/work/goleveldb/goleveldb/leveldb/db_test.go:89 +0x271
  github.com/syndtr/goleveldb/leveldb.newDbHarnessWopt()
      /home/runner/work/goleveldb/goleveldb/leveldb/db_test.go:71 +0x167
  github.com/syndtr/goleveldb/leveldb.TestDB_TransientError()
      /home/runner/work/goleveldb/goleveldb/leveldb/db_test.go:2360 +0x3e
  testing.tRunner()
      /opt/hostedtoolcache/go/1.19.0/x64/src/testing/testing.go:1446 +0x216
  testing.(*T).Run.func1()
      /opt/hostedtoolcache/go/1.19.0/x64/src/testing/testing.go:1493 +0x47
Goroutine 2253 (running) created at:
  github.com/syndtr/goleveldb/leveldb.openDB()
      /home/runner/work/goleveldb/goleveldb/leveldb/db.go:155 +0xa39
  fmt.Fscanf()
      /opt/hostedtoolcache/go/1.19.0/x64/src/fmt/scan.go:143 +0xe9
  fmt.Sscanf()
      /opt/hostedtoolcache/go/1.19.0/x64/src/fmt/scan.go:114 +0x164
  github.com/syndtr/goleveldb/leveldb/storage.fsParseName()
      /home/runner/work/goleveldb/goleveldb/leveldb/storage/file_storage.go:651 +0x26
  github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).List()
      /home/runner/work/goleveldb/goleveldb/leveldb/storage/file_storage.go:465 +0x3e6
  fmt.Fscanf()
      /opt/hostedtoolcache/go/1.19.0/x64/src/fmt/scan.go:143 +0xe9
  fmt.Sscanf()
      /opt/hostedtoolcache/go/1.19.0/x64/src/fmt/scan.go:114 +0x164
  github.com/syndtr/goleveldb/leveldb/storage.fsParseName()
      /home/runner/work/goleveldb/goleveldb/leveldb/storage/file_storage.go:651 +0x26
  github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).List()
      /home/runner/work/goleveldb/goleveldb/leveldb/storage/file_storage.go:465 +0x3e6
  fmt.Fscanf()
      /opt/hostedtoolcache/go/1.19.0/x64/src/fmt/scan.go:143 +0xe9
  fmt.Sscanf()
      /opt/hostedtoolcache/go/1.19.0/x64/src/fmt/scan.go:114 +0x164
  github.com/syndtr/goleveldb/leveldb/storage.fsParseName()
      /home/runner/work/goleveldb/goleveldb/leveldb/storage/file_storage.go:651 +0x26
  github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).List()
      /home/runner/work/goleveldb/goleveldb/leveldb/storage/file_storage.go:465 +0x3e6
  fmt.Fscanf()
      /opt/hostedtoolcache/go/1.19.0/x64/src/fmt/scan.go:143 +0xe9
  fmt.Sscanf()
      /opt/hostedtoolcache/go/1.19.0/x64/src/fmt/scan.go:114 +0x164
  github.com/syndtr/goleveldb/leveldb/storage.fsParseName()
      /home/runner/work/goleveldb/goleveldb/leveldb/storage/file_storage.go:651 +0x26
  github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).List()
      /home/runner/work/goleveldb/goleveldb/leveldb/storage/file_storage.go:465 +0x3e6
  github.com/syndtr/goleveldb/leveldb/testutil.(*Storage).List()
      /home/runner/work/goleveldb/goleveldb/leveldb/testutil/storage.go:349 +0x6d
  github.com/syndtr/goleveldb/leveldb.(*DB).checkAndCleanFiles()
      /home/runner/work/goleveldb/goleveldb/leveldb/db_util.go:52 +0x1fe
  github.com/syndtr/goleveldb/leveldb.openDB()
      /home/runner/work/goleveldb/goleveldb/leveldb/db.go:134 +0x7bd
  fmt.Fscanf()
      /opt/hostedtoolcache/go/1.19.0/x64/src/fmt/scan.go:143 +0xe9
  fmt.Sscanf()
      /opt/hostedtoolcache/go/1.19.0/x64/src/fmt/scan.go:114 +0x164
  github.com/syndtr/goleveldb/leveldb/storage.fsParseName()
      /home/runner/work/goleveldb/goleveldb/leveldb/storage/file_storage.go:651 +0x26
  github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).List()
      /home/runner/work/goleveldb/goleveldb/leveldb/storage/file_storage.go:465 +0x3e6
  fmt.Fscanf()
      /opt/hostedtoolcache/go/1.19.0/x64/src/fmt/scan.go:143 +0xe9
  fmt.Sscanf()
      /opt/hostedtoolcache/go/1.19.0/x64/src/fmt/scan.go:114 +0x164
  github.com/syndtr/goleveldb/leveldb/storage.fsParseName()
      /home/runner/work/goleveldb/goleveldb/leveldb/storage/file_storage.go:651 +0x26
  github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).List()
      /home/runner/work/goleveldb/goleveldb/leveldb/storage/file_storage.go:465 +0x3e6
  fmt.Fscanf()
      /opt/hostedtoolcache/go/1.19.0/x64/src/fmt/scan.go:143 +0xe9
  fmt.Sscanf()
      /opt/hostedtoolcache/go/1.19.0/x64/src/fmt/scan.go:114 +0x164
  github.com/syndtr/goleveldb/leveldb/storage.fsParseName()
      /home/runner/work/goleveldb/goleveldb/leveldb/storage/file_storage.go:651 +0x26
  github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).List()
      /home/runner/work/goleveldb/goleveldb/leveldb/storage/file_storage.go:465 +0x3e6
  fmt.Fscanf()
      /opt/hostedtoolcache/go/1.19.0/x64/src/fmt/scan.go:143 +0xe9
  fmt.Sscanf()
      /opt/hostedtoolcache/go/1.19.0/x64/src/fmt/scan.go:114 +0x164
  github.com/syndtr/goleveldb/leveldb/storage.fsParseName()
      /home/runner/work/goleveldb/goleveldb/leveldb/storage/file_storage.go:651 +0x26
  github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).List()
      /home/runner/work/goleveldb/goleveldb/leveldb/storage/file_storage.go:465 +0x3e6
  github.com/syndtr/goleveldb/leveldb/testutil.(*Storage).List()
      /home/runner/work/goleveldb/goleveldb/leveldb/testutil/storage.go:349 +0x6d
  github.com/syndtr/goleveldb/leveldb.(*DB).recoverJournal()
      /home/runner/work/goleveldb/goleveldb/leveldb/db.go:489 +0xb2
  github.com/syndtr/goleveldb/leveldb.openDB()
      /home/runner/work/goleveldb/goleveldb/leveldb/db.go:129 +0x7a7
  github.com/syndtr/goleveldb/leveldb.Open()
      /home/runner/work/goleveldb/goleveldb/leveldb/db.go:202 +0x35c
  fmt.Sscanf()
      /opt/hostedtoolcache/go/1.19.0/x64/src/fmt/scan.go:114 +0x164
  github.com/syndtr/goleveldb/leveldb/storage.fsParseName()
      /home/runner/work/goleveldb/goleveldb/leveldb/storage/file_storage.go:651 +0x26
  github.com/syndtr/goleveldb/leveldb/storage.(*fileStorage).List()
      /home/runner/work/goleveldb/goleveldb/leveldb/storage/file_storage.go:465 +0x3e6
  github.com/syndtr/goleveldb/leveldb/testutil.(*Storage).List()
      /home/runner/work/goleveldb/goleveldb/leveldb/testutil/storage.go:349 +0x6d
  github.com/syndtr/goleveldb/leveldb.(*session).recover.func1()
      /home/runner/work/goleveldb/goleveldb/leveldb/session.go:134 +0xd0
  runtime.deferreturn()
      /opt/hostedtoolcache/go/1.19.0/x64/src/runtime/panic.go:476 +0x32
  github.com/syndtr/goleveldb/leveldb.Open()
      /home/runner/work/goleveldb/goleveldb/leveldb/db.go:188 +0x104
  github.com/syndtr/goleveldb/leveldb.(*dbHarness).openDB0()
      /home/runner/work/goleveldb/goleveldb/leveldb/db_test.go:98 +0xb4
  github.com/syndtr/goleveldb/leveldb.(*dbHarness).init()
      /home/runner/work/goleveldb/goleveldb/leveldb/db_test.go:89 +0x271
  github.com/syndtr/goleveldb/leveldb.newDbHarnessWopt()
      /home/runner/work/goleveldb/goleveldb/leveldb/db_test.go:71 +0x167
  github.com/syndtr/goleveldb/leveldb.TestDB_TransientError()
      /home/runner/work/goleveldb/goleveldb/leveldb/db_test.go:2360 +0x3e
  testing.tRunner()
      /opt/hostedtoolcache/go/1.19.0/x64/src/testing/testing.go:1446 +0x216
  testing.(*T).Run.func1()
      /opt/hostedtoolcache/go/1.19.0/x64/src/testing/testing.go:1493 +0x47
Goroutine 2248 (running) created at:
  testing.(*T).Run()
      /opt/hostedtoolcache/go/1.19.0/x64/src/testing/testing.go:1493 +0x75d
  testing.runTests.func1()
      /opt/hostedtoolcache/go/1.19.0/x64/src/testing/testing.go:1846 +0x99
  testing.tRunner()
      /opt/hostedtoolcache/go/1.19.0/x64/src/testing/testing.go:1446 +0x216
  testing.runTests()
      /opt/hostedtoolcache/go/1.19.0/x64/src/testing/testing.go:1844 +0x7ec
  testing.(*M).Run()
      /opt/hostedtoolcache/go/1.19.0/x64/src/testing/testing.go:1726 +0xa84
  main.main()
      _testmain.go:263 +0x2e9
==================
FAIL	github.com/syndtr/goleveldb/leveldb	188.432s

iterator: optimize merged iterator (again)

specialize index heap with int type
use heap fix instead of pop&push

benchmarks:

$ benchstat master.txt pr.txt 
name                       old time/op  new time/op  delta
MergedIterator/2_iters-8   8.97µs ± 0%  6.34µs ± 0%  -29.38%  (p=0.008 n=5+5)
MergedIterator/50_iters-8   732µs ± 0%   580µs ± 1%  -20.67%  (p=0.008 n=5+5)

LevelDB key/value database in Go.

Installation

Requirements

Usage

Documentation

Owner

Suryandaru Triandana

Comments

Require explicit general database closing.

Slice bounds out of range error during compaction

Possible memory leak

Effort to reduce heap allocations

Compaction time can be quite big

Possible corruption on the block data

db corruption on 'out of disk space'

Significant memory usage when trying to insert data

disable auto compaction

Compression change behind the scenes

go test fail on Windows/386

migrate ci to github actions, test against go1.19

slice bounds out of range [:18014398510996847] with capacity 4194304

some node will pass to f() function many times, is this a bug or a feature?

data race

iterator: optimize merged iterator (again)

Related tags

Nipo is a powerful, fast, multi-thread, clustered and in-memory key-value database, with ability to configure token and acl on commands and key-regexes written by GO

pure golang key database support key have value. 非常高效实用的键值数据库。

Simple key value database that use json files to store the database

An embedded key/value database for Go.

BuntDB is an embeddable, in-memory key/value database for Go with custom indexing and geospatial support

ACID key-value database.

Graviton Database: ZFS for key-value stores.

Membin is an in-memory database that can be stored on disk. Data model smiliar to key-value but values store as JSON byte array.

An embedded, hardened key/value database for Go.

BadgerDB is an embeddable, persistent and fast key-value (KV) database written in pure Go

GalaxyDB is a hobbyist key-value database written in Go.

levigo is a Go wrapper for LevelDB

Persistent stacks and queues for Go backed by LevelDB

Fast key-value DB in Go.

🔑A high performance Key/Value store written in Go with a predictable read/write performance and high throughput. Uses a Bitcask on-disk layout (LSM+WAL) similar to Riak.

A disk-backed key-value store.

An in-memory key:value store/cache (similar to Memcached) library for Go, suitable for single-machine applications.

A simple, fast, embeddable, persistent key/value store written in pure Go. It supports fully serializable transactions and many data structures such as list, set, sorted set.

Embedded key-value store for read-heavy workloads written in Go