Go bindings for ForestDB

goforestdb

Go bindings for ForestDB

Building

  1. Obtain and build forestdb: https://github.com/couchbaselabs/forestdb (run make install to install the library)
  2. Install header files to system location
  3. On Ubuntu 14.04: cd <forestdb_project_dir> && mkdir /usr/local/include/libforestdb && cp include/libforestdb/* /usr/local/include/libforestdb
  4. go get -u -v -t github.com/couchbase/goforestdb

Documentation

See godocs

Sample usage (without proper error handling):

// Open a database
db, _ := Open("test", nil)

// Close it properly when we're done
defer db.Close()

// Store the document
doc, _ := NewDoc([]byte("key"), nil, []byte("value"))
defer doc.Close()
db.Set(doc)

// Lookup the document
doc2, _ := NewDoc([]byte("key"), nil, nil)
defer doc2.Close()
db.Get(doc2)

// Delete the document
doc3, _ := NewDoc([]byte("key"), nil, nil)
defer doc3.Close()
db.Delete(doc3)
Comments
  • error: unexpected signal during runtime execution

    error: unexpected signal during runtime execution

    Any thoughts on following error? ( I'm using go1.5.1 darwin/amd64 on OSX 10.10.5 ( Yosemite ) with latest ForestDB )

    package main
    
    import (
        "fmt"
    
        "github.com/couchbase/goforestdb"
    )
    
    func main() {
        fmt.Println("test")
    
        // Open a database
        db, _ := forestdb.Open("test", nil)
    
        // Close it properly when we're done
        defer db.Close()
    }
    
    $ go run main.go 
    test
    fatal error: unexpected signal during runtime execution
    [signal 0xb code=0x1 addr=0x0 pc=0x0]
    
    runtime stack:
    runtime.throw(0x414c320, 0x2a)
      /usr/local/go/src/runtime/panic.go:527 +0x90
    runtime.sigpanic()
      /usr/local/go/src/runtime/sigpanic_unix.go:12 +0x5a
    
    goroutine 1 [syscall, locked to thread]:
    runtime.cgocall(0x4001d40, 0xc82004dc70, 0xc800000000)
      /usr/local/go/src/runtime/cgocall.go:120 +0x11b fp=0xc82004dc50 sp=0xc82004dc20
    github.com/couchbase/goforestdb._Cfunc_fdb_get_default_config(0x100004000008, 0x8000000, 0x1000, 0x1, 0x105400001, 0x1e00000104000000, 0x100000, 0xf, 0xc82001c001, 0x1e, ...)
      ??:0 +0x45 fp=0xc82004dc70 sp=0xc82004dc50
    github.com/couchbase/goforestdb.DefaultConfig(0xc820062270)
      gocode/src/github.com/couchbase/goforestdb/config.go:201 +0x82 fp=0xc82004de30 sp=0xc82004dc70
    github.com/couchbase/goforestdb.Open(0x4123050, 0x4, 0x0, 0x0, 0x0, 0x0)
      gocode/src/github.com/couchbase/goforestdb/file.go:30 +0x51 fp=0xc82004ded0 sp=0xc82004de30
    main.main()
      main.go:13 +0xf2 fp=0xc82004df50 sp=0xc82004ded0
    runtime.main()
      /usr/local/go/src/runtime/proc.go:111 +0x2b0 fp=0xc82004dfa0 sp=0xc82004df50
    runtime.goexit()
      /usr/local/go/src/runtime/asm_amd64.s:1696 +0x1 fp=0xc82004dfa8 sp=0xc82004dfa0
    
    goroutine 17 [syscall, locked to thread]:
    runtime.goexit()
      /usr/local/go/src/runtime/asm_amd64.s:1696 +0x1
    exit status 2
    

    goforestdb tests are okay;

    go test -v .
    === RUN   TestSnapshotAndRollback
    --- PASS: TestSnapshotAndRollback (0.00s)
    === RUN   TestForestDBCrud
    --- PASS: TestForestDBCrud (0.00s)
    === RUN   TestForestDBCompact
    --- PASS: TestForestDBCompact (0.02s)
    === RUN   TestForestDBCompactUpto
    --- PASS: TestForestDBCompactUpto (0.00s)
    === RUN   TestForestDBConcurrent
    --- PASS: TestForestDBConcurrent (4.59s)
    === RUN   TestForestDBIterator
    --- PASS: TestForestDBIterator (0.00s)
    === RUN   TestForestDBIteratorSeq
    --- PASS: TestForestDBIteratorSeq (0.00s)
    === RUN   TestForestDBIteratorSeek
    --- PASS: TestForestDBIteratorSeek (0.00s)
    === RUN   TestForestDBIteratorPrev
    --- PASS: TestForestDBIteratorPrev (0.00s)
    === RUN   TestForestDBIteratorOnSnapshot
    --- PASS: TestForestDBIteratorOnSnapshot (0.00s)
    === RUN   TestForestDBIteratorPreAlloc
    --- PASS: TestForestDBIteratorPreAlloc (0.00s)
    === RUN   TestForestDBIteratorBug
    --- PASS: TestForestDBIteratorBug (0.00s)
    === RUN   TestForestDBKVCrud
    --- PASS: TestForestDBKVCrud (0.00s)
    === RUN   TestForestDBKVBatch
    --- PASS: TestForestDBKVBatch (0.00s)
    === RUN   TestPool
    --- PASS: TestPool (0.00s)
    === RUN   TestSnapshotMarkersSingleKvs
    --- PASS: TestSnapshotMarkersSingleKvs (0.00s)
    === RUN   TestSnapshotMarkersMultiKvs
    --- PASS: TestSnapshotMarkersMultiKvs (0.00s)
    === RUN   TestTx
    --- PASS: TestTx (0.00s)
    PASS
    ok      github.com/couchbase/goforestdb 4.638s
    
  • Add logging.

    Add logging.

    In order to trace what happened when debugging forestdb crashes, it is useful to log sufficient information to show sequence of operations and perhaps useful memory locations

    Change-Id: I38691c79686ded51303a1839002a1e4c408801d1

  • Getting better stack traces?

    Getting better stack traces?

    I somehow managed to cause a panic (I'm still trying to dig into the root cause), and I'm seeing a stack trace:

    fatal error: unexpected signal during runtime execution
    [signal 0xb code=0x1 addr=0x15 pc=0x4ea78bb]
    
    runtime stack:
    runtime.gothrow(0x4847e70, 0x2a)
        /usr/local/go/src/runtime/panic.go:503 +0x8e
    runtime.sigpanic()
        /usr/local/go/src/runtime/sigpanic_unix.go:14 +0x5e
    
    goroutine 7 [syscall, locked to thread]:
    runtime.cgocall_errno(0x4002df0, 0xc20819b900, 0xc200000000)
        /usr/local/go/src/runtime/cgocall.go:130 +0xf5 fp=0xc20819b8e0 sp=0xc20819b8b8
    github.com/couchbaselabs/goforestdb._Cfunc_fdb_kvs_close(0x5500090, 0x0)
        /Users/tleyden/Development/sync_gateway/src/github.com/couchbaselabs/goforestdb/:251 +0x43 fp=0xc20819b900 sp=0xc20819b8e0
    github.com/couchbaselabs/goforestdb.(*KVStore).Close(0xc20802bb10, 0x0, 0x0)
        /Users/tleyden/Development/sync_gateway/src/github.com/couchbaselabs/goforestdb/forestdb.go:26 +0x135 fp=0xc20819b978 sp=0xc20819b900
    github.com/couchbaselabs/forestdb-bucket.(*forestdbBucket).Close(0xc208092480)
        /Users/tleyden/Development/sync_gateway/src/github.com/couchbaselabs/forestdb-bucket/forestdb_bucket.go:470 +0x71 fp=0xc20819b998 sp=0xc20819b978
    

    it's a bit cryptic, and I was wondering if there was a way to get a better error message.

  • Unexpected behavior when setting a doc body to nil

    Unexpected behavior when setting a doc body to nil

    When I set a doc body to nil and the retrieve the doc, it's returning an empty slice (and I was expecting it to return a nil doc body)

    Steps to reproduce:

    • Create new doc
    • Update doc to set doc body to nil
    • Get doc via kvstore.GetKV()
    • Expected: should return nil. Actual: empty slice is returned

    Test case

    func TestSetDocBodyToNil(t *testing.T) {
    
        rawBucket, tempDir := GetTestBucket()
    
        defer os.RemoveAll(tempDir)
        defer CloseBucket(rawBucket)
    
        bucket := rawBucket.(*forestdbBucket)
    
        // set "foo" key to value "bar"
        err := bucket.kvstore.SetKV([]byte("foo"), []byte("bar"))
        assert.True(t, err == nil)
    
        err = bucket.db.Commit(forestdb.COMMIT_NORMAL)
        assert.True(t, err == nil)
    
        // now set "foo" key to nil
        doc, err := forestdb.NewDoc([]byte("foo"), nil, nil)
        assert.True(t, err == nil)
    
        defer doc.Close()
        err = bucket.kvstore.Set(doc)
        assert.True(t, err == nil)
    
        err = bucket.db.Commit(forestdb.COMMIT_NORMAL)
        assert.True(t, err == nil)
    
        // get the value of "foo" key
        docBody, err := bucket.kvstore.GetKV([]byte("foo"))
        assert.True(t, err == nil)
        assert.True(t, docBody == nil) // fails, because it's an empty slice
    
    }
    
  • Cannot build after OSX Yosemite upgrade

    Cannot build after OSX Yosemite upgrade

    I just upgraded to OSX Yosemite (yeah I know, I was waiting for them to fix all the bugs), and now I'm having issues building goforestdb. In order to build, I had to do the following workarounds:

    1. In my .bash_profile, export C_INCLUDE_PATH=$C_INCLUDE_PATH:/usr/local/include/. It seems that by default /usr/local/include is no longer in the default header search path. (or is my system messed up somehow?)
    2. In all .go files that had an #cgo LDFLAGS directive, replace with: //#cgo LDFLAGS: -L/usr/local/lib -lforestdb to explicitly add /usr/local/lib into the library search path.

    If someone else is able to build fine on OSX Yosemite, then just close this bug and I'll try to figure out how my system is messed up. Any pointers welcome.

    Btw, forestdb itself still builds fine.

    (I wonder if I just need to reinstall or rebuild go from source?)

  • Panic when read/written concurrently

    Panic when read/written concurrently

    Here's a test script I've been using:

    package main
    
    import (
        "bytes"
        "encoding/binary"
        "fmt"
        "log"
        "os"
        "sync"
    
        fdb "github.com/couchbaselabs/goforestdb"
    )
    
    var testFile = "/tmp/test.fdb"
    var testValue = bytes.Repeat([]byte{120}, 256)
    var total = int(1e5)
    
    func abortOn(err error) {
        if err != nil {
            log.Fatal(err.Error())
        }
    }
    
    func main() {
        os.RemoveAll(testFile)
    
        config := fdb.DefaultConfig()
        config.SetWalFlushBeforeCommit(true)
    
        wg := sync.WaitGroup{}
        db, err := fdb.Open(testFile, config)
        abortOn(err)
        defer db.Close()
    
        abortOn(write(db, 0))
        abortOn(read(db, 0))
    
        wg.Add(1)
        go func() {
            defer wg.Done()
            abortOn(write(db, 1))
        }()
        wg.Add(1)
        go func() {
            defer wg.Done()
            abortOn(read(db, 0))
        }()
        wg.Wait()
    }
    
    func write(db *fdb.Database, base int) error {
        fmt.Println("* writing")
        for n := 0; n < total; n++ {
            key := make([]byte, 4)
            binary.BigEndian.PutUint32(key, uint32(base*total+n))
            if err := db.SetKV(key, testValue); err != nil {
                return err
            }
        }
    
        return db.Commit(fdb.COMMIT_NORMAL)
    }
    
    func read(db *fdb.Database, base int) error {
        fmt.Println("* reading")
        for n := 0; n < total; n++ {
            key := make([]byte, 4)
            binary.BigEndian.PutUint32(key, uint32(base*total+n))
            if _, err := db.GetKV(key); err != nil {
                return err
            }
        }
        return nil
    }
    

    Causes:

    $ go run fdb-bug.go 
    * writing
    * reading
    * writing
    * reading
    *** Error in `/tmp/go-build726603444/command-line-arguments/_obj/exe/fdb-bug': double free or corruption (fasttop): 0x00000000097f6e00 ***
    SIGABRT: abort
    PC=0x7f1172a0ad27
    signal arrived during cgo execution
    
    goroutine 21 [syscall]:
    runtime.cgocall(0x402410, 0x7f1173239e08)
      $GOROOT/src/pkg/runtime/cgocall.c:143 +0xe5 fp=0x7f1173239df0 sp=0x7f1173239da8
    github.com/couchbaselabs/goforestdb._Cfunc_fdb_get_kv(0x13b62c0, 0xc208073040, 0x4, 0xc20803c758, 0xc208073048, 0x1)
      github.com/couchbaselabs/goforestdb/_obj/_cgo_defun.c:179 +0x31 fp=0x7f1173239e08 sp=0x7f1173239df0
    github.com/couchbaselabs/goforestdb.(*Database).GetKV(0xc20803c028, 0xc208073040, 0x4, 0x4, 0x0, 0x0, 0x0, 0x0, 0x0)
      $GOPATH/src/github.com/couchbaselabs/goforestdb/kv.go:32 +0xf2 fp=0x7f1173239e88 sp=0x7f1173239e08
    main.read(0xc20803c028, 0x0, 0x0, 0x0)
      fdb-bug.go:68 +0x1ee fp=0x7f1173239f68 sp=0x7f1173239e88
    main.funcĀ·002()
      fdb-bug.go:45 +0x58 fp=0x7f1173239fa8 sp=0x7f1173239f68
    runtime.goexit()
      $GOROOT/src/pkg/runtime/proc.c:1445 fp=0x7f1173239fb0 sp=0x7f1173239fa8
    created by main.main
      fdb-bug.go:46 +0x23d
    
    goroutine 16 [semacquire]:
    sync.runtime_Semacquire(0xc208096e50)
      $GOROOT/src/pkg/runtime/sema.goc:199 +0x30
    sync.(*WaitGroup).Wait(0xc20805a000)
      $GOROOT/src/pkg/sync/waitgroup.go:129 +0x14b
    main.main()
      fdb-bug.go:47 +0x24d
    
    goroutine 19 [finalizer wait]:
    runtime.park(0x4153f0, 0x77bbd8, 0x770de9)
      $GOROOT/src/pkg/runtime/proc.c:1369 +0x89
    runtime.parkunlock(0x77bbd8, 0x770de9)
      $GOROOT/src/pkg/runtime/proc.c:1385 +0x3b
    runfinq()
      $GOROOT/src/pkg/runtime/mgc0.c:2644 +0xcf
    runtime.goexit()
      $GOROOT/src/pkg/runtime/proc.c:1445
    
    goroutine 17 [syscall]:
    runtime.goexit()
      $GOROOT/src/pkg/runtime/proc.c:1445
    
    goroutine 20 [syscall]:
    github.com/couchbaselabs/goforestdb._Cfunc_fdb_set_kv(0x13b62c0, 0xc208073020, 0x4, 0xc208040000, 0x100, 0x425de3)
      github.com/couchbaselabs/goforestdb/_obj/_cgo_defun.c:305 +0x31
    github.com/couchbaselabs/goforestdb.(*Database).SetKV(0xc20803c028, 0xc208073020, 0x4, 0x4, 0xc208040000, 0x100, 0x100, 0x0, 0x0)
      $GOPATH/src/github.com/couchbaselabs/goforestdb/kv.go:58 +0xa4
    main.write(0xc20803c028, 0x1, 0x0, 0x0)
      fdb-bug.go:55 +0x215
    main.funcĀ·001()
      fdb-bug.go:40 +0x58
    created by main.main
      fdb-bug.go:41 +0x1f0
    
    rax     0x0
    rbx     0x8b
    rcx     0xffffffffffffffff
    rdx     0x6
    rdi     0x74ef
    rsi     0x74f2
    rbp     0x7f1170e9ebe0
    rsp     0x7f1170e9e848
    r8      0x3030653666373930
    r9      0x6f6974707572726f
    r10     0x8
    r11     0x202
    r12     0x7f1170e9e9f0
    r13     0x7
    r14     0x8b
    r15     0x7
    rip     0x7f1172a0ad27
    rflags  0x202
    cs      0x33
    fs      0x0
    gs      0x0
    exit status 2
    godep: go exit status 1
    
  • memory leak?

    memory leak?

    I am not sure if it's the C code or the go wrapper or maybe something I misunderstand altogether, but the following piece of code is an easy way to consume all available memory of your machine:

    package main
    
    import (
        "encoding/binary"
        "log"
    
        fdb "github.com/couchbaselabs/goforestdb"
    )
    
    func abortOn(err error) {
        if err != nil {
            log.Fatal(err)
        }
    }
    
    func main() {
        db, err := fdb.Open("/tmp/leak.fdb", fdb.DefaultConfig())
        abortOn(err)
    
        for i := 0; i < 100000000; i++ {
            key := make([]byte, 8)
            binary.BigEndian.PutUint64(key, uint64(i))
            err := db.SetKV(key, []byte("BOGUSDATA"))
            abortOn(err)
    
            if (i+1)%1000000 == 0 {
                log.Printf("written %d keys", i+1)
            }
        }
    
    }
    

    It looks like a memory leak to me. I tried both, the db.Set and the db.SetKV methods (obviously closing the documents) and they both produce the same pattern of uncontrolled memory consumption growth.

    Any ideas? Thanks!

Go bindings for the PortAudio audio I/O library

portaudio This package provides an interface to the PortAudio audio I/O library. See the package documentation for details. To build this package you

Jan 1, 2023
Go bindings for libportmidi

portmidi Want to output to an MIDI device or listen your MIDI device as an input? This package contains Go bindings for PortMidi. libportmidi (v. 217)

Dec 25, 2022
Golang bindings for the Telegram Bot API

Golang bindings for the Telegram Bot API All methods are fairly self explanatory, and reading the godoc page should explain everything. If something i

Jan 6, 2023
OpenSSL bindings for Go

OpenSSL bindings for Go Please see http://godoc.org/github.com/spacemonkeygo/openssl for more info License Copyright (C) 2017. See AUTHORS. Licensed u

Dec 29, 2022
Golang bindings for libxlsxwriter for writing XLSX files
Golang bindings for libxlsxwriter for writing XLSX files

goxlsxwriter provides Go bindings for the libxlsxwriter C library. Install goxlsxwriter requires the libxslxwriter library to be installe

Nov 18, 2022
Go bindings for GLib type system.

Go bindings for GLib type system. This package is designed for building bindings to C libraries based on GLib type system (like GTK, GStreamer, and ot

Aug 13, 2019
Source code editor written in Go using go-gtk bindings. It aims to handle navigation effectively among large number of files.
Source code editor written in Go using go-gtk bindings. It aims to handle navigation effectively among large number of files.

tabby Source code editor written in Go using go-gtk bindings. It aims to handle navigation effectively among large number of files. screenshot: depend

Nov 16, 2022
Duktape JavaScript engine bindings for Go

Duktape bindings for Go(Golang) Duktape is a thin, embeddable javascript engine. Most of the api is implemented. The exceptions are listed here. Usage

Jan 6, 2023
PHP bindings for the Go programming language (Golang)

PHP bindings for Go This package implements support for executing PHP scripts, exporting Go variables for use in PHP contexts, attaching Go method rec

Jan 1, 2023
naive go bindings to the CPython C-API

go-python Naive go bindings towards the C-API of CPython-2. this package provides a go package named "python" under which most of the PyXYZ functions

Jan 5, 2023
Go bindings for Lua C API - in progress

Go Bindings for the lua C API Simplest way to install: # go get github.com/aarzilli/golua/lua You can then try to run the examples: $ cd golua/_examp

Dec 28, 2022
Go bindings for libmagic to detect MIME types

magicmime magicmime is a Go package which allows you to discover a file's mimetype by looking for magic numbers in its content. It could be used as a

Nov 9, 2022
Go bindings for raylib, a simple and easy-to-use library to enjoy videogames programming.
Go bindings for raylib, a simple and easy-to-use library to enjoy videogames programming.

raylib-go Golang bindings for raylib, a simple and easy-to-use library to enjoy videogames programming. Requirements Ubuntu X11 apt-get install libgl1

Dec 28, 2022
Go bindings for the Cartographic Projections Library PROJ.4

The Go package proj provides a limited interface to the Cartographic Projections Library PROJ. For PROJ version 5 and beyond, see also: https://github

Nov 10, 2022
Go bindings for GLFW 3

GLFW 3.3 for Go Installation GLFW C library source is included and built automatically as part of the Go package. But you need to make sure you have d

Jan 8, 2023
Go bindings for GLFW 3

GLFW 3.3 for Go Installation GLFW C library source is included and built automatically as part of the Go package. But you need to make sure you have d

Dec 25, 2022
Go bindings for OpenGL (generated via glow)

gl This repository holds Go bindings to various OpenGL versions. They are auto-generated using Glow. Features: Go functions that mirror the C specific

Dec 12, 2022
Go bindings to OpenGL Utility Library

GLU This package offers minimal bindings for GLU functions. Usage go get github.com/go-gl-legacy/glu License Copyright 2012 The go-gl Authors. All ri

Aug 18, 2018
Go bindings for audio capture and playback with ALSA and libasound

Go ALSA bindings These bindings allow capture and playback of audio via ALSA using the alsa-lib library. Installation go get github.com/cocoonlife/goa

Nov 26, 2022
naive go bindings to GnuPlot
naive go bindings to GnuPlot

go-gnuplot Simple-minded functions to work with gnuplot. go-gnuplot runs gnuplot as a subprocess and pushes commands via the STDIN of that subprocess.

Nov 8, 2021