An embedded key/value database for Go.

Bolt Coverage Status GoDoc Version

Bolt is a pure Go key/value store inspired by Howard Chu's LMDB project. The goal of the project is to provide a simple, fast, and reliable database for projects that don't require a full database server such as Postgres or MySQL.

Since Bolt is meant to be used as such a low-level piece of functionality, simplicity is key. The API will be small and only focus on getting values and setting values. That's it.

Project Status

Bolt is stable, the API is fixed, and the file format is fixed. Full unit test coverage and randomized black box testing are used to ensure database consistency and thread safety. Bolt is currently used in high-load production environments serving databases as large as 1TB. Many companies such as Shopify and Heroku use Bolt-backed services every day.

A message from the author

The original goal of Bolt was to provide a simple pure Go key/value store and to not bloat the code with extraneous features. To that end, the project has been a success. However, this limited scope also means that the project is complete.

Maintaining an open source database requires an immense amount of time and energy. Changes to the code can have unintended and sometimes catastrophic effects so even simple changes require hours and hours of careful testing and validation.

Unfortunately I no longer have the time or energy to continue this work. Bolt is in a stable state and has years of successful production use. As such, I feel that leaving it in its current state is the most prudent course of action.

If you are interested in using a more featureful version of Bolt, I suggest that you look at the CoreOS fork called bbolt.

Table of Contents

Getting Started

Installing

To start using Bolt, install Go and run go get:

$ go get github.com/boltdb/bolt/...

This will retrieve the library and install the bolt command line utility into your $GOBIN path.

Opening a database

The top-level object in Bolt is a DB. It is represented as a single file on your disk and represents a consistent snapshot of your data.

To open your database, simply use the bolt.Open() function:

package main

import (
	"log"

	"github.com/boltdb/bolt"
)

func main() {
	// Open the my.db data file in your current directory.
	// It will be created if it doesn't exist.
	db, err := bolt.Open("my.db", 0600, nil)
	if err != nil {
		log.Fatal(err)
	}
	defer db.Close()

	...
}

Please note that Bolt obtains a file lock on the data file so multiple processes cannot open the same database at the same time. Opening an already open Bolt database will cause it to hang until the other process closes it. To prevent an indefinite wait you can pass a timeout option to the Open() function:

db, err := bolt.Open("my.db", 0600, &bolt.Options{Timeout: 1 * time.Second})

Transactions

Bolt allows only one read-write transaction at a time but allows as many read-only transactions as you want at a time. Each transaction has a consistent view of the data as it existed when the transaction started.

Individual transactions and all objects created from them (e.g. buckets, keys) are not thread safe. To work with data in multiple goroutines you must start a transaction for each one or use locking to ensure only one goroutine accesses a transaction at a time. Creating transaction from the DB is thread safe.

Read-only transactions and read-write transactions should not depend on one another and generally shouldn't be opened simultaneously in the same goroutine. This can cause a deadlock as the read-write transaction needs to periodically re-map the data file but it cannot do so while a read-only transaction is open.

Read-write transactions

To start a read-write transaction, you can use the DB.Update() function:

err := db.Update(func(tx *bolt.Tx) error {
	...
	return nil
})

Inside the closure, you have a consistent view of the database. You commit the transaction by returning nil at the end. You can also rollback the transaction at any point by returning an error. All database operations are allowed inside a read-write transaction.

Always check the return error as it will report any disk failures that can cause your transaction to not complete. If you return an error within your closure it will be passed through.

Read-only transactions

To start a read-only transaction, you can use the DB.View() function:

err := db.View(func(tx *bolt.Tx) error {
	...
	return nil
})

You also get a consistent view of the database within this closure, however, no mutating operations are allowed within a read-only transaction. You can only retrieve buckets, retrieve values, and copy the database within a read-only transaction.

Batch read-write transactions

Each DB.Update() waits for disk to commit the writes. This overhead can be minimized by combining multiple updates with the DB.Batch() function:

err := db.Batch(func(tx *bolt.Tx) error {
	...
	return nil
})

Concurrent Batch calls are opportunistically combined into larger transactions. Batch is only useful when there are multiple goroutines calling it.

The trade-off is that Batch can call the given function multiple times, if parts of the transaction fail. The function must be idempotent and side effects must take effect only after a successful return from DB.Batch().

For example: don't display messages from inside the function, instead set variables in the enclosing scope:

var id uint64
err := db.Batch(func(tx *bolt.Tx) error {
	// Find last key in bucket, decode as bigendian uint64, increment
	// by one, encode back to []byte, and add new key.
	...
	id = newValue
	return nil
})
if err != nil {
	return ...
}
fmt.Println("Allocated ID %d", id)

Managing transactions manually

The DB.View() and DB.Update() functions are wrappers around the DB.Begin() function. These helper functions will start the transaction, execute a function, and then safely close your transaction if an error is returned. This is the recommended way to use Bolt transactions.

However, sometimes you may want to manually start and end your transactions. You can use the DB.Begin() function directly but please be sure to close the transaction.

// Start a writable transaction.
tx, err := db.Begin(true)
if err != nil {
    return err
}
defer tx.Rollback()

// Use the transaction...
_, err := tx.CreateBucket([]byte("MyBucket"))
if err != nil {
    return err
}

// Commit the transaction and check for error.
if err := tx.Commit(); err != nil {
    return err
}

The first argument to DB.Begin() is a boolean stating if the transaction should be writable.

Using buckets

Buckets are collections of key/value pairs within the database. All keys in a bucket must be unique. You can create a bucket using the DB.CreateBucket() function:

db.Update(func(tx *bolt.Tx) error {
	b, err := tx.CreateBucket([]byte("MyBucket"))
	if err != nil {
		return fmt.Errorf("create bucket: %s", err)
	}
	return nil
})

You can also create a bucket only if it doesn't exist by using the Tx.CreateBucketIfNotExists() function. It's a common pattern to call this function for all your top-level buckets after you open your database so you can guarantee that they exist for future transactions.

To delete a bucket, simply call the Tx.DeleteBucket() function.

Using key/value pairs

To save a key/value pair to a bucket, use the Bucket.Put() function:

db.Update(func(tx *bolt.Tx) error {
	b := tx.Bucket([]byte("MyBucket"))
	err := b.Put([]byte("answer"), []byte("42"))
	return err
})

This will set the value of the "answer" key to "42" in the MyBucket bucket. To retrieve this value, we can use the Bucket.Get() function:

db.View(func(tx *bolt.Tx) error {
	b := tx.Bucket([]byte("MyBucket"))
	v := b.Get([]byte("answer"))
	fmt.Printf("The answer is: %s\n", v)
	return nil
})

The Get() function does not return an error because its operation is guaranteed to work (unless there is some kind of system failure). If the key exists then it will return its byte slice value. If it doesn't exist then it will return nil. It's important to note that you can have a zero-length value set to a key which is different than the key not existing.

Use the Bucket.Delete() function to delete a key from the bucket.

Please note that values returned from Get() are only valid while the transaction is open. If you need to use a value outside of the transaction then you must use copy() to copy it to another byte slice.

Autoincrementing integer for the bucket

By using the NextSequence() function, you can let Bolt determine a sequence which can be used as the unique identifier for your key/value pairs. See the example below.

// CreateUser saves u to the store. The new user ID is set on u once the data is persisted.
func (s *Store) CreateUser(u *User) error {
    return s.db.Update(func(tx *bolt.Tx) error {
        // Retrieve the users bucket.
        // This should be created when the DB is first opened.
        b := tx.Bucket([]byte("users"))

        // Generate ID for the user.
        // This returns an error only if the Tx is closed or not writeable.
        // That can't happen in an Update() call so I ignore the error check.
        id, _ := b.NextSequence()
        u.ID = int(id)

        // Marshal user data into bytes.
        buf, err := json.Marshal(u)
        if err != nil {
            return err
        }

        // Persist bytes to users bucket.
        return b.Put(itob(u.ID), buf)
    })
}

// itob returns an 8-byte big endian representation of v.
func itob(v int) []byte {
    b := make([]byte, 8)
    binary.BigEndian.PutUint64(b, uint64(v))
    return b
}

type User struct {
    ID int
    ...
}

Iterating over keys

Bolt stores its keys in byte-sorted order within a bucket. This makes sequential iteration over these keys extremely fast. To iterate over keys we'll use a Cursor:

db.View(func(tx *bolt.Tx) error {
	// Assume bucket exists and has keys
	b := tx.Bucket([]byte("MyBucket"))

	c := b.Cursor()

	for k, v := c.First(); k != nil; k, v = c.Next() {
		fmt.Printf("key=%s, value=%s\n", k, v)
	}

	return nil
})

The cursor allows you to move to a specific point in the list of keys and move forward or backward through the keys one at a time.

The following functions are available on the cursor:

First()  Move to the first key.
Last()   Move to the last key.
Seek()   Move to a specific key.
Next()   Move to the next key.
Prev()   Move to the previous key.

Each of those functions has a return signature of (key []byte, value []byte). When you have iterated to the end of the cursor then Next() will return a nil key. You must seek to a position using First(), Last(), or Seek() before calling Next() or Prev(). If you do not seek to a position then these functions will return a nil key.

During iteration, if the key is non-nil but the value is nil, that means the key refers to a bucket rather than a value. Use Bucket.Bucket() to access the sub-bucket.

Prefix scans

To iterate over a key prefix, you can combine Seek() and bytes.HasPrefix():

db.View(func(tx *bolt.Tx) error {
	// Assume bucket exists and has keys
	c := tx.Bucket([]byte("MyBucket")).Cursor()

	prefix := []byte("1234")
	for k, v := c.Seek(prefix); k != nil && bytes.HasPrefix(k, prefix); k, v = c.Next() {
		fmt.Printf("key=%s, value=%s\n", k, v)
	}

	return nil
})

Range scans

Another common use case is scanning over a range such as a time range. If you use a sortable time encoding such as RFC3339 then you can query a specific date range like this:

db.View(func(tx *bolt.Tx) error {
	// Assume our events bucket exists and has RFC3339 encoded time keys.
	c := tx.Bucket([]byte("Events")).Cursor()

	// Our time range spans the 90's decade.
	min := []byte("1990-01-01T00:00:00Z")
	max := []byte("2000-01-01T00:00:00Z")

	// Iterate over the 90's.
	for k, v := c.Seek(min); k != nil && bytes.Compare(k, max) <= 0; k, v = c.Next() {
		fmt.Printf("%s: %s\n", k, v)
	}

	return nil
})

Note that, while RFC3339 is sortable, the Golang implementation of RFC3339Nano does not use a fixed number of digits after the decimal point and is therefore not sortable.

ForEach()

You can also use the function ForEach() if you know you'll be iterating over all the keys in a bucket:

db.View(func(tx *bolt.Tx) error {
	// Assume bucket exists and has keys
	b := tx.Bucket([]byte("MyBucket"))

	b.ForEach(func(k, v []byte) error {
		fmt.Printf("key=%s, value=%s\n", k, v)
		return nil
	})
	return nil
})

Please note that keys and values in ForEach() are only valid while the transaction is open. If you need to use a key or value outside of the transaction, you must use copy() to copy it to another byte slice.

Nested buckets

You can also store a bucket in a key to create nested buckets. The API is the same as the bucket management API on the DB object:

func (*Bucket) CreateBucket(key []byte) (*Bucket, error)
func (*Bucket) CreateBucketIfNotExists(key []byte) (*Bucket, error)
func (*Bucket) DeleteBucket(key []byte) error

Say you had a multi-tenant application where the root level bucket was the account bucket. Inside of this bucket was a sequence of accounts which themselves are buckets. And inside the sequence bucket you could have many buckets pertaining to the Account itself (Users, Notes, etc) isolating the information into logical groupings.

// createUser creates a new user in the given account.
func createUser(accountID int, u *User) error {
    // Start the transaction.
    tx, err := db.Begin(true)
    if err != nil {
        return err
    }
    defer tx.Rollback()

    // Retrieve the root bucket for the account.
    // Assume this has already been created when the account was set up.
    root := tx.Bucket([]byte(strconv.FormatUint(accountID, 10)))

    // Setup the users bucket.
    bkt, err := root.CreateBucketIfNotExists([]byte("USERS"))
    if err != nil {
        return err
    }

    // Generate an ID for the new user.
    userID, err := bkt.NextSequence()
    if err != nil {
        return err
    }
    u.ID = userID

    // Marshal and save the encoded user.
    if buf, err := json.Marshal(u); err != nil {
        return err
    } else if err := bkt.Put([]byte(strconv.FormatUint(u.ID, 10)), buf); err != nil {
        return err
    }

    // Commit the transaction.
    if err := tx.Commit(); err != nil {
        return err
    }

    return nil
}

Database backups

Bolt is a single file so it's easy to backup. You can use the Tx.WriteTo() function to write a consistent view of the database to a writer. If you call this from a read-only transaction, it will perform a hot backup and not block your other database reads and writes.

By default, it will use a regular file handle which will utilize the operating system's page cache. See the Tx documentation for information about optimizing for larger-than-RAM datasets.

One common use case is to backup over HTTP so you can use tools like cURL to do database backups:

func BackupHandleFunc(w http.ResponseWriter, req *http.Request) {
	err := db.View(func(tx *bolt.Tx) error {
		w.Header().Set("Content-Type", "application/octet-stream")
		w.Header().Set("Content-Disposition", `attachment; filename="my.db"`)
		w.Header().Set("Content-Length", strconv.Itoa(int(tx.Size())))
		_, err := tx.WriteTo(w)
		return err
	})
	if err != nil {
		http.Error(w, err.Error(), http.StatusInternalServerError)
	}
}

Then you can backup using this command:

$ curl http://localhost/backup > my.db

Or you can open your browser to http://localhost/backup and it will download automatically.

If you want to backup to another file you can use the Tx.CopyFile() helper function.

Statistics

The database keeps a running count of many of the internal operations it performs so you can better understand what's going on. By grabbing a snapshot of these stats at two points in time we can see what operations were performed in that time range.

For example, we could start a goroutine to log stats every 10 seconds:

go func() {
	// Grab the initial stats.
	prev := db.Stats()

	for {
		// Wait for 10s.
		time.Sleep(10 * time.Second)

		// Grab the current stats and diff them.
		stats := db.Stats()
		diff := stats.Sub(&prev)

		// Encode stats to JSON and print to STDERR.
		json.NewEncoder(os.Stderr).Encode(diff)

		// Save stats for the next loop.
		prev = stats
	}
}()

It's also useful to pipe these stats to a service such as statsd for monitoring or to provide an HTTP endpoint that will perform a fixed-length sample.

Read-Only Mode

Sometimes it is useful to create a shared, read-only Bolt database. To this, set the Options.ReadOnly flag when opening your database. Read-only mode uses a shared lock to allow multiple processes to read from the database but it will block any processes from opening the database in read-write mode.

db, err := bolt.Open("my.db", 0666, &bolt.Options{ReadOnly: true})
if err != nil {
	log.Fatal(err)
}

Mobile Use (iOS/Android)

Bolt is able to run on mobile devices by leveraging the binding feature of the gomobile tool. Create a struct that will contain your database logic and a reference to a *bolt.DB with a initializing constructor that takes in a filepath where the database file will be stored. Neither Android nor iOS require extra permissions or cleanup from using this method.

func NewBoltDB(filepath string) *BoltDB {
	db, err := bolt.Open(filepath+"/demo.db", 0600, nil)
	if err != nil {
		log.Fatal(err)
	}

	return &BoltDB{db}
}

type BoltDB struct {
	db *bolt.DB
	...
}

func (b *BoltDB) Path() string {
	return b.db.Path()
}

func (b *BoltDB) Close() {
	b.db.Close()
}

Database logic should be defined as methods on this wrapper struct.

To initialize this struct from the native language (both platforms now sync their local storage to the cloud. These snippets disable that functionality for the database file):

Android

String path;
if (android.os.Build.VERSION.SDK_INT >=android.os.Build.VERSION_CODES.LOLLIPOP){
    path = getNoBackupFilesDir().getAbsolutePath();
} else{
    path = getFilesDir().getAbsolutePath();
}
Boltmobiledemo.BoltDB boltDB = Boltmobiledemo.NewBoltDB(path)

iOS

- (void)demo {
    NSString* path = [NSSearchPathForDirectoriesInDomains(NSLibraryDirectory,
                                                          NSUserDomainMask,
                                                          YES) objectAtIndex:0];
	GoBoltmobiledemoBoltDB * demo = GoBoltmobiledemoNewBoltDB(path);
	[self addSkipBackupAttributeToItemAtPath:demo.path];
	//Some DB Logic would go here
	[demo close];
}

- (BOOL)addSkipBackupAttributeToItemAtPath:(NSString *) filePathString
{
    NSURL* URL= [NSURL fileURLWithPath: filePathString];
    assert([[NSFileManager defaultManager] fileExistsAtPath: [URL path]]);

    NSError *error = nil;
    BOOL success = [URL setResourceValue: [NSNumber numberWithBool: YES]
                                  forKey: NSURLIsExcludedFromBackupKey error: &error];
    if(!success){
        NSLog(@"Error excluding %@ from backup %@", [URL lastPathComponent], error);
    }
    return success;
}

Resources

For more information on getting started with Bolt, check out the following articles:

Comparison with other databases

Postgres, MySQL, & other relational databases

Relational databases structure data into rows and are only accessible through the use of SQL. This approach provides flexibility in how you store and query your data but also incurs overhead in parsing and planning SQL statements. Bolt accesses all data by a byte slice key. This makes Bolt fast to read and write data by key but provides no built-in support for joining values together.

Most relational databases (with the exception of SQLite) are standalone servers that run separately from your application. This gives your systems flexibility to connect multiple application servers to a single database server but also adds overhead in serializing and transporting data over the network. Bolt runs as a library included in your application so all data access has to go through your application's process. This brings data closer to your application but limits multi-process access to the data.

LevelDB, RocksDB

LevelDB and its derivatives (RocksDB, HyperLevelDB) are similar to Bolt in that they are libraries bundled into the application, however, their underlying structure is a log-structured merge-tree (LSM tree). An LSM tree optimizes random writes by using a write ahead log and multi-tiered, sorted files called SSTables. Bolt uses a B+tree internally and only a single file. Both approaches have trade-offs.

If you require a high random write throughput (>10,000 w/sec) or you need to use spinning disks then LevelDB could be a good choice. If your application is read-heavy or does a lot of range scans then Bolt could be a good choice.

One other important consideration is that LevelDB does not have transactions. It supports batch writing of key/values pairs and it supports read snapshots but it will not give you the ability to do a compare-and-swap operation safely. Bolt supports fully serializable ACID transactions.

LMDB

Bolt was originally a port of LMDB so it is architecturally similar. Both use a B+tree, have ACID semantics with fully serializable transactions, and support lock-free MVCC using a single writer and multiple readers.

The two projects have somewhat diverged. LMDB heavily focuses on raw performance while Bolt has focused on simplicity and ease of use. For example, LMDB allows several unsafe actions such as direct writes for the sake of performance. Bolt opts to disallow actions which can leave the database in a corrupted state. The only exception to this in Bolt is DB.NoSync.

There are also a few differences in API. LMDB requires a maximum mmap size when opening an mdb_env whereas Bolt will handle incremental mmap resizing automatically. LMDB overloads the getter and setter functions with multiple flags whereas Bolt splits these specialized cases into their own functions.

Caveats & Limitations

It's important to pick the right tool for the job and Bolt is no exception. Here are a few things to note when evaluating and using Bolt:

  • Bolt is good for read intensive workloads. Sequential write performance is also fast but random writes can be slow. You can use DB.Batch() or add a write-ahead log to help mitigate this issue.

  • Bolt uses a B+tree internally so there can be a lot of random page access. SSDs provide a significant performance boost over spinning disks.

  • Try to avoid long running read transactions. Bolt uses copy-on-write so old pages cannot be reclaimed while an old transaction is using them.

  • Byte slices returned from Bolt are only valid during a transaction. Once the transaction has been committed or rolled back then the memory they point to can be reused by a new page or can be unmapped from virtual memory and you'll see an unexpected fault address panic when accessing it.

  • Bolt uses an exclusive write lock on the database file so it cannot be shared by multiple processes.

  • Be careful when using Bucket.FillPercent. Setting a high fill percent for buckets that have random inserts will cause your database to have very poor page utilization.

  • Use larger buckets in general. Smaller buckets causes poor page utilization once they become larger than the page size (typically 4KB).

  • Bulk loading a lot of random writes into a new bucket can be slow as the page will not split until the transaction is committed. Randomly inserting more than 100,000 key/value pairs into a single new bucket in a single transaction is not advised.

  • Bolt uses a memory-mapped file so the underlying operating system handles the caching of the data. Typically, the OS will cache as much of the file as it can in memory and will release memory as needed to other processes. This means that Bolt can show very high memory usage when working with large databases. However, this is expected and the OS will release memory as needed. Bolt can handle databases much larger than the available physical RAM, provided its memory-map fits in the process virtual address space. It may be problematic on 32-bits systems.

  • The data structures in the Bolt database are memory mapped so the data file will be endian specific. This means that you cannot copy a Bolt file from a little endian machine to a big endian machine and have it work. For most users this is not a concern since most modern CPUs are little endian.

  • Because of the way pages are laid out on disk, Bolt cannot truncate data files and return free pages back to the disk. Instead, Bolt maintains a free list of unused pages within its data file. These free pages can be reused by later transactions. This works well for many use cases as databases generally tend to grow. However, it's important to note that deleting large chunks of data will not allow you to reclaim that space on disk.

    For more information on page allocation, see this comment.

Reading the Source

Bolt is a relatively small code base (<3KLOC) for an embedded, serializable, transactional key/value database so it can be a good starting point for people interested in how databases work.

The best places to start are the main entry points into Bolt:

  • Open() - Initializes the reference to the database. It's responsible for creating the database if it doesn't exist, obtaining an exclusive lock on the file, reading the meta pages, & memory-mapping the file.

  • DB.Begin() - Starts a read-only or read-write transaction depending on the value of the writable argument. This requires briefly obtaining the "meta" lock to keep track of open transactions. Only one read-write transaction can exist at a time so the "rwlock" is acquired during the life of a read-write transaction.

  • Bucket.Put() - Writes a key/value pair into a bucket. After validating the arguments, a cursor is used to traverse the B+tree to the page and position where they key & value will be written. Once the position is found, the bucket materializes the underlying page and the page's parent pages into memory as "nodes". These nodes are where mutations occur during read-write transactions. These changes get flushed to disk during commit.

  • Bucket.Get() - Retrieves a key/value pair from a bucket. This uses a cursor to move to the page & position of a key/value pair. During a read-only transaction, the key and value data is returned as a direct reference to the underlying mmap file so there's no allocation overhead. For read-write transactions, this data may reference the mmap file or one of the in-memory node values.

  • Cursor - This object is simply for traversing the B+tree of on-disk pages or in-memory nodes. It can seek to a specific key, move to the first or last value, or it can move forward or backward. The cursor handles the movement up and down the B+tree transparently to the end user.

  • Tx.Commit() - Converts the in-memory dirty nodes and the list of free pages into pages to be written to disk. Writing to disk then occurs in two phases. First, the dirty pages are written to disk and an fsync() occurs. Second, a new meta page with an incremented transaction ID is written and another fsync() occurs. This two phase write ensures that partially written data pages are ignored in the event of a crash since the meta page pointing to them is never written. Partially written meta pages are invalidated because they are written with a checksum.

If you have additional notes that could be helpful for others, please submit them via pull request.

Other Projects Using Bolt

Below is a list of public, open source projects that use Bolt:

  • BoltDbWeb - A web based GUI for BoltDB files.
  • Operation Go: A Routine Mission - An online programming game for Golang using Bolt for user accounts and a leaderboard.
  • Bazil - A file system that lets your data reside where it is most convenient for it to reside.
  • DVID - Added Bolt as optional storage engine and testing it against Basho-tuned leveldb.
  • Skybox Analytics - A standalone funnel analysis tool for web analytics.
  • Scuttlebutt - Uses Bolt to store and process all Twitter mentions of GitHub projects.
  • Wiki - A tiny wiki using Goji, BoltDB and Blackfriday.
  • ChainStore - Simple key-value interface to a variety of storage engines organized as a chain of operations.
  • MetricBase - Single-binary version of Graphite.
  • Gitchain - Decentralized, peer-to-peer Git repositories aka "Git meets Bitcoin".
  • event-shuttle - A Unix system service to collect and reliably deliver messages to Kafka.
  • ipxed - Web interface and api for ipxed.
  • BoltStore - Session store using Bolt.
  • photosite/session - Sessions for a photo viewing site.
  • LedisDB - A high performance NoSQL, using Bolt as optional storage.
  • ipLocator - A fast ip-geo-location-server using bolt with bloom filters.
  • cayley - Cayley is an open-source graph database using Bolt as optional backend.
  • bleve - A pure Go search engine similar to ElasticSearch that uses Bolt as the default storage backend.
  • tentacool - REST api server to manage system stuff (IP, DNS, Gateway...) on a linux server.
  • Seaweed File System - Highly scalable distributed key~file system with O(1) disk read.
  • InfluxDB - Scalable datastore for metrics, events, and real-time analytics.
  • Freehold - An open, secure, and lightweight platform for your files and data.
  • Prometheus Annotation Server - Annotation server for PromDash & Prometheus service monitoring system.
  • Consul - Consul is service discovery and configuration made easy. Distributed, highly available, and datacenter-aware.
  • Kala - Kala is a modern job scheduler optimized to run on a single node. It is persistent, JSON over HTTP API, ISO 8601 duration notation, and dependent jobs.
  • drive - drive is an unofficial Google Drive command line client for *NIX operating systems.
  • stow - a persistence manager for objects backed by boltdb.
  • buckets - a bolt wrapper streamlining simple tx and key scans.
  • mbuckets - A Bolt wrapper that allows easy operations on multi level (nested) buckets.
  • Request Baskets - A web service to collect arbitrary HTTP requests and inspect them via REST API or simple web UI, similar to RequestBin service
  • Go Report Card - Go code quality report cards as a (free and open source) service.
  • Boltdb Boilerplate - Boilerplate wrapper around bolt aiming to make simple calls one-liners.
  • lru - Easy to use Bolt-backed Least-Recently-Used (LRU) read-through cache with chainable remote stores.
  • Storm - Simple and powerful ORM for BoltDB.
  • GoWebApp - A basic MVC web application in Go using BoltDB.
  • SimpleBolt - A simple way to use BoltDB. Deals mainly with strings.
  • Algernon - A HTTP/2 web server with built-in support for Lua. Uses BoltDB as the default database backend.
  • MuLiFS - Music Library Filesystem creates a filesystem to organise your music files.
  • GoShort - GoShort is a URL shortener written in Golang and BoltDB for persistent key/value storage and for routing it's using high performent HTTPRouter.
  • torrent - Full-featured BitTorrent client package and utilities in Go. BoltDB is a storage backend in development.
  • gopherpit - A web service to manage Go remote import paths with custom domains
  • bolter - Command-line app for viewing BoltDB file in your terminal.
  • btcwallet - A bitcoin wallet.
  • dcrwallet - A wallet for the Decred cryptocurrency.
  • Ironsmith - A simple, script-driven continuous integration (build - > test -> release) tool, with no external dependencies
  • BoltHold - An embeddable NoSQL store for Go types built on BoltDB
  • Ponzu CMS - Headless CMS + automatic JSON API with auto-HTTPS, HTTP/2 Server Push, and flexible server framework.

If you are using Bolt in a project please send a pull request to add it to the list.

Owner
BoltDB
Pure Go key/value database
BoltDB
Comments
  • panic: page 87 already freed

    panic: page 87 already freed

    I've been doing a bit of corruption testing under unexpected power off scenarios (using an external USB drive and disconnecting it midstream) and, unfortunately, I'm fairly consistently hitting unrecoverable database corruption issues due to things such as unreachable unfreed or already freed. I understand write corruption in this scenario is expected, but the database shouldn't be left in an unrecoverable state.

    EDIT: I should also note this is on Windows 7 using all default bolt options.

    I have a 4KiB database that is corrupted from this scenario that I'll keep around to help debug. To get started, I'm providing the following pertinent details:

    • A trace of the panic. NOTE: The trace is from a vendored version of bolt, but there are no code modifications and it is synced through commit ee954308d64186f0fc9b7022b6178977848c17a3.
    • Output of the bolt pages command
    • Output of the bolt check command. NOTE: This command hangs and never completes
    goroutine 12 [running]:
    github.com/btcsuite/bolt.(*freelist).free(0xc08202b680, 0xca2, 0x2af7000)
           .../bolt/freelist.go:118 +0x35c
    github.com/btcsuite/bolt.(*node).spill(0xc0820e2380, 0x0, 0x0)
            .../bolt/node.go:349 +0x280
    github.com/btcsuite/bolt.(*node).spill(0xc0820e2310, 0x0, 0x0)
            .../bolt/node.go:336 +0x134
    github.com/btcsuite/bolt.(*node).spill(0xc0820e22a0, 0x0, 0x0)
            .../bolt/node.go:336 +0x134
    github.com/btcsuite/bolt.(*Bucket).spill(0xc0820dc7c0, 0x0, 0x0)
            .../bolt/bucket.go:536 +0x1cb
    github.com/btcsuite/bolt.(*Bucket).spill(0xc0820dc780, 0x0, 0x0)
            .../bolt/bucket.go:503 +0xac9
    github.com/btcsuite/bolt.(*Bucket).spill(0xc0820e65c8, 0x0, 0x0)
            .../bolt/bucket.go:503 +0xac9
    github.com/btcsuite/bolt.(*Tx).Commit(0xc0820e65b0, 0x0, 0x0)
            .../bolt/tx.go:150 +0x1f5
    ...
    
    $ bolt pages test.db
    
    ID       TYPE       ITEMS  OVRFLW
    ======== ========== ====== ======
    0        meta       0            
    1        meta       0            
    2        leaf       14           
    3        leaf       19           
    4        leaf       19           
    5        leaf       17           
    6        leaf       17           
    7        leaf       27           
    8        leaf       22           
    9        leaf       17           
    10       leaf       17           
    11       leaf       15           
    12       leaf       19           
    13       leaf       26           
    14       leaf       14           
    15       leaf       17           
    16       leaf       25           
    17       leaf       17           
    18       leaf       16           
    19       leaf       22           
    20       leaf       17           
    21       leaf       15           
    22       leaf       27           
    23       leaf       22           
    24       leaf       16           
    25       leaf       16           
    26       leaf       24           
    27       leaf       24           
    28       leaf       27           
    29       leaf       23           
    30       leaf       23           
    31       leaf       18           
    32       leaf       26           
    33       leaf       16           
    34       leaf       22           
    35       leaf       16           
    36       leaf       25           
    37       leaf       19           
    38       leaf       23           
    39       leaf       14           
    40       leaf       25           
    41       leaf       28           
    42       leaf       16           
    43       leaf       16           
    44       leaf       16           
    45       leaf       20           
    46       leaf       18           
    47       leaf       26           
    48       leaf       18           
    49       leaf       24           
    50       leaf       28           
    51       leaf       23           
    52       leaf       21           
    53       leaf       18           
    54       leaf       24           
    55       leaf       20           
    56       leaf       27           
    57       leaf       18           
    58       leaf       18           
    59       leaf       23           
    60       leaf       28           
    61       leaf       26           
    62       leaf       17           
    63       leaf       16           
    64       leaf       26           
    65       leaf       19           
    66       leaf       15           
    67       leaf       14           
    68       leaf       17           
    69       leaf       18           
    70       leaf       27           
    71       leaf       21           
    72       leaf       23           
    73       leaf       16           
    74       leaf       19           
    75       leaf       23           
    76       leaf       19           
    77       leaf       21           
    78       leaf       17           
    79       leaf       24           
    80       leaf       25           
    81       leaf       16           
    82       leaf       27           
    83       leaf       15           
    84       leaf       21           
    85       leaf       22           
    86       leaf       23           
    87       freelist   6            
    88       leaf       16           
    89       leaf       15           
    90       leaf       22           
    91       leaf       19           
    92       leaf       14           
    93       leaf       18           
    94       leaf       19           
    95       leaf       18           
    96       leaf       21           
    97       leaf       23           
    98       leaf       19           
    99       leaf       25           
    100      leaf       19           
    101      leaf       17           
    102      leaf       22           
    103      leaf       16           
    104      leaf       25           
    105      leaf       20           
    106      leaf       17           
    107      leaf       22           
    108      leaf       15           
    109      leaf       16           
    110      leaf       21           
    111      leaf       15           
    112      leaf       14           
    113      leaf       27           
    114      leaf       17           
    115      leaf       15           
    116      leaf       21           
    117      leaf       26           
    118      leaf       26           
    119      leaf       22           
    120      leaf       20           
    121      leaf       16           
    122      leaf       25           
    123      leaf       15           
    124      leaf       18           
    125      leaf       18           
    126      leaf       16           
    127      leaf       28           
    128      leaf       18           
    129      leaf       17           
    130      leaf       19           
    131      leaf       25           
    132      leaf       26           
    133      leaf       24           
    134      leaf       18           
    135      leaf       26           
    136      leaf       20           
    137      leaf       18           
    138      leaf       23           
    139      leaf       26           
    140      leaf       15           
    141      leaf       17           
    142      leaf       24           
    143      leaf       16           
    144      leaf       16           
    145      leaf       22           
    146      leaf       15           
    147      leaf       18           
    148      leaf       26           
    149      leaf       14           
    150      leaf       14           
    151      leaf       14           
    152      leaf       15           
    153      leaf       15           
    154      leaf       17           
    155      leaf       27           
    156      leaf       22           
    157      branch     80           
    158      leaf       15           
    159      leaf       15           
    160      leaf       15           
    161      leaf       15           
    162      leaf       15           
    163      leaf       15           
    164      branch     42           
    165      branch     45           
    166      free                    
    167      branch     4            
    168      free                    
    169      leaf       2            
    170      leaf       1            
    171      free                    
    172      free                    
    173      free                    
    174      free                    
    
    $ bolt check test.db
    page 87: reachable freed
    page 105: multiple references
    page 83: multiple references
    page 87: multiple references
    page 87: reachable freed
    page 97: multiple references
    page 97: multiple references
    page 97: multiple references
    page 157: multiple references
    page 21: multiple references
    page 20: multiple references
    page 87: multiple references
    page 87: reachable freed
    page 27: multiple references
    page 35: multiple references
    page 129: multiple references
    page 132: multiple references
    page 156: multiple references
    page 82: multiple references
    page 80: multiple references
    page 139: multiple references
    page 60: multiple references
    page 121: multiple references
    page 153: multiple references
    page 30: multiple references
    page 7: multiple references
    page 112: multiple references
    page 9: multiple references
    page 127: multiple references
    page 118: multiple references
    page 74: multiple references
    page 106: multiple references
    page 15: multiple references
    page 78: multiple references
    page 65: multiple references
    page 39: multiple references
    page 159: multiple references
    page 63: multiple references
    page 124: multiple references
    page 107: multiple references
    page 3: multiple references
    page 16: multiple references
    page 67: multiple references
    page 163: multiple references
    page 29: multiple references
    page 92: multiple references
    page 105: multiple references
    page 105: multiple references
    page 13: multiple references
    page 85: multiple references
    page 109: multiple references
    page 137: multiple references
    page 48: multiple references
    page 138: multiple references
    page 75: multiple references
    page 28: multiple references
    page 83: multiple references
    page 55: multiple references
    page 32: multiple references
    page 34: multiple references
    page 84: multiple references
    page 51: multiple references
    page 53: multiple references
    page 126: multiple references
    page 19: multiple references
    page 23: multiple references
    page 95: multiple references
    page 96: multiple references
    page 5: multiple references
    page 37: multiple references
    page 130: multiple references
    page 149: multiple references
    page 161: multiple references
    page 72: multiple references
    page 73: multiple references
    page 116: multiple references
    page 113: multiple references
    page 18: multiple references
    page 125: multiple references
    page 146: multiple references
    page 102: multiple references
    page 134: multiple references
    page 76: multiple references
    page 114: multiple references
    page 62: multiple references
    page 66: multiple references
    page 103: multiple references
    page 97: multiple references
    page 83: multiple references
    page 87: multiple references
    page 87: reachable freed
    **Hangs here and never completes** 
    
  • Add transaction batching

    Add transaction batching

    Batcher makes it easy to make lots of small transactions with significantly better performance. Batcher.Update combines multiple Update calls into a single disk transaction, managing errors smartly.

    I'm aware of https://github.com/boltdb/coalescer. I wrote the original version of this before you pushed coalescer, it just took a while for me to circle back to cleaning it up. This code doesn't leave a goroutine around, and is a few percent faster ;) But the real difference to coalescer is this: with this code, callers don't need to write a for loop around ErrRollback. I pull failing updaters out of the transaction, and retry the others automatically.

    I'd really like to see this as part of bolt, and I feel like having a good batcher mechanism would help significantly with performance complaints like https://github.com/boltdb/bolt/issues/244 . If this looks promising, it could even sit alongside DB.Update as DB.Batch -- or even become DB.Update (though beware the "runs multiple times" behavior difference).

    With this in place, I see no reason why any of my apps would ever again bother doing non-batched transactions.

    Let me know what you think. If you say no, this'll live as tv42/semiauto (as opposed to bolt action..)

  • Writing to a >8gb db file

    Writing to a >8gb db file

    I'm using bolt (master) in production as a persistent store for binary files (50 to 300kb). I'm finding that now that my database has grown to 8gb (and growing), writes to the database are very slow, between 4 to 13seconds. I know batch writes are much better, but I can't do that in this case because it's an isolated operation, unless I keep them in memory somewhere first then every 30 seconds sync to bolt, which seems a bit crazy.

    Does bolt have any restrictions on db size? .. I've also wondered about how mmap'ed databases work and manage memory utilization while staying friendly with other processes on the system.

    Any suggestions?

    Does bolt write the entire db to disk each time when doing a db.Update/bucket.Put?

  • Avoid trashing page cache on Tx.Copy().

    Avoid trashing page cache on Tx.Copy().

    This commit change the database copy to use O_DIRECT so that the Linux page cache is not trashed during a backup. This is only available on Linux.

    *Note: The odirect name is used instead of o_direct to avoid golint warnings.

    /cc @snormore @mkobetic

  • Compaction

    Compaction

    Hello,

    I was just wondering what Bolt is planning for database compaction? I'm not sure if this is something designed into Lmdb, but I was just running some tests, putting a lot of data into a boltdb, then removing the keys, then adding some more (ones with the same name and others), and the database just kept growing.

    Btw, great work on Bolt! It's very clean and happy to see it so active.

    As a comparison, leveldb does compaction in a background thread that gets triggered at some event (not sure at which point).

  • Use tx.meta during Tx.WriteTo()

    Use tx.meta during Tx.WriteTo()

    Overview

    This commit changes Tx.WriteTo() to use the transaction's in-memory meta page instead of copying from the disk. This is needed because the transaction uses the size from its meta page but writes the current meta page on disk which may have allocated additional pages since the transaction started.

    Fixes #513

    /cc @xiang90

  • solaris: fix issues with mmap, munmap, madvise and flock

    solaris: fix issues with mmap, munmap, madvise and flock

    • Many thanks to @akolb1 for the FcntlFlock based flock implementation.
    • Requires recent golang.org/x/sys/unix for fixed mmap,munmap,madvise.
    • Fixes #274 and fixes #305.
    • Addresses part of #345.
    • Solaris fcntl locks don't support intra-process locking

    Three unit tests were failing, now are skipped on solaris only:

    seed: 68195
    quick settings: count=5, items=1000, ksize=1024, vsize=1024
    00010000000000000000000000000000
    db_test.go:57: 
    
    --- FAIL: TestOpen_Timeout (0.00s)
    db_test.go:86: 
    
    --- FAIL: TestOpen_Wait (0.00s)
    db_test.go:257: 
    
    --- FAIL: TestOpen_ReadOnly (0.00s)
    FAIL
    exit status 1
    FAIL    github.com/boltdb/bolt  354.617s
    
    • FcntlFlock treats separate locks on separate file descriptors differently then Flock.
    • Flock semantics are preferrable for multithreaded servers.
    • No simple replacement currently available on Solaris.

    References:

    • http://infinitesque.net/articles/2010/on%20Python%20flock/
    • https://github.com/jperkin/www.perkin.org.uk/blob/master/_posts/2013-01-08-solaris-portability-flock.markdown
    • https://gist.github.com/MerlinDMC/3197f4d13f8145c457e4
  • "unexpected fault address" on commit

    I see this, intermittently.

    go version go1.3 darwin/amd64

    Bolt 42b4cae

    unexpected fault address 0x19441ca
    fatal error: fault
    [signal 0xb code=0x1 addr=0x19441ca pc=0x45392]
    
    goroutine 33 [running]:
    runtime.throw(0x7fb7f3)
            /usr/local/go/src/pkg/runtime/panic.c:520 +0x69 fp=0x1e1eed8 sp=0x1e1eec0
    runtime.sigpanic()
            /usr/local/go/src/pkg/runtime/os_darwin.c:457 +0x13f fp=0x1e1eef0 sp=0x1e1eed8
    runtime.memmove(0xc2087f224f, 0x19441ca, 0x80)
            /usr/local/go/src/pkg/runtime/memmove_amd64.s:157 +0x192 fp=0x1e1eef8 sp=0x1e1eef0
    github.com/boltdb/bolt.(*node).write(0xc20875e230, 0xc2087f2000)
            /Users/jb/src/github.com/boltdb/bolt/node.go:202 +0x361 fp=0x1e1f088 sp=0x1e1eef8
    github.com/boltdb/bolt.(*node).spill(0xc208789340, 0x0, 0x0)
            /Users/jb/src/github.com/boltdb/bolt/node.go:335 +0x3f6 fp=0x1e1f240 sp=0x1e1f088
    github.com/boltdb/bolt.(*Bucket).spill(0xc20878e480, 0x0, 0x0)
            /Users/jb/src/github.com/boltdb/bolt/bucket.go:543 +0x1c5 fp=0x1e1f438 sp=0x1e1f240
    github.com/boltdb/bolt.(*Bucket).spill(0xc20878e330, 0x0, 0x0)
            /Users/jb/src/github.com/boltdb/bolt/bucket.go:514 +0x81e fp=0x1e1f630 sp=0x1e1f438
    github.com/boltdb/bolt.(*Bucket).spill(0xc208028aa8, 0x0, 0x0)
            /Users/jb/src/github.com/boltdb/bolt/bucket.go:514 +0x81e fp=0x1e1f828 sp=0x1e1f630
    github.com/boltdb/bolt.(*Tx).Commit(0xc208028a90, 0x0, 0x0)
            /Users/jb/src/github.com/boltdb/bolt/tx.go:159 +0x1df fp=0x1e1f960 sp=0x1e1f828
    github.com/boltdb/bolt.(*DB).Update(0xc2081b2000, 0xc2084e5fc0, 0x0, 0x0)
            /Users/jb/src/github.com/boltdb/bolt/db.go:468 +0x103 fp=0x1e1f9b8 sp=0x1e1f960
    github.com/calmh/syncthing/files.(*Set).Replace(0xc208024d80, 0x1, 0xc20870e000, 0x35a, 0x35a)
            /Users/jb/src/github.com/calmh/syncthing/files/set.go:59 +0x357 fp=0x1e1fb30 sp=0x1e1f9b8
    github.com/calmh/syncthing/model.(*Model).Index(0xc2081e0100, 0xc20803c380, 0x34, 0xc208402cd4, 0x3, 0xc208604000, 0x35a, 0x35a)
            /Users/jb/src/github.com/calmh/syncthing/model/model.go:295 +0x82e fp=0x1e1fe00 sp=0x1e1fb30
    github.com/calmh/syncthing/protocol.nativeModel.Index(0xa27998, 0xc2081e0100, 0xc20803c380, 0x34, 0xc208402cd4, 0x3, 0xc208604000, 0x35a, 0x35a)
            /Users/jb/src/github.com/calmh/syncthing/protocol/nativemodel_darwin.go:21 +0x13b fp=0x1e1fe70 sp=0x1e1fe00
    github.com/calmh/syncthing/protocol.(*nativeModel).Index(0xc2080008c0, 0xc20803c380, 0x34, 0xc208402cd4, 0x3, 0xc208604000, 0x35a, 0x35a)
            <autogenerated>:31 +0xe1 fp=0x1e1fec0 sp=0x1e1fe70
    github.com/calmh/syncthing/protocol.(*rawConnection).indexSerializerLoop(0xc2080a0b00)
            /Users/jb/src/github.com/calmh/syncthing/protocol/protocol.go:310 +0x16d fp=0x1e1ffa0 sp=0x1e1fec0
    runtime.goexit()
            /usr/local/go/src/pkg/runtime/proc.c:1445 fp=0x1e1ffa8 sp=0x1e1ffa0
    created by github.com/calmh/syncthing/protocol.NewConnection
            /Users/jb/src/github.com/calmh/syncthing/protocol/protocol.go:130 +0x58e
    
  • request: compaction api call (as bolt disk space consumption grows without bound)

    request: compaction api call (as bolt disk space consumption grows without bound)

    I don't mind that a bolt file takes up 3-4x the space on disk as the contained value, but I had a bug where my usually 500KB blob ballooned to 7MB before I caught the leak.

    Now my boltdb takes up 21MB instead of the 1.5MB, which costs alot as I ship the file around.

    Feature request: an api call to compact an existing boltdb file so that it takes up only the pages that are actually in use. I suppose I could just delete and re-write the file from scratch, but that introduces a bunch of race conditions that could have the file disappearing if the process is killed at the wrong moment.

    Would this be difficult to implement?

  • Windows: default options limit database to 64MB.

    Windows: default options limit database to 64MB.

    Environment:

    OS: Windows 7 Go: go version go1.5.1 windows/amd64

    Steps to reproduce:

    1. Open a new database with nil options.
    2. Create a bucket
    3. Repeatedly put new key/value pairs into the bucket.

    What happens:

    The database file quickly grows to 64MB and then bolt returns an error:

    file resize error: truncate test.db: The requested operation cannot be performed on a file with a user-mapped section open.
    

    Workarounds: The error does not occur if the database is opened with the NoGrowSync option enabled.

    What was expected: The default options should work better on all platforms.

  • Faster freelist updates

    Faster freelist updates

    Since freelist is kept sorted we can update it by merging the sorted pgid lists rather than appending and resorting the whole list. We have DBs in production reaching 10M pages in the freelist and about 20K pages in the pending lists. The benchmark is an attempt to prove that at these sizes the merge can be nearly 100x faster than the append/sort approach.

    Append/Sort:

    Martins-MacBook-Pro-5:bolt martin$ go test -v -bench=FreelistRelease -run None -short
    seed: 56446
    quick settings: count=5, items=1000, ksize=1024, vsize=1024
    PASS
    Benchmark_FreelistRelease10K        1000       1301767 ns/op
    Benchmark_FreelistRelease100K        100      15634815 ns/op
    Benchmark_FreelistRelease1000K        10     182048520 ns/op
    Benchmark_FreelistRelease10000K        1    2187250681 ns/op
    ok      github.com/boltdb/bolt  11.825s
    

    Merge:

    Martins-MacBook-Pro-5:bolt martin$ go test -v -bench=FreelistRelease -run None -short
    seed: 27571
    quick settings: count=5, items=1000, ksize=1024, vsize=1024
    PASS
    Benchmark_FreelistRelease10K       30000         50734 ns/op
    Benchmark_FreelistRelease100K       5000        210540 ns/op
    Benchmark_FreelistRelease1000K       500       3132026 ns/op
    Benchmark_FreelistRelease10000K       30      37413055 ns/op
    ok      github.com/boltdb/bolt  19.631s
    

    This isn't a theoretical issue, we have a production scenario with large freelists where freelist sorting is taking nearly 90% of CPU time. I suspect that slowness of the freelist updates could significantly contribute to explosive freelist growth as well if it extends transaction duration by seconds.

    @benbjohnson, I'm going to test this branch in some of our production scenarios next, but I would love your thoughts on this change, in case I'm missing something. Thanks!

    /cc @snormore

  • Cursor.Last() returns nil for non-empty bucket

    Cursor.Last() returns nil for non-empty bucket

    Sometimes after a bunch of deletes Last() for new cursor returns nil for non-empty bucket. Code to reproduce is here.

    May be it's enough to put n.rebalance() at the end of (n *node) del(key []byte) to fix this.

    Hi @benbjohnson, Could you please confirm the solution?

    I am not sure about invariants that should provide cursor.Delete(). Is it ok to get an empty leaf page after delete a key?

    Thank you.

  • permission denied in user home directory when open boltdb path

    permission denied in user home directory when open boltdb path

    I use the following code to create/open a bolt database file under the user home directory on mac:

    
    curUser, err := user.Current()
    if err != nil {
      panic(err)
    } else {
     dataPath := filepath.Join(curUser.HomeDir, dataPath)
     handle, err := bolt.Open(dataPath, 0644, defaultBoltOptions)
    if err != nil {
     log(err) // this tells me permission denied error 
    }
    }
    
    

    I wonder why this happened and how to fix it? I run my go code go run main.go as a mac user.

  • Tons of compilation errors

    Tons of compilation errors

    github.com/boltdb/bolt

    ../../../boltdb/bolt/db.go:113:11: error: reference to undefined identifier ‘sync.Pool’ pagePool sync.Pool ^ ../../../boltdb/bolt/db.go:223:21: error: reference to undefined identifier ‘sync.Pool’ db.pagePool = sync.Pool{ ^ ../../../boltdb/bolt/db.go:223:25: error: expected ‘;’ or ‘}’ or newline db.pagePool = sync.Pool{ ^ ../../../boltdb/bolt/db.go:224:27: error: expected ‘;’ or newline after top level declaration New: func() interface{} { ^ ../../../boltdb/bolt/db.go:226:3: error: expected declaration }, ^ ../../../boltdb/bolt/db.go:230:2: error: expected declaration if err := db.mmap(options.InitialMmapSize); err != nil { ^ ../../../boltdb/bolt/db.go:230:46: error: expected declaration if err := db.mmap(options.InitialMmapSize); err != nil { ^ ../../../boltdb/bolt/db.go:232:3: error: expected declaration return nil, err ^ ../../../boltdb/bolt/db.go:233:2: error: expected declaration } ^ ../../../boltdb/bolt/db.go:236:2: error: expected declaration db.freelist = newFreelist() ^ ../../../boltdb/bolt/db.go:237:2: error: expected declaration db.freelist.read(db.page(db.meta().freelist)) ^ ../../../boltdb/bolt/db.go:240:2: error: expected declaration return db, nil ^ ../../../boltdb/bolt/db.go:241:1: error: expected declaration } ^ ../../../boltdb/bolt/db.go:224:25: error: missing return at end of function New: func() interface{} { ^ ../../../boltdb/bolt/db.go:176:3: error: too many values in return statement return nil, err ^ ../../../boltdb/bolt/db.go:188:3: error: too many values in return statement return nil, err ^ ../../../boltdb/bolt/db.go:196:3: error: too many values in return statement return nil, err ^ ../../../boltdb/bolt/db.go:200:4: error: too many values in return statement return nil, err ^ ../../../boltdb/bolt/db.go:461:3: error: not enough arguments to return return db.beginRWTx() ^ ../../../boltdb/bolt/db.go:463:2: error: not enough arguments to return return db.beginTx() ^ ../../../boltdb/bolt/db.go:489:20: error: argument 1 must be a slice db.txs = append(db.txs, t) ^ ../../../boltdb/bolt/db.go:532:2: error: range clause must have array, slice, string, map, or channel type for _, t := range db.txs { ^ ../../../boltdb/bolt/db.go:553:2: error: range clause must have array, slice, string, map, or channel type for i, t := range db.txs { ^ ../../../boltdb/bolt/db.go:671:34: error: argument 1 must be a slice db.batch.calls = append(db.batch.calls, call{fn: fn, err: errCh}) ^ ../../../boltdb/bolt/db.go:739:3: error: range clause must have array, slice, string, map, or channel type for _, c := range b.calls { ^ ../../../boltdb/bolt/db.go:718:4: error: range clause must have array, slice, string, map, or channel type for i, c := range b.calls { ^ ../../../boltdb/bolt/db.go:833:27: error: non-integer len argument to make buf = make([]byte, count*db.pageSize) ^ ../../../boltdb/bolt/tx.go:71:2: error: return with value in function with no return type return tx.db ^ ../../../boltdb/bolt/tx.go:108:2: error: not enough arguments to return return tx.root.CreateBucket(name) ^ ../../../boltdb/bolt/tx.go:115:2: error: not enough arguments to return return tx.root.CreateBucketIfNotExists(name) ^ ../../../boltdb/bolt/tx.go:138:31: error: argument 1 must be a slice tx.commitHandlers = append(tx.commitHandlers, fn) ^ ../../../boltdb/bolt/tx.go:177:47: error: division by zero p, err := tx.allocate((tx.db.freelist.size() / tx.db.pageSize) + 1) ^ ../../../boltdb/bolt/tx.go:209:4: error: expected channel err, ok := <-ch ^ ../../../boltdb/bolt/tx.go:231:2: error: range clause must have array, slice, string, map, or channel type for _, fn := range tx.commitHandlers { ^ ../../../boltdb/bolt/tx.go:477:2: error: range clause must have array, slice, string, map, or channel type for _, p := range tx.pages { ^ ../../../boltdb/bolt/tx.go:574:6: error: expected map index on right hand side if p, ok := tx.pages[id]; ok { ^ ../../../boltdb/bolt/tx.go:574:6: error: invalid tuple definition ../../../boltdb/bolt/db.go:532:6: error: invalid type for range clause for _, t := range db.txs { ^ ../../../boltdb/bolt/db.go:553:6: error: invalid type for range clause for i, t := range db.txs { ^ ../../../boltdb/bolt/db.go:553:6: error: invalid type for range clause ../../../boltdb/bolt/db.go:739:7: error: invalid type for range clause for _, c := range b.calls { ^ ../../../boltdb/bolt/db.go:718:8: error: invalid type for range clause for i, c := range b.calls { ^ ../../../boltdb/bolt/db.go:718:8: error: invalid type for range clause ../../../boltdb/bolt/tx.go:231:6: error: invalid type for range clause for _, fn := range tx.commitHandlers { ^ ../../../boltdb/bolt/tx.go:477:6: error: invalid type for range clause for _, p := range tx.pages { ^

  • page already freed on certain builds

    page already freed on certain builds

    Sorry to bring up another issue with this headline... I already checked the other two issues and do not have the impression that they match my case.

    Here my error

    panic: page 4 already freed

    goroutine 416 [running]: .../vendor/github.com/boltdb/bolt.(*freelist).free(0x202f7ec0, 0x5, 0x0, 0x3014000) /ext-go/1/src/.../vendor/github.com/boltdb/bolt/freelist.go:121 +0x263 .../vendor/github.com/boltdb/bolt.(*node).spill(0x20356d40, 0x2064bc64, 0x6) /ext-go/1/src/.../vendor/github.com/boltdb/bolt/node.go:363 +0x19f .../vendor/github.com/boltdb/bolt.(*Bucket).spill(0x2032eb0c, 0xe, 0x38554104) /ext-go/1/src/.../vendor/github.com/boltdb/bolt/bucket.go:570 +0x13d .../vendor/github.com/boltdb/bolt.(*Tx).Commit(0x2032eb00, 0x0, 0x0) /ext-go/1/src/.../vendor/github.com/boltdb/bolt/tx.go:163 +0xff .../vendor/github.com/boltdb/bolt.(*DB).Update(0x20244fc0, 0x2056de80, 0x0, 0x0) /ext-go/1/src/.../vendor/github.com/boltdb/bolt/db.go:605 +0xc9

    Here my Code snippet:

    // Add adds a value to the db using NextSequence to autoincrement the key
    func (s *DB) Add(tableName string, value []byte) (uint64, error) {
    	var id uint64
    	err := s.Conn.Update(func(tx *bolt.Tx) error {
    		bt, err := tx.CreateBucketIfNotExists([]byte(tableName))
    		if err != nil {
    			return err
    		}
    
    		id, err = bt.NextSequence()
    		if err != nil {
    			return err
    		}
    
    		return bt.Put(itob(id), value)
    	})
    	return id, errors.WithStack(err)
    }
    
    // itob returns an 8-byte big endian representation of v.
    func itob(i uint64) []byte {
    	b := make([]byte, 8)
    	binary.BigEndian.PutUint64(b, i)
    	return b
    }
    

    The error occurs the second time this code is executed (every time).

    The code runs fine on MacOS (amd64) and Windows (386) without cgo. Disabling cgo disables a few packages that use windows dlls. When compiling the final version with cgo (and dlls) using xgo --targets=windows/386 . the resulting windows executable will show the above issue.

    I added some logs to the boltdb code and checked which pages are being freed. On every call the same pages are being freed. The node.pgid on node.go:363 is also the same on every call (calls are several secods apart). Why?

    Since in one of the other issues on this topic there is talk about race conditions I made sure the entire connection is used by a single goroutine using the action pattern. This did not resolve the issue either:

    // Add adds a value to the db using NextSequence to autoincrement the key
    func (s *DB) Add(tableName string, value []byte) (uint64, error) {
    	// added this to make sure there is no other goroutine using the byte array at the same time
    	var valCopy = make([]byte, 0, len(value))
    	valCopy = append(valCopy, value...)
    
    	var id uint64
    	chErr := make(chan error)
    	s.Run(s.Action)
    	s.Action <- func() {
    		err := s.Conn.Update(func(tx *bolt.Tx) error {
    			bt, err := tx.CreateBucketIfNotExists([]byte(tableName))
    			if err != nil {
    				return err
    			}
    
    			id, err = bt.NextSequence()
    			if err != nil {
    				return err
    			}
    
    			return bt.Put(itob(id), valCopy)
    		})
    		chErr <- err
    	}
    	err := <-chErr
    	return id, errors.WithStack(err)
    }
    

    Anyone any idea on how to solve this issue?

  • Meta2 make DBFile invalid

    Meta2 make DBFile invalid

    Write-Transactions update Meta1 on commit, and Meta2 only be updated at Backup (invoke tx.WriteTo);

    So, please consider this case:

    1. Assume my DB's freelist is stored in Page34, and I backup it now; Then the page id of freelist in meta2 is 34 now;
    2. Then I do some write-transactions on the new DB file, obviously the Page34 maybe reallocated as a branch page or a leaf page;
    3. If meta1 is written to broken now, then meta2 is used, and the Page34 is regard as a freelist page again;
Distributed cache and in-memory key/value data store. It can be used both as an embedded Go library and as a language-independent service.

Olric Distributed cache and in-memory key/value data store. It can be used both as an embedded Go library and as a language-independent service. With

Jan 4, 2023
Pogreb is an embedded key-value store for read-heavy workloads written in Go.
Pogreb is an embedded key-value store for read-heavy workloads written in Go.

Embedded key-value store for read-heavy workloads written in Go

Dec 29, 2022
rosedb is a fast, stable and embedded key-value (k-v) storage engine based on bitcask.
rosedb is a fast, stable and embedded key-value (k-v) storage engine based on bitcask.

rosedb is a fast, stable and embedded key-value (k-v) storage engine based on bitcask. Its on-disk files are organized as WAL(Write Ahead Log) in LSM trees, optimizing for write throughput.

Dec 28, 2022
A key-value db api with multiple storage engines and key generation
A key-value db api with multiple storage engines and key generation

Jet is a deadly-simple key-value api. The main goals of this project are : Making a simple KV tool for our other projects. Learn tests writing and git

Apr 5, 2022
LevelDB key/value database in Go.

This is an implementation of the LevelDB key/value database in the Go programming language. Installation go get github.com/syndtr/goleveldb/leveldb R

Jan 9, 2023
RocksDB/LevelDB inspired key-value database in Go

Pebble Nightly benchmarks Pebble is a LevelDB/RocksDB inspired key-value store focused on performance and internal usage by CockroachDB. Pebble inheri

Dec 29, 2022
Key-value database stored in memory with option of persistence
Key-value database stored in memory with option of persistence

Easy and intuitive command line tool allows you to spin up a database avaliable from web or locally in a few seconds. Server can be run over a custom TCP protocol or over HTTP.

Aug 1, 2022
ZedisDB - a key-value memory database written in Go

ZedisDB - a key-value memory database written in Go

Sep 4, 2022
Simple Distributed key-value database (in-memory/disk) written with Golang.

Kallbaz DB Simple Distributed key-value store (in-memory/disk) written with Golang. Installation go get github.com/msam1r/kallbaz-db Usage API // Get

Jan 18, 2022
FlashDB is an embeddable, in-memory key/value database in Go
FlashDB is an embeddable, in-memory key/value database in Go

FlashDB is an embeddable, in-memory key/value database in Go (with Redis like commands and super easy to read)

Dec 28, 2022
A HTTP RESTful type kv database which embedded some frequently-used mid-ware.

A HTTP RESTful type kv database which embedded some frequently-used mid-ware.

Apr 4, 2022
A disk-backed key-value store.

What is diskv? Diskv (disk-vee) is a simple, persistent key-value store written in the Go language. It starts with an incredibly simple API for storin

Jan 1, 2023
Distributed reliable key-value store for the most critical data of a distributed system

etcd Note: The master branch may be in an unstable or even broken state during development. Please use releases instead of the master branch in order

Dec 28, 2022
Simple, ordered, key-value persistence library for the Go Language

gkvlite gkvlite is a simple, ordered, ACID, key-value persistence library for Go. Overview gkvlite is a library that provides a simple key-value persi

Dec 21, 2022
Distributed, fault-tolerant key-value storage written in go.
Distributed, fault-tolerant key-value storage written in go.

A simple, distributed, fault-tolerant key-value storage inspired by Redis. It uses Raft protocotol as consensus algorithm. It supports the following data structures: String, Bitmap, Map, List.

Jan 3, 2023
a persistent real-time key-value store, with the same redis protocol with powerful features
a persistent real-time key-value store, with the same redis protocol with powerful features

a fast NoSQL DB, that uses the same RESP protocol and capable to store terabytes of data, also it integrates with your mobile/web apps to add real-time features, soon you can use it as a document store cause it should become a multi-model db. Redix is used in production, you can use it in your apps with no worries.

Dec 25, 2022
GhostDB is a distributed, in-memory, general purpose key-value data store that delivers microsecond performance at any scale.
GhostDB is a distributed, in-memory, general purpose key-value data store that delivers microsecond performance at any scale.

GhostDB is designed to speed up dynamic database or API driven websites by storing data in RAM in order to reduce the number of times an external data source such as a database or API must be read. GhostDB provides a very large hash table that is distributed across multiple machines and stores large numbers of key-value pairs within the hash table.

Jan 6, 2023
HA LDAP based key/value solution for projects configuration storing with multi master replication support

Recon is the simple solution for storing configs of you application. There are no specified instruments, no specified data protocols. For the full power of Recon you only need curl.

Jun 15, 2022
Fault tolerant, sharded key value storage written in GoLang
Fault tolerant, sharded key value storage written in GoLang

Ravel is a sharded, fault-tolerant key-value store built using BadgerDB and hashicorp/raft. You can shard your data across multiple clusters with mult

Nov 1, 2022