An embedded key/value database for Go.

bbolt

Go Report Card Coverage Build Status Travis Godoc Releases LICENSE

bbolt is a fork of Ben Johnson's Bolt key/value store. The purpose of this fork is to provide the Go community with an active maintenance and development target for Bolt; the goal is improved reliability and stability. bbolt includes bug fixes, performance enhancements, and features not found in Bolt while preserving backwards compatibility with the Bolt API.

Bolt is a pure Go key/value store inspired by Howard Chu's LMDB project. The goal of the project is to provide a simple, fast, and reliable database for projects that don't require a full database server such as Postgres or MySQL.

Since Bolt is meant to be used as such a low-level piece of functionality, simplicity is key. The API will be small and only focus on getting values and setting values. That's it.

Project Status

Bolt is stable, the API is fixed, and the file format is fixed. Full unit test coverage and randomized black box testing are used to ensure database consistency and thread safety. Bolt is currently used in high-load production environments serving databases as large as 1TB. Many companies such as Shopify and Heroku use Bolt-backed services every day.

Project versioning

bbolt uses semantic versioning. API should not change between patch and minor releases. New minor versions may add additional features to the API.

Table of Contents

Getting Started

Installing

To start using Bolt, install Go and run go get:

$ go get go.etcd.io/bbolt/...

This will retrieve the library and install the bolt command line utility into your $GOBIN path.

Importing bbolt

To use bbolt as an embedded key-value store, import as:

import bolt "go.etcd.io/bbolt"

db, err := bolt.Open(path, 0666, nil)
if err != nil {
  return err
}
defer db.Close()

Opening a database

The top-level object in Bolt is a DB. It is represented as a single file on your disk and represents a consistent snapshot of your data.

To open your database, simply use the bolt.Open() function:

package main

import (
	"log"

	bolt "go.etcd.io/bbolt"
)

func main() {
	// Open the my.db data file in your current directory.
	// It will be created if it doesn't exist.
	db, err := bolt.Open("my.db", 0600, nil)
	if err != nil {
		log.Fatal(err)
	}
	defer db.Close()

	...
}

Please note that Bolt obtains a file lock on the data file so multiple processes cannot open the same database at the same time. Opening an already open Bolt database will cause it to hang until the other process closes it. To prevent an indefinite wait you can pass a timeout option to the Open() function:

db, err := bolt.Open("my.db", 0600, &bolt.Options{Timeout: 1 * time.Second})

Transactions

Bolt allows only one read-write transaction at a time but allows as many read-only transactions as you want at a time. Each transaction has a consistent view of the data as it existed when the transaction started.

Individual transactions and all objects created from them (e.g. buckets, keys) are not thread safe. To work with data in multiple goroutines you must start a transaction for each one or use locking to ensure only one goroutine accesses a transaction at a time. Creating transaction from the DB is thread safe.

Transactions should not depend on one another and generally shouldn't be opened simultaneously in the same goroutine. This can cause a deadlock as the read-write transaction needs to periodically re-map the data file but it cannot do so while any read-only transaction is open. Even a nested read-only transaction can cause a deadlock, as the child transaction can block the parent transaction from releasing its resources.

Read-write transactions

To start a read-write transaction, you can use the DB.Update() function:

err := db.Update(func(tx *bolt.Tx) error {
	...
	return nil
})

Inside the closure, you have a consistent view of the database. You commit the transaction by returning nil at the end. You can also rollback the transaction at any point by returning an error. All database operations are allowed inside a read-write transaction.

Always check the return error as it will report any disk failures that can cause your transaction to not complete. If you return an error within your closure it will be passed through.

Read-only transactions

To start a read-only transaction, you can use the DB.View() function:

err := db.View(func(tx *bolt.Tx) error {
	...
	return nil
})

You also get a consistent view of the database within this closure, however, no mutating operations are allowed within a read-only transaction. You can only retrieve buckets, retrieve values, and copy the database within a read-only transaction.

Batch read-write transactions

Each DB.Update() waits for disk to commit the writes. This overhead can be minimized by combining multiple updates with the DB.Batch() function:

err := db.Batch(func(tx *bolt.Tx) error {
	...
	return nil
})

Concurrent Batch calls are opportunistically combined into larger transactions. Batch is only useful when there are multiple goroutines calling it.

The trade-off is that Batch can call the given function multiple times, if parts of the transaction fail. The function must be idempotent and side effects must take effect only after a successful return from DB.Batch().

For example: don't display messages from inside the function, instead set variables in the enclosing scope:

var id uint64
err := db.Batch(func(tx *bolt.Tx) error {
	// Find last key in bucket, decode as bigendian uint64, increment
	// by one, encode back to []byte, and add new key.
	...
	id = newValue
	return nil
})
if err != nil {
	return ...
}
fmt.Println("Allocated ID %d", id)

Managing transactions manually

The DB.View() and DB.Update() functions are wrappers around the DB.Begin() function. These helper functions will start the transaction, execute a function, and then safely close your transaction if an error is returned. This is the recommended way to use Bolt transactions.

However, sometimes you may want to manually start and end your transactions. You can use the DB.Begin() function directly but please be sure to close the transaction.

// Start a writable transaction.
tx, err := db.Begin(true)
if err != nil {
    return err
}
defer tx.Rollback()

// Use the transaction...
_, err := tx.CreateBucket([]byte("MyBucket"))
if err != nil {
    return err
}

// Commit the transaction and check for error.
if err := tx.Commit(); err != nil {
    return err
}

The first argument to DB.Begin() is a boolean stating if the transaction should be writable.

Using buckets

Buckets are collections of key/value pairs within the database. All keys in a bucket must be unique. You can create a bucket using the Tx.CreateBucket() function:

db.Update(func(tx *bolt.Tx) error {
	b, err := tx.CreateBucket([]byte("MyBucket"))
	if err != nil {
		return fmt.Errorf("create bucket: %s", err)
	}
	return nil
})

You can also create a bucket only if it doesn't exist by using the Tx.CreateBucketIfNotExists() function. It's a common pattern to call this function for all your top-level buckets after you open your database so you can guarantee that they exist for future transactions.

To delete a bucket, simply call the Tx.DeleteBucket() function.

Using key/value pairs

To save a key/value pair to a bucket, use the Bucket.Put() function:

db.Update(func(tx *bolt.Tx) error {
	b := tx.Bucket([]byte("MyBucket"))
	err := b.Put([]byte("answer"), []byte("42"))
	return err
})

This will set the value of the "answer" key to "42" in the MyBucket bucket. To retrieve this value, we can use the Bucket.Get() function:

db.View(func(tx *bolt.Tx) error {
	b := tx.Bucket([]byte("MyBucket"))
	v := b.Get([]byte("answer"))
	fmt.Printf("The answer is: %s\n", v)
	return nil
})

The Get() function does not return an error because its operation is guaranteed to work (unless there is some kind of system failure). If the key exists then it will return its byte slice value. If it doesn't exist then it will return nil. It's important to note that you can have a zero-length value set to a key which is different than the key not existing.

Use the Bucket.Delete() function to delete a key from the bucket.

Please note that values returned from Get() are only valid while the transaction is open. If you need to use a value outside of the transaction then you must use copy() to copy it to another byte slice.

Autoincrementing integer for the bucket

By using the NextSequence() function, you can let Bolt determine a sequence which can be used as the unique identifier for your key/value pairs. See the example below.

// CreateUser saves u to the store. The new user ID is set on u once the data is persisted.
func (s *Store) CreateUser(u *User) error {
    return s.db.Update(func(tx *bolt.Tx) error {
        // Retrieve the users bucket.
        // This should be created when the DB is first opened.
        b := tx.Bucket([]byte("users"))

        // Generate ID for the user.
        // This returns an error only if the Tx is closed or not writeable.
        // That can't happen in an Update() call so I ignore the error check.
        id, _ := b.NextSequence()
        u.ID = int(id)

        // Marshal user data into bytes.
        buf, err := json.Marshal(u)
        if err != nil {
            return err
        }

        // Persist bytes to users bucket.
        return b.Put(itob(u.ID), buf)
    })
}

// itob returns an 8-byte big endian representation of v.
func itob(v int) []byte {
    b := make([]byte, 8)
    binary.BigEndian.PutUint64(b, uint64(v))
    return b
}

type User struct {
    ID int
    ...
}

Iterating over keys

Bolt stores its keys in byte-sorted order within a bucket. This makes sequential iteration over these keys extremely fast. To iterate over keys we'll use a Cursor:

db.View(func(tx *bolt.Tx) error {
	// Assume bucket exists and has keys
	b := tx.Bucket([]byte("MyBucket"))

	c := b.Cursor()

	for k, v := c.First(); k != nil; k, v = c.Next() {
		fmt.Printf("key=%s, value=%s\n", k, v)
	}

	return nil
})

The cursor allows you to move to a specific point in the list of keys and move forward or backward through the keys one at a time.

The following functions are available on the cursor:

First()  Move to the first key.
Last()   Move to the last key.
Seek()   Move to a specific key.
Next()   Move to the next key.
Prev()   Move to the previous key.

Each of those functions has a return signature of (key []byte, value []byte). When you have iterated to the end of the cursor then Next() will return a nil key. You must seek to a position using First(), Last(), or Seek() before calling Next() or Prev(). If you do not seek to a position then these functions will return a nil key.

During iteration, if the key is non-nil but the value is nil, that means the key refers to a bucket rather than a value. Use Bucket.Bucket() to access the sub-bucket.

Prefix scans

To iterate over a key prefix, you can combine Seek() and bytes.HasPrefix():

db.View(func(tx *bolt.Tx) error {
	// Assume bucket exists and has keys
	c := tx.Bucket([]byte("MyBucket")).Cursor()

	prefix := []byte("1234")
	for k, v := c.Seek(prefix); k != nil && bytes.HasPrefix(k, prefix); k, v = c.Next() {
		fmt.Printf("key=%s, value=%s\n", k, v)
	}

	return nil
})

Range scans

Another common use case is scanning over a range such as a time range. If you use a sortable time encoding such as RFC3339 then you can query a specific date range like this:

db.View(func(tx *bolt.Tx) error {
	// Assume our events bucket exists and has RFC3339 encoded time keys.
	c := tx.Bucket([]byte("Events")).Cursor()

	// Our time range spans the 90's decade.
	min := []byte("1990-01-01T00:00:00Z")
	max := []byte("2000-01-01T00:00:00Z")

	// Iterate over the 90's.
	for k, v := c.Seek(min); k != nil && bytes.Compare(k, max) <= 0; k, v = c.Next() {
		fmt.Printf("%s: %s\n", k, v)
	}

	return nil
})

Note that, while RFC3339 is sortable, the Golang implementation of RFC3339Nano does not use a fixed number of digits after the decimal point and is therefore not sortable.

ForEach()

You can also use the function ForEach() if you know you'll be iterating over all the keys in a bucket:

db.View(func(tx *bolt.Tx) error {
	// Assume bucket exists and has keys
	b := tx.Bucket([]byte("MyBucket"))

	b.ForEach(func(k, v []byte) error {
		fmt.Printf("key=%s, value=%s\n", k, v)
		return nil
	})
	return nil
})

Please note that keys and values in ForEach() are only valid while the transaction is open. If you need to use a key or value outside of the transaction, you must use copy() to copy it to another byte slice.

Nested buckets

You can also store a bucket in a key to create nested buckets. The API is the same as the bucket management API on the DB object:

func (*Bucket) CreateBucket(key []byte) (*Bucket, error)
func (*Bucket) CreateBucketIfNotExists(key []byte) (*Bucket, error)
func (*Bucket) DeleteBucket(key []byte) error

Say you had a multi-tenant application where the root level bucket was the account bucket. Inside of this bucket was a sequence of accounts which themselves are buckets. And inside the sequence bucket you could have many buckets pertaining to the Account itself (Users, Notes, etc) isolating the information into logical groupings.

// createUser creates a new user in the given account.
func createUser(accountID int, u *User) error {
    // Start the transaction.
    tx, err := db.Begin(true)
    if err != nil {
        return err
    }
    defer tx.Rollback()

    // Retrieve the root bucket for the account.
    // Assume this has already been created when the account was set up.
    root := tx.Bucket([]byte(strconv.FormatUint(accountID, 10)))

    // Setup the users bucket.
    bkt, err := root.CreateBucketIfNotExists([]byte("USERS"))
    if err != nil {
        return err
    }

    // Generate an ID for the new user.
    userID, err := bkt.NextSequence()
    if err != nil {
        return err
    }
    u.ID = userID

    // Marshal and save the encoded user.
    if buf, err := json.Marshal(u); err != nil {
        return err
    } else if err := bkt.Put([]byte(strconv.FormatUint(u.ID, 10)), buf); err != nil {
        return err
    }

    // Commit the transaction.
    if err := tx.Commit(); err != nil {
        return err
    }

    return nil
}

Database backups

Bolt is a single file so it's easy to backup. You can use the Tx.WriteTo() function to write a consistent view of the database to a writer. If you call this from a read-only transaction, it will perform a hot backup and not block your other database reads and writes.

By default, it will use a regular file handle which will utilize the operating system's page cache. See the Tx documentation for information about optimizing for larger-than-RAM datasets.

One common use case is to backup over HTTP so you can use tools like cURL to do database backups:

func BackupHandleFunc(w http.ResponseWriter, req *http.Request) {
	err := db.View(func(tx *bolt.Tx) error {
		w.Header().Set("Content-Type", "application/octet-stream")
		w.Header().Set("Content-Disposition", `attachment; filename="my.db"`)
		w.Header().Set("Content-Length", strconv.Itoa(int(tx.Size())))
		_, err := tx.WriteTo(w)
		return err
	})
	if err != nil {
		http.Error(w, err.Error(), http.StatusInternalServerError)
	}
}

Then you can backup using this command:

$ curl http://localhost/backup > my.db

Or you can open your browser to http://localhost/backup and it will download automatically.

If you want to backup to another file you can use the Tx.CopyFile() helper function.

Statistics

The database keeps a running count of many of the internal operations it performs so you can better understand what's going on. By grabbing a snapshot of these stats at two points in time we can see what operations were performed in that time range.

For example, we could start a goroutine to log stats every 10 seconds:

go func() {
	// Grab the initial stats.
	prev := db.Stats()

	for {
		// Wait for 10s.
		time.Sleep(10 * time.Second)

		// Grab the current stats and diff them.
		stats := db.Stats()
		diff := stats.Sub(&prev)

		// Encode stats to JSON and print to STDERR.
		json.NewEncoder(os.Stderr).Encode(diff)

		// Save stats for the next loop.
		prev = stats
	}
}()

It's also useful to pipe these stats to a service such as statsd for monitoring or to provide an HTTP endpoint that will perform a fixed-length sample.

Read-Only Mode

Sometimes it is useful to create a shared, read-only Bolt database. To this, set the Options.ReadOnly flag when opening your database. Read-only mode uses a shared lock to allow multiple processes to read from the database but it will block any processes from opening the database in read-write mode.

db, err := bolt.Open("my.db", 0666, &bolt.Options{ReadOnly: true})
if err != nil {
	log.Fatal(err)
}

Mobile Use (iOS/Android)

Bolt is able to run on mobile devices by leveraging the binding feature of the gomobile tool. Create a struct that will contain your database logic and a reference to a *bolt.DB with a initializing constructor that takes in a filepath where the database file will be stored. Neither Android nor iOS require extra permissions or cleanup from using this method.

func NewBoltDB(filepath string) *BoltDB {
	db, err := bolt.Open(filepath+"/demo.db", 0600, nil)
	if err != nil {
		log.Fatal(err)
	}

	return &BoltDB{db}
}

type BoltDB struct {
	db *bolt.DB
	...
}

func (b *BoltDB) Path() string {
	return b.db.Path()
}

func (b *BoltDB) Close() {
	b.db.Close()
}

Database logic should be defined as methods on this wrapper struct.

To initialize this struct from the native language (both platforms now sync their local storage to the cloud. These snippets disable that functionality for the database file):

Android

String path;
if (android.os.Build.VERSION.SDK_INT >=android.os.Build.VERSION_CODES.LOLLIPOP){
    path = getNoBackupFilesDir().getAbsolutePath();
} else{
    path = getFilesDir().getAbsolutePath();
}
Boltmobiledemo.BoltDB boltDB = Boltmobiledemo.NewBoltDB(path)

iOS

- (void)demo {
    NSString* path = [NSSearchPathForDirectoriesInDomains(NSLibraryDirectory,
                                                          NSUserDomainMask,
                                                          YES) objectAtIndex:0];
	GoBoltmobiledemoBoltDB * demo = GoBoltmobiledemoNewBoltDB(path);
	[self addSkipBackupAttributeToItemAtPath:demo.path];
	//Some DB Logic would go here
	[demo close];
}

- (BOOL)addSkipBackupAttributeToItemAtPath:(NSString *) filePathString
{
    NSURL* URL= [NSURL fileURLWithPath: filePathString];
    assert([[NSFileManager defaultManager] fileExistsAtPath: [URL path]]);

    NSError *error = nil;
    BOOL success = [URL setResourceValue: [NSNumber numberWithBool: YES]
                                  forKey: NSURLIsExcludedFromBackupKey error: &error];
    if(!success){
        NSLog(@"Error excluding %@ from backup %@", [URL lastPathComponent], error);
    }
    return success;
}

Resources

For more information on getting started with Bolt, check out the following articles:

Comparison with other databases

Postgres, MySQL, & other relational databases

Relational databases structure data into rows and are only accessible through the use of SQL. This approach provides flexibility in how you store and query your data but also incurs overhead in parsing and planning SQL statements. Bolt accesses all data by a byte slice key. This makes Bolt fast to read and write data by key but provides no built-in support for joining values together.

Most relational databases (with the exception of SQLite) are standalone servers that run separately from your application. This gives your systems flexibility to connect multiple application servers to a single database server but also adds overhead in serializing and transporting data over the network. Bolt runs as a library included in your application so all data access has to go through your application's process. This brings data closer to your application but limits multi-process access to the data.

LevelDB, RocksDB

LevelDB and its derivatives (RocksDB, HyperLevelDB) are similar to Bolt in that they are libraries bundled into the application, however, their underlying structure is a log-structured merge-tree (LSM tree). An LSM tree optimizes random writes by using a write ahead log and multi-tiered, sorted files called SSTables. Bolt uses a B+tree internally and only a single file. Both approaches have trade-offs.

If you require a high random write throughput (>10,000 w/sec) or you need to use spinning disks then LevelDB could be a good choice. If your application is read-heavy or does a lot of range scans then Bolt could be a good choice.

One other important consideration is that LevelDB does not have transactions. It supports batch writing of key/values pairs and it supports read snapshots but it will not give you the ability to do a compare-and-swap operation safely. Bolt supports fully serializable ACID transactions.

LMDB

Bolt was originally a port of LMDB so it is architecturally similar. Both use a B+tree, have ACID semantics with fully serializable transactions, and support lock-free MVCC using a single writer and multiple readers.

The two projects have somewhat diverged. LMDB heavily focuses on raw performance while Bolt has focused on simplicity and ease of use. For example, LMDB allows several unsafe actions such as direct writes for the sake of performance. Bolt opts to disallow actions which can leave the database in a corrupted state. The only exception to this in Bolt is DB.NoSync.

There are also a few differences in API. LMDB requires a maximum mmap size when opening an mdb_env whereas Bolt will handle incremental mmap resizing automatically. LMDB overloads the getter and setter functions with multiple flags whereas Bolt splits these specialized cases into their own functions.

Caveats & Limitations

It's important to pick the right tool for the job and Bolt is no exception. Here are a few things to note when evaluating and using Bolt:

  • Bolt is good for read intensive workloads. Sequential write performance is also fast but random writes can be slow. You can use DB.Batch() or add a write-ahead log to help mitigate this issue.

  • Bolt uses a B+tree internally so there can be a lot of random page access. SSDs provide a significant performance boost over spinning disks.

  • Try to avoid long running read transactions. Bolt uses copy-on-write so old pages cannot be reclaimed while an old transaction is using them.

  • Byte slices returned from Bolt are only valid during a transaction. Once the transaction has been committed or rolled back then the memory they point to can be reused by a new page or can be unmapped from virtual memory and you'll see an unexpected fault address panic when accessing it.

  • Bolt uses an exclusive write lock on the database file so it cannot be shared by multiple processes.

  • Be careful when using Bucket.FillPercent. Setting a high fill percent for buckets that have random inserts will cause your database to have very poor page utilization.

  • Use larger buckets in general. Smaller buckets causes poor page utilization once they become larger than the page size (typically 4KB).

  • Bulk loading a lot of random writes into a new bucket can be slow as the page will not split until the transaction is committed. Randomly inserting more than 100,000 key/value pairs into a single new bucket in a single transaction is not advised.

  • Bolt uses a memory-mapped file so the underlying operating system handles the caching of the data. Typically, the OS will cache as much of the file as it can in memory and will release memory as needed to other processes. This means that Bolt can show very high memory usage when working with large databases. However, this is expected and the OS will release memory as needed. Bolt can handle databases much larger than the available physical RAM, provided its memory-map fits in the process virtual address space. It may be problematic on 32-bits systems.

  • The data structures in the Bolt database are memory mapped so the data file will be endian specific. This means that you cannot copy a Bolt file from a little endian machine to a big endian machine and have it work. For most users this is not a concern since most modern CPUs are little endian.

  • Because of the way pages are laid out on disk, Bolt cannot truncate data files and return free pages back to the disk. Instead, Bolt maintains a free list of unused pages within its data file. These free pages can be reused by later transactions. This works well for many use cases as databases generally tend to grow. However, it's important to note that deleting large chunks of data will not allow you to reclaim that space on disk.

    For more information on page allocation, see this comment.

Reading the Source

Bolt is a relatively small code base (<5KLOC) for an embedded, serializable, transactional key/value database so it can be a good starting point for people interested in how databases work.

The best places to start are the main entry points into Bolt:

  • Open() - Initializes the reference to the database. It's responsible for creating the database if it doesn't exist, obtaining an exclusive lock on the file, reading the meta pages, & memory-mapping the file.

  • DB.Begin() - Starts a read-only or read-write transaction depending on the value of the writable argument. This requires briefly obtaining the "meta" lock to keep track of open transactions. Only one read-write transaction can exist at a time so the "rwlock" is acquired during the life of a read-write transaction.

  • Bucket.Put() - Writes a key/value pair into a bucket. After validating the arguments, a cursor is used to traverse the B+tree to the page and position where they key & value will be written. Once the position is found, the bucket materializes the underlying page and the page's parent pages into memory as "nodes". These nodes are where mutations occur during read-write transactions. These changes get flushed to disk during commit.

  • Bucket.Get() - Retrieves a key/value pair from a bucket. This uses a cursor to move to the page & position of a key/value pair. During a read-only transaction, the key and value data is returned as a direct reference to the underlying mmap file so there's no allocation overhead. For read-write transactions, this data may reference the mmap file or one of the in-memory node values.

  • Cursor - This object is simply for traversing the B+tree of on-disk pages or in-memory nodes. It can seek to a specific key, move to the first or last value, or it can move forward or backward. The cursor handles the movement up and down the B+tree transparently to the end user.

  • Tx.Commit() - Converts the in-memory dirty nodes and the list of free pages into pages to be written to disk. Writing to disk then occurs in two phases. First, the dirty pages are written to disk and an fsync() occurs. Second, a new meta page with an incremented transaction ID is written and another fsync() occurs. This two phase write ensures that partially written data pages are ignored in the event of a crash since the meta page pointing to them is never written. Partially written meta pages are invalidated because they are written with a checksum.

If you have additional notes that could be helpful for others, please submit them via pull request.

Other Projects Using Bolt

Below is a list of public, open source projects that use Bolt:

  • Algernon - A HTTP/2 web server with built-in support for Lua. Uses BoltDB as the default database backend.
  • Bazil - A file system that lets your data reside where it is most convenient for it to reside.
  • bolter - Command-line app for viewing BoltDB file in your terminal.
  • boltcli - the redis-cli for boltdb with Lua script support.
  • BoltHold - An embeddable NoSQL store for Go types built on BoltDB
  • BoltStore - Session store using Bolt.
  • Boltdb Boilerplate - Boilerplate wrapper around bolt aiming to make simple calls one-liners.
  • BoltDbWeb - A web based GUI for BoltDB files.
  • bleve - A pure Go search engine similar to ElasticSearch that uses Bolt as the default storage backend.
  • btcwallet - A bitcoin wallet.
  • buckets - a bolt wrapper streamlining simple tx and key scans.
  • cayley - Cayley is an open-source graph database using Bolt as optional backend.
  • ChainStore - Simple key-value interface to a variety of storage engines organized as a chain of operations.
  • Consul - Consul is service discovery and configuration made easy. Distributed, highly available, and datacenter-aware.
  • DVID - Added Bolt as optional storage engine and testing it against Basho-tuned leveldb.
  • dcrwallet - A wallet for the Decred cryptocurrency.
  • drive - drive is an unofficial Google Drive command line client for *NIX operating systems.
  • event-shuttle - A Unix system service to collect and reliably deliver messages to Kafka.
  • Freehold - An open, secure, and lightweight platform for your files and data.
  • Go Report Card - Go code quality report cards as a (free and open source) service.
  • GoWebApp - A basic MVC web application in Go using BoltDB.
  • GoShort - GoShort is a URL shortener written in Golang and BoltDB for persistent key/value storage and for routing it's using high performent HTTPRouter.
  • gopherpit - A web service to manage Go remote import paths with custom domains
  • gokv - Simple key-value store abstraction and implementations for Go (Redis, Consul, etcd, bbolt, BadgerDB, LevelDB, Memcached, DynamoDB, S3, PostgreSQL, MongoDB, CockroachDB and many more)
  • Gitchain - Decentralized, peer-to-peer Git repositories aka "Git meets Bitcoin".
  • InfluxDB - Scalable datastore for metrics, events, and real-time analytics.
  • ipLocator - A fast ip-geo-location-server using bolt with bloom filters.
  • ipxed - Web interface and api for ipxed.
  • Ironsmith - A simple, script-driven continuous integration (build - > test -> release) tool, with no external dependencies
  • Kala - Kala is a modern job scheduler optimized to run on a single node. It is persistent, JSON over HTTP API, ISO 8601 duration notation, and dependent jobs.
  • Key Value Access Langusge (KVAL) - A proposed grammar for key-value datastores offering a bbolt binding.
  • LedisDB - A high performance NoSQL, using Bolt as optional storage.
  • lru - Easy to use Bolt-backed Least-Recently-Used (LRU) read-through cache with chainable remote stores.
  • mbuckets - A Bolt wrapper that allows easy operations on multi level (nested) buckets.
  • MetricBase - Single-binary version of Graphite.
  • MuLiFS - Music Library Filesystem creates a filesystem to organise your music files.
  • NATS - NATS Streaming uses bbolt for message and metadata storage.
  • Operation Go: A Routine Mission - An online programming game for Golang using Bolt for user accounts and a leaderboard.
  • photosite/session - Sessions for a photo viewing site.
  • Prometheus Annotation Server - Annotation server for PromDash & Prometheus service monitoring system.
  • Rain - BitTorrent client and library.
  • reef-pi - reef-pi is an award winning, modular, DIY reef tank controller using easy to learn electronics based on a Raspberry Pi.
  • Request Baskets - A web service to collect arbitrary HTTP requests and inspect them via REST API or simple web UI, similar to RequestBin service
  • Seaweed File System - Highly scalable distributed key~file system with O(1) disk read.
  • stow - a persistence manager for objects backed by boltdb.
  • Storm - Simple and powerful ORM for BoltDB.
  • SimpleBolt - A simple way to use BoltDB. Deals mainly with strings.
  • Skybox Analytics - A standalone funnel analysis tool for web analytics.
  • Scuttlebutt - Uses Bolt to store and process all Twitter mentions of GitHub projects.
  • tentacool - REST api server to manage system stuff (IP, DNS, Gateway...) on a linux server.
  • torrent - Full-featured BitTorrent client package and utilities in Go. BoltDB is a storage backend in development.
  • Wiki - A tiny wiki using Goji, BoltDB and Blackfriday.

If you are using Bolt in a project please send a pull request to add it to the list.

Owner
etcd-io
etcd Development and Communities
etcd-io
Comments
  • Fix incorrect unsafe usage

    Fix incorrect unsafe usage

    After checkptr fixes by 2fc6815c, it was discovered that new issues were hit in production systems, in particular when a single process opened and updated multiple separate databases. This indicates that some bug relating to bad unsafe usage was introduced during this commit.

    This commit combines several attempts at fixing this new issue. For example, slices are once again created by slicing an array of "max allocation" elements, but this time with the cap set to the intended length. This operation is espressly permitted according to the Go wiki, so it should be preferred to type converting a reflect.SliceHeader.

    Fixes #214.

  • sweep increased allocation count with v1.3.4

    sweep increased allocation count with v1.3.4

    After updating our project to the latest version of bbolt which includes #201, we've started running into this runtime error in our tests:

    runtime: nelems=21 nalloc=3 previous allocCount=2 nfreed=65535
    fatal error: sweep increased allocation count
    runtime stack:
    runtime.throw(0xc2ecf1, 0x20)
    	/home/travis/.gimme/versions/go1.13.linux.amd64/src/runtime/panic.go:774 +0x72
    runtime.(*mspan).sweep(0x7f399ffced10, 0x7f399ffced00, 0x455700)
    	/home/travis/.gimme/versions/go1.13.linux.amd64/src/runtime/mgcsweep.go:328 +0x8e1
    runtime.(*mcentral).uncacheSpan(0x11e9d60, 0x7f399ffced10)
    	/home/travis/.gimme/versions/go1.13.linux.amd64/src/runtime/mcentral.go:197 +0x79
    runtime.(*mcache).releaseAll(0x7f39a00cd008)
    	/home/travis/.gimme/versions/go1.13.linux.amd64/src/runtime/mcache.go:158 +0x6b
    runtime.(*mcache).prepareForSweep(0x7f39a00cd008)
    	/home/travis/.gimme/versions/go1.13.linux.amd64/src/runtime/mcache.go:185 +0x46
    runtime.acquirep(0xc000042000)
    	/home/travis/.gimme/versions/go1.13.linux.amd64/src/runtime/proc.go:4114 +0x3d
    runtime.stopm()
    	/home/travis/.gimme/versions/go1.13.linux.amd64/src/runtime/proc.go:1930 +0xe8
    runtime.gcstopm()
    	/home/travis/.gimme/versions/go1.13.linux.amd64/src/runtime/proc.go:2125 +0xb4
    runtime.schedule()
    	/home/travis/.gimme/versions/go1.13.linux.amd64/src/runtime/proc.go:2481 +0x473
    runtime.exitsyscall0(0xc000001680)
    	/home/travis/.gimme/versions/go1.13.linux.amd64/src/runtime/proc.go:3141 +0x116
    runtime.mcall(0x800000)
    	/home/travis/.gimme/versions/go1.13.linux.amd64/src/runtime/asm_amd64.s:318 +0x5b
    

    So far, it only seems to happen about every 1 in 5 test runs. At first we were running the tests with Go 1.14, then tried to go back to version 1.13, but the issue would still pop up. Only after downgrading to bbolt v1.3.3 does the issue seem to disappear. Given that the issue seems to persist using the last two major versions of Go, it may be the case that #201 uncovered another underlying issue, or the changes in the PR are actually triggering this new error.

    We're working on creating a reproducible program to trigger this warning.

    The full trace of this error in our tests can be found here: https://paste.ubuntu.com/p/XNgrZ49dbw/

  • go mod tidy failed

    go mod tidy failed

    when i run "go mod tidy" i got this
    github.com/coreos/etcd/etcdmain imports github.com/coreos/etcd/etcdserver imports github.com/coreos/etcd/mvcc/backend imports github.com/coreos/bbolt: github.com/coreos/[email protected]: parsing go.mod: module declares its path as: go.etcd.io/bbolt but was required as: github.com/coreos/bbolt

  • panic: runtime error: unsafe pointer conversion (Go 1.14)

    panic: runtime error: unsafe pointer conversion (Go 1.14)

    In Go 1.14 there is now checks of pointer conversions with the requirement that they should align to the "natural size".

    It seems like they decided to make this enabled when -race is specified: https://github.com/golang/go/issues/34964

    This leads to crashes like this:

        --- FAIL: TestNewBoltStore/Basic (0.00s)
    panic: runtime error: unsafe pointer conversion [recovered]
    	panic: runtime error: unsafe pointer conversion
    goroutine 6 [running]:
    testing.tRunner.func1(0xc000100360)
    	/home/travis/.gimme/versions/go/src/testing/testing.go:916 +0xaeb
    panic(0x6b5ee0, 0xc00000e260)
    	/home/travis/.gimme/versions/go/src/runtime/panic.go:967 +0x396
    github.com/etcd-io/bbolt.(*freelist).write(0xc0000fc080, 0xc0000ff000, 0xc0000ff000, 0x0)
    	/home/travis/gopath/pkg/mod/github.com/etcd-io/[email protected]/freelist.go:318 +0xe9
    github.com/etcd-io/bbolt.(*Tx).commitFreelist(0xc00011a0e0, 0x2, 0x7f603f3d1000)
    	/home/travis/gopath/pkg/mod/github.com/etcd-io/[email protected]/tx.go:234 +0x204
    github.com/etcd-io/bbolt.(*Tx).Commit(0xc00011a0e0, 0x0, 0x0)
    	/home/travis/gopath/pkg/mod/github.com/etcd-io/[email protected]/tx.go:175 +0x7d6
    github.com/etcd-io/bbolt.(*DB).Update(0xc000106000, 0xc00005abf8, 0x0, 0x0)
    	/home/travis/gopath/pkg/mod/github.com/etcd-io/[email protected]/db.go:701 +0x174
    ...
    
  • use segregated hmap to boost the freelist allocate and release performace

    use segregated hmap to boost the freelist allocate and release performace

    In this pr, I use segregated bitmap to replace the original freeids allocating and releasing approach. It is much faster than the original version, especially when the db size is large or the fragmentation in the db is large, we can gain 1000x faster performance.

  • Such keys, much RAM!

    Such keys, much RAM!

    Good morning!

    Why this code allocates so much memory? (Insert 10M records with keys "10000000"..."1999999" and empty values → 2G+ allocated RAM, output below in gist.)

    I decided to migrate 10M+ rows table from MySQL to BoltDB using UUID for keys. But when I tried, I ran out of memory, because process allocated 26G+ RAM during migration.

    Then I found that it's about key length: 4 bytes are fine, but 8 bytes and more cause pain. Maybe, hashing, caching, paging, duplication in memory, I don't really know how it works.

    Is this normal behaviour for Bolt (and other similar storages)? Can it be tuned? Or, maybe, it's not the way I should use Bolt?

    macOS 10.13.4, go version go1.10.2 darwin/amd64

  • Panic (index out of range) on writeable tx rollback with db.NoFreelistSync

    Panic (index out of range) on writeable tx rollback with db.NoFreelistSync

    If the db is opened with db.NoFreelistSync = true, if a writeable transaction is rolled-back, we get following panic:

    === RUN   TestPanicOnTxTollbackWithNoFreelistSync
    --- FAIL: TestPanicOnTxTollbackWithNoFreelistSync (0.05s)
    panic: runtime error: index out of range [recovered]
    	panic: runtime error: index out of range
    
    goroutine 52 [running]:
    testing.tRunner.func1(0xc0001a4200)
    	/usr/local/go/src/testing/testing.go:830 +0x68a
    panic(0x1b3d700, 0x2431150)
    	/usr/local/go/src/runtime/panic.go:522 +0x1b5
    github.com/nats-io/nats-streaming-server/vendor/go.etcd.io/bbolt.(*DB).page(...)
    	/Users/ivan/dev/go/src/github.com/nats-io/nats-streaming-server/vendor/go.etcd.io/bbolt/db.go:880
    github.com/nats-io/nats-streaming-server/vendor/go.etcd.io/bbolt.(*Tx).rollback(0xc0002000e0)
    	/Users/ivan/dev/go/src/github.com/nats-io/nats-streaming-server/vendor/go.etcd.io/bbolt/tx.go:267 +0x24b
    github.com/nats-io/nats-streaming-server/vendor/go.etcd.io/bbolt.(*Tx).Rollback(0xc0002000e0, 0xc00016e990, 0x1)
    	/Users/ivan/dev/go/src/github.com/nats-io/nats-streaming-server/vendor/go.etcd.io/bbolt/tx.go:257 +0x75
    github.com/nats-io/nats-streaming-server/server.TestPanicOnTxTollbackWithNoFreelistSync(0xc0001a4200)
    	/Users/ivan/dev/go/src/github.com/nats-io/nats-streaming-server/server/raft_log_test.go:459 +0x5b2
    testing.tRunner(0xc0001a4200, 0x1c50670)
    	/usr/local/go/src/testing/testing.go:865 +0x164
    created by testing.(*T).Run
    	/usr/local/go/src/testing/testing.go:916 +0x69a
    exit status 2
    FAIL	github.com/nats-io/nats-streaming-server/server	0.080s
    Error: Tests failed.
    

    Here is the test to produce such panic:

    func TestPanicOnTxTollbackWithNoFreelistSync(t *testing.T) {
    	os.Remove("bolt.db")
    	defer os.Remove("bolt.db")
    
    	db, err := bolt.Open("bolt.db", 0600, nil)
    	if err != nil {
    		t.Fatalf("Error opening db: %v", err)
    	}
    	db.NoFreelistSync = true
    	db.FreelistType = bolt.FreelistMapType
    
    	tx, err := db.Begin(true)
    	if err != nil {
    		t.Fatalf("Error starting tx: %v", err)
    	}
    	bucket := []byte("mybucket")
    	if _, err := tx.CreateBucket(bucket); err != nil {
    		t.Fatalf("Error creating bucket: %v", err)
    	}
    	if err := tx.Commit(); err != nil {
    		t.Fatalf("Error on commit: %v", err)
    	}
    
    	tx, err = db.Begin(true)
    	if err != nil {
    		t.Fatalf("Error starting tx: %v", err)
    	}
    	b := tx.Bucket(bucket)
    	if err := b.Put([]byte("k"), []byte("v")); err != nil {
    		t.Fatalf("Error on put: %v", err)
    	}
    	// Imagine there is an error and tx needs to be rolled-back
    	if err := tx.Rollback(); err != nil {
    		t.Fatalf("Error on rollback: %v", err)
    	}
    
    	tx, err = db.Begin(false)
    	if err != nil {
    		t.Fatalf("Error starting tx: %v", err)
    	}
    	b = tx.Bucket(bucket)
    	if v := b.Get([]byte("k")); v != nil {
    		t.Fatalf("Value for k should not have been stored")
    	}
    	if err := tx.Rollback(); err != nil {
    		t.Fatalf("Error on rollback: %v", err)
    	}
    }
    

    Commenting out db.NoFreelistSync = true solves the issue.

    I wonder if you should check for NoFreelistSync here

    Something like this instead:

    	if tx.writable && !tx.db.NoFreelistSync {
    		tx.db.freelist.rollback(tx.meta.txid)
    		tx.db.freelist.reload(tx.db.page(tx.db.meta().freelist))
    	}
    	tx.close()
    
  • page already freed

    page already freed

    Openig same issue as https://github.com/boltdb/bolt/issues/731, because coreos/bbolt also crashes in same way:

    package main
    
    import (
    	"log"
    	"math/rand"
    
    	"github.com/coreos/bbolt"
    	// "github.com/boltdb/bolt"
    )
    
    var BC = make(chan []byte)
    
    func main() {
    
    	go Add(BC, "/opt/test.db")
    
    	s := make([]byte, 30)
    	for i := 0; i < 100000000; i++ {
    		rand.Read(s)
    		BC <- s
    	}
    
    	close(BC)
    
    }
    
    func Add(c chan []byte, bfile string) {
    	const bname = `bc41`
    
    	db, err := bolt.Open(bfile, 0600, nil)
    	if err != nil {
    		log.Fatal(err)
    	}
    	defer db.Close()
    
    	err = db.Update(func(tx *bolt.Tx) error {
    		_, err := tx.CreateBucketIfNotExists([]byte(bname))
    		if err != nil {
    			return err //fmt.Errorf("create bucket: %s", err)
    		}
    		return nil
    	})
    	if err != nil {
    		log.Fatal(err)
    		panic(err)
    	}
    
    	more := true
    	for more {
    		err := db.Update(func(tx *bolt.Tx) error {
    			b := tx.Bucket([]byte(bname))
    			for i := 0; i < 100000; i++ {
    				v := <-c
    				if v == nil {
    					more = false
    					return nil
    				}
    				err = b.Put(v, []byte("0"))
    				if err != nil {
    					return err
    				}
    			}
    			return nil
    		})
    		if err != nil {
    			log.Fatal(err)
    			panic(err)
    		}
    	}
    }
    
    panic: page 3955 already freed
    
    goroutine 5 [running]:
    github.com/coreos/bbolt.(*freelist).free(0xc420074210, 0x7, 0x7fbb550ef000)
    	/home/jambo/golang/src/github.com/coreos/bbolt/freelist.go:143 +0x4ad
    github.com/coreos/bbolt.(*node).spill(0xc42163d810, 0xc4209c7800, 0x55f920)
    	/home/jambo/golang/src/github.com/coreos/bbolt/node.go:363 +0x210
    github.com/coreos/bbolt.(*node).spill(0xc42163d7a0, 0x0, 0x0)
    	/home/jambo/golang/src/github.com/coreos/bbolt/node.go:350 +0xbf
    github.com/coreos/bbolt.(*node).spill(0xc42163d490, 0xc4229c36e0, 0xc420045ab0)
    	/home/jambo/golang/src/github.com/coreos/bbolt/node.go:350 +0xbf
    github.com/coreos/bbolt.(*Bucket).spill(0xc420052580, 0xc4229c3600, 0xc420045d28)
    	/home/jambo/golang/src/github.com/coreos/bbolt/bucket.go:568 +0x4d3
    github.com/coreos/bbolt.(*Bucket).spill(0xc4200900f8, 0x2116cdd17, 0x5721a0)
    	/home/jambo/golang/src/github.com/coreos/bbolt/bucket.go:535 +0x417
    github.com/coreos/bbolt.(*Tx).Commit(0xc4200900e0, 0x0, 0x0)
    	/home/jambo/golang/src/github.com/coreos/bbolt/tx.go:160 +0x129
    github.com/coreos/bbolt.(*DB).Update(0xc420088000, 0xc42006bf98, 0x0, 0x0)
    	/home/jambo/golang/src/github.com/coreos/bbolt/db.go:674 +0xf2
    main.Add(0xc420072060, 0x4ea01b, 0x12)
    	/home/jambo/go/src/jambo/tests/error1/error1.go:50 +0x1ac
    created by main.main
    	/home/jambo/go/src/jambo/tests/error1/error1.go:15 +0x5a
    
    go version 
    go version go1.9.1 linux/amd64
    
    uname -a
    Linux bee 4.9.0-0.bpo.4-amd64 #1 SMP Debian 4.9.51-1~bpo8+1 (2017-10-17) x86_64 GNU/Linux
    
  • use segregated hashmap to boost the freelist allocate and release performance

    use segregated hashmap to boost the freelist allocate and release performance

    In this pr, I use segregated hashmap to replace the original freeids allocating and releasing approach. It is much faster than the original version, especially when the db size is large or the fragmentation in the db is large, we can gain 1000x faster performance.

  • Fix Windows flock/funlock race

    Fix Windows flock/funlock race

    Signed-off-by: John Howard [email protected]

    Fixes https://github.com/etcd-io/bbolt/issues/121, and downstream https://github.com/docker/libnetwork/issues/1950.

    Using the test program from https://github.com/etcd-io/bbolt/issues/121:

    package main
    
    import (
    	"fmt"
    	"os"
    	"sync"
    
    	bolt "github.com/etcd-io/bbolt"
    )
    
    func main() {
    	const instances = 30
    	var wg sync.WaitGroup
    	for instance := 0; instance < instances; instance++ {
    		wg.Add(1)
    		go opener(instance, &wg)
    
    	}
    	wg.Wait()
    	fmt.Println("Success :)")
    }
    
    func opener(instance int, wg *sync.WaitGroup) {
    	defer wg.Done()
    	db, err := bolt.Open(`c:\bolt\test.db`, 0600, nil)
    	if err != nil {
    		panic(fmt.Sprintf("instance %d failed to open: %s", instance, err))
    		os.Exit(-1)
    	}
    	db.Close()
    }
    

    Running it in two separate Windows, this now succeeds after running a significant amount of time without a panic. Pre this change, it would panic within the first few iterations.

    $lastExitCode=0;$i=0;while ($lastExitCode -eq 0) {$i++;echo $i;.\test.exe}

    Success :)
    3128
    Success :)
    3129
    Success :)
    3130
    Success :)
    3131
    Success :)
    3132
    Success :)
    3133
    Success :)
    3134
    Success :)
    3135
    Success :)
    3136
    Success :)
    3137
    Success :)
    3138
    Success :)
    3139
    Success :)
    3140
    

    etc.

    @jstarks - Could you do a quick check please.

    @dineshgovindasamy @msabansal @carlfischer1 @johnstep @mavenugo FYI.

  • fix(cursor): `Last` method needs skip empty pages

    fix(cursor): `Last` method needs skip empty pages

    In the same transaction, Detete may cause the last page to be empty. In this case, calling the Last method will return nill instead of the value of the previous page, which is incorrect.

    1. suppose there is a bucket of 1000 elements
    2. the structure diagram at this time
                              +---------------------+                                                                  
                              |     root bucket     |                                                                  
                    +---------+---------------------+---------+                                                        
                    |                                         |                                                        
                    |                                         |                                                        
                    |                                         |                                                        
                    |                                         |                                                        
                    |                                         |                                                        
                    |                                         |                                                        
             +--------------------+                 +---------------------------+                                      
             |                    |                 |                           |                                      
     leaf1   |1,2,3,4.......300 |                 |301,302,303.......1000     |  leaf2                               
             |                    |                 |                           |                                      
             +--------------------+                 +---------------------------+           
    
    1. now calling the Last method should return 1000
    2. ok, now we call the Delete method to delete the last 800 elements
                              +---------------------+                                                                  
                              |     root bucket     |                                                                  
                    +---------+---------------------+---------+                                                        
                    |                                         |                                                        
                    |                                         |                                                        
                    |                                         |                                                        
                    |                                         |                                                        
                    |                                         |                                                        
                    |                                         |                                                        
             +----------------+                             +--+                                                       
             |                |                             |  |                                                       
     leaf1   |1,2,3......200 |                             |  |  leaf2 is empty                                       
             |                |                             |  |                                                       
             +----------------+                             +--+                                                       
                                                                                                                         
                                                                                                                       
       
    
    1. bad thing happened, calling the Last method again returns nil, but I think it should return 200
  • TestDB_Close_PendingTx_RW flakes on windows

    TestDB_Close_PendingTx_RW flakes on windows

    https://github.com/ptabor/bbolt/actions/runs/3831841195/jobs/6521456595

    === RUN   TestDB_Close_PendingTx_RW
        btesting.go:42: Opening bbolt DB at: C:\Users\RUNNER~1\AppData\Local\Temp\TestDB_Close_PendingTx_RW2836126199\001\db
        btesting.go:81: Closing bbolt DB at: C:\Users\RUNNER~1\AppData\Local\Temp\TestDB_Close_PendingTx_RW2836126199\001\db
        db_test.go:720: database did not close
        btesting.go:150: 
            	Error Trace:	D:\a\bbolt\bbolt\btesting.go:150
            	            				D:\a\bbolt\bbolt\btesting.go:69
            	            				D:\a\bbolt\bbolt\testing.go:912
            	            				D:\a\bbolt\bbolt\testing.go:1049
            	            				D:\a\bbolt\bbolt\testing.go:1253
            	            				D:\a\bbolt\bbolt\panic.go:642
            	            				D:\a\bbolt\bbolt\testing.go:756
            	            				D:\a\bbolt\bbolt\testing.go:824
            	            				D:\a\bbolt\bbolt\db_test.go:720
            	            				D:\a\bbolt\bbolt\db_test.go:671
            	Error:      	Received unexpected error:
            	            	database not open
            	Test:       	TestDB_Close_PendingTx_RW
    --- FAIL: TestDB_Close_PendingTx_RW (0.31s)
    
  • `bbolt pages` should distinguish reachable (vs. free) pages

    `bbolt pages` should distinguish reachable (vs. free) pages

    1. Add an option to bbolt.Options that forces load of freepages e
    2. Make bbolt pages tool to set this flag when opening file

    Please see for context: https://github.com/etcd-io/bbolt/pull/371

  • RO db should allow access to tx.Page(...) thus bbolt pages

    RO db should allow access to tx.Page(...) thus bbolt pages

    I discoved that after change https://github.com/etcd-io/bbolt/pull/365 the 'bbolt pages' command stopped working.

    The problem is this check: https://github.com/etcd-io/bbolt/blob/b654ce922133ff49bb385297f30835ca357bac5f/tx.go#L663 that assumes that freepages has been loaded.

    We can either preload the freepages when opening a RO file: https://github.com/etcd-io/bbolt/blob/b654ce922133ff49bb385297f30835ca357bac5f/db.go#L280-L284 but this would slow-down opening RO files.

    Thus for now I disable checking whether page is free...

    Signed-off-by: Piotr Tabor [email protected]

  • bbolt CLI surgery commands

    bbolt CLI surgery commands

    After https://github.com/etcd-io/bbolt/pull/361 is submitted it would be good to expose additional bbolt file 'surgery' commands:

    1. bbolt surgery revert-meta-page. db.inputFile db.outputFile

      • should execute https://github.com/etcd-io/bbolt/blob/d13096eccc8deb93488897990957a8ac82df8ada/internal/surgeon/surgeon.go#L41
    2. bbolt surgery copy-page db.inputFile db.outputFile 17 14

    • should execute https://github.com/etcd-io/bbolt/blob/d13096eccc8deb93488897990957a8ac82df8ada/internal/surgeon/surgeon.go#L10

    3 bbolt surgery write-empty-branch-page db.inputFile db.outputFile 17

    • should execute a new command that overwites given page with an empty node.

    Follow up to: https://github.com/etcd-io/bbolt/issues/335

  • Exploratory test for freepages allocator

    Exploratory test for freepages allocator

    Free pages allocation is extremely sensitive piece of bbolt logic. Double releasing a page has catastrophic consequences for bbolt file consistency and data durability.
    Current code with 2 implementations (array & map) using the same struct is very hard to follow and justifies bigger refactor. But I would be afraid to touch this code without investing in extensive unit/exploratory tests for this data structure.

    I think we should have a test that uses just free-pages allocator:

    It randomly keep performing one of the following action.

    • performs a RW TX that: - releases some of the nodes present at the beginning of transaction. - allocates random number of nodes. Some of this nodes are multi-page. The closer to DB size limit, the less should get allocated. - sometimes parforms a rollback after such changes.

    • start a RO transaction. Copy the list of 'available' pages to per-tx data.

    • closes one of the RO transactions [and verifies all the nodes present at the beginning of the transactions are still available]

    I think this could be based on 2 data structures:

    map [pageId] -> (txId, allocated) - a map that stores for each page, the last RW transaction that 'wrote' (allocated) the page list[ (pageid, size) ] - list of pages that are 'in-use' at the current RW transaction

    Freelist is not locked - so the test should be executed from a single thread.

  • skip rollback if db or db.data is nil

    skip rollback if db or db.data is nil

    Provent bbolt from panicking in case https://github.com/etcd-io/bbolt/issues/262

    In (*DB) mmap, it munmap the file firstly, then mmap it again. If the munmap somehow fails, then the db.data is reset to nil. In this case, there is no need to execute rollback.

    Signed-off-by: Benjamin Wang [email protected]

Related tags
Embedded key-value store for read-heavy workloads written in Go
Embedded key-value store for read-heavy workloads written in Go

Pogreb Pogreb is an embedded key-value store for read-heavy workloads written in Go. Key characteristics 100% Go. Optimized for fast random lookups an

Jan 3, 2023
Nipo is a powerful, fast, multi-thread, clustered and in-memory key-value database, with ability to configure token and acl on commands and key-regexes written by GO

Welcome to NIPO Nipo is a powerful, fast, multi-thread, clustered and in-memory key-value database, with ability to configure token and acl on command

Dec 28, 2022
pure golang key database support key have value. 非常高效实用的键值数据库。
pure golang key database support key have value.  非常高效实用的键值数据库。

orderfile32 pure golang key database support key have value The orderfile32 is standard alone fast key value database. It have two version. one is thi

Apr 30, 2022
Simple key value database that use json files to store the database

KValDB Simple key value database that use json files to store the database, the key and the respective value. This simple database have two gRPC metho

Nov 13, 2021
BuntDB is an embeddable, in-memory key/value database for Go with custom indexing and geospatial support
BuntDB is an embeddable, in-memory key/value database for Go with custom indexing and geospatial support

BuntDB is a low-level, in-memory, key/value store in pure Go. It persists to disk, is ACID compliant, and uses locking for multiple readers and a sing

Dec 30, 2022
ACID key-value database.

Coffer Simply ACID* key-value database. At the medium or even low latency it tries to provide greater throughput without losing the ACID properties of

Dec 7, 2022
LevelDB key/value database in Go.

This is an implementation of the LevelDB key/value database in the Go programming language. Installation go get github.com/syndtr/goleveldb/leveldb R

Jan 1, 2023
Graviton Database: ZFS for key-value stores.
Graviton Database: ZFS for key-value stores.

Graviton Database: ZFS for key-value stores. Graviton Database is simple, fast, versioned, authenticated, embeddable key-value store database in pure

Dec 31, 2022
Membin is an in-memory database that can be stored on disk. Data model smiliar to key-value but values store as JSON byte array.

Membin Docs | Contributing | License What is Membin? The Membin database system is in-memory database smiliar to key-value databases, target to effici

Jun 3, 2021
BadgerDB is an embeddable, persistent and fast key-value (KV) database written in pure Go
BadgerDB is an embeddable, persistent and fast key-value (KV) database written in pure Go

BadgerDB BadgerDB is an embeddable, persistent and fast key-value (KV) database written in pure Go. It is the underlying database for Dgraph, a fast,

Dec 10, 2021
GalaxyDB is a hobbyist key-value database written in Go.

GalaxyDB GalaxyDB is a hobbyist key-value database written in Go Author: Andrew N ([email protected]) Features Data is stored via keys Operations Grafana

Mar 30, 2022
rosedb is an embedded and fast k-v database based on LSM + WAL
rosedb is an embedded and fast k-v database based on LSM + WAL

A simple k-v database in pure Golang, supports string, list, hash, set, sorted set.

Dec 30, 2022
Fast key-value DB in Go.
Fast key-value DB in Go.

BadgerDB BadgerDB is an embeddable, persistent and fast key-value (KV) database written in pure Go. It is the underlying database for Dgraph, a fast,

Dec 29, 2022
🔑A high performance Key/Value store written in Go with a predictable read/write performance and high throughput. Uses a Bitcask on-disk layout (LSM+WAL) similar to Riak.

bitcask A high performance Key/Value store written in Go with a predictable read/write performance and high throughput. Uses a Bitcask on-disk layout

Sep 26, 2022
A disk-backed key-value store.

What is diskv? Diskv (disk-vee) is a simple, persistent key-value store written in the Go language. It starts with an incredibly simple API for storin

Jan 7, 2023
An in-memory key:value store/cache (similar to Memcached) library for Go, suitable for single-machine applications.

go-cache go-cache is an in-memory key:value store/cache similar to memcached that is suitable for applications running on a single machine. Its major

Dec 29, 2022
A simple, fast, embeddable, persistent key/value store written in pure Go. It supports fully serializable transactions and many data structures such as list, set, sorted set.

NutsDB English | 简体中文 NutsDB is a simple, fast, embeddable and persistent key/value store written in pure Go. It supports fully serializable transacti

Jan 1, 2023
Fast and simple key/value store written using Go's standard library
Fast and simple key/value store written using Go's standard library

Table of Contents Description Usage Cookbook Disadvantages Motivation Benchmarks Test 1 Test 4 Description Package pudge is a fast and simple key/valu

Nov 17, 2022
Low-level key/value store in pure Go.
Low-level key/value store in pure Go.

Description Package slowpoke is a simple key/value store written using Go's standard library only. Keys are stored in memory (with persistence), value

Jan 2, 2023