Package ring provides a high performance and thread safe Go implementation of a bloom filter.

ring - high performance bloom filter

Build Status codecov Go Report Card PkgGoDev GitHub license Mentioned in Awesome Go

Package ring provides a high performance and thread safe Go implementation of a bloom filter.

Usage

Please see the godoc for usage.

Accuracy

Running make will perform unit tests, comparing the target false positive rate with the actual rate. Here is a test against 1 million elements with a targeted false positive rate of 0.1%. Tests fail if the number of false positives exceeds the target.

=== RUN   TestBadParameters
--- PASS: TestBadParameters (0.00s)
=== RUN   TestReset
--- PASS: TestReset (0.26s)
=== RUN   TestData
--- PASS: TestData (14.07s)
=== RUN   TestMerge
--- PASS: TestMerge (13.78s)
=== RUN   TestMarshal
--- PASS: TestMarshal (14.48s)
PASS
>> Number of elements:  1000000
>> Target false positive rate:  0.001000
>> Number of false positives:  99
>> Actual false positive rate:  0.000099
>> Number of false negatives:  0
>> Actual false negative rate:  0.000000
>> Benchmark Add():  10000000          158 ns/op
>> Benchmark Test():  10000000         173 ns/op
ok      command-line-arguments  47.914s

License

Copyright (c) 2019 Tanner Ryan. All rights reserved. Use of this source code is governed by a BSD-style license that can be found in the LICENSE file.

Owner
Tanner Ryan
Network strategy @cloudflare. Cyborg, network and security stuff. Flowing with @golang.
Tanner Ryan
Comments
  • Module path should be

    Module path should be "github.com/TheTannerRyan/ring", not "github.com/thetannerryan/ring"

    Background

    Module path is inconsistent with go import path. GO111MODULE=on, as doc said, import "github.com/TheTannerRyan/ring", then get this error:

    go: finding module for package github.com/TheTannerRyan/ring
    go: downloading github.com/TheTannerRyan/ring v1.1.1
    go: found github.com/TheTannerRyan/ring in github.com/TheTannerRyan/ring v1.1.1
    go: test1 imports
            github.com/TheTannerRyan/ring: github.com/TheTannerRyan/[email protected]: parsing go.mod:
            module declares its path as: github.com/thetannerryan/ring
                    but was required as: github.com/TheTannerRyan/ring 
    

    Solution

    Fix the module path:

    1. Rename the module path to "github.com/TheTannerRyan/ring": https://github.com/TheTannerRyan/ring/blob/master/go.mod#L1
    module github.com/TheTannerRyan/ring
    go 1.12 
    
    1. Consider reverting the thetannerryan -> TheTannerRyan rename.
    2. Change the doc document to use import "github.com/thetannerryan/ring".
  • The bitset doesn't return the 0s and 1s

    The bitset doesn't return the 0s and 1s

    Hello! Sorry for concerning, but can't understand why Marshal.Binary returns [1 0 0 0 0 0 0 3 191 0 0 0 0 0 0 0 7 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 32 0 0 0 0 0 0 0 0 32 0 0 128 16 0 0 0 0 0 0 0 36 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 16 0 0 0 0 0 0 0 4 0 0 0 64 0 0 0 0 0 0 16 0 64 4 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 4 0] this, fro where this numbers "191, 3,36, ...", Bloom Filter isn't bitset which should give "101010101" this type of bitset? Thank you in advance.

  • Added Merge, MarshalBinary, UnmarshalBinary

    Added Merge, MarshalBinary, UnmarshalBinary

    We need to be able to merge multiple different Ring's from different hours of data since we process our data in hourly chunks concurrently. The Marshal/Unmarshal methods will be necessary so we can store already-processed hours in a database so we don't need to re-compute them all of the time.

    I added tests to test the Merge and Marshal methods but I made them separate tests. They're still ran by TestMain but they don't affect the stats output.

    /cc @TheTannerRyan

  • generateMultiHash changes source byte array

    generateMultiHash changes source byte array

    unit test below fails

    func TestRing(t *testing.T) {
            assert := assert.New(t)
    
            filter, err := ring.Init(1000, 0.03)
            assert.NoError(err)
    
            b := []byte{0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00}
            d := []byte{0x77, 0x50, 0x52, 0x48, 0x57, 0x32, 0x32, 0x77}
    
            copy(b, d)
            assert.False(filter.Test(b[:3]))
            assert.Equal(b, d)
    
            copy(b, d)
            filter.Add(b[:3])
            assert.Equal(b, d)
    
            copy(b, d)
            filter.Add(b[:3])
            assert.Equal(b, d)
    
    }
    

    part of output:

                    Error:          Not equal:
                                    expected: []byte{0x77, 0x50, 0x52, 0x1, 0x57, 0x32, 0x32, 0x77}
                                    actual  : []byte{0x77, 0x50, 0x52, 0x48, 0x57, 0x32, 0x32, 0x77}
    
                                    Diff:
                                    --- Expected
                                    +++ Actual
                                    @@ -1,3 +1,3 @@
                                     ([]uint8) (len=8) {
                                    - 00000000  77 50 52 01 57 32 32 77                           |wPR.W22w|
                                    + 00000000  77 50 52 48 57 32 32 77                           |wPRHW22w|
                                     }
                    Test:           TestRing
    
    

    It caused by this line:

    h3, h4 := murmur128(append(data, single))
    
  • Add parameterizable Init funciton

    Add parameterizable Init funciton

    I am an engineer at Elixxir, and we are considering using your bloom filter as part of our system. However, we want to use a more customizable bloom filter, in which we can input our own values for the size of the bit array and the number of hash rounds.Thus, we are presenting this pull request to you for your consideration.

go:embed and the golang-standards project layout

An example of using the golang-standards project layout and the go:embed directive.

May 17, 2022
Go concurrent-safe, goroutine-safe, thread-safe queue
Go concurrent-safe, goroutine-safe, thread-safe queue

goconcurrentqueue - Concurrent safe queues The package goconcurrentqueue offers a public interface Queue with methods for a queue. It comes with multi

Dec 31, 2022
Collection of high performance, thread-safe, lock-free go data structures

Garr - Go libs in a Jar Collection of high performance, thread-safe, lock-free go data structures. adder - Data structure to perform highly-performant

Dec 26, 2022
A Go implementation of an in-memory bloom filter, with support for boltdb and badgerdb as optional data persistent storage.

Sprout A bloom filter is a probabilistic data structure that is used to determine if an element is present in a set. Bloom filters are fast and space

Jul 4, 2022
A simple Bloom Filter implementation in Go

This is a simple Bloom filter implementation written in Go. For the theory behind Bloom filters, read http://en.wikipedia.org/wiki/Bloom_filter This

Apr 26, 2018
This provides the lru package which implements a fixed-size thread safe LRU cache

golang-lru This provides the lru package which implements a fixed-size thread sa

Dec 22, 2021
Cuckoo Filter: Practically Better Than Bloom

Cuckoo Filter Cuckoo filter is a Bloom filter replacement for approximated set-membership queries. While Bloom filters are well-known space-efficient

Jan 4, 2023
go/golang: fast bit set Bloom filter

package implements a fast bloom filter with real 'bitset' and JSONMarshal/JSONUnmarshal to store/reload the Bloom filter.

Nov 3, 2022
Leaked password check library with bloom filter

Leaked password check With this library you can check the password is probably leaked or not. Pre generated bitset DB includes 6 Million leaked passwo

Aug 24, 2022
A bloom filter is a probabilistic data structure that is used to determine if an element is present in a set

A Go implementation of a bloom filter, with support for boltdb and badgerdb as optional in-memory persistent storage.

Jul 4, 2022
go.fifo provides a simple fifo thread-safe queue for the Go programming language

go.fifo Description go.fifo provides a simple FIFO thread-safe queue. *fifo.Queue supports pushing an item at the end with Add(), and popping an item

Aug 29, 2022
Routines was a fixed number thread pool to process the user task, and it would respawn a corresponding new thread when panic

Routines Routines was a fixed number thread pool to process the user task, and it would respawn a corresponding new thread when panic. It supports the

Dec 16, 2021
Lightweight, Simple, Quick, Thread-Safe Golang Stack Implementation

stack Lightweight, Simple, Quick, Thread-Safe Golang Stack Implementation Purpose Provide a fast, thread safe, and generic Golang Stack API with minim

May 3, 2022
Supports the safe and convenient execution of asynchronous computations with goroutines and provides facilities for the safe retrieval of the computation results.

Rendezvous The Rendezvous library supports the safe and convenient execution of asynchronous computations with goroutines and provides facilities for

Dec 29, 2021
🔑A high performance Key/Value store written in Go with a predictable read/write performance and high throughput. Uses a Bitcask on-disk layout (LSM+WAL) similar to Riak.

bitcask A high performance Key/Value store written in Go with a predictable read/write performance and high throughput. Uses a Bitcask on-disk layout

Sep 26, 2022
LinDB is an open-source Time Series Database which provides high performance, high availability and horizontal scalability.
LinDB is an open-source Time Series Database which provides high performance, high availability and horizontal scalability.

LinDB is an open-source Time Series Database which provides high performance, high availability and horizontal scalability. LinDB stores all monitoring data of ELEME Inc, there is 88TB incremental writes per day and 2.7PB total raw data.

Jan 1, 2023
Caddy log filter module with a log field filter to extract the user from a basic Authorization HTTP-Header

caddy-basic-auth-filter This packages contains a log field filter to extract the user from a basic Authorization HTTP-Header. Installation xcaddy buil

May 10, 2022
the pluto is a gateway new time, high performance, high stable, high availability, easy to use

pluto the pluto is a gateway new time, high performance, high stable, high availability, easy to use Acknowledgments thanks nbio for providing low lev

Sep 19, 2021
A simple and efficient thread-safe sharded hashmap for Go

shardmap A simple and efficient thread-safe sharded hashmap for Go. This is an alternative to the standard Go map and sync.Map, and is optimized for w

Dec 17, 2022