go/golang: fast bit set Bloom filter

Casting to a uintptr, saving the uintptr, and then casting it back is unsound.

https://github.com/AndreasBriese/bbloom/blob/e2d15f34fcf99d5dbb871c820ec73f710fca9815/bbloom.go#L80-L88

https://github.com/AndreasBriese/bbloom/blob/e2d15f34fcf99d5dbb871c820ec73f710fca9815/bbloom.go#L232-L246

Hi Andreas,

congrats for your work on bbloom, I am integrating it at the core of my new project. I was wondering if you were planning to add methods to calculate union and intersection of 2 bloom filters or if you could put me in the right direction to learn how to do it.

Let me know, best, Romain.

Hey @AndreasBriese , for travis:

Your's travis builds were failing because of uncompatible old version of go defined in .travis.yml file and because words.txt is not uploaded to the repository, so travis can't actually find when it runs the TestMain test method (the last is fixed by stopping the test without error if os.Open("words.txt") failed).

go: 1.1 will make use the go.1 version not the go.11, the correct way for what you wanted to do with your commit: 343706a is to represent the version as 1.11.x instead.

This may be a stupid question, but I'd very much appreciate your help. I want to implement a page view counter that will be pushed to a database every X hours. Before it gets pushed, then I intend to store it in memory and make sure that the same device (checked using IP + UA) doesn't count for multiple views (unless 1000 other people have visited the page, or it has been X hours since last visit)

I have the following code:

package main

import (
	"fmt"
	"sync"
)

type page struct {
	mu     *sync.Mutex
	values map[string]int64
}

var pageCounters = page{
	values: make(map[string]int64),
}

func (p *page) Delete(key string) {
	p.mu.Lock()
	defer p.mu.Unlock()
	delete(pageCounters.values, key)
}

func (p *page) Get(key string) int64 {
	p.mu.Lock()
	defer p.mu.Unlock()
	return p.values[key]
}

func (p *page) Incr(key string) int64 {
	p.mu.Lock()
	defer p.mu.Unlock()
	p.values[key]++
	return p.values[key]
}

func main() {
	_ = pageCounters.Incr("/some/url/here")
	_ = pageCounters.Incr("/another/url")
	_ = pageCounters.Incr("/some/url/here")
	
	fmt.Println(pageCounters.Get("/some/url/here"))
	fmt.Println(pageCounters.Get("/another/url"))
	fmt.Println(pageCounters)
	pageCounters.Delete("/another/url")
	fmt.Println(pageCounters)

}

The plan now is to update the page struct to include a bloomfilter field, I'm thinking it would be done like this:

type page struct {
	mu     *sync.Mutex
	values map[string]int64
	bloom  bbloom.Bloom
}

I'm now wondering how I should update the Incr func so that:

• if a bloom filter doesn't exist, then it will create one containing 1000 items
• each item will be a combination of user IP address + User Agent (curious how you would recommend storing it?)
• when the bloom filters items have been filled up, then the bloom filter should either be deleted and replaced by a new bloom filter, or have the previous items be overridden (I assume this is impossible)

I'm thinking it would look something like this:

func (p *page) Incr(key, ipUA string) int64 {
	p.mu.Lock()
	defer p.mu.Unlock()
	if (p.values[key] == 0) {
		p.bloom = bbloom.New(1000.0, 7.0) // probably use wrong values here
	}
	added := p.bloom.AddIfNotHasTS([]byte(ipUA))
	if added == true {
		p.values[key]++
	}
	return p.values[key]
}

Right now then I don't really know what will happen if the bloomfilter is full, and I'm not sure how to check for this and how to replace it. I guess if (p.values[key] == 0) could become if (p.values[key] == 0 || p.bloom.Size == 999)?

I also wonder, would there be any benefit to store the bloom as JSON? Keep in mind that each website page will have its own filter and they'll only be deleted after X hours (meaning there could potentially be many thousands of page structs initialized at any given moment)

I hope you don't mind me abusing your repo & time with this question