Multi producer and multi-reader lockless ring buffer in go using generics from the go 1.18beta release

LocklessGenericRingBuffer

This is an implementation of a multi producer, multi reader lockless ring buffer utilizing the new generics available in go 1.18. Instead of passing typeless interface{} which we have to assert or deserialized []byte we are able to pass serialized structs between go routines in a type safe manner.

Requirements

  • golang 1.18beta
Owner
Gavin Clarke
Western Computer Science
Gavin Clarke
Comments
  • Remove Consumer Lock, Simplify Indexing of Read/Write And Reduce Write Branching Operation

    Remove Consumer Lock, Simplify Indexing of Read/Write And Reduce Write Branching Operation

    => Change reader, writer index logic to now mean that the value of the index is the last value read/written to. => Change loop operations on readers to now yield on first indication write is not currently possible. Replace loop and if statements with goto to retry write on failure reducing branching on successful and yielding write operations. This improves speed significantly in concurrent environments. => refactor buffer itself to use consistent naming for reader value => make tests use same implementation to ensure read/write order is consistent => Update readme with new benchmarks

  • Simplify Consumer Logic, Add more tests for removing consumers, fix race conditions

    Simplify Consumer Logic, Add more tests for removing consumers, fix race conditions

    • add new flag method to determine if a consumer index is in use, converting pointer array to int array improving latency/cache hits
    • add test for removing a consumer in a concurrent environment
    • improve readme with additional examples + benchmark explanation.
    • remove channel/ringbuffer creation time from benchmark to better simulate use case
  • Add benchmarks

    Add benchmarks

    • Fixed race conditions due to not atomically reading on atomically written variables. Test -race shows no race conditions for current tests.
    • Fixed race conditions between writing and add/remove consumers that currently, tests do not exist for, but are theoretically identified. This will need more work.

    TODO:

    • create tests for theoretical race conditions between writes/reads and add/remove consumers
  • Pointer consumers

    Pointer consumers

    Allowed removing consumers and determining if they are active via comparison with nil. Pointer Array and skipping comparisons on nil has sufficient speed-ups to allow approach.

  • Multi-process support?

    Multi-process support?

    I am curious if this supports multi-process cases where the ring buffer is shared memory (either shared mmap or shmem). Have you tested this, is it meant to be supported?

High-performance Redis-Server multi-threaded framework, based on rawepoll model.
High-performance Redis-Server multi-threaded framework, based on rawepoll model.

RedHub High-performance Redis-Server multi-threaded framework, based on RawEpoll model. Ultra high performance Fully multi-threaded support Low CPU re

Dec 18, 2022
NoSql DB using fileSystems using golang

No SQL DB using Go Prerequisite go 1.15 Run test go test -v ./... Env Var Variable Description Default Value Possible Values DB_DATA dir location to s

Nov 8, 2021
Fast and simple key/value store written using Go's standard library
Fast and simple key/value store written using Go's standard library

Table of Contents Description Usage Cookbook Disadvantages Motivation Benchmarks Test 1 Test 4 Description Package pudge is a fast and simple key/valu

Nov 17, 2022
Simple DB using yaml. A project for managing the content of yaml files.

DB Yaml Simple DB using yaml. A project for managing the content of yaml files. Table of Contents DB Yaml Features Usage Write to DB Query DB Get Firs

Dec 27, 2022
Toy project to test golang toolset, using as pretext the "cyclic number" problem (famous 142857 number)

go-cyclic-number Toy project to test golang toolset, using as pretext the "cyclic number" problem (famous 142857 number) First version: https://github

Feb 12, 2022
Owl is a db manager platform,committed to standardizing the data, index in the database and operations to the database, to avoid risks and failures.

Owl is a db manager platform,committed to standardizing the data, index in the database and operations to the database, to avoid risks and failures. capabilities which owl provides include Process approval、sql Audit、sql execute and execute as crontab、data backup and recover .

Nov 9, 2022
Being played at The Coffee House and try to find and play it on Spotify
Being played at The Coffee House and try to find and play it on Spotify

The Coffee House Muzik Follow the music that is being played at The Coffee House and try to find and play it on Spotify. Installation Clone this proje

May 25, 2022
Walrus - Fast, Secure and Reliable System Backup, Set up in Minutes.
Walrus - Fast, Secure and Reliable System Backup, Set up in Minutes.

Walrus is a fast, secure and reliable backup system suitable for modern infrastructure. With walrus, you can backup services like MySQL, PostgreSQL, Redis, etcd or a complete directory with a short interval and low overhead. It supports AWS S3, digitalocean spaces and any S3-compatible object storage service.

Jan 5, 2023
🔑A high performance Key/Value store written in Go with a predictable read/write performance and high throughput. Uses a Bitcask on-disk layout (LSM+WAL) similar to Riak.

bitcask A high performance Key/Value store written in Go with a predictable read/write performance and high throughput. Uses a Bitcask on-disk layout

Sep 26, 2022
BuntDB is an embeddable, in-memory key/value database for Go with custom indexing and geospatial support
BuntDB is an embeddable, in-memory key/value database for Go with custom indexing and geospatial support

BuntDB is a low-level, in-memory, key/value store in pure Go. It persists to disk, is ACID compliant, and uses locking for multiple readers and a sing

Dec 30, 2022
Concurrency-safe Go caching library with expiration capabilities and access counters

cache2go Concurrency-safe golang caching library with expiration capabilities. Installation Make sure you have a working Go environment (Go 1.2 or hig

Dec 31, 2022
groupcache is a caching and cache-filling library, intended as a replacement for memcached in many cases.

groupcache Summary groupcache is a distributed caching and cache-filling library, intended as a replacement for a pool of memcached nodes in many case

Dec 29, 2022
Scalable datastore for metrics, events, and real-time analytics

InfluxDB InfluxDB is an open source time series platform. This includes APIs for storing and querying data, processing it in the background for ETL or

Jan 5, 2023
Kivik provides a common interface to CouchDB or CouchDB-like databases for Go and GopherJS.

Kivik Package kivik provides a common interface to CouchDB or CouchDB-like databases. The kivik package must be used in conjunction with a database dr

Dec 29, 2022
A simple, fast, embeddable, persistent key/value store written in pure Go. It supports fully serializable transactions and many data structures such as list, set, sorted set.

NutsDB English | 简体中文 NutsDB is a simple, fast, embeddable and persistent key/value store written in pure Go. It supports fully serializable transacti

Jan 1, 2023
The Prometheus monitoring system and time series database.

Prometheus Visit prometheus.io for the full documentation, examples and guides. Prometheus, a Cloud Native Computing Foundation project, is a systems

Dec 31, 2022
Fast specialized time-series database for IoT, real-time internet connected devices and AI analytics.
Fast specialized time-series database for IoT, real-time internet connected devices and AI analytics.

unitdb Unitdb is blazing fast specialized time-series database for microservices, IoT, and realtime internet connected devices. As Unitdb satisfy the

Jan 1, 2023
VictoriaMetrics: fast, cost-effective monitoring solution and time series database
VictoriaMetrics: fast, cost-effective monitoring solution and time series database

VictoriaMetrics VictoriaMetrics is a fast, cost-effective and scalable monitoring solution and time series database. It is available in binary release

Jan 8, 2023
The MySQL Cluster Autopilot Management with GTID and Raft
The MySQL Cluster Autopilot Management with GTID and Raft

Xenon Overview Xenon is a MySQL HA and Replication Management tool using Raft protocol. Xenon has many cool features, such as: Fast Failover with no l

Jan 3, 2023