golang实现的分布式唯一ID生成器distributed id generator,有全局趋势递增、严防时钟漂移、高可用、高性能等特点

ekko-idgenerator是什么

顾名思义,ekko是一个分布式唯一ID生成器,参考了snowFlake思想,但是并不局限于其设计。

名称由来

英雄联盟的时间刺客ekko

arch

特点

  • 易用,最大限度保证系统的易用性,支持Get与MultiGet;
  • 高并发,单机每秒100w个唯一ID生成;
  • 高可用,理论上没有强依赖任何其他服务;最多可拓展至达4096个物理机;且有生成ID的兜底方案;
  • 多维度,支持机房级别、产品线级别区分;
  • 不受时钟漂移影响,通过多种方式规避时钟漂移;
  • 轻量级,支持多种方式分配WorkerID进行部署,总有一种方式适合你;

UID组成

唯一ID是uint64类型,并保证ID在趋势上全局递增,其中:

39 bits: 从发布至今经历的毫秒数,系统可运行17年;

3 bits: 机房标识,可支持8个机房;

12 bits: 物理机workerID + 产品线,两者共占12bits,具体划分是可配置的。

  • 产品线可以配置0-4bits, 0表示不区分产品线
  • 剩余bits用于表示workerID,故可支持最多256~4096台物理机部署

10 bits:毫秒内并发计数,一毫秒内可支持1024个ID,即QPS = 1024*1000 约为 100W

|------------------------------------------------------------------------------|
|                     Ekko ID generator design                                 |
|------------------------------------------------------------------------------|
| 64bits                                                                       |
| : 39bits for timestamp                                                       |
| : 3bits for idc                                                              |
| : 8~12bits for workerID                                                      |
| : 0~4bits for products                                                       |
| : 10bits for currency                                                        |
|------------------------------------------------------------------------------|
|                                                                              |
| @if 8bits for workerID                                                       |
|  [.......................................][...][........][....][..........]  |
|  [                timestamp              ][idc][workerID][prod][ currency ]  |
|                                                                              |
| @if 12bits for workerID, no product                                          |
|  [.......................................][...][........][....][..........]  |
|  [                timestamp              ][idc][   workerID   ][ currency ]  |
|                                                                              |
|------------------------------------------------------------------------------|

workerID分配

支持多种方式为物理机分配workerID:

  • 手动分配;
  • 基于mysql主键方式自动分配;
  • 基于redis incrby方式自动分片;
  • 基于etcd自动分配;

时间漂移

有两种手段共同判定时间漂移

  • 本地共享内存:记录上次成功生成UID的时间戳+并发,下次生成UID的时间戳+并发必须比共享内存中的值大,否则不服务;
  • 周期上报(default 3s):周期性将本地时间上报到mysql/redis/etcd

防止时间漂移主要依赖第一种手段;放在共享内存可以使得服务重启后数据也有效;

而第二种手段更多用于服务启动时检查,是否此时时间大于上次上报时间,以及是否与其他机器保持了合理的时间戳diff;

TODO

  • 各模式的测试
Similar Resources

An enum generator for go

go-enum An enum generator for go How it works The goal of go-enum is to create an easy to use enum generator that will take a decorated type declarati

Dec 22, 2022

Jennifer is a code generator for Go

Jennifer Jennifer is a code generator for Go. package main import ( "fmt" . "github.com/dave/jennifer/jen" ) func main() { f := NewFile("m

Jan 4, 2023

A protoc-gen-go wrapper including an RPC stub generator

// Copyright 2013 Google. All rights reserved. // // Use of this source code is governed by a BSD-style // license that can be found in the LICENSE fi

Nov 17, 2022

Port of D. J. Bernstein's primegen prime number generator to Go

primegen primegen is a Go package that generates prime numbers in order using the Sieve of Atkin instead of the traditional Sieve of Eratosthenes. It

Jul 20, 2021

Random fake data and struct generator for Go.

Faker Random fake data and struct generator for Go. More than 100 generator functions Struct generator Unique data generator Builtin types support Eas

Oct 3, 2022

Random fake data generator written in go

Random fake data generator written in go

Gofakeit Random data generator written in go Features 160+ Functions!!! Concurrent Global Rand Struct Generator Custom Functions Http Server Command L

Jan 1, 2023

A distributed unique ID generator of using Sonyflake and encoded by Base58

Indigo About A distributed unique ID generator of using Sonyflake and encoded by Base58. ID max length is 11 characters by unsigned int64 max value. A

Nov 24, 2022

Password generator written in Go

go-generate-password Password generator written in Go. Use as a library or as a CLI. Usage CLI go-generate-password can be used on the cli, just insta

Dec 19, 2022

A PDF document generator with high level support for text, drawing and images

A PDF document generator with high level support for text, drawing and images

GoFPDF document generator Package gofpdf implements a PDF document generator with high level support for text, drawing and images. Features UTF-8 supp

Dec 28, 2022

iTunes and RSS 2.0 Podcast Generator in Golang

podcast Package podcast generates a fully compliant iTunes and RSS 2.0 podcast feed for GoLang using a simple API. Full documentation with detailed ex

Dec 23, 2022

A tiny and fast Go unique string generator

Nano ID A tiny and fast Go unique string generator Safe. It uses cryptographically strong random APIs and tests distribution of symbols. Compact. It u

Nov 11, 2022

Parser and generator of M3U8-playlists for Apple HLS. Library for Go language. :cinema:

M3U8 This is the most complete opensource library for parsing and generating of M3U8 playlists used in HTTP Live Streaming (Apple HLS) for internet vi

Dec 20, 2022

PlantUML Class Diagram Generator for golang projects

PlantUML Class Diagram Generator for golang projects

GoPlantUML PlantUML Class Diagram Generator for golang projects. Generates class diagram text compatible with plantuml with the information of all str

Dec 31, 2022

:guardsman: A teeny tiny and somewhat opinionated generator for your next golang project

A Yeoman Golang Generator We are very sorry Gophers, but other names for the generator where taken, so we choose go-lang. But we have gocreate as an a

Sep 27, 2022

Unit tests generator for Go programming language

Unit tests generator for Go programming language

GoUnit GoUnit is a commandline tool that generates tests stubs based on source function or method signature. There are plugins for Vim Emacs Atom Subl

Jan 1, 2023

XSD (XML Schema Definition) parser and Go/C/Java/Rust/TypeScript code generator

xgen Introduction xgen is a library written in pure Go providing a set of functions that allow you to parse XSD (XML schema definition) files. This li

Jan 1, 2023

HTTP load generator, ApacheBench (ab) replacement, formerly known as rakyll/boom

HTTP load generator, ApacheBench (ab) replacement, formerly known as rakyll/boom

hey is a tiny program that sends some load to a web application. hey was originally called boom and was influenced from Tarek Ziade's tool at tarekzia

Jan 3, 2023

GObject-introspection based bindings generator

WARNING! This project is no longer maintained. Probably doesn't even compile. GObject-introspection based bindings generator for Go. Work in progress

Jan 5, 2022

Typo/error resilient, human-readable token generator

toktok A human-friendly token generator Creates tokens which avoid characters that can be easily misinterpreted, like '1' and 'I' or '8' and 'B', as w

Sep 16, 2022
High performance unique number generator powered by Go

SEQSVR High performance unique number generator powered by Go 中文 README Features Distributed: Can be scaled horizontally High performance: Allocation

Nov 16, 2022
❄ An Lock Free ID Generator for Golang based on Snowflake Algorithm (Twitter announced).
❄ An Lock Free ID Generator for Golang based on Snowflake Algorithm (Twitter announced).

❄ An Lock Free ID Generator for Golang based on Snowflake Algorithm (Twitter announced).

Dec 14, 2022
Snowflake - Simple twitter's snowflake uniquely identifiable descriptors (IDs) format generator for Go

Snowflake Dead simple and fast Twitter's snowflake id generator in Go. Installation go get github.com/HotPotatoC/snowflake Usage Generating a snowflak

Oct 6, 2022
A generator library for concise, unambiguous and URL-safe UUIDs

shortuuid A Go library that generates concise, unambiguous, URL-safe UUIDs. Based on and compatible with the Python library shortuuid. Often, one need

Jan 4, 2023
DND-magic-item-Generator - D&D magic item generator like in Diablo

DND-magic-item-Generator D&D magic item generator like in Diablo Legendary items

Mar 28, 2022
A fast data generator that's multi-table aware and supports multi-row DML.
A fast data generator that's multi-table aware and supports multi-row DML.

If you need to generate a lot of random data for your database tables but don't want to spend hours configuring a custom tool for the job, then datage

Dec 26, 2022
Super fast static photo and video gallery generator (written in Go and HTML/CSS/native JS)

fastgallery Fast static photo and video gallery generator Super fast (written in Go and C, concurrent, uses fastest image/video libraries, 4-8 times f

Dec 4, 2022
OpenGL binding generator for Go

GoGL GoGL is an OpenGL binding generator for Go. No external dependencies like GLEW are needed. Install the OpenGL bindings For example, OpenGL 2.1 bi

Dec 25, 2022
An avatar generator for Go.

Cameron An avatar generator for Go. Oh, by the way, the name of this project came from the Avatar's director James Cameron. Features Identicon Install

Dec 25, 2022
Versatile Go code generator.
Versatile Go code generator.

Generis Versatile Go code generator. Description Generis is a lightweight code preprocessor adding the following features to the Go language : Generic

Nov 30, 2022