gopkg is a universal utility collection for Go, it complements offerings such as Boost, Better std, Cloud tools.

gopkg

gopkg is a universal utility collection for Go, it complements offerings such as Boost, Better std, Cloud tools.

Table of Contents

Introduction

gopkg is a universal utility collection for Go, it complements offerings such as Boost, Better std, Cloud tools. It is migrated from the internal code base at ByteDance and has been extensively adopted in production.

We depend on the same code(this repo) in our production environment.

Catalogs

  • cache: Caching Mechanism
  • cloud: Cloud Computing Design Patterns
  • collection: Data Structures
  • lang: Enhanced Standard Libraries
  • util: Utilities Useful across Domains

Releases

gopkg recommends users to "live-at-head" (update to the latest commit from the main branch as often as possible). We develop at develop branch and will only merge to main when develop is stable.

How To Use

You can use go get -u github.com/bytedance/gopkg@main to get or update gopkg.

License

gopkg is licensed under the terms of the Apache license 2.0. See LICENSE for more information.

Owner
Comments
  • feat(skipmap): generate the level lazily in LoadOrStore

    feat(skipmap): generate the level lazily in LoadOrStore

    The key problem is that the LoadOrStore uses an outdated highest level to search the skip list, if the new generated level(say newLevel) is bigger than the outdated highest level(say oldLevel), all slots in the slices preds[newLevel-oldLevel:newLevel] are nil, the function will panic.

    The solution is that if the newLevel has updated the highest level, we can just continue the loop, and find a new path. At this time, the latest highest level used by the findNode is always bigger than or equal to the newLevel, the function won't panic.

    name                            old time/op  new time/op  delta
    LoadOrStoreExist-16             4.43ns ±11%  1.06ns ±21%  -76.05%  (p=0.000 n=10+10)
    LoadOrStoreLazyExist-16         4.61ns ± 6%  1.16ns ± 0%  -74.90%  (p=0.000 n=9+7)
    LoadOrStoreExistSingle-16       37.5ns ± 6%  10.3ns ± 0%  -72.61%  (p=0.000 n=9+9)
    LoadOrStoreLazyExistSingle-16   38.3ns ±11%  10.6ns ± 0%  -72.43%  (p=0.000 n=10+10)
    LoadOrStoreRandom-16             209ns ±14%   206ns ±14%     ~     (p=0.684 n=10+10)
    LoadOrStoreLazyRandom-16         215ns ± 8%   207ns ± 9%     ~     (p=0.139 n=10+10)
    LoadOrStoreRandomSingle-16      1.05µs ± 1%  1.04µs ± 4%     ~     (p=0.535 n=9+10)
    LoadOrStoreLazyRandomSingle-16  1.07µs ± 1%  1.06µs ± 3%   -1.09%  (p=0.029 n=8+9)
    
  • ci: add github workflow for performance regression check

    ci: add github workflow for performance regression check

    This PR add a github workflow for performance regression check (using benchdiff, a wrapper of benchstat).

    Close #28.

    Usage

    The workflow will be triggered:

    • when a PR is created, or
    • anyone comments "/benchdiff" on PR

    It takes a few minutes to run (the time is depends on the go packages modified by this PR), once it is finished, benchdiff result will be posted on this PR:

    image

    Non-full amount

    This workflow won't do a full amount benchmarking, it only do bench on modified go packages (get by git diff <default_branch> <head_of_pr>, for details, refer to jobs.benchdiff.steps.head). In the aboved case, only packages collection/skipset and collection/skipmap are modified. If there is no go package modified, the following comment will be posted:

    image

    Github Checks

    This workflow also creates a check for the HEAD commit of PR.

    image

    The details of check looks like this:

    image

  • About gopoll benchmark

    About gopoll benchmark

    Question

    请问有没有gopool和通过channel实现goroutine pool的性能对比,个人认为这个锁太大了: 1、 P的数量越多,channel的优势会优于锁 2、 如果goroutine创建速度快,锁等待会更高

    感觉参照runtime,通过runtime.LockOSThread单独占几个系统线程,去处理这件事情(每次从全局的tasks中拿1/n的task),效率会更高,因为不会被runtime抢占。

  • feat(lscq): add arm64 support

    feat(lscq): add arm64 support

    Using CASP instruction to implement double-width CAS for arm64. The CASP instruction is available for instruction set Armv8.1+ (inclusive, Apple M1/M2/A12, Snapdragon 845, etc). All tests in lscq_test.go passed on Macbook Air M1 2020, macOS 12.5.1.

    Change-Id: Ieb89fa9361f1fce8fb52b102dca556867f7e8e8a

  • feat: add zset

    feat: add zset

    Intro

    Package zset provides a concurrent-safety sorted set, can be used as a local replacement of Redis' zset (https://redis.com/ebook/part-2-core-concepts/chapter-3-commands-in-redis/3-5-sorted-sets/).

    The main different to other sets is, every value of set is associated with a score, that is used in order to take the sorted set ordered, from the smallest to the greatest score.

    The sorted set has O(log(N)) time complexity when doing Add(ZADD) and Remove(ZREM) operations and O(1) time complexity when doing Contains operations.

    Package zset is a go implmentation of https://github.com/redis/redis/blob/unstable/src/t_zset.c

    Comparison

    | Redis command | Go function | |-----------------------|---------------------| | ZADD | Add | | ZINCRBY | IncrBy | | ZREM | Remove | | ZREMRANGEBYSCORE | RemoveRangeByScore | | ZREMRANGEBYRANK | RemoveRangeByRank | | ZUNION | Union | | ZINTER | Inter | | ZINTERCARD | TODO | | ZDIFF | TODO | | ZRANGE | Range | | ZRANGEBYSCORE | IncrBy | | ZREVRANGEBYSCORE | RevRangeByScore | | ZCOUNT | Count | | ZREVRANGE | RevRange | | ZCARD | Len | | ZSCORE | Score | | ZRANK | Rank | | ZREVRANK | RevRank | | ZPOPMIN | TODO | | ZPOPMAX | TODO | | ZRANDMEMBER | TODO |

    List of redis commands are generated from the following command:

    cat redis/src/server.c | grep -o '"z.*",z.*Command' | grep -o '".*"' | cut -d '"' -f2
    

    You may find that not all redis commands have corresponding go implementations, the reason is as follows:

    Unsupported Commands

    Redis' zset can operates elements in lexicographic order, which is not commonly used function, so zset does not support commands like ZREMRANGEBYLEX, ZLEXCOUNT and so on.

    | Redis command | |-----------------------| | ZREMRANGEBYLEX | | ZRANGEBYLEX | | ZREVRANGEBYLEX | | ZLEXCOUNT |

    In redis, user accesses zset via a string key. We do not need such string key because we have variable. so the following commands are not implemented:

    | Redis command | |-----------------------| | ZUNIONSTORE | | ZINTERSTORE | | ZDIFFSTORE | | ZRANGESTORE | | ZMSCORE | | ZSCAN |

    Tests

    Unittests

    coverage: 94.0% of statements

    Benchmark

    go test -v -cpuprofile cpu.pprof -cpu=1 -run=NOTEST -bench=. -benchmem -benchtime=100000x -count=10 -timeout=60m > y.txt && benchstat y.txt
    name                                  time/op
    Contains100Hits/sortedset             39.5ns ± 1%
    Contains50Hits/sortedset              43.4ns ± 4%
    ContainsNoHits/sortedset              21.9ns ± 5%
    Add/sortedset                         53.6ns ± 1%
    1Add99Contains/sortedset              38.1ns ±12%
    10Add90Contains/sortedset             47.6ns ± 5%
    50Add50Contains/sortedset             56.9ns ± 3%
    1Add3Incr6Remove90Contains/sortedset  49.3ns ±11%
    
    name                                  alloc/op
    Contains100Hits/sortedset              0.00B
    Contains50Hits/sortedset               0.00B
    ContainsNoHits/sortedset               0.00B
    Add/sortedset                          2.00B ± 0%
    1Add99Contains/sortedset               0.00B
    10Add90Contains/sortedset              0.00B
    50Add50Contains/sortedset              1.00B ± 0%
    1Add3Incr6Remove90Contains/sortedset  0.50B ±100%
    
    name                                  allocs/op
    Contains100Hits/sortedset               0.00
    Contains50Hits/sortedset                0.00
    ContainsNoHits/sortedset                0.00
    Add/sortedset                           0.00
    1Add99Contains/sortedset                0.00
    10Add90Contains/sortedset               0.00
    50Add50Contains/sortedset               0.00
    1Add3Incr6Remove90Contains/sortedset    0.00
    

    TODOs

    • [x] implementation
    • [x] godoc
    • [x] unittest
    • [x] benchmark
    • [x] reamde
  • feat(lang/fastrand): add Read([]byte) function

    feat(lang/fastrand): add Read([]byte) function

    What

    This PR add a new function func Read([]byte) (n int, err error) for fastrand package. It has the same signature with math/rand.Read and math/rand.(*Rand).Read.

    Why

    fastrand is so good that I want to use it everywhere~ And in my benchmark, it's at least 30% faster than math/rand.(*Rand).Read in single goroutine.

    How

    Just like math/rand.Read.

  • fix: logger cannot correctly handle varargs

    fix: logger cannot correctly handle varargs

    the defaultLogger.CtxXXXf would pass varargs by slice as first argument, without keeping its original form.

    Example: logger.CtxInfof(ctx "%v %v", 1, 2) would produce [Info] [1, 2] %!v(MISSING). It prints the whole varargs as one slice, so the second vararg is MISSING.

  • perf(fastrand): optimize Read

    perf(fastrand): optimize Read

    Use wyrand, see https://github.com/wangyi-fudan/wyhash.

    name                                old time/op  new time/op  delta
    SingleCore/math/rand-Uint32()-16    10.8ns ± 0%  10.8ns ± 0%     ~     (p=0.913 n=7+10)
    SingleCore/fast-rand-Uint32()-16    2.26ns ± 0%  2.25ns ± 0%     ~     (p=0.015 n=10+10)
    SingleCore/math/rand-Uint64()-16    11.1ns ± 0%  11.1ns ± 0%     ~     (p=0.014 n=10+8)
    SingleCore/fast-rand-Uint64()-16    5.03ns ± 0%  4.75ns ± 0%   -5.50%  (p=0.000 n=10+9)
    SingleCore/math/rand-Read1000-16     682ns ± 0%   682ns ± 0%     ~     (p=0.927 n=10+10)
    SingleCore/fast-rand-Read1000-16     298ns ± 1%   150ns ± 0%  -49.69%  (p=0.000 n=10+9)
    MultipleCore/math/rand-Uint32()-16   114ns ± 3%   113ns ± 4%     ~     (p=0.306 n=10+10)
    MultipleCore/fast-rand-Uint32()-16  0.18ns ± 1%  0.18ns ± 2%     ~     (p=0.006 n=9+10)
    MultipleCore/math/rand-Uint64()-16   115ns ± 6%   118ns ± 3%     ~     (p=0.018 n=10+9)
    MultipleCore/fast-rand-Uint64()-16  0.39ns ± 1%  0.38ns ± 0%   -1.55%  (p=0.000 n=10+8)
    MultipleCore/math/rand-Read1000-16  1.02µs ± 3%  1.03µs ± 4%     ~     (p=0.644 n=10+10)
    MultipleCore/fast-rand-Read1000-16   112ns ± 0%   102ns ± 1%   -9.38%  (p=0.000 n=10+10)
    
  • possible to show an example of passing value / reference?

    possible to show an example of passing value / reference?

    Question

    possible to show an example of passing value / reference?

    nice work by the way, i've tested against https://github.com/panjf2000/ants

    and found this to be faster but not sure if there are any significant difference between the two?

  • prevent gopool from printing stack infos when panic is handled by panicHandler

    prevent gopool from printing stack infos when panic is handled by panicHandler

    Summary

    Once the go routine from gopool panics, gopool would print stack info, and try to invoke panicHandler.
    Even if we already set a panicHandler , there's no way to prevent the log printing.
    Can we just let the panicHandler handle the log printing if it exists.
    from

    if r := recover(); r != nil {
      msg := fmt.Sprintf("GOPOOL: panic in pool: %s: %v: %s", w.pool.name, r, debug.Stack())
      logger.CtxErrorf(t.ctx, msg)
      if w.pool.panicHandler != nil {
    	  w.pool.panicHandler(t.ctx, r)
      }
    }
    

    to

    if r := recover(); r != nil {
      if w.pool.panicHandler != nil {
    	  w.pool.panicHandler(t.ctx, r)
      } else {
        msg := fmt.Sprintf("GOPOOL: panic in pool: %s: %v: %s", w.pool.name, r, debug.Stack())
        logger.CtxErrorf(t.ctx, msg)
      }
    }
    

    Thanka a lot.

    Motivation

    two many logs in log files and test logs

    Explanation

    No response

  • Fix small typo (string value)

    Fix small typo (string value)

    • passed the tests in cloud/metainfo package.
    • no direct reference of the string (RPC_TRANSIT.../RPC_TRANSIENT...) anywhere in the package. Only the corresponding variable names are referenced, and they remain unchanged (PrefixTransient and PrefixTransientUpstream) .
  • Broken change of golang.org/x/sys causes build failed in go1.16

    Broken change of golang.org/x/sys causes build failed in go1.16

    Question

    Current go.mod use go1.16 with golang.org/x/sys v0.0.0-20221010170243-090e33056c14 package which introduced go1.17's unsafe.Slice in 20220910 (see history https://cs.opensource.google/go/x/sys/+/master:unix/syscall.go;bpv=1). This will cause build failed in go1.16 with following error messges.

    # golang.org/x/sys/unix
    ../compile_path/pkg/mod/golang.org/x/[email protected]/unix/syscall.go:83:16: undefined: unsafe.Slice
    ../compile_path/pkg/mod/golang.org/x/[email protected]/unix/syscall_linux.go:2255:9: undefined: unsafe.Slice
    ../compile_path/pkg/mod/golang.org/x/[email protected]/unix/syscall_unix.go:118:7: undefined: unsafe.Slice
    ../compile_path/pkg/mod/golang.org/x/[email protected]/unix/sysvshm_unix.go:33:7: undefined: unsafe.Slice
    note: module requires Go 1.17
    
  • perf: improve SaveMetaInfoToMap performance

    perf: improve SaveMetaInfoToMap performance

    Benchmark

    Before:

    BenchmarkAllParallel/SaveMetaInfoToMap_10
    BenchmarkAllParallel/SaveMetaInfoToMap_10-12         	  912304	      1723 ns/op	    3720 B/op	      28 allocs/op
    BenchmarkAllParallel/SaveMetaInfoToMap_20
    BenchmarkAllParallel/SaveMetaInfoToMap_20-12         	  530864	      2380 ns/op	    8026 B/op	      50 allocs/op
    BenchmarkAllParallel/SaveMetaInfoToMap_50
    BenchmarkAllParallel/SaveMetaInfoToMap_50-12         	  205908	      5524 ns/op	   18170 B/op	     118 allocs/op
    BenchmarkAllParallel/SaveMetaInfoToMap_100
    BenchmarkAllParallel/SaveMetaInfoToMap_100-12        	   87831	     14197 ns/op	   36655 B/op	     226 allocs/op
    

    After:

    BenchmarkAllParallel/SaveMetaInfoToMap_10
    BenchmarkAllParallel/SaveMetaInfoToMap_10-12         	 1258407	       897.6 ns/op	    2331 B/op	      22 allocs/op
    BenchmarkAllParallel/SaveMetaInfoToMap_20
    BenchmarkAllParallel/SaveMetaInfoToMap_20-12         	  639684	      1971 ns/op	    5404 B/op	      44 allocs/op
    BenchmarkAllParallel/SaveMetaInfoToMap_50
    BenchmarkAllParallel/SaveMetaInfoToMap_50-12         	  273380	      4734 ns/op	   12570 B/op	     109 allocs/op
    BenchmarkAllParallel/SaveMetaInfoToMap_100
    BenchmarkAllParallel/SaveMetaInfoToMap_100-12        	   98899	     11902 ns/op	   25629 B/op	     214 allocs/op
    
  • util/gctuner: test fail randomly

    util/gctuner: test fail randomly

    Operating System

    linux

    Go Version

    go1.18.6 linux/amd64 AND go1.19.1 linux/amd64 AND go latest

    Package Version

    latest

    Affected Packages

    util/gctuner

    Expected Behavior

    ALL TEST PASSED

    Actual Behavior

    
    --- FAIL: TestTuner (0.20s)
        tuner_test.go:37: 
                    Error Trace:    tuner_test.go:37
                    Error:          Not equal: 
                                    expected: 0x1f4
                                    actual  : 0x32
                    Test:           TestTuner
    FAIL
    exit status 1
    FAIL    github.com/bytedance/gopkg/util/gctuner 0.217s
    
    

    Reproduction Steps

    cd util/gctuner && go test -count=1 OR cd util/gctuner && go test -count=10

    Other Information

    @joway PTAL, thanks!

  • test: fix TestPoolPanic in gopool

    test: fix TestPoolPanic in gopool

    This PR tries to fix the TestPoolPanic function in gopool. Currently it only contains the following test logic

    p := NewPool("test", 100, NewConfig())
    p.Go(testPanicFunc)
    

    Maybe the original idea is to see if any panic occurs. But actually, the p.Go goroutine is not guaranteed to be executed before the main goroutine exits. Chances are testPanicFunc never get called. So this test case is actually useless.

    I add sync.WaitGroup to wait for sub goroutines to finish. Also we need to check if SetPanicHandler is working correctly, so I compared the actual execute times with expected times to see if provided panic handler does get called every time when panic occurs.

  • gopool的RegisterPool使用不了

    gopool的RegisterPool使用不了

    Question

    大佬您好,我使用了一下gopool.RegisterPool 好像仅仅只是将 这个 Pool 存入了一个sync.Map中,在后续的 gopool.Go 中并没有使用到自己 RegisterPool。是否考虑给 gopool.go文件中init 函数加一个check,如果调用者没有 RegisterPool,在使用 defaultPool。还有就是整个capacity容量是固定的,是否可以加一些动态机制,在大流量的时候taskList中的足够多了,可以进行capacity扩容,当task少的时候capacity可以进行缩容。还望大佬指正观点

Related tags
A better Generic Pool (sync.Pool)
A better Generic Pool (sync.Pool)

This package is a thin wrapper over the Pool provided by the sync package. The Pool is an essential package to obtain maximum performance by reducing the number of memory allocations.

Dec 1, 2022
redis-util business-friendly encapsulation of redis operations, such as the common cache set get operation

redis-util 方便业务使用的redis操作封装,比如常见的缓存set get操作, 一行代码搞定,不像开源库需要写好多行 使用方法

Oct 22, 2021
Goterators - A util library that Supports aggregate & transforms functions Go. Such as filter, map, reduce, find, exist
Goterators - A util library that Supports aggregate & transforms functions Go. Such as filter, map, reduce, find, exist

Goterators Goterators is util library that Supports aggregate & transforms functions Go, including: for-each find exist reduce filter map API and func

Dec 19, 2022
A collection of small Go utilities to make life easier.

The simplego package provides a collection of Go utilities for common tasks.

Jan 4, 2023
Collection of unusual generics usecases in Go

Unusual Generics Type parameters or Generics in Go designed to reduce boilerplate for container data types like lists, graphs, etc. and functions like

Dec 14, 2022
A collection of functional operators for golang with generics

fn fn is a collection of go functional operators with generics Getting Started P

Jul 8, 2022
Go-Utils is a library containing a collection of Golang utilities

Go-Utils is a library containing a collection of Golang utilities

Jun 2, 2022
Code generation tools for Go.

interfaces Code generation tools for Go's interfaces. Tools available in this repository: cmd/interfacer cmd/structer cmd/interfacer Generates an inte

Dec 23, 2022
A directory of hardware related libs, tools, and tutorials for Go

Go + hardware This repo is a directory of tools, packages and tutorials to let you introduce Go in your hardware projects. Why Go? Go can target platf

Dec 30, 2022
gqlanalysis makes easy to develop static analysis tools for GraphQL in Go.
gqlanalysis makes easy to develop static analysis tools for GraphQL in Go.

gqlanalysis gqlanalysis defines the interface between a modular static analysis for GraphQL in Go. gqlanalysis is inspired by go/analysis. gqlanalysis

Dec 14, 2022
Little Bug Bounty & Hacking Tools⚔️

Little Bug Bounty & Hacking Tools ⚔️

Jan 7, 2023
common tools for golang

utils common tools for golang package main

Dec 27, 2021
Source code of Liteloader Tools

LiteLoader Tools This repository store the source code of some LiteLoader Tools Prebuilt Binary see /bin folder Image2Binary [Golang] convert Image(jp

Aug 30, 2022
Go tools sourcecode read and customize

Go Tools This subrepository holds the source for various packages and tools that support the Go programming language. Some of the tools, godoc and vet

Oct 24, 2021
A fully Go userland with Linux bootloaders! u-root can create a one-binary root file system (initramfs) containing a busybox-like set of tools written in Go.

u-root Description u-root embodies four different projects. Go versions of many standard Linux tools, such as ls, cp, or shutdown. See cmds/core for m

Dec 29, 2022
Mackerel SLI/SLO tools

shimesaba For SRE to operate and monitor services using Mackerel. Description shimesaba is a tool for tracking SLO/ErrorBudget using Mackerel as an SL

Nov 21, 2022
Like tools/cmd/stringer with bitmask features

Bitmasker Bitmasker is a tool used to automate the creation of helper methods when dealing with bitmask-type constant flags. Given the name of an unsi

Nov 25, 2021
Functional tools in Go 1.18 using newly introduced generics

functools functools is a simple Go library that brings you your favourite functi

Dec 5, 2022
A super simple Lodash like utility library with essential functions that empowers the development in Go
A super simple Lodash like utility library with essential functions that empowers the development in Go

A simple Utility library for Go Go does not provide many essential built in functions when it comes to the data structure such as slice and map. This

Jan 4, 2023