a generic object pool for golang

jolestar

Last update: Jan 5, 2023

Comments: 13

Go Commons Pool

The Go Commons Pool is a generic object pool for Golang, direct rewrite from Apache Commons Pool.

Features

Support custom PooledObjectFactory.
Rich pool configuration option, can precise control pooled object lifecycle. See ObjectPoolConfig.
- Pool LIFO (last in, first out) or FIFO (first in, first out)
- Pool cap config
- Pool object validate config
- Pool object borrow block and max waiting time config
- Pool object eviction config
- Pool object abandon config

Pool Configuration Option

Configuration option table, more detail description see ObjectPoolConfig

Option	Default	Description
LIFO	true	If pool is LIFO (last in, first out)
MaxTotal	8	The cap of pool
MaxIdle	8	Max "idle" instances in the pool
MinIdle	0	Min "idle" instances in the pool
TestOnCreate	false	Validate when object is created
TestOnBorrow	false	Validate when object is borrowed
TestOnReturn	false	Validate when object is returned
TestWhileIdle	false	Validate when object is idle, see TimeBetweenEvictionRuns
BlockWhenExhausted	true	Whether to block when the pool is exhausted
MinEvictableIdleTime	30m	Eviction configuration,see DefaultEvictionPolicy
SoftMinEvictableIdleTime	math.MaxInt64	Eviction configuration,see DefaultEvictionPolicy
NumTestsPerEvictionRun	3	The maximum number of objects to examine during each run evictor goroutine
TimeBetweenEvictionRuns	0	The number of milliseconds to sleep between runs of the evictor goroutine, less than 1 mean not run

Usage

Use Simple Factory

import (
	"context"
	"fmt"
	"strconv"
	"sync/atomic"

	"github.com/jolestar/go-commons-pool/v2"
)

func Example_simple() {
	type myPoolObject struct {
		s string
	}

	v := uint64(0)
	factory := pool.NewPooledObjectFactorySimple(
		func(context.Context) (interface{}, error) {
			return &myPoolObject{
					s: strconv.FormatUint(atomic.AddUint64(&v, 1), 10),
				},
				nil
		})

	ctx := context.Background()
	p := pool.NewObjectPoolWithDefaultConfig(ctx, factory)

	obj, err := p.BorrowObject(ctx)
	if err != nil {
		panic(err)
	}

	o := obj.(*myPoolObject)
	fmt.Println(o.s)

	err = p.ReturnObject(ctx, obj)
	if err != nil {
		panic(err)
	}

	// Output: 1
}

Use Custom Factory

import (
	"context"
	"fmt"
	"strconv"
	"sync/atomic"

	"github.com/jolestar/go-commons-pool/v2"
)

type MyPoolObject struct {
	s string
}

type MyCustomFactory struct {
	v uint64
}

func (f *MyCustomFactory) MakeObject(ctx context.Context) (*pool.PooledObject, error) {
	return pool.NewPooledObject(
			&MyPoolObject{
				s: strconv.FormatUint(atomic.AddUint64(&f.v, 1), 10),
			}),
		nil
}

func (f *MyCustomFactory) DestroyObject(ctx context.Context, object *pool.PooledObject) error {
	// do destroy
	return nil
}

func (f *MyCustomFactory) ValidateObject(ctx context.Context, object *pool.PooledObject) bool {
	// do validate
	return true
}

func (f *MyCustomFactory) ActivateObject(ctx context.Context, object *pool.PooledObject) error {
	// do activate
	return nil
}

func (f *MyCustomFactory) PassivateObject(ctx context.Context, object *pool.PooledObject) error {
	// do passivate
	return nil
}

func Example_customFactory() {
	ctx := context.Background()
	p := pool.NewObjectPoolWithDefaultConfig(ctx, &MyCustomFactory{})
	p.Config.MaxTotal = 100
    
	obj1, err := p.BorrowObject(ctx)
	if err != nil {
		panic(err)
	}

	o := obj1.(*MyPoolObject)
	fmt.Println(o.s)

	err = p.ReturnObject(ctx, obj1)
	if err != nil {
		panic(err)
	}

	// Output: 1
}

For more examples please see pool_test.go and example_simple_test.go, example_customFactory_test.go.

Note

PooledObjectFactory.MakeObject must return a pointer, not value. The following code will complain error.

p := pool.NewObjectPoolWithDefaultConfig(ctx, pool.NewPooledObjectFactorySimple(
    func(context.Context) (interface{}, error) {
        return "hello", nil
    }))
obj, _ := p.BorrowObject()
p.ReturnObject(obj)

The right way is:

p := pool.NewObjectPoolWithDefaultConfig(ctx, pool.NewPooledObjectFactorySimple(
    func(context.Context) (interface{}, error) {
        s := "hello"
        return &s, nil
    }))

For more examples please see example_simple_test.go.

Dependency

testify for test

PerformanceTest

The results of running the pool_perf_test is almost equal to the java version PerformanceTest

go test --perf=true

For Apache commons pool user

Direct use pool.Config.xxx to change pool config
Default config value is same as java version
If TimeBetweenEvictionRuns changed after ObjectPool created, should call ObjectPool.StartEvictor to take effect. Java version do this on set method.
No KeyedObjectPool (TODO)
No ProxiedObjectPool
No pool stats (TODO)

FAQ

How to contribute

Choose one open issue you want to solve, if not create one and describe what you want to change.
Fork the repository on GitHub.
Write code to solve the issue.
Create PR and link to the issue.
Make sure test and coverage pass.
Wait maintainers to merge.

中文文档

License

Go Commons Pool is available under the Apache License, Version 2.0.

Owner

jolestar

https://github.com/jolestar/go-commons-pool

Comments

Cannot update to v2.1 when using gomod
Using go get -u github.com/jolestar/[email protected] to attempt to update to 2.1 results in this error:

go: finding github.com/jolestar/go-commons-pool v2.1.0 go: finding github.com/jolestar/go-commons-pool v2.1.0 go get github.com/jolestar/[email protected]: github.com/jolestar/[email protected]: invalid version: module contains a go.mod file, so major version must be compatible: should be v0 or v1, not v2

To resolve this, the module needs to refer to itself as github.com/jolestar/go-commons-pool/v2, as per the documentation, and likely the commit should be tagged as a new release (v2.1.1, at least).

dead lock when goroutines exceeds MaxTotal

Seeing

dead lock when goroutines exceeds MaxTotal:

Environment

go 1.11 go-commons-pool newest ubuntu 18.04

Code

package main

import (
	"bytes"
	"context"
	"crypto/md5"
	"encoding/binary"
	"fmt"
	"github.com/gomodule/redigo/redis"
	"github.com/jolestar/go-commons-pool"
	"log"
	"math/rand"
	"net/http"
	_ "net/http/pprof"
	"strconv"
	"sync"
	"time"
)

func main() {
	// dump stack info by http://localhost:6060/debug/pprof/goroutine?debug=2
	go func() {
		log.Println(http.ListenAndServe("localhost:6060", nil))
	}()
	// create commons pool
	factory := pool.NewPooledObjectFactorySimple(
		func(context.Context) (interface{}, error) {
			return redis.Dial("tcp", ":6400")
		})
	ctx := context.TODO()
	p := pool.NewObjectPoolWithDefaultConfig(ctx, factory)
	p.Config.MaxTotal = 100
	p.Config.MaxIdle = 100
	defer p.Close(ctx)
	// write to redis parallel
	total := 100000
	idCount := 10
	// when <= 100 (MaxTotal) is OK, or else dead lock
	parallel := 200
	today := time.Now().Format("20060102")
	rand.Seed(time.Now().Unix())
	log.Println("begin write to redis")
	for i := 0; i < total; {
		batchWg := &sync.WaitGroup{}
		for j := 0; j < parallel/idCount && i < total; j++ {
			uuid := createUuid()
			for k := 0; k < idCount; k++ {
				idInt := rand.Intn(1000000)
				id := strconv.Itoa(idInt)
				field := ":" + today + ":" + id
				batchWg.Add(1)
				go incrCounter(batchWg, p, uuid+field)
			}
			i++
		}
		batchWg.Wait()
	}
	log.Println("end write to redis")
}

func incrCounter(wg *sync.WaitGroup, pool *pool.ObjectPool, key string) {
	defer wg.Done()
	ctx := context.TODO()
	conn, _ := pool.BorrowObject(ctx)
	if conn != nil {
		defer pool.ReturnObject(ctx, conn)
	} else {
		return
	}
	redisConn := conn.(redis.Conn)
	_, err := redisConn.Do("incr", key)
	if err != nil {
		log.Printf("incr error %v", err)
	}
}

func createUuid() string {
	randInt := rand.Int63()
	intBytes := make([]byte, 8)
	binary.BigEndian.PutUint64(intBytes, uint64(randInt))
	md5Bytes := md5.Sum(intBytes)
	buf := new(bytes.Buffer)
	fmt.Fprintf(buf, "%x", md5Bytes)
	return buf.String()
}

Stack Trace

goroutine 228 [running]:
runtime/pprof.writeGoroutineStacks(0x857620, 0xc00048e000, 0x40d62f, 0xc000392360)
	/home/zhyhang/workdoc/dev/go/src/runtime/pprof/pprof.go:678 +0xa7
runtime/pprof.writeGoroutine(0x857620, 0xc00048e000, 0x2, 0xae41e0, 0x0)
	/home/zhyhang/workdoc/dev/go/src/runtime/pprof/pprof.go:667 +0x44
runtime/pprof.(*Profile).WriteTo(0xad5d60, 0x857620, 0xc00048e000, 0x2, 0xc00048e000, 0x7f5c13)
	/home/zhyhang/workdoc/dev/go/src/runtime/pprof/pprof.go:328 +0x3e4
net/http/pprof.handler.ServeHTTP(0xc00037cb51, 0x9, 0x85a5c0, 0xc00048e000, 0xc0002ce000)
	/home/zhyhang/workdoc/dev/go/src/net/http/pprof/pprof.go:245 +0x210
net/http/pprof.Index(0x85a5c0, 0xc00048e000, 0xc0002ce000)
	/home/zhyhang/workdoc/dev/go/src/net/http/pprof/pprof.go:268 +0x723
net/http.HandlerFunc.ServeHTTP(0x80d5b0, 0x85a5c0, 0xc00048e000, 0xc0002ce000)
	/home/zhyhang/workdoc/dev/go/src/net/http/server.go:1964 +0x44
net/http.(*ServeMux).ServeHTTP(0xae3400, 0x85a5c0, 0xc00048e000, 0xc0002ce000)
	/home/zhyhang/workdoc/dev/go/src/net/http/server.go:2361 +0x127
net/http.serverHandler.ServeHTTP(0xc0002b60d0, 0x85a5c0, 0xc00048e000, 0xc0002ce000)
	/home/zhyhang/workdoc/dev/go/src/net/http/server.go:2741 +0xab
net/http.(*conn).serve(0xc0002ae8c0, 0x85a940, 0xc000371a80)
	/home/zhyhang/workdoc/dev/go/src/net/http/server.go:1847 +0x646
created by net/http.(*Server).Serve
	/home/zhyhang/workdoc/dev/go/src/net/http/server.go:2851 +0x2f5

goroutine 1 [semacquire]:
sync.runtime_Semacquire(0xc00043c558)
	/home/zhyhang/workdoc/dev/go/src/runtime/sema.go:56 +0x39
sync.(*WaitGroup).Wait(0xc00043c550)
	/home/zhyhang/workdoc/dev/go/src/sync/waitgroup.go:130 +0x64
main.main()
	/home/zhyhang/code-go/src/github.com/zhyhang/gofirst/tools/deadpool.go:56 +0x36c

goroutine 20 [IO wait]:
internal/poll.runtime_pollWait(0x7faf71b262b0, 0x72, 0x0)
	/home/zhyhang/workdoc/dev/go/src/runtime/netpoll.go:173 +0x66
internal/poll.(*pollDesc).wait(0xc0001d1718, 0x72, 0xc0001aa100, 0x0, 0x0)
	/home/zhyhang/workdoc/dev/go/src/internal/poll/fd_poll_runtime.go:85 +0x9a
internal/poll.(*pollDesc).waitRead(0xc0001d1718, 0xffffffffffffff00, 0x0, 0x0)
	/home/zhyhang/workdoc/dev/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(*FD).Accept(0xc0001d1700, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/home/zhyhang/workdoc/dev/go/src/internal/poll/fd_unix.go:384 +0x1a0
net.(*netFD).accept(0xc0001d1700, 0x40d62f, 0xc0002ae960, 0xa0)
	/home/zhyhang/workdoc/dev/go/src/net/fd_unix.go:238 +0x42
net.(*TCPListener).accept(0xc0002a4078, 0xc000042df0, 0x8dcf19d1, 0x6b34adfcd281e637)
	/home/zhyhang/workdoc/dev/go/src/net/tcpsock_posix.go:139 +0x2e
net.(*TCPListener).AcceptTCP(0xc0002a4078, 0xc000042e18, 0x491246, 0x5bbc77ec)
	/home/zhyhang/workdoc/dev/go/src/net/tcpsock.go:247 +0x47
net/http.tcpKeepAliveListener.Accept(0xc0002a4078, 0xc000042e68, 0x18, 0xc000062900, 0x6a6825)
	/home/zhyhang/workdoc/dev/go/src/net/http/server.go:3232 +0x2f
net/http.(*Server).Serve(0xc0002b60d0, 0x85a7c0, 0xc0002a4078, 0x0, 0x0)
	/home/zhyhang/workdoc/dev/go/src/net/http/server.go:2826 +0x22f
net/http.(*Server).ListenAndServe(0xc0002b60d0, 0xc0002b60d0, 0x0)
	/home/zhyhang/workdoc/dev/go/src/net/http/server.go:2764 +0xb6
net/http.ListenAndServe(0x7f66e8, 0xe, 0x0, 0x0, 0x0, 0x0)
	/home/zhyhang/workdoc/dev/go/src/net/http/server.go:3004 +0x74
main.main.func1()
	/home/zhyhang/code-go/src/github.com/zhyhang/gofirst/tools/deadpool.go:23 +0x3e
created by main.main
	/home/zhyhang/code-go/src/github.com/zhyhang/gofirst/tools/deadpool.go:22 +0x47

goroutine 229 [IO wait]:
internal/poll.runtime_pollWait(0x7faf71aea958, 0x72, 0xc000167858)
	/home/zhyhang/workdoc/dev/go/src/runtime/netpoll.go:173 +0x66
internal/poll.(*pollDesc).wait(0xc0001d1f18, 0x72, 0xffffffffffffff00, 0x8581c0, 0xaa6618)
	/home/zhyhang/workdoc/dev/go/src/internal/poll/fd_poll_runtime.go:85 +0x9a
internal/poll.(*pollDesc).waitRead(0xc0001d1f18, 0xc000406000, 0x1000, 0x1000)
	/home/zhyhang/workdoc/dev/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(*FD).Read(0xc0001d1f00, 0xc000406000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/home/zhyhang/workdoc/dev/go/src/internal/poll/fd_unix.go:169 +0x179
net.(*netFD).Read(0xc0001d1f00, 0xc000406000, 0x1000, 0x1000, 0xc0001aba00, 0xc000167968, 0x69cff0)
	/home/zhyhang/workdoc/dev/go/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc0002a4098, 0xc000406000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/home/zhyhang/workdoc/dev/go/src/net/net.go:177 +0x68
net/http.(*connReader).Read(0xc0001cd080, 0xc000406000, 0x1000, 0x1000, 0x435c7c, 0xc0001b9680, 0x298)
	/home/zhyhang/workdoc/dev/go/src/net/http/server.go:786 +0xfc
bufio.(*Reader).fill(0xc0001c39e0)
	/home/zhyhang/workdoc/dev/go/src/bufio/bufio.go:100 +0x106
bufio.(*Reader).ReadSlice(0xc0001c39e0, 0xc0002ca00a, 0xc000167ac8, 0x40d1a3, 0x7faf71b6db30, 0x0, 0x203000)
	/home/zhyhang/workdoc/dev/go/src/bufio/bufio.go:341 +0x36
bufio.(*Reader).ReadLine(0xc0001c39e0, 0xc0002ca000, 0x100, 0xf8, 0x7e4760, 0x1, 0x1000000ae41e0)
	/home/zhyhang/workdoc/dev/go/src/bufio/bufio.go:370 +0x34
net/textproto.(*Reader).readLineSlice(0xc0001cd0b0, 0xc000404701, 0xc000167b98, 0x40deb8, 0x100, 0x7e4760)
	/home/zhyhang/workdoc/dev/go/src/net/textproto/reader.go:55 +0x6f
net/textproto.(*Reader).ReadLine(0xc0001cd0b0, 0xc0002ca000, 0x0, 0x0, 0x0)
	/home/zhyhang/workdoc/dev/go/src/net/textproto/reader.go:36 +0x2b
net/http.readRequest(0xc0001c39e0, 0x0, 0xc0002ca000, 0x0, 0x0)
	/home/zhyhang/workdoc/dev/go/src/net/http/request.go:958 +0x8c
net/http.(*conn).readRequest(0xc0002ae960, 0x85a940, 0xc0001ab980, 0x0, 0x0, 0x0)
	/home/zhyhang/workdoc/dev/go/src/net/http/server.go:966 +0x162
net/http.(*conn).serve(0xc0002ae960, 0x85a940, 0xc0001ab980)
	/home/zhyhang/workdoc/dev/go/src/net/http/server.go:1788 +0x49e
created by net/http.(*Server).Serve
	/home/zhyhang/workdoc/dev/go/src/net/http/server.go:2851 +0x2f5

goroutine 2642 [runnable]:
syscall.Syscall(0x0, 0x69, 0xc000209241, 0x1, 0xffffffffffffffff, 0x0, 0xb)
	/home/zhyhang/workdoc/dev/go/src/syscall/asm_linux_amd64.s:18 +0x5
syscall.read(0x69, 0xc000209241, 0x1, 0x1, 0xc00005a000, 0x0, 0x0)
	/home/zhyhang/workdoc/dev/go/src/syscall/zsyscall_linux_amd64.go:732 +0x5a
syscall.Read(0x69, 0xc000209241, 0x1, 0x1, 0x0, 0xc000060000, 0xc00044ef00)
	/home/zhyhang/workdoc/dev/go/src/syscall/syscall_unix.go:172 +0x49
internal/poll.(*FD).Read(0xc0001d1e80, 0xc000209241, 0x1, 0x1, 0x0, 0x0, 0x0)
	/home/zhyhang/workdoc/dev/go/src/internal/poll/fd_unix.go:165 +0x119
net.(*netFD).Read(0xc0001d1e80, 0xc000209241, 0x1, 0x1, 0x1, 0xc00039adc0, 0xc00001e7b8)
	/home/zhyhang/workdoc/dev/go/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc0002a4090, 0xc000209241, 0x1, 0x1, 0x0, 0x0, 0x0)
	/home/zhyhang/workdoc/dev/go/src/net/net.go:177 +0x68
net/http.(*connReader).backgroundRead(0xc000209230)
	/home/zhyhang/workdoc/dev/go/src/net/http/server.go:676 +0x5a
created by net/http.(*connReader).startBackgroundRead
	/home/zhyhang/workdoc/dev/go/src/net/http/server.go:672 +0xd2

goroutine 2624 [select]:
github.com/jolestar/go-commons-pool/concurrent.(*TimeoutCond).Wait(0xc00008a420, 0x85a980, 0xc000094018, 0x0)
	/home/zhyhang/code-go/src/github.com/jolestar/go-commons-pool/concurrent/cond.go:55 +0x14f
github.com/jolestar/go-commons-pool/collections.(*LinkedBlockingDeque).PollFirstWithContext(0xc00009c340, 0x85a980, 0xc000094018, 0x0, 0x0, 0x0, 0x0)
	/home/zhyhang/code-go/src/github.com/jolestar/go-commons-pool/collections/queue.go:289 +0xba
github.com/jolestar/go-commons-pool.(*ObjectPool).borrowObject(0xc000122000, 0x85a980, 0xc000094018, 0x8, 0xc0001aab40, 0x1, 0xc00019a718)
	/home/zhyhang/code-go/src/github.com/jolestar/go-commons-pool/pool.go:276 +0x45d
github.com/jolestar/go-commons-pool.(*ObjectPool).BorrowObject(0xc000122000, 0x85a980, 0xc000094018, 0xc0004558c0, 0x1, 0x1, 0xc000218320)
	/home/zhyhang/code-go/src/github.com/jolestar/go-commons-pool/pool.go:141 +0x3f
main.incrCounter(0xc00043c550, 0xc000122000, 0xc000442f00, 0x30)
	/home/zhyhang/code-go/src/github.com/zhyhang/gofirst/tools/deadpool.go:64 +0x8a
created by main.main
	/home/zhyhang/code-go/src/github.com/zhyhang/gofirst/tools/deadpool.go:52 +0x2f1

unit test failed on respberry, 32bit ARM, linux system

--- FAIL: TestBaseInvalidateObject (0.00s) panic: runtime error: invalid memory address or nil pointer dereference [recovered] panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x12208]

goroutine 27 [running]: testing.tRunner.func1(0x24cce60) /usr/local/go/src/testing/testing.go:792 +0x30c panic(0x2f5e68, 0x59bfb0) /usr/local/go/src/runtime/panic.go:513 +0x194 runtime/internal/atomic.goLoad64(0x250b7ac, 0x243ed44, 0x243ed4c) /usr/local/go/src/runtime/internal/atomic/atomic_arm.go:124 +0x1c github.com/jolestar/go-commons-pool/concurrent.(*TimeoutCond).HasWaiters(0x250b7a0, 0x344f10) /home/pi/go/src/github.com/jolestar/go-commons-pool/concurrent/cond.go:42 +0x24 github.com/jolestar/go-commons-pool/collections.(*LinkedBlockingDeque).HasTakeWaiters(0x250b7e0, 0x2479100) /home/pi/go/src/github.com/jolestar/go-commons-pool/collections/queue.go:441 +0x60 github.com/jolestar/go-commons-pool.(*ObjectPool).ensureIdle(0x249c720, 0x38dea0, 0x24160a8, 0x1, 0x2479000) /home/pi/go/src/github.com/jolestar/go-commons-pool/pool.go:339 +0x124 github.com/jolestar/go-commons-pool.(*ObjectPool).InvalidateObject(0x249c720, 0x38dea0, 0x24160a8, 0x2ceac0, 0x251c8c8, 0x2ceac0, 0x251c8cc) /home/pi/go/src/github.com/jolestar/go-commons-pool/pool.go:477 +0x118 github.com/jolestar/go-commons-pool.(*PoolTestSuite).TestBaseInvalidateObject(0x24ea040) /home/pi/go/src/github.com/jolestar/go-commons-pool/pool_test.go:426 +0x234 reflect.Value.call(0x2510540, 0x24ee450, 0x13, 0x32da11, 0x4, 0x24fafac, 0x1, 0x1, 0x24faf78, 0xc, ...) /usr/local/go/src/reflect/value.go:447 +0x368 reflect.Value.Call(0x2510540, 0x24ee450, 0x13, 0x24fafac, 0x1, 0x1, 0x2d1b6e3, 0x0, 0xb9930664) /usr/local/go/src/reflect/value.go:308 +0x74 github.com/jolestar/go-commons-pool/vendor/github.com/stretchr/testify/suite.Run.func2(0x24cce60) /home/pi/go/src/github.com/jolestar/go-commons-pool/vendor/github.com/stretchr/testify/suite/suite.go:95 +0x190 testing.tRunner(0x24cce60, 0x2511e80) /usr/local/go/src/testing/testing.go:827 +0xac created by testing.(*T).Run /usr/local/go/src/testing/testing.go:878 +0x2b0 exit status 2 FAIL github.com/jolestar/go-commons-pool 0.073s

go test github.com/jolestar/go-commons-pool/concurrent panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x120a8]

goroutine 23 [running]: runtime/internal/atomic.goXadd64(0x145310c, 0x1, 0x0, 0x0, 0x0) /usr/local/go/src/runtime/internal/atomic/atomic_arm.go:96 +0x1c github.com/jolestar/go-commons-pool/concurrent.(*TimeoutCond).addWaiter(0x1453100) /home/pi/go/src/github.com/jolestar/go-commons-pool/concurrent/cond.go:25 +0x34 github.com/jolestar/go-commons-pool/concurrent.(*TimeoutCond).Wait(0x1453100, 0x2d2690, 0x1454b00, 0x71a00) /home/pi/go/src/github.com/jolestar/go-commons-pool/concurrent/cond.go:47 +0x24 github.com/jolestar/go-commons-pool/concurrent.(*LockTestObject).lockAndWaitWithTimeout(0x140c240, 0x498d0000, 0x71afd, 0x0) /home/pi/go/src/github.com/jolestar/go-commons-pool/concurrent/cond_test.go:31 +0xc4 github.com/jolestar/go-commons-pool/concurrent.TestTimeoutCondWaitTimeoutNotify.func1(0x140c240, 0x77359400, 0x0, 0x1418e40, 0x14166f0) /home/pi/go/src/github.com/jolestar/go-commons-pool/concurrent/cond_test.go:96 +0x5c created by github.com/jolestar/go-commons-pool/concurrent.TestTimeoutCondWaitTimeoutNotify /home/pi/go/src/github.com/jolestar/go-commons-pool/concurrent/cond_test.go:94 +0x1a0 FAIL github.com/jolestar/go-commons-pool/concurrent 0.034s
Added support for contexts (including for canceling and timing out)

In order to support more modern ways of timing out or otherwise canceling calls to borrow an object from the pool, this PR adds support for passing a context to the new method, BorrowObjectWithContext.

deadlock in BorrowObject

Problem

When creating object fails and using n goroutines (n > MaxTotal) to call BorrowObject, it will deadlock. But when I lock BorrowObject, the deadlock will not happen. Is this a bug?

Env

go version 1.13 windows10 go-commons-pool version v2.11

Code

package pool

import (
	"context"
	"errors"
	"fmt"
	"github.com/jolestar/go-commons-pool/v2"
	"sync"
	"testing"
	"time"
)

func TestPool(t *testing.T) {
	factory := pool.NewPooledObjectFactorySimple(
		func(context.Context) (interface{}, error) {
			time.Sleep(time.Second)
			return nil, errors.New("create object fail") // simulate the failure of creating objects
		})

	ctx := context.Background()
	conf := pool.NewDefaultPoolConfig()
	conf.MaxTotal = 1
	p := pool.NewObjectPool(ctx, factory, conf)

	n := 2
	wg := sync.WaitGroup{}
	wg.Add(n)
	//lock := sync.Mutex{}
	for i := 0; i < n; i++ {
		id := i
		go func() { // when using goroutines without locking BorrowObject, it will deadlock
			defer wg.Done()
			fmt.Println(id, "before borrow")
			//lock.Lock()
			obj, err := p.BorrowObject(ctx)
			//lock.Unlock()
			fmt.Println(id, "after borrow")
			if err != nil {
				fmt.Println(id, "borrow err:", err)
			}

			err = p.ReturnObject(ctx, obj)
			if err != nil {
				fmt.Println(id, "return err:", err)
			}
		}()
	}
	wg.Wait()
}

Context Support

It would be nice if context.Context could be passed to methods on the pool, and also if the context was passed to the factory methods.

I already have a PR that adds a BorrowWithContext method that will cancel based on the supplied context (#26), but this is only a first step.

If the author is willing to break backwards compatibility (which means releasing this as v2.0.0), I can change the package to pass a context down all the way to the factory methods.
Added support for contexts (including for canceling and timing out)

In order to support more modern ways of timing out or otherwise canceling function calls, this PR adds support for passing a context to the new method, BorrowObjectWithContext.
Bump github.com/stretchr/testify from 1.7.5 to 1.8.0
Bumps github.com/stretchr/testify from 1.7.5 to 1.8.0.

Commits

181cea6 impr: CallerInfo should print full paths to the terminal (#1201)

cf1284f Allow mock expectations to be ordered (#1106)

66eef0e fix: assert.MapSubset (or just support maps in assert.Subset) (#1178)

2fab6df Add WithinTimeRange method (#1188)

See full diff in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

@dependabot rebase will rebase this PR

@dependabot recreate will recreate this PR, overwriting any edits that have been made to it

@dependabot merge will merge this PR after your CI passes on it

@dependabot squash and merge will squash and merge this PR after your CI passes on it

@dependabot cancel merge will cancel a previously requested merge and block automerging

@dependabot reopen will reopen this PR if it is closed

@dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually

@dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
Bump github.com/stretchr/testify from 1.7.4 to 1.7.5
Bumps github.com/stretchr/testify from 1.7.4 to 1.7.5.

Commits

b5ce165 fixing panic in calls to assertion with nil m.mutex (#1212)

c206b2e Mock can be deadlocked by a panic (#1157)

1b73601 suite: correctly set stats on test panic (#1195)

ba1076d Add .Unset method to mock (#982)

c31ea03 Support comparing byte slice (#1202)

See full diff in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

@dependabot rebase will rebase this PR

@dependabot recreate will recreate this PR, overwriting any edits that have been made to it

@dependabot merge will merge this PR after your CI passes on it

@dependabot squash and merge will squash and merge this PR after your CI passes on it

@dependabot cancel merge will cancel a previously requested merge and block automerging

@dependabot reopen will reopen this PR if it is closed

@dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually

@dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
Bump github.com/stretchr/testify from 1.7.2 to 1.7.4
Bumps github.com/stretchr/testify from 1.7.2 to 1.7.4.

Commits

48391ba Fix panic in AssertExpectations for mocks without expectations (#1207)

840cb80 arrays value types in a zero-initialized state are considered empty (#1126)

07dc7ee Bump actions/setup-go from 3.1.0 to 3.2.0 (#1191)

c33fc8d Bump actions/checkout from 2 to 3 (#1163)

3c33e07 Added Go 1.18.1 as a build/supported version (#1182)

e2b56b3 Bump github.com/stretchr/objx from 0.1.0 to 0.4.0

See full diff in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

@dependabot rebase will rebase this PR

@dependabot recreate will recreate this PR, overwriting any edits that have been made to it

@dependabot merge will merge this PR after your CI passes on it

@dependabot squash and merge will squash and merge this PR after your CI passes on it

@dependabot cancel merge will cancel a previously requested merge and block automerging

@dependabot reopen will reopen this PR if it is closed

@dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually

@dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
Bump github.com/stretchr/testify from 1.7.1 to 1.7.2
Bumps github.com/stretchr/testify from 1.7.1 to 1.7.2.

Commits

41453c0 Update gopkg.in/yaml.v3

285adcc Update go versions in build matrix

6e7fab4 Bump actions/setup-go from 2 to 3.1.0

106ec21 use RWMutex

a409ccf fix data race in the suit

3586478 assert: fix typo

7797738 Update versions supported to include go 1.16

See full diff in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

@dependabot rebase will rebase this PR

@dependabot recreate will recreate this PR, overwriting any edits that have been made to it

@dependabot merge will merge this PR after your CI passes on it

@dependabot squash and merge will squash and merge this PR after your CI passes on it

@dependabot cancel merge will cancel a previously requested merge and block automerging

@dependabot reopen will reopen this PR if it is closed

@dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually

@dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
Bump github.com/stretchr/testify from 1.8.0 to 1.8.1
Bumps github.com/stretchr/testify from 1.8.0 to 1.8.1.

Commits

b747d7c Bump github.com/stretchr/objx from 0.4.0 to 0.5.0 (#1283)

See full diff in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

@dependabot rebase will rebase this PR

@dependabot recreate will recreate this PR, overwriting any edits that have been made to it

@dependabot merge will merge this PR after your CI passes on it

@dependabot squash and merge will squash and merge this PR after your CI passes on it

@dependabot cancel merge will cancel a previously requested merge and block automerging

@dependabot reopen will reopen this PR if it is closed

@dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually

@dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
Optimization: prepare config in init-time rather than in run-time
Introduction: There are some instructions in the code, that can be executed once in init-time rather than every time in running:

evictionConfig is been recreated every time in eviction-phase.

EvictionPolicy is being looked in map every time in eviction-phase. Map operations aren't so fast as we might think: they calculate hash, look object in buckets by the hash.

Something else?

Problem: Probably, it's not reasonable to do these operations every time, because the config is been changed hardly ever after pool is created.

Possible solution:

Create such objects once in startEvictor(). So, when you change config - just restart evictor;

Make a new method to apply config changes. So, when you change config - just call the method;

Another solution?

a generic object pool for golang

Go Commons Pool

Features

Pool Configuration Option

Usage

Use Simple Factory

Use Custom Factory

Note

Dependency

PerformanceTest

For Apache commons pool user

FAQ

How to contribute

中文文档

License

Owner

jolestar

Comments

Cannot update to v2.1 when using gomod

dead lock when goroutines exceeds MaxTotal

Seeing

Environment

Code

Stack Trace

unit test failed on respberry, 32bit ARM, linux system

Added support for contexts (including for canceling and timing out)

deadlock in BorrowObject

Problem

Env

Code

Context Support

Added support for contexts (including for canceling and timing out)

Bump github.com/stretchr/testify from 1.7.5 to 1.8.0

Bump github.com/stretchr/testify from 1.7.4 to 1.7.5

Bump github.com/stretchr/testify from 1.7.2 to 1.7.4

Bump github.com/stretchr/testify from 1.7.1 to 1.7.2

Bump github.com/stretchr/testify from 1.8.0 to 1.8.1

Optimization: prepare config in init-time rather than in run-time

Related tags

Go-generic-unboxing - A quick ready to ship demo for go generic using the official example

[TOOL, CLI] - Filter and examine Go type structures, interfaces and their transitive dependencies and relationships. Export structural types as TypeScript value object or bare type representations.

State observer - StateObserver used to synchronize the local(cached) state of the remote object with the real state

Go worker pool

Optimistic rollup tech, minimal and generic.

Generic mapStringInterface tool for extracting of data for CSV output

Generic slices for Go 1.8+

Generic tools for go 1.18+

Ecsgo - Cache friendly, Multi threading Entity Component System in Go (with Generic)

Go-generic - A collection of experiments using Go Generics coming out in Go 1.18

Nune - High-performance numerical engine based on generic tensors

Generic-list-go - Go container/list but with generics

Nune-go - High-performance numerical engine based on generic tensors

Golang CS:GO external base. Development currently halted due to compiler/runtime Golang bugs.

Belajar Golang Install Golang

Golang-module-references - A reference for how to setup a Golang project with modules - Task Management + Math Examples

Golang-echo-sample - Make an out-of-the-box backend based on golang-echo

Minimalistic, pluggable Golang evloop/timer handler with dependency-injection

GoLang Library for Browser Capabilities Project