Kafka implemented in Golang with built-in coordination (No ZK dep, single binary install, Cloud Native)

Jocko

ci gitter codecov

Kafka/distributed commit log service in Go.

Goals of this project:

  • Implement Kafka in Go
  • Protocol compatible with Kafka so Kafka clients and services work with Jocko
  • Make operating simpler
  • Distribute a single binary
  • Use Serf for discovery, Raft for consensus (and remove the need to run ZooKeeper)
  • Smarter configuration settings
    • Able to use percentages of disk space for retention policies rather than only bytes and time kept
    • Handling size configs when you change the number of partitions or add topics
  • Learn a lot and have fun

TODO

  • Producing
  • Fetching
  • Partition consensus and distribution
  • Protocol
    • Produce
    • Fetch
    • Metadata
    • Create Topics
    • Delete Topics
    • Consumer group [current task]
  • Discovery
  • API versioning [more API versions to implement]
  • Replication [first draft done - testing heavily now]

Hiatus Writing Book

I’m writing a book for PragProg called Building Distributed Services with Go. You can sign up on this mailing list and get updated when the book’s available. It walks you through building a distributed commit log from scratch. I hope it will help Jocko contributors and people who want to work on distributed services.

Reading

Project Layout

├── broker        broker subsystem
├── cmd           commands
│   └── jocko     command to run a Jocko broker and manage topics
├── commitlog     low-level commit log implementation
├── examples      examples running/using Jocko
│   ├── cluster   example booting up a 3-broker Jocko cluster
│   └── sarama    example producing/consuming with Sarama
├── protocol      golang implementation of Kafka's protocol
├── prometheus    wrapper around Prometheus' client lib to handle metrics
├── server        API subsystem
└── testutil      test utils
    └── mock      mocks of the various subsystems

Building

Local

  1. Clone Jocko

    $ go get github.com/travisjeffery/jocko

  2. Build Jocko

    $ cd $GOPATH/src/github.com/travisjeffery/jocko
$ make build

    (If you see an error about dep not being found, ensure that $GOPATH/bin is in your PATH)

Docker

docker build -t travisjeffery/jocko:latest .

Contributing

See CONTRIBUTING for details on submitting patches and the contribution workflow.

License

Jocko is under the MIT license, see the LICENSE file for details.

Owner
Travis Jeffery
Worked at Basecamp, Segment, Confluent on Kafka/Cloud. Author of Distributed Services with Go.
Travis Jeffery
https://github.com/travisjeffery/jocko https://twitter.com/travisjeffery
Comments
  • broker tests #67

    broker tests #67

    • Test broker: New()
      • Broker struct is initialized properly
      • Serf.Bootstrap is called correctly
      • Raft.Bootstrap is called correctly
      • Handle all error cases
    • Test broker: Join()
      • Add test for serf error
    • Test broker: partition()
      • Add test for empty topic
    • Test broker: startReplica()
      • Add test for starting as leader
      • Add test for starting as follower
      • Add test for existing topic
      • Add test for duplicate partition
      • Add test for commitlog New() error
  • Cleanup cluster example

    Cleanup cluster example

    While trying to reproduce https://github.com/travisjeffery/jocko/issues/20 I've noticed that:

    • the cluster example is using outdated config flags - I found replacement for log-dir and prometheus-addr to be data-dir and http-addr, but I couldn't find replacement for debug so its usage is just removed (maybe DevMode is planned replacement, but that one is not configurable through CLI at the moment and it doesn't change zap logger so probably should be in separate commit)
    • broker-id and data-dir config flags were configuring config struct properties which weren't actually used, there were duplicate struct properties which were used but not configurable throgh CLI
    • jocko binary produced from both make build and by following cluster example is not gitignored so could be by accident committed to version control

    This PR fixes the 3 mentioned issues in 3 separate commits.

  • Move leadership from broker to raft

    Move leadership from broker to raft

    Changes:

    • moved leader.go from broker package to raft package to finish the separation between broker and raft

    Next steps:

    • add raft test (mock out serf)
    • add broker tests (mock out serf and raft)
    • modify server tests (mock out broker)
  • Refactor tests

    Refactor tests

    Changes done:

    • Moved tests to their own package
    • Exposed type replicator, re-wrote replicator tests without internals (using mocks)
    • Re-wrote serf tests without broker
    • Minor code changes in commitlog and server: stop accepting connections after server shutdown

    Next steps (TODO):

    • Raft: move leader.go from broker to raft package to finish their separation, write tests for raft
    • Broker: write tests for broker (mocking out serf/raft) to cover more cases
    • Server: modify server tests by mocking out broker, add tests to cover remaining request/response cases
  • Error opening raft store: missing port in address

    Error opening raft store: missing port in address

    Hi,

    I just followed the samara/README.md and got this error:

    $ go run main.go
Error opening raft store: missing port in address
exit status 1

    I'm using go version go1.7.5 darwin/amd64 on macOS Sierra 10.12.3 (16D32).

    I also tried go get -u github.com/kardianos/govendor and govendor sync - inspired by your Makefile - assuming some dependency version issues but got

    $ govendor sync
# cd .; git clone https://gopkg.in/alecthomas/kingpin.v2 /Users/stefanscheidt/Code/Go/.cache/govendor/gopkg.in/alecthomas/kingpin.v2
Cloning into '/Users/stefanscheidt/Code/Go/.cache/govendor/gopkg.in/alecthomas/kingpin.v2'...
error: RPC failed; HTTP 301 curl 22 The requested URL returned error: 301
fatal: The remote end hung up unexpectedly
Error: Remotes failed for:
	Failed for "gopkg.in/alecthomas/kingpin.v2" (failed to clone repo): exit status 128
  • Reader giving incorrect results

    Reader giving incorrect results

    It seems that the io.Reader created with NewReader can give incorrect results back. My understanding is that NewReader(offset, maxBytes) will create a reader that begins returning messages starting with the given offset. The following test shows a case where that is not true:

    func TestReader(t *testing.T) {
	var err error
	l := setup(t)
	defer cleanup(t)

	numMsgs := 10
	msgs := make([]commitlog.MessageSet, numMsgs)
	for i := 0; i < numMsgs; i++ {
		msgs[i] = commitlog.NewMessageSet(
			uint64(i), commitlog.NewMessage([]byte(strconv.Itoa(i))),
		)
	}
	for _, ms := range msgs {
		_, err := l.Append(ms)
		require.NoError(t, err)
	}
	idx := 4
	maxBytes := msgs[idx].Size()
	r, err := l.NewReader(int64(idx), maxBytes)
	require.NoError(t, err)

	p := make([]byte, maxBytes)
	_, err = r.Read(p)
	require.NoError(t, err)
	act := commitlog.MessageSet(p)
	require.Equal(t, int64(idx), act.Offset())
}

    --- FAIL: TestReader (0.00s)
        commitlog_test.go:55:
                        Error Trace:    commitlog_test.go:55
                        Error:          Not equal:
                                        expected: 4
                                        actual  : 7
                        Test:           TestReader

    This test inserts 10 MessageSets (offsets 0 to 9), then creates a reader at offset 4. We then check that the first MessageSet returned is offset 4. As you can see in this particular test, it's actually returning a message with offset 7.

    From a brief review of the code, there may be problems with findSegment and Segment.findEntry?

  • Working replication and broker failover

    Working replication and broker failover

    This PR marks a big milestone for Jocko adding a first working draft of replication and broker failover:

    • Working/tested replication/partition reassignment of failed brokers
    • Added broker healthchecks
    • Added tracing You can run Uber's jaegar with $ docker run -d -p5775:5775/udp -p6831:6831/udp -p6832:6832/udp -p5778:5778 -p16686:16686 -p14268:14268 -p9411:9411 jaegertracing/all-in-one, run jocko's tests with $ make test, and open localhost:16686 to see the traces.
    • Added verbose logs which you can enable with JOCKODEBUG="server=1 broker=1 fsm=1"

    Example of a trace:

    screen shot 2018-03-04 at 8 29 22 pm

  • Different logging library

    Different logging library

    Is there a reason for using travisjeffery/simplelog library for logging? Logrus would give us much more logging features. For jocko specific, I think different logging levels and logging output options that Logrus provides would come in handy.

  • Broker cleanup 2

    Broker cleanup 2

    Part 2 of Issue #22 @travisjeffery Made the changes suggested in PR #25

    Change set: Added wrapper over serf.MemberStatus and raft.FSM to cleanly hide hashicorp raft and serf from jocko behind interfaces

    Next steps: add/edit tests Will open a new issue for this

  • Please consider adding vendor dir to next release

    Please consider adding vendor dir to next release

    Hi!

    I am developing a port for FreeBSD where network access is not allowed during build phase. Thus I have no possibility of running go get or deps ensure. Therefor it would be very cool if vendor dir could be added. It would not only benefit FreeBSD, but it'd ensure consistency with other operating systems as well.

  • Ditch the gommap

    Ditch the gommap

    Hi, I just wanted to test the commit log but it looks like the index depends on the https://github.com/tysonmote/gommap which hasn't been updated for almost 5 years and does not work on Windows at all.

  • example: fix Sarama example

    example: fix Sarama example

    Sarama example did not run correctly, notably because the second value returned by NewTestServer is not a cancel function but a string with the directory containing Kafka data.

  • Project Status

    Project Status

    Hello team,

    Great job for putting such a lib. I would like to know whether the project is still ongoing or it is abandoned. I do know how difficult it is to combine day-job and open source project.

  • codahale/hdrhistogram repo url has been transferred under the github HdrHstogram umbrella

    codahale/hdrhistogram repo url has been transferred under the github HdrHstogram umbrella

    Problem

    The codahale/hdrhistogram repo has been transferred under the github HdrHstogram umbrella with the help from the original author in Sept 2020 (new repo url https://github.com/HdrHistogram/hdrhistogram-go). The main reasons are to group all implementations under the same roof and to provide more active contribution from the community as the original repository was archived several years ago.

    The dependency URL should be modified to point to the new repository URL. The tag "v0.9.0" was applied at the point of transfer and will reflect the exact code that was frozen in the original repository.

    If you are using Go modules, you can update to the exact point of transfer using the @v0.9.0 tag in your go get command.

    go mod edit -replace github.com/codahale/hdrhistogram=github.com/HdrHistogram/[email protected]

    Performance Improvements

    From the point of transfer, up until now (mon 16 aug 2021), we've released 3 versions that aim support the standard HdrHistogram serialization/exposition formats, and deeply improve READ performance. We recommend to update to the latest version.

  • Write to CommitLog and Index does not in a transaction.

    Write to CommitLog and Index does not in a transaction.

    Thanks a lot for your project, it is very helpful for me to learn kafka. When I see the code, I find that, writeing a meesage to CommitLog and Index does not in a transaction, so how to ensure the consistency, or maybe I miss something? Thanks again. I am a Chinese student, so forgive my poor English.

  • Add support for flex versions

    Add support for flex versions

    Flex version support as per the kafka protocol specification and as implemented in the confluent librdkafka library

    https://github.com/edenhill/librdkafka/blob/master/src/rdkafka_buf.h

  • bump gommap to 6edfc90 to support FreeBSD amd64

    bump gommap to 6edfc90 to support FreeBSD amd64

    I'd like to add jocko to FreeBSD ports - with this patch, we already get amd64 support for mmap. I will see what's missing to add arm64 support as well, but that will take a bit longer to work through.

    $ uname -a
FreeBSD 14.0-CURRENT main-n244674-a4f26914d4d5 GENERIC-NODEBUG  amd64

$ gmake
?   	github.com/travisjeffery/jocko/cmd/jocko	[no test files]
=== RUN   TestIndex
--- PASS: TestIndex (0.00s)
=== RUN   TestIndexScanner
--- PASS: TestIndexScanner (0.00s)
=== RUN   TestNewCommitLog
--- PASS: TestNewCommitLog (0.00s)
=== RUN   TestTruncate
--- PASS: TestTruncate (0.00s)
=== RUN   TestCleaner
--- PASS: TestCleaner (0.00s)
=== RUN   TestCompactCleaner
--- PASS: TestCompactCleaner (0.00s)
=== RUN   TestDeleteCleaner
--- PASS: TestDeleteCleaner (0.00s)
=== RUN   TestMessageSet
--- PASS: TestMessageSet (0.00s)
=== RUN   TestMessages
--- PASS: TestMessages (0.00s)
=== RUN   TestMessage
--- PASS: TestMessage (0.00s)
=== RUN   TestReader
=== RUN   TestReader/6
=== RUN   TestReader/60
=== RUN   TestReader/600
=== RUN   TestReader/6000
--- PASS: TestReader (0.00s)
    --- PASS: TestReader/6 (0.00s)
    --- PASS: TestReader/60 (0.00s)
    --- PASS: TestReader/600 (0.00s)
    --- PASS: TestReader/6000 (0.00s)
=== RUN   TestSegmentScanner
--- PASS: TestSegmentScanner (0.00s)
PASS
ok  	github.com/travisjeffery/jocko/commitlog	0.011s
=== RUN   TestNewBrokerLookup
--- PASS: TestNewBrokerLookup (0.00s)
=== RUN   TestBroker_Run
=== RUN   TestBroker_Run/api_versions
2021-02-11T12:58:35.975Z [INFO]  raft: Initial configuration (index=1): [{Suffrage:Voter ID:1 Address:127.0.0.1:10002}]
2021-02-11T12:58:35.975Z [INFO]  raft: Node at 127.0.0.1:10002 [Leader] entering Leader state
2021/02/11 12:58:35.975651 serf/1: [INFO] serf: EventMemberJoin: TestBroker_Run/api_versions-node-1 127.0.0.1
2021/02/11 12:58:35.975759 broker_test: leader/1: cluster leadership acquired
2021/02/11 12:58:35.975835 broker_test: broker/1: adding LAN server: 1
2021/02/11 12:58:35.975898 broker_test: broker/1: raft data found, disabling bootstrap mode: index: 3, path: /tmp/jocko-test-server-1962560940/raft
2021/02/11 12:58:35.975916 broker_test: leader/1: skipping self join since cluster is too small: servers: 1
2021/02/11 12:58:35.975946 broker_test: leader/1: member joined, marking health alive: TestBroker_Run/api_versions-node-1
2021/02/11 12:58:35.976237 broker_test: leader/1: skipping self join since cluster is too small: servers: 1
2021/02/11 12:58:36.002462 broker_test: broker/1: request: ctx: correlation id: 1, api key: 0, client: , size: 0
2021/02/11 12:58:36.002517 broker_test: broker/1: run done
2021/02/11 12:58:36.002650 broker_test: broker/1: shutting down broker
2021/02/11 12:58:36.002667 serf/1: [WARN] serf: Shutdown without a Leave
=== RUN   TestBroker_Run/create_topic_ok
2021-02-11T12:58:36.003Z [INFO]  raft: Initial configuration (index=1): [{Suffrage:Voter ID:1 Address:127.0.0.1:10006}]
2021-02-11T12:58:36.004Z [INFO]  raft: Node at 127.0.0.1:10006 [Leader] entering Leader state
2021/02/11 12:58:36.004368 serf/1: [INFO] serf: EventMemberJoin: TestBroker_Run/create_topic_ok-node-2 127.0.0.1
2021/02/11 12:58:36.004438 broker_test: leader/1: cluster leadership acquired
2021/02/11 12:58:36.004439 broker_test: broker/1: adding LAN server: 1
2021/02/11 12:58:36.004462 broker_test: broker/1: raft data found, disabling bootstrap mode: index: 2, path: /tmp/jocko-test-server-2946557211/raft
2021/02/11 12:58:36.004674 broker_test: leader/1: skipping self join since cluster is too small: servers: 1
2021/02/11 12:58:36.004707 broker_test: leader/1: member joined, marking health alive: TestBroker_Run/create_topic_ok-node-2
2021/02/11 12:58:36.004962 broker_test: leader/1: skipping self join since cluster is too small: servers: 1
2021/02/11 12:58:36.029736 broker_test: broker/1: request: ctx: correlation id: 1, api key: 0, client: , size: 0
2021/02/11 12:58:36.029788 broker_test: broker/1: run done
2021/02/11 12:58:36.029959 broker_test: broker/1: shutting down broker
2021/02/11 12:58:36.029974 serf/1: [WARN] serf: Shutdown without a Leave
=== RUN   TestBroker_Run/create_topic_invalid_replication_factor_error
2021-02-11T12:58:36.032Z [INFO]  raft: Initial configuration (index=1): [{Suffrage:Voter ID:1 Address:127.0.0.1:10010}]
2021-02-11T12:58:36.032Z [INFO]  raft: Node at 127.0.0.1:10010 [Leader] entering Leader state
2021/02/11 12:58:36.032578 serf/1: [INFO] serf: EventMemberJoin: TestBroker_Run/create_topic_invalid_replication_factor_error-node-3 127.0.0.1
2021/02/11 12:58:36.032635 broker_test: leader/1: cluster leadership acquired
2021/02/11 12:58:36.032643 broker_test: broker/1: adding LAN server: 1
2021/02/11 12:58:36.032708 broker_test: broker/1: raft data found, disabling bootstrap mode: index: 2, path: /tmp/jocko-test-server-3266246590/raft
2021/02/11 12:58:36.032860 broker_test: leader/1: skipping self join since cluster is too small: servers: 1
2021/02/11 12:58:36.032883 broker_test: leader/1: member joined, marking health alive: TestBroker_Run/create_topic_invalid_replication_factor_error-node-3
2021/02/11 12:58:36.033096 broker_test: leader/1: skipping self join since cluster is too small: servers: 1
2021/02/11 12:58:36.057713 broker_test: broker/1: request: ctx: correlation id: 1, api key: 0, client: , size: 0
2021/02/11 12:58:36.057764 broker_test: broker/1: run done
2021/02/11 12:58:36.057905 broker_test: broker/1: shutting down broker
2021/02/11 12:58:36.057923 serf/1: [WARN] serf: Shutdown without a Leave
=== RUN   TestBroker_Run/delete_topic
2021-02-11T12:58:36.059Z [INFO]  raft: Initial configuration (index=1): [{Suffrage:Voter ID:1 Address:127.0.0.1:10014}]
2021-02-11T12:58:36.059Z [INFO]  raft: Node at 127.0.0.1:10014 [Leader] entering Leader state
2021/02/11 12:58:36.059707 serf/1: [INFO] serf: EventMemberJoin: TestBroker_Run/delete_topic-node-4 127.0.0.1
2021/02/11 12:58:36.059762 broker_test: leader/1: cluster leadership acquired
2021/02/11 12:58:36.059763 broker_test: broker/1: adding LAN server: 1
2021/02/11 12:58:36.059785 broker_test: broker/1: raft data found, disabling bootstrap mode: index: 2, path: /tmp/jocko-test-server-4213653509/raft
2021/02/11 12:58:36.059921 broker_test: leader/1: skipping self join since cluster is too small: servers: 1
2021/02/11 12:58:36.059943 broker_test: leader/1: member joined, marking health alive: TestBroker_Run/delete_topic-node-4
2021/02/11 12:58:36.060146 broker_test: leader/1: skipping self join since cluster is too small: servers: 1
2021/02/11 12:58:36.085370 broker_test: broker/1: request: ctx: correlation id: 1, api key: 0, client: , size: 0
2021/02/11 12:58:36.085410 broker_test: broker/1: request: ctx: correlation id: 2, api key: 0, client: , size: 0
2021/02/11 12:58:36.085443 broker_test: broker/1: run done
2021/02/11 12:58:36.085563 broker_test: broker/1: shutting down broker
2021/02/11 12:58:36.085587 serf/1: [WARN] serf: Shutdown without a Leave
=== RUN   TestBroker_Run/offsets
2021-02-11T12:58:36.087Z [INFO]  raft: Initial configuration (index=1): [{Suffrage:Voter ID:1 Address:127.0.0.1:10018}]
2021-02-11T12:58:36.087Z [INFO]  raft: Node at 127.0.0.1:10018 [Leader] entering Leader state
2021/02/11 12:58:36.087735 serf/1: [INFO] serf: EventMemberJoin: TestBroker_Run/offsets-node-5 127.0.0.1
2021/02/11 12:58:36.087787 broker_test: leader/1: cluster leadership acquired
2021/02/11 12:58:36.087793 broker_test: broker/1: adding LAN server: 1
2021/02/11 12:58:36.087820 broker_test: broker/1: raft data found, disabling bootstrap mode: index: 2, path: /tmp/jocko-test-server-5947463584/raft
2021/02/11 12:58:36.087946 broker_test: leader/1: skipping self join since cluster is too small: servers: 1
2021/02/11 12:58:36.087969 broker_test: leader/1: member joined, marking health alive: TestBroker_Run/offsets-node-5
2021/02/11 12:58:36.088172 broker_test: leader/1: skipping self join since cluster is too small: servers: 1
2021/02/11 12:58:36.112922 broker_test: broker/1: request: ctx: correlation id: 1, api key: 0, client: , size: 0
2021/02/11 12:58:36.113463 broker_test: broker/1: request: ctx: correlation id: 2, api key: 0, client: , size: 0
2021/02/11 12:58:36.113487 broker_test: broker/1: produce: &protocol.ProduceRequest{APIVersion:0, TransactionalID:(*string)(nil), Acks:0, Timeout:100000000, TopicData:[]*protocol.TopicData{(*protocol.TopicData)(0xc0001904e0)}}
2021/02/11 12:58:36.113499 broker_test: broker/1: produce to partition: 0: &{test-topic [0xc00000f860]}
2021/02/11 12:58:36.113576 broker_test: broker/1: request: ctx: correlation id: 3, api key: 0, client: , size: 0
2021/02/11 12:58:36.113605 broker_test: broker/1: request: ctx: correlation id: 4, api key: 0, client: , size: 0
2021/02/11 12:58:36.113628 broker_test: broker/1: run done
2021/02/11 12:58:36.113785 broker_test: broker/1: shutting down broker
2021/02/11 12:58:36.113802 serf/1: [WARN] serf: Shutdown without a Leave
=== RUN   TestBroker_Run/fetch
2021-02-11T12:58:36.115Z [INFO]  raft: Initial configuration (index=1): [{Suffrage:Voter ID:1 Address:127.0.0.1:10022}]
2021-02-11T12:58:36.115Z [INFO]  raft: Node at 127.0.0.1:10022 [Leader] entering Leader state
2021/02/11 12:58:36.115475 serf/1: [INFO] serf: EventMemberJoin: TestBroker_Run/fetch-node-6 127.0.0.1
2021/02/11 12:58:36.115533 broker_test: leader/1: cluster leadership acquired
2021/02/11 12:58:36.115535 broker_test: broker/1: adding LAN server: 1
2021/02/11 12:58:36.115575 broker_test: broker/1: raft data found, disabling bootstrap mode: index: 2, path: /tmp/jocko-test-server-6039209599/raft
2021/02/11 12:58:36.115731 broker_test: leader/1: skipping self join since cluster is too small: servers: 1
2021/02/11 12:58:36.115774 broker_test: leader/1: member joined, marking health alive: TestBroker_Run/fetch-node-6
2021/02/11 12:58:36.115987 broker_test: leader/1: skipping self join since cluster is too small: servers: 1
2021/02/11 12:58:36.140719 broker_test: broker/1: request: ctx: correlation id: 1, api key: 0, client: , size: 0
2021/02/11 12:58:36.141226 broker_test: broker/1: request: ctx: correlation id: 2, api key: 0, client: , size: 0
2021/02/11 12:58:36.141246 broker_test: broker/1: produce: &protocol.ProduceRequest{APIVersion:0, TransactionalID:(*string)(nil), Acks:0, Timeout:100000000, TopicData:[]*protocol.TopicData{(*protocol.TopicData)(0xc0001907e0)}}
2021/02/11 12:58:36.141253 broker_test: broker/1: produce to partition: 0: &{test-topic [0xc00000f960]}
2021/02/11 12:58:36.141329 broker_test: broker/1: request: ctx: correlation id: 3, api key: 0, client: , size: 0
2021/02/11 12:58:36.141401 broker_test: broker/1: run done
2021/02/11 12:58:36.142078 broker_test: broker/1: shutting down broker
2021/02/11 12:58:36.142100 serf/1: [WARN] serf: Shutdown without a Leave
=== RUN   TestBroker_Run/metadata
2021-02-11T12:58:36.143Z [INFO]  raft: Initial configuration (index=1): [{Suffrage:Voter ID:1 Address:127.0.0.1:10026}]
2021-02-11T12:58:36.143Z [INFO]  raft: Node at 127.0.0.1:10026 [Leader] entering Leader state
2021/02/11 12:58:36.143552 serf/1: [INFO] serf: EventMemberJoin: TestBroker_Run/metadata-node-7 127.0.0.1
2021/02/11 12:58:36.143605 broker_test: leader/1: cluster leadership acquired
2021/02/11 12:58:36.143607 broker_test: broker/1: adding LAN server: 1
2021/02/11 12:58:36.143636 broker_test: broker/1: raft data found, disabling bootstrap mode: index: 2, path: /tmp/jocko-test-server-7717650898/raft
2021/02/11 12:58:36.143757 broker_test: leader/1: skipping self join since cluster is too small: servers: 1
2021/02/11 12:58:36.143792 broker_test: leader/1: member joined, marking health alive: TestBroker_Run/metadata-node-7
2021/02/11 12:58:36.143985 broker_test: leader/1: skipping self join since cluster is too small: servers: 1
2021/02/11 12:58:36.168698 broker_test: broker/1: request: ctx: correlation id: 1, api key: 0, client: , size: 0
2021/02/11 12:58:36.169167 broker_test: broker/1: request: ctx: correlation id: 2, api key: 0, client: , size: 0
2021/02/11 12:58:36.169184 broker_test: broker/1: produce: &protocol.ProduceRequest{APIVersion:0, TransactionalID:(*string)(nil), Acks:0, Timeout:100000000, TopicData:[]*protocol.TopicData{(*protocol.TopicData)(0xc000190a20)}}
2021/02/11 12:58:36.169191 broker_test: broker/1: produce to partition: 0: &{test-topic [0xc00000fa80]}
2021/02/11 12:58:36.169265 broker_test: broker/1: request: ctx: correlation id: 3, api key: 0, client: , size: 0
2021/02/11 12:58:36.169333 broker_test: broker/1: run done
2021/02/11 12:58:36.169514 broker_test: broker/1: shutting down broker
2021/02/11 12:58:36.169531 serf/1: [WARN] serf: Shutdown without a Leave
=== RUN   TestBroker_Run/produce_topic/partition_doesn't_exist_error
2021-02-11T12:58:36.170Z [INFO]  raft: Initial configuration (index=1): [{Suffrage:Voter ID:1 Address:127.0.0.1:10030}]
2021-02-11T12:58:36.170Z [INFO]  raft: Node at 127.0.0.1:10030 [Leader] entering Leader state
2021/02/11 12:58:36.171629 serf/1: [INFO] serf: EventMemberJoin: TestBroker_Run/produce_topic/partition_doesn't_exist_error-node-8 127.0.0.1
2021/02/11 12:58:36.171695 broker_test: broker/1: adding LAN server: 1
2021/02/11 12:58:36.171708 broker_test: leader/1: cluster leadership acquired
2021/02/11 12:58:36.171712 broker_test: broker/1: raft data found, disabling bootstrap mode: index: 2, path: /tmp/jocko-test-server-8714297609/raft
2021/02/11 12:58:36.171828 broker_test: leader/1: skipping self join since cluster is too small: servers: 1
2021/02/11 12:58:36.171864 broker_test: leader/1: member joined, marking health alive: TestBroker_Run/produce_topic/partition_doesn't_exist_error-node-8
2021/02/11 12:58:36.172057 broker_test: leader/1: skipping self join since cluster is too small: servers: 1
2021/02/11 12:58:36.196770 broker_test: broker/1: request: ctx: correlation id: 2, api key: 0, client: , size: 0
2021/02/11 12:58:36.196796 broker_test: broker/1: produce: &protocol.ProduceRequest{APIVersion:0, TransactionalID:(*string)(nil), Acks:0, Timeout:100000000, TopicData:[]*protocol.TopicData{(*protocol.TopicData)(0xc000190bd0)}}
2021/02/11 12:58:36.196805 broker_test: broker/1: produce to partition: 0: &{another-topic [0xc00000fba0]}
2021/02/11 12:58:36.196838 broker_test: broker/1: produce to partition error: unknown topic
2021/02/11 12:58:36.196889 broker_test: broker/1: run done
2021/02/11 12:58:36.197004 broker_test: broker/1: shutting down broker
2021/02/11 12:58:36.197023 serf/1: [WARN] serf: Shutdown without a Leave
=== RUN   TestBroker_Run/find_coordinator
2021-02-11T12:58:36.198Z [INFO]  raft: Initial configuration (index=1): [{Suffrage:Voter ID:1 Address:127.0.0.1:10034}]
2021-02-11T12:58:36.198Z [INFO]  raft: Node at 127.0.0.1:10034 [Leader] entering Leader state
2021/02/11 12:58:36.198644 serf/1: [INFO] serf: EventMemberJoin: TestBroker_Run/find_coordinator-node-9 127.0.0.1
2021/02/11 12:58:36.198708 broker_test: broker/1: adding LAN server: 1
2021/02/11 12:58:36.198746 broker_test: broker/1: raft data found, disabling bootstrap mode: index: 2, path: /tmp/jocko-test-server-9243442388/raft
2021/02/11 12:58:36.198780 broker_test: leader/1: cluster leadership acquired
2021/02/11 12:58:36.198852 broker_test: leader/1: skipping self join since cluster is too small: servers: 1
2021/02/11 12:58:36.198888 broker_test: leader/1: member joined, marking health alive: TestBroker_Run/find_coordinator-node-9
2021/02/11 12:58:36.199090 broker_test: leader/1: skipping self join since cluster is too small: servers: 1
2021/02/11 12:58:36.223912 broker_test: broker/1: request: ctx: correlation id: 1, api key: 0, client: , size: 0
2021/02/11 12:58:36.224412 broker_test: broker/1: request: ctx: correlation id: 3, api key: 0, client: , size: 0
2021/02/11 12:58:36.231678 broker_test: broker/1: run done
2021/02/11 12:58:36.231852 broker_test: broker/1: shutting down broker
2021/02/11 12:58:36.231873 serf/1: [WARN] serf: Shutdown without a Leave
--- PASS: TestBroker_Run (0.26s)
    --- PASS: TestBroker_Run/api_versions (0.03s)
    --- PASS: TestBroker_Run/create_topic_ok (0.03s)
    --- PASS: TestBroker_Run/create_topic_invalid_replication_factor_error (0.03s)
    --- PASS: TestBroker_Run/delete_topic (0.03s)
    --- PASS: TestBroker_Run/offsets (0.03s)
    --- PASS: TestBroker_Run/fetch (0.03s)
    --- PASS: TestBroker_Run/metadata (0.03s)
    --- PASS: TestBroker_Run/produce_topic/partition_doesn't_exist_error (0.03s)
    --- PASS: TestBroker_Run/find_coordinator (0.03s)
=== RUN   TestBroker_Run_JoinSyncGroup
    broker_test.go:569: 
--- SKIP: TestBroker_Run_JoinSyncGroup (0.00s)
=== RUN   TestBroker_Shutdown
=== RUN   TestBroker_Shutdown/shutdown_ok
2021-02-11T12:58:36.233Z [INFO]  raft: Initial configuration (index=1): [{Suffrage:Voter ID:10 Address:127.0.0.1:10038}]
2021-02-11T12:58:36.233Z [INFO]  raft: Node at 127.0.0.1:10038 [Follower] entering Follower state (Leader: "")
2021/02/11 12:58:36.233913 serf/10: [INFO] serf: EventMemberJoin: TestBroker_Shutdown/shutdown_ok-node-10 127.0.0.1
2021/02/11 12:58:36.233968 broker_test: server/10: run handler
2021/02/11 12:58:36.233974 broker_test: broker/10: shutting down broker
2021/02/11 12:58:36.233982 serf/10: [WARN] serf: Shutdown without a Leave
2021/02/11 12:58:36.234005 broker_test: broker/10: adding LAN server: 10
--- PASS: TestBroker_Shutdown (0.00s)
    --- PASS: TestBroker_Shutdown/shutdown_ok (0.00s)
=== RUN   TestBroker_JoinLAN
2021-02-11T12:58:36.235Z [INFO]  raft: Initial configuration (index=0): []
2021/02/11 12:58:36.235566 serf/11: [INFO] serf: EventMemberJoin: TestBroker_JoinLAN-node-11 127.0.0.1
2021-02-11T12:58:36.236Z [INFO]  raft: Node at 127.0.0.1:10042 [Follower] entering Follower state (Leader: "")
2021/02/11 12:58:36.236285 broker_test: broker/11: adding LAN server: 11
2021-02-11T12:58:36.236Z [INFO]  raft: Initial configuration (index=0): []
2021-02-11T12:58:36.236Z [INFO]  raft: Node at 127.0.0.1:10046 [Follower] entering Follower state (Leader: "")
2021/02/11 12:58:36.237309 serf/12: [INFO] serf: EventMemberJoin: TestBroker_JoinLAN-node-12 127.0.0.1
2021/02/11 12:58:36.237388 broker_test: broker/12: adding LAN server: 12
2021/02/11 12:58:36.237468 memberlist/12: [DEBUG] memberlist: Initiating push/pull sync with: 127.0.0.1:10043
2021/02/11 12:58:36.237491 memberlist/11: [DEBUG] memberlist: Stream connection from=127.0.0.1:22171
2021/02/11 12:58:36.237990 serf/11: [INFO] serf: EventMemberJoin: TestBroker_JoinLAN-node-12 127.0.0.1
2021/02/11 12:58:36.238066 broker_test: broker/11: adding LAN server: 12
2021/02/11 12:58:36.238115 serf/12: [INFO] serf: EventMemberJoin: TestBroker_JoinLAN-node-11 127.0.0.1
2021/02/11 12:58:36.238146 broker_test: broker/12: adding LAN server: 11
2021/02/11 12:58:36.238222 broker_test: broker/12: shutting down broker
2021/02/11 12:58:36.238234 serf/12: [WARN] serf: Shutdown without a Leave
2021/02/11 12:58:36.238468 broker_test: broker/11: shutting down broker
2021/02/11 12:58:36.238485 serf/11: [WARN] serf: Shutdown without a Leave
--- PASS: TestBroker_JoinLAN (0.00s)
=== RUN   TestBroker_RegisterMember
2021-02-11T12:58:36.239Z [INFO]  raft: Initial configuration (index=1): [{Suffrage:Voter ID:13 Address:127.0.0.1:10050}]
2021-02-11T12:58:36.239Z [INFO]  raft: Node at 127.0.0.1:10050 [Follower] entering Follower state (Leader: "")
2021/02/11 12:58:36.239914 serf/13: [INFO] serf: EventMemberJoin: TestBroker_RegisterMember-node-13 127.0.0.1
2021/02/11 12:58:36.239970 broker_test: broker/13: adding LAN server: 13
2021/02/11 12:58:36.239988 broker_test: broker/13: raft data found, disabling bootstrap mode: index: 1, path: /tmp/jocko-test-server-13386614088/raft
2021-02-11T12:58:36.240Z [INFO]  raft: Initial configuration (index=0): []
2021-02-11T12:58:36.240Z [INFO]  raft: Node at 127.0.0.1:10054 [Follower] entering Follower state (Leader: "")
2021/02/11 12:58:36.241086 serf/14: [INFO] serf: EventMemberJoin: TestBroker_RegisterMember-node-14 127.0.0.1
2021/02/11 12:58:36.241172 broker_test: broker/14: adding LAN server: 14
2021/02/11 12:58:36.241193 broker_test: broker/14: maybe bootstrap: need more brokers: got: 1: expect: 3
2021/02/11 12:58:36.241229 memberlist/13: [DEBUG] memberlist: Initiating push/pull sync with: 127.0.0.1:10055
2021/02/11 12:58:36.241256 memberlist/14: [DEBUG] memberlist: Stream connection from=127.0.0.1:22172
2021/02/11 12:58:36.241924 serf/14: [INFO] serf: EventMemberJoin: TestBroker_RegisterMember-node-13 127.0.0.1
2021/02/11 12:58:36.242087 broker_test: broker/14: adding LAN server: 13
2021/02/11 12:58:36.242183 broker_test: broker/14; member TestBroker_RegisterMember-node-13 has bootstrap mode. expect disabled
2021/02/11 12:58:36.242189 serf/13: [INFO] serf: EventMemberJoin: TestBroker_RegisterMember-node-14 127.0.0.1
2021/02/11 12:58:36.242408 broker_test: broker/13: adding LAN server: 14
2021/02/11 12:58:36.340469 serf/14: [DEBUG] serf: messageJoinType: TestBroker_RegisterMember-node-13
2021/02/11 12:58:36.340604 serf/14: [DEBUG] serf: messageJoinType: TestBroker_RegisterMember-node-13
2021/02/11 12:58:36.341461 serf/13: [DEBUG] serf: messageJoinType: TestBroker_RegisterMember-node-13
2021/02/11 12:58:36.341502 serf/13: [DEBUG] serf: messageJoinType: TestBroker_RegisterMember-node-13
2021/02/11 12:58:36.490092 broker_test: broker/13: state:
	members:
		- 0:
			name: TestBroker_RegisterMember-node-13
			addr: 127.0.0.1
			status: alive
		- 1:
			name: TestBroker_RegisterMember-node-14
			addr: 127.0.0.1
			status: alive
	nodes:
	topics:
2021/02/11 12:58:36.491327 broker_test: broker/14: state:
	members:
		- 0:
			name: TestBroker_RegisterMember-node-14
			addr: 127.0.0.1
			status: alive
		- 1:
			name: TestBroker_RegisterMember-node-13
			addr: 127.0.0.1
			status: alive
	nodes:
	topics:
2021-02-11T12:58:36.498Z [WARN]  raft: no known peers, aborting election
2021-02-11T12:58:36.638Z [WARN]  raft: Heartbeat timeout from "" reached, starting election
2021-02-11T12:58:36.638Z [INFO]  raft: Node at 127.0.0.1:10050 [Candidate] entering Candidate state in term 2
2021-02-11T12:58:36.639Z [DEBUG] raft: Votes needed: 1
2021-02-11T12:58:36.639Z [DEBUG] raft: Vote granted from 13 in term 2. Tally: 1
2021-02-11T12:58:36.639Z [INFO]  raft: Election won. Tally: 1
2021-02-11T12:58:36.639Z [INFO]  raft: Node at 127.0.0.1:10050 [Leader] entering Leader state
2021/02/11 12:58:36.639176 broker_test: leader/13: cluster leadership acquired
2021/02/11 12:58:36.639282 broker_test: leader/13: join cluster: add voter: 14
2021-02-11T12:58:36.639Z [INFO]  raft: Updating configuration with AddStaging (14, 127.0.0.1:10054) to [{Suffrage:Voter ID:13 Address:127.0.0.1:10050} {Suffrage:Voter ID:14 Address:127.0.0.1:10054}]
2021-02-11T12:58:36.639Z [INFO]  raft: Added peer 14, starting replication
2021/02/11 12:58:36.639441 broker_test: leader/13: member joined, marking health alive: TestBroker_RegisterMember-node-14
2021/02/11 12:58:36 [DEBUG] raft-net: 127.0.0.1:10054 accepted connection from: 127.0.0.1:22182
2021-02-11T12:58:36.639Z [WARN]  raft: Failed to get previous log: 4 log not found (last: 0)
2021-02-11T12:58:36.639Z [WARN]  raft: AppendEntries to {Voter 14 127.0.0.1:10054} rejected, sending older logs (next: 1)
2021-02-11T12:58:36.640Z [INFO]  raft: pipelining replication to peer {Voter 14 127.0.0.1:10054}
2021/02/11 12:58:36.640478 broker_test: leader/13: skipping self join since cluster is too small: servers: 2
2021/02/11 12:58:36.640512 broker_test: leader/13: member joined, marking health alive: TestBroker_RegisterMember-node-13
2021/02/11 12:58:36.646671 broker_test: broker/14: shutting down broker
2021/02/11 12:58:36.646693 serf/14: [WARN] serf: Shutdown without a Leave
2021/02/11 12:58:36.647044 broker_test: broker/13: shutting down broker
2021/02/11 12:58:36.647066 serf/13: [WARN] serf: Shutdown without a Leave
2021-02-11T12:58:36.647Z [INFO]  raft: aborting pipeline replication to peer {Voter 14 127.0.0.1:10054}
--- PASS: TestBroker_RegisterMember (0.41s)
=== RUN   TestBroker_FailedMember
2021-02-11T12:58:36.649Z [INFO]  raft: Initial configuration (index=1): [{Suffrage:Voter ID:15 Address:127.0.0.1:10058}]
2021-02-11T12:58:36.649Z [INFO]  raft: Node at 127.0.0.1:10058 [Leader] entering Leader state
2021/02/11 12:58:36.649731 serf/15: [INFO] serf: EventMemberJoin: TestBroker_FailedMember-node-15 127.0.0.1
2021/02/11 12:58:36.649798 broker_test: leader/15: cluster leadership acquired
2021/02/11 12:58:36.649817 broker_test: broker/15: adding LAN server: 15
2021/02/11 12:58:36.649840 broker_test: broker/15: raft data found, disabling bootstrap mode: index: 2, path: /tmp/jocko-test-server-15115152314/raft
2021/02/11 12:58:36.649895 broker_test: leader/15: skipping self join since cluster is too small: servers: 1
2021/02/11 12:58:36.649933 broker_test: leader/15: member joined, marking health alive: TestBroker_FailedMember-node-15
2021/02/11 12:58:36.650128 broker_test: leader/15: skipping self join since cluster is too small: servers: 1
2021-02-11T12:58:36.650Z [INFO]  raft: Initial configuration (index=0): []
2021-02-11T12:58:36.650Z [INFO]  raft: Node at 127.0.0.1:10062 [Follower] entering Follower state (Leader: "")
2021/02/11 12:58:36.650988 serf/16: [INFO] serf: EventMemberJoin: TestBroker_FailedMember-node-16 127.0.0.1
2021/02/11 12:58:36.651034 broker_test: broker/16: adding LAN server: 16
2021/02/11 12:58:36.651126 memberlist/15: [DEBUG] memberlist: Initiating push/pull sync with: 127.0.0.1:10063
2021/02/11 12:58:36.651163 memberlist/16: [DEBUG] memberlist: Stream connection from=127.0.0.1:22183
2021/02/11 12:58:36.651557 serf/16: [INFO] serf: EventMemberJoin: TestBroker_FailedMember-node-15 127.0.0.1
2021/02/11 12:58:36.651606 broker_test: broker/16: adding LAN server: 15
2021/02/11 12:58:36.651717 serf/15: [INFO] serf: EventMemberJoin: TestBroker_FailedMember-node-16 127.0.0.1
2021/02/11 12:58:36.651758 broker_test: broker/15: adding LAN server: 16
2021/02/11 12:58:36.651775 broker_test: broker/16: shutting down broker
2021/02/11 12:58:36.651787 serf/16: [WARN] serf: Shutdown without a Leave
2021-02-11T12:58:36.651Z [INFO]  raft: Updating configuration with AddNonvoter (16, 127.0.0.1:10062) to [{Suffrage:Voter ID:15 Address:127.0.0.1:10058} {Suffrage:Nonvoter ID:16 Address:127.0.0.1:10062}]
2021-02-11T12:58:36.651Z [INFO]  raft: Added peer 16, starting replication
2021/02/11 12:58:36.651919 broker_test: leader/15: member joined, marking health alive: TestBroker_FailedMember-node-16
2021/02/11 12:58:36.677177 broker_test: broker/15: shutting down broker
2021/02/11 12:58:36.677196 serf/15: [WARN] serf: Shutdown without a Leave
2021-02-11T12:58:46.652Z [ERROR] raft: Failed to AppendEntries to {Nonvoter 16 127.0.0.1:10062}: dial tcp 127.0.0.1:10062: i/o timeout
2021-02-11T12:58:46.673Z [ERROR] raft: Failed to heartbeat to 127.0.0.1:10062: dial tcp 127.0.0.1:10062: i/o timeout
--- PASS: TestBroker_FailedMember (10.03s)
=== RUN   TestBroker_LeftMember
2021-02-11T12:58:46.676Z [INFO]  raft: Initial configuration (index=1): [{Suffrage:Voter ID:17 Address:127.0.0.1:10066}]
2021-02-11T12:58:46.676Z [INFO]  raft: Node at 127.0.0.1:10066 [Leader] entering Leader state
2021/02/11 12:58:46.676858 serf/17: [INFO] serf: EventMemberJoin: TestBroker_LeftMember-node-17 127.0.0.1
2021/02/11 12:58:46.676932 broker_test: leader/17: cluster leadership acquired
2021/02/11 12:58:46.676975 broker_test: broker/17: adding LAN server: 17
2021/02/11 12:58:46.677002 broker_test: broker/17: raft data found, disabling bootstrap mode: index: 2, path: /tmp/jocko-test-server-17924346108/raft
2021/02/11 12:58:46.677107 broker_test: leader/17: skipping self join since cluster is too small: servers: 1
2021/02/11 12:58:46.677143 broker_test: leader/17: member joined, marking health alive: TestBroker_LeftMember-node-17
2021/02/11 12:58:46.677416 broker_test: leader/17: skipping self join since cluster is too small: servers: 1
2021-02-11T12:58:46.678Z [INFO]  raft: Initial configuration (index=0): []
2021-02-11T12:58:46.678Z [INFO]  raft: Node at 127.0.0.1:10070 [Follower] entering Follower state (Leader: "")
2021/02/11 12:58:46.678576 serf/18: [INFO] serf: EventMemberJoin: TestBroker_LeftMember-node-18 127.0.0.1
2021/02/11 12:58:46.678638 broker_test: broker/18: adding LAN server: 18
2021/02/11 12:58:46.678729 memberlist/17: [DEBUG] memberlist: Initiating push/pull sync with: 127.0.0.1:10071
2021/02/11 12:58:46.678766 memberlist/18: [DEBUG] memberlist: Stream connection from=127.0.0.1:22272
2021/02/11 12:58:46.679156 serf/18: [INFO] serf: EventMemberJoin: TestBroker_LeftMember-node-17 127.0.0.1
2021/02/11 12:58:46.679201 broker_test: broker/18: adding LAN server: 17
2021/02/11 12:58:46.679247 serf/17: [INFO] serf: EventMemberJoin: TestBroker_LeftMember-node-18 127.0.0.1
2021/02/11 12:58:46.679291 broker_test: broker/17: adding LAN server: 18
2021-02-11T12:58:46.679Z [INFO]  raft: Updating configuration with AddNonvoter (18, 127.0.0.1:10070) to [{Suffrage:Voter ID:17 Address:127.0.0.1:10066} {Suffrage:Nonvoter ID:18 Address:127.0.0.1:10070}]
2021-02-11T12:58:46.679Z [INFO]  raft: Added peer 18, starting replication
2021/02/11 12:58:46.679462 broker_test: leader/17: member joined, marking health alive: TestBroker_LeftMember-node-18
2021/02/11 12:58:46 [DEBUG] raft-net: 127.0.0.1:10070 accepted connection from: 127.0.0.1:22273
2021-02-11T12:58:46.679Z [WARN]  raft: Failed to get previous log: 5 log not found (last: 0)
2021-02-11T12:58:46.679Z [WARN]  raft: AppendEntries to {Nonvoter 18 127.0.0.1:10070} rejected, sending older logs (next: 1)
2021-02-11T12:58:46.680Z [INFO]  raft: pipelining replication to peer {Nonvoter 18 127.0.0.1:10070}
2021/02/11 12:58:46.704442 broker_test: broker/18: starting leave
2021/02/11 12:58:46 [DEBUG] raft-net: 127.0.0.1:10070 accepted connection from: 127.0.0.1:22274
2021/02/11 12:58:46.778088 serf/18: [DEBUG] serf: messageJoinType: TestBroker_LeftMember-node-17
2021/02/11 12:58:46.778564 serf/18: [DEBUG] serf: messageJoinType: TestBroker_LeftMember-node-17
2021/02/11 12:58:46.778684 serf/17: [DEBUG] serf: messageJoinType: TestBroker_LeftMember-node-17
2021/02/11 12:58:46.778718 serf/18: [INFO] serf: EventMemberLeave: TestBroker_LeftMember-node-18 127.0.0.1
2021/02/11 12:58:46.778721 serf/17: [DEBUG] serf: messageLeaveType: TestBroker_LeftMember-node-18
2021/02/11 12:58:46.778755 broker_test: broker/18: removing LAN server: TestBroker_LeftMember-node-18
2021/02/11 12:58:46.779045 serf/17: [DEBUG] serf: messageJoinType: TestBroker_LeftMember-node-17
2021/02/11 12:58:46.779092 serf/18: [DEBUG] serf: messageLeaveType: TestBroker_LeftMember-node-18
2021/02/11 12:58:46.779119 serf/17: [DEBUG] serf: messageLeaveType: TestBroker_LeftMember-node-18
2021/02/11 12:58:46.877073 serf/18: [DEBUG] serf: messageLeaveType: TestBroker_LeftMember-node-18
2021/02/11 12:58:46.878873 serf/17: [INFO] serf: EventMemberLeave: TestBroker_LeftMember-node-18 127.0.0.1
2021/02/11 12:58:46.878902 broker_test: broker/17: removing LAN server: TestBroker_LeftMember-node-18
2021/02/11 12:58:46.878931 broker_test: leader/17: removing server by id: 18
2021-02-11T12:58:46.878Z [INFO]  raft: Updating configuration with RemoveServer (18, ) to [{Suffrage:Voter ID:17 Address:127.0.0.1:10066}]
2021-02-11T12:58:46.879Z [INFO]  raft: Removed peer 18, stopping replication after 7
2021/02/11 12:58:46.879231 broker_test: leader/17: member is deregistering: reason: left; node: 18
2021-02-11T12:58:46.879Z [INFO]  raft: aborting pipeline replication to peer {Nonvoter 18 127.0.0.1:10070}
2021/02/11 12:58:46 [ERR] raft-net: Failed to flush response: write tcp 127.0.0.1:10070->127.0.0.1:22273: write: broken pipe
2021/02/11 12:58:46.927226 broker_test: broker/17: state:
	members:
		- 0:
			name: TestBroker_LeftMember-node-17
			addr: 127.0.0.1
			status: alive
		- 1:
			name: TestBroker_LeftMember-node-18
			addr: 127.0.0.1
			status: left
	nodes:
		- 0:
			id: 17
			status: passing
	topics:
2021/02/11 12:58:46.928842 broker_test: broker/18: state:
	members:
		- 0:
			name: TestBroker_LeftMember-node-17
			addr: 127.0.0.1
			status: alive
		- 1:
			name: TestBroker_LeftMember-node-18
			addr: 127.0.0.1
			status: left
	nodes:
		- 0:
			id: 17
			status: passing
		- 1:
			id: 18
			status: passing
	topics:
2021/02/11 12:58:46.977115 broker_test: leader/17: skipping self join since cluster is too small: servers: 1
2021/02/11 12:58:47.177017 broker_test: broker/17: state:
	members:
		- 0:
			name: TestBroker_LeftMember-node-17
			addr: 127.0.0.1
			status: alive
		- 1:
			name: TestBroker_LeftMember-node-18
			addr: 127.0.0.1
			status: left
	nodes:
		- 0:
			id: 17
			status: passing
	topics:
2021/02/11 12:58:47.178704 broker_test: broker/18: state:
	members:
		- 0:
			name: TestBroker_LeftMember-node-18
			addr: 127.0.0.1
			status: left
		- 1:
			name: TestBroker_LeftMember-node-17
			addr: 127.0.0.1
			status: alive
	nodes:
		- 0:
			id: 17
			status: passing
		- 1:
			id: 18
			status: passing
	topics:
2021/02/11 12:58:47.278541 broker_test: leader/17: skipping self join since cluster is too small: servers: 1
2021-02-11T12:58:47.349Z [WARN]  raft: not part of stable configuration, aborting election
2021/02/11 12:58:47.427102 broker_test: broker/17: state:
	members:
		- 0:
			name: TestBroker_LeftMember-node-17
			addr: 127.0.0.1
			status: alive
		- 1:
			name: TestBroker_LeftMember-node-18
			addr: 127.0.0.1
			status: left
	nodes:
		- 0:
			id: 17
			status: passing
	topics:
2021/02/11 12:58:47.428784 broker_test: broker/18: state:
	members:
		- 0:
			name: TestBroker_LeftMember-node-18
			addr: 127.0.0.1
			status: left
		- 1:
			name: TestBroker_LeftMember-node-17
			addr: 127.0.0.1
			status: alive
	nodes:
		- 0:
			id: 17
			status: passing
		- 1:
			id: 18
			status: passing
	topics:
2021/02/11 12:58:47.578682 broker_test: leader/17: skipping self join since cluster is too small: servers: 1
2021/02/11 12:58:47.677040 broker_test: broker/17: state:
	members:
		- 0:
			name: TestBroker_LeftMember-node-17
			addr: 127.0.0.1
			status: alive
		- 1:
			name: TestBroker_LeftMember-node-18
			addr: 127.0.0.1
			status: left
	nodes:
		- 0:
			id: 17
			status: passing
	topics:
2021/02/11 12:58:47.678681 broker_test: broker/18: state:
	members:
		- 0:
			name: TestBroker_LeftMember-node-18
			addr: 127.0.0.1
			status: left
		- 1:
			name: TestBroker_LeftMember-node-17
			addr: 127.0.0.1
			status: alive
	nodes:
		- 0:
			id: 17
			status: passing
		- 1:
			id: 18
			status: passing
	topics:
2021/02/11 12:58:47.878796 broker_test: leader/17: skipping self join since cluster is too small: servers: 1
2021/02/11 12:58:47.927054 broker_test: broker/17: state:
	members:
		- 0:
			name: TestBroker_LeftMember-node-17
			addr: 127.0.0.1
			status: alive
		- 1:
			name: TestBroker_LeftMember-node-18
			addr: 127.0.0.1
			status: left
	nodes:
		- 0:
			id: 17
			status: passing
	topics:
2021/02/11 12:58:47.928782 broker_test: broker/18: state:
	members:
		- 0:
			name: TestBroker_LeftMember-node-18
			addr: 127.0.0.1
			status: left
		- 1:
			name: TestBroker_LeftMember-node-17
			addr: 127.0.0.1
			status: alive
	nodes:
		- 0:
			id: 17
			status: passing
		- 1:
			id: 18
			status: passing
	topics:
2021/02/11 12:58:48.030607 broker_test: broker/18: shutting down broker
2021/02/11 12:58:48.031079 broker_test: broker/17: shutting down broker
2021/02/11 12:58:48.031144 serf/17: [WARN] serf: Shutdown without a Leave
--- PASS: TestBroker_LeftMember (1.36s)
=== RUN   TestBroker_ReapLeader
2021-02-11T12:58:48.033Z [INFO]  raft: Initial configuration (index=1): [{Suffrage:Voter ID:19 Address:127.0.0.1:10074}]
2021-02-11T12:58:48.033Z [INFO]  raft: Node at 127.0.0.1:10074 [Follower] entering Follower state (Leader: "")
2021/02/11 12:58:48.034025 serf/19: [INFO] serf: EventMemberJoin: TestBroker_ReapLeader-node-19 127.0.0.1
2021/02/11 12:58:48.034396 broker_test: broker/19: adding LAN server: 19
2021-02-11T12:58:48.035Z [INFO]  raft: Initial configuration (index=0): []
2021/02/11 12:58:48.036356 serf/20: [INFO] serf: EventMemberJoin: TestBroker_ReapLeader-node-20 127.0.0.1
2021-02-11T12:58:48.036Z [INFO]  raft: Node at 127.0.0.1:10078 [Follower] entering Follower state (Leader: "")
2021/02/11 12:58:48.036560 broker_test: broker/20: adding LAN server: 20
2021-02-11T12:58:48.037Z [INFO]  raft: Initial configuration (index=0): []
2021-02-11T12:58:48.037Z [INFO]  raft: Node at 127.0.0.1:10082 [Follower] entering Follower state (Leader: "")
2021/02/11 12:58:48.038149 serf/21: [INFO] serf: EventMemberJoin: TestBroker_ReapLeader-node-21 127.0.0.1
2021/02/11 12:58:48.038285 broker_test: broker/21: adding LAN server: 21
2021/02/11 12:58:48.038352 memberlist/20: [DEBUG] memberlist: Initiating push/pull sync with: 127.0.0.1:10075
2021/02/11 12:58:48.038515 memberlist/19: [DEBUG] memberlist: Stream connection from=127.0.0.1:22283
2021/02/11 12:58:48.038986 serf/19: [INFO] serf: EventMemberJoin: TestBroker_ReapLeader-node-20 127.0.0.1
2021/02/11 12:58:48.039166 serf/20: [INFO] serf: EventMemberJoin: TestBroker_ReapLeader-node-19 127.0.0.1
2021/02/11 12:58:48.039178 broker_test: broker/19: adding LAN server: 20
2021/02/11 12:58:48.039291 broker_test: broker/20: adding LAN server: 19
2021/02/11 12:58:48.039462 memberlist/21: [DEBUG] memberlist: Initiating push/pull sync with: 127.0.0.1:10075
2021/02/11 12:58:48.039645 memberlist/19: [DEBUG] memberlist: Stream connection from=127.0.0.1:22284
2021/02/11 12:58:48.041132 serf/19: [INFO] serf: EventMemberJoin: TestBroker_ReapLeader-node-21 127.0.0.1
2021/02/11 12:58:48.041320 broker_test: broker/19: adding LAN server: 21
2021/02/11 12:58:48.041673 serf/21: [INFO] serf: EventMemberJoin: TestBroker_ReapLeader-node-20 127.0.0.1
2021/02/11 12:58:48.041842 serf/21: [INFO] serf: EventMemberJoin: TestBroker_ReapLeader-node-19 127.0.0.1
2021/02/11 12:58:48.041862 broker_test: broker/21: adding LAN server: 20
2021/02/11 12:58:48.042174 broker_test: broker/21: adding LAN server: 19
2021/02/11 12:58:48.135683 serf/20: [INFO] serf: EventMemberJoin: TestBroker_ReapLeader-node-21 127.0.0.1
2021/02/11 12:58:48.135892 broker_test: broker/20: adding LAN server: 21
2021/02/11 12:58:48.136122 serf/19: [DEBUG] serf: messageJoinType: TestBroker_ReapLeader-node-20
2021/02/11 12:58:48.136877 serf/21: [DEBUG] serf: messageJoinType: TestBroker_ReapLeader-node-20
2021/02/11 12:58:48.137086 serf/20: [DEBUG] serf: messageJoinType: TestBroker_ReapLeader-node-20
2021/02/11 12:58:48.138664 serf/20: [DEBUG] serf: messageJoinType: TestBroker_ReapLeader-node-21
2021/02/11 12:58:48.138692 serf/20: [DEBUG] serf: messageJoinType: TestBroker_ReapLeader-node-20
2021/02/11 12:58:48.138940 serf/21: [DEBUG] serf: messageJoinType: TestBroker_ReapLeader-node-20
2021/02/11 12:58:48.138972 serf/19: [DEBUG] serf: messageJoinType: TestBroker_ReapLeader-node-21
2021/02/11 12:58:48.138991 serf/19: [DEBUG] serf: messageJoinType: TestBroker_ReapLeader-node-20
2021/02/11 12:58:48.234410 serf/20: [DEBUG] serf: messageJoinType: TestBroker_ReapLeader-node-21
2021/02/11 12:58:48.234594 serf/21: [DEBUG] serf: messageJoinType: TestBroker_ReapLeader-node-21
2021/02/11 12:58:48.236661 serf/19: [DEBUG] serf: messageJoinType: TestBroker_ReapLeader-node-21
2021/02/11 12:58:48.236810 serf/21: [DEBUG] serf: messageJoinType: TestBroker_ReapLeader-node-21
2021-02-11T12:58:48.238Z [WARN]  raft: Heartbeat timeout from "" reached, starting election
2021-02-11T12:58:48.238Z [INFO]  raft: Node at 127.0.0.1:10074 [Candidate] entering Candidate state in term 2
2021-02-11T12:58:48.238Z [DEBUG] raft: Votes needed: 1
2021-02-11T12:58:48.238Z [DEBUG] raft: Vote granted from 19 in term 2. Tally: 1
2021-02-11T12:58:48.238Z [INFO]  raft: Election won. Tally: 1
2021-02-11T12:58:48.238Z [INFO]  raft: Node at 127.0.0.1:10074 [Leader] entering Leader state
2021/02/11 12:58:48.238867 broker_test: leader/19: cluster leadership acquired
2021/02/11 12:58:48.238989 broker_test: leader/19: skipping self join since cluster is too small: servers: 1
2021/02/11 12:58:48.239052 broker_test: leader/19: member joined, marking health alive: TestBroker_ReapLeader-node-19
2021/02/11 12:58:48.239385 broker_test: leader/19: join cluster: add voter: 20
2021-02-11T12:58:48.239Z [INFO]  raft: Updating configuration with AddStaging (20, 127.0.0.1:10078) to [{Suffrage:Voter ID:19 Address:127.0.0.1:10074} {Suffrage:Voter ID:20 Address:127.0.0.1:10078}]
2021-02-11T12:58:48.239Z [INFO]  raft: Added peer 20, starting replication
2021/02/11 12:58:48.239561 broker_test: leader/19: member joined, marking health alive: TestBroker_ReapLeader-node-20
2021/02/11 12:58:48 [DEBUG] raft-net: 127.0.0.1:10078 accepted connection from: 127.0.0.1:22285
2021-02-11T12:58:48.239Z [WARN]  raft: Failed to get previous log: 5 log not found (last: 0)
2021-02-11T12:58:48.240Z [WARN]  raft: AppendEntries to {Voter 20 127.0.0.1:10078} rejected, sending older logs (next: 1)
2021-02-11T12:58:48.240Z [INFO]  raft: pipelining replication to peer {Voter 20 127.0.0.1:10078}
2021/02/11 12:58:48.240952 broker_test: leader/19: join cluster: add voter: 21
2021-02-11T12:58:48.241Z [INFO]  raft: Updating configuration with AddStaging (21, 127.0.0.1:10082) to [{Suffrage:Voter ID:19 Address:127.0.0.1:10074} {Suffrage:Voter ID:20 Address:127.0.0.1:10078} {Suffrage:Voter ID:21 Address:127.0.0.1:10082}]
2021-02-11T12:58:48.241Z [INFO]  raft: Added peer 21, starting replication
2021/02/11 12:58:48 [DEBUG] raft-net: 127.0.0.1:10082 accepted connection from: 127.0.0.1:22286
2021-02-11T12:58:48.241Z [WARN]  raft: Failed to get previous log: 7 log not found (last: 0)
2021/02/11 12:58:48.241646 broker_test: leader/19: member joined, marking health alive: TestBroker_ReapLeader-node-21
2021-02-11T12:58:48.241Z [WARN]  raft: AppendEntries to {Voter 21 127.0.0.1:10082} rejected, sending older logs (next: 1)
2021-02-11T12:58:48.242Z [INFO]  raft: pipelining replication to peer {Voter 21 127.0.0.1:10082}
2021/02/11 12:58:48.247242 broker_test: leader/19: removing server by id: 21
2021-02-11T12:58:48.247Z [INFO]  raft: Updating configuration with RemoveServer (21, ) to [{Suffrage:Voter ID:19 Address:127.0.0.1:10074} {Suffrage:Voter ID:20 Address:127.0.0.1:10078}]
2021-02-11T12:58:48.247Z [INFO]  raft: Removed peer 21, stopping replication after 9
2021-02-11T12:58:48.247Z [INFO]  raft: aborting pipeline replication to peer {Voter 21 127.0.0.1:10082}
2021/02/11 12:58:48.247822 broker_test: leader/19: member is deregistering: reason: reaped; node: 21
2021/02/11 12:58:48.248474 broker_test: broker/21: shutting down broker
2021/02/11 12:58:48.248500 serf/21: [WARN] serf: Shutdown without a Leave
2021/02/11 12:58:48.248919 broker_test: broker/20: shutting down broker
2021/02/11 12:58:48.248950 serf/20: [WARN] serf: Shutdown without a Leave
2021/02/11 12:58:48.249283 broker_test: broker/19: shutting down broker
2021/02/11 12:58:48.249343 serf/19: [WARN] serf: Shutdown without a Leave
2021-02-11T12:58:48.249Z [INFO]  raft: aborting pipeline replication to peer {Voter 20 127.0.0.1:10078}
--- PASS: TestBroker_ReapLeader (0.22s)
=== RUN   TestBroker_ReapMember
2021-02-11T12:58:48.251Z [INFO]  raft: Initial configuration (index=1): [{Suffrage:Voter ID:22 Address:127.0.0.1:10086}]
2021-02-11T12:58:48.251Z [INFO]  raft: Node at 127.0.0.1:10086 [Follower] entering Follower state (Leader: "")
2021/02/11 12:58:48.252253 serf/22: [INFO] serf: EventMemberJoin: TestBroker_ReapMember-node-22 127.0.0.1
2021/02/11 12:58:48.252425 broker_test: broker/22: adding LAN server: 22
2021-02-11T12:58:48.253Z [INFO]  raft: Initial configuration (index=0): []
2021-02-11T12:58:48.253Z [INFO]  raft: Node at 127.0.0.1:10090 [Follower] entering Follower state (Leader: "")
2021/02/11 12:58:48.254108 serf/23: [INFO] serf: EventMemberJoin: TestBroker_ReapMember-node-23 127.0.0.1
2021/02/11 12:58:48.254233 broker_test: broker/23: adding LAN server: 23
2021/02/11 12:58:48.254322 memberlist/23: [DEBUG] memberlist: Initiating push/pull sync with: 127.0.0.1:10087
2021/02/11 12:58:48.254345 memberlist/22: [DEBUG] memberlist: Stream connection from=127.0.0.1:22287
2021/02/11 12:58:48.254869 serf/22: [INFO] serf: EventMemberJoin: TestBroker_ReapMember-node-23 127.0.0.1
2021/02/11 12:58:48.254944 broker_test: broker/22: adding LAN server: 23
2021/02/11 12:58:48.255170 serf/23: [INFO] serf: EventMemberJoin: TestBroker_ReapMember-node-22 127.0.0.1
2021/02/11 12:58:48.255223 broker_test: broker/23: adding LAN server: 22
2021/02/11 12:58:48.353174 serf/22: [DEBUG] serf: messageJoinType: TestBroker_ReapMember-node-23
2021/02/11 12:58:48.354419 serf/22: [DEBUG] serf: messageJoinType: TestBroker_ReapMember-node-23
2021/02/11 12:58:48.354549 serf/23: [DEBUG] serf: messageJoinType: TestBroker_ReapMember-node-23
2021/02/11 12:58:48.452518 serf/23: [DEBUG] serf: messageJoinType: TestBroker_ReapMember-node-23
2021-02-11T12:58:48.459Z [WARN]  raft: Heartbeat timeout from "" reached, starting election
2021-02-11T12:58:48.459Z [INFO]  raft: Node at 127.0.0.1:10086 [Candidate] entering Candidate state in term 2
2021-02-11T12:58:48.459Z [DEBUG] raft: Votes needed: 1
2021-02-11T12:58:48.459Z [DEBUG] raft: Vote granted from 22 in term 2. Tally: 1
2021-02-11T12:58:48.459Z [INFO]  raft: Election won. Tally: 1
2021-02-11T12:58:48.459Z [INFO]  raft: Node at 127.0.0.1:10086 [Leader] entering Leader state
2021/02/11 12:58:48.459596 broker_test: leader/22: cluster leadership acquired
2021/02/11 12:58:48.459703 broker_test: leader/22: skipping self join since cluster is too small: servers: 1
2021/02/11 12:58:48.459756 broker_test: leader/22: member joined, marking health alive: TestBroker_ReapMember-node-22
2021/02/11 12:58:48.460055 broker_test: leader/22: join cluster: add voter: 23
2021-02-11T12:58:48.460Z [INFO]  raft: Updating configuration with AddStaging (23, 127.0.0.1:10090) to [{Suffrage:Voter ID:22 Address:127.0.0.1:10086} {Suffrage:Voter ID:23 Address:127.0.0.1:10090}]
2021-02-11T12:58:48.460Z [INFO]  raft: Added peer 23, starting replication
2021/02/11 12:58:48.460228 broker_test: leader/22: member joined, marking health alive: TestBroker_ReapMember-node-23
2021/02/11 12:58:48 [DEBUG] raft-net: 127.0.0.1:10090 accepted connection from: 127.0.0.1:22288
2021-02-11T12:58:48.460Z [WARN]  raft: Failed to get previous log: 5 log not found (last: 0)
2021-02-11T12:58:48.460Z [WARN]  raft: AppendEntries to {Voter 23 127.0.0.1:10090} rejected, sending older logs (next: 1)
2021-02-11T12:58:48.461Z [INFO]  raft: pipelining replication to peer {Voter 23 127.0.0.1:10090}
2021/02/11 12:58:48.482137 broker_test: leader/22: removing server by id: 23
2021-02-11T12:58:48.482Z [INFO]  raft: Updating configuration with RemoveServer (23, ) to [{Suffrage:Voter ID:22 Address:127.0.0.1:10086}]
2021-02-11T12:58:48.482Z [INFO]  raft: Removed peer 23, stopping replication after 7
2021/02/11 12:58:48.482405 broker_test: leader/22: member is deregistering: reason: reaped; node: 23
2021-02-11T12:58:48.482Z [INFO]  raft: aborting pipeline replication to peer {Voter 23 127.0.0.1:10090}
2021/02/11 12:58:48.482634 broker_test: broker/23: shutting down broker
2021/02/11 12:58:48.482664 serf/23: [WARN] serf: Shutdown without a Leave
2021/02/11 12:58:48 [ERR] raft-net: Failed to flush response: write tcp 127.0.0.1:10090->127.0.0.1:22288: write: broken pipe
2021/02/11 12:58:48.483153 broker_test: broker/22: shutting down broker
2021/02/11 12:58:48.483179 serf/22: [WARN] serf: Shutdown without a Leave
--- PASS: TestBroker_ReapMember (0.23s)
=== RUN   TestBroker_LeftLeader
2021-02-11T12:58:48.485Z [INFO]  raft: Initial configuration (index=1): [{Suffrage:Voter ID:24 Address:127.0.0.1:10094}]
2021-02-11T12:58:48.485Z [INFO]  raft: Node at 127.0.0.1:10094 [Follower] entering Follower state (Leader: "")
2021/02/11 12:58:48.485659 serf/24: [INFO] serf: EventMemberJoin: TestBroker_LeftLeader-node-24 127.0.0.1
2021/02/11 12:58:48.485784 broker_test: broker/24: adding LAN server: 24
2021/02/11 12:58:48.485845 broker_test: broker/24: raft data found, disabling bootstrap mode: index: 1, path: /tmp/jocko-test-server-24293516697/raft
2021-02-11T12:58:48.486Z [INFO]  raft: Initial configuration (index=0): []
2021-02-11T12:58:48.486Z [INFO]  raft: Node at 127.0.0.1:10098 [Follower] entering Follower state (Leader: "")
2021/02/11 12:58:48.487334 serf/25: [INFO] serf: EventMemberJoin: TestBroker_LeftLeader-node-25 127.0.0.1
2021/02/11 12:58:48.487438 broker_test: broker/25: adding LAN server: 25
2021/02/11 12:58:48.487462 broker_test: broker/25: maybe bootstrap: need more brokers: got: 1: expect: 3
2021-02-11T12:58:48.488Z [INFO]  raft: Initial configuration (index=0): []
2021-02-11T12:58:48.488Z [INFO]  raft: Node at 127.0.0.1:10102 [Follower] entering Follower state (Leader: "")
2021/02/11 12:58:48.488986 serf/26: [INFO] serf: EventMemberJoin: TestBroker_LeftLeader-node-26 127.0.0.1
2021/02/11 12:58:48.489062 broker_test: broker/26: adding LAN server: 26
2021/02/11 12:58:48.489078 broker_test: broker/26: maybe bootstrap: need more brokers: got: 1: expect: 3
2021/02/11 12:58:48.489168 memberlist/24: [DEBUG] memberlist: Initiating push/pull sync with: 127.0.0.1:10099
2021/02/11 12:58:48.489210 memberlist/25: [DEBUG] memberlist: Stream connection from=127.0.0.1:22289
2021/02/11 12:58:48.489649 serf/25: [INFO] serf: EventMemberJoin: TestBroker_LeftLeader-node-24 127.0.0.1
2021/02/11 12:58:48.489718 broker_test: broker/25: adding LAN server: 24
2021/02/11 12:58:48.489746 broker_test: broker/25; member TestBroker_LeftLeader-node-24 has bootstrap mode. expect disabled
2021/02/11 12:58:48.489861 serf/24: [INFO] serf: EventMemberJoin: TestBroker_LeftLeader-node-25 127.0.0.1
2021/02/11 12:58:48.489917 broker_test: broker/24: adding LAN server: 25
2021/02/11 12:58:48.490118 memberlist/24: [DEBUG] memberlist: Initiating push/pull sync with: 127.0.0.1:10103
2021/02/11 12:58:48.490161 memberlist/26: [DEBUG] memberlist: Stream connection from=127.0.0.1:22290
2021/02/11 12:58:48.491058 serf/26: [INFO] serf: EventMemberJoin: TestBroker_LeftLeader-node-25 127.0.0.1
2021/02/11 12:58:48.491166 serf/26: [INFO] serf: EventMemberJoin: TestBroker_LeftLeader-node-24 127.0.0.1
2021/02/11 12:58:48.491343 broker_test: broker/26: adding LAN server: 25
2021/02/11 12:58:48.491412 serf/24: [INFO] serf: EventMemberJoin: TestBroker_LeftLeader-node-26 127.0.0.1
2021/02/11 12:58:48.491434 broker_test: broker/26; member TestBroker_LeftLeader-node-24 has bootstrap mode. expect disabled
2021/02/11 12:58:48.491561 broker_test: broker/26: adding LAN server: 24
2021/02/11 12:58:48.491606 broker_test: broker/26; member TestBroker_LeftLeader-node-24 has bootstrap mode. expect disabled
2021/02/11 12:58:48.491687 broker_test: broker/24: adding LAN server: 26
2021/02/11 12:58:48.586551 serf/25: [INFO] serf: EventMemberJoin: TestBroker_LeftLeader-node-26 127.0.0.1
2021/02/11 12:58:48.586597 serf/25: [DEBUG] serf: messageJoinType: TestBroker_LeftLeader-node-24
2021/02/11 12:58:48.586620 serf/25: [DEBUG] serf: messageJoinType: TestBroker_LeftLeader-node-24
2021/02/11 12:58:48.586650 broker_test: broker/25: adding LAN server: 26
2021/02/11 12:58:48.586704 broker_test: broker/25; member TestBroker_LeftLeader-node-24 has bootstrap mode. expect disabled
2021/02/11 12:58:48.586828 serf/26: [DEBUG] serf: messageJoinType: TestBroker_LeftLeader-node-24
2021/02/11 12:58:48.586851 serf/26: [DEBUG] serf: messageJoinType: TestBroker_LeftLeader-node-24
2021/02/11 12:58:48.587807 serf/24: [DEBUG] serf: messageJoinType: TestBroker_LeftLeader-node-24
2021/02/11 12:58:48.587827 serf/24: [DEBUG] serf: messageJoinType: TestBroker_LeftLeader-node-24
2021/02/11 12:58:48.587988 serf/24: [DEBUG] serf: messageJoinType: TestBroker_LeftLeader-node-24
2021/02/11 12:58:48.588011 serf/24: [DEBUG] serf: messageJoinType: TestBroker_LeftLeader-node-24
2021/02/11 12:58:48.589492 serf/25: [DEBUG] serf: messageJoinType: TestBroker_LeftLeader-node-24
2021/02/11 12:58:48.589556 serf/24: [DEBUG] serf: messageJoinType: TestBroker_LeftLeader-node-24
2021/02/11 12:58:48.735927 broker_test: broker/24: state:
	members:
		- 0:
			name: TestBroker_LeftLeader-node-24
			addr: 127.0.0.1
			status: alive
		- 1:
			name: TestBroker_LeftLeader-node-25
			addr: 127.0.0.1
			status: alive
		- 2:
			name: TestBroker_LeftLeader-node-26
			addr: 127.0.0.1
			status: alive
	nodes:
	topics:
2021/02/11 12:58:48.737564 broker_test: broker/25: state:
	members:
		- 0:
			name: TestBroker_LeftLeader-node-25
			addr: 127.0.0.1
			status: alive
		- 1:
			name: TestBroker_LeftLeader-node-24
			addr: 127.0.0.1
			status: alive
		- 2:
			name: TestBroker_LeftLeader-node-26
			addr: 127.0.0.1
			status: alive
	nodes:
	topics:
2021/02/11 12:58:48.739225 broker_test: broker/26: state:
	members:
		- 0:
			name: TestBroker_LeftLeader-node-26
			addr: 127.0.0.1
			status: alive
		- 1:
			name: TestBroker_LeftLeader-node-25
			addr: 127.0.0.1
			status: alive
		- 2:
			name: TestBroker_LeftLeader-node-24
			addr: 127.0.0.1
			status: alive
	nodes:
	topics:
2021-02-11T12:58:48.800Z [WARN]  raft: no known peers, aborting election
2021-02-11T12:58:48.841Z [WARN]  raft: Heartbeat timeout from "" reached, starting election
2021-02-11T12:58:48.841Z [INFO]  raft: Node at 127.0.0.1:10094 [Candidate] entering Candidate state in term 2
2021-02-11T12:58:48.841Z [DEBUG] raft: Votes needed: 1
2021-02-11T12:58:48.841Z [DEBUG] raft: Vote granted from 24 in term 2. Tally: 1
2021-02-11T12:58:48.841Z [INFO]  raft: Election won. Tally: 1
2021-02-11T12:58:48.841Z [INFO]  raft: Node at 127.0.0.1:10094 [Leader] entering Leader state
2021/02/11 12:58:48.841667 broker_test: leader/24: cluster leadership acquired
2021/02/11 12:58:48.841831 broker_test: leader/24: skipping self join since cluster is too small: servers: 1
2021/02/11 12:58:48.841903 broker_test: leader/24: member joined, marking health alive: TestBroker_LeftLeader-node-24
2021/02/11 12:58:48.842205 broker_test: leader/24: join cluster: add voter: 25
2021-02-11T12:58:48.842Z [INFO]  raft: Updating configuration with AddStaging (25, 127.0.0.1:10098) to [{Suffrage:Voter ID:24 Address:127.0.0.1:10094} {Suffrage:Voter ID:25 Address:127.0.0.1:10098}]
2021-02-11T12:58:48.842Z [INFO]  raft: Added peer 25, starting replication
2021/02/11 12:58:48.842381 broker_test: leader/24: member joined, marking health alive: TestBroker_LeftLeader-node-25
2021/02/11 12:58:48 [DEBUG] raft-net: 127.0.0.1:10098 accepted connection from: 127.0.0.1:22300
2021-02-11T12:58:48.842Z [WARN]  raft: Failed to get previous log: 5 log not found (last: 0)
2021-02-11T12:58:48.842Z [WARN]  raft: AppendEntries to {Voter 25 127.0.0.1:10098} rejected, sending older logs (next: 1)
2021-02-11T12:58:48.843Z [INFO]  raft: pipelining replication to peer {Voter 25 127.0.0.1:10098}
2021/02/11 12:58:48.843565 broker_test: leader/24: join cluster: add voter: 26
2021-02-11T12:58:48.843Z [INFO]  raft: Updating configuration with AddStaging (26, 127.0.0.1:10102) to [{Suffrage:Voter ID:24 Address:127.0.0.1:10094} {Suffrage:Voter ID:25 Address:127.0.0.1:10098} {Suffrage:Voter ID:26 Address:127.0.0.1:10102}]
2021-02-11T12:58:48.843Z [INFO]  raft: Added peer 26, starting replication
2021/02/11 12:58:48 [DEBUG] raft-net: 127.0.0.1:10102 accepted connection from: 127.0.0.1:22301
2021-02-11T12:58:48.844Z [WARN]  raft: Failed to get previous log: 7 log not found (last: 0)
2021-02-11T12:58:48.844Z [WARN]  raft: AppendEntries to {Voter 26 127.0.0.1:10102} rejected, sending older logs (next: 1)
2021/02/11 12:58:48.844236 broker_test: leader/24: member joined, marking health alive: TestBroker_LeftLeader-node-26
2021/02/11 12:58:48.851151 broker_test: broker/24: starting leave
2021-02-11T12:58:48.851Z [INFO]  raft: Updating configuration with RemoveServer (24, ) to [{Suffrage:Voter ID:25 Address:127.0.0.1:10098} {Suffrage:Voter ID:26 Address:127.0.0.1:10102}]
2021-02-11T12:58:48.851Z [INFO]  raft: pipelining replication to peer {Voter 26 127.0.0.1:10102}
2021-02-11T12:58:48.852Z [INFO]  raft: Removed ourself, shutting down
2021/02/11 12:58:48.852260 broker_test: leader/24: shutting down leader loop
2021-02-11T12:58:48.852Z [INFO]  raft: aborting pipeline replication to peer {Voter 25 127.0.0.1:10098}
2021-02-11T12:58:48.852Z [INFO]  raft: aborting pipeline replication to peer {Voter 26 127.0.0.1:10102}
2021/02/11 12:58:48.852273 broker_test: leader/24: cluster leadership lost
2021/02/11 12:58:48.885970 serf/26: [DEBUG] serf: messageLeaveType: TestBroker_LeftLeader-node-24
2021/02/11 12:58:48.885986 serf/24: [INFO] serf: EventMemberLeave: TestBroker_LeftLeader-node-24 127.0.0.1
2021/02/11 12:58:48.886022 broker_test: broker/24: removing LAN server: TestBroker_LeftLeader-node-24
2021/02/11 12:58:48.886202 serf/25: [DEBUG] serf: messageLeaveType: TestBroker_LeftLeader-node-24
2021/02/11 12:58:48.887550 serf/24: [DEBUG] serf: messageLeaveType: TestBroker_LeftLeader-node-24
2021/02/11 12:58:48.887764 serf/24: [DEBUG] serf: messageLeaveType: TestBroker_LeftLeader-node-24
2021/02/11 12:58:48.887842 serf/25: [INFO] serf: EventMemberLeave: TestBroker_LeftLeader-node-24 127.0.0.1
2021/02/11 12:58:48.887873 broker_test: broker/25: removing LAN server: TestBroker_LeftLeader-node-24
2021/02/11 12:58:48.889278 serf/24: [DEBUG] serf: messageLeaveType: TestBroker_LeftLeader-node-24
2021/02/11 12:58:48.889408 serf/24: [DEBUG] serf: messageLeaveType: TestBroker_LeftLeader-node-24
2021/02/11 12:58:48.889476 serf/26: [INFO] serf: EventMemberLeave: TestBroker_LeftLeader-node-24 127.0.0.1
2021/02/11 12:58:48.889501 broker_test: broker/26: removing LAN server: TestBroker_LeftLeader-node-24
2021/02/11 12:58:48.985843 broker_test: broker/24: state:
	members:
		- 0:
			name: TestBroker_LeftLeader-node-24
			addr: 127.0.0.1
			status: left
		- 1:
			name: TestBroker_LeftLeader-node-25
			addr: 127.0.0.1
			status: alive
		- 2:
			name: TestBroker_LeftLeader-node-26
			addr: 127.0.0.1
			status: alive
	nodes:
		- 0:
			id: 24
			status: passing
		- 1:
			id: 25
			status: passing
		- 2:
			id: 26
			status: passing
	topics:
2021/02/11 12:58:48.987458 broker_test: broker/25: state:
	members:
		- 0:
			name: TestBroker_LeftLeader-node-24
			addr: 127.0.0.1
			status: left
		- 1:
			name: TestBroker_LeftLeader-node-26
			addr: 127.0.0.1
			status: alive
		- 2:
			name: TestBroker_LeftLeader-node-25
			addr: 127.0.0.1
			status: alive
	nodes:
		- 0:
			id: 24
			status: passing
		- 1:
			id: 25
			status: passing
		- 2:
			id: 26
			status: passing
	topics:
2021/02/11 12:58:48.989141 broker_test: broker/26: state:
	members:
		- 0:
			name: TestBroker_LeftLeader-node-26
			addr: 127.0.0.1
			status: alive
		- 1:
			name: TestBroker_LeftLeader-node-25
			addr: 127.0.0.1
			status: alive
		- 2:
			name: TestBroker_LeftLeader-node-24
			addr: 127.0.0.1
			status: left
	nodes:
		- 0:
			id: 24
			status: passing
		- 1:
			id: 25
			status: passing
		- 2:
			id: 26
			status: passing
	topics:
2021-02-11T12:58:49.181Z [WARN]  raft: Heartbeat timeout from "127.0.0.1:10094" reached, starting election
2021-02-11T12:58:49.181Z [INFO]  raft: Node at 127.0.0.1:10098 [Candidate] entering Candidate state in term 3
2021-02-11T12:58:49.181Z [DEBUG] raft: Votes needed: 2
2021-02-11T12:58:49.181Z [DEBUG] raft: Vote granted from 25 in term 3. Tally: 1
2021/02/11 12:58:49 [DEBUG] raft-net: 127.0.0.1:10102 accepted connection from: 127.0.0.1:22302
2021-02-11T12:58:49.181Z [WARN]  raft: Rejecting vote request from 127.0.0.1:10098 since we have a leader: 127.0.0.1:10094
2021-02-11T12:58:49.218Z [WARN]  raft: Heartbeat timeout from "127.0.0.1:10094" reached, starting election
2021-02-11T12:58:49.218Z [INFO]  raft: Node at 127.0.0.1:10102 [Candidate] entering Candidate state in term 3
2021-02-11T12:58:49.218Z [DEBUG] raft: Votes needed: 2
2021-02-11T12:58:49.218Z [DEBUG] raft: Vote granted from 26 in term 3. Tally: 1
2021/02/11 12:58:49 [DEBUG] raft-net: 127.0.0.1:10098 accepted connection from: 127.0.0.1:22303
2021-02-11T12:58:49.218Z [INFO]  raft: Duplicate RequestVote for same term: 3
2021/02/11 12:58:49.235858 broker_test: broker/24: state:
	members:
		- 0:
			name: TestBroker_LeftLeader-node-24
			addr: 127.0.0.1
			status: left
		- 1:
			name: TestBroker_LeftLeader-node-25
			addr: 127.0.0.1
			status: alive
		- 2:
			name: TestBroker_LeftLeader-node-26
			addr: 127.0.0.1
			status: alive
	nodes:
		- 0:
			id: 24
			status: passing
		- 1:
			id: 25
			status: passing
		- 2:
			id: 26
			status: passing
	topics:
2021/02/11 12:58:49.237512 broker_test: broker/25: state:
	members:
		- 0:
			name: TestBroker_LeftLeader-node-25
			addr: 127.0.0.1
			status: alive
		- 1:
			name: TestBroker_LeftLeader-node-24
			addr: 127.0.0.1
			status: left
		- 2:
			name: TestBroker_LeftLeader-node-26
			addr: 127.0.0.1
			status: alive
	nodes:
		- 0:
			id: 24
			status: passing
		- 1:
			id: 25
			status: passing
		- 2:
			id: 26
			status: passing
	topics:
2021/02/11 12:58:49.239146 broker_test: broker/26: state:
	members:
		- 0:
			name: TestBroker_LeftLeader-node-26
			addr: 127.0.0.1
			status: alive
		- 1:
			name: TestBroker_LeftLeader-node-25
			addr: 127.0.0.1
			status: alive
		- 2:
			name: TestBroker_LeftLeader-node-24
			addr: 127.0.0.1
			status: left
	nodes:
		- 0:
			id: 24
			status: passing
		- 1:
			id: 25
			status: passing
		- 2:
			id: 26
			status: passing
	topics:
2021-02-11T12:58:49.403Z [WARN]  raft: Election timeout reached, restarting election
2021-02-11T12:58:49.403Z [INFO]  raft: Node at 127.0.0.1:10098 [Candidate] entering Candidate state in term 4
2021-02-11T12:58:49.403Z [DEBUG] raft: Votes needed: 2
2021-02-11T12:58:49.403Z [DEBUG] raft: Vote granted from 25 in term 4. Tally: 1
2021-02-11T12:58:49.403Z [DEBUG] raft: lost leadership because received a requestvote with newer term
2021-02-11T12:58:49.403Z [INFO]  raft: Node at 127.0.0.1:10102 [Follower] entering Follower state (Leader: "")
2021-02-11T12:58:49.403Z [DEBUG] raft: Vote granted from 26 in term 4. Tally: 2
2021-02-11T12:58:49.403Z [INFO]  raft: Election won. Tally: 2
2021-02-11T12:58:49.403Z [INFO]  raft: Node at 127.0.0.1:10098 [Leader] entering Leader state
2021-02-11T12:58:49.403Z [INFO]  raft: Added peer 26, starting replication
2021/02/11 12:58:49.403863 broker_test: leader/25: cluster leadership acquired
2021-02-11T12:58:49.404Z [INFO]  raft: pipelining replication to peer {Voter 26 127.0.0.1:10102}
2021/02/11 12:58:49.404401 broker_test: leader/25: member is deregistering: reason: left; node: 24
2021/02/11 12:58:49.404885 broker_test: leader/25: skipping self join since cluster is too small: servers: 2
2021/02/11 12:58:49 [DEBUG] raft-net: 127.0.0.1:10102 accepted connection from: 127.0.0.1:22304
2021/02/11 12:58:49.485822 broker_test: broker/24: state:
	members:
		- 0:
			name: TestBroker_LeftLeader-node-24
			addr: 127.0.0.1
			status: left
		- 1:
			name: TestBroker_LeftLeader-node-25
			addr: 127.0.0.1
			status: alive
		- 2:
			name: TestBroker_LeftLeader-node-26
			addr: 127.0.0.1
			status: alive
	nodes:
		- 0:
			id: 24
			status: passing
		- 1:
			id: 25
			status: passing
		- 2:
			id: 26
			status: passing
	topics:
2021/02/11 12:58:49.487478 broker_test: broker/25: state:
	members:
		- 0:
			name: TestBroker_LeftLeader-node-25
			addr: 127.0.0.1
			status: alive
		- 1:
			name: TestBroker_LeftLeader-node-24
			addr: 127.0.0.1
			status: left
		- 2:
			name: TestBroker_LeftLeader-node-26
			addr: 127.0.0.1
			status: alive
	nodes:
		- 0:
			id: 25
			status: passing
		- 1:
			id: 26
			status: passing
	topics:
2021/02/11 12:58:49.489119 broker_test: broker/26: state:
	members:
		- 0:
			name: TestBroker_LeftLeader-node-26
			addr: 127.0.0.1
			status: alive
		- 1:
			name: TestBroker_LeftLeader-node-25
			addr: 127.0.0.1
			status: alive
		- 2:
			name: TestBroker_LeftLeader-node-24
			addr: 127.0.0.1
			status: left
	nodes:
		- 0:
			id: 24
			status: passing
		- 1:
			id: 25
			status: passing
		- 2:
			id: 26
			status: passing
	topics:
2021/02/11 12:58:49.704396 broker_test: leader/25: skipping self join since cluster is too small: servers: 2
2021/02/11 12:58:49.735886 broker_test: broker/24: state:
	members:
		- 0:
			name: TestBroker_LeftLeader-node-24
			addr: 127.0.0.1
			status: left
		- 1:
			name: TestBroker_LeftLeader-node-25
			addr: 127.0.0.1
			status: alive
		- 2:
			name: TestBroker_LeftLeader-node-26
			addr: 127.0.0.1
			status: alive
	nodes:
		- 0:
			id: 24
			status: passing
		- 1:
			id: 25
			status: passing
		- 2:
			id: 26
			status: passing
	topics:
2021/02/11 12:58:49.737495 broker_test: broker/25: state:
	members:
		- 0:
			name: TestBroker_LeftLeader-node-26
			addr: 127.0.0.1
			status: alive
		- 1:
			name: TestBroker_LeftLeader-node-25
			addr: 127.0.0.1
			status: alive
		- 2:
			name: TestBroker_LeftLeader-node-24
			addr: 127.0.0.1
			status: left
	nodes:
		- 0:
			id: 25
			status: passing
		- 1:
			id: 26
			status: passing
	topics:
2021/02/11 12:58:49.739200 broker_test: broker/26: state:
	members:
		- 0:
			name: TestBroker_LeftLeader-node-26
			addr: 127.0.0.1
			status: alive
		- 1:
			name: TestBroker_LeftLeader-node-25
			addr: 127.0.0.1
			status: alive
		- 2:
			name: TestBroker_LeftLeader-node-24
			addr: 127.0.0.1
			status: left
	nodes:
		- 0:
			id: 25
			status: passing
		- 1:
			id: 26
			status: passing
	topics:
2021/02/11 12:58:49.890382 broker_test: broker/24: shutting down broker
2021/02/11 12:58:49.890615 broker_test: broker/26: shutting down broker
2021/02/11 12:58:49.890639 serf/26: [WARN] serf: Shutdown without a Leave
2021/02/11 12:58:49.890936 broker_test: broker/25: shutting down broker
2021/02/11 12:58:49.890958 serf/25: [WARN] serf: Shutdown without a Leave
2021-02-11T12:58:49.891Z [INFO]  raft: aborting pipeline replication to peer {Voter 26 127.0.0.1:10102}
2021/02/11 12:58:49.891326 broker_test: broker/24: shutting down broker
--- PASS: TestBroker_LeftLeader (1.41s)
=== RUN   TestConn
2021-02-11T12:58:49.892Z [INFO]  raft: Initial configuration (index=1): [{Suffrage:Voter ID:27 Address:127.0.0.1:10106}]
2021-02-11T12:58:49.892Z [INFO]  raft: Node at 127.0.0.1:10106 [Leader] entering Leader state
2021/02/11 12:58:49.893226 serf/27: [INFO] serf: EventMemberJoin: TestConn-node-27 127.0.0.1
2021/02/11 12:58:49.893290 broker_test: server/27: run handler
=== RUN   TestConn/close_immediately
2021/02/11 12:58:49.893303 broker_test: leader/27: cluster leadership acquired
2021/02/11 12:58:49.893312 broker_test: broker/27: adding LAN server: 27
2021/02/11 12:58:49.893334 broker_test: broker/27: raft data found, disabling bootstrap mode: index: 2, path: /tmp/jocko-test-server-27020760310/raft
2021/02/11 12:58:49.893463 broker_test: leader/27: skipping self join since cluster is too small: servers: 1
=== RUN   TestConn/create_topice
2021/02/11 12:58:49.893505 broker_test: leader/27: member joined, marking health alive: TestConn-node-27
2021/02/11 12:58:49.893709 broker_test: server/27: handle request: ctx: correlation id: 1, api key: 19, client: , size: 44
2021/02/11 12:58:49.893739 broker_test: broker/27: request: ctx: correlation id: 1, api key: 19, client: , size: 44
2021/02/11 12:58:49.893778 broker_test: leader/27: skipping self join since cluster is too small: servers: 1
2021/02/11 12:58:49.893789 broker_test: server/27: handle response: ctx: correlation id: 1, api key: 19, client: , size: 44
=== RUN   TestConn/leader_and_isr
2021/02/11 12:58:49.894100 broker_test: server/27: handle request: ctx: correlation id: 1, api key: 4, client: , size: 74
2021/02/11 12:58:49.894116 broker_test: broker/27: request: ctx: correlation id: 1, api key: 4, client: , size: 74
2021/02/11 12:58:49.894142 broker_test: server/27: handle response: ctx: correlation id: 1, api key: 4, client: , size: 74
=== RUN   TestConn/fetch
2021/02/11 12:58:49.894440 broker_test: server/27: handle request: ctx: correlation id: 1, api key: 1, client: , size: 58
2021/02/11 12:58:49.894464 broker_test: broker/27: request: ctx: correlation id: 1, api key: 1, client: , size: 58
2021/02/11 12:58:49.894494 broker_test: server/27: handle response: ctx: correlation id: 1, api key: 1, client: , size: 58
=== RUN   TestConn/alter_configs
    conn_test.go:187: 
2021/02/11 12:58:49.894750 broker_test: broker/27: shutting down broker
2021/02/11 12:58:49.894766 serf/27: [WARN] serf: Shutdown without a Leave
2021/02/11 12:58:49.894906 broker_test: server/27: listener accept error: accept tcp 127.0.0.1:10105: use of closed network connection
--- PASS: TestConn (0.00s)
    --- PASS: TestConn/close_immediately (0.00s)
    --- PASS: TestConn/create_topice (0.00s)
    --- PASS: TestConn/leader_and_isr (0.00s)
    --- PASS: TestConn/fetch (0.00s)
    --- SKIP: TestConn/alter_configs (0.00s)
=== RUN   TestNewReplicaLookup
--- PASS: TestNewReplicaLookup (0.00s)
=== RUN   TestBroker_Replicate
--- PASS: TestBroker_Replicate (0.01s)
=== RUN   TestProduceConsume
    server_test.go:30: 
--- SKIP: TestProduceConsume (0.00s)
=== RUN   TestConsumerGroup
    server_test.go:176: 
--- SKIP: TestConsumerGroup (0.00s)
PASS
ok  	github.com/travisjeffery/jocko/jocko	13.944s
?   	github.com/travisjeffery/jocko/jocko/config	[no test files]
=== RUN   TestRegisterNode
--- PASS: TestRegisterNode (0.00s)
=== RUN   TestRegisterTopic
--- PASS: TestRegisterTopic (0.00s)
=== RUN   TestRegisterPartition
--- PASS: TestRegisterPartition (0.00s)
=== RUN   TestRegisterGroup
--- PASS: TestRegisterGroup (0.00s)
=== RUN   TestStore_maxIndex
--- PASS: TestStore_maxIndex (0.00s)
=== RUN   TestStore_Abandon
--- PASS: TestStore_Abandon (0.00s)
=== RUN   TestStore_DeleteNode
--- PASS: TestStore_DeleteNode (0.00s)
=== RUN   TestStore_RegisterTopic
--- PASS: TestStore_RegisterTopic (0.00s)
=== RUN   TestStore_RegisterPartition
--- PASS: TestStore_RegisterPartition (0.00s)
=== RUN   TestStore_RegisterGroup
--- PASS: TestStore_RegisterGroup (0.00s)
PASS
ok  	github.com/travisjeffery/jocko/jocko/fsm	(cached)
=== RUN   TestIsBroker
=== RUN   TestIsBroker/minumum_config
--- PASS: TestIsBroker (0.00s)
    --- PASS: TestIsBroker/minumum_config (0.00s)
PASS
ok  	github.com/travisjeffery/jocko/jocko/metadata	(cached)
?   	github.com/travisjeffery/jocko/jocko/state	[no test files]
=== RUN   TestEncodeDecode
=== RUN   TestEncodeDecode/partition
=== RUN   TestEncodeDecode/group
--- PASS: TestEncodeDecode (0.00s)
    --- PASS: TestEncodeDecode/partition (0.00s)
    --- PASS: TestEncodeDecode/group (0.00s)
PASS
ok  	github.com/travisjeffery/jocko/jocko/structs	(cached)
?   	github.com/travisjeffery/jocko/jocko/util	[no test files]
?   	github.com/travisjeffery/jocko/log	[no test files]
?   	github.com/travisjeffery/jocko/mock	[no test files]
=== RUN   TestAlterConfigsRequest
--- PASS: TestAlterConfigsRequest (0.00s)
=== RUN   TestAlterConfigsResponse
--- PASS: TestAlterConfigsResponse (0.00s)
=== RUN   TestCreateTopicRequests
--- PASS: TestCreateTopicRequests (0.00s)
=== RUN   TestCreateTopicResponse
--- PASS: TestCreateTopicResponse (0.00s)
=== RUN   TestDescribeConfigsRequest
--- PASS: TestDescribeConfigsRequest (0.00s)
=== RUN   TestDescribeConfigsResponse
--- PASS: TestDescribeConfigsResponse (0.00s)
=== RUN   TestFetchRequest
--- PASS: TestFetchRequest (0.00s)
=== RUN   TestFetchResponse
--- PASS: TestFetchResponse (0.00s)
=== RUN   TestFindCoordinatorRequest
--- PASS: TestFindCoordinatorRequest (0.00s)
=== RUN   TestFindCoordinatorResponse
--- PASS: TestFindCoordinatorResponse (0.00s)
=== RUN   TestHeartbeatRequest
--- PASS: TestHeartbeatRequest (0.00s)
=== RUN   TestLeaderAndISRRequest
--- PASS: TestLeaderAndISRRequest (0.00s)
=== RUN   TestLeaderAndISRResponse
--- PASS: TestLeaderAndISRResponse (0.00s)
PASS
ok  	github.com/travisjeffery/jocko/protocol	(cached)
?   	github.com/travisjeffery/jocko/testutil	[no test files]
Related tags
Distributed Systems go distributed-systems streaming kafka queue messaging
Micro is a platform for cloud native development
 Micro is a platform for cloud native development

Micro Overview Micro addresses the key requirements for building services in the cloud. It leverages the microservices architecture pattern and provid

Dec 31, 2022
High-Performance server for NATS, the cloud native messaging system.
 High-Performance server for NATS, the cloud native messaging system.

NATS is a simple, secure and performant communications system for digital systems, services and devices. NATS is part of the Cloud Native Computing Fo

Jan 8, 2023
CockroachDB - the open source, cloud-native distributed SQL database.
 CockroachDB - the open source, cloud-native distributed SQL database.

CockroachDB is a cloud-native distributed SQL database designed to build, scale, and manage modern, data-intensive applications. What is CockroachDB?

Dec 29, 2022
A distributed systems library for Kubernetes deployments built on top of spindle and Cloud Spanner.

hedge A library built on top of spindle and Cloud Spanner that provides rudimentary distributed computing facilities to Kubernetes deployments. Featur

Nov 9, 2022
A distributed locking library built on top of Cloud Spanner and TrueTime.

A distributed locking library built on top of Cloud Spanner and TrueTime.

Sep 13, 2022
A library built to provide support for defining service health for golang services. It allows you to register async health checks for your dependencies and the service itself, provides a health endpoint that exposes their status, and health metrics.
 A library built to provide support for defining service health for golang services. It allows you to register async health checks for your dependencies and the service itself, provides a health endpoint that exposes their status, and health metrics.

go-sundheit A library built to provide support for defining service health for golang services. It allows you to register async health checks for your

Dec 27, 2022
A distributed MySQL binlog storage system built on Raft
 A distributed MySQL binlog storage system built on Raft

What is kingbus? 中文 Kingbus is a distributed MySQL binlog store based on raft. Kingbus can act as a slave to the real master and as a master to the sl

Dec 31, 2022
The lightweight, distributed relational database built on SQLite
 The lightweight, distributed relational database built on SQLite

rqlite is a lightweight, distributed relational database, which uses SQLite as its storage engine. Forming a cluster is very straightforward, it grace

Jan 2, 2023
JuiceFS is a distributed POSIX file system built on top of Redis and S3.
 JuiceFS is a distributed POSIX file system built on top of Redis and S3.

JuiceFS is a high-performance POSIX file system released under GNU Affero General Public License v3.0. It is specially optimized for the cloud-native

Jan 4, 2023
Dapr is a portable, event-driven, runtime for building distributed applications across cloud and edge.
 Dapr is a portable, event-driven, runtime for building distributed applications across cloud and edge.

Dapr is a portable, serverless, event-driven runtime that makes it easy for developers to build resilient, stateless and stateful microservices that run on the cloud and edge and embraces the diversity of languages and developer frameworks.

Jan 5, 2023
SeaweedFS is a distributed storage system for blobs, objects, files, and data lake, to store and serve billions of files fast! Blob store has O(1) disk seek, local tiering, cloud tiering. Filer supports cross-cluster active-active replication, Kubernetes, POSIX, S3 API, encryption, Erasure Coding for warm storage, FUSE mount, Hadoop, WebDAV.
 SeaweedFS is a distributed storage system for blobs, objects, files, and data lake, to store and serve billions of files fast! Blob store has O(1) disk seek, local tiering, cloud tiering. Filer supports cross-cluster active-active replication, Kubernetes, POSIX, S3 API, encryption, Erasure Coding for warm storage, FUSE mount, Hadoop, WebDAV.

SeaweedFS Sponsor SeaweedFS via Patreon SeaweedFS is an independent Apache-licensed open source project with its ongoing development made possible ent

Jan 5, 2023
Golang client library for adding support for interacting and monitoring Celery workers, tasks and events.

Celeriac Golang client library for adding support for interacting and monitoring Celery workers and tasks. It provides functionality to place tasks on

Oct 28, 2022
Consistent hashing with bounded loads in Golang

consistent This library provides a consistent hashing function which simultaneously achieves both uniformity and consistency. For detailed information

Dec 29, 2022
Compute cluster (HPC) job submission library for Go (#golang) based on the open DRMAA standard.

go-drmaa This is a job submission library for Go (#golang) which is compatible to the DRMAA standard. The Go library is a wrapper around the DRMAA C l

Nov 17, 2022
Simple, fast and scalable golang rpc library for high load

gorpc Simple, fast and scalable golang RPC library for high load and microservices. Gorpc provides the following features useful for highly loaded pro

Dec 19, 2022
Hprose is a cross-language RPC. This project is Hprose for Golang.
 Hprose is a cross-language RPC. This project is Hprose for Golang.

Hprose 3.0 for Golang Introduction Hprose is a High Performance Remote Object Service Engine. It is a modern, lightweight, cross-language, cross-platf

Dec 26, 2022
Golang implementation of the Raft consensus protocol

raft raft is a Go library that manages a replicated log and can be used with an FSM to manage replicated state machines. It is a library for providing

Jan 9, 2023
The pure golang implementation of nanomsg (version 1, frozen)
 The pure golang implementation of nanomsg (version 1, frozen)

mangos NOTE: This is the legacy version of mangos (v1). Users are encouraged to use mangos v2 instead if possible. No further development is taking pl

Dec 7, 2022
A Golang implementation of the Umee network, a decentralized universal capital facility in the Cosmos ecosystem.

Umee A Golang implementation of the Umee network, a decentralized universal capital facility in the Cosmos ecosystem. Umee is a Universal Capital Faci

Jan 3, 2023