Modern CLI for Apache Kafka, written in Go.

Kaf

Kafka CLI inspired by kubectl & docker

Actions Status GoReportCard GoDoc

asciicinema

Install

Install from source:

go get -u github.com/birdayz/kaf/cmd/kaf

Install binary:

curl https://raw.githubusercontent.com/birdayz/kaf/master/godownloader.sh | BINDIR=$HOME/bin bash

Install on Archlinux via AUR:

yay -S kaf

Usage

Add a local Kafka with no auth

kaf config add-cluster local -b localhost:9092

Select cluster from dropdown list

kaf config select-cluster

Describe and List nodes

kaf node ls

List topics, partitions and replicas

kaf topics

Describe a given topic called mqtt.messages.incoming

kaf topic describe mqtt.messages.incoming

List consumer groups

kaf groups

Describe a given consumer group called dispatcher

kafa group describe dispatcher

Write message into given topic from stdin

echo test | kaf produce mqtt.messages.incoming

Configuration

See the examples folder

Shell autocompletion

Source the completion script in your shell commands file:

Bash Linux:

kaf completion bash > /etc/bash_completion.d/kaf

Bash MacOS:

kaf completion bash > /usr/local/etc/bash_completion.d/kaf

Zsh

kaf completion zsh > "${fpath[1]}/_kaf"

Fish

kaf completion fish > ~/.config/fish/completions/kaf.fish

Powershell

Invoke-Expression (@(kaf completion powershell) -replace " ''\)$"," ' ')" -join "`n")

Owner
Johannes Brüderl
Software Engineer
Johannes Brüderl
Comments
  • strimzi SSL

    strimzi SSL

    Default Strimzi operator installs with TLS enabled with a crt certificate but no user or password at all. I could not make Kaf to work with just the TLS certificate. I could not find anything in the documentation or examples folder explaining this connectivity mode.

    So far, I am configuring my cluster like:

    - name: strimzi
      brokers:
      - my-cluster-kafka-bootstrap-kafka-strimzi.apps.openshift.mydomain.com:443
      SASL: null
      TLS:
        cafile: /Users/user/ca.crt
        clientfile: ""
        clientkeyfile: ""
        insecure: true
      security-protocol: SASL_SSL
      schema-registry-url: ""
    

    The output for that configuration is:

    $ kaf topics               
    panic: runtime error: invalid memory address or nil pointer dereference
    [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x1641e92]
    
    goroutine 1 [running]:
    main.getConfig(0xc000161260)
    	/home/j0e/projects/kaf/cmd/kaf/kaf.go:88 +0x172
    main.getClusterAdmin(0xc0002540e8, 0xc000161260)
    	/home/j0e/projects/kaf/cmd/kaf/kaf.go:177 +0x26
    main.glob..func16(0x1e183e0, 0x1e3c468, 0x0, 0x0)
    	/home/j0e/projects/kaf/cmd/kaf/topic.go:53 +0x37
    github.com/spf13/cobra.(*Command).execute(0x1e183e0, 0x1e3c468, 0x0, 0x0, 0x1e183e0, 0x1e3c468)
    	/home/j0e/gopath/pkg/mod/github.com/spf13/[email protected]/command.go:830 +0x2aa
    github.com/spf13/cobra.(*Command).ExecuteC(0x1e165e0, 0x17d2f3e, 0xc000207f50, 0x100543f)
    	/home/j0e/gopath/pkg/mod/github.com/spf13/[email protected]/command.go:914 +0x2fb
    github.com/spf13/cobra.(*Command).Execute(...)
    	/home/j0e/gopath/pkg/mod/github.com/spf13/[email protected]/command.go:864
    main.main()
    	/home/j0e/projects/kaf/cmd/kaf/kaf.go:105 +0x31
    

    If I change the secutiry-protocol to SSL I get the following:

    $ kaf topics      
    Unable to read Clientfile :open : no such file or directory
    

    Thanks

  • cmd/consume: extend offsets flag and add count

    cmd/consume: extend offsets flag and add count

    This extends the consume command with flags for setting specific offsets, or offsets relative to newest/oldest, setting a count of messages to read, and setting a flag to exit when the end of the partition is reached.

    The changes are roughly based on https://github.com/birdayz/kaf/issues/39, except it does support setting different offsets and counts on different partitions - you just need to make sure your lists of partitions, offsets, and counts are the same length.

    This incidentally fixes a race condition on --follow which was due to the offset variable being shared between goroutines. It mostly showed up for me when partitions had widely different offsets. I can make a PR just fixing that if you don't want to accept this entire change right away.

  • Adding support to produce avro encoded messages

    Adding support to produce avro encoded messages

    As it turns out, most of the heavy lifting was already done by the schemaCache implementation. If you think this approach is fine I would like to add support for avro keys as well.

    Fixes #181

  • Only allow group commit on empty consumergroups

    Only allow group commit on empty consumergroups

    Adds a simple check to the kaf group commit cmd that exits if the consumer group is active. In local testing it was possible for me to set the offset for an active consumer 😬 , thus the change.

    Also added a small example in the readme so its more explicit

    I've been having problems running the tests locally so no tests atm, if anyone has any advice, please share! 🙇 Would be happy to add some test coverage in a follow up MR

  • Add integration tests for some of the commands

    Add integration tests for some of the commands

    This commit adds integration tests for a part of available commands. This is achieved by using Gnomock Kafka preset which spins up a temporary Kafka container, and executing CLI commands against it.

    To support integration tests, I had to make a couple of changes to non-test code:

    1. I added context to a couple of places to support cobra's ExecuteContext function. The reason behind it was that consume command does not exit automatically, and there needed to be a way to cancel it after some time. For that I used context.WithTimeout, and pushed this context into consumer code. Now consumer code either passes context forward, or listens to cancellation event itself.
    2. I changed all of the commands to use configurable stdin, stdout and stderr streams. Cobra supports using custom input/output objects, but all the actual commands used os.Stdout directly. I implemented a PersistentPreRun function on rootCmd that decides which reader/writer to use: by default we always use os descriptors.
  • Possibility to produce avro encoded message to schema regustry powered topic

    Possibility to produce avro encoded message to schema regustry powered topic

    Hello

    Some tools allow to produce avro encoded messges to topic using schema from schema registry using JSON file as input message.

    For example this https://github.com/tchiotludo/akhq

  • Add bash/zsh autocompletion

    Add bash/zsh autocompletion

    Issue https://github.com/birdayz/kaf/issues/14

    I've added bash autocompletion and fixed some minor issues.

    note that it can be slow to get the list of topics. maybe add caching?

    Not sure if this is a good idea. How would you manage such cache? What user should do to update his list of topics?

  • Consumption Improvements

    Consumption Improvements

    • I was getting "broker not connected" errors frequently when starting up that end up being a race between requesting partition information and having the broker's connection established. This implements a simply retry+backoff so that it can hopefully succeed.
    • I was also having issues with the output being interleaved on account of the multiple goroutines writing to same writer without synchronization (in some places). I fixed that with a single channel which handles all the writing. This is mentioned in #23
    • I wonder if we can use sarama's newer ConsumerGroup interface which might simplify this -- I have to dig a bit more into that one...

    Thoughts welcome... :-)

  • Add version flag to show the current version of the tool

    Add version flag to show the current version of the tool

    Hi @birdayz

    I finally decided to try a first contribution for this project that has been so useful in the last year of my work. I'm no go expert but I hope this is a good contribution and a good start.

    As requested in #158 this PR:

    Output:

    $> go build -v -ldflags "-s -w -X main.version=v2.0 -X main.commit=abc" ./cmd/kaf/
    $> ./kaf --version
    kaf version v2.0 (abc)
    
  • add TLS config

    add TLS config

    clusters:
    - name: test
      brokers:
      - localhost:9092
      SASL:
        mechanism: PLAIN
        username: admin
        password: mypasswordisnotsosimple
      TLS:
        cafile: /path/ca.pem
        verify: true
      security-protocol: SASL_SSL
    
  • Add ability to reset consumer group offsets

    Add ability to reset consumer group offsets

    Hey there! First of all, thanks for the great project, the kubectl syntax is a stroke of genius, makes it really friendly to use and stand out compared to the other tools out there.

    I was wondering if there was any demand for adding the ability to reset consumer group offsets? Some other admin-related tools offer this ability, notably kowl and I think it would provide a lot of value to the project.

    I'm happy to own this as I require this functionality, just want to run it past y'all before commencing work :)

  • Bump github.com/spf13/cobra from 1.5.0 to 1.6.1

    Bump github.com/spf13/cobra from 1.5.0 to 1.6.1

    Bumps github.com/spf13/cobra from 1.5.0 to 1.6.1.

    Release notes

    Sourced from github.com/spf13/cobra's releases.

    v1.6.1

    Bug fixes 🐛

    • Fixes a panic when AddGroup isn't called before AddCommand(my-sub-command) is executed. This can happen within more complex cobra file structures that have many different inits to be executed. Now, the check for groups has been moved to ExecuteC and provides more flexibility when working with grouped commands - @​marckhouzam (and shout out to @​aawsome, @​andig and @​KINGSABRI for a deep investigation into this! 👏🏼)

    v1.6.0

    Summer 2022 Release

    Some exciting changes make their way to Cobra! Command completions continue to get better and better (including adding --help and --version automatic flags to the completions list). Grouping is now possible in your help output as well! And you can now use the OnFinalize method to cleanup things when all "work" is done. Checkout the full changelog below:


    Features 🌠

    Deprecation 👎🏼

    • ExactValidArgs is deprecated (but not being removed entirely). This is abit nuanced, so checkout #1643 for further information and the updated user_guide.md on how this may affect you (and how you can take advantage of the correct behavior in the validators): @​umarcor #1643

    Bug fixes 🐛

    Dependencies 🗳️

    Testing 🤔

    Docs ✏️

    Misc 💭

    ... (truncated)

    Commits

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
  • [Bug] Create topic - show in-correct logs

    [Bug] Create topic - show in-correct logs

    • Version: v0.2.3
    • OS: Mac

    Currently

    $ kaf topic create my_topic
    
    ✅ Created topic!
          Topic Name:            my_topic
          Partitions:            -1
          Replication Factor:    1
          Cleanup Policy:        delete
    

    Expected

    $ kaf topic create my_topic
                                                                               
    ✅ Created topic!
          Topic Name:            my_topic
          Partitions:            1
          Replication Factor:    1
          Cleanup Policy:        delete
    

    -1 -> 1

    Topics list

    $ kaf topics
    
    NAME             PARTITIONS   REPLICAS   
    my_topic         1            1          
    
  • Bump github.com/Shopify/sarama from 1.36.0 to 1.37.2

    Bump github.com/Shopify/sarama from 1.36.0 to 1.37.2

    Bumps github.com/Shopify/sarama from 1.36.0 to 1.37.2.

    Release notes

    Sourced from github.com/Shopify/sarama's releases.

    Version 1.37.2 (2022-10-04)

    What's Changed

    :bug: Fixes

    :heavy_plus_sign: Other Changes

    Full Changelog: https://github.com/Shopify/sarama/compare/v1.37.1...v1.37.2

    Version 1.37.1 (2022-10-04)

    What's Changed

    :bug: Fixes

    New Contributors

    Full Changelog: https://github.com/Shopify/sarama/compare/v1.37.0...v1.37.1

    Version 1.37.0 (2022-09-28)

    What's Changed

    :rotating_light: Breaking Changes

    • Due to a change in github.com/klauspost/compress v1.15.10, Sarama v1.37.0 requires Go 1.17 going forward, unfortunately due to an oversight this wasn't reflected in the go.mod declaration at time of release.

    :tada: New Features / Improvements

    :bug: Fixes

    :package: Dependency updates

    ... (truncated)

    Commits
    • 610514e fix: bump go.mod specification to go 1.17 (#2357)
    • fb9a931 fix: ensure updateMetaDataMs is 64-bit aligned (#2356)
    • 0162486 fix(producer): replace time.After with time.Timer to avoid high memory usage ...
    • 50ceb93 fix(test): consumer group rebalance strategy compatibility (#2353)
    • eaf1def fix: support existing deprecated Rebalance.Strategy field usage (#2352)
    • 619d4ef Merge pull request #2327 from joewreschnig/mock-config-validate
    • 9a14762 Validate the Config when creating a mock producer/consumer
    • b923960 Merge pull request #2350 from Shopify/deps/github.com-pierrec-lz4-v4-4.1.x
    • fcc0267 Merge pull request #2349 from Shopify/deps/github.com-klauspost-compress-1.15.x
    • c9db08b Merge pull request #2336 from Shopify/deps/github.com-shopify-toxiproxy-v2-2.x
    • Additional commits viewable in compare view

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
  • How to use protobuf, an example will be appreciated

    How to use protobuf, an example will be appreciated

    We are trying to call with following command, but it seems the result is the same as '--raw'.

    kaf consume example.topic.name --proto-include /home/somebody/ --proto-type com.example.model.ClassName

  • Cluster config flags without the config file

    Cluster config flags without the config file

    Hi there,

    is it possible to use the commands without the config file for a cluster?

    I would like to dynamically use kaf in our CI for some housecleaning work and would like to avoid creating the config file manually.

    e.g. use it in a similar fashion to this:

    kaf groups --broker <broker-here> --username <username-here> --password <password-here> ... other needed configs follow
    
Related tags
franz-go - A complete Apache Kafka client written in Go

franz-go contains a feature complete, pure Go library for interacting with Kafka from 0.8.0 through 2.8.0+. Producing, consuming, transacting, administrating, etc.

Dec 29, 2022
Study project that uses Apache Kafka as syncing mechanism between two databases, with producers and consumers written in Go.

Kafka DB Sync Study project that uses Apache Kafka as syncing mechanisms between a monolith DB and a microservice. The main purpose of this project is

Dec 5, 2021
A CLI tool for interacting with Kafka through the Confluent Kafka Rest Proxy

kafkactl Table of contents kafkactl Table of contents Overview Build Development Overview kafkactl is a CLI tool to interact with Kafka through the Co

Nov 1, 2021
Confluent's Apache Kafka Golang client

Confluent's Golang Client for Apache KafkaTM confluent-kafka-go is Confluent's Golang client for Apache Kafka and the Confluent Platform. Features: Hi

Dec 30, 2022
Sarama is a Go library for Apache Kafka 0.8, and up.

sarama Sarama is an MIT-licensed Go client library for Apache Kafka version 0.8 (and later). Getting started API documentation and examples are availa

Jan 1, 2023
Apache Kafka Web UI for exploring messages, consumers, configurations and more with a focus on a good UI & UX.
Apache Kafka Web UI for exploring messages, consumers, configurations and more with a focus on a good UI & UX.

Kowl - Apache Kafka Web UI Kowl (previously known as Kafka Owl) is a web application that helps you to explore messages in your Apache Kafka cluster a

Jan 3, 2023
Cluster extensions for Sarama, the Go client library for Apache Kafka 0.9

Cluster extensions for Sarama, the Go client library for Apache Kafka 0.9 (and later).

Dec 28, 2022
A quick introduction to how Apache Kafka works and differs from other messaging systems using an example application.
A quick introduction to how Apache Kafka works and differs from other messaging systems using an example application.

Apache Kafka in 6 minutes A quick introduction to how Apache Kafka works and differs from other messaging systems using an example application. In thi

Oct 27, 2021
Testing Apache Kafka using Go.

Apache Kafka Go Testing Apache Kafka using Go. Instructions Provision the single node Kafka cluster using Docker: docker-compose -p apache-kafka-go up

Dec 17, 2021
provider-kafka is a Crossplane Provider that is used to manage Kafka resources.

provider-kafka provider-kafka is a Crossplane Provider that is used to manage Kafka resources. Usage Create a provider secret containing a json like t

Oct 29, 2022
Apache Pulsar Go Client Library

Apache Pulsar Go Client Library A Go client library for the Apache Pulsar project. Goal This projects is developing a pure-Go client library for Pulsa

Jan 4, 2023
Implementation of the NELI leader election protocol for Go and Kafka
Implementation of the NELI leader election protocol for Go and Kafka

goNELI Implementation of the NELI leader election protocol for Go and Kafka. goNELI encapsulates the 'fast' variation of the protocol, running in excl

Dec 8, 2022
ChanBroker, a Broker for goroutine, is simliar to kafka

Introduction chanbroker, a Broker for goroutine, is simliar to kafka In chanbroker has three types of goroutine: Producer Consumer(Subscriber) Broker

Aug 12, 2021
franz-go contains a high performance, pure Go library for interacting with Kafka from 0.8.0 through 2.7.0+. Producing, consuming, transacting, administrating, etc.

franz-go - Apache Kafka client written in Go Franz-go is an all-encompassing Apache Kafka client fully written Go. This library aims to provide every

Dec 29, 2022
Easy to use distributed event bus similar to Kafka
Easy to use distributed event bus similar to Kafka

chukcha Easy to use distributed event bus similar to Kafka. The event bus is designed to be used as a persistent intermediate storage buffer for any k

Dec 30, 2022
Kafka implemented in Golang with built-in coordination (No ZooKeeper, single binary install, Cloud Native)

Jocko Distributed commit log service in Go that is wire compatible with Kafka. Created by @travisjeffery, continued by nash. Goals: Protocol compatibl

Aug 9, 2021
kafka watcher for casbin library

Casbin Kafka Watcher Casbin watcher for kafka This watcher library will enable users to dynamically change casbin policies through kakfa messages Infl

May 8, 2021
Go gRPC Kafka CQRS microservices with tracing

Golang CQRS Kafka gRPC Postgresql MongoDB Redis microservices example ?? ??‍?? Full list what has been used: Kafka as messages broker gRPC Go implemen

Jan 1, 2023
pubsub controller using kafka and base on sarama. Easy controll flow for actions streamming, event driven.

Psub helper for create system using kafka to streaming and events driven base. Install go get github.com/teng231/psub have 3 env variables for config

Sep 26, 2022