Hi,

I think go-kit/kit/log reflects my personal thoughts on logging, but I am hesitant to use it directly due to Logger.log() returning an error. In my opinion errors are intended to be checked, not ignored. And it's unreasonable to ask users to check log messages. I feel a better philosophy is for the formatters/writers that eventually output messages to disk/tcp/stdout/etc to allow registering handlers for error types. Then, when I setup logging I can say "Here is a logger that logs JSON to a file. If there is an error, call this function (maybe try stdout or maybe trigger some application "debug me" state).

Maybe something like

type jsonLogger struct {
  io.Writer
  onError func(err error, keyvals ...interface{})
}

I'd like to start thinking about a design for package pubsub/msgbus/...

Important use cases:

• An async transport for existing/typical RPC messaging
• A foundation for event sourcing
• A way to manage streaming
• Others?

Would like support for:

• Kafka
• RabbitMQ
• NATS
• NSQ
• AWS SQS
• GCP PubSub
• Weave Mesh
• Others?

Prior art:

• NYTimes/gizmo/pubsub
• micro/go-micro/broker
• Micro on NATS
• Others?

Open questions:

• What are the core abstractions and interfaces?
• Is streaming significantly different than event sourcing and transport use cases?
• What is the lowest common denominator for serialization?
• Others?

Inspired by #16 and the discussion there. @ChrisHines, this isn't very far along, but I'd appreciate your feedback to the general approach, which closely mirrors package metrics...

I have several use cases, now, where it would be beneficial if projects could use Go kit's package log without pulling in anything else. I would like to extract it to it's own module and package, tentatively named kitlog to avoid the name collision. And then I would like to point the existing package at it, maybe with type aliases if they are up for the task.

@ChrisHines — Do you think this is a reasonable idea, and is feasible? Can you identify any blockers?

Hello,

I noticed that here had been interest in AMQP transport similiar to NATS (#681) as well as talks of pubsub support (#295) so I thought I might like to share an implementation of a simple AMQP client & server transport.

Comments/Reviews would be appreciated.

As a foil to #357 and following up on #368, demonstrate an alternate approach to the "level" package offered via the "github.com/go-kit/kit/log/experimental_package" import path, allowing log event level filtering via instances of the (unexported) level.leveler interface, created by functions that specify the filtering threshold.

Features not accommodated here for now:

• Returning an error from log.Log when squelching an event
• Returning an error from log.Log for events that lack a level
  (This feature doesn't make much sense with this approach.)
• Squelching events that lack a level
  (This feature doesn't make much sense with this approach.)
• Documentation
  (We could adapt much of this from the sibling package.)

See the commit message for then benchstat comparison.

Hello, It will be great into example to ensure that http connection will be re-use by ensuring that Body has be completely purged. That avoid performance problem for whom that use this example as primary guide.

For example for https://github.com/go-kit/kit/blob/master/examples/stringsvc3/transport.go#L50, add something like this :

func decodeUppercaseResponse(_ context.Context, r *http.Response) (interface{}, error) {
    // This defer ensures to reuse the tcp connection via HTTP/1.1
    defer io.Copy(ioutil.Discard, r.Body)

    var response uppercaseResponse

Thx.

So far what I've got is simple generation based on a basic interface definition.

Not complete: different layout engines.

Specific design decision here was to construct the generated AST and then format it out, as opposed to using a text/template. There's some things that are made much easier this way (i.e. the output is more likely to be correct code), but the generation code is a bit more opaque. In working with this process I've found a few things that make the process clearer, and I'm looking to implement those before this is merged. Specifically, I think many of the repeated structures could be snagged from an example codefile and then processed into target structures, rather than built out of whole cloth or partial inline parses.

Filing an issue so we consider unified configuration of some kind.

Motivating example (though definitely not the only case): let's say a gokit process does want to do Dapper/Zipkin style tracing. The process will need to know where to forward the tracing information to, which won't be known at compile time.

It would be nice if all of the gokit packages were configured in the same or similar way, so that as they are used modularly, configuration of subpackages happens automatically, with little programmer assistance.

Twirp (https://github.com/twitchtv/twirp) is an RPC system that works over HTTP/1.1 using either JSON or Protobuf. It's leverages protobuf for generating servers and clients.

I still need to do a few things, but wanted to get initial feedback.

Todo

• [ ] Add tests (planning to mirror what GRPC has)
• [ ] Add Twirp support to addsvc example
• [ ] Add documentation

Edit: Due to lots of feature churn in the last few weeks/months, I've been convinced that it makes more sense to hold off on a 1.0.0 until things settle. Notes in comments below.

—

Hello! I want to cut a v1.0.0 release in the near future. This issue is created per the request of @ChrisHines to serve as a notice to anyone who may be interested, for example to raise questions, concerns, or propose specific additions.

The only thing outstanding that I'd like to get in is a promotion of log/experimental_level to log/level. I've asked @seh to weigh in on that, as he's previously commented on the package. Otherwise, let me know your thoughts. I'll leave this open until at least the weekend, so another 5 days.

edit: outstanding issues I'd want to resolve before a 1.0.0

• [x] #476 — Merge log/experimental_level to level; move existing levels to deprecated_levels
• [x] #420 / #421 — use stdlib context where possible
• [x] #432 / #437 — log: StdlibAdapter "file" key to "caller"
• [x] Optional: #434 — log: normalize StdlibAdapter timestamp format
• [x] Optional: #431 — transport: implement http.EncodeJSONRequest
• [x] Evaluate breaking change: #474 — Replace log.Context with log.With/WithPrefix?

If anyone feels strongly about either of those two Optionals, (cc: @willfaught) please get your PRs in within the next few days...

What would you like?

Overview

Since release v1.11.0, nats.go supports message headers. headers are very useful to transfer metadata together with messages.

I'm proposing to add basic support for headers in the default response encoder.

Proposed API

The idea is to do something similar it's done for transport/http with the Headerer interface. API change proposal is backward compatible (no public API change)

type Headerer interface {
    Headers() Header
}

func EncodeJSONResponse(_ context.Context, reply string, nc *nats.Conn, response interface{}) error {
        msg := nats.NewMessage(reply)
        if nc. HeadersSupported() {
                if headerer, ok := response.(Headerer); ok {
                         msg.Header = headerer.Headers()  // I wrote that to make it shorter
                }
        }

	msg.Data, err := json.Marshal(response)
	if err != nil {
		return err
	}

	return nc.PublishMsg(msg)
}

If the proposal is accepted I can raise a PR

Description

This PR fixes an issue that occurred when the Dogstatsd client could not reach the Datadog agent to send metrics. It would fail to send counters, return and never reset timings and histograms, resulting in an ever-increasing memory consumption while the client cannot reach the Datadog agent.

What did you do?

Using the GoKit implementation of Instancer for Consul in GoKit 0.11 and up results in Instancer spamming Consul with hundreds of RPC requests/sec on any register/deregister event that occurs after the Instancer is created. This resulted in our Consul agent pods going to 100% CPU and overwhelming our Consul cluster.

Steps to re-create:

1. Start Consul

2. Register a dummy service

curl --location --request PUT 'http://localhost:8500/v1/agent/service/register' \ --header 'Content-Type: application/json' \ --data-raw '{ "name": "db-example", "id": "db-example", "address": "127.0.0.1", "port": 5432, "tags": [ "database" ] }'

3. Open a terminal and run the following command to get insights into what the Consul agent is doing

consul monitor -log-level=trace

4. Create Instancer for db-example service (see sample code below) and run app/code

5. Deregister the service

curl --location --request PUT 'http://localhost:8500/v1/agent/service/deregister/db-example'

Observe the behavior from consul monitor, the agent is getting slammed with requests and it never stops until the Instancer is stopped or application exits.

Simple example application.

package main

import (
	"fmt"
	"github.com/go-kit/kit/log"
	"github.com/go-kit/kit/sd"
	"github.com/hashicorp/consul/api"
	"time"

	"github.com/go-kit/kit/sd/consul"
)

func main() {

	apiClient, err := api.NewClient(api.DefaultConfig())
	if err != nil {
		panic(err)
	}
	client := consul.NewClient(apiClient)

	instancer := consul.NewInstancer(client, log.NewNopLogger(), "db-example", []string{}, true)

	events := make(chan sd.Event, 1)
	go func() {
		instancer.Register(events)
	}()

	go func() {
		for event := range events {
			fmt.Println(event.Instances)
		}
	}()

	time.Sleep(1 * time.Hour)
}

What did you expect?

Consul agent should not be getting spammed with hundreds of requests a second.

What happened instead?

Instancer spammed Consul with way too many requests and drove the CPU utilization to 100%.

7.0.0.1:53702 latency=9.375µs 2022-02-12T18:06:20.710-0500 [DEBUG] agent.http: Request finished: method=GET url=/v1/health/service/db-example?index=155&passing=1 from=127.0.0.1:53702 latency=12.208µs 2022-02-12T18:06:20.710-0500 [DEBUG] agent.http: Request finished: method=GET url=/v1/health/service/db-example?index=155&passing=1 from=127.0.0.1:53702 latency=162.792µs 2022-02-12T18:06:20.710-0500 [DEBUG] agent.http: Request finished: method=GET url=/v1/health/service/db-example?index=155&passing=1 from=127.0.0.1:53702 latency=54.791µs

What would you like?

I have noticed a couple of differences between the tutorial section of the stringsvc3 example and its full source code. It is worth mentioning the differences are about different imports, or different number of parameters of a particular function. I was wondering which one is the most up to date one?