Go Micro is a framework for distributed systems development

Last update: Jan 8, 2023

Comments: 16

Go Micro

Go Micro is a framework for distributed systems development.

Overview

Go Micro provides the core requirements for distributed systems development including RPC and Event driven communication. The Micro philosophy is sane defaults with a pluggable architecture. We provide defaults to get you started quickly but everything can be easily swapped out.

Features

Go Micro abstracts away the details of distributed systems. Here are the main features.

Authentication - Auth is built in as a first class citizen. Authentication and authorization enable secure zero trust networking by providing every service an identity and certificates. This additionally includes rule based access control.
Dynamic Config - Load and hot reload dynamic config from anywhere. The config interface provides a way to load application level config from any source such as env vars, file, etcd. You can merge the sources and even define fallbacks.
Data Storage - A simple data store interface to read, write and delete records. It includes support for memory, file and CockroachDB by default. State and persistence becomes a core requirement beyond prototyping and Micro looks to build that into the framework.
Service Discovery - Automatic service registration and name resolution. Service discovery is at the core of micro service development. When service A needs to speak to service B it needs the location of that service. The default discovery mechanism is multicast DNS (mdns), a zeroconf system.
Load Balancing - Client side load balancing built on service discovery. Once we have the addresses of any number of instances of a service we now need a way to decide which node to route to. We use random hashed load balancing to provide even distribution across the services and retry a different node if there's a problem.
Message Encoding - Dynamic message encoding based on content-type. The client and server will use codecs along with content-type to seamlessly encode and decode Go types for you. Any variety of messages could be encoded and sent from different clients. The client and server handle this by default. This includes protobuf and json by default.
RPC Client/Server - RPC based request/response with support for bidirectional streaming. We provide an abstraction for synchronous communication. A request made to a service will be automatically resolved, load balanced, dialled and streamed.
Async Messaging - PubSub is built in as a first class citizen for asynchronous communication and event driven architectures. Event notifications are a core pattern in micro service development. The default messaging system is a HTTP event message broker.
Synchronization - Distributed systems are often built in an eventually consistent manner. Support for distributed locking and leadership are built in as a Sync interface. When using an eventually consistent database or scheduling use the Sync interface.
Pluggable Interfaces - Go Micro makes use of Go interfaces for each distributed system abstraction. Because of this these interfaces are pluggable and allows Go Micro to be runtime agnostic. You can plugin any underlying technology.

Getting Started

To make use of Go Micro

import "github.com/asim/go-micro/v3"

// create a new service
service := micro.NewService(
    micro.Name("helloworld"),
)

// initialise flags
service.Init()

// start the service
service.Run()

See the docs for detailed information on the architecture, installation and use of go-micro.

Code Generation

See cmd/protoc-gen-micro for protobuf code generation.

Example Usage

See examples directory for usage examples.

Plugins

See plugins directory for all the plugins.

License

Go Micro is Apache 2.0 licensed.

Upgrade

Micro v3 consolidates Go Micro into it as a single unified platform. Please see the upgrade guide.

Owner

Asim Aslam

Working on Micro

https://github.com/micro/go-micro https://go-micro.dev

Comments

example of compose two or more services with single binary

I like go-micro design, but if i want to provide single binary with all my services together. For example that implements some api with 10 different services. How can i do that with go-micro? All this services need to communicate via message bus (nats or etcd).
RPC stream client/server mutex fix

As described in #878 (and #532) it is not possible to send messages while waiting to receive them as well (or vice-versa), due to RPC stream client and server being locked by both Send() and Recv() functions.

In order to solve this issue, I unlock the Mutex before calling the codec to receive data, and locking it again after codec returned. This way we still have the Mutex protection of the client/server structs, but we can send/receive at same time.

:warning: Note that this assumes that codec does not need protection while being active.

I did not update Send() because I assumed that it won't block. By only fixing Recv() the Send() worked as welll. Maybe depending on transport/codec it can block as well?

I only considered these 2 components/files, as we are (currently) not having issues in other parts of the go-micro framework regarding to this. If needed I can update other parts as well to stay in sync with this change.

I have tested this in our micro-service environment and it all seems to work fine. I did not trigger race conditions or other panics. But I did not yet perform (high) load or long running tests, which would make race conditions more likely to pop up (if there are any).
build command-line-arguments: cannot load crypto/ed25519: cannot find module providing package crypto/ed2551

When I run the go-micro,it tips build command-line-arguments: cannot load crypto/ed25519: cannot find module providing package crypto/ed25519,but i find the module is exists in gopath.
consul registry broken: mismatch in type due to import being expected from "github.com/hashicorp/consul/vendor"
My Go version is 1.12.5.

I'm building my package binary using the golang1.12.5-alpine3.9 Docker image as a package builder and I'm running the following commands in my Dockerfile to install all the necessary dependencies -

# install source dependencies RUN go get -d -v ./... RUN go install -v ./...

the go get command works as expected but the go install command crashes with an exit code: 2 and the following message -

# github.com/micro/go-micro/registry/consul /go/src/github.com/micro/go-micro/registry/consul/watcher.go:48:33: cannot use cr.Client (type *"github.com/hashicorp/consul/api".Client) as type *"github.com/hashicorp/consul/vendor/github.com/hashicorp/consul/api".Client in argument to wp.RunWithClientAndLogger /go/src/github.com/micro/go-micro/registry/consul/watcher.go:213:37: cannot use cw.r.Client (type *"github.com/hashicorp/consul/api".Client) as type *"github.com/hashicorp/consul/vendor/github.com/hashicorp/consul/api".Client in argument to wp.RunWithClientAndLogger

Just FYI, I was able to generate the Docker based package builder successfully up till 3 days ago.
gossip registry

I'm experiment with gossip registry and may be found some specific issue. When i'm start first service1 with specify gossip.Address("172.16.1.254:4223") and on other server service2 with registry.Addrs("172.16.1.254:4223") and gossip.Address("172.16.1.1:0")

i see that members equal 2 on both sides, but the second service does not registered in registry. But if i stop service1 and start it with registry.Addrs("172.16.1.254:4223") and gossip.Address("172.16.1.1:xxx") where xxx is port provided by service2 all works fine. So issue appears only on first service1 when it start first without other members.

[BUG] Micro API fails on streaming endpoint unexpected nil error

Describe the bug

What are you trying to do? Run micro api gateway and call a streaming endpoint
What did you expect to happen? Success, actual response
What happens instead?

{
    "code": 500,
    "detail": "grpc: client streaming protocol violation: get <nil>, want <EOF>",
    "id": "go.micro.client",
    "status": "Internal Server Error"
}

How to reproduce the bug: micro version 2.0.0 micro api --address :8085 --namespace some.ns --handler rpc http localhost:8085/my/streaming/endpoint

If possible, please include a minimal code snippet here.

       ctx, cancel := context.WithCancel(ctx)
	ch, errCh := data.GetSomeData(ctx, cancel, s.db, s.service)

	for {
		select {
		case <-ctx.Done():
			stream.Close()
			return nil
		case err := <-errCh:
			return err
		case msg := <-ch:
			if msg == nil {
				return nil
			}
			err := stream.Send(msg)
			if err != nil {
				cancel()
				for len(ch) > 0 {
					<-ch
				}
				return errors.InternalServerError(s.service.Name(), err.Error())
			}
		}
	}

Environment: Go Version: please paste go version output here

go version go1.13.1 darwin/amd64

Disable timeout on RPC stream connection
For a stream RPC connection I'd like to get rid of timeout, I see/found two solutions:

Set timeout to some huge value and hope the client never stays connected long enough.

Set timeout to -1 which 'somehow' seems to work. (Actually strconv.ParseUint() chokes on it and context timeout is not set (and no error is issued...))

Both options feel hacky to me... Is there a better/recommended solution instead?

go get -u fails, apparently due to coreos/etcd move

Go version: go1.13 Problem: when I run go get -u github.com/micro/go-micro I get the following error:

# github.com/coreos/etcd/clientv3
src/github.com/coreos/etcd/clientv3/auth.go:121:72: cannot use auth.callOpts (type []"github.com/coreos/etcd/vendor/google.golang.org/grpc".CallOption) as type []"go.etcd.io/etcd/vendor/google.golang.org/grpc".CallOption in argument to auth.remote.AuthEnable
src/github.com/coreos/etcd/clientv3/auth.go:126:74: cannot use auth.callOpts (type []"github.com/coreos/etcd/vendor/google.golang.org/grpc".CallOption) as type []"go.etcd.io/etcd/vendor/google.golang.org/grpc".CallOption in argument to auth.remote.AuthDisable
src/github.com/coreos/etcd/clientv3/auth.go:131:152: cannot use auth.callOpts (type []"github.com/coreos/etcd/vendor/google.golang.org/grpc".CallOption) as type []"go.etcd.io/etcd/vendor/google.golang.org/grpc".CallOption in argument to auth.remote.UserAdd
src/github.com/coreos/etcd/clientv3/auth.go:136:144: cannot use auth.callOpts (type []"github.com/coreos/etcd/vendor/google.golang.org/grpc".CallOption) as type []"go.etcd.io/etcd/vendor/google.golang.org/grpc".CallOption in argument to auth.remote.UserAdd
src/github.com/coreos/etcd/clientv3/auth.go:141:86: cannot use auth.callOpts (type []"github.com/coreos/etcd/vendor/google.golang.org/grpc".CallOption) as type []"go.etcd.io/etcd/vendor/google.golang.org/grpc".CallOption in argument to auth.remote.UserDelete
src/github.com/coreos/etcd/clientv3/auth.go:146:122: cannot use auth.callOpts (type []"github.com/coreos/etcd/vendor/google.golang.org/grpc".CallOption) as type []"go.etcd.io/etcd/vendor/google.golang.org/grpc".CallOption in argument to auth.remote.UserChangePassword
src/github.com/coreos/etcd/clientv3/auth.go:151:104: cannot use auth.callOpts (type []"github.com/coreos/etcd/vendor/google.golang.org/grpc".CallOption) as type []"go.etcd.io/etcd/vendor/google.golang.org/grpc".CallOption in argument to auth.remote.UserGrantRole
src/github.com/coreos/etcd/clientv3/auth.go:156:80: cannot use auth.callOpts (type []"github.com/coreos/etcd/vendor/google.golang.org/grpc".CallOption) as type []"go.etcd.io/etcd/vendor/google.golang.org/grpc".CallOption in argument to auth.remote.UserGet
src/github.com/coreos/etcd/clientv3/auth.go:161:72: cannot use auth.callOpts (type []"github.com/coreos/etcd/vendor/google.golang.org/grpc".CallOption) as type []"go.etcd.io/etcd/vendor/google.golang.org/grpc".CallOption in argument to auth.remote.UserList
src/github.com/coreos/etcd/clientv3/auth.go:166:106: cannot use auth.callOpts (type []"github.com/coreos/etcd/vendor/google.golang.org/grpc".CallOption) as type []"go.etcd.io/etcd/vendor/google.golang.org/grpc".CallOption in argument to auth.remote.UserRevokeRole
src/github.com/coreos/etcd/clientv3/auth.go:166:106: too many errors

We first saw this error when our CI build began to fail, and at first couldn't reproduce it locally, but then were able to by adding the -u to the command. Alternately, here's a simple Dockerfile that demonstrates the issue when building it.

This seems to be caused by the moving of github.com/coreos/etcd to github.com/etcd-io/etcd, as described here: https://github.com/etcd-io/etcd/pull/10044#issuecomment-417125341

That change and the issue are from last year, so I'm not sure why this is just showing up now, but my colleagues and I (and our CI) can all reproduce the problem now.

I will look into creating a PR that changes the imports from coreos/etcd to etcd-io/etcd.

Adding TLS to RPC

I'm feeling a little bit dumb, I don't suppose you have an example of adding TLS to a RPC service?

	security.Setup(log, security.Vault())
	tlsConfig := &tls.Config{
		GetCertificate: security.Vault().CertificateGetter(),
	}

	// initialise service
	service.Init(
		micro.Action(func(c *cli.Context) {
			if c.Bool("debug") {
				log.Level = logrus.DebugLevel
				log.Debug("Debug log level enabled")
			}
		}),
// transport.TLSConfig(tlsConfig),
	)

	countryHandler, err := handler.NewCountryHandler()
	if err != nil {
		log.Fatal(err)
	}

	country.RegisterCountriesHandler(service.Server(), countryHandler)

	if err := service.Run(); err != nil {
		log.Fatal(err)
	}

I tried adding transport options to the service.Init but it's clearly not that simple?

How to add value to metadata?

Hello,

I want to add some value to metadata from grpc method, then process in wrapper, however I found since grpc 1.40, we can't modify the context, the proposed solution is to use metadata.NewIncomingContext.

https://github.com/grpc/grpc-go/issues/4363

I don't know how to do it in micro?

func (s *ServerImpl) Location(ctx context.Context, req *pb.LocationRequest, rsp *pb.City) error {
// add value to context
	md, _ := metadata.FromContext(ctx)
	if md != nil {
		md.Set("tag", "value")
	}
}

func LogWrapper(fn server.HandlerFunc) server.HandlerFunc {
	return func(ctx context.Context, req server.Request, rsp interface{}) error {
 		err := fn(ctx, req, rsp)
 		value, _ := md.Get("tag")  // value is empty
}
}

Documentation unavailable : not displayed due to license restrictions

Hello,

This is not a #1956 or #1953 duplicate. :)

I tried to access : https://pkg.go.dev/github.com/micro/go-micro/v3?tab=overview But the documentation is unavailable due to license restrictions.

Thanks,
[FEATURE] Adding Sentry reporter

Is your feature request related to a problem? Please describe. Hi Guys, i'm really need adding sentry for error reporter but i cannot get recover error data because recover already called when program had error on this file https://github.com/go-micro/plugins/blob/0682c1c82ec59cf5c67c3f859e2e25e011ee0c7a/v4/server/grpc/grpc.go#L401

Describe the solution you'd like if posible can we add 2 lines code of sentry like this image below

Additional context

timestamppb.timestamp cause "json: cannot unmarshal object into Go value of type string" after upgrade v4.6

Describe the bug

I use golang and upgrade go micro from v4.5 to v4.6
I expected that all of my integration test should pass
What happens instead? My logger print errors below: {"id":"go.micro.client.codec","code":500,"detail":"json: cannot unmarshal object into Go value of type string","status":"Internal Server Error"} In general it is said go.micro.client.codec has error json: cannot unmarshal object into Go value of type string.

How to reproduce the bug

My message structure in proto file

message Meeting {
    // @inject_tag: json:"start" validate:"required"
    google.protobuf.Timestamp start = 1;
}

The golang structure created by protoc

type Meeting struct {
	state         protoimpl.MessageState
	sizeCache     protoimpl.SizeCache
	unknownFields protoimpl.UnknownFields

	// @inject_tag: json:"start" validate:"required"
	Start *timestamppb.Timestamp `protobuf:"bytes,1,opt,name=start,proto3" json:"start" validate:"required"`
}

When I assign value to meeting.Start, the code like:

	meeting.Start = timestamp.New(model.Start) // timestamp imported from "google.golang.org/protobuf/types/known/timestamppb"

There is no such value if I continue using go micro v4.5. I am using libprotoc 3.21.6 and protoc-gen-go v1.27.1

Go Version output:

go version go1.17.13 darwin/amd64

The go env output:

GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/zhihong/Library/Caches/go-build"
GOENV="/Users/zhihong/Library/Application Support/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOINSECURE=""
GOMODCACHE="/Users/zhihong/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/zhihong/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/darwin_amd64"
GOVCS=""
GOVERSION="go1.17.13"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/dev/null"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -arch x86_64 -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/r7/f018pp653m3_fw0xv9jx_tgh0000gn/T/go-build2635807831=/tmp/go-build -gno-record-gcc-switches -fno-common"

Any suggestions welcomed, thanks,

James

[BUG] go-micro error reported during press test

go-micro

Concurrency: 5000 Connection pooling: 10

2022/09/01 17:17:03 client.go:47: INFO : concurrency: 5000
requests per client: 200
 
2022/09/01 17:17:03 client.go:49: INFO : Servers: 10.130.12.14:8900
 
2022/09/01 17:17:03 client.go:57: INFO : message size: 581 bytes
 
2022/09/01 17:17:07 client.go:125: ERROR: 失败请求, 错误信息: {"id":"go.micro.client","code":503,"detail":"connection closed before server preface received","status":"Service Unavailable"}, 返回值: <nil>
2022/09/01 17:17:07 client.go:125: ERROR: 失败请求, 错误信息: {"id":"go.micro.client","code":503,"detail":"connection closed before server preface received","status":"Service Unavailable"}, 返回值: <nil>
2022/09/01 17:17:07 client.go:125: ERROR: 失败请求, 错误信息: {"id":"go.micro.client","code":503,"detail":"connection closed before server preface received","status":"Service Unavailable"}, 返回值: <nil>
2022/09/01 17:17:08 client.go:125: ERROR: 失败请求, 错误信息: {"id":"go.micro.client","code":503,"detail":"connection closed before server preface received","status":"Service Unavailable"}, 返回值: <nil>
2022/09/01 17:17:09 client.go:125: ERROR: 失败请求, 错误信息: {"id":"go.micro.client","code":503,"detail":"connection closed before server preface received","status":"Service Unavailable"}, 返回值: <nil>
2022/09/01 17:17:09 client.go:125: ERROR: 失败请求, 错误信息: {"id":"go.micro.client","code":503,"detail":"connection closed before server preface received","status":"Service Unavailable"}, 返回值: <nil>
2022/09/01 17:17:09 client.go:125: ERROR: 失败请求, 错误信息: {"id":"go.micro.client","code":503,"detail":"connection closed before server preface received","status":"Service Unavailable"}, 返回值: <nil>
2022/09/01 17:17:09 client.go:125: ERROR: 失败请求, 错误信息: {"id":"go.micro.client","code":503,"detail":"connection closed before server preface received","status":"Service Unavailable"}, 返回值: <nil>
2022/09/01 17:17:09 client.go:125: ERROR: 失败请求, 错误信息: {"id":"go.micro.client","code":503,"detail":"connection closed before server preface received","status":"Service Unavailable"}, 返回值: <nil>
2022/09/01 17:17:09 client.go:125: ERROR: 失败请求, 错误信息: {"id":"go.micro.client","code":503,"detail":"connection closed before server preface received","status":"Service Unavailable"}, 返回值: <nil>
2022/09/01 17:17:09 client.go:125: ERROR: 失败请求, 错误信息: {"id":"go.micro.client","code":503,"detail":"connection closed before server preface received","status":"Service Unavailable"}, 返回值: <nil>
2022/09/01 17:17:09 client.go:125: ERROR: 失败请求, 错误信息: {"id":"go.micro.client","code":503,"detail":"connection closed before server preface received","status":"Service Unavailable"}, 返回值: <nil>
2022/09/01 17:17:09 client.go:125: ERROR: 失败请求, 错误信息: {"id":"go.micro.client","code":503,"detail":"connection closed before server preface received","status":"Service Unavailable"}, 返回值: <nil>
2022/09/01 17:17:09 client.go:125: ERROR: 失败请求, 错误信息: {"id":"go.micro.client","code":503,"detail":"connection closed before server preface received","status":"Service Unavailable"}, 返回值: <nil>
2022/09/01 17:17:09 client.go:125: ERROR: 失败请求, 错误信息: {"id":"go.micro.client","code":503,"detail":"connection closed before server preface received","status":"Service Unavailable"}, 返回值: <nil>
2022/09/01 17:17:09 client.go:125: ERROR: 失败请求, 错误信息: {"id":"go.micro.client","code":503,"detail":"connection closed before server preface received","status":"Service Unavailable"}, 返回值: <nil>
2022/09/01 17:17:09 client.go:125: ERROR: 失败请求, 错误信息: {"id":"go.micro.client","code":503,"detail":"connection closed before server preface received","status":"Service Unavailable"}, 返回值: <nil>
2022/09/01 17:17:09 client.go:125: ERROR: 失败请求, 错误信息: {"id":"go.micro.client","code":503,"detail":"connection closed before server preface received","status":"Service Unavailable"}, 返回值: <nil>
2022/09/01 17:17:09 client.go:125: ERROR: 失败请求, 错误信息: {"id":"go.micro.client","code":503,"detail":"connection closed before server preface received","status":"Service Unavailable"}, 返回值: <nil>
2022/09/01 17:17:09 client.go:125: ERROR: 失败请求, 错误信息: {"id":"go.micro.client","code":503,"detail":"connection closed before server preface received","status":"Service Unavailable"}, 返回值: <nil>
2022/09/01 17:17:09 client.go:125: ERROR: 失败请求, 错误信息: {"id":"go.micro.client","code":503,"detail":"connection closed before server preface received","status":"Service Unavailable"}, 返回值: <nil>
2022/09/01 17:17:09 client.go:125: ERROR: 失败请求, 错误信息: {"id":"go.micro.client","code":503,"detail":"connection closed before server preface received","status":"Service Unavailable"}, 返回值: <nil>
2022/09/01 17:17:09 client.go:125: ERROR: 失败请求, 错误信息: {"id":"go.micro.client","code":503,"detail":"connection closed before server preface received","status":"Service Unavailable"}, 返回值: <nil>
2022/09/01 17:17:09 client.go:125: ERROR: 失败请求, 错误信息: {"id":"go.micro.client","code":503,"detail":"connection closed before server preface received","status":"Service Unavailable"}, 返回值: <nil>
2022/09/01 17:17:09 client.go:125: ERROR: 失败请求, 错误信息: {"id":"go.micro.client","code":503,"detail":"connection closed before server preface received","status":"Service Unavailable"}, 返回值: <nil>
2022/09/01 17:17:09 client.go:125: ERROR: 失败请求, 错误信息: {"id":"go.micro.client","code":503,"detail":"connection closed before server preface received","status":"Service Unavailable"}, 返回值: <nil>
2022/09/01 17:17:09 client.go:125: ERROR: 失败请求, 错误信息: {"id":"go.micro.client","code":503,"detail":"connection closed before server preface received","status":"Service Unavailable"}, 返回值: <nil>
2022/09/01 17:17:09 client.go:125: ERROR: 失败请求, 错误信息: {"id":"go.micro.client","code":503,"detail":"connection closed before server preface received","status":"Service Unavailable"}, 返回值: <nil>
2022/09/01 17:17:09 client.go:125: ERROR: 失败请求, 错误信息: {"id":"go.micro.client","code":503,"detail":"connection closed before server preface received","status":"Service Unavailable"}, 返回值: <nil>
2022/09/01 17:17:09 client.go:125: ERROR: 失败请求, 错误信息: {"id":"go.micro.client","code":503,"detail":"connection closed before server preface received","status":"Service Unavailable"}, 返回值: <nil>
2022/09/01 17:17:09 client.go:125: ERROR: 失败请求, 错误信息: {"id":"go.micro.client","code":503,"detail":"connection closed before server preface received","status":"Service Unavailable"}, 返回值: <nil>
2022/09/01 17:17:09 client.go:125: ERROR: 失败请求, 错误信息: {"id":"go.micro.client","code":503,"detail":"connection closed before server preface received","status":"Service Unavailable"}, 返回值: <nil>
2022/09/01 17:17:09 client.go:125: ERROR: 失败请求, 错误信息: {"id":"go.micro.client","code":503,"detail":"connection closed before server preface received","status":"Service Unavailable"}, 返回值: <nil>
2022/09/01 17:17:09 client.go:125: ERROR: 失败请求, 错误信息: {"id":"go.micro.client","code":503,"detail":"connection closed before server preface received","status":"Service Unavailable"}, 返回值: <nil>
2022/09/01 17:17:09 client.go:125: ERROR: 失败请求, 错误信息: {"id":"go.micro.client","code":503,"detail":"connection closed before server preface received","status":"Service Unavailable"}, 返回值: <nil>
2022/09/01 17:17:09 client.go:125: ERROR: 失败请求, 错误信息: {"id":"go.micro.client","code":503,"detail":"connection closed before server preface received","status":"Service Unavailable"}, 返回值: <nil>
2022/09/01 17:17:09 client.go:125: ERROR: 失败请求, 错误信息: {"id":"go.micro.client","code":503,"detail":"connection closed before server preface received","status":"Service Unavailable"}, 返回值: <nil>
2022/09/01 17:17:35 stats.go:16: INFO : 花了 31302 ms 用于 1000000 请求
2022/09/01 17:17:35 stats.go:38: INFO : 发送请求    : 1000000
2022/09/01 17:17:35 stats.go:39: INFO : 收到的请求    : 1000000
2022/09/01 17:17:35 stats.go:40: INFO : 收到的成功请求 : 999963
2022/09/01 17:17:35 stats.go:48: INFO : 吞吐量  (TPS)    : 31946
 
2022/09/01 17:17:35 stats.go:60: INFO : 平均值: 137153203.650 ns, 中位数: 4337898.500 ns, 最大值: 4303066047.000 ns, 最小值: 243587.000 ns, p99.9: 1817774720.500 ns
2022/09/01 17:17:35 stats.go:61: INFO : 平均值: 137.153 ms, 中位数: 4.338 ms, 最大值: 4303.066 ms, 最小值: 0.244 ms, p99.9: 1817.775 ms
2022/09/01 17:17:35 stats.go:62: INFO : [go-micro]: TPS: 31946.84, TP99: 887.68ms, TP999: 1817.77ms (b=581 Byte, c=5000, n=1000000)
Info: [go-micro]: TPS: 31946.84, TP99: 887.68ms, TP999: 1817.77ms (b=581 Byte, c=5000, n=1000000)

grpc

Concurrency: 5000 Connection pooling: 10

2022/09/01 17:18:27 grpc_client.go:43: INFO : concurrency: 5000
requests per client: 200

2022/09/01 17:18:27 grpc_client.go:46: INFO : Servers: 10.130.12.14:8900

2022/09/01 17:18:27 grpc_client.go:52: INFO : message size: 581 bytes


2022/09/01 17:18:33 stats.go:16: INFO : 花了 5953 ms 用于 1000000 请求
2022/09/01 17:18:34 stats.go:38: INFO : 发送请求    : 1000000
2022/09/01 17:18:34 stats.go:39: INFO : 收到的请求    : 1000000
2022/09/01 17:18:34 stats.go:40: INFO : 收到的成功请求 : 1000000
2022/09/01 17:18:34 stats.go:48: INFO : 吞吐量  (TPS)    : 167982

2022/09/01 17:18:34 stats.go:60: INFO : 平均值: 28944916.637 ns, 中位数: 25278866.000 ns, 最大值: 255435674.000 ns, 最小值: 207720.000 ns, p99.9: 137116793.500 ns
2022/09/01 17:18:34 stats.go:61: INFO : 平均值: 28.945 ms, 中位数: 25.279 ms, 最大值: 255.436 ms, 最小值: 0.208 ms, p99.9: 137.117 ms
2022/09/01 17:18:34 stats.go:62: INFO : [grpc]: TPS: 167982.53, TP99: 80.10ms, TP999: 137.12ms (b=581 Byte, c=5000, n=1000000)
Info: [grpc]: TPS: 167982.53, TP99: 80.10ms, TP999: 137.12ms (b=581 Byte, c=5000, n=1000000)

A grpc press test on a similarly configured machine shows no errors

@asim

[BUG] go-micro registry node nil pointer

Describe the bug

What are you trying to do?

Build a Rpc Client to request a Rpc Server

What did you expect to happen?

when rpc client try to get rpc server addr and request, "nil pointer exception" should not happen

What happens instead?

a panic occur which have already get a Rlock wihout unlock

/usr/local/go17/src/runtime/panic.go:221 (0xc76946)
/usr/local/go17/src/runtime/signal_unix.go:735 (0xc76916)
mydir/pkg/mod/go-micro.dev/[email protected]/util/registry/util.go:65 (0x1226a8c)
mydir/pkg/mod/go-micro.dev/[email protected]/util/registry/util.go:85 (0x1226cfc)
mydir/pkg/mod/go-micro.dev/[email protected]/registry/cache/cache.go:126 (0x1228a78)
mydir/pkg/mod/go-micro.dev/[email protected]/registry/cache/cache.go:436 (0x122a95d)
mydir/pkg/mod/go-micro.dev/[email protected]/selector/default.go:52 (0x127feeb)
mydir/pkg/mod/github.com/asim/go-micro/plugins/client/grpc/[email protected]/grpc.go:90 (0x1de1242)
mydir/pkg/mod/github.com/asim/go-micro/plugins/client/grpc/[email protected]/grpc.go:403 (0x1de4df6)
[email protected]/gomicro/wrap.go:595 (0x1e0c50c)
[email protected]/gomicro/wrap.go:575 (0x1e0b99e)
[email protected]/gomicro/wrap.go:312 (0x1e0936f)
myproject/cmu.pb.micro.go:238 (0x1b71d4e)
myproject/upcw_service.go:53 (0x1c0f57a)
myproject/upcw_common.go:33 (0x1e52904)
mydir/pkg/mod/github.com/gin-gonic/[email protected]/context.go:147 (0x1abd0ea)
[email protected]/middleware/http/recovery.go:39 (0x1abd0d4)
mydir/pkg/mod/github.com/gin-gonic/[email protected]/context.go:147 (0x1af3dca)
myproject/comm/middleware/session/session.go:82 (0x1af3dad)
mydir/pkg/mod/github.com/gin-gonic/[email protected]/context.go:147 (0x1af714d)
myproject/comm/metadata/metadata.go:113 (0x1af7134)
mydir/pkg/mod/github.com/gin-gonic/[email protected]/context.go:147 (0x1ab8ef3)
[email protected]/middleware/http/gin.go:293 (0x1ab8d73)
mydir/pkg/mod/github.com/gin-gonic/[email protected]/context.go:147 (0x116c369)
mydir/pkg/mod/github.com/gin-gonic/[email protected]/gin.go:403 (0x116bfb9)
mydir/pkg/mod/github.com/gin-gonic/[email protected]/gin.go:364 (0x116bb33)
mydir/pkg/mod/github.com/gorilla/[email protected]/mux.go:210 (0x1cb0b6e)
/usr/local/go17/src/net/http/server.go:2879 (0xf0ee7a)
/usr/local/go17/src/net/http/server.go:1930 (0xf0a467)
/usr/local/go17/src/runtime/asm_amd64.s:1581 (0xc945a0)

a goroutine panic with lock and other goroutine can't get the lock: pprof
cause other goroutine can not get lock resource until I restart it.

How to reproduce the bug: I do not know how to reproduce and why happen. I found it in product environment sometime and I have to restart the program to fix it or the program can not make rpc request.

Environment: Go Version: please paste go version output here


GO111MODULE="on"
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/myname/.cache/go-build"
GOENV="/home/myname/.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/myname/goproject17/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/myname/goproject17"
GOPRIVATE=""
GOPROXY=""
GOROOT="/usr/local/go17"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go17/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="go1.17.6"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/dev/null"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build4281984978=/tmp/go-build -gno-record-gcc-switches"

Remove cmd package

Looking at removing the cmd package. I think flag and env var parsing was initially quite useful when the surface area of go-micro was small. Over time that's gotten broader and more complex to manage. Ultimately if go-micro is pluggable, then either the cmd package needs to be totally rewritten or removed entirely.

Distributed-Services - Distributed Systems with Golang to consequently build a fully-fletched distributed service

Distributed-Services This project is essentially a result of my attempt to under

Jun 1, 2022

Distributed lock manager. Warning: very hard to use it properly. Not because it's broken, but because distributed systems are hard. If in doubt, do not use this.

What Dlock is a distributed lock manager [1]. It is designed after flock utility but for multiple machines. When client disconnects, all his locks are

Dec 24, 2019

Tarmac is a unique framework designed for the next generation of distributed systems

Framework for building distributed services with Web Assembly

Dec 31, 2022

Micro is a platform for cloud native development

Micro Overview Micro addresses the key requirements for building services in the cloud. It leverages the microservices architecture pattern and provid

Dec 31, 2022

A distributed systems library for Kubernetes deployments built on top of spindle and Cloud Spanner.

hedge A library built on top of spindle and Cloud Spanner that provides rudimentary distributed computing facilities to Kubernetes deployments. Featur

Nov 9, 2022

MIT 6.824: Distributed Systems

MIT 6.824 is a core 12-unit graduate subject with lectures, readings, programming labs, an optional project, a mid-term exam, and a final exam.

Jul 6, 2022

Distributed Systems 2021 - Miniproject 3

Distributed Systems 2021 -- Miniproject 3 Hand-in Date: 1 December 2021 (at 23:59) What to submit on learnit: a single zip-compressed file containing:

Dec 11, 2021

Labs, solutions and related materials from the MIT 6.824 Distributed Systems course.

MIT 6.824 Distributed Systems Labs, solutions and related materials from the MIT 6.824 Distributed Systems course. Overview From the official website:

Nov 5, 2022

MIT6.824 Distributed Systems

MIT6.824-Distributed-Systems My Solutions for MIT6.824

Jan 28, 2022

Distributed reliable key-value store for the most critical data of a distributed system

etcd Note: The main branch may be in an unstable or even broken state during development. For stable versions, see releases. etcd is a distributed rel

Dec 30, 2022

Skynet is a framework for distributed services in Go.

##Introduction Skynet is a communication protocol for building massively distributed apps in Go. It is not constrained to Go, so it will lend itself n

Nov 18, 2022

a dynamic configuration framework used in distributed system

go-archaius This is a light weight configuration management framework which helps to manage configurations in distributed system The main objective of

Dec 9, 2022

A Distributed Content Licensing Framework (DCLF) using Hyperledger Fabric permissioned blockchain.

Nov 4, 2022

Flowgraph package for scalable asynchronous system development

flowgraph Getting Started go get -u github.com/vectaport/flowgraph go test Links Wiki Slides from Minneapolis Golang Meetup, May 22nd 2019 Overview F

Dec 22, 2022

distributed data sync with operational transformation/transforms

DOT The DOT project is a blend of operational transformation, CmRDT, persistent/immutable datastructures and reactive stream processing. This is an im

Dec 16, 2022

High performance, distributed and low latency publish-subscribe platform.

Emitter: Distributed Publish-Subscribe Platform Emitter is a distributed, scalable and fault-tolerant publish-subscribe platform built with MQTT proto

Jan 2, 2023

Fast, efficient, and scalable distributed map/reduce system, DAG execution, in memory or on disk, written in pure Go, runs standalone or distributedly.

Gleam Gleam is a high performance and efficient distributed execution system, and also simple, generic, flexible and easy to customize. Gleam is built

Jan 1, 2023

Glow is an easy-to-use distributed computation system written in Go, similar to Hadoop Map Reduce, Spark, Flink, Storm, etc. I am also working on another similar pure Go system, https://github.com/chrislusf/gleam , which is more flexible and more performant.

glow Purpose Glow is providing a library to easily compute in parallel threads or distributed to clusters of machines. This is written in pure Go. I a