Draft change of merging v2 into master. Probably have to freeze v2 and master from now until I can merge this. I'm going to come back to this and update things that are missing/incorrect before we can make the release.

Fixes #1223

This PR adds support optional response_body field for google.api.HttpRule. Also see PR: https://github.com/google/go-genproto/pull/87 and https://github.com/googleapis/googleapis/pull/512

This version of code can work with current HttpRule implementation and with forked implementation with new field. This implementation does not break existing code.

Also allow_repeated_fields_in_body flag added to make it possible using repeated fields in body and response_body. This flag can be used for slices in request and response.

Also flag json_name_in_swgdef added to make it possible to use Field.GetJsonName() instead of Field.GetName() for generating swagger definitions. See: https://github.com/grpc-ecosystem/grpc-gateway/pull/681 for more info.

I figured we should have an issue to track the outstanding work for tagging a first stable v2 version of the gateway and switching the default branches. I've looked through the issue tracker and I think these are some of the issues we want to tackle:

• [x] Issues mentioned in https://github.com/grpc-ecosystem/grpc-gateway/pull/546, including:
  • [x] Emitting defaults (https://github.com/grpc-ecosystem/grpc-gateway/issues/233)
  • [x] Using jsonpb marshaller by default
  • [x] Changing the output of errors to match the JSON representation of a gRPC status (https://github.com/grpc-ecosystem/grpc-gateway/issues/1098).
  • [x] Adopt correct behaviour for custom verbs (see https://github.com/grpc-ecosystem/grpc-gateway/issues/224#issuecomment-426091809).
  • [x] Change default JSON marshalling to use camelCase (https://github.com/grpc-ecosystem/grpc-gateway/pull/540) (and update swagger: https://github.com/grpc-ecosystem/grpc-gateway/issues/375)
• [x] Switch to using the new Go protobuf stack (depends on https://github.com/grpc/grpc-go/pull/3453) (https://github.com/grpc-ecosystem/grpc-gateway/pull/1165 is the first part of this). #1147 #1719
• [x] Use rules_go version which support APIv2 well known types (https://github.com/bazelbuild/rules_go/issues/2514)
• [x] Minimize public API per https://github.com/grpc-ecosystem/grpc-gateway/pull/1146
• [x] Rename swagger to openapi (https://github.com/grpc-ecosystem/grpc-gateway/issues/675)
• [x] Consolidate error handler configuration.
• [x] Make HTTPBodyMarshaler behaviour the default
• [x] Remove runtime.DisallowUnknownFields

I will update this issue as we find more things we want to have in a v2 release. I've also updated the v2 milestone: https://github.com/grpc-ecosystem/grpc-gateway/milestone/3.

this addresses #379

If a binding is mapped to patch and the request message has exactly one FieldMask message in it, additional code is rendered for the gateway handler that will populate the FieldMask based on the request body.

This handles two scenarios: The FieldMask is hidden from the REST request (as in the UpdateV2 example). In this case, the FieldMask is updated from the request body and set in the gRPC request message.

The FieldMask is exposed to the REST request (as in the PatchWithFieldMaskInBody example). For this case, a check is made as to whether the FieldMask is nil/empty prior to populating with the request body. If it's not nil, then it converts the FieldMask paths from the REST (snake_case) to gRPC (PascalCase) format. Otherwise, it acts like the previous case.

Additional comments of interest are inline on this PR.

I have the following endpoint and user_ids of the format "user:"

  rpc GetUser(GetUserRequest) returns (GetUserResponse) {
    option (google.api.http) = {
      get: "/v0/users/{user_id}"
    };
  };

I'm getting 404 status code and it doesn't hit my GetUser handler when I call this endpoint with user_ids of the format above. However, if I call the endpoint with a user_id that doesn't have a colon it (e.g. "user123") it does hit my GetUser handler. Url encoding the colon doesn't work either.

What's even more interesting is if I add another colon to the end of the path, grpc-gateway parses the user_id into the correct format that I want. ("/v0/users/user:123:" -> user_id = "user:123"). Given this data point, I believe grpc-gateway is treating colons differently in the last path segment.

For example here: https://github.com/grpc-ecosystem/grpc-gateway/blob/master/protoc-gen-grpc-gateway/httprule/parse.go#L86 it strips off the last colon in the segment. Though this looks like the rule for parsing the endpoint path definition and not what is being used to parse incoming requests.

sometimes we don't need the HTTP gateway to be a RPC. just convert the gRPC service to an HTTP service. this reduces one remote call.

example code

func main() {
	s := grpc.NewServer(
	)
	srv := service{}
	pb.RegisterExampleServiceServer(s, &srv)

	ctx := context.Background()
	ctx, cancel := context.WithCancel(ctx)
	defer cancel()

	mux := runtime.NewServeMux()
	err := pb.RegisterExampleServiceHandlerServer(ctx, mux, &srv)
	if err != nil {
		log.Panic(err)
	}

	if err := http.ListenAndServe(serveAddr, mux); err != nil {
		log.Fatalf("http failed to serve: %v", err)
	}
}

• swaggerSchemaObject.Properties is now optional (pointer), because google.protobuf.Empty requires us to set Properties{} instead of nil, because this equals in Swagger to an empty JSON object (which is exactly what Empty represents and how the gateway treats it). If it's not a pointer, we can't distinguish between "not set" (in most cases, we don't want Properties to be set, instead usually just .Ref is set), and "set to an empty value e.g. Properties{} with length 0). We don't want Properties{} to occur except for specific cases, e.g. for Empty.
• Wrappers and Empty are not rendered as definitions, now also for RPC Method input/output. instead, all necessary schema information is provided "in-line" via schema.type and schema.properties.
• Empty is now omitted as well if it's input/output of a RPC Method.

Steps you follow to reproduce the error:

1. Using go1.11 outside of go path
2. Run go get -u github.com/grpc-ecosystem/grpc-gateway/protoc-gen-grpc-gateway

Returns

go: finding github.com/grpc-ecosystem/grpc-gateway/protoc-gen-grpc-gateway latest

# github.com/grpc-ecosystem/grpc-gateway/protoc-gen-grpc-gateway/descriptor

../../go/pkg/mod/github.com/grpc-ecosystem/[email protected]/protoc-gen-grpc-gateway/descriptor/services.go:146:49: opts.ResponseBody undefined (type *annotations.HttpRule has no field or method ResponseBody)
-->

What did you expect to happen instead: Successfully added to mod file

It seems like it's able to find the latest version 1.5.0, but then isn't up to date? The error that's output seems to be what was fixed with #731

Introduce GO Templates into proto comments to allow more advanced documentation such as from external (markdown) files and automatically generated content based on the proto definitions.

For example: The comments in this proto file:

syntax = "proto3";

package LoginTest;

import "google/api/annotations.proto";

service LoginService {

// Logout
// 
// {{.MethodDescriptorProto.Name}} is a call with the method(s) {{$first := true}}{{range .Bindings}}{{if $first}}{{$first = false}}{{else}}, {{end}}{{.HTTPMethod}}{{end}} within the "{{.Service.Name}}" service.
// It takes in "{{.RequestType.Name}}" and returns a "{{.ResponseType.Name}}".
// 
// ## {{.RequestType.Name}}
// | Field ID    | Name      | Type                                                       | Description                  |
// | ----------- | --------- | ---------------------------------------------------------  | ---------------------------- | {{range .RequestType.Fields}}
// | {{.Number}} | {{.Name}} | {{if eq .Label.String "LABEL_REPEATED"}}[]{{end}}{{.Type}} | {{fieldcomments .Message .}} | {{end}}  
//  
// ## {{.ResponseType.Name}}
// | Field ID    | Name      | Type                                                       | Description                  |
// | ----------- | --------- | ---------------------------------------------------------- | ---------------------------- | {{range .ResponseType.Fields}}
// | {{.Number}} | {{.Name}} | {{if eq .Label.String "LABEL_REPEATED"}}[]{{end}}{{.Type}} | {{fieldcomments .Message .}} | {{end}}  
rpc Logout (LogoutRequest) returns (LogoutReply) {
  option (google.api.http) = {
    post: "/v1/example/ekko"
    body: "*"
    additional_bindings {
      post: "/test/test/test"
    }
    };
  }
}

message LogoutRequest {
  // This field only contains this title
  string timeoflogout = 1;
  // This is the title
  //
  // This is the "Description" of field test
  // you can use as many newlines as you want
  //
  //
  // it will still format the same in the table
  int32 test = 2;
  // Array title
  repeated string stringarray = 3;
}

message LogoutReply {
  // Message that displays whether the logout was succesful or not
  string message = 1;
}

would generate the following documentation in Swagger UI:

or when imported in Postman:

• OpenAPI 3.0 support nullable [link] field to define schema can be null value.
• go-swagger support vendor extensions for x-nullable [link]

Should protoc-gen-swagger convert well know types to nullable type?

This PR prepends the service name to the method name to generate unique operation IDs. Prior to this change, the swagger generator would produce spec non-compliant swagger when multiple services had shared method names. This is quite problematic for generic CRUD services with method names like Create, Delete, etc.

References to other Issues or PRs

Fixes #3095

Have you read the Contributing Guidelines?

Yes.

Brief description of what is fixed or changed

Support JSON examples in YAML output, add a test.

Other comments

📚 Documentation

The documentation says:

HTTP headers that start with 'Grpc-Metadata-' are mapped to gRPC metadata (prefixed with grpcgateway-).

I expected the metadata being prefixed with grpcgateway-. However they are not. After digging into the code, it turns out that the code doesn't add the prefix to the metadata:

// DefaultHeaderMatcher is used to pass http request headers to/from gRPC context. This adds permanent HTTP header
// keys (as specified by the IANA) to gRPC context with grpcgateway- prefix. HTTP headers that start with
// 'Grpc-Metadata-' are mapped to gRPC metadata after removing prefix 'Grpc-Metadata-'.
func DefaultHeaderMatcher(key string) (string, bool) {
	key = textproto.CanonicalMIMEHeaderKey(key)
	if isPermanentHTTPHeader(key) {
		return MetadataPrefix + key, true
	} else if strings.HasPrefix(key, MetadataHeaderPrefix) {
		return key[len(MetadataHeaderPrefix):], true
	}
	return "", false
}

Since our code base relies on this behavior, fixing the code may break existing users. So maybe it's good to just change the documentation to align with the code.

Removes json5. It's no longer used after updating ancestor dependency webpack-stream. These dependencies need to be updated together.

Removes json5

Updates webpack-stream from 3.2.0 to 7.0.0

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

References to other Issues or PRs

Have you read the Contributing Guidelines?

Yes

Brief description of what is fixed or changed

Allow extensions to be used with the recently added AllOf feature

Other comments

🐛 Bug Report

Hello, this is my first issue, I just try using protoc-gen-openapiv2 and write some proto but I found when write the example of response and render it, the output will render []byte.

Sorry for my english.

To Reproduce

(Write your steps here:)

1. Write a example like this one:

examples: {
        key: "application/json"
        value: "{\"value\": \"the input value\"}"
      }

2. After generate, the value is like []byte:

My example proto

service Signup {
    rpc Signup (SignupRequest) returns (SampleResponse) {
      option (google.api.http) = {
        post: "/v1/signup",
        body: "*"
      };
      option (grpc.gateway.protoc_gen_openapiv2.options.openapiv2_operation) = {
        consumes: [""]
        responses: {
            key: "200"
            value: {
                description: "Success"
            }
        }
        responses: {
            key: "500"
            value: {
                description: "internal Server Error",
                schema: {
                  json_schema: {
                    ref: ".Error";
                  }
                }
                examples: {
                  key: "application/json"
                  value: "{\"value\": \"the input value\"}"
                }
            }
        }
      };
    }
}

i;m using buf to generate, this is my configuration:

version: v1
managed:
  enabled: true
  go_package_prefix:
    default: github.com/example/example-proto
plugins:
  - remote: buf.build/grpc-ecosystem/plugins/openapiv2:v2.15.0
    opt:
      - merge_file_name=config
      - allow_merge
      - use_allof_for_refs
    out: ../../gen/openapiv2

It would be very nice if there is a type which can receive raw binary data as body, and all according metadata via headers (Content-Type video/mp4).

Browsers of today can send real binary data (const binaryData = new Uint8Array(buffer);).

Curl scripts also (curl --data-binary "@filepath").

This would reduce the transferred data size by 33%.

I refer to HttpBody, because this type is only usable as a "root" response type in a transcoded context, why not bidirectional?

Originally posted by @veith in https://github.com/grpc-ecosystem/grpc-gateway/issues/1781#issuecomment-1364121185