A Protocol Buffers compiler that generates optimized marshaling & unmarshaling Go code for ProtoBuf APIv2

vtprotobuf, the Vitess Protocol Buffers compiler

This repository provides the protoc-gen-go-vtproto plug-in for protoc, which is used by Vitess to generate optimized marshall & unmarshal code.

The code generated by this compiler is based on the optimized code generated by gogo/protobuf, although this package is not a fork of the original gogo compiler, as it has been implemented to support the new ProtoBuf APIv2 packages.

Available features

vtprotobuf is implemented as a helper plug-in that must be run alongside the upstream protoc-gen-go generator, as it generates fully-compatible auxiliary code to speed up (de)serialization of Protocol Buffer messages.

The following features can be generated:

  • size: generates a func (p *YourProto) SizeVT() int helper that behaves identically to calling proto.Size(p) on the message, except the size calculation is fully unrolled and does not use reflection. This helper function can be used directly, and it'll also be used by the marshal codegen to ensure the destination buffer is properly sized before ProtoBuf objects are marshalled to it.

  • marshal: generates the following helper methods

    • func (p *YourProto) MarshalVT() ([]byte, error): this function behaves identically to calling proto.Marshal(p), except the actual marshalling has been fully unrolled and does not use reflection or allocate memory. This function simply allocates a properly sized buffer by calling SizeVT on the message and then uses MarshalToSizedBufferVT to marshal to it.

    • func (p *YourProto) MarshalToVT(data []byte) (int, error): this function can be used to marshal a message to an existing buffer. The buffer must be large enough to hold the marshalled message, otherwise this function will panic. It returns the number of bytes marshalled. This function is useful e.g. when using memory pooling to re-use serialization buffers.

    • func (p *YourProto) MarshalToSizedBufferVT(data []byte) (int, error): this function behaves like MarshalTo but expects that the input buffer has the exact size required to hold the message, otherwise it will panic.

  • unmarshal: generates a func (p *YourProto) UnmarshalVT(data []byte) that behaves similarly to calling proto.Unmarshal(data, p) on the message, except the unmarshalling is performed by unrolled codegen without using reflection and allocating as little memory as possible. If the receiver p is not fully zeroed-out, the unmarshal call will actually behave like proto.Merge(data, p). This is because the proto.Unmarshal in the ProtoBuf API is implemented by resetting the destionation message and then calling proto.Merge on it. To ensure proper Unmarshal semantics, ensure you've called proto.Reset on your message before calling UnmarshalVT, or that your message has been newly allocated.

  • pool: generates the following helper methods

    • func (p *YourProto) ResetVT(): this function behaves similarly to proto.Reset(p), except it keeps as much memory as possible available on the message, so that further calls to UnmarshalVT on the same message will need to allocate less memory. This an API meant to be used with memory pools and does not need to be used directly.

    • func (p *YourProto) ReturnToVTPool(): this function returns message p to a local memory pool so it can be reused later. It clears the object properly with ResetVT before storing it on the pool. This method should only be used on messages that were obtained from a memory pool by calling YourProtoFromVTPool. Using p after calling this method will lead to undefined behavior.

    • func YourProtoFromVTPool() *YourProto: this function returns a YourProto message from a local memory pool, or allocates a new one if the pool is currently empty. The returned message is always empty and ready to be used (e.g. by calling UnmarshalVT on it). Once the message has been processed, it must be returned to the memory pool by calling ReturnToVTPool() on it. Returning the message to the pool is not mandatory (it does not leak memory), but if you don't return it, that defeats the whole point of memory pooling.

Usage

  1. Install protoc-gen-go-vtproto:
go install github.com/planetscale/vtprotobuf/cmd/protoc-gen-go-vtproto
  1. Ensure your project is already using the ProtoBuf v2 API (i.e. google.golang.org/protobuf). The vtprotobuf compiler is not compatible with APIv1 generated code.

  2. Update your protoc generator to use the new plug-in. Example from Vitess:

for name in $(PROTO_SRC_NAMES); do \
    $(VTROOT)/bin/protoc \
    --go_out=. --plugin protoc-gen-go="${GOBIN}/protoc-gen-go" \
    --go-grpc_out=. --plugin protoc-gen-go-grpc="${GOBIN}/protoc-gen-go-grpc" \
    --go-vtproto_out=. --plugin protoc-gen-go-vtproto="${GOBIN}/protoc-gen-go-vtproto" \
    --go-vtproto_opt=features=marshal+unmarshal+size \
    proto/$${name}.proto; \
done

Note that the vtproto compiler runs like an auxiliary plug-in to the protoc-gen-go in APIv2, just like the new GRPC compiler plug-in, protoc-gen-go-grpc. You need to run it alongside the upstream generator, not as a replacement.

  1. (Optional) Pass the features that you want to generate as --go-vtproto_opt. If no features are given, all the codegen steps will be performed.

  2. Compile the .proto files in your project. You should see _vtproto.pb.go files next to the .pb.go and _grpc.pb.go files that were already being generated.

  3. (Optional) Switch your RPC framework to use the optimized helpers (see following sections)

Using the optimized code with RPC frameworks

The protoc-gen-go-vtproto compiler does not overwrite any of the default marshalling or unmarshalling code for your ProtoBuf objects. Instead, it generates helper methods that can be called explicitly to opt-in to faster (de)serialization.

vtprotobuf with GRPC

To use vtprotobuf with the new versions of GRPC, you need to register the codec provided by the github.com/planetscale/vtprotobuf/codec/grpc package.

package servenv

import (
    "github.com/planetscale/vtprotobuf/codec/grpc"
	"google.golang.org/grpc/encoding"
	_ "google.golang.org/grpc/encoding/proto"
)

func init() {
	encoding.RegisterCodec(grpc.Codec{})
}

Note that we perform a blank import _ "google.golang.org/grpc/encoding/proto" of the default proto coded that ships with GRPC to ensure it's being replaced by us afterwards. The provided Codec will serialize & deserialize all ProtoBuf messages using the optimized codegen.

Twirp

I actually have no idea of how to switch encoders in Twirp. Maybe it's not even possible.

DRPC

To use vtprotobuf as a DRPC encoding, simply pass github.com/planetscale/vtprotobuf/codec/drpc as the protolib flag in your protoc-gen-go-drpc invocation.

Example:

protoc --go_out=. --go-vtproto_out=. --go-drpc_out=. --go-drpc_opt=protolib=github.com/planetscale/vtprotobuf/codec/drpc
Owner
PlanetScale
We help operationalize Vitess in the cloud and on premise.
PlanetScale
Comments
  • bug fix in reset slice

    bug fix in reset slice

    Hi! We are using your plugin and its working really great, but recently we have discovered one little bug about memory pooling, connected with this issue https://github.com/planetscale/vtprotobuf/issues/8 . When the message contains repeated field you just reduse size to zero when ResetVT not to overload GC and its cool. The problem is that if repeated element contains pointer field or other repeated field, firstly when you unmarshall data to it and these fields are present in request, fields will be filled in. Secondly, when another request comes without pointer field or slice in element, you take element from pool with this fields set, unmarshal data to it, and then you have a kind of merged data from previous request. Thats why we thought that it may be good to reduse size to zero not only at top level slice, but on lower levels too, and we came with pr as one of the options to solve this issue without overhead of complexity. I hope that i could explain everything clearly. Waiting very much for your feedback

  • buf.build support?

    buf.build support?

    Unfortunately when using buf.build, #19 pops up again, except since buf is doing the work there's no easy way just run a single protoc.

    I'm not sure if this is something vtprotobuf can work around, or if this requires a feature in buf but it's a shame that these two tools don't play nicely together.

  • Build and test in CI

    Build and test in CI

    I had to build protobuf as this is the only way I found to obtain conformance-test-runner. This build is cached but first runs (for example any new pull request) will take a 30min-ish hit. I can remove this time to zero by caching somewhere less temporary such as some Docker image in this repo's registry. What do you think?

    While setting this up I noticed some generated files where out of sync with git. Should I add their diff to this PR?

    Details
    diff --git a/conformance/internal/conformance/conformance_vtproto.pb.go b/conformance/internal/conformance/conformance_vtproto.pb.go
    index 980bbd5..8d6971f 100644
    --- a/conformance/internal/conformance/conformance_vtproto.pb.go
    +++ b/conformance/internal/conformance/conformance_vtproto.pb.go
    @@ -89,19 +89,9 @@ func (m *ConformanceRequest) MarshalToSizedBufferVT(dAtA []byte) (int, error) {
     		i -= len(m.unknownFields)
     		copy(dAtA[i:], m.unknownFields)
     	}
    -	if m.PrintUnknownFields {
    -		i--
    -		if m.PrintUnknownFields {
    -			dAtA[i] = 1
    -		} else {
    -			dAtA[i] = 0
    -		}
    -		i--
    -		dAtA[i] = 0x48
    -	}
     	if vtmsg, ok := m.Payload.(interface {
    -		MarshalToVT([]byte) (int, error)
    -		SizeVT() int
    +		MarshalTo([]byte) (int, error)
    +		Size() int
     	}); ok {
     		{
     			size := vtmsg.SizeVT()
    @@ -[11](https://github.com/fenollp/vtprotobuf/runs/5318361134?check_suite_focus=true#step:7:11)1,6 +101,16 @@ func (m *ConformanceRequest) MarshalToSizedBufferVT(dAtA []byte) (int, error) {
     			}
     		}
     	}
    +	if m.PrintUnknownFields {
    +		i--
    +		if m.PrintUnknownFields {
    +			dAtA[i] = 1
    +		} else {
    +			dAtA[i] = 0
    +		}
    +		i--
    +		dAtA[i] = 0x48
    +	}
     	if m.JspbEncodingOptions != nil {
     		size, err := m.JspbEncodingOptions.MarshalToSizedBufferVT(dAtA[:i])
     		if err != nil {
    @@ -228,8 +228,8 @@ func (m *ConformanceResponse) MarshalToSizedBufferVT(dAtA []byte) (int, error) {
     		copy(dAtA[i:], m.unknownFields)
     	}
     	if vtmsg, ok := m.Result.(interface {
    -		MarshalToVT([]byte) (int, error)
    -		SizeVT() int
    +		MarshalTo([]byte) (int, error)
    +		Size() int
     	}); ok {
     		{
     			size := vtmsg.SizeVT()
    diff --git a/conformance/internal/conformance/test_messages_proto2_vtproto.pb.go b/conformance/internal/conformance/test_messages_proto2_vtproto.pb.go
    index 23972ed..119b4be 100644
    --- a/conformance/internal/conformance/test_messages_proto2_vtproto.pb.go
    +++ b/conformance/internal/conformance/test_messages_proto2_vtproto.pb.go
    @@ -259,6 +259,18 @@ func (m *TestAllTypesProto2) MarshalToSizedBufferVT(dAtA []byte) (int, error) {
     		i -= len(m.unknownFields)
     		copy(dAtA[i:], m.unknownFields)
     	}
    +	if vtmsg, ok := m.OneofField.(interface {
    +		MarshalTo([]byte) (int, error)
    +		Size() int
    +	}); ok {
    +		{
    +			size := vtmsg.SizeVT()
    +			i -= size
    +			if _, err := vtmsg.MarshalToVT(dAtA[i:]); err != nil {
    +				return 0, err
    +			}
    +		}
    +	}
     	if m.FieldName18__ != nil {
     		i = encodeVarint(dAtA, i, uint64(*m.FieldName18__))
     		i--
    @@ -400,18 +4[12](https://github.com/fenollp/vtprotobuf/runs/5318361134?check_suite_focus=true#step:7:12),6 @@ func (m *TestAllTypesProto2) MarshalToSizedBufferVT(dAtA []byte) (int, error) {
     		i--
     		dAtA[i] = 0xcb
     	}
    -	if vtmsg, ok := m.OneofField.(interface {
    -		MarshalToVT([]byte) (int, error)
    -		SizeVT() int
    -	}); ok {
    -		{
    -			size := vtmsg.SizeVT()
    -			i -= size
    -			if _, err := vtmsg.MarshalToVT(dAtA[i:]); err != nil {
    -				return 0, err
    -			}
    -		}
    -	}
     	if len(m.UnpackedNestedEnum) > 0 {
     		for iNdEx := len(m.UnpackedNestedEnum) - 1; iNdEx >= 0; iNdEx-- {
     			i = encodeVarint(dAtA, i, uint64(m.UnpackedNestedEnum[iNdEx]))
    diff --git a/conformance/internal/conformance/test_messages_proto3_vtproto.pb.go b/conformance/internal/conformance/test_messages_proto3_vtproto.pb.go
    index 76f05a8..d5fecd2 100644
    --- a/conformance/internal/conformance/test_messages_proto3_vtproto.pb.go
    +++ b/conformance/internal/conformance/test_messages_proto3_vtproto.pb.go
    @@ -104,6 +104,18 @@ func (m *TestAllTypesProto3) MarshalToSizedBufferVT(dAtA []byte) (int, error) {
     		i -= len(m.unknownFields)
     		copy(dAtA[i:], m.unknownFields)
     	}
    +	if vtmsg, ok := m.OneofField.(interface {
    +		MarshalTo([]byte) (int, error)
    +		Size() int
    +	}); ok {
    +		{
    +			size := vtmsg.SizeVT()
    +			i -= size
    +			if _, err := vtmsg.MarshalToVT(dAtA[i:]); err != nil {
    +				return 0, err
    +			}
    +		}
    +	}
     	if m.FieldName18__ != 0 {
     		i = encodeVarint(dAtA, i, uint64(m.FieldName18__))
     		i--
    @@ -10[13](https://github.com/fenollp/vtprotobuf/runs/5318361134?check_suite_focus=true#step:7:13),18 +1025,6 @@ func (m *TestAllTypesProto3) MarshalToSizedBufferVT(dAtA []byte) (int, error) {
     		i--
     		dAtA[i] = 0xca
     	}
    -	if vtmsg, ok := m.OneofField.(interface {
    -		MarshalToVT([]byte) (int, error)
    -		SizeVT() int
    -	}); ok {
    -		{
    -			size := vtmsg.SizeVT()
    -			i -= size
    -			if _, err := vtmsg.MarshalToVT(dAtA[i:]); err != nil {
    -				return 0, err
    -			}
    -		}
    -	}
     	if len(m.UnpackedNestedEnum) > 0 {
     		for iNdEx := len(m.UnpackedNestedEnum) - 1; iNdEx >= 0; iNdEx-- {
     			i = encodeVarint(dAtA, i, uint64(m.UnpackedNestedEnum[iNdEx]))
    diff --git a/testproto/proto3opt/opt_vtproto.pb.go b/testproto/proto3opt/opt_vtproto.pb.go
    index 726b[15](https://github.com/fenollp/vtprotobuf/runs/5318361134?check_suite_focus=true#step:7:15)c..[16](https://github.com/fenollp/vtprotobuf/runs/5318361134?check_suite_focus=true#step:7:16)a5e8b 100644
    --- a/testproto/proto3opt/opt_vtproto.pb.go
    +++ b/testproto/proto3opt/opt_vtproto.pb.go
    @@ -50,6 +50,[19](https://github.com/fenollp/vtprotobuf/runs/5318361134?check_suite_focus=true#step:7:19)8 @@ func (m *OptionalFieldInProto3) MarshalToSizedBufferVT(dAtA []byte) (int, error)
     		i -= len(m.unknownFields)
     		copy(dAtA[i:], m.unknownFields)
     	}
    +	if vtmsg, ok := m.XOptionalEnum.(interface {
    +		MarshalTo([]byte) (int, error)
    +		Size() int
    +	}); ok {
    +		{
    +			size := vtmsg.SizeVT()
    +			i -= size
    +			if _, err := vtmsg.MarshalToVT(dAtA[i:]); err != nil {
    +				return 0, err
    +			}
    +		}
    +	}
    +	if vtmsg, ok := m.XOptionalBytes.(interface {
    +		MarshalTo([]byte) (int, error)
    +		Size() int
    +	}); ok {
    +		{
    +			size := vtmsg.SizeVT()
    +			i -= size
    +			if _, err := vtmsg.MarshalToVT(dAtA[i:]); err != nil {
    +				return 0, err
    +			}
    +		}
    +	}
    +	if vtmsg, ok := m.XOptionalString.(interface {
    +		MarshalTo([]byte) (int, error)
    +		Size() int
    +	}); ok {
    +		{
    +			size := vtmsg.SizeVT()
    +			i -= size
    +			if _, err := vtmsg.MarshalToVT(dAtA[i:]); err != nil {
    +				return 0, err
    +			}
    +		}
    +	}
    +	if vtmsg, ok := m.XOptionalBool.(interface {
    +		MarshalTo([]byte) (int, error)
    +		Size() int
    +	}); ok {
    +		{
    +			size := vtmsg.SizeVT()
    +			i -= size
    +			if _, err := vtmsg.MarshalToVT(dAtA[i:]); err != nil {
    +				return 0, err
    +			}
    +		}
    +	}
    +	if vtmsg, ok := m.XOptionalDouble.(interface {
    +		MarshalTo([]byte) (int, error)
    +		Size() int
    +	}); ok {
    +		{
    +			size := vtmsg.SizeVT()
    +			i -= size
    +			if _, err := vtmsg.MarshalToVT(dAtA[i:]); err != nil {
    +				return 0, err
    +			}
    +		}
    +	}
    +	if vtmsg, ok := m.XOptionalFloat.(interface {
    +		MarshalTo([]byte) (int, error)
    +		Size() int
    +	}); ok {
    +		{
    +			size := vtmsg.SizeVT()
    +			i -= size
    +			if _, err := vtmsg.MarshalToVT(dAtA[i:]); err != nil {
    +				return 0, err
    +			}
    +		}
    +	}
    +	if vtmsg, ok := m.XOptionalSfixed64.(interface {
    +		MarshalTo([]byte) (int, error)
    +		Size() int
    +	}); ok {
    +		{
    +			size := vtmsg.SizeVT()
    +			i -= size
    +			if _, err := vtmsg.MarshalToVT(dAtA[i:]); err != nil {
    +				return 0, err
    +			}
    +		}
    +	}
    +	if vtmsg, ok := m.XOptionalSfixed[32](https://github.com/fenollp/vtprotobuf/runs/5318361134?check_suite_focus=true#step:7:32).(interface {
    +		MarshalTo([]byte) (int, error)
    +		Size() int
    +	}); ok {
    +		{
    +			size := vtmsg.SizeVT()
    +			i -= size
    +			if _, err := vtmsg.MarshalToVT(dAtA[i:]); err != nil {
    +				return 0, err
    +			}
    +		}
    +	}
    +	if vtmsg, ok := m.XOptionalFixed[64](https://github.com/fenollp/vtprotobuf/runs/5318361134?check_suite_focus=true#step:7:64).(interface {
    +		MarshalTo([]byte) (int, error)
    +		Size() int
    +	}); ok {
    +		{
    +			size := vtmsg.SizeVT()
    +			i -= size
    +			if _, err := vtmsg.MarshalToVT(dAtA[i:]); err != nil {
    +				return 0, err
    +			}
    +		}
    +	}
    +	if vtmsg, ok := m.XOptionalFixed32.(interface {
    +		MarshalTo([]byte) (int, error)
    +		Size() int
    +	}); ok {
    +		{
    +			size := vtmsg.SizeVT()
    +			i -= size
    +			if _, err := vtmsg.MarshalToVT(dAtA[i:]); err != nil {
    +				return 0, err
    +			}
    +		}
    +	}
    +	if vtmsg, ok := m.XOptionalSint64.(interface {
    +		MarshalTo([]byte) (int, error)
    +		Size() int
    +	}); ok {
    +		{
    +			size := vtmsg.SizeVT()
    +			i -= size
    +			if _, err := vtmsg.MarshalToVT(dAtA[i:]); err != nil {
    +				return 0, err
    +			}
    +		}
    +	}
    +	if vtmsg, ok := m.XOptionalSint32.(interface {
    +		MarshalTo([]byte) (int, error)
    +		Size() int
    +	}); ok {
    +		{
    +			size := vtmsg.SizeVT()
    +			i -= size
    +			if _, err := vtmsg.MarshalToVT(dAtA[i:]); err != nil {
    +				return 0, err
    +			}
    +		}
    +	}
    +	if vtmsg, ok := m.XOptionalUint64.(interface {
    +		MarshalTo([]byte) (int, error)
    +		Size() int
    +	}); ok {
    +		{
    +			size := vtmsg.SizeVT()
    +			i -= size
    +			if _, err := vtmsg.MarshalToVT(dAtA[i:]); err != nil {
    +				return 0, err
    +			}
    +		}
    +	}
    +	if vtmsg, ok := m.XOptionalUint32.(interface {
    +		MarshalTo([]byte) (int, error)
    +		Size() int
    +	}); ok {
    +		{
    +			size := vtmsg.SizeVT()
    +			i -= size
    +			if _, err := vtmsg.MarshalToVT(dAtA[i:]); err != nil {
    +				return 0, err
    +			}
    +		}
    +	}
    +	if vtmsg, ok := m.XOptionalInt64.(interface {
    +		MarshalTo([]byte) (int, error)
    +		Size() int
    +	}); ok {
    +		{
    +			size := vtmsg.SizeVT()
    +			i -= size
    +			if _, err := vtmsg.MarshalToVT(dAtA[i:]); err != nil {
    +				return 0, err
    +			}
    +		}
    +	}
    +	if vtmsg, ok := m.XOptionalInt32.(interface {
    +		MarshalTo([]byte) (int, error)
    +		Size() int
    +	}); ok {
    +		{
    +			size := vtmsg.SizeVT()
    +			i -= size
    +			if _, err := vtmsg.MarshalToVT(dAtA[i:]); err != nil {
    +				return 0, err
    +			}
    +		}
    +	}
     	if m.OptionalEnum != nil {
     		i = encodeVarint(dAtA, i, uint64(*m.OptionalEnum))
     		i--
    
  • ReturnToVTPool() recursive?

    ReturnToVTPool() recursive?

    When func (p *YourProto) ReturnToVTPool() is called, children of YourProto that implement method ReturnToVTPool() should also be returned to the pool.

  • Error compiling generated code with third party imports

    Error compiling generated code with third party imports

    Hello,

    I was looking to experiment with this and ran into the following error:

    type *"google.golang.org/genproto/googleapis/rpc/status".Status has no field or method MarshalToSizedBufferVT
    

    The protobuf files that are complaining import files like:

    import "google/rpc/status.proto";
    

    with messages that look like:

    message TestMessage {
       google.rpc.Status status = 2;
    }
    

    The google/rpc/status.proto file is copied locally for the code generation, but the generated code is importing the Go module from google.golang.org/genproto/googleapis/rpc/status so it's not part of the vtproto generation steps.

    Is this an issue that you've had to resolve or any suggestions on how to approach this?

  • Codec with GRPC Server Support

    Codec with GRPC Server Support

    Thought/Question - Why does the GRPC codec not try and ReturnToVTPool on the way out? Is anyone else doing this. My thoughts are to have this for optimising the return of complex payloads - build them in the handler with ...FromVTPool() then let the codec release them once the wire-work has finished and the bytes are sent?

    Are there reasons for this not being in the Codec, and if not, is there an opening for a PR with a PoolAwareCodec to be added to the project?

  • equal: code doesn't distinguish between oneof fields when zero-valued

    equal: code doesn't distinguish between oneof fields when zero-valued

    Because the comparison logic for oneof fields relies on the getters for the individual fields, it cannot differentiate between a field not being set, and a field being set to the zero value. While the nil checks allow distinguishing protos where one has a oneof field set to a zero value, while the other doesn't have any field in the oneof set, the code fails to distinguish protos where different fields in a oneof are set to the respective zero value.

    Test case:

    func TestEqualVT_Oneof_AbsenceVsZeroValue(t *testing.T) {
    	a := &TestAllTypesProto3{
    		OneofField: &TestAllTypesProto3_OneofUint32{
    			OneofUint32: 0,
    		},
    	}
    	b := &TestAllTypesProto3{
    		OneofField: &TestAllTypesProto3_OneofString{
    			OneofString: "",
    		},
    	}
    
    	aJson, err := protojson.Marshal(a)
    	require.NoError(t, err)
    	bJson, err := protojson.Marshal(b)
    	require.NoError(t, err)
    
    	if a.EqualVT(b) {
    		assert.JSONEq(t, string(aJson), string(bJson))
    		err := fmt.Errorf("these %T should not be equal:\nmsg = %+v\noriginal = %+v", a, a, b)
    		require.NoError(t, err)
    	}
    }
    

    This is similar to #48 , but applies to oneofs and exercises different paths in the code generation.

  • Duplicate Functions and Variables in Generated Package

    Duplicate Functions and Variables in Generated Package

    First, just want to say thank you for working on and releasing vtprotobuf. We're working on transitioning out of gogo/protobuf and read your great blog post announcing this alternative.

    We've found a slight issue with our use case. We have a few proto packages that have multiple files in them. The generated _vtproto.pb.go files redeclare some utility functions/variables (e.g. sov, skip, ErrInvalidLength, etc.). As an example:

    hellopb/service.proto:

    syntax = "proto3";
    
    package hellopb;
    
    ...
    
    message HelloRequest {
      string q = 1;
    }
    
    message HelloResponse {
      string response = 2;
    }
    
    service HelloService {
      rpc Hello(HelloRequest) returns (HelloResponse) {}
    }
    

    hellopb/db.proto:

    syntax = "proto3";
    
    package hellopb;
    
    ...
    
    message MessageEntry {
      string q = 1;
      ...
    }
    

    We then run:

    protoc --proto_path=. --proto_path=../../../  \
      --go_out=../../../ --plugin protoc-gen-go=/go/bin/protoc-gen-go \
      --go-grpc_out=../../../ --plugin protoc-gen-go-grpc=/go/bin/protoc-gen-go-grpc \
      --grpc-gateway_out=../../../ \
      --go-vtproto_out=../../../ --plugin protoc-gen-go-vtproto=/go/bin/protoc-gen-go-vtproto \
      --go-vtproto_opt=features=marshal+unmarshal+size hellopb/service.proto
    

    and

    protoc --proto_path=. --proto_path=../../../  \
      --go_out=../../../ --plugin protoc-gen-go=/go/bin/protoc-gen-go \
      --go-grpc_out=../../../ --plugin protoc-gen-go-grpc=/go/bin/protoc-gen-go-grpc \
      --grpc-gateway_out=../../../ \
      --go-vtproto_out=../../../ --plugin protoc-gen-go-vtproto=/go/bin/protoc-gen-go-vtproto \
      --go-vtproto_opt=features=marshal+unmarshal+size hellopb/db.proto
    

    If we run go vet we get:

    service_vtproto.pb.go:214:6: encodeVarint redeclared in this block
        db_vtproto.pb.go:125:54: previous declaration
    service_vtproto.pb.go:308:6: sov redeclared in this block
        db_vtproto.pb.go:188:23: previous declaration
    service_vtproto.pb.go:311:6: soz redeclared in this block
        db_vtproto.pb.go:191:23: previous declaration
    service_vtproto.pb.go:694:6: skip redeclared in this block
        db_vtproto.pb.go:545:36: previous declaration
    service_vtproto.pb.go:774:2: ErrInvalidLength redeclared in this block
        db_vtproto.pb.go:625:2: previous declaration
    service_vtproto.pb.go:775:2: ErrIntOverflow redeclared in this block
        db_vtproto.pb.go:626:2: previous declaration
    service_vtproto.pb.go:776:2: ErrUnexpectedEndOfGroup redeclared in this block
        db_vtproto.pb.go:627:2: previous declaration
    

    Is there anyway to avoid this? Post protoc cleanup on this is pretty tough. Is there anyway those functions/variables could just be imported from vtprotobuf?

  • Inconsistent (de)serialization behavior

    Inconsistent (de)serialization behavior

    I am building an application protocol with protobufs, and I'm using vtprotobuf exclusively to marshal and unmarshal the messages. Currently, I'm experiencing strange behavior I'm not understanding that I think is related to vtprotobuf.

    Here are my message definitions:

    message Header {
      fixed32 Size = 1; // Size of the next message
      fixed32 Checksum = 2; // Checksum of the serialized message
    }
    
    message RaftControlPayload {
      oneof Types {
        GetLeaderIDRequest GetLeaderIdRequest = 1;
        GetLeaderIDResponse GetLeaderIdResponse = 2;
        IdRequest IdRequest = 3;
        IdResponse IdResponse = 4;
        IndexState IndexState = 5;
        ModifyNodeRequest ModifyNodeRequest = 6;
        ReadIndexRequest ReadIndexRequest = 7;
        ReadLocalNodeRequest ReadLocalNodeRequest = 8;
        RequestLeaderTransferResponse RequestLeaderTransferResponse = 9;
        RequestSnapshotRequest RequestSnapshotRequest = 10;
        SnapshotOption SnapshotOption = 12;
        StopNodeResponse StopNodeResponse = 13;
        StopRequest StopRequest = 14;
        StopResponse StopResponse = 15;
        SysOpState SysOpState = 16;
        DBError Error = 17;
      }
      enum MethodName {
          ADD_NODE = 0;
          ADD_OBSERVER = 1;
          ADD_WITNESS = 2;
          GET_ID = 3;
          GET_LEADER_ID = 4;
          READ_INDEX = 5;
          READ_LOCAL_NODE = 6;
          REQUEST_COMPACTION = 7;
          REQUEST_DELETE_NODE = 8;
          REQUEST_LEADER_TRANSFER = 9;
          REQUEST_SNAPSHOT = 10;
          STOP = 11;
          STOP_NODE = 12;
      }
      MethodName Method = 18;
    }
    

    This message serializes to 10 bytes, which I send across a network stream as a header for whatever unknown message payload is coming next. This allows me to simply pass raw protobuf messages across a network stream without having to leverage gRPC or other RPC frameworks.

    Sending a message across the network stream is pretty straightforward. I prepare a message, serialize the message, create a header with all of the appropriate values, serialize the header, send the header, then send the message.

    idReqPayload := &database.RaftControlPayload{
    	Method: database.RaftControlPayload_GET_ID,
    	Types: &database.RaftControlPayload_IdRequest{
    		IdRequest: &database.IdRequest{},
    	},
    }
    payloadBuf, _ := idReqPayload.MarshalVT()
    
    initialHeader := &transportv1.Header{
    	Size: uint32(len(payloadBuf)),
    	Checksum: crc32.ChecksumIEEE(payloadBuf),
    }
    headerBuf, _ := initialHeader.MarshalVT()
    
    stream.Write(headerBuf)
    stream.Write(payloadBuf)
    

    Receiving a message on the network stream is also pretty straightforward. I read the header into a buffer, deserialize it, read the next N bytes from the stream based off the Size field in the header message, and verify some checksums, then serialize the byte array into the equivalent messages.

    headerBuf := make([]byte, 10)
    if _, err := io.ReadFull(stream, headerBuf); err != nil {
    	logger.Error().Err(err).Msg("cannot readAndHandle raft control header")
    	continue
    }
    
    // marshall the header
    header := &transportv1.Header{}
    if err := header.UnmarshalVT(headerBuf); err != nil {
    	logger.Error().Err(err).Msg("cannot unmarshal header")
    	return
    }
    
    // prep the message buffer
    msgBuf := make([]byte, header.Size)
    if _, err := io.ReadFull(stream, msgBuf); err != nil {
    	logger.Error().Err(err).Msg("cannot read message payload")
    	return
    }
    
    // verify the message is intact
    checked := crc32.ChecksumIEEE(msgBuf)
    if checked != header.GetChecksum() {
    	logger.Error().Msg("checksums do not match")
    }
    
    // unmarshal the payload
    msg := &database.RaftControlPayload{}
    if err := msg.UnmarshalVT(msgBuf); err != nil {
    	logger.Error().Err(err).Msg("cannot unmarshal payload")
    }
    

    Here's where things start to get confusing. When I serialize idReqPayload via MarshalVT() and run a checksum against it, I'll get uint32(1298345897); when I send the header as you see here, the Size field is uint32(5) and Checksum is uint32(1298345897). When the header message gets deserialized on the receiving end of a localhost connection, it looks very different.

    The header message gets deserialized with the Size field being uint32(5) and the Checksum field being uint(1). That's the first strange thing.

    When I run a checksum against the next 5 bytes of the serialized idReqPayload payload which followed, it checksums to uint32(737000948) even though there was no change to the byte array from the time it was serialized to the time it was received. That's the second strange thing.

    When I run an equality check against the value of the deserialised header Checksum field against a local checksum of the serialized idReqPayload payload with checked := crc32.ChecksumIEEE(msgBuf); if checked != header.GetChecksum() { // ... }, it passes an equality check - the deserialized header Checksum field's value is uint(1) whereas the calculated checksum of the received message is uint32(737000948). That's the third strange thing.

    When I deserialize the serialized idReqPayload byte array, it deserializes without an error. However, the message information is incorrectly serialized. When I serialize protobuf with this configuration:

    idReqPayload := &database.RaftControlPayload{
    	Method: database.RaftControlPayload_GET_ID,
    	Types: &database.RaftControlPayload_IdRequest{
    		IdRequest: &database.IdRequest{},
    	},
    }
    

    It deserializes into this equivalent:

    msg := &database.RaftControlPayload{
    	Method: database.RaftControlPayload_ADD_NODE,
    	Types: nil,
    }
    

    The Method field is reset so the enum is defaulted to 0, and the Types field is nil.

    I'm fairly positive this could partially be related to #51, but I updated my local protoc-gen-go-vtproto binary to 0ae748f and the problem still persists. I've also eliminated the network stream as it's a localhost network stream, so nothing is intercepting it or modifying it in transit.

    Am I doing something wrong or is this a bug of some kind?

  • clone: generate unrolled code for cloning messages

    clone: generate unrolled code for cloning messages

    This adds a new clone option to the generator that, when enabled, generates the following optimized methods for each message type:

    • func (p *YourProto) CloneVT() *YourProto, which returns a cloned message, i.e., the resulting message is equal to p but does not share any memory.
    • func (p *YourProto) CloneGenericVT() proto.Message, which does the same as the above, with the benefit of offering a type-independent signature such that it can be invoked via an interface type assertion on any proto.Message (in a dynamic setting where the type is not known at compile time, CloneVT can only be accessed via reflection/MethodByName)

    Fixes #45. (actually, just one half, I can easily add the CopyVT logic if this is desired)

  • features: equal

    features: equal

    This begins the work of porting [a variant of] gogo's equal plugin

    An unrolled non-reflective implementation of proto.Equal is about twice faster in our testing.

    Note: this is WIP. It is hard to test right now (I am working on a CI PR that should help here) mainly because this command cannot be used with forks:

    go install github.com/planetscale/vtprotobuf/cmd/protoc-gen-go-vtproto@latest
    

    Any help around this is very welcomed!

  • unknown feature:

    unknown feature: "clone"

    Works fine...

    --go-vtproto_opt=features=marshal+unmarshal+size+pool \
    

    Making sure to get latest go install github.com/planetscale/vtprotobuf/cmd/protoc-gen-go-vtproto@latest

    --go-vtproto_opt=features=marshal+unmarshal+size+pool+clone \
    

    Gives --go-vtproto_out: unknown feature: "clone"

  • Repeated field with nulls break pool

    Repeated field with nulls break pool

    It appears there is no null checks for repeated field with returning to the pool, causing panic. Is this not supported or am I missing how to handle properly?

  • New extension: scrub

    New extension: scrub

    Proposal: new extension called scrub which adds a function Scrub() to messages.

    Similar to Reset(), except it recursively overwrites all buffers & fields in the message with zeros.

  • Update README.md to add instructions to use with connect

    Update README.md to add instructions to use with connect

    Hello, So this adds instructions to add vtprotobuf to connect.

    connect-go uses the same interface as the grpc for codec and so it does work grpc.Codec{} but another alternative would be to copy the grpc directory to make a connect directory to decouple them.

    Let me know what you think!

  • Nil pointer panic marshalling nil oneof field

    Nil pointer panic marshalling nil oneof field

    Hi 👋 Ran into a bit of a weird one. Given the following proto:

    syntax = "proto3";
    
    package proto;
    
    option go_package = "github.com/pfouilloux/vttest/proto";
    
    message TestMsg {
      oneof Test {
        string a = 1;
        string b = 2;
        string c = 3;
      }
    }
    

    and the following code:

    package main_test
    
    import (
    	"testing"
    
    	"github.com/planetscale/vtprotobuf/codec/grpc"
    	_ "google.golang.org/grpc/encoding/proto"
    	"vttest/proto/github.com/pfouilloux/vttest/proto"
    )
    
    //go:generate protoc --proto_path=proto --go_out=proto --go_opt=paths=source_relative --go-vtproto_out=proto --go-vtproto_opt=features=marshal+unmarshal+size oneof.proto
    
    func TestMarshal(t *testing.T) {
    	test := &proto.TestMsg{Test: getA()}
    	_, err := grpc.Codec{}.Marshal(test)
    	if err != nil {
    		panic(err)
    	}
    }
    
    func getA() *proto.TestMsg_A {
    	return nil
    }
    

    I'm seeing the following error:

    panic: runtime error: invalid memory address or nil pointer dereference [recovered]
    	panic: runtime error: invalid memory address or nil pointer dereference
    [signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x1239f1d]
    
    goroutine 4 [running]:
    testing.tRunner.func1.2({0x126e880, 0x14b3f20})
    	/usr/local/opt/go/libexec/src/testing/testing.go:1396 +0x24e
    testing.tRunner.func1()
    	/usr/local/opt/go/libexec/src/testing/testing.go:1399 +0x39f
    panic({0x126e880, 0x14b3f20})
    	/usr/local/opt/go/libexec/src/runtime/panic.go:884 +0x212
    vttest/proto/github.com/pfouilloux/vttest/proto.(*TestMsg_A).MarshalToSizedBufferVT(0x126e580?, {0x14efc18?, 0xc000057601?, 0x0?})
    	/Users/pfouilloux/code/vttest/proto/github.com/pfouilloux/vttest/proto/oneof_vtproto.pb.go:72 +0x1d
    vttest/proto/github.com/pfouilloux/vttest/proto.(*TestMsg_A).MarshalToVT(0x1240e01?, {0x14efc18?, 0x0?, 0x123a5a4?})
    	/Users/pfouilloux/code/vttest/proto/github.com/pfouilloux/vttest/proto/oneof_vtproto.pb.go:68 +0x6a
    vttest/proto/github.com/pfouilloux/vttest/proto.(*TestMsg).MarshalToSizedBufferVT(0xc0001049c0?, {0x14efc18, 0x0, 0x0})
    	/Users/pfouilloux/code/vttest/proto/github.com/pfouilloux/vttest/proto/oneof_vtproto.pb.go:58 +0x133
    vttest/proto/github.com/pfouilloux/vttest/proto.(*TestMsg).MarshalVT(0xc0001049c0)
    	/Users/pfouilloux/code/vttest/proto/github.com/pfouilloux/vttest/proto/oneof_vtproto.pb.go:27 +0x58
    github.com/planetscale/vtprotobuf/codec/grpc.Codec.Marshal({}, {0x12a20c0, 0xc0001049c0})
    	/Users/pfouilloux/go/pkg/mod/github.com/planetscale/[email protected]/codec/grpc/grpc_codec.go:20 +0x42
    vttest_test.TestMarshal(0x0?)
    	/Users/pfouilloux/code/vttest/oneof_test.go:15 +0x47
    testing.tRunner(0xc0000076c0, 0x12c8b20)
    	/usr/local/opt/go/libexec/src/testing/testing.go:1446 +0x10b
    created by testing.(*T).Run
    	/usr/local/opt/go/libexec/src/testing/testing.go:1493 +0x35f
    
    
    Process finished with the exit code 1
    

    It looks like there is a nil check missing in the implementation of MarshalToVT for *TestMsg_A

    func (m *TestMsg_A) MarshalToVT(dAtA []byte) (int, error) {
    	size := m.SizeVT()
    	return m.MarshalToSizedBufferVT(dAtA[:size])
    }
    

    I'm more than happy to raise a PR to address this if you could give me some guidance on where to add the appropriate tests.

    Kind regards & thanks for sharing your work with the community!

protoCURL is cURL for Protobuf: The command-line tool for interacting with Protobuf over HTTP REST endpoints using human-readable text formats

protoCURL protoCURL is cURL for Protobuf: The command-line tool for interacting with Protobuf over HTTP REST endpoints using human-readable text forma

Jan 6, 2023
A new way of working with Protocol Buffers.

Buf All documentation is hosted at https://buf.build. Please head over there for more details. Goal Buf’s long-term goal is to enable schema-driven de

Jan 1, 2023
Protocol Buffers - Google's data interchange format

Protocol Buffers - Google's data interchange format Copyright 2008 Google Inc. https://developers.google.com/protocol-buffers/ Overview Protocol Buffe

Jan 3, 2023
A plugin of protoc that for using a service of Protocol Buffers as http.Handler definition

protoc-gen-gohttp protoc-gen-gohttp is a plugin of protoc that for using a service of Protocol Buffers as http.Handler definition. The generated inter

Dec 9, 2021
Go support for Protocol Buffers - Google's data interchange format

Go support for Protocol Buffers - Google's data interchange format Google's data interchange format. Copyright 2010 The Go Authors. https://github.com

Dec 15, 2021
Estudos com Golang, GRPC e Protocol Buffers

Golang, GRPC e Protocol Buffers Estudos com Golang, GRPC e Protocol Buffers Projeto feito para fins de estudos. Para rodar basta seguir os passos abai

Feb 10, 2022
This is a golang C2 + Implant that communicates via Protocol Buffers (aka. protobufs).

Br4vo6ix DISCLAIMER: This tool is for educational, competition, and training purposes only. I am in no way responsible for any abuse of this tool This

Nov 9, 2022
Compiler as a Service is a compiler that is available over http/https and gRPC

BlakBoks(CaaS) Elasticsearch but for compiling untrusted code Compiler as a Service is a compiler that is available over http/2 and gRPC. Setup First

Nov 24, 2021
Control your Flipper Zero over Protobuf RPC protocol.

go-flipper Control your Flipper Zero over Protobuf RPC protocol. This library is designed to be transport agnostic, though I've tested it with RPC ove

Dec 17, 2022
wire protocol for multiplexing connections or streams into a single connection, based on a subset of the SSH Connection Protocol

qmux qmux is a wire protocol for multiplexing connections or streams into a single connection. It is based on the SSH Connection Protocol, which is th

Dec 26, 2022
A simple tool to convert socket5 proxy protocol to http proxy protocol

Socket5 to HTTP 这是一个超简单的 Socket5 代理转换成 HTTP 代理的小工具。 如何安装? Golang 用户 # Required Go 1.17+ go install github.com/mritd/s2h@master Docker 用户 docker pull m

Jan 2, 2023
Assembly-optimized MD4 hash algorithm in Go

md4 MD4 hash algorithm in Go. Assembly-optimized for amd64 platforms. MD4 is cryptographically broken and should should only be used where compatibili

Apr 14, 2022
Schema-free, document-oriented streaming database that optimized for monitoring network traffic in real-time

Basenine Schema-free, document-oriented streaming database that optimized for monitoring network traffic in real-time. Featured Aspects Has the fastes

Nov 2, 2022
GoScan is a port-scanner made entirely in Go-lang. The purpose of the tool is to be fast, dynamic and simple so that a professional in the CyberSecurity area can make an optimized list of ports
GoScan is a port-scanner made entirely in Go-lang. The purpose of the tool is to be fast, dynamic and simple so that a professional in the CyberSecurity area can make an optimized list of ports

?? GoScan GoScan is a port-scanner made entirely in Go-lang. The purpose of the tool is to be fast, dynamic and simple so that a professional in the C

Jul 19, 2022
Protobuf files manager

Prot - protobuf files manager. It application can help your manage protobuf files and generate code based on him. !!! Before use Prot you must install

Jun 22, 2022
Generate types and service clients from protobuf definitions annotated with http rules.

protoc-gen-typescript-http Generates Typescript types and service clients from protobuf definitions annotated with http rules. The generated types fol

Nov 22, 2022
A simple RPC framework with protobuf service definitions

Twirp is a framework for service-to-service communication emphasizing simplicity and minimalism. It generates routing and serialization from API defin

Jan 7, 2023
protobuf ではなく JSON でやり取りするファイルを出力する protoc プラグイン

protoc-gen-jsonif proto ファイルから、JSON フォーマットでやりとりする型定義ファイルを出力する protoc プラグインです。 proto ファイルで言語を越えて型定義が出来るのはとても良い しかし protobuf ライブラリを入れるのが面倒 今のプロジェクトには既に

Feb 28, 2022
WIP protobuf support for Gleam ✨

gleam_pb WIP protobuf support for Gleam ✨ Progress Gleam Type generation custom functions that better handle default values stop including unnecessary

Feb 26, 2022