Complete the options handling in the compiler for all Options types, all options value types and for complex nested options. In the process, remove the special handling of all the known option fields since the method for parsing extension fields of options also applies to known fields.

This parses a superset of options compared to what protoc handles. protoc does not allow the same option to be specified multiple times, even if that comes from first specifying a nested field of a message-based option value and then another specifying the whole message. The code here does parse that and just merges all the specified options. Buyer beware.

The PR is split into separate commits which should make it easier to understand the separate bits of functionality.

make lint is currently failing because the current version (1.43.0) is incompatible with Go 1.18 (latest). We'll either need to go back to 1.17 or wait for a new lint.

Rework compiler package for imports. Create new function

compiler.NewFileDescriptorSet(filenames, importPaths, includeImports)

and load .proto dependencies from file.

Apply a bunch of fixes so that all the conformance tests pass, and remove the code that skips the listed ones.

• Extend the current patch (and rename) to turn off message_set_wire_format in all places, enabling the tests that were skipped purely because of that.
• Fix handling of groups in oneofs.
• Fix unquoting in the parser.
• Fix formatting of field default values.
• Specially handle the json_name field option.

Fixes: https://github.com/alecthomas/protobuf/issues/38

A group is both a message and a field. Handle options on the field portion of the group. The message options within the group braces are already handled.

Fixes: https://github.com/alecthomas/protobuf/issues/35

The syntax for extension ranges is:

  extensions a to max [options];

or

  extensions a, b to c, d to max [options];

When there are comma separated ranges, the options apply to all the ranges, not just the last one. The current parse tree attaches the options to the last of the range rather than all the ranges.

Also the parser.Range struct includes the Options, which is also used for reserved ranges. These do not have options.

I think the current code:

type Extensions struct {
        Pos lexer.Position
        Extensions []*Range `"extensions" @@ { "," @@ }`
}

type Range struct {
        Start   int     `@Int`
        End     *int    `  [ "to" ( @Int`
        Max     bool    `           | @"max" ) ]`
        Options Options `[ "[" @@ { "," @@ } "]" ]`
}

should be

type Extensions struct {
        Pos lexer.Position
        Extensions []*Range `"extensions" @@ { "," @@ }`
        Options Options `[ "[" @@ { "," @@ } "]" ]`
}

type Range struct {
        Start   int     `@Int`
        End     *int    `  [ "to" ( @Int`
        Max     bool    `           | @"max" ) ]`
}

Parse trailing comments after the last field in a prototext option. This is useful for commenting out the last field of a prototext option, which is a time-honoured debugging tradition. It must be supported.

Add map<key,value> to compiler which again required slightly bigger surgery as it is represented in protos as follows:

For maps fields:
    map<KeyType, ValueType> map_field = 1;
The parsed descriptor looks like:
    message MapFieldEntry {
        option map_entry = true;
        optional KeyType key = 1;
        optional ValueType value = 2;
    }
    repeated MapFieldEntry map_field = 1;

Add proto2 group to FileDescriptor in compiler package. This was bigger surgery than expected as groups can nest other messages, groups or enums. Additionally when groups are referenced as types they are classified as Messages. I've tried to cover most cases in the golden tests.

This PR is part 1 of many to get protobuf to generate FileDescriptor output.

This PR deals with: scalar types, packages, Message definition and all kinds of nesting.

It does not yet deal with imports, enums, extensions, services, options,.... 😅

This fails to parse

optional bytes escaped_bytes = 1 [default = "\0\001\a\b\f\n\r\t\v\\\'\"\xfe"];

with error

testdata/conformance/unittest.proto:619:47: invalid quoted string "\"\\0\\001\\a\\b\\f\\n\\r\\t\\v\\\\\\'\\\"\\xfe\"": invalid syntax

We will need to write a custom unquoter to fix this

When looking up the Registry for extensions and messages, first look in protoreflect.GlobalTypes so that concrete types are used where they are present in the program binary. This helps when the compiler is used as library in another program that has some concrete types built in (i.e. it has linked with generated protobuf code).

This issue has become apparent when working github.com/foxygoat/jig directly with proto files. The dynamically generated FileDescriptorSet could not be type asserted against *annotations.HttpRule (source).

Initial performance tests have shown that single proto files with big dependency try (~2000 files) perform as was as protoc (seconds). However when compiling all these files individually, e.g. as

protoc --include_imports -o all-protoc.pb -I . $(find . -name '*.proto')
protobuf --include-imports -o all-protobuf.pb -I . $(find . -name '*.proto')

protoc performs significantly faster (still seconds, whereas protobuf takes 10s of seconds).

The cause for this may be found with how we build up ASTs in the compiler (possible duplicates!): https://github.com/alecthomas/protobuf/blob/master/compiler/compiler.go#L480

The compiler has a global types map that it populates with fully-qualified type names as it walks the parse tree. A subsequent pass then looks up this global map to resolve relative names to full names and their types. This means a file will see all types in the entire import hierarchy as valid. However the only types available to a .proto file are those defined in that file, those in any files it imports and those in any public imports of our imports.

The types map should not be global, but per-file instead, tracking the types each file provides (either by defining them or publicly importing them) and looking up that and the type maps of the imports of a file.