PowerProto
English | 中文
PowerProto is used to solve the following three main problems:
- lower the usage threshold and usage cost of gRPC.
- solve the version control problem of protoc and its related plugins (such as protoc-gen-go, protoc-gen-grpc-gateway).
- efficiently manage the compilation of proto to achieve multi-platform compatibility, one-click installation and compilation.
🎉
Features
- one-click installation and multi-version management of protoc.
- one-click installation and multi-version management of protoc related plugins (such as protoc-gen-go).
- manage the compilation of proto through config file instead of shell script to improve readability and compatibility.
- bootstrap generation of config files, cross-platform compatibility, a config can be compiled in multiple platforms with one click.
- support batch and recursive compilation of proto files to improve efficiency.
- cross-platform support PostAction, you can perform some routine operations (such as replacing "omitempty" in all generated files) after the compilation.
- support PostShell, execute specific shell scripts after the compilation.
- one-click installation and version control of google apis and gogo protobuf etc。
Installation and Dependencies
- The current version of
PowerProtorelies ongo(>=1.16) andgit(in the future it may use CDN to pull built binaries directly), please make sure the runtime environment contains these two commands. protocdownload source is Github,PowerProtorespectsHTTP_PROXY,HTTPS_PROXYenvironment variables when downloadingprotoc, if you encounter network problems, please configure your own proxy.- When querying the version list of
protoc,git ls-remoteis used forgithub.com, if you encounter network problems, please configure the proxy forgitby yourself. - In the current version, downloading and querying plugin versions rely on the
gocommand, so if you encounter network problems, please configure theGOPROXYenvironment variable yourself. - By default,
user directory/.powerprotois used as the installation directory, which is used to place the downloaded plug-ins and global config. - If you think the name
powerprotois too long, you canaliasit into a simpler name to improve the input efficiency, for example, no one will mind if you call itpp.
I. Installation via Go
Installation can be performed by executing the following command directly:
go install github.com/storyicon/powerproto/cmd/powerproto@latest
II. out-of-the-box version
You can download the out-of-the-box version via the Github Release Page
Command Introduction
You can view help with powerproto -h, e.g.
powerproto -h
powerproto init -h
powerproto tidy -h
powerproto build -h
powerproto env -h
It has the advantage that the documentation on the command line is always consistent with your binary version.
I. Initial Config
The config can be initialized with the following command.
powerproto init
II. Tidy Config
The config can be tidied with the following command.
powerproto tidy
It will search for a config file named powerproto.yaml from the current directory to the parent directory, and will read and tidy the config.
You can also specify which config file to tidy.
powerproto tidy [the path of proto file]
Tidy the config consists of two main operations:
- replacing the latest in the version with the real latest version number by querying.
- install all dependencies defined in the config file.
Supports entering debug mode by appending the -d argument to see more detailed logs.
III. Compiling Proto files
The Proto file can be compiled with the following command.
// Compile the specified proto file
powerproto build xxxx.proto
// Compile all the proto files in the current directory
powerproto build .
// Compile all proto files in the current directory recursively, including subfolders.
powerproto build -r .
The execution logic is that for each proto file, the powerproto.yaml config file will be searched from the directory where the proto file is located to the ancestor directory:
- For the found config file, match it with the
scopein it and use it if it matches. - Check and install the dependencies declared in the config file.
- Compile the proto file according to the
plugins,protoc,options,importPathsand other configs in the config file。 After all the proto files are compiled, if you specify the-pargument,PostActionandPostShellwill also be executed.
Note: The default working directory of PowerProto is the directory where the proto file matches to the config file, it is equivalent to the directory where you execute the protoc command. You can change it via protocWorkDir in the config file.
Supports entering debug mode by appending the -d argument to see more detailed logs.
Supports entering dryRun mode by appending the -y argument, in this mode the commands are not actually executed, but just printed out, which is very useful for debugging.
IV. View environment variables
If your command keeps getting stuck in a certain state, there is a high probability that there is a network problem.
You can check if the environment variables are configured successfully with the following command:
powerproto env
Examples
For example, you have the following file structure in the /mnt/data/hello directory:
$ pwd
/mnt/data/hello
$ tree
./apis
└── hello.proto
powerproto.yaml
The contents of the powerproto.yaml file (you can easily generate the config file with the powerproto init command) are:
scopes:
- ./
protoc: latest
protocWorkDir: ""
plugins:
protoc-gen-go: google.golang.org/protobuf/cmd/protoc-gen-go@latest
protoc-gen-go-grpc: google.golang.org/grpc/cmd/protoc-gen-go-grpc@latest
repositories:
GOOGLE_APIS: https://github.com/googleapis/googleapis@75e9812478607db997376ccea247dd6928f70f45
options:
- --go_out=.
- --go_opt=paths=source_relative
- --go-grpc_out=.
- --go-grpc_opt=paths=source_relative
importPaths:
- .
- $GOPATH
- $POWERPROTO_INCLUDE
- $GOOGLE_APIS/github.com/googleapis/googleapis
postActions: []
postShell: ""
Execute in any directory:
powerproto build -r /mnt/data/hello/apis
You can get the compiled file:
$ pwd
/mnt/data/hello
$ tree
./apis
├── hello.pb.go
├── hello.proto
└── hello_grpc.pb.go
powerproto.yaml
It is equivalent to if you were in the directory where powerproto.yaml is located and executed:
$POWERPROTO_HOME/protoc/3.17.3/protoc --go_out=. \
--go_opt=paths=source_relative \
--go-grpc_out=. \
--go-grpc_opt=paths=source_relative \
--proto_path=/mnt/data/hello \
--proto_path=$GOPATH \
--proto_path=$POWERPROTO_HOME/include \
--proto_path=$POWERPROTO_HOME/gits/75e9812478607db997376ccea247dd6928f70f45/github.com/googleapis/googleapis \
--plugin=protoc-gen-go=$POWERPROTO_HOME/plugins/google.golang.org/protobuf/cmd/[email protected]/protoc-gen-go \
--plugin=protoc-gen-go-grpc=$POWERPROTO_HOME/plugins/google.golang.org/grpc/cmd/[email protected]/protoc-gen-go-grpc \
/mnt/data/hello/apis/hello.proto
More examples can be found in examples.
Config File
The config file is used to describe the versions of various dependencies and parameters when compiling the proto file.
It can be easily initialized with powerproto init.
Definition
Take the following config file as an example:
# required. scopes is used to define scopes.
# i.e. which directories in the project the current config item is valid for
scopes:
- ./
# required. the version of protoc.
# you can fill in the 'latest', will be automatically converted to the latest version
protoc: 3.17.3
# optional. The working directory for executing the protoc command,
# the default is the directory where the config file is located.
# support mixed environment variables in path, such as $GOPATH
protocWorkDir: ""
# optional. define dependent Git repositories
# Generally used for dependency control of public protobuf libraries
repositories:
# Definition depends on the 27156597fdf4fb77004434d4409154a230dc9a32 version of https://github.com/googleapis/googleapis
# and defines its name as GOOGLE_APIS
# It can be referenced in importPaths by $GOOGLE_APIS
GOOGLE_APIS: https://github.com/googleapis/googleapis@27156597fdf4fb77004434d4409154a230dc9a32
# Definition depends on the 226206f39bd7276e88ec684ea0028c18ec2c91ae version of https://github.com/gogo/protobuf
# and defines its name as GOGO_PROTOBUF
# It can be referenced in the importPaths by $GOGO_PROTOBUF
GOGO_PROTOBUF: https://github.com/gogo/protobuf@226206f39bd7276e88ec684ea0028c18ec2c91ae
# required. it is used to describe which plug-ins are required for compilation
plugins:
# the name, path, and version number of the plugin.
# the address of the plugin must be in path@version format,
# and version can be filled with 'latest', which will be automatically converted to the latest version.
protoc-gen-deepcopy: istio.io/tools/cmd/protoc-gen-deepcopy@latest
protoc-gen-go: google.golang.org/protobuf/cmd/protoc-gen-go@latest
protoc-gen-go-json: github.com/mitchellh/[email protected]
protoc-gen-grpc-gateway: github.com/grpc-ecosystem/grpc-gateway/v2/[email protected]
# required. defines the parameters of protoc when compiling proto files
options:
- --go_out=paths=source_relative:.
- --go-json_out=.
- --deepcopy_out=source_relative:.
- --grpc-gateway_out=.
- --go-grpc_out=paths=source_relative:.
# required. defines the path of the proto dependency, which will be converted to the --proto_path (-I) parameter.
importPaths:
# Special variables. Will be replaced with the folder where the current configuration file is located.
- .
# Environment variables. Environment variables can be used in importPaths.
# Support mixed writing like $GOPATH/include
- $GOPATH
# Special variables. Will be replaced with the local path to the public proto file that comes with protoc by default
- $POWERPROTO_INCLUDE
# Special variables. Reference to the directory where the proto file to be compiled is located
# For example, if /a/b/data.proto is to be compiled, then the /a/b directory will be automatically referenced
- $SOURCE_RELATIVE
# References GOOGLE_APIS as defined in repositories
- $GOOGLE_APIS/github.com/googleapis/googleapis
# References GOGO_PROTOBUF as defined in repositories
- $GOGO_PROTOBUF
# optional. The operation is executed after compilation.
# its working directory is the directory where the config file is located.
# postActions is cross-platform compatible.
# Note that the "-p" parameter must be appended to the "powerproto build" to allow execution of the postActions in the config file
postActions: []
# optional. The shell script that is executed after compilation.
# its working directory is the directory where the config file is located.
# postShell is not cross-platform compatible.
# Note that the "-p" parameter must be appended to the "powerproto build" to allow execution of the postShell in the config file
postShell: |
// do something
Matching patterns and working directory
When building the proto file, the powerproto.yaml config file will be searched from the directory where the proto file is located to the ancestor directory, match with the scope in. The first matched config item will be used for the compilation of this proto file. When PowerProto executes protoc (and also when it executes postActions and postShell), the default is to use the directory where the config file is located as the working directory. (working directory is equivalent to the directory where you execute the protoc command.)
Multi-config
A config file can be filled with multiple configs, which are separated by "---".
In the example below, the apis1 directory uses protoc-gen-go with v1.25.0, while the apis2 directory uses protoc-gen-go with v1.27.0.
scopes:
- ./apis1
protoc: v3.17.3
protocWorkDir: ""
plugins:
protoc-gen-go: google.golang.org/protobuf/cmd/[email protected]
protoc-gen-go-grpc: google.golang.org/grpc/cmd/[email protected]
repositories:
GOOGLE_APIS: https://github.com/googleapis/googleapis@75e9812478607db997376ccea247dd6928f70f45
options:
- --go_out=.
- --go_opt=paths=source_relative
- --go-grpc_out=.
- --go-grpc_opt=paths=source_relative
importPaths:
- .
- $GOPATH
- $POWERPROTO_INCLUDE
- $GOOGLE_APIS/github.com/googleapis/googleapis
postActions: []
postShell: ""
---
scopes:
- ./apis2
protoc: v3.17.3
protocWorkDir: ""
plugins:
protoc-gen-go: google.golang.org/protobuf/cmd/[email protected]
protoc-gen-go-grpc: google.golang.org/grpc/cmd/[email protected]
repositories:
GOOGLE_APIS: https://github.com/googleapis/googleapis@75e9812478607db997376ccea247dd6928f70f45
options:
- --go_out=.
- --go_opt=paths=source_relative
- --go-grpc_out=.
- --go-grpc_opt=paths=source_relative
importPaths:
- .
- $GOPATH
- $POWERPROTO_INCLUDE
- $GOOGLE_APIS/github.com/googleapis/googleapis
postActions: []
postShell: ""
PostAction
PostAction allows to perform specific actions after all proto files have been compiled. In contrast to PostShell, it is cross-platform supported. For security reasons, the PostAction and PostShell defined in the config file will only be executed if the -p argument is appended to the execution of powerproto build.
Currently, PostAction supports the following commands:
| Command | Description | Function Prototype |
|---|---|---|
| copy | Copy file or folder | copy(src string, dest string) error |
| move | Move file or folder | move(src string, dest string) error |
| remove | Delete file or folder | remove(path ...string) error |
| replace | Batch replacement of strings in files | replace(pattern string, from string, to string) error |
1. copy
For copying files or folders, the function prototype is:
copy(src string, dest string) error
For security and config compatibility, only relative paths are allowed in the parameters.
If the target folder already exists, it will be merged.
The following example will copy 'a' from the directory where the config file is located to 'b'.
postActions:
- name: copy
args:
- ./a
- ./b
2. move
For moving files or folders, the function prototype is:
move(src string, dest string) error
For security and config compatibility, only relative paths are allowed in the parameters.
If the target folder already exists, it will be merged.
The following example will move 'a' in the directory where the config file is located to 'b'.
postActions:
- name: move
args:
- ./a
- ./b
3. remove
For deleting files or folders, the function prototype is:
remove(path ...string) error
For security and config compatibility, only relative paths are allowed in the parameters.
The following example will remove 'a', 'b' and 'c' from the directory where the config file is located:
postActions:
- name: remove
args:
- ./a
- ./b
- ./c
4. replace
Used for batch replacement of strings in files. Its function prototype is:
replace(pattern string, from string, to string) error
patternis a relative path that supports wildcard characters.fromis the string to be replaced.tois the string to replace with.
The following example will replace ,omitempty with the empty string in all go files in the apis directory and its subdirectories:
postActions:
- name: replace
args:
- ./apis/**/*.go
- ',omitempty'
- ""
