protoCURL
protoCURL is cURL for Protobuf: The command-line tool for interacting with Protobuf over HTTP REST endpoints using human-readable text formats.
Install
protocurl includes and uses a bundled protoc by default. It is recommended to install curl into PATH for configurable http requests. Otherwise protocurl will use a simple non-configurable fallback http implementation.
Native CLI
Archive
- Download the latest release archive for your platform from https://github.com/qaware/protocurl/releases
- Extract the archive into a folder, e.g.
/opt/protocurl. - Add symlink to the binary in the folder. e.g.
ln -s /opt/protocurl/bin/protocurl /usr/bin/protocurlOr add the binary folder/opt/protocurl/binto your system-wide path. - Test it via
protocurl -h
Debian .deb package
- Download the latest release
.debfor your architecture from https://github.com/qaware/protocurl/releases - Install dependency curl:
sudo apt install curl - Install
sudo dpkg -i <downloaded-release>.deb - Test it via
protocurl -h
Alpine .apk package
- Download the latest release
.apkfor your architecture from https://github.com/qaware/protocurl/releases - Install dependencies curl and gcompat:
sudo apk add curl gcompat - Install
sudo apk add --allow-untrusted <downloaded-release>.apk - Test it via
protocurl -h
Docker
Simply run docker run -v "/path/to/proto/files:/proto" qaware/protocurl <args>. See Quick Start for how to use.
Quick Start
After installing protocurl a request is as simple as:
protocurl -I test/proto -i ..HappyDayRequest -o ..HappyDayResponse \
-u http://localhost:8080/happy-day/verify -d "includeReason: true"
where
-I test/protopoints to the directory of protobuf files of your service- with docker one needs to instead mount the directory to
/protovia-v $PWD/test/proto:/proto
- with docker one needs to instead mount the directory to
-i ..HappyDayRequestand-o ..HappyDayResponseare Protobuf message types. The..makes protocurl infer their full package paths.-u http://localhost:8080/happy-day/verifyis the url to the HTTP REST endpoint accepting and returning binary protobuf payloads- with docker one may additionally need
--network host
- with docker one may additionally need
-d "includeReason: true"is the protobuf payload in Protobuf Text or JSON Format
Then protocurl will
- encode the textual Protobuf message to a binary request payload
- send the binary request to the HTTP REST endpoint (via
curl, if possible) and receive the binary response payload - decode the binary response payload back to text and display it
and produce the following output:
=========================== Request Text =========================== >>>
includeReason: true
=========================== Response Text =========================== <<<
isHappyDay: true
reason: "Thursday is a Happy Day! ⭐"
formattedDate: "Thu, 01 Jan 1970 00:00:00 GMT"
See below for usage notes and EXAMPLES.md for more information.
Usage and Example
See usage notes and EXAMPLES.md.
Protobuf JSON Format
protoCURL supports the Protobuf JSON Format. Note, that the JSON format is not a straightforward 1:1 mapping as it is in the case of the Protobuf Text Format (described below). For instance, the JSON mapping for timestamp.proto uses a human-readable string representation, whereas the payload itself and the text format use a representation with seconds and nanoseconds.
Protobuf Text Format
Aside from JSON, Protobuf primarily and natively supports a text format which represents the fields 1:1 like in the wire-format. For instance, repeated fields are condensed into an array in the JSON format - whereas they are simply ' repeated' without an array type in the text format.
This text format's syntax is hardly documented (protobuf issue), so this section will shortly describe how to write Protobuf messages in the text format.
Given the following .proto file
syntax = "proto3";
import "google/protobuf/timestamp.proto";
message HappyDayRequest {
google.protobuf.Timestamp date = 1;
bool includeReason = 2;
double myDouble = 3;
int64 myInt64 = 5;
repeated string myString = 6;
repeated NestedMessage messages = 9;
}
message NestedMessage {
Foo fooEnum = 1;
repeated int32 i = 4;
}
enum Foo {
BAR = 0;
BAZ = 1;
}
A HappyDayRequest message in text format might look like this:
includeReason: true,
myInt64: 123123123123,
myString: "hello world"
myString: 'single quotes are also possible'
myDouble: 123.456
messages: { fooEnum: BAR, i: 0, i: 1, i: 1337 },
messages: { i: 15, fooEnum: BAZ, i: -1337 },
messages: { },
date: { seconds: 123, nanos: 321 }
In summary:
- No encapsulating
{ ... }are used for the top level message (in contrast to JSON). - fields are comma separated and described via
<fieldname>: <value>.- Strictly speaking, the commas are optional and whitespace is sufficient
- repeated fields are simply repeated multiple times (instead of using an array) and they do not need to appear consecutively.
- nested messages are described with
{ ... }opening a new context and describing their fields recursively - scalar values are describes similar to JSON. Single and double quotes are both possible for strings.
- enum values are referenced by their name
- built-in messages (such as google.protobuf.Timestamp are described just like user-defined custom messages via
{ ... }and their message fields
This page shows more details on the text format.
Development
For development it is recommended to use the a bash-like Terminal either natively (Linux, Mac) or via MinGW on Windows.
About the CI/CD tests: TESTS.md
How to make a release: RELEASE.md
Setup
- As for script utilities, one needs
bash,jq,zip,unzipandcurl. - One also needs to download the protoc binaries for the local development via
release/10-get-protoc-binaries.sh.
For development the generated-local.Dockerfile (via generate-local.Dockerfile.sh) is used. To build the image simply run source test/suite/setup.sh and then buildProtocurl
Updating Docs after changes
Generate the main docs (.md files etc.) in bash/WSL via doc/generate-docs.sh <absolute-path-to-protocurl-repository>.
Once a pull request is ready, run this to generate updated docs.
Enhancements and Bugs
See issues.
FAQ
- How is protocurl different from grpccurl? grpccurl only works with gRPC services with corresponding endpoints. However, classic REST HTTP endpoints with binary Protobuf payloads are only possible with
protocurl. - Why is the use of a runtime curl recommended with protocurl? curl is a simple, flexible and mature command line tool to interact with HTTP endpoints. In principle, we could simply use the HTTP implementation provided by the host programming language (Go) - and this is what we do if no curl was found in the PATH. However, as more people use protocurl, they will request for more features - leading to a feature creep in such a 'simple' tool as protocurl. We would like to avoid implementing the plentiful features which are necessary for a proper HTTP CLI tool, because HTTP can be complex. Since is essentially what curl already does, we recommend using curl and all advanced features are only possible with curl.
- What are some nice features of protocurl?
- The implementation is well tested with end-2-end approval tests (see TESTS.md). All features are tested based on their effect on the behavior/output. Furthermore, there are also a few cross-platform native CI tests running on Windows and MacOS runners.
- The build and release process is optimised for minimal maintenance efforts. During release build, the latest versions of many dependencies are taken automatically (by looking up the release tags via the GitHub API).
- The documentation and examples are generated via scripts and enable one to update the examples automatically rather than manually. The consistency of the outputs of the code with the checked in documentation is further tested in CI.
