webrpc is a schema-driven approach to writing backend services for modern Web apps and networks

Hi, I've been evaluating this project and noticed that this kind of tag value doesn't parse anymore:

    - CreatedAt: timestamp
        + go.tag.pg = default**:**now**()**,use_zero

The code seems to be:

https://github.com/webrpc/webrpc/blob/3a0c9f0ee2f9eeda68b5fade4454c1a11791e94b/schema/ridl/parser.go#L342

and gives

parse error: "unexpected token" near ":" (line: 14, col: 30)

Goal: Allow WebRPC to parse schema from a Go source file instead of RIDL/JSON.

I'm seeking early feedback on this from @pkieltyka. I'll start adding test cases and examples in the upcoming days.

Based on work at https://github.com/vcilabs/webrpc/pull/2

This post might me more appropriate for a discussions tab in GitHub, but this project does not have one. My question is if the return type is configurable? Currently, it seems like all slices of structs/message are mapped to slices of pointers? Is this by design? Any way to create slices of structs instead?

save this one for v2, but some ideas below.

for v1, we will just use a json file like: https://github.com/webrpc/webrpc-go/blob/master/_example/proto/example.webrpc.json

implementing the schema at https://github.com/webrpc/webrpc-go/blob/master/schema/README.md

note, the data types of webrpc are inspired by protobuf's, Go, TypeScript and WASM :) intended to be serialized as JSON

Discovered an issue today when the webrpc handler has code that panic's or you manually add a panic("stop") the stack trace is wrong/broken and not helpful in debugging.

I tested today with one of your example apps, running latest webrpc generator.

With go1.16 or lower, the panic stack trace looks correct with:

 panic: stop
 
 -> main.(*ExampleServiceRPC).GetUser
 ->   /Users/johneberly/go/src/github.com/jeberly/webrpc-golang-basics/main.go:57

    main.(*exampleServiceServer).serveGetUserJSON.func1
      /Users/johneberly/go/src/github.com/jeberly/webrpc-golang-basics/example.gen.go:374
.... <snip>

With go1.17 or higher, panic stack trace seems broken:

2022/09/28 15:51:06 http: panic serving 127.0.0.1:50847: runtime error: slice bounds out of range [-1:]
goroutine 4 [running]:
net/http.(*conn).serve.func1(0x14000117d60)
	/Users/johneberly/sdk/go1.17/src/net/http/server.go:1801 +0xdc
panic({0x1027e35a0, 0x1400001a210})
	/Users/johneberly/sdk/go1.17/src/runtime/panic.go:1052 +0x2ac
github.com/go-chi/chi/middleware.prettyStack.decorateFuncCallLine({}, {0x1400019c321, 0x21}, 0x1, 0x8)
	/Users/johneberly/go/src/github.com/jeberly/webrpc-golang-basics/vendor/github.com/go-chi/chi/middleware/recoverer.go:130 +0x508
github.com/go-chi/chi/middleware.prettyStack.decorateLine({}, {0x1400019c321, 0x21}, 0x1, 0x8)
... <snip>

Steps to reproduce:

1. install and run this example https://github.com/webrpc/webrpc/tree/master/_examples/golang-basics
2. install go version 1.16 via these instructions https://go.dev/doc/manage-install
3. run go1.16 run . and then curl -X POST -H"Content-Type: application/json" -d '{"userID":1234}' http://localhost:4242/rpc/ExampleService/GetUser and verify 200 response
4. add panic("stop") to first line inGetUser` handler, call curl again, verify panic("stop") included in output
5. install go version 1.17 via these instructions https://go.dev/doc/manage-install
6. run go1.17 run . and then curl -X POST -H"Content-Type: application/json" -d '{"userID":1234}' http://localhost:4242/rpc/ExampleService/GetUser and verify incorrect panic stack trace with http: panic serving 127.0.0.1:50847: runtime error: slice bounds out of range [-1:]

Thanks in advance and let me know if I can test anything or I am just missing something.

First off, huge thanks to this library!! It looks like exactly what I was looking for after evaluating Twirp, et al. I love that it is simple, and right to the point with little dependencies and easy to grep the source. Keep up the good work.

Although RIDL is fairly self documenting (good job), has there been any thought into a doc generator with outputs eg. swagger, etc? Just interested in your thoughts/recommendations as I was thinking about potentially writing a generator to build docs we could publish for our API.

This ridl-code:

message AlertL
  - alerts: []Alert
    + go.field.type = []AlertDetails

service AppService
  - ListAlerts(limit: int) => (alerts: AlertL)

generates the following go-code:

type AlertL struct {
	Alerts []*AlertDetails `json:"alerts"`
}

I thought it was supposed to result in the the forced "go type" []AlertDetails, but it inserts an extra "*" here. The whole purpose of using the AlertLstructure was to avoid the "slice of pointers" problem, but that was unsuccessful.

I believe webrpc behaved different in v0.6 with -target=go

webrpc-gen -schema=./rpc/app.ridl -target=golang -pkg=rpc -server -out=./rpc/app.gen.go

If I have an rpc method like this...

  - Delete(id: string) => ()

... then the client always fails (after actually succeeding server-side) because the generated code looks like this:

      return buildResponse(res).then(_data => {
        return {
        }
      })

And buildResponse always tries parse the JSON body, which is empty.

An easy workaround is to just return something from the rpc call:

  - Delete(id: string) => (ok: boolean)

note: My server-side is Go and my client is TS.

I got this error on my program when using a struct as function parameter and then I saw the same situation in the golang-basics example.

You can reproduce the error by running the example:

cd _examples/golang-basics
go run .

And when you try to find a user, it issues a fatal error:

curl -v -X POST -H"Content-Type: application/json" -v -d '{"q":"123"}' http://localhost:4244/rpc/ExampleService/FindUser

I figured out that the error is in the serveFindUserJSON function on the generated Go file example.gen.go:

// line 350
reqContent := struct {
  Arg0 *SearchFilter `json:"s"`
}{}

// line 362
err = json.Unmarshal(reqBody, &reqContent)

It's trying to unmarshall Arg0 from the received json, that's why it's crashing.

To fix it I just added .Arg0 to reqContent:

err = json.Unmarshal(reqBody, &reqContent.Arg0)

(The crash itself is not caused by this error, it's due to the lack of proper checking on the received parameters at the FindUser function.)

As part of webrpc v2, I'd like to remove the templates inside of gen/ and instead have them as part of their own repos which would be fetched during the time of code-generation. The webrpc v2 cli would offer just the schema / RPC object tree, and a "gen" sub-package for the methods available for the templates.

Currently in v1 to generate webrpc code we call: webrpc-gen -schema=example.ridl -target=go -pkg=main -server -client -out=./example.gen.go

In v2, the -target=X flag can receive a uri (assuming https) to a git repo where it would look for client.tmpl and server.tmpl.

The command would be: webrpc-gen -schema=example.ridl -target=github.com/webrpc/gen-golang -pkg=main -server -client -out=./example.gen.go

As a short-hand, we will support -target=X as -target=golang, -target=typescript, etc. and it will by default search https://github.com/webrpc/gen-X for example: https://github.com/webrpc/gen-golang

The logic for parsing the target would be to check for "/", and if we have any "/"'s we assume this is a URI as otherwise it would just be a webrpc-default target which we'll just prepend the "github.com/webrpc/gen-" to the target and continue. Otherwise we will use the remote target.

This will be really cool because anyone can fork a generator and tweak it as they like, it it would be very easy for them to add new support for another language.

TODO

• [x] Add remote git support for -target flag
• [x] Pull/fetch git repos remotely, and also cache them locally somewhere, perhaps in ~/.webrpc/
• [x] Update webrpc/gen package to use a well-defined set of function maps used by the templates
• [x] Copy golang/typescript/javascript templates and push to respective new repos under github.com/webrpc/gen-XXX

presently rpc methods with no response will return a 0 byte body, which is annoying. it should either return an empty object, ie. {}, or we return always a response value..

ie.

- SomeMethod(in: string)

or

- SomeMethod(in: string) => ()

will cause a 0 byte response body..

for now, the fix is to always return a value.. like - SomeMethod(in: string) => (ok: bool) .. and perhaps we should enforce that, but another approach is to just return an empty body which is okay too I think

Introduced in Peter's "v2" branch

• https://github.com/webrpc/webrpc/tree/schema-v2
• https://github.com/webrpc/webrpc/pull/134
• https://github.com/webrpc/webrpc/pull/123

struct FlattenRequest
  - name: string
    + go.tag.db = name
  - amount?: Balance
    + go.tag.db = amount

struct FlattenResponse
  - id: uint64
    + go.field.name = ID
  - count: uint64
    + json = counter

service Another
  - Flatten(FlattenRequest) => (FlattenResponse)

Introduced in Peter's "v2" branch

• https://github.com/webrpc/webrpc/tree/schema-v2
• https://github.com/webrpc/webrpc/pull/134
• https://github.com/webrpc/webrpc/pull/123

alias Balance: string
  + go.field.type = BigInt
  + go.tag.db = balance
  + json = b_a_l_a_n_c_e
  
struct User
  - id: uint64
  - username: string
  - amount: Balance

failed to load templates from ../webrpc/gen-golang: template: struct.go.tmpl:23: function "trimPrefix" not defined

I thought we'd be able to require specific Template Functions API from within the templates & that's why we have introduced minVersion .WebrpcGenVersion "v0.7.0" function. However, it looks like Go templates look up all the functions up-front and fail early before even executing the template (no matter if the missing function would be executed or not).

We'll need to support this in some other way, so we have better & user-friendly errors.

int64 is too big for JavaScript's Number type (IEEE-754 double-precision floating point).

JavaScript looses precision on values higher than Number.MAX_SAFE_INTEGER (9007199254740991) and rounds them up. Technically, we could error out in JS runtime, if we notice values bigger than that if we decide to keep Number in JS/TS generators.

Current workarounds:

message User
  - id: string
    + go.field.type = int64
    + go.tag.json = id,string

message User
  - id: string
    + go.field.type = prototyp.BigInt # sequence prototyp

Proposal - switch to BigInt

I think it would make sense to represent int64 as BigInt in JavaScript and TypeScript generators. BigInt looks to be well supported.

I propose we remove the + json = {CUSTOM_JSON_FIELD_NAME} meta for webrpc v2.

JSON is the core of webrpc and thus I don't see why we'd need different field names in RIDL vs. in JSON.

1. Field names over the wire (JSON)

Instead of being able to "overwrite" RIDL field names with json meta, why don't we simply rely on the RIDL field name itself?

Example:

struct User
  - id: uint64
  - FIELD_name: string

would send

{ "user": {"id": 0, "FIELD_name": ""} }

2. Custom field names in TypeScript/JavaScript

• not supported (aka, same as with json meta)
• RIDL field name is what you see in JSON, and what you get in TS/JS

3. A special + json = - value

See https://github.com/webrpc/webrpc/issues/66 for a related bug report about + json = -

I believe this special value was added because of Go's json:"-" struct tag, which means "hide" this field in JSON. I think it could be replaced by some new option to "hide" server fields from the API clients.

Example:

struct User
  - id: uint64
  - FIELD_name: string
  - privateID: string
    + private = true

would still send

{ "user": {"id": 0, "FIELD_name": ""} }

since the privateID field was "hidden", it would not make it to the generated code for clients. However, the generated server code would still have the privateID field available (Go, Node.js or any other server).

new PR replacing https://github.com/webrpc/webrpc/tree/schema-v2

this is easier then rebasing, but its missing updates in _examples/ which we should copy over from schema-v2 once we update the generator