Motivation

In some cultures, the pronunciation of kok may lead to misunderstanding. (See comments here and here.)

New name

The new name under consideration is kin, which means "Kin Is Not just a kit of Go kit".

Other related names:

• Module path
  • github.com/RussellLuo/kin
• Code generation tool
  • kingen
• Annotation directives
  • //kin:op
  • //kin:param
  • //kin:body
  • //kin:success
  • //kin:oas
  • //kin:alias
  • //kin:grpc
• Struct tags
  • kin:"..."

Tasks

• Rename the repo
• Update the code
• Update the docs

Hi @RussellLuo thanks creating kun, great toolkit! i'm running into a (minor) issue with the generated returnErr when generating an http client.

As an example, suppose I have this service with a method that contains an interface (i.e. Animal)

package service 

//go:generate kungen -force ./service.go Service
type Service interface {
	//kun:op GET /animal/{id}
	Find(id string) (Animal, error)
}

type Animal interface {
	IsAnimal()
}

After generating the code http_client.go will contain the following:

req, err := http.NewRequest("GET", u.String(), nil)
if err != nil {
	return Animal{}, err
}

returning Animal{} results in an InvalidLiteral error.

If you're open to contributions I created a PR: https://github.com/RussellLuo/kun/pull/29 Feel free to give feedback, or let me know if you'd like to see tests for contributions:)

Sometimes, a Go method needs to correspond to more than one URI (or HTTP method). For example:

type Service interface {
	//kun:op GET /messages/{messageID}
	//kun:op GET /users/{userID}/messages/{messageID}
	GetMessage(ctx context.Context, userID string, messageID string) (text string, err error)
}

This will enable the following two alternative HTTP to Go Method mappings:

HTTP | Go Method -- | -- GET /messages/123456 | GetMessage(ctx, "", "123456") GET /users/me/messages/123456 | GetMessage(ctx, "me", "123456")

This feature is much like (and is partially inspired by) gRPC's additional_bindings option (also see Multiple URI bindings).

This closes #25.

Sometimes, a Go method needs to correspond to more than one URI (or HTTP method). For example:

type Service interface {
	//kun:op GET /messages/{messageID}
	//kun:op GET /users/{userID}/messages/{messageID}
	GetMessage(ctx context.Context, userID string, messageID string) (text string, err error)
}

This will enable the following two alternative HTTP to Go-method mappings:

HTTP | Go Method -- | -- GET /messages/123456 | GetMessage(ctx, "", "123456") GET /users/me/messages/123456 | GetMessage(ctx, "me", "123456")

This feature is much like (and is partially inspired by) gRPC's additional_bindings option (also see Multiple URI bindings).

Syntax improvements

Original syntax (after #18):

• //kok:param <argName> [<parameter> [; <parameter2> [; ...]]]
  • parameter: in=<in> name=<name> required=<required> type=<type> descr=<descr>

New syntax:

• //kok:param <argName> [<parameter> [, <parameter2> [, ...]]]
  • parameter: in=<in> name=<name> required=<required> type=<type> descr=<descr>
• NOTE: If multiple method arguments are involved, you may need to apply multiple bindings. This can be done by adding a new //kok:param directive, or by appending the binding to the end of the last //kok:param directive in a semicolon-separated list.

Example

After the above syntax improvements, here are the equivalent annotations:

One binding per //kok:param

type Service interface {
    //kok:op POST /users
    //kok:param name
    //kok:param age in=query
    //kok:param ip in=header name=X-Forwarded-For, in=request name=RemoteAddr
    CreateUser(ctx context.Context, name string, age int, ip net.IP) (err error)
}

// HTTP request:
// $ http POST /users?name=tracey&age=1

Multiple bindings in a single //kok:param

type Service interface {
    //kok:op POST /users
    //kok:param name; age in=query; ip in=header name=X-Forwarded-For, in=request name=RemoteAddr
    CreateUser(ctx context.Context, name string, age int, ip net.IP) (err error)
}

// HTTP request:
// $ http POST /users?name=tracey&age=1

Multiple bindings in a single //kok:param (but using backslash for better readability)

type Service interface {
    //kok:op POST /users
    //kok:param name; \
    //          age in=query; \
    //          ip in=header name=X-Forwarded-For, in=request name=RemoteAddr
    CreateUser(ctx context.Context, name string, age int, ip net.IP) (err error)
}

// HTTP request:
// $ http POST /users?name=tracey&age=1

Motivation

1. Follow the Go conventions for comment directives (and this).
2. Provide a unified annotation syntax for both comments and struct tags.
3. Make it possible to apply Argument aggregation in a single-line annotation.
4. Improve the comment syntax for manipulating HTTP request/response body fields.

Proposed annotation syntax

HTTP

1. Define the HTTP operation

   • Directive: //kok:op

   • Arguments: <method> <pattern>

     • ...

   • Examples:

     type Service interface {
         //kok:op DELETE /users/{id}
         DeleteUser(ctx context.Context, id int) (err error)
     }
     
     // HTTP request:
     // $ http DELETE /users/101
     

2. Define the HTTP request parameters

   • Directive: //kok:param
   • Arguments: <argName> [<parameter> [; <parameter2> [; ...]]]
     • parameter: in=<in> name=<name> required=<required> type=<type> descr=<descr>
       • ...
   • Examples:

     • Bind request parameters to simple arguments:

       type Service interface {
           //kok:op PUT /users/{id}
           //kok:param name in=header name=X-User-Name
           UpdateUser(ctx context.Context, id int, name string) (err error)
       }
       
       // HTTP request:
       // $ http PUT /users/101 X-User-Name:tracey
       

     • Bind multiple request parameters to a struct according to tags:

       type User struct {
           ID   int    `kok:"in=path"`  // name defaults to snake case `id`
           Name string `kok:"in=query"` // name defaults to snake case `name`
           Age  int    `kok:"in=header name=X-User-Age"`
       }
       
       type Service interface {
           //kok:op PUT /users/{id}
           //kok:param user
           UpdateUser(ctx context.Context, user User) (err error)
       }
       
       // HTTP request:
       // $ http PUT /users/101?name=tracey X-User-Age:1
       

     • Bind multiple query parameters to a struct with no tags:

       type User struct {
           Name    string   // equivalent to `kok:"in=query name=name"`
           Age     int      // equivalent to `kok:"in=query name=age"`
           Hobbies []string // equivalent to `kok:"in=query name=hobbies"`
       }
       
       type Service interface {
           //kok:op POST /users
           //kok:param user
           CreateUser(ctx context.Context, user User) (err error)
       }
       
       // HTTP request:
       // $ http POST /users?name=tracey&age=1&hobbies=music&hobbies=sport
       

     • Argument aggregation:

       type Service interface {
           //kok:op POST /logs
           //kok:param ip in=header name=X-Forwarded-For; in=request name=RemoteAddr
           Log(ctx context.Context, ip net.IP) (err error)
       }
       
       // The equivalent annotations =>
       // (using backslash `\` for line continuation)
       
       type Service interface {
           //kok:op POST /logs
           //kok:param ip in=header name=X-Forwarded-For; \
           //             in=request name=RemoteAddr
           Log(ctx context.Context, ip net.IP) (err error)
       }
       
       // HTTP request:
       // $ http POST /logs
       

3. Define the HTTP request body

   • Directive: //kok:body
   • Arguments: <field> or <manipulation> [; <manipulation2> [; ...]]
     • manipulation: <argName> name=<name> type=<type> descr=<descr>
       • ...
   • Examples:

     • Omitted:

       type Service interface {
           //kok:op POST /users
           CreateUser(ctx context.Context, name string, age int) (err error)
       }
       
       // HTTP request:
       // $ http POST /users name=tracey age=1
       

     • Specified as a normal argument:

       type User struct {
           Name string `json:"name"`
           Age  int    `json:"age"`
       }
       
       type Service interface {
           //kok:op POST /users
           //kok:body user
           CreateUser(ctx context.Context, user User) (err error)
       }
       
       // HTTP request:
       // $ http POST /users name=tracey age=1
       

     • Specified as -:

       type User struct {
           Name    string
           Age     int
           Hobbies []string `kok:"name=hobby"`
       }
       
       type Service interface {
           //kok:op POST /users
           //kok:body -
           CreateUser(ctx context.Context, user User) (err error)
       }
       
       // HTTP request:
       // $ http POST /users?name=tracey&age=1&hobby=music&hobby=sport
       

4. Define the success HTTP response

   • Directive: //kok:success

   • Arguments: statusCode=<statusCode> body=<body> manip=`<manipulation> [; <manipulation2> [; ...]]`

     • manipulation: <argName> name=<name> type=<type> descr=<descr>
       • ...

   • Examples:

     type User struct {
         Name string `json:"name"`
         Age  int    `json:"age"`
     }
     
     type Service interface {
         //kok:op POST /users
         //kok:success statusCode=201 body=user
         CreateUser(ctx context.Context) (user User, err error)
     }
     

5. Define the OAS metadata

   • Directive: //kok:oas

   • Arguments: <property>=<value>

   • Examples:

     // This is the API documentation of User.
     //kok:oas docsPath=/api-docs
     //kok:oas title=User API
     //kok:oas version=1.0.0
     //kok:oas basePath=/v1
     //kok:oas tags=user
     type Service interface {
         //kok:op POST /users
         CreateUser(ctx context.Context, name string, age int) (err error)
     }
     

6. Define the annotation alias

   • Directive: //kok:alias

   • Arguments: <name>=`<value>`

   • Examples:

     type Service interface {
         //kok:op POST /users
         //kok:param operatorID in=header name=Authorization required=true
         CreateUser(ctx context.Context, operatorID int) (err error)
     
         //kok:op DELETE /users/{id}
         //kok:param operatorID in=header name=Authorization required=true
         DeleteUser(ctx context.Context, id, operatorID int) (err error)
     }
     
     // The equivalent annotations =>
     
     //kok:alias opID=`operatorID in=header name=Authorization required=true`
     type Service interface {
         //kok:op POST /users
         //kok:param $opID
         CreateUser(ctx context.Context, operatorID int) (err error)
     
         //kok:op DELETE /users/{id}
         //kok:param $opID
         DeleteUser(ctx context.Context, id, operatorID int) (err error)
     }
     

gRPC

• Directive: //kok:grpc
• Arguments: request=<request> response=<response>
• Examples:

  • Omitted:

    type Service interface {
        //kok:grpc
        CreateUser(ctx context.Context, name string, age int) (err error)
    }
    
    // gRPC request:
    // $ grpcurl -d '{"name": "tracey", "age": 1}' ... pb.Service/CreateUser
    

  • Specified:

    type User struct {
        Name string `json:"name"`
        Age  int    `json:"age"`
    }
    
    type Service interface {
        //kok:grpc request=user
        CreateUser(ctx context.Context, user User) (err error)
    }
    
    // gRPC request:
    // $ grpcurl -d '{"name": "tracey", "age": 1}' ... pb.Service/CreateUser
    

Since the underlying interface parser is borrowed from the old version of moq, kokgen has the same legacy issues as moq:

• https://github.com/matryer/moq/issues/68
• https://github.com/matryer/moq/issues/94

In order to address these issues, we must improve the interface parser by leveraging the new version of moq (thanks to https://github.com/matryer/moq/pull/141).

Since the core packages for interface parsing in moq are internal, we have to copy the code from moq instead.

Problem

Sometimes, an HTTP request parameter is only used in the Transport/Endpoint layer, and no further argument binding (in the Service layer) is needed.

Taking the following GetUser operation as an example:

type Service interface {
	// @kok(op): GET /users/{id}
	// @kok(success): body:user
	GetUser(ctx context.Context, id int) (user User, err error)
}

In order to protect the operation, we may implement service-to-service authentication via the Authorization header. Meanwhile, there is no corresponding argument in GetUser to be mapped to.

So how to represent this Authorization header in kok's HTTP annotation?

Proposed Solution

Add support for the blank identifier __ (double underscore, similar to Go's blank identifier) in @kok(param):

type Service interface {
	// @kok(op): GET /users/{id}
	// @kok(param): __ < in:header,name:Authorization,required:true
	// @kok(success): body:user
	GetUser(ctx context.Context, id int) (user User, err error)
}

And here is the possible codec.go for the Authorization header:

import (
	"fmt"

	"github.com/RussellLuo/kok/pkg/codec/httpcodec"
)

const (
	authPrivateKey = "AUTH_PRIVATE_KEY"
)

// AuthCodec is used to encode and decode the `Authorization` header. It can be reused wherever needed.
type AuthCodec struct{}

func (c AuthCodec) Decode(in []string, out interface{}) error {
	// NOTE: never use out, which is nil here.

	if len(in) == 0 || in[0] != authPrivateKey {
		return fmt.Errorf("authentication failed")
	}
	return nil
}

func (c AuthCodec) Encode(in interface{}) (out []string) {
	// NOTE: never use in, which is nil here.

	return []string{authPrivateKey}
}

func NewCodecs() *httpcodec.DefaultCodecs {
	// Use AuthCodec to encode and decode the argument named "__", if exists,
	// for the operation named "GetUser".
	return httpcodec.NewDefaultCodecs(nil,
		httpcodec.Op("GetUser", httpcodec.NewPatcher(httpcodec.JSON{}).Param("__", AuthCodec{})))
}

Multiple blank identifiers

In complicated cases, where multiple blank identifiers are involved and need to be differentiated from one another (typically for applying different encoders and decoders), we can use meaningful names, each with a double underscore prefix __ (e.g. __auth).

In summary, any name with a double underscore prefix __ is a valid blank identifier:

• __
• __auth
• __other

• [x] Generate the .proto file from the service definition in Go
• [x] Compile the .proto file to the gRPC definition
• [x] Adapt the gRPC definition to Go kit

Lengthy

type User interface {
    // @kok(op): POST /users
    // @kok(param): operatorID < in:header,name:Authorization,required:true
    Create(ctx context.Context, operatorID int) (err error)

    // @kok(op): GET /users/{id}
    // @kok(param): operatorID < in:header,name:Authorization,required:true
    Read(ctx context.Context, id, operatorID int) (err error)

    // @kok(op): PUT /users/{id}
    // @kok(param): operatorID < in:header,name:Authorization,required:true
    Update(ctx context.Context, id, operatorID int) (err error)

    // @kok(op): DELETE /users/{id}
    // @kok(param): operatorID < in:header,name:Authorization,required:true
    Delete(ctx context.Context, id, operatorID int) (err error)
}

Simplified

// @kok(alias): auth=`in:header,name:Authorization,required:true`
type User interface {
    // @kok(op): POST /users
    // @kok(param): operatorID < $auth
    Create(ctx context.Context, operatorID int) (err error)

    // @kok(op): GET /users/{id}
    // @kok(param): operatorID < $auth
    Read(ctx context.Context, id, operatorID int) (err error)

    // @kok(op): PUT /users/{id}
    // @kok(param): operatorID < $auth
    Update(ctx context.Context, id, operatorID int) (err error)

    // @kok(op): DELETE /users/{id}
    // @kok(param): operatorID < $auth
    Delete(ctx context.Context, id, operatorID int) (err error)
}

Or

// @kok(alias): opID=`operatorID < in:header,name:Authorization,required:true`
type User interface {
    // @kok(op): POST /users
    // @kok(param): $opID
    Create(ctx context.Context, operatorID int) (err error)

    // @kok(op): GET /users/{id}
    // @kok(param): $opID
    Read(ctx context.Context, id, operatorID int) (err error)

    // @kok(op): PUT /users/{id}
    // @kok(param): $opID
    Update(ctx context.Context, id, operatorID int) (err error)

    // @kok(op): DELETE /users/{id}
    // @kok(param): $opID
    Delete(ctx context.Context, id, operatorID int) (err error)
}

Take helloworld as an example, this is the original construction code:

https://github.com/RussellLuo/kok/blob/1a22aefc486c8250d788a4d36012af5b391425ed/examples/helloworld/cmd/main.go#L22

If we want to add request validation for the SayHello operation, we can change the above code as follows:

1. Validation rules written in Go

validators := []httpoption.NamedValidator{
    httpoption.Op("SayHello", httpoption.FuncValidator(func(value interface{}) error {
        req := value.(*helloworld.SayHelloRequest)
        if len(req.Name) > 10 {
            return werror.Wrapf(gcode.ErrInvalidArgument, "name: length exceeds 10")
        }

        reVarName := regexp.MustCompile(`^\w+$`)
        if !reVarName.MatchString(req.Name) {
            return werror.Wrapf(gcode.ErrInvalidArgument, "name: invalid name format")
        }

        return nil
    })),
}
r := helloworld.NewHTTPRouter(svc, httpcodec.NewDefaultCodecs(nil), httpoption.RequestValidators(validators...))

2. Validation rules written in DSL of validating

validators := []httpoption.NamedValidator{
    httpoption.Op("SayHello", helloworld.ValidateSayHelloRequest(func(req *helloworld.SayHelloRequest) v.Schema {
        return v.Schema{
            v.F("name", &req.Name): v.All(
                v.Len(0, 10).Msg("length exceeds 10"),
                v.Match(regexp.MustCompile(`^\w+$`)).Msg("invalid name format"),
            ),
        }
    })),
}
r := helloworld.NewHTTPRouter(svc, httpcodec.NewDefaultCodecs(nil), httpoption.RequestValidators(validators...))

NOTE:

• The ValidateSayHelloRequest() function is designed to be generated by kokgen
• The validation of each validator will be triggered in its corresponding DecodeRequestFunc (https://github.com/go-kit/kit/issues/908#issuecomment-526874228)

i really like the approach this framework takes of using annotations.

i would like to try to add async communication generators .

i use NATS quite a bit . It’s a message bus and broker .

https://github.com/nats-io/nats.go

It has HTTP / web socket and tcp transports built in.

So for iot people use the tcp layer . But for web they use the http and web sockets .

Nats is globally clustered , self load balancing and has build in security. It also has a clustered kv that replaced etc and consul. So it remove the need for a lot of other layers typically needed for global internet scale systems engineering .

It’s 100% golang and runs on anything .

It can do RPC and PUB SUB also.

You can use protobufs, json , anything you want serialisation . It does not care .

I don’t know how familiar you are with NATS . But just let me know if this is something if interest.

i currently use it in iot, web, desktop and servers .

—-

there is a standard approach to async communications that is emerging also called “async api”.

https://www.asyncapi.com/

It’s like openapi but for async.

it also has generators but like openapi takes the approach of you first writing your api description in json or yaml and then code generating from that .

it’s quite possible however to generate an async api document using kok !!!! This I think would be pretty cool because then others can generate the client in whatever language they want .

https://github.com/orgs/asyncapi/repositories?q=&type=&language=go&sort=