Type-driven code generation for Go

It seems that gen will not if the current package is not compilable. This caused me some confusion as I was refactoring some hand-duplicated code to use gen. I first tried it with one type and it worked. So then I deleted all my hand implementations and added the annotations to generate them.

But now gen would fail with what looks like a compile error that my types I wanted to generate were not found (because the rest of my original package depended on what I now wanted gen to generate for me). Confusing to be told that it is failing because Foo is not defined when gen is supposed to generate Foo.

Finally I figured out that gen must be parsing the Go code to read out the annotations and that was what was failing.

It'd be nice to get an error message that gen needs a compilable project before it can generate code instead of just getting what looks like a Go build error.

On the bright side, overall gen is easy to use and extend. Even with this I was able to refactor my hand-duplicated code to use gen (including a custom typewriter) in about an hour and a half.

This allows gen to generate code that is already being used in the original program. This is a first step towards fully automated partial generation as discussed in #46.

For example, gen can now create things_gen.go from this directly, without first commenting out the contents of main:

// thing.go
package main
// +gen
type Thing int
func main() {
    t := Things{2,1,3}
    x, _ := t.Min()
    println(x)
}

Output:

>gen
typecheck error: thing.go:6:7: undeclared name: Things
attempting to continue...
  Writing thing_gen.go
>go build
>

The previous code called types.Check, which calls types.Config.Check and explicitly returns a nil Package on any error. This change calls types.Config.Check directly which will return a partial result if possible. If there is an error, instead of completely bailing out, it still attempts to generate the templates. This should succeed as long as the type itself can be evaluated.

That a typecheck error was detected is still printed out just in case the error is unrelated to gen and the user should take action on it. This may not be necessary, and doesn't allow code that calls getTypes to handle it explicitly (instead of being printed out).

With a fresh checkout:

% go test
_gen_test.go:4:4: could not import github.com/clipperhouse/gen/typewriters/foowriter (can't find import: github.com/clipperhouse/gen/typewriters/foowriter)
exit status 1
--- FAIL: TestList (1.01 seconds)
    list_test.go:60: exit status 1
    list_test.go:65: standard list should output 4 lines, got 0
  Writing dummy_gen_test.go
_gen_test.go:4:4: could not import github.com/clipperhouse/gen/typewriters/foowriter (can't find import: github.com/clipperhouse/gen/typewriters/foowriter)
exit status 1
--- FAIL: TestRun (0.97 seconds)
    run_test.go:63: exit status 1
    run_test.go:68: open dummy_foo_test.go: no such file or directory
FAIL
exit status 1
FAIL    github.com/clipperhouse/gen 2.144s

Like import and var, the type declaration in go supports multiple declarations in a single statement like so:

// This comment is in GenDecl.Doc
type (
    // This comment in GenDecl.Spec[0].Doc
    Thing1 int

    // This comment in GenDecl.Spec[1].Doc
    Thing2 float
)

(Here it is, parsed with go/ast)

Should gen support such type declarations? If so, what should the semantics be?

My opinion on the semantics: Directives found in GenDecl.Doc applies to all TypeSpecs it contains. A directive found directly in a TypeSpec.Doc applies to that TypeSpec and overrides any GenDecl.Doc directive, if present.

Although the LINQ-like features are useful, I find myself more often desiring and building containers to hold my structs. container/list provides stack and queue-like functions, although I've always just done it myself because it's not too complicated and I'd prefer to not type assert. There is no set or hash container, so I use maps to achieve the same functionality. Would it make sense to have gen able to generate such containers? I'm opening this up as more of an idea discussion than a feature request.

The current gen API is designed to get you going with a minimum of effort. Simply by adding a // +gen directive above your type, you get a whole bunch of Linq-y (underscore-y) methods right out of the box.

While that’s a nice demo and first experience, I am thinking it’s not ideal.

First, it creates a bunch of methods you may never use, which is bloat.

Second, it’s not very Go-ish. I think we value explicitness over magic. You can subset them, // +gen methods:"Any,Count,Where", but…

If you also want projection methods, you use a special tag, indicating types, eg projections:"int,Foo". This tag then looks at the methods you’ve specified (or not) and basically multiplies the types and the methods.

Combinatorially, it can be a lot. Explaining (and implementing) the interaction of methods × types is messy.

I propose that all methods should be explicit (no magic) and combined into a single tag. It would look something like:

// +gen slice:"Count,Where,GroupBy<Foo>,SortBy<int>"

This is a breaking change, but perhaps better for the long term. Thoughts?

Nice project! :+1:

My package utils contains test files with the package name utils_test to test the public API.

File stringSlice.go contains e.g.:

// +gen slice:"Where,Count,GroupBy[int64]"
type Int64Slice []int64

Running then gen gives me the following error:

stringSlice_test.go:20:2: could not import github.com/project/csfw/utils (can't find import: github.com/project/csfw/utils)

Renaming util_test to utils and then running gen does work. But I don't want to do that.

Am I doing something wrong?

In my package github.com/pierrre/geohash I try to create a slice of Point (https://github.com/pierrre/geohash/blob/master/geohash.go#L113-L116).

But gen returns an error:

➜  geohash git:(master) ✗ gen
geohash_benchmark_test.go:6:18: could not import github.com/Codefor/geohash (can't find import: github.com/Codefor/geohash)
exit status 1

or

➜  geohash git:(master) ✗ gen
cstest_test.go:6:2: could not import github.com/pierrre/cstest (can't find import: github.com/pierrre/cstest)
exit status 1

I've been trying to toy with using this under Go 1.6 however no matter what I do I get some-file.go: ... could not import any/imported/package (Config.Importer not installed) from any gen command I run in any non-trivial package. (Even under the gen package itself!)

This is rather sad, as I was hoping to toy with this for some personal projects. I've tried looking through the issues of this and typewriter, and havn't found any comparable bugs, so I filed this.

See the typewriter branch for context.

On this branch, we split out a typewriter package which subsumes most of gen’s core functionality.

I think the name is good and the concept is clear, but it doesn’t need to be in a separate package, does it? I suggest we just bring all that functionality back into the main gen package.

The types and concept remain, they’ll just be gen.TypeWriter instead of typewriter.TypeWriter. gen will be an install and an import.

It’s one less thing to keep in sync, too.

Individual typewriters (“codecs”) remain their own packages and can live anywhere.

Thoughts?

Go lint will complain if the comments of func/methods doesn't follow the MethodName does something something, as defined in Effective Go:

http://golang.org/doc/effective_go.html#commentary

And enforced by the tool:

https://github.com/golang/lint/blob/master/lint.go#L485

// Add does the addition of a with b
func Add(a, b int) int {
    return a + b
}

I'm opening the issue so that:

1. If you care and want to do it, you can.
2. If you care but don't want to do it, I can.
3. If you don't care, then at least that will settle the question ^^.

# page doesn't actually exist and I don't see any Where language in your docs except for a single comment.
# What am I missing?

// Where returns a new MyTypeSlice whose elements return true for func. See: http://clipperhouse.github.io/gen/#Where

thanks!

Hi. I think I caught some type of runtime bug.

$ ./gen unexpected fault address 0x7f79d82f2000 fatal error: fault [signal SIGSEGV: segmentation violation code=0x2 addr=0x7f79d82f2000 pc=0x7f79d7eb4de0]

goroutine 1 [running]: runtime.dopanic_m /home/oceanfis81/workarea/llvm-project/llvm/tools/gollvm/gofrontend/libgo/go/runtime/panic.go:1210 runtime.fatalthrow /home/oceanfis81/workarea/llvm-project/llvm/tools/gollvm/gofrontend/libgo/go/runtime/panic.go:1071 runtime.throw /home/oceanfis81/workarea/llvm-project/llvm/tools/gollvm/gofrontend/libgo/go/runtime/panic.go:1042 runtime.sigpanic /home/oceanfis81/workarea/llvm-project/llvm/tools/gollvm/gofrontend/libgo/go/runtime/signal_unix.go:671 runtime.readUnaligned64 /home/oceanfis81/workarea/llvm-project/llvm/tools/gollvm/gofrontend/libgo/go/runtime/alg.go:516 runtime.memhash /home/oceanfis81/workarea/llvm-project/llvm/tools/gollvm/gofrontend/libgo/go/runtime/hash64.go:73 runtime.strhash /home/oceanfis81/workarea/llvm-project/llvm/tools/gollvm/gofrontend/libgo/go/runtime/alg.go:91 main.struct.4Name.0string.2Path.0string.5..hash /home/oceanfis81/go_projects/gen/:1 runtime.mapiternext /home/oceanfis81/workarea/llvm-project/llvm/tools/gollvm/gofrontend/libgo/go/runtime/map.go:974 runtime.mapiterinit /home/oceanfis81/workarea/llvm-project/llvm/tools/gollvm/gofrontend/libgo/go/runtime/map.go:896 github.x2ecom..z2fclipperhouse..z2ftypewriter.ImportSpecSet.ToSlice /home/oceanfis81/go/pkg/mod/github.com/clipperhouse/[email protected]/importspec_set.go:26 ffi_call_unix64 /home/oceanfis81/workarea/llvm-project/llvm/tools/gollvm/libgo/libffi/src/x86/unix64.S:106 ffi_call_int /home/oceanfis81/workarea/llvm-project/llvm/tools/gollvm/libgo/libffi/src/x86/ffi64.c:669 runtime.reflectcall /home/oceanfis81/workarea/llvm-project/llvm/tools/gollvm/gofrontend/libgo/runtime/go-reflect-call.c:226 reflect.Value.call /home/oceanfis81/workarea/llvm-project/llvm/tools/gollvm/gofrontend/libgo/go/reflect/value.go:486 reflect.Value.Call /home/oceanfis81/workarea/llvm-project/llvm/tools/gollvm/gofrontend/libgo/go/reflect/value.go:334 template.safeCall /home/oceanfis81/workarea/llvm-project/llvm/tools/gollvm/gofrontend/libgo/go/text/template/funcs.go:365 text..z2ftemplate.state.evalCall /home/oceanfis81/workarea/llvm-project/llvm/tools/gollvm/gofrontend/libgo/go/text/template/exec.go:722 text..z2ftemplate.state.evalField /home/oceanfis81/workarea/llvm-project/llvm/tools/gollvm/gofrontend/libgo/go/text/template/exec.go:606 text..z2ftemplate.state.evalFieldChain /home/oceanfis81/workarea/llvm-project/llvm/tools/gollvm/gofrontend/libgo/go/text/template/exec.go:567 text..z2ftemplate.state.evalFieldNode /home/oceanfis81/workarea/llvm-project/llvm/tools/gollvm/gofrontend/libgo/go/text/template/exec.go:531 text..z2ftemplate.state.evalCommand /home/oceanfis81/workarea/llvm-project/llvm/tools/gollvm/gofrontend/libgo/go/text/template/exec.go:459 text..z2ftemplate.state.evalPipeline /home/oceanfis81/workarea/llvm-project/llvm/tools/gollvm/gofrontend/libgo/go/text/template/exec.go:433 text..z2ftemplate.state.walkRange /home/oceanfis81/workarea/llvm-project/llvm/tools/gollvm/gofrontend/libgo/go/text/template/exec.go:340 text..z2ftemplate.state.walk /home/oceanfis81/workarea/llvm-project/llvm/tools/gollvm/gofrontend/libgo/go/text/template/exec.go:268 text..z2ftemplate.state.walk /home/oceanfis81/workarea/llvm-project/llvm/tools/gollvm/gofrontend/libgo/go/text/template/exec.go:265 text..z2ftemplate.state.walkIfOrWith /home/oceanfis81/workarea/llvm-project/llvm/tools/gollvm/gofrontend/libgo/go/text/template/exec.go:295 text..z2ftemplate.state.walk /home/oceanfis81/workarea/llvm-project/llvm/tools/gollvm/gofrontend/libgo/go/text/template/exec.go:262 text..z2ftemplate.state.walk /home/oceanfis81/workarea/llvm-project/llvm/tools/gollvm/gofrontend/libgo/go/text/template/exec.go:265 text..z2ftemplate.Template.execute /home/oceanfis81/workarea/llvm-project/llvm/tools/gollvm/gofrontend/libgo/go/text/template/exec.go:222 text..z2ftemplate.Template.Execute /home/oceanfis81/workarea/llvm-project/llvm/tools/gollvm/gofrontend/libgo/go/text/template/exec.go:205 main.executeCustom /home/oceanfis81/go_projects/gen/execute.go:63 main.execute /home/oceanfis81/go_projects/gen/execute.go:24 main.run /home/oceanfis81/go_projects/gen/run.go:19 main.runMain /home/oceanfis81/go_projects/gen/main.go:40 main.main /home/oceanfis81/go_projects/gen/main.go:22 runtime.main /home/oceanfis81/workarea/llvm-project/llvm/tools/gollvm/gofrontend/libgo/go/runtime/proc.go:236

$ go env && go version GO111MODULE="" GOARCH="amd64" GOBIN="" GOCACHE="/home/oceanfis81/.cache/go-build" GOENV="/home/oceanfis81/.config/go/env" GOEXE="" GOFLAGS="" GOHOSTARCH="amd64" GOHOSTOS="linux" GOINSECURE="" GOMODCACHE="/home/oceanfis81/go/pkg/mod" GONOPROXY="" GONOSUMDB="" GOOS="linux" GOPATH="/home/oceanfis81/go" GOPRIVATE="" GOPROXY="https://proxy.golang.org,direct" GOROOT="/home/oceanfish81/gollvm_dist" GOSUMDB="sum.golang.org" GOTMPDIR="" GOTOOLDIR="/home/oceanfish81/gollvm_dist/tools" GCCGO="/home/oceanfish81/gollvm_dist/bin/llvm-goc" AR="ar" CC="/usr/bin/clang" CXX="/usr/bin/clang++" CGO_ENABLED="1" GOMOD="/home/oceanfis81/go_projects/gen/go.mod" CGO_CFLAGS="-g -O2" CGO_CPPFLAGS="" CGO_CXXFLAGS="-g -O2" CGO_FFLAGS="-g -O2" CGO_LDFLAGS="-g -O2" PKG_CONFIG="pkg-config" GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build551695759=/tmp/go-build -gno-record-gcc-switches -funwind-tables" go version go1.15.2 gollvm LLVM 12.0.0git linux/amd64

CC @thanm @cherrymui

In an attempt to avoid emulating php, and instead being more like stringer...

Could we have the option to pass the structs we want as command line arguments?

This would probably help people with https://github.com/clipperhouse/gen/issues/99 sort of issues too

go get failed.

$ go get -v github.com/clipperhouse/gen
Fetching https://golang.org/x/tools/go/gcimporter15?go-get=1
Parsing meta tags from https://golang.org/x/tools/go/gcimporter15?go-get=1 (status code 200)
get "golang.org/x/tools/go/gcimporter15": found meta tag get.metaImport{Prefix:"golang.org/x/tools", VCS:"git", RepoRoot:"https://go.googlesource.com/tools"} at https://golang.org/x/tools/go/gcimporter15?go-get=1
get "golang.org/x/tools/go/gcimporter15": verifying non-authoritative meta tag
Fetching https://golang.org/x/tools?go-get=1
Parsing meta tags from https://golang.org/x/tools?go-get=1 (status code 200)
golang.org/x/tools (download)
package golang.org/x/tools/go/gcimporter15: cannot find package "golang.org/x/tools/go/gcimporter15" in any of:
        /usr/local/Cellar/go/1.10.1/libexec/src/golang.org/x/tools/go/gcimporter15 (from $GOROOT)
        /{{**masked**}}/src/golang.org/x/tools/go/gcimporter15 (from $GOPATH)

Package renamed by go/internal/gcimporter: rename from go/gcimporter15

Hey there,

Great work again. =) Just wanted to say it'd be sweet if the slice typewriter could support codegen'ing a PriorityQueue implementation too, via the Heap interface:

https://golang.org/pkg/container/heap/ (see PriorityQueue example)

Although if it's helpful, what I'm really looking to do is something like a "top N by <func>". An ideal implementation would use a heap/priority queue with capacity N, so discard anything ranked lower than N as I iterate over the slice and add things to this heap/priority queue.

Thanks for the consideration!