The use of modules in go-codec introduced more headaches than necessary.

It has been a disaster, and I really wish I had waited until the dust settled before making this module aware, especially since so many people depended on this package and it has the atypical "package name differs from directory name" issue.

I regret jumping on modules that early, and having to constantly try to correct my mistakes.

It seems the dust has settled now and we have the tools to:

• support the simulation of a multi-module repository

We have 2 options:

1. Make go/codec its own module using a tag: codec/vX.Y.Z
2. Keep go as the only module, using a tag: vX.Y.Z
3. Have both as modules using "same'ish" parallel tags (vX.Y.Z and codec/vX.Y.Z) and corresponding go.mod files

I am not sure if 3. above is supported by go modules. I read that a module doesn't include modules within it.

I will appreciate any thoughts on how best to resolve this. I am including references to places where I have seen folks have issues. I hope that you folks can respond and help me settle this once and for all with the best solution.

https://github.com/golang/go/wiki/Modules#publishing-a-release https://golang.org/wiki/Modules#faqs--multi-module-repositories

https://github.com/gin-gonic/gin/issues/1897

@dmitshur @arizvisa @bvisness @HZ89 @bcmills @amarox @thepudds

Here's a simplified repro:

package main

import (
        "bytes"
        "fmt"

        "github.com/ugorji/go/codec"
)

// We have defined Selfers for a type (either with codecgen or manually)
type Embedded struct {
        Embedded1 string
        Embedded2 string
}

func (e Embedded) CodecEncodeSelf(encoder *codec.Encoder) {
        // phon4y output, in a real example this would encode the struct properly
        encoder.Encode("['phony']")
}

func (e *Embedded) CodecDecodeSelf(decoder *codec.Decoder) {
}

// We use embedded in a different structure
type Foo struct {
        Bar string
        Embedded
}

func main() {
        foo := Foo{
                Bar: "bar",
                Embedded: Embedded{
                        Embedded1: "embedded1",
                        Embedded2: "embedded2",
                },
        }

        // The Decoder only encodes the Embedded struct
        // and ignores Bar!
        buffer := &bytes.Buffer{}
        codec.NewEncoder(buffer, &codec.JsonHandle{}).Encode(foo)

         fmt.Printf("Output: %s\n", buffer.String())
}

The program above outputs:

Output: "['phony']"

completely ignoring the Bar element.

I would expect something in the lines of:

Output: { Bar: "Bar", Embedded: "phony" }

Or, with a properly defined selfer:

Output: { Bar: "Bar", Embedded1: "embedded1", Embedded2: "embedded2"  }

I guess that, since the embedded Selfer makes the full struct a Selfer too, the encoder is blindly applying (e Embedded) CodecEncodeSelf(), ignoring other fields. I think the Encoder must somehow distinguish between the case in which the struct embeds a Selfer and when the struct itself implements a Selfer directly.

I haven't tried it yet it but, as a workaround, I guess one can define a Selfer for Foo.

I haven't tested the decoding, but maybe that's broken too.

It would be nice if the import path mapped directly to a URL to the source code.

Currently, it doesn't, and for no good reason. Initially, we used a repository name of go, with the hope of putting multiple packages into the same repository. However, in retrospect, that is not a good idea; one repository should only hold one package, with possible sub-packages which relate directly to that package.

The plan of action is:

• Use github to change repository name from go to go-codec
• Move all files from the codec sub-directory of the repository to the top-level directory
• Keep the codec subdirectory there, and create a codec/fail.go file

The new codec/fail.go file will contain

• package docs that tell users the new import path and where to find docs
• init method that panics with an appropriate error message
• possibly use the // import label to allow go get fail early

We hope to do this soon, likely by the end of january.

To reproduce I'm using: https://github.com/kubernetes/kubernetes/compare/master...wojtek-t:use_ugorij#diff-22207499f9ab0d60831b9c388397f034R1

Input data: https://github.com/kubernetes/kubernetes/compare/master...wojtek-t:use_ugorij#diff-29f2371dad30835b344a6ca05b473eaeR1

I did NOT run codecgen for the file that contains definition of that jsons: https://github.com/kubernetes/kubernetes/blob/master/plugin/pkg/scheduler/api/types.go

IIUC, this should result in falling back to encoding/json (which works fine). However this is not the case (i.e. the output is empty)

@ugorji

Formal releases are useful to define dependency relationships. 3rd party developers would be able to bundle somewhat "stable" version of your software instead of just top of "master". Also it is helpful for downstream maintainers for numerous reasons.

Please consider tagging. Thanks.

Here's my test:

n := 5

log.Print("encoding/json")
for i := 0; i < n; i++ {
  t := time.Now()
  var v interface{}
  for j := 0; j < 20; j++ {
    if err = json.Unmarshal(b, &v); err != nil {
      panic(err)
    }
  }
  log.Print(time.Now().Sub(t))
}

log.Print("go-codec")
for i := 0; i < n; i++ {
  t := time.Now()
  for j := 0; j < 20; j++ {
    var dec *codec.Decoder = codec.NewDecoderBytes(b, new(codec.JsonHandle))
    var v interface{}
    if err := dec.Decode(&v); err != nil {
      panic(err)
    }
  }
  log.Print(time.Now().Sub(t))
}

For a large, minified file with long keys (4.7MB), go-codec is ~ 31% slower than encoding/json:

encoding/json
1.617601942s
1.576523967s
1.610622356s
1.608594081s
1.647141938s
go-codec
2.129585437s
2.166288614s
2.1520013s
2.061147463s
2.102718168s

For a file containing the same values, but with every key shortened to just 2 characters (970KB), it's 80% slower:

encoding/json
474.933239ms
451.410297ms
452.822768ms
462.508272ms
456.032676ms
go-codec
852.508337ms
849.04073ms
830.897711ms
852.40816ms
826.848954ms

This is from byte arrays into interface{} values. Is there anything I can tweak to make go-codec be faster? Is this by design, somehow?

Go 1.5.1 on OS X.

When I was trying to use Ugorij for Kubernetes: https://github.com/kubernetes/kubernetes/blob/master/pkg/api/types.go I received the following error: "reflect.Value.Addr of unaddressable value"

How to reproduce it:

• I used the following code: https://github.com/kubernetes/kubernetes/compare/master...wojtek-t:use_ugorij#diff-a880abe356116a717967d2246a1ffc36R29

• This diff also contains a file that was generated for our structures.

  Please let me know if you need more information on reproducing it

The sample code below doesn't seem to work as it should:

package main

import (
	"bytes"
	"fmt"

	"github.com/ugorji/go/codec"
)

func main() {

	h := &codec.JsonHandle{}
	h.MapValueReset = true

	d := bytes.NewBuffer([]byte(`{"map": {"a": 1}}`))
	dec := codec.NewDecoder(d, h)

	o := struct {
		Map map[string]interface{} `json:"map"`
	}{}

	o.Map = map[string]interface{}{"b": 1}

	if err := dec.Decode(&o); err != nil {
		panic(err)
	}

	fmt.Println(o.Map)
}

This prints:

map[a:1 b:1]

Since I set MapValueReset = true, I would expect it to print:

map[a:1]

In this way, we can get the right tag like v1.1.3, instead of pseudo-version

A good example would be shown below

https://github.com/stretchr/testify/blob/master/go.mod https://github.com/andrewstuart/go-robinhood/blob/master/go.mod#L8

go-robinhood imports github.com/stretchr/testify/assert and got the right version

github.com/stretchr/testify v1.2.2

As far as I've managed to dig into the docs, registering an extension (cbor tag) requires you to also manually en/decode the data yourself. Both the InterfaceExt and Ext interfaces assume you implement the en/decoding.

I was wondering if there is a way to register a struct as an extension, and let the normal un/marshalling of the struct to take place for its payload.

eg.

type Something struct {
	Foo string `json:"foo"`
}

I'd like for example to be able to register Foo as extension 123 and have the handler marshal the tag, followed by a map with a string element with key foo.

Is there a way to do something simple like this in the current state of the lib that I am missing? If not, could point me to the right direction of what could be done to add it?

Thank you for an amazing library.

This is usability issue (not correctness one). I observed that when I'm using codecgen, the output produced for a given file (I was using: https://github.com/kubernetes/kubernetes/blob/master/pkg/api/types.go is non deterministic. Few things that change:

1. constants "codecSelferC*" has different names, e.g.:

codecSelferC_UTF85477         = 1

vs

 codecSelferC_UTF85165         = 1

1. ordering of generated function (and thus the names of variables they are using, because IIUC, you're using consecutive integers for variable names)

It would be very convenient if the output would be deterministic.

As of this commit: https://github.com/ugorji/go/commit/ebaaab4fb661237a0e4f376a751ae0eb26445b3d arrays with zero values ar treated as empty. Whether this is correct or not is up for debate (see https://github.com/golang/go/issues/29310), however it makes the current behaviour inconsistent with the standard json.Marshal() which makes it more difficult to use ugorji as a replacement.

I personally would argue that array of zero values is not empty. It's true that for Go arrays there is no distinction (meaning a Go array cannot be empty), however when doing un-typed unmarshalling the difference is between an undefined/missing value and an array of zeros.

go-codec currently supports

• Binary: Msgpack
• Binary: CBOR
• Text: JSON

For completeness, it should support XML also.

Though XML has fallen out-of-favor, it is still widely used. The encoding/xml in the standard library is not efficient, and does not conform to the simplicity used in encoding/json. We have a chance to add a performant XML encoder/decoder as just another format.

We are attempting this due to perceived issues with encoding/xml:

• Complicated. It tried to do too much, and is not as simple to use as json.
• Due to over-engineering, reflection is over-used AND performance suffers: java is 6X faster:http://fabsk.eu/blog/category/informatique/dev/golang/ even PYTHON performs better: http://outgoing.typepad.com/outgoing/2014/07/exploring-golang.html

codec framework will offer the following benefits

• VASTLY improved performance (when using reflection-mode or codecgen)
• simplicity and consistency: with the rest of the supported formats
• all other benefits of codec framework (streaming, codegeneration, etc)

go-codec is not a drop-in replacement for encoding/xml, as the tags and style will not transfer over. Having said that, it is a replacement, based on the simplicity and performance of codec.

The goal is for this to be like JAXB for Go.