Go library for decoding generic map values into native Go structures and vice versa.

mapstructure Godoc

mapstructure is a Go library for decoding generic map values to structures and vice versa, while providing helpful error handling.

This library is most useful when decoding values from some data stream (JSON, Gob, etc.) where you don't quite know the structure of the underlying data until you read a part of it. You can therefore read a map[string]interface{} and use this library to decode it into the proper underlying native Go structure.

Installation

Standard go get:

$ go get github.com/mitchellh/mapstructure

Usage & Example

For usage and examples see the Godoc.

The Decode function has examples associated with it there.

But Why?!

Go offers fantastic standard libraries for decoding formats such as JSON. The standard method is to have a struct pre-created, and populate that struct from the bytes of the encoded format. This is great, but the problem is if you have configuration or an encoding that changes slightly depending on specific fields. For example, consider this JSON:

{
  "type": "person",
  "name": "Mitchell"
}

Perhaps we can't populate a specific structure without first reading the "type" field from the JSON. We could always do two passes over the decoding of the JSON (reading the "type" first, and the rest later). However, it is much simpler to just decode this into a map[string]interface{} structure, read the "type" key, then use something like this library to decode it into the proper structure.

Comments
  • add support for 'required' tag value

    add support for 'required' tag value

    Implementation for checking that required fields got filled in map -> struct decoding direction.

    A basic test has been added as well.

    type Required struct {
    	RequiredBar string `mapstructure:"bar,required"`
    	Value       string `mapstructure:"foo"`
    }
    

    Fixes: #7

  • Create a tag

    Create a tag

    Can a tag be created of this so that dependent packages can pull a specific version rather than having to use current sources? That would be much appreciated!

    While this shows a 0.1, I made that version up. You can see in the ebuild it pulls the master.zip file rather than a tagged version.

    https://github.com/Obsidian-StudiosInc/os-xtoo/blob/master/dev-go/mapstructure/mapstructure-0.1.ebuild

  • time.Time encoded as map[string]interface{}{}

    time.Time encoded as map[string]interface{}{}

    Noticed that if encoding a struct into a map, time.Time entries are passed into empty maps, which doesn't seam very useful.

    I could add a hook to encode it into a string instance, but because hooks run as pre-processors, it's not possible to preserve it as a time.Time.

    Example code:

    package main
    
    import (
    	"fmt"
    	"time"
    
    	"github.com/mitchellh/mapstructure"
    )
    
    func PayloadEncoder(target *map[string]interface{}) (*mapstructure.Decoder, error) {
    	return mapstructure.NewDecoder(&mapstructure.DecoderConfig{
    		//DecodeHook: mapstructure.ComposeDecodeHookFunc(
    		// <- no hook can be added here to solve this.
    		//),
    		Result: target,
    	})
    }
    
    type MyStruct struct {
    	CreatedAt time.Time
    }
    
    func main() {
    	p1 := MyStruct{
    		CreatedAt: time.Now(),
    	}
    	m1 := make(map[string]interface{})
    	d1, err := PayloadEncoder(&m1)
    	fmt.Println("\nerr:", err, "\ndec:", d1, "\nmap:", m1)
    	err2 := d1.Decode(p1)
    	fmt.Printf("\nerr: %v\noutput: %#v\n", err2, m1)
    }
    

    Output:

    
    err: <nil> 
    dec: &{0xc0001be000} 
    map: map[]
    
    err: <nil>
    output: map[string]interface {}{"CreatedAt":map[string]interface {}{}}
    
  • Decode to int slice but input is string slice

    Decode to int slice but input is string slice

    I have specific case that my struct have []int32 , but values for it goes from url via fields=xxx&fields=yyy so input map have map['fields']=[]string{'xxx','yyy'}

    to decode it i need to get map with names and ints. How ca i do that via this package? P.S. Now i have handmade stuff that uses reflection and get values via proto.EnumValueMap via passing to it slice element type name

  • Feature: Decode structs to maps

    Feature: Decode structs to maps

    This feature adds the ability to decode structs to maps (essentially allowing the converse of the conventional use of this library). I've tried to be as complete in the test cases that I could add.

    One thing this doesn't do (yet) is map structs recursively. I'm not sure if this library should even do that, the containers for those inner structs can't really be supplied elegantly. If this was to be done - and i'd be happy to do it if that's the case, all structs would be initialized and decoded to map[string]interface{}. What do you think @mitchellh?

    Closes #53

  • Accept string to time.Time as weak input

    Accept string to time.Time as weak input

    It would be nice if the import can handle strings with a timestamp or a date time format as string. Something like that: "2015-09-30T01:18:56.096224525+02:00"

  • No encoding ?

    No encoding ?

    mapstructure is a Go library for decoding generic map values to structures and vice versa

    At first glance, there is no way to encode a struct to a map with this library ..?

  • failed to decode config

    failed to decode config

    Use below code to reproduce:

    package main
    
    import (
    	"fmt"
    	"github.com/mitchellh/mapstructure"
    )
    
    type Config struct {
    	ClientConfig  clientConfig   `yaml:"client,omitempty"`
    	ClientConfigs []clientConfig `yaml:"clients,omitempty"`
    }
    
    type clientConfig struct {
    	URL string `"yaml:url,omitempty"`
    }
    
    func main() {
    
    	input := map[string]interface{}{
    		"client": map[string]string{
    			"url": "http://198.168.181.61:3100/api/prom/push",
    		},
    	}
    
    	var result Config
    	err := mapstructure.Decode(input, &result)
    	if err != nil {
    		panic(err)
    	}
    
    	fmt.Printf("%#v", result)
    
    }
    

    output:

    main.Config{ClientConfig:main.clientConfig{URL:""}, ClientConfigs:[]main.clientConfig(nil)}
    
  • Would you be open to support array types?

    Would you be open to support array types?

    I have added in a case to handle array type because I need to deal with uuid.UUID type

    	var err error
    	dataKind := getKind(val)
    	fmt.Println(name, val, dataKind)
    	switch dataKind {
    	case reflect.Bool:
    		err = d.decodeBool(name, data, val)
    	case reflect.Interface:
    		err = d.decodeBasic(name, data, val)
    	case reflect.String:
    		err = d.decodeString(name, data, val)
    	case reflect.Int:
    		err = d.decodeInt(name, data, val)
    	case reflect.Uint:
    		err = d.decodeUint(name, data, val)
    	case reflect.Float32:
    		err = d.decodeFloat(name, data, val)
    	case reflect.Struct:
    		err = d.decodeStruct(name, data, val)
    	case reflect.Map:
    		err = d.decodeMap(name, data, val)
    	case reflect.Ptr:
    		err = d.decodePtr(name, data, val)
    	case reflect.Slice:
    		err = d.decodeSlice(name, data, val)
    	case reflect.Array: // required to handle uuid.UUID type
    		err = d.decodeArray(name, data, val)
    	default:
    		// If we reached this point then we weren't able to decode it
    		return fmt.Errorf("%s: unsupported type: %s", name, dataKind)
    	}
    

    Would you accept this new feature?

  • Support struct wrapped in interface as target for decoding

    Support struct wrapped in interface as target for decoding

    Similar to #186, mapstructure is not handling structs assigned to interfaces:

    var cc interface{}
    cc = struct{ Foo, Bar string }{}
    
    decoderConfig := &mapstructure.DecoderConfig{Result: &cc, ErrorUnused: true}
    decoder, err := mapstructure.NewDecoder(decoderConfig)
    if err != nil {
    	log.Fatal(err)
    }
    
    conf := map[string]string{
    	"foo": "bar",
    	"bar": "baz",
    }
    
    if err := decoder.Decode(conf); err != nil {
    	log.Fatal(err)
    }
    

    Will give

    * '' has invalid keys: bar, foo
    

    Would this be something worthwile to address with a PR or are there constraints that would prevent handling this case?

  • Fix data race in decodeStruct

    Fix data race in decodeStruct

    Using *reflect.StructField as map key results in a data race. I honestly do not understand why yet and can only reproduce this in the consul codebase with high concurrency but this patch fixes it.

    The same problem also exists in hashicorp/hcl.

    https://github.com/hashicorp/hcl/pull/213

  • Tag rename missing is encoding

    Tag rename missing is encoding

    If I'm not mistaken, the main.go below should output:

    sub_config:
        first_children:
            - name: parent
              grand_children:
                - name: foo
                - name: bar
    

    Instead, it is outputting:

    sub_config:
        first_children:
            - name: parent
              children:
                - name: foo
                - name: bar
    

    Notice that the second slice is intended to be named grand_children but is really being named children as it is actually named in the struct.

    If I step through the mapstructure.Decode function, I can see the tag get read and build an appropriate fieldName as sub_config.first_children[0].grand_children, but that at some point gets shuffled back to children.

    We can see in this screenshot the result of the Decode prior to rendering the yaml: image

    main.go
    package main
    
    import (
    	"fmt"
    	"os"
    
    	"github.com/mitchellh/mapstructure"
    	"gopkg.in/yaml.v3"
    )
    
    type Config struct {
    	Subconfig SubSection `mapstructure:"sub_config"`
    }
    
    type SubSection struct {
    	Children []Child `mapstructure:"first_children"`
    }
    
    type Child struct {
    	Name     string
    	Children []Grandchild `mapstructure:"grand_children"`
    }
    
    type Grandchild struct {
    	Name string
    }
    
    func main() {
    	config := &Config{
    		Subconfig: SubSection{
    			Children: []Child{
    				{Name: "parent", Children: []Grandchild{
    					{Name: "foo"},
    					{Name: "bar"},
    				}},
    			},
    		},
    	}
    
    	encodedData := map[string]interface{}{}
    	err := mapstructure.Decode(config, &encodedData)
    	if err != nil {
    		err = fmt.Errorf("unable to encode configuration: %w", err)
    		fmt.Println(err)
    		os.Exit(1)
    	}
    
    	data, err := yaml.Marshal(encodedData)
    	if err != nil {
    		err = fmt.Errorf("unable to marshal to yaml: %w", err)
    		fmt.Println(err)
    		os.Exit(1)
    	}
    
    	fmt.Println(string(data))
    }
    
    go.mod
    module foo
    
    go 1.19
    
    require (
    	github.com/mitchellh/mapstructure v1.5.0 // indirect
    	gopkg.in/yaml.v3 v3.0.1 // indirect
    )
    
  • Unable to unmarshal custom type hcl data

    Unable to unmarshal custom type hcl data

    Struct format

    type (
    	Specification struct {
    		Environment
    		Templates
    	}
    
    	Environment struct {
    		Build   image.Image
    		Runtime image.Image
    	}
    
    	Templates struct {
    		ArtifactPathTemplate          string                      `mapstructure:"artifact-path"`
    		StartCommandTemplate          *executable.TemplateCommand `mapstructure:"start-command"`
    		ComponentInstallationTemplate *executable.TemplateCommand `mapstructure:"component-install"`
    		ExecutableBinary              string                      `mapstructure:"executable-binary"`
    	}
    )
    
    

    getting error error decoding 'Specification': unsupported entry[type= []map[string]interface {}, value= ] Anyone can help how to decode these value

  • Add decode hooks for netip Addr and AddrPort

    Add decode hooks for netip Addr and AddrPort

    Technically this package still supports Go 1.14 which is why I decided to add a guard for the new functions. I can easily get rid of it, but I'd recommend bumping the Go version in the go.mod file in that case.

  • Slice of structures deep mapping

    Slice of structures deep mapping

    Structs in slices were simply copyed to the target map. With this PR they can also be mapped, using the "deep" tag, or the global "Deep" option.

    Partially solves #249

  • Pointers decoding inconsistency (and discussion)

    Pointers decoding inconsistency (and discussion)

    When decoding a struct containing a string pointer and a struct pointer into a map, the resulting map has a pointer to the string but not a pointer to the struct.

    func TestDecode_structToMap_PointersConsistency(t *testing.T) {
    	type SourceChild struct {
    		String string `mapstructure:"string"`
    	}
    
    	type SourceParent struct {
    		ChildPtr  *SourceChild `mapstructure:"child-ptr"`
    		StringPtr *string      `mapstructure:"string-ptr"`
    	}
    
    	var target map[string]interface{}
    
    	var value = "goodbye"
    
    	source := SourceParent{
    		ChildPtr: &SourceChild{
    			String: "hello",
    		},
    		StringPtr: &value,
    	}
    
    	if err := Decode(source, &target); err != nil {
    		t.Fatalf("got error: %s", err)
    	}
    
    	expected := map[string]interface{}{
    		"child-ptr": map[string]interface{}{
    			"string": "hello",
    		},
    		"string-ptr": "goodbye",
    	}
    
    	expectedPointers := map[string]interface{}{
    		"child-ptr": &map[string]interface{}{
    			"string": "hello",
    		},
    		"string-ptr": stringPtr("goodbye"),
    	}
    
    	if !reflect.DeepEqual(target, expected) && !reflect.DeepEqual(target, expectedPointers) {
    		t.Fatalf("bad: \nexpected        : %#v\nexpectedPointers: %#v\nresult          : %#v", expected, expectedPointers, target)
    	}
    }
    

    Produces:

    === RUN   TestDecode_structToMap_PointersConsistency
        mapstructure_test.go:2941: bad: 
            expected        : map[string]interface {}{"child-ptr":map[string]interface {}{"string":"hello"}, "string-ptr":"goodbye"}
            expectedPointers: map[string]interface {}{"child-ptr":(*map[string]interface {})(0xc0000a8058), "string-ptr":(*string)(0xc00008eec0)}
            result          : map[string]interface {}{"child-ptr":map[string]interface {}{"string":"hello"}, "string-ptr":(*string)(0xc00008ede0)}
    --- FAIL: TestDecode_structToMap_PointersConsistency (0.00s)
    

    mapstructure version https://github.com/mitchellh/mapstructure/commit/bf980b35cac4dfd34e05254ee5aba086504c3f96 go version go1.18.7 linux/amd64

    I can't see a reason why structures and base types should behave differently. I would expect to either keep the pointers or remove them. Can you please explain to me if this is a design choice or not? My guess is that struct pointers where not supported before PR #271 which made it dereference struct pointers systematically.

    Discussion

    Should all pointers be represented in the target map or none?

    From the perspective of my use case, the pointer representation in the map is not necessary.

    Here are the benefits of the pointers usage in this context:

    • Data representation : allows the representation of "no information". E.g. a nul pointer is different from an empty string. -> nul pointer can be represented as an unset key
    • Memory and speed optimization : redundent data structures can be referenced and large data structure can be referenced so that they are not copied on return -> this can be usefull when the decoded structure contains big data that should not be copied

    For these reasons, I would suggest to:

    • Handle pointers of any type in the same way (e.g. struct pointers and string pointers)
    • Create a global option tho choose whether to keep or not the pointer representation in the target map (in the other direction, it's the type of the struct data that defines if it's a pointer or not). Defaults to "remove pointers". Name suggestion : "PreserveReferences"(true/false)
    • For retro-compatibility, add an option to keep the pointer representation on non-struct types. Eventually set this by default until next major. Name suggestion : "LegacyReferencesHandling" (true/false)
    • Add a property tag option to keep pointer representation and not copy data but only the pointer value (used for optimiazation in case of a large data structure, for example a string, that should be referenced in the map instead of copied). Name suggestion: "nocopy"

    As an alternative, the options could be merge in a "ReferencesHandling" option with values legacy/preserve/discard.

    Please let me know what you think and if there are any thechnical constraints that would not allow this. I would be happy to contribute.

GED - Global-purpose Encoding / Decoding library

GED - Global-purpose Encoding / Decoding library This library lets you use common encoding/decoding schemes and allows you to define custom ones. Use

Nov 28, 2021
An optimal, byte-aligned, LZ+RLE hybrid encoder, designed to maximize decoding speed on NMOS 6502 and derived CPUs
An optimal, byte-aligned, LZ+RLE hybrid encoder, designed to maximize decoding speed on NMOS 6502 and derived CPUs

TSCrunch TSCrunch is an optimal, byte-aligned, LZ+RLE hybrid encoder, designed to maximize decoding speed on NMOS 6502 and derived CPUs, while keeping

Dec 21, 2022
Easily and dynamically generate maps from Go static structures

structomap This package helps you to transform your struct into map easily. It provides a structomap.Serializer interface implemented by the structoma

Dec 9, 2022
csvutil provides fast and idiomatic mapping between CSV and Go (golang) values.
csvutil provides fast and idiomatic mapping between CSV and Go (golang) values.

csvutil Package csvutil provides fast and idiomatic mapping between CSV and Go (golang) values. This package does not provide a CSV parser itself, it

Jan 6, 2023
generic sort for slices in golang

slices generic sort for slices in golang basic API func BinarySearch[E constraints.Ordered](list []E, x E) int func IsSorted[E constraints.Ordered](li

Nov 3, 2022
Golang binary decoder for mapping data into the structure

binstruct Golang binary decoder to structure Install go get -u github.com/ghostiam/binstruct Examples ZIP decoder PNG decoder Use For struct From file

Dec 17, 2022
Cap'n Proto library and parser for go. This is go-capnproto-1.0, and does not have rpc. See https://github.com/zombiezen/go-capnproto2 for 2.0 which has rpc and capabilities.

Version 1.0 vs 2.0 Update 2015 Sept 20: Big news! Version 2.0 of the go-bindings, authored by Ross Light, is now released and newly available! It feat

Nov 29, 2022
Asn.1 BER and DER encoding library for golang.

WARNING This repo has been archived! NO further developement will be made in the foreseen future. asn1 -- import "github.com/PromonLogicalis/asn1" Pac

Nov 14, 2022
A go library that facilitates the implementation of decorator pattern.

go-decorator go-decorator is a library that facilitates the implementation of decorator pattern. Installation To install go-decorator, use go get: go

Nov 25, 2021
A library that provides dynamic features of Go language.

go-dynamic go-dynamic is a library that provides dynamic features of Go language. Installation To install go-dynamic, use go get: go get -u github.com

Dec 8, 2021
Encode and decode binary message and file formats in Go

Encode and Decode Binary Formats in Go This module wraps the package encoding/binary of the Go standard library and provides the missing Marshal() and

Dec 22, 2022
idiomatic codec and rpc lib for msgpack, cbor, json, etc. msgpack.org[Go]

go-codec This repository contains the go-codec library, the codecgen tool and benchmarks for comparing against other libraries. This is a High Perform

Dec 19, 2022
Go package for dealing with maps, slices, JSON and other data.

Objx Objx - Go package for dealing with maps, slices, JSON and other data. Get started: Install Objx with one line of code, or update it with another

Dec 27, 2022
Encode and decode Go (golang) struct types via protocol buffers.

protostructure protostructure is a Go library for encoding and decoding a struct type over the wire. This library is useful when you want to send arbi

Nov 15, 2022
Simple, specialised, and efficient binary marshaling

?? surge Documentation A library for fast binary (un)marshaling. Designed to be used in Byzantine networks, ?? surge never explicitly panics, protects

Oct 4, 2022
Go library for decoding generic map values into native Go structures and vice versa.

mapstructure mapstructure is a Go library for decoding generic map values to structures and vice versa, while providing helpful error handling. This l

Dec 28, 2022
Go library for encoding native Go structures into generic map values.

wstructs origin: github.com/things-go/structs Go library for encoding native Go structures into generic map values. Installation Use go get. go ge

Jan 10, 2022
:steam_locomotive: Decodes url.Values into Go value(s) and Encodes Go value(s) into url.Values. Dual Array and Full map support.

Package form Package form Decodes url.Values into Go value(s) and Encodes Go value(s) into url.Values. It has the following features: Supports map of

Dec 26, 2022
Simple Relay between a Unix socket and a TCP socket, and vice versa.
Simple Relay between a Unix socket and a TCP socket, and vice versa.

Simple TCP <-> Unix Relay simpletcpunixrelay is a program which exposes a TCP endpoint as a Unix socket and vice versa. Usecase Let's say you are runn

Nov 23, 2022
Lookup or replace AWS account IDs with their names and vice versa

awsacc A trusty helper for working with AWS account IDs. Working with AWS account IDs often involves more manual effort than necessary. Often account

Oct 14, 2021