A better way to marshal and unmarshal YAML in Golang

YAML marshaling and unmarshaling support for Go

Build Status

Introduction

A wrapper around go-yaml designed to enable a better way of handling YAML when marshaling to and from structs.

In short, this library first converts YAML to JSON using go-yaml and then uses json.Marshal and json.Unmarshal to convert to or from the struct. This means that it effectively reuses the JSON struct tags as well as the custom JSON methods MarshalJSON and UnmarshalJSON unlike go-yaml. For a detailed overview of the rationale behind this method, see this blog post.

Compatibility

This package uses go-yaml and therefore supports everything go-yaml supports.

Caveats

Caveat #1: When using yaml.Marshal and yaml.Unmarshal, binary data should NOT be preceded with the !!binary YAML tag. If you do, go-yaml will convert the binary data from base64 to native binary data, which is not compatible with JSON. You can still use binary in your YAML files though - just store them without the !!binary tag and decode the base64 in your code (e.g. in the custom JSON methods MarshalJSON and UnmarshalJSON). This also has the benefit that your YAML and your JSON binary data will be decoded exactly the same way. As an example:

BAD:
	exampleKey: !!binary gIGC

GOOD:
	exampleKey: gIGC
... and decode the base64 data in your code.

Caveat #2: When using YAMLToJSON directly, maps with keys that are maps will result in an error since this is not supported by JSON. This error will occur in Unmarshal as well since you can't unmarshal map keys anyways since struct fields can't be keys.

Installation and usage

To install, run:

$ go get github.com/ghodss/yaml

And import using:

import "github.com/ghodss/yaml"

Usage is very similar to the JSON library:

package main

import (
	"fmt"

	"github.com/ghodss/yaml"
)

type Person struct {
	Name string `json:"name"` // Affects YAML field names too.
	Age  int    `json:"age"`
}

func main() {
	// Marshal a Person struct to YAML.
	p := Person{"John", 30}
	y, err := yaml.Marshal(p)
	if err != nil {
		fmt.Printf("err: %v\n", err)
		return
	}
	fmt.Println(string(y))
	/* Output:
	age: 30
	name: John
	*/

	// Unmarshal the YAML back into a Person struct.
	var p2 Person
	err = yaml.Unmarshal(y, &p2)
	if err != nil {
		fmt.Printf("err: %v\n", err)
		return
	}
	fmt.Println(p2)
	/* Output:
	{John 30}
	*/
}

yaml.YAMLToJSON and yaml.JSONToYAML methods are also available:

package main

import (
	"fmt"

	"github.com/ghodss/yaml"
)

func main() {
	j := []byte(`{"name": "John", "age": 30}`)
	y, err := yaml.JSONToYAML(j)
	if err != nil {
		fmt.Printf("err: %v\n", err)
		return
	}
	fmt.Println(string(y))
	/* Output:
	name: John
	age: 30
	*/
	j2, err := yaml.YAMLToJSON(y)
	if err != nil {
		fmt.Printf("err: %v\n", err)
		return
	}
	fmt.Println(string(j2))
	/* Output:
	{"age":30,"name":"John"}
	*/
}
Comments
  • Please tag a new release after #49

    Please tag a new release after #49

    #49 adds basic support for Go modules, but the latest current release tag (v1.0.0) is dated March 2017. Could you please find a suitable point at or after #49 to add a newer tag? (I think v1.1.0 is a reasonable choice, since there don't seem to be any API changes, but I didn't check exhaustively).

  • Panic when trying to unmarshal specific yaml

    Panic when trying to unmarshal specific yaml

    This is a pretty bizarre one. When trying to unmarshal this yaml:

    Port: 7746
    Processes:
        Something:
            Command: some command
    

    I get this panic:

    panic: reflect: reflect.Value.Set using unaddressable value
    
    goroutine 1 [running]:
    panic(0x3cdb80, 0xc820132680)
        /usr/local/Cellar/go/1.6.2/libexec/src/runtime/panic.go:481 +0x3e6
    reflect.flag.mustBeAssignable(0x16)
        /usr/local/Cellar/go/1.6.2/libexec/src/reflect/value.go:229 +0x221
    reflect.Value.Set(0x3a98e0, 0x0, 0x16, 0x3a98e0, 0xc82016e2c0, 0x16)
        /usr/local/Cellar/go/1.6.2/libexec/src/reflect/value.go:1328 +0x25
    dj/vendor/github.com/ghodss/yaml.indirect(0x3a98e0, 0x0, 0x16, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        /Users/craig/go-clean/src/dj/vendor/github.com/ghodss/yaml/fields.go:48 +0x528
    dj/vendor/github.com/ghodss/yaml.convertToJSONableObject(0x3ca880, 0xc820136630, 0xc8201346a0, 0x0, 0x0, 0x0, 0x0)
        /Users/craig/go-clean/src/dj/vendor/github.com/ghodss/yaml/yaml.go:106 +0xc9
    dj/vendor/github.com/ghodss/yaml.convertToJSONableObject(0x3ca880, 0xc8201365d0, 0xc8200d7170, 0x0, 0x0, 0x0, 0x0)
        /Users/craig/go-clean/src/dj/vendor/github.com/ghodss/yaml/yaml.go:205 +0xf3a
    dj/vendor/github.com/ghodss/yaml.convertToJSONableObject(0x3ca880, 0xc8201365a0, 0xc8200d7628, 0x0, 0x0, 0x0, 0x0)
        /Users/craig/go-clean/src/dj/vendor/github.com/ghodss/yaml/yaml.go:195 +0xb15
    dj/vendor/github.com/ghodss/yaml.yamlToJSON(0xc820162000, 0x43, 0x243, 0xc8200d78a0, 0x0, 0x0, 0x0, 0x0, 0x0)
        /Users/craig/go-clean/src/dj/vendor/github.com/ghodss/yaml/yaml.go:89 +0x110
    dj/vendor/github.com/ghodss/yaml.Unmarshal(0xc820162000, 0x43, 0x243, 0x3a9880, 0x78c480, 0x0, 0x0)
        /Users/craig/go-clean/src/dj/vendor/github.com/ghodss/yaml/yaml.go:32 +0xa8
    

    Interestingly it seems like it might be something specific to the word Processes cause if I change that to Process or really any other word it works fine. I was previously using an older version of your library c3eb24aeea63668ebdac08d2e252f20df8b6b1ae and it works fine there.

  • Return typed errors from exported funcs

    Return typed errors from exported funcs

    By adding a YAMLError type and returning typed errors from exported functions, downstream users will be able to easily distinguish yaml-specific errors.

    Related downstream PR: https://github.com/kubernetes/kubernetes/pull/38905

    cc @deads2k @fabianofranz

  • "y" argument not parsed

    After parsing the following yaml :

    x: 10
    y: 20
    

    x is being extracted, but y is not. Changing 'y' to anything else works.

    Reproducible sample:

    package main
    
    import (
        "fmt"
        "github.com/ghodss/yaml"
    )
    
    type Position struct {
        X int `json:"x"`
        Y int `json:"y"`
    }
    
    func main() {
        s := []byte(`
    x: 10
    y: 20
    `)
    
        var p Position
        err := yaml.Unmarshal(s, &p)
        if err != nil {
            fmt.Printf("err: %v\n", err)
            return
        }
        fmt.Println(p) // Should be {10 20}, but prints {10 0}
    }
    
    
  • Updating go.mod to take the latest release of go-yaml

    Updating go.mod to take the latest release of go-yaml

    Updating go-yaml to prevent abuse of high CPU/ memory consumption: https://github.com/go-yaml/yaml/releases/tag/v2.2.4

    Issue where it was mentioned: https://github.com/kubernetes/kubernetes/issues/83253

  • Add UnmarshalStrict for strict yaml unmarshaling

    Add UnmarshalStrict for strict yaml unmarshaling

    Reopening a fresh PR with code from @tanner-bruce in: https://github.com/ghodss/yaml/pull/36

    Please note that i had another one similar to Tanner's in: https://github.com/ghodss/yaml/pull/37

    This is NOT the same as in: https://github.com/ghodss/yaml/pull/33

    and we need it to fix Kubernetes issue: https://github.com/kubernetes/kubernetes/issues/67477

  • Adds a hook to specify unmarshal options.

    Adds a hook to specify unmarshal options.

    This makes it possible to configure the parser to error out if an unknown tagged field has been found in the byte stream that is being unmarshaled to a go object.

    For example like this (also see ExampleUnknown in yaml_test.go):

    func ExampleUnknown() {
    	type WithTaggedField struct {
    		Field string `json:"field"`
    	}
    	y := []byte(`unknown: "hello"`)
    	v := WithTaggedField{}
    	fmt.Printf("%v\n", Unmarshal(y, &v, DisallowUnknownFields))
    	// Ouptut:
    	// unmarshaling JSON: while decoding JSON: json: unknown field "unknown"
    }
    

    If you want to manipulate the decoder used in the unmarshal function, you can define a custom option to your liking and apply it in the Unmarshal call.

    The way I found out about this is kind of comical. Some nonprintable characters made its way into my YAML file. You wouldn't see them in the editor, but they'd be there. Now when the YAML file is parsed, the nonprintable chars together with the key characters were made into a weird UTF-8 key (e.g. like "\u00c2some_key"), which would then become a key unknown to the tags of my struct, and would silently get dropped in the parsing steps. So as result, you would get a perfectly normal parse, except the data that you wanted to be in the struct is not there.

    This meant that silently dropping unknown keys is not always a good idea, so I went to fix it, while retaining the same interface. JsonOpt type was fairly easy to add in the Unmarshal because it can be silently ignored by the old code.

  • Add UnmarshalStrict.

    Add UnmarshalStrict.

    UnmarshalStrict is like Unmarshal except that any mapping keys that are duplicates, will result in an error.

    Since https://godoc.org/gopkg.in/yaml.v2 provides both Unmarshal and UnmarshalStrict, this library should provide the same interface.

  • Use pkg/errors to wrap extended errors

    Use pkg/errors to wrap extended errors

    Creating a new error with the content from the previous one loses context. By wrapping the error, we can still extend it while allowing to retrieve the base error.

    For example, the json package defines SyntaxError which carries more than just the message (by being it's own struct, we can do a switch on it's type). Wrapping the error avoids losing that information.

  • Difference in behaviour parsing numbers compared to gopkg.in/yaml.v2

    Difference in behaviour parsing numbers compared to gopkg.in/yaml.v2

    gopkg.in/yaml.v2 parses yaml integers into ints and yaml floats into float64s, which I believe is correct. github.com/ghodss/yaml parses both into float64s, which is a difference in behaviour, and which I believe is wrong.

  • How to decode yaml with ---

    How to decode yaml with ---

    kind: Namespace
    metadata:
       name: test
    ---
    name: aaa
    
    type Namespace struct {
        Kind     string `json:"kind"`
        Metadata struct {
            Name string `json:"name"`
        } `json:"metadata"`
    }
    
    type Name struct {
        Name string `json:"name"`
    }
    

    Can't decode the name struct, How?

  • 🚚 New fork available at invopop/yaml

    🚚 New fork available at invopop/yaml

    I've been using this package extensively over the last few years as it makes working with YAML in Go much easier. YAML is great, but it's flexibility also makes it hard to work with sometimes in Go; converting first to JSON makes a ton of sense for most use-cases. I was quite sad to see the number of unresolved issues and felt that the latest security scare (CVE-2022-28948) was enough to warrant a fork, so I did:

    https://github.com/invopop/yaml

    Most of the issues I see are related to the fact that the library is still using yaml.v2, so our first step was to upgrade to yaml.v3: https://github.com/invopop/yaml/pull/2. It was more complex than expected due to the change in handling of keys in maps, but it's working now.

    We also maintain the jsonschema package so are quite used to the messy reflection stuff that needs to happen behind the scenes.

    I do hope that the original maintainers don't mind, and I'd like to take this opportunity to thank them for all their work! I hope we can continue to maintain our fork effectively, and welcome any contributions.

  • ⚡ Is anyone maintaining the project ?

    ⚡ Is anyone maintaining the project ?

    My apologies but I'd like to know if someone takes care of this project. ghodss/yaml is apparently used in many projects and would deserve at least minimal maintenance (at least for future CVE dependabot reporting).

    thx

    cc @ghodss

  • fix: CVE-2022-28948 by upgrading go-yaml/yaml

    fix: CVE-2022-28948 by upgrading go-yaml/yaml

    This PR fixes https://snyk.io/vuln/SNYK-GOLANG-GOPKGINYAMLV2-2840885 by upgrading go-yaml/yaml to v3.

    It is dependent on https://github.com/go-yaml/yaml/pull/856.

  • Keys like

    Keys like "yes" and "on" are read as true

    template = `
    on:
     - RUNNING
    `
    template, _ = yaml.YAMLToJSON(template)
    fmt.Println(string(template))
    // => {"true":["DELETED"]}
    
  • How about cutting a new release?

    How about cutting a new release?

    The latest release is v1.0.0 which is released in the year 2017, I think it's time to cut a new release, e.g. v1.0.1 or v1.1.0(only if there are new features).

    cc @bradfitz @ghodss

Related tags
Go-yaml - Yaml parsing Toolkit For Golang

go-yaml 介绍 gopkg.in/yaml.v3 已经是个非常好用的包,但是在实际开发中总有类型转换带来的麻烦,go-yaml只是在它的基础上,简单的一层

Jan 13, 2022
Golang library for reading properties from configuration files in JSON and YAML format or from environment variables.

go-config Golang library for reading properties from configuration files in JSON and YAML format or from environment variables. Usage Create config in

Aug 22, 2022
Library providing routines to merge and validate JSON, YAML and/or TOML files
Library providing routines to merge and validate JSON, YAML and/or TOML files

CONFLATE Library providing routines to merge and validate JSON, YAML, TOML files and/or structs (godoc) Typical use case: Make your application config

Sep 26, 2022
Golang Configuration tool that support YAML, JSON, TOML, Shell Environment

Configor Golang Configuration tool that support YAML, JSON, TOML, Shell Environment (Supports Go 1.10+) Usage package main import ( "fmt" "github.c

Dec 29, 2022
Golang config.yaml loader

Description goconfig is a configuration library designed using the following pri

May 31, 2022
Generic templating tool with support of JSON, YAML and TOML data

gotempl Small binary used to generate files from Go Templates and data files. The following formats are supported: JSON YAML TOML Usage usage: gotempl

Jun 15, 2022
Go-config - Config parser for go that supports environment vars and multiple yaml files

go-multiconfig This package is able to parse yaml config files. It supports gett

Jun 23, 2022
JSON or YAML configuration wrapper with convenient access methods.

Config Package config provides convenient access methods to configuration stored as JSON or YAML. This is a fork of the original version. This version

Dec 16, 2022
Light weight, extensible configuration management library for Go. Built in support for JSON, TOML, YAML, env, command line, file, S3 etc. Alternative to viper.
Light weight, extensible configuration management library for Go. Built in support for JSON, TOML, YAML, env, command line, file, S3 etc. Alternative to viper.

koanf (pronounced conf; a play on the Japanese Koan) is a library for reading configuration from different sources in different formats in Go applicat

Jan 8, 2023
YAML support for the Go language.

YAML support for the Go language Introduction The yaml package enables Go programs to comfortably encode and decode YAML values. It was developed with

Jan 8, 2023
create a bootable disk image from Docker image or a yaml config

docker2boot docker2boot creates a bootable disk from either a Docker image or a config yaml file Features status dns Y cloud-init Y network Y ssh TODO

Oct 30, 2022
It syncronizes the configuration described in a YAML file against your GitHub Organization

It syncronizes the configuration described in a YAML file against your GitHub Organization. Combined with a CI system, it can be used to implement GitOps for GitHub.

Jul 19, 2021
YAML support for the Go language
YAML support for the Go language

YAML support for the Go language

Dec 31, 2022
Tmpl - A tool to apply variables from cli, env, JSON/TOML/YAML files to templates

tmpl allows to apply variables from JSON/TOML/YAML files, environment variables or CLI arguments to template files using Golang text/template and functions from the Sprig project.

Nov 14, 2022
SmartYAML - Go package to handle YAML

SmartYAML - Go package to handle YAML The smartyaml is a go package to handle parsed YAML files more confortable. This package is not a parser, it use

Feb 25, 2022
VINYL Inscribes Nettlesome YAML Legibly

VINYL Inscribes Nettlesome YAML Legibly VINYL formats yaml files into a canonical format retaining comments and setting the indentation by default to

Jan 18, 2022
Quick and easy way to load config files based on a simple set of rules.
Quick and easy way to load config files based on a simple set of rules.

config Quick and easy way to load config files based on a simple set of rules. Project inspired by https://github.com/lorenwest/node-config Important

Apr 9, 2021
Jul 4, 2022