Decode / encode XML to/from map[string]interface{} (or JSON); extract values with dot-notation paths and wildcards. Replaces x2j and j2x packages.

I've encountered a case where 'skey' seems to be empty and golang throws 'panic: assignment to entry in nil map'. Adding a check for 'skey' fixes that.

When getting the following perfectly valid XML (a response from PANW Wildfire component): ` 2.0 <file_info> yes b918d3b98f590d99ac3694b373958017256a22b4 PE dd8e64991f68eeebb52c9947ea56f39a776185d1a076741533aa6843022c03b0 7f638f13d0797ef9b1a393808dc93b94 55296 </file_info> <task_info> 3.0 2 Windows XP, Adobe Reader 9.4.0, Flash 10, Office 2007 dd8e64991f68eeebb52c9947ea56f39a776185d1a076741533aa6843022c03b0 7f638f13d0797ef9b1a393808dc93b94 55296 yes

map.Xml() time grows exponentially for large map. For example if we read 1MB XML and serialize it back it take almost 20s and if we double the size (duplicate the xml nodes from same input) then time taken is almost 90s.

I tested with this xml (http://aiweb.cs.washington.edu/research/projects/xmltk/xmldata/data/mondial/mondial-3.0.xml) with following code.

package main

import (
	"fmt"
	"io/ioutil"
	"time"

	mxj "github.com/clbanning/mxj"
)

func main() {
	b, _ := ioutil.ReadFile("/tmp/large.xml")
	start := time.Now()
	m, _ := mxj.NewMapXml(b)
	fmt.Printf("XML Read time %v\n", time.Now().Sub(start))
	start = time.Now()
	b, _ = m.Xml()
	fmt.Printf("XML Write time %v\n", time.Now().Sub(start))
}

I debugged further and I think it is because of string concatenation (*s += ). I created a simple code to string concatenation and found that it is very slow as string grows. When I replaced it by bytes.Buffer.Read then there was significant improvement.

There seems to be a couple issues with this function.

Getting a runtime error: index out of range [recovered] First the switch is on the length of the split string. If the length of the string array is 3 (case 3), then you can't reference vv[3] which would be the 4th element right? It must be vv[2] to access the third string.

Second, if your text has any ":" in it, it will get confused. My text had a "http://blah/blah/blah" in it. So, after I made the above change, I got the error: "unknown subkey conversion spec:..."

Basically I was trying to read in a kml file, change the description and name fields and then print it back out. I was trying to use the paths= PathsForKey("description") values =ValuesForPath(paths[0]).
UpdateValuesForPath(newVal, path[0], "description:"+value[0]) with a double loop in there to get all the paths and values. But, because description can have any kind of text, this will not be possible. Is there a better way of doing this?

Thanks, Lorie

hi,

I'm dealing with some XML files that have <?xml version="1.0" encoding="utf-8"?> at top of the file by using the NewMapXmlSeq cause I want to preserve the format and some comments in the file.

But I could only get the result map[#procinst:map[#target:xml #inst:version="1.0" encoding="utf-8"]], and the returned error is no root key

As you mentioned in the doc:

comments, directives, and procinsts that are NOT part of a document with a root key will be returned as
  map[string]interface{} and the error value 'NoRoot'.

How can I get the whole map of the XML file? I thought it's quite common that <?xml version="1.0" encoding="utf-8"?> not belong to any root

Input json: "ipv4": { "address-mode": "manual", "address": { "ip": "10.10.10.10", "prefix-length": 30 }

Output xml: <ipv4><address><ip>10.0.22.14</ip><prefix-length>30.000000</prefix-length></address><address-mode>manual</address-mode><enabled>true</enabled><mtu>1500.000000</mtu><vti_peer>10.10.10.10</vti_peer></ipv4>

In this case, prefix-length:30 became 30.000000 Do anyone have the same problem with me?

But it should not, of course. E.g. serializing this mxj-map

        m[""] = "empty"
	m["$invalid"] = "hex$"
	m["entities"] = "<>&"
	m["nil"] = nil

produces this string, which is formally incorrect XML:

<root>
  <>empty</>
  <$invalid>hex$</$invalid>
  <entities><>&</entities>
  <nil><nil></nil>
</root>

Errors are:

• <> is not a valid XML element
• $ is not valid in XML element names
• the three basic XML entities <, > and & must be escaped as <, > and &
• the nil interface{} value is serialized as <nil> without escaping < and >

Additionally there are some decisions to be drawn for how to mitigate these problems. Probably some strategies need to be implemented for a client to customize the behavior of the API

• empty element names should either be omitted or given a customizeable name
• invalid element name characters could be either dropped completely or somehow encoded by valid entity characters like a hex string representation decorated with the valid _ char
• nil valued elements could be serialized as <element-name/>, <element-name></element-name>, omitted, or marked with xsi:nil="true" (see e.g. https://www.oreilly.com/library/view/xml-in-a/0596007647/re166.html)

test fixture (example)

package web

import (
	"bytes"
	"testing"

	"github.com/clbanning/mxj/v2"
)

func TestJSON2XMLConverter(t *testing.T) {
	m := mxj.New()

	m[""] = "empty"
	m["$invalid"] = "hex$"
	m["entities"] = "<>&"
	m["nil"] = nil

	buf := new(bytes.Buffer)
	err := m.XmlWriter(buf, "root")
	if err != nil {
		t.Fatalf("writer: %v", err)
	}

	if got, want := buf.String(), `<root><_24_invalid>hex$</_24_invalid><entities>&lt;&gt;&amp;</entities><nil></nil></root>`; got != want {
		// want <root><_>empty</_><_24_invalid>hex$</_24_invalid><entities>&lt;&gt;&amp;</entities><nil></nil></root>
		// got <root><>empty</><$invalid>hex$</$invalid><entities><>&</entities><nil><nil></nil></root>
		t.Errorf("want %v, got %v", want, got)
	}
}

Hi there, mxj is a great project and I've been using it in my gf project in XML marshal/unmarshal. It worked quite well with map[string]interface{} type, but when the parameter contains other map type (eg: map[string]string/map[string]int etc...), it failed working.

Here's the simple codes to reproduce this issue:

package main

import (
    "fmt"
    "github.com/clbanning/mxj"
)

func main() {
    m := make(map[string]interface{})
    m["m"] = map[string]string {
        "k" : "v",
    }
    b, _ := mxj.Map(m).Xml()
    fmt.Println(string(b))
}

It's supposed to give me {"m":{"k":"v"}}, but actually I got >UNKNOWN/>.

Would you please take a look, and kindly give me a nice feedback, thanks!

Is there any way to get the ordering of inner tags, if they are variable in their tag name?

For example:

Obj Attr3 Attr1 Attr2 Obj

This ends up a map

{Obj:{Attr1:"",Attr2:"",Attr3:""}}, and there is no way to tell that the order was 3,1,2. I've run into a situation where I need to know the order, is there some flag or something that would allow me to turn the attributes into an ordered array, or to write out the order the parser ran into them as an attribute or something?

Using NewMapXml with data that contains HTML tags results in truncated output.

package main

import (
 "fmt"

 "github.com/clbanning/mxj"
)

func main() {
 bytes := []byte(`<?xml version="1.0"?>
<article>
	<body>
    		<p>hello world<b>truncate me</b></p>
	</body>
</article>`)

 m, err := mxj.NewMapXml(bytes)
 if err != nil {
  fmt.Println("err:", err)
  return
 }
 fmt.Println("mxj.Map:\n",m.StringIndent(3))

 mxj.XMLEscapeChars(false)
 v, err := m.Xml()
 if err != nil {
  fmt.Println("err:", err)
  return
 }
 fmt.Println("\nm.Xml:\n", string(v))
}

mxj.Map:
       article : 
        body : 
          p : [string] hello world

m.Xml:
 <article><body><p>hello world</p></body></article>

The expected behaviour could be to leave the content as it is without the truncation. @clbanning what do you think?

Using NewMapXml with data that contains entities (e.g. <) results in unexpected output.

bytes := []byte(`<?xml version="1.0"?>
<systemList>
  <system>
    <name>retropie</name>
    <fullname>RetroPie</fullname>
    <path>/home/fozz/RetroPie/retropiemenu</path>
    <extension>.rp .sh</extension>
    <command>sudo /home/fozz/RetroPie-Setup/retropie_packages.sh retropiemenu launch %ROM% &lt;/dev/tty &gt;/dev/tty</command>
    <platform/>
    <theme>retropie</theme>
  </system>
</systemList>`)

val, _ := mxj.NewMapXml(bytes)

// Print val["systemList"]["system"]["command"]

fmt.Println(val["systemList"].(map[string]interface{})["system"].(map[string]interface{})["command"])

//  sudo /home/fozz/RetroPie-Setup/retropie_packages.sh retropiemenu launch %ROM% </dev/tty >/dev/tty

I expected the value of systemList > system > command to be: sudo /home/fozz/RetroPie-Setup/retropie_packages.sh retropiemenu launch %ROM% </dev/tty >/dev/tty

But I can see that it's actual value is: sudo /home/fozz/RetroPie-Setup/retropie_packages.sh retropiemenu launch %ROM% </dev/tty >/dev/tty

If value of XML attribute contains double quotes (") and mxj.XMLEscapeChars is false, the library produces invalid XML through XML() function (unescaped double quotes are placed into value of attributes).

I'm currently using mxj.MapSeq to modify XML documents, but I find adding new nodes cumbersome. It's likely that I'm doing something wrong, so I come to you for advice :sweat_smile:

Right now, when I want to add a new child node to some XML element, I assume that I need to figure out manually what the sequence number of that child node should be. To do so, I walk the MapSeq to find the largest number, then I add one to that. Is there a better way to do this?

In code, that looks like:

// mapSeqNumber returns the sequence number associated with this map
func mapSeqNumber(m map[string]interface{}) (int, bool) {
	if seq, ok := m["#seq"].(int); ok {
		return seq, true
	}

	if seq, ok := m["#seq"].(float64); ok {
		return int(seq), true
	}

	return 0, false
}

// mapSeqMaxSeqNumber returns the maximum sequence number of all child maps
func mapSeqMaxSeqNumber(m map[string]interface{}) int {
	maxSeqNumber := -1
	for _, val := range m {
		switch data := val.(type) {
		case map[string]interface{}:
			seqNumber, ok := mapSeqNumber(data)
			if !ok {
				continue
			}
			if seqNumber <= maxSeqNumber {
				continue
			}
			maxSeqNumber = seqNumber
		case []interface{}:
			for _, v := range data {
				vMap, ok := v.(map[string]interface{})
				if !ok {
					continue
				}
				seqNumber, ok := mapSeqNumber(vMap)
				if !ok {
					continue
				}
				if seqNumber <= maxSeqNumber {
					continue
				}
				maxSeqNumber = seqNumber
			}
		}
	}
	return maxSeqNumber
}

// Load some XML
m, _ := mxj.NewMapXmlSeq(/*... XML DATA ...*/)
// Get the maximum child sequence number
maxSeqNumber := mapSeqMaxSeqNumber(m)
// Create a new child with the next sequence number
m["NewChildNode"] = map[string]interface{
    "#seq": maxSeqNumber + 1,
}