Split out from https://github.com/johnkerl/miller/issues/369. See also https://github.com/johnkerl/miller/blob/master/go/README.md.

Pre-release/rough-draft docs are at http://johnkerl.org/miller6.

Things which may change:

As noted in go/README.md, I want to preserve as much user experience as possible. That said:

• --jvstack and --jsonx will still be supported as command-line flags, but JSON output will be pretty-printed (like --jvstack) by default.
• --csvlite will still be different from --csv, as detailed below.
• emitf and emitp were invented before I had for-loops in the DSL. If people really want to keep these and are using these, I can keep them; but maybe we're better off leaving them behind. Please let me know.
• CR vs CR/LF (line-endings) will be platform-appropriate using Go's own portability -- Windows files will be written correctly on Windows, and likewise for Linux and MacOS. That said, I don't know if we need any longer to preserve CR/LF-to-CR/LF even on Linux (line endings which are non-standard for the platform) -- again, please let me know.
• mlr put -S and mlr put -F will become unnecessary since string-conversion will be done just-in -time as suggested by @gromgit on https://github.com/johnkerl/miller/issues/151.

Please include here any thoughts you have on the Go port.

Hello,

On a gnu/linux system, make is failing with a simple make:

make: *** No targets specified and no makefile found.  Stop.

cc --version
gcc-4.8.real (Ubuntu 4.8.4-2ubuntu1~14.04) 4.8.4
Copyright (C) 2013 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Is there a new method to make mlr now?

I suggest to reuse a higher level build system than your current small make file so that powerful checks for software features will become easier.

• CMake
• Autotools

I'd like to do some kind of regex based parsing, like that:

mlr filter '$FIELD =~ "([A-Z]+)([0-9]+)" ' then put '$F1  = "\1"; $F2 = "\2" '

how can I do that? I've succeeded with sub() like this, but it's not optimal :

mlr filter '$FIELD =~ "([A-Z]+)([0-9]+)" ' then put '$F1  = sub($FIELD, "([A-Z]+)([0-9]+)", '\1") '

is there a shorter way?

I love the idea of Miller. It is clearly a needed tool that is missing from the standard unix toolbox.

However, you really cannot say you have a tool that is designed to support csv, without supporting csv.

CSV is a standard file format, and has an RFC: https://tools.ietf.org/html/rfc4180

Not supporting double quotes is the same thing as saying that you do not support csv, since double quotes are central to the way that the standard handles other characters... comma being just one example. Your tool is young enough that supporting the standard now will make later development much simpler. This will prevent the situation years from now where you have a 'normal mode' and a 'standards mode'. If you make the change now you can just have the one correct mode.

You have an ambitious work-list, but I would suggest taking a pause and thinking about how you will support the RFC version of the file format.

People like me (open data advocates) spend alot of time trying to ensure that organizations that release csv do so under the standard format, rather than releasing unparsable garbage. Having a library like yours that supported the standard too would be a huge boon.. I could say things like:

"See by using the RFC for your data output, all kinds of open tools will work out of the box on your data... like Miller (link)"

Thank you for working on such a clever tool...

Regards, -FT

More documentation details. flatten method does not have a complete description. It explains what it does, but it does not explain which are the arguments. You have to look and understand the examples to grasp the meaning. The usage should be clear from the description itself.

Similarly, for join* and split*, the different arguments are not clear without looking at the examples. Some assumptions could be made, like the parameters for joink are the array/map keys and the string to use to make the join. But what about joinkv? Which parameter is the separator of key and value and which parameter is the separator of records? In fact, the current implementation is antiintuitive: the joink and joinv methods use the second argument for the record separator while the joinkv uses the third field for that and the second field is used for the key-value separator.

When I compile the latest version (5.10.2) without using anaconda, it successfully compiles and runs the unit tests, but when I try and build it as an anaconda package, I get the following error while it runs make:

/bin/sh ../libtool  --tag=CC   --mode=link $BUILD_PREFIX/bin/x86_64-conda-linux-gnu-cc -Wall -std=gnu99 -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem $PREFIX/include -fdebug-prefix-map=$SRC_DIR=/usr/local/src/conda/miller-5.10.2 -fdebug-prefix-map=$PREFIX=/usr/local/src/conda-prefix -static -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--disable-new-dtags -Wl,--gc-sections -Wl,-rpath,$PREFIX/lib -Wl,-rpath-link,$PREFIX/lib -L$PREFIX/lib -o mlr mlrmain.o cli/libcli.la containers/libcontainers.la stream/libstream.la input/libinput.la dsl/libdsl.la mapping/libmapping.la output/liboutput.la lib/libmlr.la parsing/libdsl.la auxents/libauxents.la -lm
libtool: link: $BUILD_PREFIX/bin/x86_64-conda-linux-gnu-cc -Wall -std=gnu99 -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem $PREFIX/include -fdebug-prefix-map=$SRC_DIR=/usr/local/src/conda/miller-5.10.2 -fdebug-prefix-map=$PREFIX=/usr/local/src/conda-prefix -Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z -Wl,relro -Wl,-z -Wl,now -Wl,--disable-new-dtags -Wl,--gc-sections -Wl,-rpath -Wl,$PREFIX/lib -Wl,-rpath-link -Wl,$PREFIX/lib -o mlr mlrmain.o  -L$PREFIX/lib cli/.libs/libcli.a containers/.libs/libcontainers.a stream/.libs/libstream.a input/.libs/libinput.a dsl/.libs/libdsl.a mapping/.libs/libmapping.a output/.libs/liboutput.a lib/.libs/libmlr.a parsing/.libs/libdsl.a auxents/.libs/libauxents.a -lm
/sc/arion/work/fultob01/conda/envs/py3.9/conda-bld/miller_1636495473849/_build_env/bin/../lib/gcc/x86_64-conda-linux-gnu/9.3.0/../../../../x86_64-conda-linux-gnu/bin/ld: parsing/.libs/libdsl.a(mlr_dsl_wrapper.o): in function `mlr_dsl_parse':
mlr_dsl_wrapper.c:(.text.mlr_dsl_parse+0x103): undefined reference to `mlr_dsl_ParseTrace'

My build script is like so:

#!/bin/sh

./configure --prefix=$PREFIX
make
make check
make install

I have made the gcc toolchain, make and flex available. It also fails when I install gcc, make and flex using Anaconda then make manually.

Do you have any idea what's going on?

Wow, Miller seems like a great command-line tool!

I would love to use it, but there doesn't seem to be a Windows version yet... Could you make/compile one?

I found miller to be very useful with SRE tasks and often use it in Docker containers.

Sadly, there doesn't seem to be a package for Alpine available, a distribution very popular with Docker because of its small footprint. So for now I'm stuck with the large debian and ubuntu images.

Let's fix this: https://wiki.alpinelinux.org/wiki/Creating_an_Alpine_package

Hello,

How would I go about compiling your program with clang?

clang -v
FreeBSD clang version 3.4.1 (tags/RELEASE_34/dot1-final 208032) 20140512
Target: i386-unknown-freebsd10.2
Thread model: posix
Selected GCC installation:

Thanks, Sean

Create a discussion forum where Miller users may talk about Miller. Hosting suggestions:

• Discourse
• Fleep
• FreeLists
• Google Groups
• Keybase Team
• Slack
• Stride

Hi, i have 2 files, both have same format.

mlr count us_leads_woocommerce_2.csv
count=423732

cat us_leads_woocommerce_2.csv | wc -
423732

As you can see above, the numbers aligned. But for a different file, same format:

mlr count us_leads_shopify.csv
count=305141

cat us_leads_shopify.csv | wc -l
971200

As you can see, mlr returned 305,141 and wc -l returned 971,200 (which is the correct result) What can cause this discrepancy?

In Miller 6.5.0, function strftime prints imprecise fractional seconds:

$ # Milliseconds precision 
$ echo unix_timestamp=1454176342.303 | mlr put '$date_and_time = strftime($unix_timestamp, "%FT%H:%M:%3S");'
unix_timestamp=1454176342.303,date_and_time=2016-01-30T17:52:22.302

Expected result: date_and_time=2016-01-30T17:52:22.303

$ # Microseconds precision
$ echo unix_timestamp=1454176342.303 | mlr put '$date_and_time = strftime($unix_timestamp, "%FT%H:%M:%6S");'
unix_timestamp=1454176342.303,date_and_time=2016-01-30T17:52:22.302999

Expected result: date_and_time=2016-01-30T17:52:22.303000

$ # Nanoseconds precision 
$ echo unix_timestamp=1454176342.303 | mlr put '$date_and_time = strftime($unix_timestamp, "%FT%H:%M:%9S");'
unix_timestamp=1454176342.303,date_and_time=2016-01-30T17:52:22.302999973

Expected result: date_and_time=2016-01-30T17:52:22.303000000

$ mlr --version
mlr 6.5.0

I might have reported this behaviour in a comment on an earlier issue or discussion related to strftime or strptime, but I can't find the comment. Anyway, I think this behaviour deserves its own separate issue.

@johnkerl I open an issue, because in your answer you talk about bug and regression.

Thank you

Discussed in https://github.com/johnkerl/miller/discussions/1117

tail -f input.csv | mlr --icsvlite --otsv --ifs "," cat

mlr --icsvlite --opprint --ifs "," cat input.csv

data,stato,codice_regione
5,ITA,08
2,ITA,08
3,ITA,10
```</div>

A next step within Miller would be to have a command-line alternative like mlr --generate-zsh-completions which would create this kind of information automatically

Originally posted by @johnkerl in https://github.com/johnkerl/miller/discussions/1124#discussioncomment-4241025

I've searched the manual if I've missed an option and the problem is so trivial, that I would be surprised no one raised it before:

echo -e 'a,b\n"hello","004.56"' | mlr --icsv --ojson cat

miller 6 gives: [{ "a": "hello", "b": 004.56}]

removing the quotes which should mark the b column value as string so when further reading the value from json the value is read as floating point 4.56

Or is there an other option to force a column to be a string when outputting as json?