dyff verison 1.5.4 on linux

I configure dyff to be the external diff for kubectl export KUBECTL_EXTERNAL_DIFF="dyff between --omit-header --set-exit-code" Then I ask kubectl to show the diff (before applying) of kustomize output kubectl diff -k . The error is

Error occurred
unknown command "/tmp/LIVE-1245488198" for "dyff"

If I instead extract the current deployment and the result of kubectl kustomize . I can show the diff just fine.

Thanks.

This hopefully solves issue 232. When comparing two YAMLs with a document removed and using the --exclude flag I noticed that dyff threw a panic. I think this is because those Diff objects don't have a Path set (must be nil) so when we do --exclude (or any of the other exclude/filter flags) it will panic trying to produce the report. Let me know if there is anything else you'd like me to do :)

Diffing two files that contain the same set of documents, but in different orders shows a diff. At least with e.g. Kubernetes resources we can identify the document uniquely using kind and metadata.name, which would allow dyff to pair up documents.

I realise this is probably related to #16 as it's dealing with multiple documents.

a.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: x
---
apiVersion: v1
kind: Service
metadata:
  name: y

b.yaml:

apiVersion: v1
kind: Service
metadata:
  name: y
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: x

dyff between --ignore-order-changes --detect-kubernetes a.yaml b.yaml:

     _        __  __
   _| |_   _ / _|/ _|  between a.yaml, two documents
 / _' | | | | |_| |_       and b.yaml, two documents
| (_| | |_| |  _|  _|
 \__,_|\__, |_| |_|   returned six differences
        |___/

apiVersion  (document #1)
  ± value change
    - apps/v1
    + v1

kind  (document #1)
  ± value change
    - Deployment
    + Service

metadata.name  (document #1)
  ± value change
    - x
    + y

apiVersion  (document #2)
  ± value change
    - v1
    + apps/v1

kind  (document #2)
  ± value change
    - Service
    + Deployment

metadata.name  (document #2)
  ± value change
    - y
    + x

I have some issue installing on macOS with m1:

$ curl --silent --location https://git.io/JYfAY | sudo bash
Unsupported operating system darwin or machine type arm64: Please check https://github.com/homeport/dyff/releases manually.

arm64 is supported in goreleaser https://github.com/goreleaser/goreleaser/pull/1956 with go 1.16 version

fixes #171

If implemented, an inline diff will be processed for multilines strings, and diff hunks will be highlighted.

Signed-off-by: Soule BA [email protected]

I've just tried to use this as a git-dffer and get the following errors having simply copy-pasted from the readme.

I just created a small repo to test this in:

$ git log
commit 93fb76bc6324c8a45e3764478634e1977ecef5ba (HEAD -> master)
Author: Avi Greenbury <[email protected]>
Date:   Thu Dec 23 14:22:35 2021 +0000

    add stuff

commit 6507158842315aa75213cc41a6305ac475b11086
Author: Avi Greenbury <[email protected]>
Date:   Thu Dec 23 14:22:13 2021 +0000

    testing

$ git show
commit 93fb76bc6324c8a45e3764478634e1977ecef5ba (HEAD -> master)
Author: Avi Greenbury <[email protected]>
Date:   Thu Dec 23 14:22:35 2021 +0000

    add stuff

diff --git a/test.yml b/test.yml
index 2eb7df4..acf0792 100644
--- a/test.yml
+++ b/test.yml
@@ -3,3 +3,5 @@ test: true
 what:
   - dyff
   - diff
+  - git-diff
+  - things

$ git config --local diff.dyff.command 'dyff_between() { dyff --color on between --omit-header "$2" "$5"; }; dyff_between'

$ echo '*.yml diff=dyff' >> .gitattributes

$ git log --ext-diff -u
commit 93fb76bc6324c8a45e3764478634e1977ecef5ba (HEAD -> master)
Author: Avi Greenbury <[email protected]>
Date:   Thu Dec 23 14:22:35 2021 +0000

    add stuff

╭ Error: failed to load input files
│ unable to load data from
│ /tmp/YUaX0G_test.yml: Get
│ "/tmp/YUaX0G_test.yml": unsupported protocol scheme ""
╵
fatal: external diff died, stopping at test.yml

$ git show --ext-diff HEAD
commit 93fb76bc6324c8a45e3764478634e1977ecef5ba (HEAD -> master)
Author: Avi Greenbury <[email protected]>
Date:   Thu Dec 23 14:22:35 2021 +0000

    add stuff

╭ Error: failed to load input files
│ unable to load data from
│ /tmp/hbpFpg_test.yml: Get
│ "/tmp/hbpFpg_test.yml": unsupported protocol scheme ""
╵
fatal: external diff died, stopping at test.yml
 

This is installed from Snap on ubuntu 20.04; dyff version 1.4.6 and git 2.25.1

seems like kubectl is expecting the external diff variable to be just the executable name? is there a workaround? thanks

% export KUBECTL_EXTERNAL_DIFF="dyff between --omit-header --set-exit-code"
% kubectl diff -f foo.yaml
error: failed to run "dyff between --omit-header --set-exit-code": executable file not found in $PATH

% kubectl version Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.7", GitCommit:"1dd5338295409edcfff11505e7bb246f0d325d15", GitTreeState:"clean", BuildDate:"2021-01-13T13:23:52Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"darwin/amd64"} Server Version: version.Info{Major:"1", Minor:"17+", GitVersion:"v1.17.17-gke.4900", GitCommit:"2812f9fb0003709fc44fc34166701b377020f1c9", GitTreeState:"clean", BuildDate:"2021-03-19T09:19:27Z", GoVersion:"go1.13.15b4", Compiler:"gc", Platform:"linux/amd64"}

I'm using dyff for comparing k8s manifests which can be quite huge. So I think it'll be useful if we can filter out results using some pattern like below.

dyff between file1.json file2.json --filter deployment.spec.template.spec.containers.env

PS: Great job with the visualisation 🥇

I am comparing YAML files that have embedded lists that use key identifiers other than "name", "key", id". It would be useful to be able to configure list identifiers through the settings (in a more flexible way than settings.KubernetesEntityDetection).

I think that setting "branch" as an identifier will solve a problem I am having with the following false positive: (sequence ordering is disabled):

repository.protectedbranches
  - two list entries removed:
    - allowedtoforcepush: false
      allowedtomerge:
      - name: vcdc-developers
        level: 40
      allowedtopush:
      - name: "No one"
        level: 0
      - name: "Build Systems"
        level: 40
      branch: main
      codeownerapproval: true
    - allowedtoforcepush: false
      allowedtomerge:
      - name: vcdc-developers
        level: 40
      allowedtopush:
      - name: "No one"
        level: 0
      - name: "Build Systems"
        level: 40
      branch: go1.17-boringcrypto
      codeownerapproval: true
    
  
  + two list entries added:
    - branch: main
      allowedtomerge:
      - name: vcdc-developers
        level: 40
      allowedtopush:
      - name: "Build Systems"
        level: 40
      - name: "No one"
        level: 0
      allowedtoforcepush: false
      codeownerapproval: true
    - branch: go1.17-boringcrypto
      allowedtomerge:
      - name: vcdc-developers
        level: 40
      allowedtopush:
      - name: "Build Systems"
        level: 40
      - name: "No one"
        level: 0
      allowedtoforcepush: false
      codeownerapproval: true

See for example:

in actuality, those fields are like:

  string x_forwarded_host = 53;
  string worker_status = 54;
  uint64 worker_cpu_time_micro = 55;

I'll take a look at raising a fix myself, but just wanted to raise the issue :)

Hi,

This tool is really good and it helped me a lot. My request is to actually run the diff logic on the "expanded' version after expanding all the anchors and aliases.

Thx Gil

This bug has morphed a bit. I had originally tried to do --exclude-regexp metadata\.annotations\.checksum/.*, mistakenly thinking that the dots were part of a string that needed to be escaped in order not to match too many strings, and not the separators between path segments.

But that raises the question of how those two syntaxes interact. For instance, if I want to exclude any item whose path starts with metadata and ends with generation, how do I express that? It seems that metadata.*generation works, but possibly by accident? At least, I wouldn't know how to filter

metadata:
  something:
    something-else:
      generation: 5

distinctly from

metadata.something.something-else.generation: 5

or

metadataXsomethingXsomething-elseXgeneration: 5

There are a similar set of ambiguous patterns using / as the separator, too.

In tracking this down, I noticed that the tests are also missing examples which contain regular expression metacharacters (not that they would have caught this particular problem).

Ambiguity is probably fine 99% of the time, but even having that much documented would be really handy. Unfortunately, that seems like it's getting into manpage territory ....

I'll note with a crazy look in my eye that helm uses (AFAIK completely undocumented) vertical tab (\v) as a separator internally, so if you have, say, a datadog repo configured, which has datadog, datadog-crds, and datadog-operator charts, you can search for \vdatadog/datadog\v to find just the first. Perhaps it might make sense to treat \v as a segment separator here, too, assuming it's not possible to use that character in a YAML key.

I have files with these contents:

top: |
  object:
    aa_aaa: true
    bbb_bb_bbbbbbb: true
    ccccccc_ccccccc_ccccccc: true
  is_longenough123:
    # Stupid comment one
    # Stupid comment two
    # Stupid comment three

and

top: |
  is_longenough123:
    - name
  object:
    aa_aaa: true
    bbb_bb_bbbbbbb: true
    ccccccc_ccccccc_ccccccc: true

If I dyff them, I get

     _        __  __
   _| |_   _ / _|/ _|  between dyffbug-a.yaml
 / _' | | | | |_| |_       and dyffbug-b.yaml
| (_| | |_| |  _|  _|
 \__,_|\__, |_| |_|   returned one difference
        |___/

top
  ± value change in multiline text (three inserts, one deletion)
      is_longenough123:
      object:
      object:
      object:
        aa_aaa: true
      object:
        bbb_bb_bbbbbbb: true
      object:
        ccccccc_ccccccc_ccccccc: true
      is_longenough123:
        # Stupid comment one
        # Stupid comment two
        # Stupid comment three

which I'm pretty sure is not right.

The issue is very sensitive to the contents of the string. I haven't been able to get it to reproduce with fewer than three lines in both object and is_longenough123, or changing the keys very much. The keys in object can't be shortened (or at least not much; I haven't tried all possible combinations) either, which is weird. Other than the length, it doesn't seem like the key names matter.

For instance, if I drop the ccccccc_ccccccc_ccccccc key, I get what I'd expect:

top
  ± value change in multiline text (one insert, one deletion)
      is_longenough123:
        - name
      object:
        aa_aaa: true
        bbb_bb_bbbbbbb: true
      is_longenough123:
        # Stupid comment one
        # Stupid comment two
        # Stupid comment three

(Screenshots are necessary, since the colors are what actually indicate what dyff thinks is being added or removed, and the problem goes away with --color=no. The blue bits are just iTerm2 being helpful.)

From my understanding is that --exclude only works for specifing the key name.

Would it be possbile to exclude on the value?

For example:

from.yaml:

some_key:
  clientID: aaa49a1-9162-bbb6-bf06-d7c787ccc

to.yaml

some_key:
  clientID: DRY_RUN

dyff between from.yaml to.yaml --exclude-value DRY_RUN

Should return no differences.

I follow the instructions to install dyff through homebrew and config it to git.

brew install homeport/tap/dyff

git config --local diff.dyff.command 'dyff_between() { dyff --color on between --omit-header "$2" "$5"; }; dyff_between'
echo '*.yml diff=dyff' >> .gitattributes
echo '*.yaml diff=dyff' >> .gitattributes

But when I try to view the diff, dyff shows the following error.

git log --ext-diff -u
Date:   Mon Aug 22 09:10:40 2022 +0800

    Use dyff to restructure the dry run result
    
    Signed-off-by: R0CKSTAR <[email protected]>

diff --git a/Makefile b/Makefile
index 8b3491d..403161d 100644
--- a/Makefile
+++ b/Makefile
@@ -60,6 +60,7 @@ go-test:
 0200: build
        bin/kustohelmize create --from=test/testdata/0200_sample.yaml test/output/0200/mychart
        helm install --dry-run xyz test/output/0200/mychart > test/output/0200/mychart.yaml
+       dyff yaml -i --restructure test/output/0200/mychart.yaml
 
 ##@ Tools
 

LAST DEPLOYED  (document #1)
  ± value change
    - Mon Aug 22 08:56:48 2022
    + Mon Aug 22 09:09:58 2022

╭ Error: failed to load input files
│ unable to parse data from
│ /var/folders/p9/dbq3q88x7c39y7tmwqgj6yjr0000gn/T//HsEOwh_xyz-yourchart-deployment.yaml:
│ yaml: line 6: did not find expected node content
╵
fatal: external diff died, stopping at test/output/0200/mychart/templates/xyz-yourchart-deployment.yaml

BTW, dyff yaml -i --restructure test/output/0200/mychart.yaml doesn't work either. mychart.yaml can be found at https://github.com/yeahdongcn/kustohelmize/blob/main/test/output/0200/mychart.yaml

Hey all, thanks for this great tool!

Can you please help me with the following questions?:

• How can I get diff types from YAML files?
• Is there a set of categories for this? Docs?

I have been looking for docs for this, but I didn't find it.

Thank you