The skopeo sync command can sync images between a SOURCE and a destination.

The purpose of this command is to assist with the mirroring of container images from different docker registries to a single docker registry.

Right now the following transport matrix is implemented:

• docker:// -> docker://
• docker:// -> dir:
• dir: -> docker://

The dir: transport is supported to handle the use case of air-gapped environments. In this context users can perform an initial sync on a trusted machine connected to the internet; that would be a docker:// -> dir: sync. The target directory can be copied to a removable drive that can then be plugged into a node of the air-gapped environment. From there a dir: -> docker:// sync will import all the images into the registry serving the air-gapped environment.

The image namespace is changed during the docker:// to docker:// or dir: copy. The FQDN of the registry hosting the image will be added as new root namespace of the image. For example, the image registry.example.com/busybox:latest will be copied to registry.local.lan/registry.example.com/busybox:latest.

The image namespace is not changed when doing a dir: -> docker:// sync operation.

The alteration of the image namespace is used to nicely scope images coming from different registries (the Docker Hub, quay.io, gcr, other registries). That allows all of them to be hosted on the same registry without incurring in clashes and making their origin explicit.

TODO

I hope you like this feature and the direction it's going. Once we agree on its final design we will update this PR to extend the current test suites.

Future work

Currently sync will keep adding missing content from SOURCE to DESTINATION. It would be nice to add a --delete flag to remove from DESTINATION contents that are no longer available inside of SOURCE. That would be a bit like rsync's--delete` flag.

If wanted, that should be addressed with a separate PR.

Signed-off-by: Flavio Castelli [email protected] Co-authored-by: Marco Vedovati [email protected]

We already use global flags for docker specific stuff. This patch enables --username and --password to be passed down to containers/image to setup docker's registries auth.

Fixes #253

@mtrmac @cyphar PTAL

Signed-off-by: Antonio Murdaca [email protected]

In order to meet achievable deadlines converting from Travis to Cirrus CI, one significant artifact was carried forward (instead of fixing):

Depending on a --privileged container to execute all/most automated checks/tests.

Prior attempts to remove this aspect resulted in several test failures. Fixing the problems was viewed as more time-consuming than simply preserving this runtime environment.

Time has passed, and the code has since moved on. This commit removes the legacy need to execute CI operations in a --privileged container, instead running them directly on the host. At the same time, the necessary test binaries are obtained from the same container used for development/local testing purposes. This ensures the two experiences are virtually always identical.

I am trying to use 'skopeo copy' to copy an image from my local registry to a different registry that I can login to using 'aws ecr get-login' command.

After running the login command I get the credentials saved under $HOME/.docker/config.json.

Running this: docker run -v ~/.docker:/root/.docker:Z --rm --net=host luebken/skopeo skopeo copy docker://rackattack-nas.dc1:5000/kubernetes-manager:bd6c5759f401652fc938a239b73631756b60879f_kubernetes-manager docker://registry.maestro.stratoscale.com/kubernetes-manager:bd6c5759f401652fc938a239b73631756b60879f_kubernetes-manager

gets me this response: time="2017-08-27T14:23:27Z" level=fatal msg="Error writing blob: Error initiating layer upload to kubernetes-manager/blobs/uploads/, status 403"

I entered the container and saw that the the config.json is where it suppose to be and was mounted correctly.

Is there any way I can use 'skopeo copy' using aws ecr credentials?

Thanks!

The commit in this PR introduces a new option to the inspect command which allows users to specify whether they want to disable querying and displaying all available tags for the image being inspected. This has been requested in https://github.com/containers/skopeo/issues/785

The new option is --query-tags which defaults to true (to preserve backward compatibility) and is an optional optional. When set to --query-tags=false the implementation skips querying the tags for the repository and the displayed output will contain an empty RepoTags[] property:

{
    "Name": "docker.io/library/python",
    "Digest": "sha256:5ca194a80ddff913ea49c8154f38da66a41d2b73028c5cf7e46bc3c1d6fda572",
    "RepoTags": [],
    ...

I couldn't find existing tests for this command so I haven't updated/added any. However, I've run a bunch of manual tests (with --query-tags=false, with --query-tags, with --query-tags=true, without --query-tags) and they all have gone fine.

Additionally, from a performance point of view, when not using or when not setting this to false and doing an inspect on a image which is expected to have a large number of tags (for example docker.io/library/python) results in a very noticable and big improvement. For example, the following command without this new option returns in around 15-16 seconds consistently (and displays all available tags):

time bin/skopeo inspect   --override-arch=amd64 --override-os=linux docker://docker.io/library/python
....
real	0m15.884s
user	0m0.173s
sys	0m0.131s

Whereas when run with --query-tags=false for the same image, it returns consistently in around 8-9 seconds (of course with RepoTags: []):

time bin/skopeo inspect --query-tags=false  --override-arch=amd64 --override-os=linux docker://docker.io/library/python
...

real    0m8.444s
user    0m0.153s
sys 0m0.120s

Hello,

in our use cases we need to get only information about image, like labels, but we don't need all repoTags. We have repositories with many tags (1700+) and it is resource consuming to get all tags from registry and just drop that.

Could you please provide a way how to avoid fetching repotags?

Thank you

The format of an image copied from the Docker Hub to docker-archive://my-image is not what is produced by docker save. docker save produces a image that follows this spec. Basically, the image layout is such as:

├── 47bcc53f74dc94b1920f0b34f6036096526296767650f223433fe65c35f149eb.json
├── 5f29f704785248ddb9d06b90a11b5ea36c534865e9035e4022bb2e71d4ecbb9a
│   ├── VERSION
│   ├── json
│   └── layer.tar
├── a65da33792c5187473faa80fa3e1b975acba06712852d1dea860692ccddf3198
│   ├── VERSION
│   ├── json
│   └── layer.tar
├── manifest.json
└── repositories

While the format of the image produced by Skopeo only contains tar.gz file and a manifest.json such as:

sha256:5a132a7e7af11f304041e93efb9cb2a0a7839bccaec5a03cfbdc9a3f5d0eb481
sha256:fd2731e4c50ce221d785d4ce26a8430bca9a95bfe4162fafc997a1cc65682cce
sha256:28a2f68d1120598986362662445c47dce7ec13c2662479e7aab9f0ecad4a7416
sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
sha256:07c86167cdc4264926fa5d2894e34a339ad27f730e8cc81a16cd21b7479e8eac
manifest.json

What is this format? Is there a spec somewhere? Or would it be possible to be compliant with docker save ?

Currently if you're not using Fedora, you're kinda outta luck and you have to build from source. When I see a project written in Go, it's sad to see it just cannot be built directly with go tool, or doesn't offer binary builds.

Travis CI can attach binaries to your releases. For example: https://github.com/ahmetb/govvv/releases

You can use https://github.com/mitchellh/gox to provide cross platform builds in your travis build or so.

Please consider distributing binary builds.

Mainly this is to confirm some changes needed for the podman-py CI setup don't disrupt operations here. Ref:

https://github.com/containers/automation_images/pull/111

Signed-off-by: Chris Evich [email protected]

Service accounts (a.k.a. robots) in quay.io are forcably namespaced to the user or orginization under which they are created. Therefore, it is impossible to use a common login/password to push images for both skopeo and containers namespaces. Worse, because the authentication is recorded against quay.io, multiple login sessions are required.

Fix this by adding a function definition which verifies non-empty username/password arguments, before logging in. Call this function as needed from relevant targets, prior to pushing images.

Signed-off-by: Chris Evich [email protected]

Adding skopeo spec file to repo to provide public access to files required to build rpm. Location in repo follows buildah's existing convention/location.

Spec file is from the srpm from the following repo with the commitid changed to REPLACEWITHCOMMITID to match buildah .spec convention: https://cbs.centos.org/repos/virt7-container-common-candidate/source/SRPMS/

Please advise if there is an updated version available or if there is an alternate location to access the spec.

Signed-off-by: pixdrift [email protected]

Migrate our OSX build to a M1 instance, since Cirrus is sunsetting Intel-based macOS instances.

Signed-off-by: Ashley Cui [email protected] (cherry picked from commit b5ac534960bd4188f7fd847cec3225f55714abc4) Signed-off-by: Lokesh Mandvekar [email protected]

For a use case I have, I'd like to execute a skopeo binary from inside a container image, but have it fetch data from containers-storage: which requires entering the host mount namespace.

Several low-level commands have gradually gained support for calling setns(); e.g. there's now mount -N: -N, --namespace <ns> perform mount in another namespace.

The key benefit of this is that it's performed after dynamic linking is done, so assuming no further external binaries are run it avoids a host dependency.

But a general well known problem with setns() and Go is that the runtime will happily spawn threads in the background to service goroutines which may not propagate the namespace.

I came across https://cs.github.com/containers/podman/blob/864288b8dabbe3eb89854b737cc7fbd93077aa1e/libpod/container_copy_linux.go?q=org%3Acontainers+setns+lang%3Ago#L17 which seems related.

Thinking about this, I wonder if we could add support to containers/storage for having it take an explicit mount namespace and perform operations there?

(I tried the below code, which worked for skopeo inspect --mount-namespace 1 containers-storage:docker.io/library/busybox but skopeo copy --mount-namespace 1 containers-storage:docker.io/library/busybox oci:/tmp/busybox fails trying to access some files, I think because the accesses are running on threads spawned before we were able to unshare...so to do this right we'd need to have the main entrypoint be either C or Rust)

diff --git a/cmd/skopeo/main.go b/cmd/skopeo/main.go
index 3f8a9621..af8132c8 100644
--- a/cmd/skopeo/main.go
+++ b/cmd/skopeo/main.go
@@ -3,6 +3,7 @@ package main
 import (
 	"context"
 	"fmt"
+	"os"
 	"strings"
 	"time"
 
@@ -13,6 +14,7 @@ import (
 	"github.com/containers/storage/pkg/reexec"
 	"github.com/sirupsen/logrus"
 	"github.com/spf13/cobra"
+	"golang.org/x/sys/unix"
 )
 
 // gitCommit will be the hash that the binary was built from
@@ -27,6 +29,7 @@ type globalOptions struct {
 	policyPath         string                  // Path to a signature verification policy file
 	insecurePolicy     bool                    // Use an "allow everything" signature verification policy
 	registriesDirPath  string                  // Path to a "registries.d" registry configuration directory
+	mountns            string                  // PID or path to specified mount namespace
 	overrideArch       string                  // Architecture to use for choosing images, instead of the runtime one
 	overrideOS         string                  // OS to use for choosing images, instead of the runtime one
 	overrideVariant    string                  // Architecture variant to use for choosing images, instead of the runtime one
@@ -83,6 +86,7 @@ func createApp() (*cobra.Command, *globalOptions) {
 	rootCommand.PersistentFlags().BoolVar(&opts.debug, "debug", false, "enable debug output")
 	rootCommand.PersistentFlags().StringVar(&opts.policyPath, "policy", "", "Path to a trust policy file")
 	rootCommand.PersistentFlags().BoolVar(&opts.insecurePolicy, "insecure-policy", false, "run the tool without any policy check")
+	rootCommand.PersistentFlags().StringVar(&opts.mountns, "mount-namespace", "", "Enter target mount namespace, specified via PID or magic link /proc/<pid>/ns/mnt")
 	rootCommand.PersistentFlags().StringVar(&opts.registriesDirPath, "registries.d", "", "use registry configuration files in `DIR` (e.g. for container signature storage)")
 	rootCommand.PersistentFlags().StringVar(&opts.overrideArch, "override-arch", "", "use `ARCH` instead of the architecture of the machine for choosing images")
 	rootCommand.PersistentFlags().StringVar(&opts.overrideOS, "override-os", "", "use `OS` instead of the running OS for choosing images")
@@ -121,6 +125,25 @@ func (opts *globalOptions) before(cmd *cobra.Command) error {
 	if opts.tlsVerify.Present() {
 		logrus.Warn("'--tls-verify' is deprecated, please set this on the specific subcommand")
 	}
+	if opts.mountns != "" {
+		if !strings.HasPrefix(opts.mountns, "/") {
+			opts.mountns = fmt.Sprintf("/proc/%s/ns/mnt", opts.mountns)
+		}
+
+		// AIUI, we need to unshare() because the process is already threaded
+		if err := unix.Unshare(unix.CLONE_NEWNS); err != nil {
+			return fmt.Errorf("failed to unshare mount namespace: %w", err)
+		}
+		fd, err := os.Open(opts.mountns)
+		if err != nil {
+			return err
+		}
+		defer fd.Close()
+
+		if err := unix.Setns(int(fd.Fd()), unix.CLONE_NEWNS); err != nil {
+			return fmt.Errorf("failed to enter mount namespace: %w", err)
+		}
+	}
 	return nil
 }
 

This PR contains the following updates:

| Package | Type | Update | Change | |---|---|---|---| | gopkg.in/yaml.v2 | require | major | v2.4.0 -> v3.0.1 |

Release Notes

Configuration

📅 Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).

🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.

♻ Rebasing: Whenever PR is behind base branch, or you tick the rebase/retry checkbox.

🔕 Ignore: Close this PR and you won't be reminded about this update again.

• [ ] If you want to rebase/retry this PR, check this box

This PR has been generated by Mend Renovate. View repository job log here.

$ podman images nodeimage

REPOSITORY TAG IMAGE ID CREATED SIZE localhost/nodeimage 9.arm64 95d9fe375ed3 17 hours ago 888 MB localhost/nodeimage 9.amd64 a523734ebdb1 17 hours ago 864 MB localhost/nodeimage 9 410c25c08493 17 hours ago 1.09 kB

$ podman inspect localhost/nodeimage:9.arm64|grep Digest

      "Digest": "sha256:6db6e95b26c56459ffd63539e8f7b1418d15a8ba35e979bd61374779c20b80d3",

$ podman inspect localhost/nodeimage:9.amd64|grep Digest

      "Digest": "sha256:e4cb67eb7189c6d7e02d7f394f3d28ea50af723ea853b984618b57d96aba4940",

$ podman manifest inspect localhost/nodeimage:9|grep sha256

        "digest": "sha256:e4cb67eb7189c6d7e02d7f394f3d28ea50af723ea853b984618b57d96aba4940",

        "digest": "sha256:6db6e95b26c56459ffd63539e8f7b1418d15a8ba35e979bd61374779c20b80d3",

$ podman manifest inspect localhost/nodeimage:9

{ "schemaVersion": 2, "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json", "manifests": [ { "mediaType": "application/vnd.oci.image.manifest.v1+json", "size": 1682, "digest": "sha256:e4cb67eb7189c6d7e02d7f394f3d28ea50af723ea853b984618b57d96aba4940", "platform": { "architecture": "amd64", "os": "linux" } }, { "mediaType": "application/vnd.oci.image.manifest.v1+json", "size": 1682, "digest": "sha256:6db6e95b26c56459ffd63539e8f7b1418d15a8ba35e979bd61374779c20b80d3", "platform": { "architecture": "arm64", "os": "linux", "variant": "v8" } } ] }

$ skopeo --debug copy --dest-registry-token $gcr_auth_token containers-storage:localhost/nodeimage:9 docker://us.gcr.io/localhost/nodeimage:9 --multi-arch all

DEBU[0000] [graphdriver] trying provided driver "overlay"

DEBU[0000] Cached value indicated that overlay is supported

DEBU[0000] Cached value indicated that overlay is supported

DEBU[0000] Cached value indicated that metacopy is being used

DEBU[0000] Cached value indicated that native-diff is not being used

INFO[0000] Not using native diff for overlay, this may cause degraded performance for building images: kernel has CONFIG_OVERLAY_FS_REDIRECT_DIR enabled

DEBU[0000] backingFs=xfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=true

DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/run/containers/storage:overlay.mountopt=nodev,metacopy=on]localhost/nodeimage:9"

DEBU[0000] Using registries.d directory /etc/containers/registries.d

DEBU[0000] Loading registries configuration "/etc/containers/registries.conf"

DEBU[0000] Loading registries configuration "/etc/containers/registries.conf.d/000-shortnames.conf"

DEBU[0000] Loading registries configuration "/etc/containers/registries.conf.d/001-rhel-shortnames.conf"

DEBU[0000] Loading registries configuration "/etc/containers/registries.conf.d/002-rhel-shortnames-overrides.conf"

DEBU[0000] Found credentials for us.gcr.io/localhost/nodeimage in credential helper containers-auth.json in file /run/user/0/containers/auth.json

DEBU[0000] Lookaside configuration: using "default-docker" configuration

DEBU[0000] Using "sigstore-staging" file:///var/lib/containers/sigstore

DEBU[0000] Looking for TLS certificates and private keys in /etc/docker/certs.d/us.gcr.io

DEBU[0000] Sigstore attachments: using "default-docker" configuration

DEBU[0000] Using blob info cache at /var/lib/containers/cache/blob-info-cache-v1.boltdb

DEBU[0000] Source is a manifest list; copying all instances

Getting image list signatures

DEBU[0000] Manifest list has MIME type application/vnd.oci.image.index.v1+json, ordered candidate list [application/vnd.oci.image.manifest.v1+json, application/vnd.docker.distribution.manifest.v2+json, application/vnd.oci.image.index.v1+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.docker.distribution.manifest.v1+prettyjws, application/vnd.docker.distribution.manifest.v1+json]

DEBU[0000] ... will use the original manifest list type, and then try [application/vnd.docker.distribution.manifest.list.v2+json]

Copying 2 of 2 images in list

DEBU[0000] Copying instance sha256:e4cb67eb7189c6d7e02d7f394f3d28ea50af723ea853b984618b57d96aba4940 (1/2) Copying image sha256:e4cb67eb7189c6d7e02d7f394f3d28ea50af723ea853b984618b57d96aba4940 (1/2)

FATA[0000] copying image 1/2 from manifest list: determining manifest MIME type for containers-storage:[overlay@/var/lib/containers/storage+/run/containers/storage:overlay.mountopt=nodev,metacopy=on]localhost/nodeimage:9@410c25c084931697396bd851fb1c726d1d7d8cfb7de98c91ed1bd10e8b3226ea: reading manifest for image instance "sha256:e4cb67eb7189c6d7e02d7f394f3d28ea50af723ea853b984618b57d96aba4940": locating item named "manifest-sha256:e4cb67eb7189c6d7e02d7f394f3d28ea50af723ea853b984618b57d96aba4940" for image with ID "410c25c084931697396bd851fb1c726d1d7d8cfb7de98c91ed1bd10e8b3226ea" (consider removing the image to resolve the issue): file does not exist

In the bootc/ostree-container effort, I am trying to enforce signatures being enabled by default. The thing is, we kind of say that e.g. podman run <some image from docker hub or whatever> is "secure" - in the sense I'm using the word, we can and do fix security problems we find (mostly in the kernel) in a relatively timely fashion.

But booting a container (or running with --privileged as well as some more subtle options) completely change that.

As part of the proxy, I'd like to add an OpenImageWithRequiredSignatures API that requires that the remote image is signed in some way configured in containers-policy.json - IOW that the policy for fetching the image does not fall through to insecureAcceptAnything.

(I think it would make sense to also add podman pull --sigpolicy=required or so)

When I looked at this, it seemed feasible but would require some changes in c/image. Let me know if you have any thoughts.