OCI Registry As Storage

OCI Registry As Storage

🚧 This project is currently under active development. The API may and will change incompatibly from one commit to another. 🚧

GitHub Actions status Go Report Card GoDoc

ORAS

ORAS Go Library

Using the ORAS Go library, you can develop your own push/pull experience: myclient push artifacts.azurecr.io/myartifact:1.0 ./mything.thang

The package github.com/oras-project/oras-go/pkg/oras can quickly be imported in other Go-based tools that wish to benefit from the ability to store arbitrary content in container registries.

ORAS Go Library Example

Source

package main

import (
	"context"
	"fmt"

	"github.com/oras-project/oras-go/pkg/content"
	"github.com/oras-project/oras-go/pkg/oras"

	"github.com/containerd/containerd/remotes/docker"
	ocispec "github.com/opencontainers/image-spec/specs-go/v1"
)

func check(e error) {
	if e != nil {
		panic(e)
	}
}

func main() {
	ref := "localhost:5000/oras:test"
	fileName := "hello.txt"
	fileContent := []byte("Hello World!\n")
	customMediaType := "my.custom.media.type"

	ctx := context.Background()
	resolver := docker.NewResolver(docker.ResolverOptions{})

	// Push file(s) w custom mediatype to registry
	memoryStore := content.NewMemoryStore()
	desc := memoryStore.Add(fileName, customMediaType, fileContent)
	pushContents := []ocispec.Descriptor{desc}
	fmt.Printf("Pushing %s to %s...\n", fileName, ref)
	desc, err := oras.Push(ctx, resolver, ref, memoryStore, pushContents)
	check(err)
	fmt.Printf("Pushed to %s with digest %s\n", ref, desc.Digest)

	// Pull file(s) from registry and save to disk
	fmt.Printf("Pulling from %s and saving to %s...\n", ref, fileName)
	fileStore := content.NewFileStore("")
	defer fileStore.Close()
	allowedMediaTypes := []string{customMediaType}
	desc, _, err = oras.Pull(ctx, resolver, ref, fileStore, oras.WithAllowedMediaTypes(allowedMediaTypes))
	check(err)
	fmt.Printf("Pulled from %s with digest %s\n", ref, desc.Digest)
	fmt.Printf("Try running 'cat %s'\n", fileName)
}
Owner
OCI Registry As Storage (ORAS)
Tools and libraries to enable leveraging OCI registries for arbitrary artifacts
OCI Registry As Storage (ORAS)
Comments
  • Missing manifest annotations.

    Missing manifest annotations.

    I don't know if I'm doing anything wrong, but I've used oras.WithManifestAnnotations() to push my artefact with some metadata, that worked well because if I curl the repo directly I see it there:

    $ curl -sL -u"user:pass" https://my-repo/v2/oso-plugins/manifests/oso-legacy | jq .
    {
      "schemaVersion": 2,
      "config": {
        "mediaType": "application/vnd.unknown.config.v1+json",
        "digest": "sha256:44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a",
        "size": 2
      },
      "layers": [
        {
          "mediaType": "application/vnd.oci.image.layer.v1.tar",
          "digest": "sha256:8e985dc2c81bd5e8bfafb2858b9a9cb420c02a29d5a96f32eb00f5850556410a",
          "size": 755,
          "annotations": {
            "hook": "chmod +x ~/.oso/bin/oso-legacy",
            "org.opencontainers.image.title": "oso-legacy"
          }
        }
      ],
      "annotations": {
        "caveats": "to get auto-completions do this...",
        "description": "a bridge to the past",
        "shortDescription": "a plugin"
      }
    }
    

    But then when I pull it using desc, artefacts, err := oras.Pull(...) I can loop through the artefacts and get the annotations from each of them, but desc.Annotations is empty...

    The result of fmt.Println(desc.Annotations) is:

    Annotations: map[]
    
  • Feat: Adding discoverer & example copy command

    Feat: Adding discoverer & example copy command

    This is the minimum viable product for the recursive copy/discover. I restored some files that were removed for pkg/oras/copy.go namely pull.go, since it has the semantics I'm looking for in this implementation, and added back the ProviderIngester interface.

    Signed-off-by: Julius Liu [email protected]

  • `Transfer-Encoding: chunked` responses fail

    `Transfer-Encoding: chunked` responses fail

    Hello,

    I have been working on a project using the oras client (v2) against a repository hosted on ECR. We are storing an artifact with quite a lot of individual items (431 layers on the final manifest), which seems to cause ECR to chunk the response:

    # REPO contains the API of the target repository, private in this case
    ❯ curl -X GET -I --user "AWS:$(aws ecr get-login-password)" "${REPO}/manifests/latest"
    HTTP/1.1 200 OK
    Content-Type: application/vnd.oci.image.manifest.v1+json
    Docker-Distribution-Api-Version: registry/2.0
    Sizes:
    Date: Fri, 23 Sep 2022 09:14:00 GMT
    Transfer-Encoding: chunked
    

    The library currently explicitly checks for net/http.Response.ContentLength when validating the response, which is for these cases -1 and errors out: GET "${REPO}/manifests/latest": unknown response Content-Length, coming from registry/remote/repository.go

    I tried to replicate the behavior with the open source registry image, but even with a manifest with >10000 layers it responds non-chunked, so this might unfortunately be a bit annoying to replicate fully locally.

  • Copy API

    Copy API

    This is an implementation of the "copy API". It solves several outstanding issues with the oras go library, with the intent of making it easier to work with, easier to understand, and more flexible.

    • Instead of Push and Pull, there is a single func Copy(). You copy from a ref in one Target to a ref (which may be the same as the first) in another Target
    • Target is suspiciously like remotes.Resolver because, as of now, that is precisely what it is. That is likely to change going forward

    This makes the interface much simpler to use and understand.

    This also opens possibilities like using different URLs or different authentication for different targets. You treat a local bunch of files as a target (from or to) just like a remote registry. Memory, file, registry, oci layout, all are just targets.

    The directory examples/advanced/ contains some good examples of how this is used.

  • Proper naming for UpEdges/DownEdges methods

    Proper naming for UpEdges/DownEdges methods

    Currently, we have an UpEdges method which returns the nodes directly pointing to the current node, and a DownEdges method which returns the nodes directly pointed by the current node. However, the UpEdges / DownEdges names may not be appropriate, and we are considering using other names.

    The current candidates are:

    • UpEdges / DownEdges:
      • Pros: Currently being used.
      • Cons: There is no academic terms for "up edges" and "down edges" of a Directed Acyclic Graph (DAG).
    • Referrers / References:
      • Pros: Closer to the domain.
      • Cons: The potential concern is that we have too many things named references.
    • Parents / Children:
      • Pros: Straightforward.
      • Cons: DAGs are not trees.

    Let's brainstorm for more names and do a vote!

  • [WIP] Re-tooling remotes resolver dependency

    [WIP] Re-tooling remotes resolver dependency

    I am trying to decouple dependencies on containerd, to make the oras-go pkg a bit friendlier to take a dependency on. I'm starting on the resolver code since it is in the center of everything. And I plan on working my way towards images next.

    For implementing the resolver I am following the oci-spec line by line, hence my notes in http.go (don't want to miss details). For auth, I started out with some basic oauth2 flows so that I can work without having to install docker. I haven't put much thought into anything beyond that just yet.

    I have not started debugging this PR, but I'm anxious about getting too far ahead of myself so I wanted to put it up early. I'm pretty happy with the direction it's headed in so far, but I am open to discuss.

  • Feat: Adding discoverer

    Feat: Adding discoverer

    This adds the discoverer interface back to the main branch. Some caveats:

    • Needed to port over some docker code but plan on removing with my re-tooling work

    @sajayantony @shizhMSFT

  • fix: oras pull error `empty response Docker-Content-Digest`

    fix: oras pull error `empty response Docker-Content-Digest`

    See https://github.com/oras-project/oras-go/issues/225 for details, but here's the tldr:

    % oras pull 080565210187.dkr.ecr.us-west-2.amazonaws.com/my.repo:my.tag
    Error: GET "https://080565210187.dkr.ecr.us-west-2.amazonaws.com/v2/my.repo/manifests/my.tag": empty response Docker-Content-Digest
    

    The expected outcome (what this PR claims to fix):

    % oras pull 080565210187.dkr.ecr.us-west-2.amazonaws.com/my.repo:my.tag
    Downloading 3cd22ae067a3
    Downloaded empty artifact
    Pulled 080565210187.dkr.ecr.us-west-2.amazonaws.com/my.repo:my.tag
    Digest: sha256:32aef68c20b6f25d69ecbc2f3f98e28f6f6fdacade75ee879459b61f610583f0
    

    Resolves #225

  • Update Example

    Update Example

    Once we have the least code working, we should update the examples so that people can try themselves.

    Examples (originally from @sajayantony 's list):

    • [x] Download Manifest
    • [x] Download a Blob given the digest
    • [x] Put Manifest
    • [x] Create manifest with Annotation
    • [x] Upload Blob
    • [x] Upload Manifest with blob
    • [x] Copy Artifact
    • [x] Insecure TLS / Plain HTTP
    • [ ] HTTP Client Configuration
  • fix: avoid panic from releasing unacquired semaphore

    fix: avoid panic from releasing unacquired semaphore

    When using this pattern:

    eg, egCtx := errgroup.WithContext(ctx)
    for /* ... */ {
    	limiter.Acquire(ctx, 1)
    	eg.Go(func() error {
    		defer limiter.Release(1)
    		// ...
    		return nil
    	})
    }
    if err := eg.Wait(); err != nil {
    	return err
    }
    

    If ctx is cancelled then Acquire may return an error, which will then cause Release to panic after it's called more times than the semaphore was successfully acquired.

    This change uses the SetLimit functionality which has been added to errgroup.Group and provides a small wrapper around it in the internal/syncutil package to improve the ergonomics of its usage.

  • feat: improve the error message if non-distributable(foreign) layer is not supported

    feat: improve the error message if non-distributable(foreign) layer is not supported

    Some artifacts(e.g., Windows base images) contain layers whose distribution is restricted by license. When these images are pushed to a registry, restricted artifacts are not included.

    For now, oras.Copy and oras.Fetch will fail if the target artifact contains an non-distributable(foreign) layer since the layer blob doesn't exist in the remote registry. We should

    1. By default skip blob copying if a layer is non-distributable
    2. Provide option to enforce copying a non-distributable layer and follows the foreign link for fetching
  • Enable the dependabot for oras-go v1 branch

    Enable the dependabot for oras-go v1 branch

    Per discussion in the #oras slack channel, we need to enable the dependabot for oras-go v1 branch. It can help to bump up the dependencies for oras-go v1 automatically.

    @lucacome Would you be able to help with this configuration?

  • Referrers API support for OCI Layout

    Referrers API support for OCI Layout

  • Enhance `content/oci` for content discovery

    Enhance `content/oci` for content discovery

    Given an OCI layout, the current implementation of oras-go has no means to discover content so that users cannot copy or extract any information from the OCI layout unless they know the entry point.

    Proposal:

    1. Refactor https://github.com/oras-project/oras-go/blob/258bfba7524104a5a76a597184ca0d038f4e3086/registry/repository.go#L51-L66 of the registry.Repository interface as TagFinder.
    2. Let content/oci.Store and content/oci.ReadOnlyStore implement TagFinder.
  • Vote for ORAS-go 1.2.2 release:

    Vote for ORAS-go 1.2.2 release:

    Hi team @oras-project/oras-go-maintainers ,

    Considering ORAS-go v2 is still experimental, ORAS-go v1 is stable and will be maintained by the ORAS community until ORAS v2 GA.

    Currently, ORAS-go v1 is not under active development, the ORAS community might only cut new releases to include necessary bug fixes.

    There were four PRs merged since the ORAS-go 1.2.1 release:

    image

    Meanwhile, @lucacome also requested a new patch release of v1. So I want to request a new release v1.2.2.

    We need at least 4 approvals from 6 @oras-project/oras-go-maintainers to vote for releasing and cutting the tag v1.2.1 based on the v1 branch.

    • [ ] Avi Deitcher (@deitch)
    • [x] Josh Dolitsky (@jdolitsky)
    • [ ] Sajay Antony (@sajayantony)
    • [x] Shiwei Zhang (@shizhMSFT)
    • [x] Steve Lasker (@stevelasker)
    • [ ] Sylvia Lei (@Wwwsylvia)

    Please respond LGTM or REJECT (with reasoning).

  • Remove `@main` from the godoc URLs in README

    Remove `@main` from the godoc URLs in README

    Such as https://github.com/oras-project/oras-go/blob/7dcd0973b194e89a51ac02bec91354fcba1f881a/README.md?plain=1#L29 and https://github.com/oras-project/oras-go/blob/7dcd0973b194e89a51ac02bec91354fcba1f881a/README.md?plain=1#L41-L44

Related tags
πŸ€–πŸ€A tool to test and analyze storage and retrieval deal capability on the Filecoin network.

Dealbot A tool to test and analyze storage and retrieval deal capability on the Filecoin network. Getting Started Clone the repo and build: git clone

Sep 10, 2022
A pure Golang implementation of Rockchip rknand vendor storage interface.

go-rkvendorstorage A pure Golang implementation of Rockchip rknand vendor storage interface. Usage package main import ( "fmt" "github.com/jamesits

Nov 8, 2022
The OCI Service Operator for Kubernetes (OSOK) makes it easy to connect and manage OCI services from a cloud native application running in a Kubernetes environment.

OCI Service Operator for Kubernetes Introduction The OCI Service Operator for Kubernetes (OSOK) makes it easy to create, manage, and connect to Oracle

Sep 27, 2022
An Oracle Cloud (OCI) Pulumi resource package, providing multi-language access to OCI

Oracle Cloud Infrastructure Resource Provider The Oracle Cloud Infrastructure (OCI) Resource Provider lets you manage OCI resources. Installing This p

Dec 2, 2022
Versioned model registry suitable for temporary in-training storage and permanent storage

Cogment Model Registry Cogment is an innovative open source AI platform designed to leverage the advent of AI to benefit humankind through human-AI co

May 26, 2022
registry-tools: Prints image digest from a registry

registry-tools: Prints image digest from a registry

Dec 23, 2021
A REST API for the DN42 registry, written in Go, to provide a bridge between interactive applications and the registry.

dn42regsrv A REST API for the DN42 registry, written in Go, to provide a bridge between interactive applications and registry data. A public instance

Apr 21, 2022
πŸ€– Prune old images on GitHub (ghcr.io) and GitLab (registry.gitlab.com) container registry
πŸ€– Prune old images on GitHub (ghcr.io) and GitLab (registry.gitlab.com) container registry

✨ Prune container images in a CLI way ✨ Prune old images on GitHub (ghcr.io) and GitLab (registry.gitlab.com) Container Registry Getting Started | Des

Dec 15, 2022
The Container Storage Interface (CSI) Driver for Fortress Block Storage This driver allows you to use Fortress Block Storage with your container orchestrator

fortress-csi The Container Storage Interface (CSI) Driver for Fortress Block Storage This driver allows you to use Fortress Block Storage with your co

Jan 23, 2022
Vilicus is an open source tool that orchestrates security scans of container images(docker/oci) and centralizes all results into a database for further analysis and metrics.
Vilicus is an open source tool that orchestrates security scans of container images(docker/oci) and centralizes all results into a database for further analysis and metrics.

Vilicus Table of Contents Overview How does it work? Architecture Development Run deployment manually Usage Example of analysis Overview Vilicus is an

Dec 6, 2022
Podman: A tool for managing OCI containers and pods

Podman: A tool for managing OCI containers and pods Podman (the POD MANager) is a tool for managing containers and images, volumes mounted into those

Jan 1, 2023
A tool that facilitates building OCI images
A tool that facilitates building OCI images

Buildah - a tool that facilitates building Open Container Initiative (OCI) container images The Buildah package provides a command line tool that can

Jan 3, 2023
OCI Image Encryption Package

imgcrypt image encryption library and command line tool Project imgcrypt is a non-core subproject of containerd. The imgcrypt library provides API exe

Jan 5, 2023
Handy little CLI for interacting with OCI data

oci-tool Handy little CLI for interacting with OCI data Installation go get github.com/csweichel/oci-tool I use Gitpod for developing this tool; you s

May 17, 2022
OCI transport plugin for apt-get (i.e., apt-get over ghcr.io)

apt-transport-oci: OCI transport plugin for apt-get (i.e., apt-get over ghcr.io) apt-transport-oci is an apt-get plugin to support distributing *.deb

Nov 1, 2022
Simple, rootless, "FROM scratch" OCI image builder

zeroimage zeroimage some-program is like building the following Docker image: FROM scratch COPY some-program /some-program ENTRYPOINT ["/some-program"

Jun 26, 2022
OCI drive, available from home

OCI Drive ... use your storage with Oracle Object Store Quick Start Make sure you have the Object Storage, bucket and you know the compartment id wher

Nov 10, 2021
Cache oci login token for kubectl

oci-token-cache Cache oci login token. This command cache oci login token into ~/.oci/token-cache.json and re-use for kubectl. Usage Currently, your ~

Nov 20, 2021
Executes an OCI image using firecracker.

oci-image-executor Executes an OCI image using Firecracker. Logs from the executed process (both stdout and stderr) are sent to stdout. Logs from the

Dec 28, 2022
Docker for Your ML/DL Models Based on OCI Artifacts
Docker for Your ML/DL Models Based on OCI Artifacts

English | δΈ­ζ–‡ ORMB is an open-source model registry to manage machine learning model. ORMB helps you manage your Machine Learning/Deep Learning models

Dec 30, 2022