L/S tags
Utility and API to manipulate (analyze, synchronize and aggregate) images across different Docker registries.
Example invocation
$ lstags alpine~/^3\\./
<STATE> <DIGEST> <(local) ID> <Created At> <TAG>
ABSENT sha256:9363d03ef12c8c25a2def8551e609f146 n/a 2017-09-13T16:32:00 alpine:3.1
CHANGED sha256:9866438860a1b28cd9f0c944e42d3f6cd 39be345c901f 2017-09-13T16:32:05 alpine:3.2
ABSENT sha256:ae4d16d132e3c93dd09aec45e4c13e9d7 n/a 2017-09-13T16:32:10 alpine:3.3
CHANGED sha256:0d82f2f4b464452aac758c77debfff138 f64255f97787 2017-09-13T16:32:15 alpine:3.4
PRESENT sha256:129a7f8c0fae8c3251a8df9370577d9d6 074d602a59d7 2017-09-13T16:32:20 alpine:3.5
PRESENT sha256:f006ecbb824d87947d0b51ab8488634bf 76da55c8019d 2017-09-13T16:32:26 alpine:3.6
NB! You can specify many images to operate on, e.g: lstags nginx~/^1\\.13/ mesosphere/chronos alpine~/^3\\./
Why would someone use this?
You could use lstags, if you ...
- ... aggregate images from different external registries into your own registry for speed and locality reasons.
- ... compare images present locally with the registry ones (e.g.: know if image tagged "latest" was re-pushed).
- ... continuously pull Docker images from some public or private registry to speed-up Docker run on your system.
How?
... pull Ubuntu 14.04 & 16.04, all the Alpine images and Debian "stretch" to have the latest software to play with:
lstags --pull ubuntu~/^1[46]\\.04$/ alpine debian~/stretch/
... pull and re-push CoreOS-related images from quay.io to your own registry (in case these hipsters will break everything):
lstags -P /quay -r registry.company.io quay.io/coreos/hyperkube quay.io/coreos/flannel
NB! In case you use private registry with authentication, make sure your Docker client knows how to authenticate against it! lstags will reuse credentials saved by Docker client in its config.json file, one usually found at ~/.docker/config.json
Possible image states
lstags distinguishes five states of Docker image:
ABSENT- present in registry, but absent locallyPRESENT- present in registry, present locally, with local and remote digests being equalCHANGED- present in registry, present locally, but with different local and remote digestsLOCAL_ONLY- present locally, absent in registryNOT_FOUND- absent in registry, absent locally, probably does not exist at all
Authentication
You can either:
- rely on
lstagsdiscovering credentials "automagically"🎩 - load credentials from any Docker JSON config file specified
Assume tags
Sometimes registry may contain tags not exposed to any kind of search though still existing. lstags is unable to discover these tags, but if you need to pull or push them, you may "assume" they exist and make lstags blindly try to pull these tags from the registry. To inject assumed tags into the registry query you need to extend repository specification with a = followed by a comma-separated list of tags you want to assume.
e.g. we assume tags v1.6.1 and v1.7.0 exist like this: lstags quay.io/calico/cni=v1.6.1,v1.7.0
Repository specification
Full repository specification looks like this:
[REGISTRY[:PORT]/]REPOSITORY[~/FILTER_REGEXP/][=TAG1,TAG2,TAGn]
You may provide infinite number of repository specifications to lstags
Push prefix
When you [re]push images to your "push" registry, you can control the destination repository path prefix:
- by default, repository path prefix will be auto-generated from the source registry hostname, e.g.:
alpine▶️ /registry/hub/docker/com/localhost:5000/nginx▶️ /localhost/registry.company.com/hype/kubernetes▶️ /registry/company/com/
- passing
--push-prefix=/will push images "as is", with no additional repository path prefix - passing
--push-prefix=/my/prefix/will push images appending/my/prefix/to the repository path - specifying
/my/prefixwithout trailing slash is OK, as long as path would still be formatted correctly by API✨ - passing
--push-prefix=""would trigger "default" behavior with prefix being auto-generated
To fail or not to fail?
By default application exits after encountering any errors. To make it more tolerant to subsequent failures, you may use CLI option -N, --do-not-fail or set environment variable DO_NOT_FAIL=true before running application. HINT: Option -d, --daemon-mode always implies activation of --do-not-fail.
YAML
lstags -f file.yaml
A valid YAML file looks like this (mandatory lstags root key is here to be able to use "shared" YAMLs):
lstags:
repositories:
- busybox
- nginx:stable
- mesosphere/marathon-lb~/^v1/
- quay.io/coreos/awscli=master,latest,edge
- gcr.io/google-containers/hyperkube~/^v1\.(9|10)\./
NB! lstags can load repositories from YAML or from CLI args, but not from both at the same time!
Install: Binaries
https://github.com/ivanilves/lstags/releases
Install: Wrapper
git clone [email protected]:ivanilves/lstags.git
cd lstags
sudo make wrapper
lstags -h
A special wrapper script will be installed to manage lstags invocation and updates.
Install: From source
git clone [email protected]:ivanilves/lstags.git
cd lstags
dep ensure
go build
./lstags -h
NB! I assume you have current versions of Go & dep installed and also have set up GOPATH correctly.
Using it with Docker
docker run --rm -it -v /var/run/docker.sock:/var/run/docker.sock ivanilves/lstags
Usage:
lstags [OPTIONS] REPO1 REPO2 REPOn...
Application Options:
-j, --docker-json= JSON file with credentials (default:
~/.docker/config.json) [$DOCKER_JSON]
-p, --pull Pull Docker images matched by filter (will use
local Docker deamon) [$PULL]
-P, --push Push Docker images matched by filter to some
registry (See 'push-registry') [$PUSH]
-r, --push-registry= [Re]Push pulled images to a specified remote
registry [$PUSH_REGISTRY]
--- OUTPUT WAS CUT HERE TO SAVE SPACE ---
Analyze an image
docker run --rm -it -v /var/run/docker.sock:/var/run/docker.sock ivanilves/lstags alpine~/^3\\./
ANALYZE alpine
FETCHED alpine
-
<STATE> <DIGEST> <(local) ID> <Created At> <TAG>
CHANGED sha256:b40e202395eaec699f2d0c5e01e6d6cb8 76da55c8019d 2017-10-25T23:19:51Z alpine:3.6
ABSENT sha256:d95da16498d5d6fb4b907cbe013f95032 n/a 2017-10-25T23:20:18Z alpine:3.1
ABSENT sha256:cb275b62f789b211114f28b391fca3cc2 n/a 2017-10-25T23:20:32Z alpine:3.2
ABSENT sha256:27af7da847283a947c008592f2b2cd6d2 n/a 2017-10-25T23:20:45Z alpine:3.3
CHANGED sha256:246bbbaa81b28837b64cb9dfc574de958 1a19a71e5d38 2017-10-25T23:20:59Z alpine:3.4
CHANGED sha256:aa96c8dc3815c44d4aceaf1ee7903ce58 37c7be7a096b 2017-10-25T23:21:13Z alpine:3.5
-
Development
You are very welcome to open pull requests to this repository!
To maximize our collaboration efficiency we would humbly ask you to follow these recommendations:
- Please add reasonable description (what?/why?/etc) to your pull request
❗ - Your code should pass CI (CircleCI) and a [pretty liberal] code review
🔍 - If code adds or changes some logic, it should be covered by a unit test
- Please, put meaningful and semantic messages on your commits
🙏
NB! Not a requirement, but a GIF included in PR description would make our world a happier place!
'NORELEASE' branches and commits
We have automatic release system. Every PR merge will create a new application release with a changelog generated from PR branch commits. For the most cases it is OK. However, if you work with things that do not need to be released (e.g. non user-facing changes), you have following options:
- If you don't want to create release from your PR, make it from branch containing "NORELEASE" keyword in its name.
- If you want to prevent single commit from appearing in a changelog, please start commit message with "NORELEASE".
lstags as a single binary or as a Docker container.
API
You may use lstags either as a standalone CLI or as a Golang package inside your own application.
Set up and build PoC application with our v1 API:
make poc-app APP_PATH=../lstags-api
cd ../lstags-api
go build
# run "./lstags-api" binary to see PoC in action (examine main.go first to ensure no "rm -rf /" is there)
- This installs all necessary dependencies and sets up PoC application at the path
../lstags-api/ - We assume you already have recent Golang version installed on your system https://golang.org/dl/
GoDoc
- https://godoc.org/github.com/ivanilves/lstags/api/v1
- https://godoc.org/github.com/ivanilves/lstags/api/v1/collection
- https://godoc.org/github.com/ivanilves/lstags/repository
- https://godoc.org/github.com/ivanilves/lstags/tag
NB! Far more complete API usage example could be found in main.go
