What does this PR do?

Deprecate shell-escape based LocalDockerCompose implementation and provide new docker/compose based one using the Docker API. Introduce a new API that takes context.Context into account. Furthermore it exposes more typed options.

#425

What is this PR doing?

This is the very beginning of an initiative to port the Docker-Compose approach from the Java implementation to Golang.

It simply creates the Local Docker Compose approach (no socat container at this moment), which forces developers to have the docker-compose binary in PATH.

Following Java's approach, a LocalDockerCompose (LDC) object needs to be created with an array of paths to docker-compose.yml files, and an identifier (maybe this second field could be internally created).

It's possible to configure the LDC with a command (i.e. "up -d"), which will be executed after the method Invoke is called.

We have added a convenient Down() method, to be executed after tests run to destroy resources created by docker-compose.

Besides that, we are parsing Docker Compose YAML file to capture service names, and put into an array, so that it's possible to perform operations over the services in the compose file, accessing the compose.Services field.

Why is it important?

This will allow developers to configure more elaborated setups for testing, like an Elasticsearch + Kibana, or Metricbeat + MySQL, as an example, being those services coordinated and scheduled by Docker Compose under the hood, with no need to orchestrate them on the programatic side.

On the other hand, using declarative way of describing services is more familiar to developers (🤞), so the entry barrier could be lower.

How to test it?

I added a few unit tests, which verify that there are no errors after executing the compose, checking that environment variables are properly propagated from compose to the containers, and destroying it in a deferred manner.

Related Issues

Closes #25

What does this PR do?

Although Podman claims to be API compatible with Docker and for instance in #336 the consent seems to be let's wait until all problems are gone :sweat_smile: I thought I'd see what can be done as kind of a compromise.

The problems I could identify (and solve) are:

• The current implementation assumes the default network name is bridge which isn't the case for Podman. Even worse, bridge is not even a valid network name with Podman because it's also a network mode and Podman does not allow the creation of networks conflicting with network modes.
• The current implementation of the reaper has a hard coded path where it expects the Docker socket. Neither DOCKER_HOST env variable nor any other customization the testcontainers docs are describing are honored. This is also an issue with rootless Docker.
• Podman requires at least one stream to be attached when a command is executed. It doesn't hurt to attach stdout and/or stderr but otherwise it breaks the exec wait strategy.

DefaultNetwork option

The DefaultNetwork option allows Podman users to configure the name of the default network. If it's not set the previous auto-detection is applied so no breaking change.

PodmanProvider provider type

Podman does not allow ambiguities between network mode names and network names therefore the default Podman network is called podman unlike the bridge network of Docker. To be able to detect both cases without a breaking change the second ProviderType was introduced. Current test implementations will just behave like before but with this addition users of testcontainers-go can either explicitly set the provider type or setup a GenericProvider to their likes and use that one.

Honoring DOCKER_HOST and TESTCONTAINERS_DOCKER_SOCKET_OVERRIDE

The DOCKER_HOST env variable was already considered when creating the Docker API client. Now its value is propagated (via the context.Context) to the reaper as a fallback if TESTCONTAINERS_DOCKER_SOCKET_OVERRIDE isnt' set. If both env variables are empty the default socket path /var/run/docker.sock will be assumed.

Default NetworkMode if not set

In cases when the NetworkMode isn't set it will be set to bridge to avoid issues with Podman. Docker apparently assumes bridge if nothing's set therefore this shouldn't break anything either.

Parallel tests

To speed up the test suite I marked as many tests as possible to be executed in parallel. Some tests (like the one running in host mode) cannot be executed in parallel because that would result in port conflicts.

Compatibility state

Almost all tests are now compatible with Podman. Some are skipped because they are not compatible with Podman in rootless mode but this would be the same for Docker rootless. Some docker-compose tests are failing in the CI pipeline with Podman 3.x but are running on my local machine with Podman 4.x so I assume they will be compatible eventually.

Changes in test suite

Add Podman pipeline

As stated by @mdelapenya testcontainers-java has a PoC of a Podman pipeline that I 'borrowed' :smile: They are installing Podman 3.x in rootless mode which has some consequences:

• Docker-in-Docker does not work
• the new Netavark/Aardvark network is not yet included

compose_test.go

• Replace CLI calls withAPI calls to not depend on the availability of the docker CLI. This way validations are also possible against the Podman API.
• Handle different naming schemes like <identifier>-<service-name>-<instance-count> vs <identifier>_<service-name>_<instance-count> by filtering with regular expressions

container_test.go

Use full qualified image names (e.g. docker.io/alpine) because some Podman distributions do not fallback to docker.io by default and this way the container image registry becomes explicit.

logconsumer_test.go

Skip test that cannot be executed with rootless Podman because it depends on Docker-in-Docker

docker_test.go

• Use docker.io/nginx:alpine where possible
• Use a custom nginx config opening port 8080 for host mode tests to be able to test them with rootless Podman
• Unify container termination in all test cases and replace defer calls with *testing.T.Cleanup()
• Set new Podman provider type to correctly handle default network bridge vs. podman
• Use full qualified image names (means including registry)

testresources

• Use full qualified image names in all Dockerfiles and docker-compose files
• Add custom nginx highport config

I'm deeply sorry about the total size of the PR. If you have a good idea how to split it into different parts I will look into it! Some parts are for sure not necessary but I thought it would be good for example to unify how images are referenced, how containers are cleaned in test cases any many things more I stumbled upon while I was trying to get all tests working. Also updating all GitHub actions could be moved to another PR but adding a new pipeline with outdated actions seemed weird and having different versions of the actions in different pipelines too so I also unified that :sweat_smile: Let me know what you think and I'm sure we can figure out something :blush:

Canned containers are a good way to have pre-built container that can be re-used and can support a deep implementation with the application they are running.

When I started to do them I thought it was a good idea but now I would like to avoid testcontainers to become full of third-party dependency to download. That's why at the moment I am not merging them.

Why I like canned containers:

1. It is a good way to share the effort we do in order to build a great experience with the library
2. It helps new contributors to get onboard

What I do not like:

1. To many dependencies to download with the library

Do you have any suggestions?! I don't know how go mod works with submodules and if they are supported at all. One idea can be to have a foldering like:

canned/mysql
canned/etcd
canned/kubernetes
canned/postgres

And we should figure out a way to have the dependency only for that directory and not for the root o the project.

Affected issue/pr until now:

https://github.com/testcontainers/testcontainers-go/pull/108 https://github.com/testcontainers/testcontainers-go/pull/107 https://github.com/testcontainers/testcontainers-go/pull/105 https://github.com/testcontainers/testcontainers-go/pull/102 https://github.com/testcontainers/testcontainers-go/pull/98 https://github.com/testcontainers/testcontainers-go/pull/59

I'm attempting to run tests from a container without docker installed. This isn't an issue for ContainerRequests since it's using the API, but compose relies on docker-compose being present on the machine. It would be beneficial to be able to deploy a compose file without the requirement for external dependencies.

I have an implicit dependency on this project and when I run go get -u, I get this error:

go: github.com/testcontainers/[email protected]: parsing go.mod: unexpected module path "github.com/testcontainers/testcontainers-go"

With the older version of go within the local docker container, I found this issue: https://github.com/testcontainers/testcontainers-go/issues/260

I'm updating the docs to fix it.

Why ?

While working on a k3s-container I needed to extract the kubeconfig from the running container and discovered that there is no CopyFileFromContainer available.

What ?

Added CopyFileFromContainer to DockerContainer. Added Unit-test for the new Function.

I wondered if you had considered supporting podman for running containers, as an alternative backend to docker?

Using podman (and buildah for creating container images) lets you get rid of the docker daemon entirely.

There is a go API. This does involve running a podman service: but unlike docker, the service runs as your own userid, with no elevated privileges, and can be started and stopped by the user themselves.

Describe the bug Trying to run a test results in a panic when trying to get mapped ports.

To Reproduce https://github.com/todanni/account/blob/main/internal/repository/repository_test.go

Expected behavior Not panic.

** docker info ** output of the command:

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Build with BuildKit (Docker Inc., v0.5.1-docker)
  scan: Docker Scan (Docker Inc., v0.5.0)

Server:
 Containers: 6
  Running: 0
  Paused: 0
  Stopped: 6
 Images: 7
 Server Version: 20.10.5
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 269548fa27e0089a8b8278fc4fc781d7f65a939b
 runc version: ff819c7e9184c13b7c2607fe6c30ae19403a7aff
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 5.4.72-microsoft-standard-WSL2
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64
 CPUs: 16
 Total Memory: 25.01GiB
 Name: docker-desktop
 ID: K2UZ:4XFZ:UKRM:UXYT:MSZF:OGA3:QDVU:GVOR:EYU5:54DD:Q3J4:KSCK
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No blkio throttle.read_bps_device support
WARNING: No blkio throttle.write_bps_device support
WARNING: No blkio throttle.read_iops_device support
WARNING: No blkio throttle.write_iops_device support

Additional context Output from running go test

❯ go test
2021/03/16 13:17:45 Starting container id: dd41a0e8a506 image: quay.io/testcontainers/ryuk:0.2.3
2021/03/16 13:17:46 Waiting for container id dd41a0e8a506 image: quay.io/testcontainers/ryuk:0.2.3
--- FAIL: TestRunAuthenticationTestSuite (0.28s)
    suite.go:63: test panicked: runtime error: index out of range [0] with length 0
        goroutine 6 [running]:
        runtime/debug.Stack(0xc00014e218, 0xc99160, 0xc000128c60)
                /usr/local/go/src/runtime/debug/stack.go:24 +0x9f
        github.com/stretchr/testify/suite.failOnPanic(0xc0001ce780)
                /home/dannipo/go/pkg/mod/github.com/stretchr/[email protected]/suite/suite.go:63 +0x5b
        panic(0xc99160, 0xc000128c60)
                /usr/local/go/src/runtime/panic.go:965 +0x1b9
        github.com/testcontainers/testcontainers-go.(*DockerContainer).MappedPort(0xc00045e100, 0xddd700, 0xc0005244e0, 0xcede63, 0x8, 0x0, 0x0, 0x3132303200000000, 0x63)
                /home/dannipo/go/pkg/mod/github.com/testcontainers/[email protected]/docker.go:133 +0x485
        github.com/testcontainers/testcontainers-go/wait.(*HostPortStrategy).WaitUntilReady(0xc00011c600, 0xddd6c8, 0xc0005244e0, 0xddd9a0, 0xc00045e100, 0x0, 0x0)
                /home/dannipo/go/pkg/mod/github.com/testcontainers/[email protected]/wait/host_port.go:59 +0x16f
        github.com/testcontainers/testcontainers-go.(*DockerContainer).Start(0xc00045e100, 0xddd6c8, 0xc0000360d0, 0x0, 0x0)
                /home/dannipo/go/pkg/mod/github.com/testcontainers/[email protected]/docker.go:165 +0x315
        github.com/testcontainers/testcontainers-go.(*DockerProvider).RunContainer(0xc0005060f0, 0xddd6c8, 0xc0000360d0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xd00532, ...)
                /home/dannipo/go/pkg/mod/github.com/testcontainers/[email protected]/docker.go:727 +0xeb
        github.com/testcontainers/testcontainers-go.NewReaper(0xddd6c8, 0xc0000360d0, 0xc000152180, 0x24, 0xdcd3a0, 0xc0005060f0, 0x0, 0x0, 0x0, 0x0, ...)
                /home/dannipo/go/pkg/mod/github.com/testcontainers/[email protected]/reaper.go:78 +0x455
        github.com/testcontainers/testcontainers-go.(*DockerProvider).CreateContainer(0xc0005060f0, 0xddd6c8, 0xc0000360d0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xcee503, ...)
                /home/dannipo/go/pkg/mod/github.com/testcontainers/[email protected]/docker.go:534 +0x18c8
        github.com/testcontainers/testcontainers-go.GenericContainer(0xddd6c8, 0xc0000360d0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xcee503, 0x8, ...)
                /home/dannipo/go/pkg/mod/github.com/testcontainers/[email protected]/generic.go:43 +0xb5
        github.com/todanni/authentication/test/container.NewPGContainer(0xcf23fe, 0xf, 0xcea328, 0xcea318, 0xc0000bfd88)
                /mnt/c/Projects/todanni/authentication/test/container/postgres.go:41 +0x3db
        github.com/todanni/authentication/internal/repository.(*AccountRepoTestSuite).SetupSuite(0xc0004385a0)
                /mnt/c/Projects/todanni/authentication/internal/repository/repository_test.go:55 +0x48
        github.com/stretchr/testify/suite.Run(0xc0001ce780, 0xdd4880, 0xc0004385a0)
                /home/dannipo/go/pkg/mod/github.com/stretchr/[email protected]/suite/suite.go:118 +0x5c7
        github.com/todanni/authentication/internal/repository.TestRunAuthenticationTestSuite(0xc0001ce780)
                /mnt/c/Projects/todanni/authentication/internal/repository/repository_test.go:25 +0x51
        testing.tRunner(0xc0001ce780, 0xd1bd78)
                /usr/local/go/src/testing/testing.go:1194 +0xef
        created by testing.(*T).Run
                /usr/local/go/src/testing/testing.go:1239 +0x2b3
FAIL
exit status 1
FAIL    github.com/todanni/authentication/internal/repository   0.287s

Installing Testcontainers v0.9.0 currently fails to work, because Go is automatically pulling in Docker v20.10.1 instead of v17.12.0 and the API has changed.

As such, when I run anything I get:

/Users/sazzer/.gvm/pkgsets/go1.15.6/global/pkg/mod/github.com/testcontainers/[email protected]/docker.go:645:39: not enough arguments in call to p.client.ContainerCreate
	have (context.Context, *container.Config, *container.HostConfig, *network.NetworkingConfig, string)
	want (context.Context, *container.Config, *container.HostConfig, *network.NetworkingConfig, *v1.Platform, string)

I can fix this by running:

$ go mod edit -require github.com/docker/[email protected]+incompatible

But the next time I run go get -u ./... to update dependencies then it all breaks again.

Bumps github.com/docker/compose/v2 from 2.14.2 to 2.15.0.

You can trigger a rebase of this PR by commenting @dependabot rebase.

Bumps google.golang.org/api from 0.105.0 to 0.106.0.

You can trigger a rebase of this PR by commenting @dependabot rebase.

Bumps google.golang.org/api from 0.105.0 to 0.106.0.

You can trigger a rebase of this PR by commenting @dependabot rebase.

Bumps google.golang.org/api from 0.105.0 to 0.106.0.

You can trigger a rebase of this PR by commenting @dependabot rebase.

Bumps google.golang.org/api from 0.105.0 to 0.106.0.

You can trigger a rebase of this PR by commenting @dependabot rebase.

Bumps google.golang.org/api from 0.105.0 to 0.106.0.

You can trigger a rebase of this PR by commenting @dependabot rebase.