Bug Report

• Concourse version: 3.1.0
• Deployment type (Docker):
• Infrastructure/IaaS: Kubernetes

After upgrade to 3.1.0 all git and time resources are failing checks with:

runc create: exit status 1: container_linux.go:264: starting container process caused "process_linux.go:339: container init caused \"rootfs_linux.go:57: mounting \\\"/worker-state/3.1.0/assets/bin/init\\\" to rootfs \\\"/worker-state/volumes/live/26e7c69d-69fc-4f0f-507d-4b30c461a78f/volume\\\" at \\\"/worker-state/volumes/live/26e7c69d-69fc-4f0f-507d-4b30c461a78f/volume/tmp/garden-init\\\" caused \\\"open /worker-state/volumes/live/26e7c69d-69fc-4f0f-507d-4b30c461a78f/volume/tmp/garden-init: permission denied\\\"\""

Other resources seems to check fine

Motivation

There are so many credentials flowing through a Concourse server at the moment that it's keeping me up at night. Some of this risk can be mitigated by fetching the secrets as late as possible and using pipeline templates to insert them into the pipeline. This at least means that you'll never have the pipeline secrets stored on local workstations.

This isn't good enough.

Credentials are still stored in the Concourse database, aren't internally rotated, and do not have any audit trail as to which jobs and builds used them. Let's fix these things.

Goals

• Build credentials are never stored on a disk (apart from in Vault where they are stored encrypted).
• Build credentials are never stored locally when setting the pipeline using fly.
• An audit trail of which build used a secret and when it used it should be visible.

Proposal

I don't want to have to build a secret management system. This means that we're looking to use one of the existing ones. There are a few to choose from in this regard but I'd like to go forward by using Vault. Vault seems to strike the right balance of confidence in implementation, good technical decisions, and generically usable. I'm not ruling out the ability to use a different secret store in the future. In fact, a good implementation of this proposal would allow pluggable secret stores with minimal effort.

So, what would this look like?

resources:
- name: s3-file
  type: s3
  source:
    access_key_id: {{!vault.s3-file.access_key_id}}
    secret_access_key: {{!vault.s3-file.secret_access_key}}

Here we have a snippet of pipeline configuration. Everything looks normal except the secrets on the last 2 lines. This would be a new syntactic convention that the regular templating would ignore and pass directly up to the server. When the server needed to do something that required a realized value of one of these references it would fetch it as late as possible from the server and then handle revoking it once it isn't required anymore. This means that (assuming Vault is set up correctly) each check run and each build would get a single use credential that would become invalid immediately. This is rad. :100:

The syntax shown above is not finalized but needs to have a few components:

• vault refers to the name of the secret store from which the credential should be fetched. This would allow multiple stores per Concourse server as long as they were all named uniquely. A secret store should probably belong to a team rather than being global.
• s3-file refers to the name of the secret inside Vault.
• access_key_id and secret_access_key refer to the attributes on the secret.

Open Questions

• What if the server goes down in the middle of using one of these secrets. Who revokes it?
• How does Concourse authenticate with the secret store? Does this just move the problem?

Risks

• Yet another thing to deploy next to Concourse in order to use the feature. This moves away from secure by default. This could be partially mitigated by making it easy for people to get started with Vault. (BOSH release, adding something to the monolithic binaries)

When using the new version: every configuration for a get task, it is possible to arrive at a state where you have multiple builds of the same job running at the same time. If an earlier build fails, there is no way to re-trigger it with the same set of inputs. We haven't been able to determine a useful workaround; setting serial: true doesn't really help in this scenario, because the next build will start as soon as the first one fails.

It would be helpful if there were a way to re-trigger the job with the same inputs as a particular build (failed or otherwise).

Let us know if you need more details on this scenario or our desired fix. Thanks!

@davewalter and @rmasand

Proposed changes

1. Rename in and out scripts to get and put. These are confusingly named something different because a long time ago jobs just had _in_puts and _out_puts.
2. Run ./check, ./get, and ./put, and ./info (see next point) rather than /opt/resource/X. Having them in /opt is a bit Linux-specific and assumes containers have their own chroot, which is not the case on Darwin or Windows.
3. Add a info script which prints the resource's API version, e.g. {"version":"2.0"}. This will start at 2.0. If /info does not exist we'll execute today's resource interface behavior.
4. Change get to not emit the version or metadata. This was from when in didn't know what it was fetching until it was done. It should now be an error if it can't fetch the requested version. We'll also move all metadata collection to check (see point 8).
5. Add a delete action, which looks like put (can be given files and params to determine what to delete) but returns the set of versions that it deleted. This is to support bulk deletes, e.g. to garbage collect intermediate artifacts after a final build is shipped.
6. Remove the source and destination arguments passed to get and put - just make it the working directory.
7. Give check an official scratch space, which is the current working directory. No real semantics change here, just consistency with get and put, and potential for future behavior like reusing the scratch space but not reusing check containers for extended periods. Note: we've introduced /scratch recently so this change may just mean making that the work-dir.
8. Move all metadata emitting to check, so that it's always present. The original thought was that metadata collection may be expensive, but so far we haven't seen that to be the case.
9. Change put to emit an array of versions, rather than just one, and without metadata. Technically the git resource may push many commits, so this is necessary to track them all as outputs of a build. This could also support batch creation. Notably, the versions emitted by put are ordered chronologically, so that the dependent get knows to fetch the latest version. We would save them internally, along with an association with the build, and rely on check to determine the final ordering and metadata, being careful to not leave gaps in the version history (i.e. other commits pushed slightly before the put registered the build's).
10. Change put to write to a specified file, rather than stdout, so that we don't have to be attached to process its response. This is one of the few ways a build can error after the ATC reattaches (unexpected end of JSON). With it written to a file, we can just try to read the file when we re-attach after seeing that the process exited. This also frees up stdout/stderr for normal logging, which has been an occasional pitfall during resource development/debugging.
11. Remove the distinction between source and params; resources will receive a single config. The distinction will remain in the pipeline. This makes it easier to implement a resource without planning ahead for interesting dynamic vs. static usage patterns, and will get more powerful with #684.

There have been a few reports lately of jobs being unable to schedule. It's been difficult for us to progress on this as we never see it ourselves and there's generally not enough information provided to find a smoking gun. Which is no one's fault really - we haven't given great instructions as to what information would be valuable and how to collect it! So, that's what this issue is for.

If you're seeing this problem, please include a screenshot of the build preparation section of the build view.

Also answer the following:

• Are all of your workers present? (fly workers)
• Is there a check container present for each of the inputs to your job? (fly hijack -c pipeline-name/resource-name)
  • Do any of the check containers have a running /opt/resource/check process? If so, that may be hanging. What resource type is it?
• What is the uptime of your workers and ATC?
• Are your workers registered directly (BOSH default) or forwarded through the TSA (binary default/external workers registering with BOSH deployment)?
• Which IaaS?
  • If you're on GCP, have you configured the MTU of your workers to be 1460 to match the VM? If not, it defaults to 1500, which would cause things to hang.
  • Can you reach the workers from your ATC? (curl http://<worker ip:port>/containers) You can collect the IP + port from fly workers -d.

In addition to that, the most valuable information will be stack dumps of both the ATC and the TSA.

You can collect a stack dump from the ATC by running:

curl http://127.0.0.1:8079/debug/pprof/goroutine?debug=2

...and from the TSA by sending SIGQUIT and collecting the output from stderr. Note that if you're running the binaries, the above curl command will include the TSA's stack, so don't worry about getting it separately. Also note that SIGQUIT will kill the TSA process, so you'll need to bring it back after. (While you're at it, let us know if that fixed it. :P)

Thanks all for your patience, sorry we're taking so long to get to the bottom of this.

Bug Report

In our large-scale algorithm test environment, workers gradually stalled over the course of 2 days:

In investigating one of the workers, the machine seemed fine, but its logs showed that it simply stopped heartbeating. It must have been stuck somewhere.

lsof -p <worker pid> revealed that there wasn't even an active connection to the TSA anymore (port 222).

Through this time, web node memory usage and goroutines crept up:

Upon restarting all of the workers (i.e. bosh restart worker), the web goroutines dropped dramatically, and the memory usage slightly recovered (note: these graphs are over the past hour):

Steps to Reproduce

Not sure yet. May need to just chaos test this.

Expected Results

I would expect the worker client to notice that the connection is broken and to recover/reconnect.

Actual Results

The worker client hung forever and stopped heartbeating/registering.

Additional Context

Goroutine dumps: https://gist.github.com/vito/a53e8b484a3d254a0872ebe7f5d15773

The dumps are large; I would recommend cloning the gist instead:

$ git clone https://gist.github.com/vito/a53e8b484a3d254a0872ebe7f5d15773

Version Info

• Concourse version: master
• Deployment type (BOSH/Docker/binary): BOSH
• Infrastructure/IaaS: GCP
• Browser (if applicable): n/a
• Did this used to work? probably not

After upgrading to 5.4.0, it looks like our workers are seeing seemingly random DNS resolution failures.

14 minutes into the build (after downloading things successfully from a host), dns resolution randomly fails.

Here's an example:

Resolving package dependencies ...
--2019-07-23 18:38:54--  https://eu.hpkg.haiku-os.org/haikuports/master/build-packages//packages/groff-1.22.3-1-x86_64.hpkg
Resolving eu.hpkg.haiku-os.org (eu.hpkg.haiku-os.org)... 62.210.188.85
Connecting to eu.hpkg.haiku-os.org (eu.hpkg.haiku-os.org)|62.210.188.85|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 7633254 (7.3M) [application/octet-stream]
Saving to: 'download/groff-1.22.3-1-x86_64.hpkg'

download/groff-1.22.3-1-x86_64.hpkg                                                                                          100%[==============================================================================================================================================================================================================================================================================================================================================>]   7.28M  6.30MB/s    in 1.2s    

2019-07-23 18:38:56 (6.30 MB/s) - 'download/groff-1.22.3-1-x86_64.hpkg' saved [7633254/7633254]

--2019-07-23 18:38:56--  https://eu.hpkg.haiku-os.org/haikuports/master/build-packages//packages/libiconv-1.15-4-x86_64.hpkg
Resolving eu.hpkg.haiku-os.org (eu.hpkg.haiku-os.org)... failed: Temporary failure in name resolution.
wget: unable to resolve host address 'eu.hpkg.haiku-os.org'

export imagePath="haiku.image"
export isImage="1"
export isVMwareImage=""
../haiku-git/build/scripts/build_haiku_image /tmp/build/e05da604/generated.x86_64/./haiku.image-init-vars /tmp/build/e05da604/generated.x86_64/./haiku.image-make-dirs /tmp/build/e05da604/generated.x86_64/./haiku.image-copy-files /tmp/build/e05da604/generated.x86_64/./haiku.image-extract-files

...failed BuildHaikuImage1 haiku.image ...
...removing haiku.image

BUILD FAILURE:
...failed updating 1 target(s)...
...skipped 1 target(s)...
...updated 14957 target(s)...

a wget works as expected, then 5 seconds later a wget from the same host fails to resolve.

i have the following set for my web config

CONCOURSE_BASIC_AUTH_USERNAME=admin
CONCOURSE_BASIC_AUTH_PASSWORD=xxxxxx
CONCOURSE_EXTERNAL_URL=https://ci.xxxx.com

CONCOURSE_MAIN_TEAM_LOCAL_USER=admin
CONCOURSE_ADD_LOCAL_USER=admin:xxxxxx
CONCOURSE_MAIN_TEAM_ALLOW_ALL_USERS=true

now when i try and login, it just sits there spinning. if i look in the inspector i can see it pending on

https://ci.xxxx.com/sky/callback?code=btkstm4h47f23yugenht5v5s2&state=eyJyZWRpcmVjdF91cmkiOiIvIiwiZW50cm9weSI6IjQzNWY1N2NiZDNiMWZmYTMzNGZmNGUxYmRhOThmYjMxNzUxNTg4YThhYzFkNDQ2N2QxMGJkMmYyMzkyOTg1MzIifQ%3D%3D

it will finally time out after a while.

in the logs i see

Aug  4 00:21:36 ip-10-200-1-205 concourse[4491]: {"timestamp":"1533342096.544905424","source":"atc","message":"atc.dex.event","log_level":1,"data":{"fields":{},"message":"login successful: connector \"local\", name=\"\", email=\"admin\", groups=[]","session":"5"}}

if i put in the wrong login info it instantly comes back and says bad login/password. when it does work it seems to just timeout.

after it times out i see this in the log:

Aug 4 00:27:10 ip-10-200-1-205 concourse[4491]: {"timestamp":"1533342430.388325930","source":"atc","message":"atc.sky.callback.failed-to-fetch-dex-token","log_level":2,"data":{"error":"Post https://ci.wizr.com/sky/issuer/token: dial tcp 34.210.127.211:443: i/o timeout","session":"4.277"}}

Feature Request

What challenge are you facing?

I want to build a pipeline that has a deploy step that can deploy to many different environments, eg:

build image -> test image -> deploy

where the 'deploy' job can deploy to one of several environments. We have a bunch of servers running that different teams use (around 20). Currently, it seems like with concourse, to be able to quickly deploy to any one of those 20 environments, we need to create 20 separate deploy jobs, which is not the ideal (especially when I think about having pipelines for multiple microservices as we build them out).

A Modest Proposal

The ideal for us would be if when someone kicks off the 'deploy' job, there would be a UI prompt for them to enter what environment they want (eg 'environment-12') and that gets passed to the appropriate task

We have GHE admins who are having their deployment of GHE face-melted by some over eager polling and a somewhat large dev organization.

Would it be possible to have concourse accept github post-commit hooks to trigger git-resources? Maybe for other repo resources as well.

Tentative issue to track known issues with the current algorithm and track any progress/thoughts on a new algorithm to replace it.

Problems with the current algorithm:

• It loads up all versions and build inputs and outputs for the entire pipeline, meaning memory use and network transfer from the DB will increase as this data set grows. This is also slow as it involves a bunch of joins across a ton of data. See #2624, #3487, #3426.
  • We are also careful to cache this so it's only loaded when the data changes, but that can only help so much - periodic jobs or resources will invalidate the cache anyway.
• The algorithm itself is fairly brutal; it's hella optimized but prone to worst-case scenarios where it churns and uses a ton of CPU. And it's difficult to instrument without slowing it down.

An experimental new algorithm lives on the algorithm-experiment branch.

Summary

When performing the web node upgrade to 7.9.0, in a rolling fashion, a deadlock occurs for the migration with the bigint change.

Steps to reproduce

1. Create a multiple web deployment running on 7.8.3, using a PostgreSQL 12 database (POSTGRES_12_11.R20220710.01_00).
2. Performing a rolling upgrade to 7.9.0, running a single web node at a time.
3. Use the "migrate", "--migrate-to-latest-version" command when performing the first upgrade.

Expected results

The migrate tool is able to acquire the appropriate locks to perform the upgrade, even while older web nodes are still running.

My expected results are interpreted from: https://concourse-ci.org/upgrading-concourse.html#upgrading-the-web-node

Actual results

Container logs: error: migration '1653924132_int_to_bigint.up.sql' failed and was rolled back: pq: deadlock detected

DB Logs:

2022-12-20 22:05:10.887 UTC [793672]: [38-1] db=concourse,user=concourse_rw STATEMENT:  ALTER SEQUENCE resource_config_versions_id_seq AS bigint;

ALTER TABLE build_comments
    ALTER COLUMN build_id TYPE bigint;

ALTER TABLE successful_build_outputs
    ALTER COLUMN rerun_of TYPE bigint;

ALTER TABLE jobs
    ALTER COLUMN first_logged_build_id TYPE bigint;

ALTER TABLE containers
    ALTER COLUMN meta_build_id TYPE bigint;

db=concourse,user=concourse_rw ERROR: deadlock detected

INFO 2022-12-20T22:05:11.891887Z 2022-12-20 22:05:11.890 UTC [793672]: [40-1] db=concourse,user=concourse_rw DETAIL: Process 793672 waits for AccessExclusiveLock on relation 18103 of database 16426; blocked by process 791090. Process 791090 waits for AccessShareLock on relation 18054 of database 16426; blocked by process 793672. Process 793672: ALTER SEQUENCE resource_config_versions_id_seq AS bigint; ALTER TABLE build_comments ALTER COLUMN build_id TYPE bigint; ALTER TABLE successful_build_outputs ALTER COLUMN rerun_of TYPE bigint; ALTER TABLE jobs ALTER COLUMN first_logged_build_id TYPE bigint; ALTER TABLE containers ALTER COLUMN meta_build_id TYPE bigint; Process 791090: SELECT p.id, p.name, p.groups, p.var_sources, p.display, p.nonce, p.version, p.team_id, t.name, p.paused, p.public, p.archived, p.last_updated, p.parent_job_id, p.parent_build_id, p.instance_vars, p.paused_by, p.paused_at FROM pipelines p LEFT JOIN teams t ON p.team_id = t.id LEFT JOIN jobs j ON (j.id = p.parent_job_id) LEFT JOIN pipelines parent ON (j.pipeline_id = parent.id) WHERE (p.parent_job_id IS NOT NULL AND p.archived = $1 AND (j.active = $2 OR j.id IS NULL OR parent.archived = $3 OR (p.parent_build_id < j.latest_completed_build_id AND EXISTS ( SELECT 1 FROM builds lb WHERE lb.id = j.latest_completed_build_id AND lb.status = $4 ))))

Additional context

The blocking process is:

2022-12-20 22:02:03.201 UTC [791090]: [4-1] db=concourse,user=concourse_rw STATEMENT:  DELETE FROM resource_caches WHERE id NOT IN (SELECT resource_cache_id FROM resource_cache_uses UNION SELECT resource_cache_id FROM resource_configs WHERE resource_cache_id IS NOT NULL UNION SELECT resource_cache_id FROM build_image_resource_caches UNION SELECT r_cache.id FROM next_build_inputs nbi JOIN resources r ON r.id = nbi.resource_id JOIN resource_config_versions rcv ON rcv.version_md5 = nbi.version_md5 AND rcv.resource_config_scope_id = r.resource_config_scope_id JOIN resource_caches r_cache ON r_cache.resource_config_id = r.resource_config_id AND r_cache.version_md5 = rcv.version_md5 JOIN jobs j ON nbi.job_id = j.id JOIN pipelines p ON j.pipeline_id = p.id WHERE p.paused = false) RETURNING id

Triaging info

• Concourse version: 7.9.0
• Browser (if applicable): N/A
• Did this used to work? First time for this specific migration

Steps for a new patch release:

• [ ] Ensure each resource type is pinned to whatever version was last shipped within the MAJOR.MINOR series. This is to avoid accidentally shipping breaking changes in resource types with patch releases.

  • If a patch release is being shipped in order to bump a resource type (e.g. for a CVE or bug fix), pin it to the appropriate version instead.

• [ ] Go through all the needs-documentation PRs in the release page for your milestone https://project.concourse-ci.org/releases/concourse?milestone=v<M.m.p> and make sure that everything has proper documentation within concourse/docs (if needed). You can organize which PRs by clicking on the button to add whichever label best fits that PR.

  • If it is already documented within concourse/docs, add a release/documented label
  • If there is no documentation and the changes have user impact that should be documented, add the documentation to concourse/docs(or delegate) then add a release/documented label after finished. E.g. the addition of a new step type ( set_pipeline step).
  • If there is no documentation and the changes have user impact that do not need to be documented, add a release/undocumented label. E.g. an experimental feature.
  • If there is no documentation and the changes do not have user impact, add a release/no-impact label. E.g. refactors.

• [ ] Once the final commit has made it through the pipeline, the create-draft-release job can be triggered. This job will create a draft release within the concourse GitHub release page where you can make any final adjustments or arrangements to the generated release notes. PLEASE NOTE that any manual changes made on the draft release WILL BE OVERWRITTEN if you retrigger the create-draft-release job. Please be sure to only make manual edits AFTER you are sure this is the final run of the job.

  • If you would like to edit the content, you can directly edit the PRs that it was generated from. The title is used for each PR and also the body within the Release Note header in the PR. After you have made your edits within the PR, you can rerun the create-draft-release job in order to regenerate a new release note.

  • If you would like to change the arrangement of the PRs within the release note, you can make the edits directly on the release note of the draft release.

• [ ] Once everything is ready, the shipit job can be triggered. The publish-binaries job will convert your draft release into a final release including the body of your draft release (which will persist any changes you made to the draft release body). Subsequently, the promote concourse job will run automatically. The publish-docs job runs automatically, as long as the version is actually the latest version available.

• [ ] The helm-chart pipeline is used to bump & then publish the chart.

  • Merge the release/ branch into master.
  • Next, run the concourse-app-bump job (bumps the app version and image to point to the latest release)
  • Finally, run the publish-chart-{major|minor|patch} job, depending on what has changed in the chart
  • If you make a major bump, be sure to update the CHANGELOG.md in the concourse-chart repo

We ran in to problems trying to install v7.9.0 bosh release on a concourse deployment when using bionic 18.04 ubuntu stemcells. The error was encountered when a worker vm was being updated, relating to the version of glibc, requiring glibc_2.34 or above - but 18.04 has glibc_2.27 (and 20.04 has glibc_2.31). It looks like this requires Jammy 22.04.

Error from worker.stderr.log:

+ /var/gdn/assets/linux/sbin/iptables -w -F w--input
/var/gdn/assets/linux/sbin/iptables: /lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.29' not found (required by /var/gdn/assets/linux/sbin/iptables)
/var/gdn/assets/linux/sbin/iptables: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.28' not found (required by /var/gdn/assets/linux/sbin/iptables)
/var/gdn/assets/linux/sbin/iptables: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.33' not found (required by /var/gdn/assets/linux/sbin/iptables)
/var/gdn/assets/linux/sbin/iptables: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found (required by /var/gdn/assets/linux/sbin/iptables)

error: Exit trace for group:
garden exited with error: Exit trace for group:
gdn exited with error: exit status 1

Instead we successfully installed v7.8.3 with no problems, speculating that the dependancy was introduced with the listed feature 'Bump dependencies for worker runtime to support Ubuntu Jammy Jellyfish'. Is this intended?

Summary

When I use an in_parallel step with the parallel steps defined under 'steps:', the on_failure modifier is not taken into account.

Steps to reproduce

Config like this:

      - in_parallel:
          fast_fail: true
          steps:
            - task: execute-tu-and-sonar
              file: repo-common-code/sldng/execute-sonar.yml
            - task: analyze-enf-tester
              file: repo-common-code/sldng/execute-enf-tester.yml
              ensure:
                try:
                  task: sync-test-report-to-s3
                  file: repo-common-code/s3/s3_copy.yml
            - put: image-docker
          on_failure:
            try:
              task: delete-temporary-image
              file: repo-common-code/tools/artifactory-delete-image.yml

Expected results

$ alias fsp='fly -t $CONCOURSE_TARGET sp -c ci/pipeline.yml -p $(basename "$(printf "%q\n" $(pwd))") $CONCOURSE_VARS'
$ f sp -p test -c ci/pipeline.yml
$ f gp -p test | yq .jobs.[1].plan.[2]
in_parallel:
  steps:
    - file: repo-common-code/sldng/execute-sonar.yml
      task: execute-tu-and-sonar
    - ensure:
        try:
          file: repo-common-code/s3/s3_copy.yml
          task: sync-test-report-to-s3
      file: repo-common-code/sldng/execute-enf-tester.yml
      task: analyze-enf-tester
    - put: image-docker
on_failure:
  try:
    file: repo-common-code/tools/artifactory-delete-image.yml
    task: delete-temporary-image

Actual results

$ f gp -p test | yq .jobs.[1].plan.[2]
in_parallel:
  steps:
    - file: repo-common-code/sldng/execute-sonar.yml
      task: execute-tu-and-sonar
    - ensure:
        try:
          file: repo-common-code/s3/s3_copy.yml
          task: sync-test-report-to-s3
      file: repo-common-code/sldng/execute-enf-tester.yml
      task: analyze-enf-tester
    - put: image-docker

Additional context

Triaging info

• Concourse version: 7.8.3
• Browser (if applicable):
• Did this used to work? never tried

Summary

When managing the resource trough set_pipeline in concourse pipeline itself, the version history for that resource is wiped since the whole resource changes. After set_pipeline the input, of this resource, for the following job, doesn't trigger the job itself. The resource gets the latest version of the resource (by design I guess). In order for the pipeline to resume, you can trigger the next job manually or pause/unpause pipeline and the next job starts.

Steps to reproduce

resources:
- name: repo
  type: git
  source: 
    branch: main
    password: <password-for-git>
    skip_ssl_verification: true
    submodule_credentials:
    - host: <some-host>
      password: <some-password>
      username: <some-user>
    uri: <uri>

jobs:
  - name: configure-self
    plan:
    - get: repo
      trigger: true
    - set_pipeline: self
      file: <some/path/to/pipeline>

  - name: hello
    plan:
      - get: repo
        trigger: true
        passed: [configure-self]
      - task: echo-hello-world
        config:
          platform: linux
          image_resource:
            type: registry-image
            source: { repository: busybox }
          run:
            path: echo
            args: ["Hello world!"]

If you for example change the branch of the resource. The pipeline starts as expected it reconfigures itself and then it freezes before getting the repo in the second task. After set_pipeline, the resource cleans the version history and fetches the latest resource. This seems by design looking at last sentence of version here As I said above even a simple pause/unpause after it triggers the get step, otherwise it remains stuck.

Expected results

Concourse pipeline continues with get step, even with a short delay for fetching the latest version.

Actual results

Concourse pipeline freezes before get step, although the get step is looking at the previous commit and the resource to the latest

Additional context

N/A

Triaging info

• Concourse version: 7.8.2
• Browser (if applicable): N/A
• Did this used to work? I am not sure, I guess not.

Summary

This is not the issue of Concourse worker itself, but seems to be a combination of how AWS ECS does things and what kernel (and maybe other things) the Amazon Linux 2 AMI has. It can be summarized as follows:

• Concourse 7.8.3 (worker mode) works just fine if deployed as an ECS task on an Amazon-Linux-2-based ECS EC2 backend (or the ECS-optimized AMI which is also based on Amazon Linux 2).
• Concourse 7.9.0 (worker mode) does not work properly with the config described above
• Concourse 7.9.0 (worker mode) does work properly deployed as an ECS task on Ubuntu-based ECS EC2 backend

Considering that there is an easy workaround (just using Ubuntu), I am not sure if it is worth investigating and fixing but just wanted to let you guys know :-)

Steps to reproduce

This bug cannot be reproduced locally (as long as you do not use an Amazon Linux 2 VM). The steps would be:

• On AWS, deploy an EC2 instance using the Amazon Linux 2 AMI (tested with the latest one) or the ECS-optimized AMI
• Create an ECS task that runs the concourse worker. As an alternative, you can use the simple golang docker client from this repo which would create a container in a way ECS does. With this, you won't need to bother to create an ECS Cluster etc. The client needs to run on the corresponding EC2 instance.
• Note that if you use docker directly (e.g. docker run), the worker will work as well.
• Create a pipeline with a resource derived by this worker. The repo contains a simple example for this.

Expected results

The git repo defined in the resource can be cloned.

Actual results

The worker based on 7.8.3 can clone the repo and execute tasks as expected. The worker based on 7.9.0 cannot do the same because containers running on it cannot access any non-local host.

Additional context

The worker container itself has a properly working networking and DNS. The containers started in the worker container have a broken networking, however, and cannot reach any non-local host at all (inluding the worker container itself).

Triaging info

• Concourse version: 7.9.0
• Browser (if applicable):
• Did this used to work? Yes, works with 7.8.3