name: Add TLS (SSL) support. about: This adds TLS (SSL) support for s3-proxy, allowing us to encrypt the link between the load balancer and the S3 Proxy instances.

Issue/Feature

This enables both the server and internal server to encrypt data using Go's builtin TLS routines from crypto/tls.

Additional Information

Compliance folks are increasingly scrutinizing the link between the load balancer and servers in AWS, with some deeming security groups and NACLs insufficient controls and demanding this link be encrypted.

This enables the link to be encrypted using either generated, self-signed certificates/private keys or certificates and private keys specified directly, in S3, in AWS Secrets Manager, or AWS Systems Manager (SSM) Parameter Store.

Note that AWS Certificate Manager is not supported because it does not allow you to obtain the private key (by design).

Verification Steps

Add a new section to server or internalServer in your config named ssl:

server:
    ssl:
        enabled: true # Optional; defaults to true if certificates/selfSignedHostnames is set, false otherwise.
        certificates:
            -  # Exactly one of certificate/certificateUrl and privateKey/privateKeyUrl must be specified.
                certificate: |
                    -----BEGIN CERTIFICATE-----
                    ....
                    -----END CERTIFICATE-----
                certificateUrl: arn:aws:ssm:region:accountId:parameter/name
                privateKey: |
                    -----BEGIN RSA PRIVATE KEY-----
                    -----END RSA PRIVATE KEY-----
                privateKeyUrl: arn:aws:secretsmanager:region:accountId:secret/name
        selfSignedHostnames:
            - localhost
            - localhost.localdomain
        minTLSVersion: TLSv1.2  # Defaults to TLSv1.2
        maxTLSVersion: TLSv1.3  # Defaults to no limit
        cipherSuites: # See https://pkg.go.dev/crypto/tls#pkg-constants for valid names; the current default is shown below.
            - TLS_AES_128_GCM_SHA256
            - TLS_AES_256_GCM_SHA384
            - TLS_CHACHA20_POLY1305_SHA256
            - TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
            - TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
            - TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256

Checklist:

• [x] Verified by team member
• [x] Tests were added
• [x] Comments where necessary
• [x] Documentation changes if necessary

Describe the bug I am trying to disable the index page by configuring ListTargetsConfiguration

To Reproduce I have tried the following configurations. But they all show the index page.

log:
  level: debug
  format: text
listTargets:
  enabled: false
  mount:
    path:
      - /
targets:
  first-bucket:
    mount:
      path:
        - /
    bucket:
      name: foo-common.example.com
      prefix:
      region: gra
      s3Endpoint: https://s3.gra.cloud.ovh.net
      disableSSL: false
      s3ListMaxKeys: 1000
      credentials:
        accessKey:
          value: testing
        secretKey:
          value: testing

log:
  level: debug
  format: text
listTargets:
  enabled: false
  mount:
    path:
      - /foo-common.example.com/
targets:
  first-bucket:
    mount:
      path:
        - /foo-common.example.com/
    bucket:
      name: foo-common.example.com
      prefix:
      region: gra
      s3Endpoint: https://s3.gra.cloud.ovh.net
      disableSSL: false
      s3ListMaxKeys: 1000
      credentials:
        accessKey:
          value: testing
        secretKey:
          value: testing

log:
  level: debug
  format: text
listTargets:
  enabled: false
  mount:
    path:
      - /
targets:
  first-bucket:
    mount:
      path:
        - /foo-common.example.com/
    bucket:
      name: foo-common.example.com
      prefix:
      region: gra
      s3Endpoint: https://s3.gra.cloud.ovh.net
      disableSSL: false
      s3ListMaxKeys: 1000
      credentials:
        accessKey:
          value: testing
        secretKey:
          value: testing

Expected behavior Do not show and index page with files when I visit https://mys3proxy.example.com

Version and platform (please complete the following information):

• Helm chart: 2.7.0
• Image: oxynozeta/s3-proxy:4.1.0
• Kubernetes 1.21.1 (created with kind, running in Docker Desktop on Windows using WSL 2)

• Platform: Linux
• Arch: amd64
• Version: 4.1.0

Hey there, First off this is a great tool, thanks for building it! I'm using it with Google OAuth as a proxy for S3 artifacts for Buildkite builds. I'm having an issue where the OAuth redirect gets stuck after a successful login. It works properly when I hit the s3-proxy URL directly, but if I try and use the links created by Buildkite it gets stuck. If I close the tab and refresh the page I can download the artifacts though, so the authentication has actually worked.

I've put the logs into debug mode and I think it's because it's trying to use HTTP whereas we redirect all traffic to HTTPS on our load balancers? It sounds kind of similar to this issue.

This is the requested URL which throws a 307:

https://<my_domain>/auth/google/callback?state=<state>:http://<my_domain>/b1258790-5738-4f46-aa8c-44b04c99df8f/<artifact_name>&code=<code>&scope=email%20profile%20https://www.googleapis.com/auth/userinfo.profile%20openid%20https://www.googleapis.com/auth/userinfo.email&authuser=0&hd=<my_domain>&prompt=none

This is a log from one of the pods:

level=info msg="request complete" client_ip=10.0.20.71 http_method=GET http_proto=HTTP/1.1 http_scheme=http remote_addr=10.0.20.71 req_id=s3-proxy-6bdb5545fb-k5d9n/BYqOqpjgJ4-002249 resp_bytes_length=13756449 resp_elapsed_ms=1332.687785 resp_status=200 uri="http://<my_domain>/b1258790-5738-4f46-aa8c-44b04c99df8f/<artifact_name>" user_agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.61 Safari/537.36"

As you can see part of the redirect uses http:// and the http_scheme in the pod's logs is http too. I'm not 100% sure if this is the actual issue or if it's something I need to raise with buildkite.

My config:

    log:
      level: error
      format: text

    authProviders:
      oidc:
        google:
          clientID: **************
          clientSecret:
            value: **************
          state: **************
          issuerUrl: https://accounts.google.com
          redirectUrl: https://**************
          emailVerified: true

    targets:
      buildkite-artifacts:
        mount:
          path:
            - /
        resources:
        - path: /*
          methods:
            - GET
          provider: google
          oidc:
            authorizationAccesses:
              - email: "^([^@]+)@<my_domain>\\.<tld>$"
                regexp: true

        bucket:
          name: **************
          region: ap-southeast-2

Hi, is it possible to add features for response handling?

I tried your proxy to provide a static angular website to an internal vpc only. We need to route back requests with 404 or 403 code to index.html and code 200. Or is this already possible and I'm not seeing the right config part for this?

Thanks in advance.

Describe the bug I am trying to integrate oidc in s3-proxy and after that whats happening is - The app is redirecting me to the authorization server and after authenticating with the authorization server, it's redirecting me to the redirect url and its the point where the process stops with this URL in chrome URL box myapp.com/callback?code=fnkfnwj... and with this error ERR_TOO_MANY_REDIRECTS on screen.
To Reproduce Steps to reproduce the behavior: 1.) Configure an app in the authorization server. I am using fusion auth here. From there you will get client id, issuer URL, client secret. Also set redirect url there in the authorization app to the url where you want the authorization app to redirect you after the authentication is done. In my case, I setup this url as redirect URL in the auth app - https://reports.app.mydomain.com/api/.

2.) Now configure s3-proxy to use oidc as authorization and put those values that you got in step 1 from your auth app here in the config of s3-proxy oidc. The config looks something like this authProviders: oidc: provider1: clientID: fsdfdnfwjfnwjfwkfwkfkfwrfwkfnwkfnwrfnwrj clientSecret: env: CLIENT_SECRET state: mqeklfnrjfnejfnjw issuerUrl: https://auth.demo.app.mydomain.com/ redirectUrl: https://reports.app.mydomain.com/api/ # /auth/oidc/callback will be added automatically scopes: # OIDC Scopes (defaults: oidc, email, profile) - oidc - email - profile groupClaim: groups # path in token cookieSecure: true # Is the cookie generated secure ? cookieName: oidc # Cookie generated name emailVerified: true # check email verified field from token loginPath: / # Override login path dynamically generated from provider key callbackPath: /callback # Override callback path dynamically generated from provider key

and the target block looks like

targets: - name: api mount: path: - /api/ # A specific host can be added for filtering. Otherwise, all hosts will be accepted # host: reports.app.mydomain.com resources: - path: /api/* # HTTP Methods authorized (Must be in GET, PUT or DELETE) methods: - GET - PUT - DELETE # A authentication provider declared in section before, here is the key name provider: provider1 # OIDC section for access filter oidc: # NOTE: This list can be empty ([]) for authentication only and no group filter authorizationAccesses: # Authorization accesses : groups or email or regexp - email: "[email protected]"

also my bucket config bucket: name: api-tests.reports.app.mydomain.com prefix: region: us-east-1 s3Endpoint: disableSSL: false credentials: accessKey: env: AWS_ACCESS_KEY secretKey: env: AWS_SECRET_KEY

Expected behavior Now when I go to https://reports.app.mydomain.com/api/ ----- the client (s3-proxy) should redirect me to auth server and auth server after authenticating should redirect me to this URL https://reports.app.mydomain.com/api/callback with code and all, and app should let me in to view whatever is there behind it.

** But ** Everything goes as expected until the very last step. The app does catch me back after I have authenticated with auth-server. And I am shown an error instead. See the following screenshot. Screenshots https://imgur.com/52my0wP

Using an nginx reverse proxy to route https into s3-proxy (localhost:8080) with SSL certs server by nginx as well. However the port is still open, so using it explicitly will render an insecure connection... I thought using the internalServer port was the solution, but I may have missunderstood?

Issue/Feature

When running s3-proxy behind a TLS-terminating Load Balancer, s3-proxy generates a http:// value for the OIDC redirect_uri. In HTTPS-only environments (or with secure cookies enabled), this causes the OIDC flow to fail.

This PR adds a simple check for the de-facto standard HTTP Header X-Forwarded-Proto. If this header is set and has value https, the GetRequestURI function will use the https scheme instead of http.

Checklist:

• [ ] Verified by team member
• [ ] Tests were added
• [ ] Comments where necessary
• [ ] Documentation changes if necessary

Hello friend!

I'm trying to run your s3-proxy tool from docker but getting this error:

$ docker run -d --name s3-proxy -p 8080:8080 -p 9090:9090 -v $PWD/config:/config oxynozeta/s3-proxy fe4fb5a7c43324170e8879fd9516fa860556dc166e8043bec3baef2e4bcee582

$ docker logs fe4fb5a7c43324170e8879fd9516fa860556dc166e8043bec3baef2e4bcee582 {"level":"fatal","msg":"Config File "config" Not Found in "[/conf /]"","time":"2020-05-22T11:40:54Z"}

However I have config directory in my $HOME dir with config.yml inside: $ ls -la $HOME/config/config.yaml -rw-rw-r-- 1 db db 6291 May 22 14:17 /home/db/config/config.yaml

Any ideas what can be wrong here?

Thank you

Is your feature request related to a problem? Please describe. I don't want to reconfigure s3-proxy everytime someone creates a new bucket in our cloud storage. I don't always know what our buckets are named beforehand. And the people creating the buckets don't have access to reconfigure s3-proxy.

The endpoint, region, credentials, etc, are the same for all buckets.

Describe the solution you'd like Instead of configuring multiple buckets in the configuration file for s3-proxy in advance, the bucket name is acquired from the URL-path.

Example

# HTTPS call:
GET https://mys3proxy.example.com/bucket1/folder/file.bin

# Result:
bucket: bucket1
key: /folder/file.bin

# HTTPS call:
GET https://mys3proxy.example.com/bucket2/foo/bar.bin

# Result:
bucket: bucket2
key: /foo/bar.bin

In case the client is dropping the connection, we do not cancel the requests made to the S3 backend. Proposed fix is to forward the context coming from the client request to the AWS SDK.

Is your feature request related to a problem? Please describe. Currently we're using docusaurus to deploy static sites to s3, some urls that Docusaurus generates do not contain trailing slashes (ex. mysite.com/some_section), though clicking through from a parent to the sub-section page will work due to JS/HTML magic, a hard-link will not redirect to mysite.com/some_section/ which correctly serves up the indexDocument (in this case: index.html).

Describe the solution you'd like An option to force the proxy to check if a document is a folder, if so recurse into it and serve up indexDocument etc.

Describe alternatives you've considered An alternate solution is to just use URL rewriting, but we're using AWS ALBs which won't work in that scenario and though other reverse-proxies will remedy the situation, I am hesitant to put another layer in between the ALB and S3Proxy.

This PR contains the following updates:

| Package | Type | Update | Change | |---|---|---|---| | squidfunk/mkdocs-material | final | major | 8.5.3 -> 9.0.1 |

Configuration

📅 Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).

🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.

♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

🔕 Ignore: Close this PR and you won't be reminded about this update again.

• [ ] If you want to rebase/retry this PR, check this box

This PR has been generated by Mend Renovate. View repository job log here.

This PR contains the following updates:

| Package | Type | Update | Change | |---|---|---|---| | github.com/thoas/go-funk | require | patch | v0.9.2 -> v0.9.3 |

Release Notes

Configuration

📅 Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).

🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.

♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

🔕 Ignore: Close this PR and you won't be reminded about this update again.

• [ ] If you want to rebase/retry this PR, check this box

This PR has been generated by Mend Renovate. View repository job log here.

Is your feature request related to a problem? Please describe. I would like to build the docker image without first needing to build it on the host.

Describe the solution you'd like I would like to build the image without dependencies, like:

docker build -t s3-proxy .

Describe alternatives you've considered I have made a multi-stage Dockerfile that builds the application and image in one step: see #327

Additional context

For example its really nice and dependency free from docker-compose, just give it the URL/branch and it will build it itself:

# docker-compose.yaml
services:
  s3-proxy:
    build:
      context: https://github.com/EnigmaCurry/s3-proxy.git#self-buildable

name: Pull request about: Pull request for this project title: "Makes Docker image self-buildable" labels: "" assignees: ""

Issue/Feature

I do not want to build s3-proxy on my host computer, but only as a docker image. This addition creates a multi-stage Dockerfile to build the s3-proxy binary and make an image in one step.

Additional Information

This does not require you to prebuild the s3-proxy binary. To build a Docker image from scratch, just run:

docker build -t s3-proxy .

Verification Steps

Checklist:

• [ ] Verified by team member
• [ ] Tests were added
• [ ] Comments where necessary
• [ ] Documentation changes if necessary

Is it possible to use the ephemeral/short-lived credentials provided by the AWS IAM role security credentials endpoint as a form of authentication for s3-proxy?

It would likely require tracking the expiration of the token and requesting a new one when it expires.

Ref: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html

It if is not possible currently, is it a feature you would consider developing and including in this project?

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

• [ ] fix(deps): update module github.com/thoas/go-funk to v0.9.3
• [ ] chore(deps): update squidfunk/mkdocs-material docker tag to v9

Ignored or Blocked

These are blocked by an existing closed PR and will not be recreated unless you click a checkbox below.

• [ ] chore(deps): update squidfunk/mkdocs-material docker tag to v8.5.11

Detected dependencies

• [ ] Check this box to trigger a request for Renovate to run again on this repository