Log4Shell: a middleware plugin for Traefik which blocks JNDI attacks based on HTTP header values

Last update: Dec 26, 2022

Comments: 17

Traefik (pronounced traffic) is a modern HTTP reverse proxy and load balancer that makes deploying microservices easy. Traefik integrates with your existing infrastructure components (Docker, Swarm mode, Kubernetes, Marathon, Consul, Etcd, Rancher, Amazon ECS, ...) and configures itself automatically and dynamically. Pointing Traefik at your orchestrator should be the only configuration step you need.

. Overview . Features . Supported backends . Quickstart . Web UI . Documentation .

. Support . Release cycle . Contributing . Maintainers . Credits .

⚠️ Please be aware that the old configurations for Traefik v1.x are NOT compatible with the v2.x config as of now. If you're running v2, please ensure you are using a v2 configuration.

Overview

Imagine that you have deployed a bunch of microservices with the help of an orchestrator (like Swarm or Kubernetes) or a service registry (like etcd or consul). Now you want users to access these microservices, and you need a reverse proxy.

Traditional reverse-proxies require that you configure each route that will connect paths and subdomains to each microservice. In an environment where you add, remove, kill, upgrade, or scale your services many times a day, the task of keeping the routes up to date becomes tedious.

This is when Traefik can help you!

Traefik listens to your service registry/orchestrator API and instantly generates the routes so your microservices are connected to the outside world -- without further intervention from your part.

Run Traefik and let it do the work for you! (But if you'd rather configure some of your routes manually, Traefik supports that too!)

Features

Continuously updates its configuration (No restarts!)
Supports multiple load balancing algorithms
Provides HTTPS to your microservices by leveraging Let's Encrypt (wildcard certificates support)
Circuit breakers, retry
See the magic through its clean web UI
Websocket, HTTP/2, GRPC ready
Provides metrics (Rest, Prometheus, Datadog, Statsd, InfluxDB)
Keeps access logs (JSON, CLF)
Fast
Exposes a Rest API
Packaged as a single binary file (made with ❤️ with go) and available as an official docker image

Supported Backends

Quickstart

To get your hands on Traefik, you can use the 5-Minute Quickstart in our documentation (you will need Docker).

Web UI

You can access the simple HTML frontend of Traefik.

Documentation

You can find the complete documentation of Traefik v2 at https://doc.traefik.io/traefik/.

If you are using Traefik v1, you can find the complete documentation at https://doc.traefik.io/traefik/v1.7/.

A collection of contributions around Traefik can be found at https://awesome.traefik.io.

Support

To get community support, you can:

join the Traefik community forum:

If you need commercial support, please contact Traefik.io by mail: mailto:[email protected].

Download

Grab the latest binary from the releases page and run it with the sample configuration file:

./traefik --configFile=traefik.toml

Or use the official tiny Docker image and run it with the sample configuration file:

docker run -d -p 8080:8080 -p 80:80 -v $PWD/traefik.toml:/etc/traefik/traefik.toml traefik

Or get the sources:

git clone https://github.com/traefik/traefik

Introductory Videos

You can find high level and deep dive videos on videos.traefik.io.

Maintainers

We are strongly promoting a philosophy of openness and sharing, and firmly standing against the elitist closed approach. Being part of the core team should be accessible to anyone who is motivated and want to be part of that journey! This document describes how to be part of the core team as well as various responsibilities and guidelines for Traefik maintainers. You can also find more information on our process to review pull requests and manage issues in this document.

Contributing

If you'd like to contribute to the project, refer to the contributing documentation.

Please note that this project is released with a Contributor Code of Conduct. By participating in this project, you agree to abide by its terms.

Release Cycle

We usually release 3/4 new versions (e.g. 1.1.0, 1.2.0, 1.3.0) per year.
Release Candidates are available before the release (e.g. 1.1.0-rc1, 1.1.0-rc2, 1.1.0-rc3, 1.1.0-rc4, before 1.1.0).
Bug-fixes (e.g. 1.1.1, 1.1.2, 1.2.1, 1.2.3) are released as needed (no additional features are delivered in those versions, bug-fixes only).

Each version is supported until the next one is released (e.g. 1.1.x will be supported until 1.2.0 is out).

We use Semantic Versioning.

Mailing Lists

General announcements, new releases: mail at [email protected] or on the online viewer.
Security announcements: mail at [email protected] or on the online viewer.

Credits

Kudos to Peka for his awesome work on the gopher's logo!.

The gopher's logo of Traefik is licensed under the Creative Commons 3.0 Attributions license.

The gopher's logo of Traefik was inspired by the gopher stickers made by Takuya Ueda. The original Go gopher was designed by Renee French.

Owner

Traefik Labs

Makes Networking Boring

https://github.com/traefik/traefik https://traefik.io

Comments

v2.8.2 go panic

Welcome!

[X] Yes, I've searched similar issues on GitHub and didn't find any.
[X] Yes, I've searched similar issues on the Traefik community forum and didn't find any.

What did you do?

Watchtower upgraded to 2.8.2, I'm sourcing latest. Upgrade should have gone smoothly as usual.

What did you see instead?

Go panic, can post full stack trace if necessary, its very large and hard to bound.

What version of Traefik are you using?

Version:      2.8.2
Codename:     vacherin
Go version:   go1.19
Built:        2022-08-11T14:55:50Z
OS/Arch:      linux/amd64

What is your environment & configuration?

Docker provider, cannot provide config (company/org). 2.8.1 works as expected.

If applicable, please paste the log output in DEBUG level

 time="2022-08-11T17:16:16-03:00" level=error msg="Error in Go routine: runtime error: slice bounds out of range [2:1]"
traefik-traefik-1  | time="2022-08-11T17:16:16-03:00" level=error msg="Stack: goroutine 29 [running]:\nruntime/debug.Stack()\n\truntime/debug/stack.go:24 +0x65\ngithub.com/traefik/traefik/v2/pkg/safe.defaultRecoverGoroutine({0x36a75c0?, 0xc0007e40c0})\n\tgithub.com/traefik/traefik/v2/pkg/safe/routine.go:66 +0xa5\ngithub.com/traefik/traefik/v2/pkg/safe.GoWithRecover.func1.1()\n\tgithub.com/traefik/traefik/v2/pkg/safe/routine.go:56 +0x36\npanic({0x36a75c0, 0xc0007e40c0})\n\truntime/panic.go:884 +0x212\ngithub.com/traefik/paerser/parser.filler.setSlice({{0x19?, {0x3989c0e?, 0x0?}}}, {0x2ff49c0?, 0xc0004c91f8?, 0x355146f?}, 0xc0004d86c0)\n\tgithub.com/traefik/[email protected]/parser/element_fill.go:157 +0xaa5\ngithub.com/traefik/paerser/pa

lots more, typical go stack trace. I can't reproduce this frequently, I need to get this server back to production.

Add Support for Consul Connect
What does this PR do?

The change set introduces support for Consul Connect enabled services.

Motivation

There is no edge proxy available that can route traffic to a connect enabled service. Consul Connect, despite being a powerful and easy to use service mesh, is useless to a lot of people who are mainly looking to route traffic from internet to private services. A service running inside Connect service mesh can only receive traffic via its sidecar, and sidecar will only communicate with a network peer using mutual TLS. The solution is easy, but haven't been implemented in any form.

Traefik already supports Consul Catalog, it is only a matter of utilizing the certificates for upstream connection wherever applicable and it becomes the perfect edge proxy for connect mesh.

More

[ ] Added/updated tests

[ ] Added/updated documentation

Additional Notes

This PR is in progress, I need some help to figure out how to set the TLS configuration on a connection without specifying it in service tags.

Related: https://github.com/containous/traefik/issues/3544

Continues consul connect integration from #6373

Co-authored-by: Florian Apolloner [email protected]
Specify backend servers' weight via annotation for kubernetes
What does this PR do?

Fixes #2729. Also previous discussions.

Provides a new ingress annotation ingress.kubernetes.io/backend-weights which specifies a YAML-encoded, percentage-based weight distribution. With this annotation, we can do canary release by dynamically adjust the weight of ingress backends.

Since that currently the weight of types.Server is integer, so I created a simple allocator to make the weight of the server as average as possible.

Motivation

Introduce weight-based canary release to kubernetes provider with minimal change.

More

[X] Added/updated tests

[x] Added/updated documentation

Additional Notes
enable custom plugins/middlewares for Traefik

After seeing the Go1.8 new plugin feature I though that this could help a lot o people to add specific functionalities to Traefik.

Instead of building/compiling/shipping a custom-made version of Traefik to enable a custom functionality it would be possible to write way simpler custom-made middlewares with this approach, doesn't it ?

Try imagine creating a package that receives the request at a parameter without having to recompile the whole Traefik repository just to add a small change. Does it sounds like a middleware ? Because for me it is ! It's just a go1.8-plugin-based-middleware !

What do you guys think ?
Need URL rewrite to add trailing slash
I have a simple app, which has the following file structure at root

script.js

style.css

index.html (load the other two files using relative path script.js and style.css)

Since I want to access the app via URL http://example/app, I proxied the web app with rule PathStripPrefix:/app. The problem is when I try to access URL http://example/app (without trailing slash), it will load "index.html" fine, but not the JS and CSS file. When I look into the debugger, it tries to load:

http://example/script.js

http://exmaple/style.css

Instead of (the correct one):

http://example/app/script.js

http://example/app/style.css

It only works when I type the original URL with a trailing slash, so http://example/app/. This is not a big deal for me but users sometimes find it annoying since we used to use Nginx, who sends an "301 Moved Permanently" to a URL with trailing slash when it's not there. I wonder if it is possible / makes sense to implement this in Traefik?

Thank you!
New web ui
First of all, sorry for making PR this huge, I did read the contributing guide but in that case I believe small PR is not possible.

Hi guys, I really like the project and I decided to help you with transition to latest version of Angular rather than using first version.

I also updated UI and started to working on be a slightly more modern but didn't finish things already. First, I would like to know if you guys are even interested of upgrading and improving user interface? If so, this PR is not finished yet, its a work in progress, but can be done in a day.

I have question if there is a reason why you are sending xhr requests to server in time interval (3000ms)? I believe this is a more or less anti-pattern and should be done with websockets. All live data on Web UI should transfer data through websockets and if someone is interested of creating server I can update the frontend accordingly.

TODO:

[x] health

[ ] frondend implementation of reconnecting websocket

[ ] e2e tests

[ ] karma tests

Cheers, Jan
Support FastCGI protocol

I want to use traefik as loadbalancer in front of some rok4 intances (http://www.rok4.org/documentation/rok4-deploiement).

Rok4 only support fastCGI. It would be nice if traefik support this protocol :)
#504 Initial support for Docker 1.12 Swarm Mode

This new provide just work with one network and traefik.port label.

I include a provide swarm it`s quite similar with docker provider but this swarm provide watch for services data.

ACME HTTP-01 challenge fails by timeout

Do you want to request a feature or report a bug?

Bug

What did you do?

I am trying to fetch automatic certificates from Let's Encrypt with HTTP-01.

What did you expect to see?

Fetching certificates like before TLS-SNI problems.

What did you see instead?

No new certificates.

Possible problems / fixes

It looks like it has something to do with adding the http route to each domain (domain.com/.well-known/acme-challenge/[token]). When visiting the same route over https I receive an 404 directly. But via http timeouts.

https://github.com/containous/traefik/blob/5140bbe99a79b45f98c27fbb8e9b6833194af4cb/acme/challenge_http_provider.go#L22

Via Slack someone (maverick) tried my same configuration but with a consul backend. Maybe it has something to do with that?

When checking de debug logs it seems it "CleansUp" token for that domain before hitting the timeout. Maybe it has something to do with that?

Output of `traefik version`: (What version of Traefik are you using?)

Traefik version v1.5.0 built on 2018-01-23_04:42:32PM

What is your environment & configuration (arguments, toml, provider, platform, ...)?

defaultEntryPoints = ["http", "https"]
debug = true
logLevel = "DEBUG"

[entryPoints]
  [entryPoints.http]
  address = ":80"
#    [entryPoints.http.redirect]
#    entryPoint = "https"
  compress = true
  [entryPoints.https]
    address = ":443"
    compress = true
    [entryPoints.https.tls]

[acme]
  email = "[email protected]"
  caServer = "https://acme-staging.api.letsencrypt.org/directory"
  # Tried it on production as well
  storage = "/etc/traefik/acme/acme.json"
  entryPoint = "https"
  OnHostRule = true
  acmeLogging = true
  [acme.httpChallenge]
    entryPoint = "http"

# Enable Docker configuration backend
[docker]
  endpoint = "unix:///var/run/docker.sock"
  domain = "sandbox.domain.com"
  watch = true
  swarmmode = true
  exposedbydefault = true

[api]
  entryPoint = "traefik"
  dashboard = true
  address = ":8080"

  [api.statistics]
    recentErrors = 10

docker-compose.yml

version: '3'
services:
  nginx:
    image: nginx:1.13
    volumes:
      - "../workspace:/srv"
      - "./nginx/default.conf:/etc/nginx/conf.d/default.conf"
    deploy:
      labels:
        - "traefik.backend=rest-api"
        - "traefik.port=80"
        - "traefik.frontend.rule=Host:rest-api.sandbox.domain.com"
        - "traefik.docker.network=frontend"
        - "traefik.backend.loadbalancer.method=drr"
    networks:
      - frontend
      - backend

  php:
    image: php-fpm:7.1
    volumes:
      - "../workspace:/srv"
    networks:
      - backend

networks:
  backend:
    external:
      name: rest-api
  frontend:
    external:
      name: frontend

If applicable, please paste the log output in debug mode (`--debug` switch)

logs

time="2018-01-25T10:05:56Z" level=debug msg="LoadCertificateForDomains [rest-api.sandbox.domain.com]..." 
time="2018-01-25T10:05:56Z" level=debug msg="Looking for provided certificate to validate [rest-api.sandbox.domain.com]..." 
time="2018-01-25T10:05:56Z" level=debug msg="No provided certificate found for domains [rest-api.sandbox.domain.com], get ACME certificate." 
time="2018-01-25T10:05:56Z" level=debug msg="Loading ACME certificates [rest-api.sandbox.domain.com]..." 
legolog: 2018/01/25 10:05:56 [INFO][rest-api.sandbox.domain.com] acme: Obtaining bundled SAN certificate
legolog: 2018/01/25 10:05:56 [INFO][rest-api.sandbox.domain.com] AuthURL: https://acme-staging.api.letsencrypt.org/acme/authz/w3M__oDqozE[...]T_SPCiF7p5CYLFI
legolog: 2018/01/25 10:05:56 [INFO][rest-api.sandbox.domain.com] acme: Could not find solver for: dns-01
legolog: 2018/01/25 10:05:56 [INFO][rest-api.sandbox.domain.com] acme: Trying to solve HTTP-01
time="2018-01-25T10:05:56Z" level=debug msg="Challenge Present rest-api.sandbox.domain.com" 
time="2018-01-25T10:06:07Z" level=debug msg="Challenge CleanUp rest-api.sandbox.domain.com" 
time="2018-01-25T10:06:07Z" level=error msg="map[rest-api.sandbox.domain.com:acme: Error 400 - urn:acme:error:connection - Fetching http://rest-api.sandbox.domain.com/.well-known/acme-challenge/GECQ9JRWb4pA[...]Bc3rmeveJd611YowU: Timeout
Error Detail:
	Validation for rest-api.sandbox.domain.com:80
	Resolved to:
		***.***.***.***
		***:*:*:*::*
	Used: ***:*:*:*::*

]" 
time="2018-01-25T10:06:07Z" level=error msg="Error getting ACME certificates [rest-api.sandbox.domain.com] : cannot obtain certificates map[rest-api.sandbox.domain.com:acme: Error 400 - urn:acme:error:connection - Fetching http://rest-api.sandbox.domain.com/.well-known/acme-challenge/GECQ9JRWb4pA0OlC[...]eJd611YowU: Timeout
Error Detail:
	Validation for rest-api.sandbox.domain.com:80
	Resolved to:
		***.***.***.***
		***:*:*:*::*
	Used: ***:*:*:*::*

]" 
time="2018-01-25T10:06:07Z" level=debug msg="LoadCertificateForDomains []..." 
legolog: 2018/01/25 10:06:07 [INFO][exceptions.sandbox.domain.com] acme: Obtaining bundled SAN certificate
time="2018-01-25T10:06:07Z" level=debug msg="LoadCertificateForDomains [exceptions.sandbox.domain.com]..." 
time="2018-01-25T10:06:07Z" level=debug msg="Looking for provided certificate to validate [exceptions.sandbox.domain.com]..." 
time="2018-01-25T10:06:07Z" level=debug msg="No provided certificate found for domains [exceptions.sandbox.domain.com], get ACME certificate." 
time="2018-01-25T10:06:07Z" level=debug msg="Loading ACME certificates [exceptions.sandbox.domain.com]..." 
legolog: 2018/01/25 10:06:07 [INFO][exceptions.sandbox.domain.com] AuthURL: https://acme-staging.api.letsencrypt.org/acme/authz/oUlowLzxA9hKGib[...]MpTqEWA4ksu345xc
legolog: 2018/01/25 10:06:07 [INFO][exceptions.sandbox.domain.com] acme: Could not find solver for: dns-01
legolog: 2018/01/25 10:06:07 [INFO][exceptions.sandbox.domain.com] acme: Trying to solve HTTP-01
time="2018-01-25T10:06:07Z" level=debug msg="Challenge Present exceptions.sandbox.domain.com" 
time="2018-01-25T10:06:09Z" level=debug msg="Filtering container without port and no traefik.port label traefik.1 : strconv.Atoi: parsing "": invalid syntax" 
time="2018-01-25T10:06:09Z" level=debug msg="Filtering container without port and no traefik.port label payment_php.1 : strconv.Atoi: parsing "": invalid syntax" 
time="2018-01-25T10:06:09Z" level=debug msg="Filtering container without port and no traefik.port label my_php.1 : strconv.Atoi: parsing "": invalid syntax" 
time="2018-01-25T10:06:09Z" level=debug msg="Filtering container without port and no traefik.port label webfrontend_php.1 : strconv.Atoi: parsing "": invalid syntax" 
time="2018-01-25T10:06:09Z" level=debug msg="Filtering container without port and no traefik.port label rest-api_php.1 : strconv.Atoi: parsing "": invalid syntax" 
time="2018-01-25T10:06:09Z" level=debug msg="Filtering container without port and no traefik.port label order_php.1 : strconv.Atoi: parsing "": invalid syntax" 
time="2018-01-25T10:06:09Z" level=debug msg="Filtering container without port and no traefik.port label catalog_php.1 : strconv.Atoi: parsing "": invalid syntax" 
time="2018-01-25T10:06:09Z" level=debug msg="Filtering container without port and no traefik.port label price_php.1 : strconv.Atoi: parsing "": invalid syntax" 
time="2018-01-25T10:06:09Z" level=debug msg="Filtering container without port and no traefik.port label notifications_php.1 : strconv.Atoi: parsing "": invalid syntax" 
time="2018-01-25T10:06:09Z" level=debug msg="Filtering container without port and no traefik.port label exceptions_php.1 : strconv.Atoi: parsing "": invalid syntax" 
time="2018-01-25T10:06:09Z" level=debug msg="Could not load traefik.frontend.whitelistSourceRange labels" 
time="2018-01-25T10:06:09Z" level=debug msg="Could not load traefik.frontend.entryPoints labels" 
time="2018-01-25T10:06:09Z" level=debug msg="Could not load traefik.frontend.auth.basic labels" 
time="2018-01-25T10:06:09Z" level=debug msg="Could not load traefik.frontend.whitelistSourceRange labels" 
time="2018-01-25T10:06:09Z" level=debug msg="Could not load traefik.frontend.auth.basic labels" 
time="2018-01-25T10:06:09Z" level=debug msg="Could not load traefik.frontend.whitelistSourceRange labels" 
time="2018-01-25T10:06:09Z" level=debug msg="Could not load traefik.frontend.entryPoints labels" 
time="2018-01-25T10:06:09Z" level=debug msg="Could not load traefik.frontend.auth.basic labels" 
time="2018-01-25T10:06:09Z" level=debug msg="Could not load traefik.frontend.whitelistSourceRange labels" 
time="2018-01-25T10:06:09Z" level=debug msg="Could not load traefik.frontend.entryPoints labels" 
time="2018-01-25T10:06:09Z" level=debug msg="Could not load traefik.frontend.auth.basic labels" 
time="2018-01-25T10:06:09Z" level=debug msg="Could not load traefik.frontend.whitelistSourceRange labels" 
time="2018-01-25T10:06:09Z" level=debug msg="Could not load traefik.frontend.entryPoints labels" 
time="2018-01-25T10:06:09Z" level=debug msg="Could not load traefik.frontend.auth.basic labels" 
time="2018-01-25T10:06:09Z" level=debug msg="Could not load traefik.frontend.whitelistSourceRange labels" 
time="2018-01-25T10:06:09Z" level=debug msg="Could not load traefik.frontend.entryPoints labels" 
time="2018-01-25T10:06:09Z" level=debug msg="Could not load traefik.frontend.auth.basic labels" 
time="2018-01-25T10:06:09Z" level=debug msg="Could not load traefik.frontend.whitelistSourceRange labels" 
time="2018-01-25T10:06:09Z" level=debug msg="Could not load traefik.frontend.entryPoints labels" 
time="2018-01-25T10:06:09Z" level=debug msg="Could not load traefik.frontend.auth.basic labels" 
time="2018-01-25T10:06:09Z" level=debug msg="Could not load traefik.frontend.whitelistSourceRange labels" 
time="2018-01-25T10:06:09Z" level=debug msg="Could not load traefik.frontend.entryPoints labels" 
time="2018-01-25T10:06:09Z" level=debug msg="Could not load traefik.frontend.auth.basic labels" 
time="2018-01-25T10:06:09Z" level=debug msg="Could not load traefik.frontend.whitelistSourceRange labels" 
time="2018-01-25T10:06:09Z" level=debug msg="Could not load traefik.frontend.entryPoints labels" 
time="2018-01-25T10:06:09Z" level=debug msg="Could not load traefik.frontend.auth.basic labels" 
time="2018-01-25T10:06:09Z" level=debug msg="Could not load traefik.frontend.whitelistSourceRange labels" 
time="2018-01-25T10:06:09Z" level=debug msg="Could not load traefik.frontend.entryPoints labels" 
time="2018-01-25T10:06:09Z" level=debug msg="Could not load traefik.frontend.auth.basic labels"

Sporadic 502 response only when running through traefik

Do you want to request a feature or report a bug?

Bug

What did you do?

I have a graphql API running on NodeJS using Apollo and Express, with traefik in front.

When proxying through traefik I get sporadic 502 responses that I have not been able to resolve.

I does never happen when I bypass the proxy and connect directly to the backend node server.

I am running all tests locally on my dev machine.

My first attempt to force the error was load testing with the locust framework. However, even when sending large amounts of request through the proxy I was unable to replicate the error. It only happens when I use the frontend application in the browser.

After reading this oxy issue I started suspecting cancelled connections.

I added a custom HTTP header with a UUID to be able to trace all requests, which I print on the backend.

  app.use((req, res, next) => {
    const id = req.headers['x-request-id'];
    if (id) {
      console.log(`Request id: ${id}`);
    }
    next();
  });

Then I also added the following event listener to the express server to detect cancelled requests

  app.use((req, res, next) => {
    req.connection.on('close', () => {
      const id = req.headers['x-request-id'];
      console.log(`Cancelled request ${id}`);
    });

    next();
  });

What I can see is that I do get cancelled requests when running the application in the browser, and at some point i get a 502 response from traefik. And in the traefik log this is

DEBU[2018-04-26T13:43:51+02:00] vulcand/oxy/forward/http: Round trip: http://localhost:6543, code: 502, Length: 11, duration: 66.352475ms

And the nodejs backend log looks something like this:

...
Request id: 7455804b-490a-4361-98e5-43d12bf4aca8
Request id: 737f8d9d-3300-461b-858b-07006582a937
POST /graphql 200 83.542 ms - 310
POST /graphql 200 16.441 ms - 682
Request id: 096e0e39-90e6-475c-b8ad-0aa2dfd2e345
POST /graphql 200 5.338 ms - 163
Request id: 69f17cb2-cdf1-4db5-a9f5-08e46d795892
Request id: 50d3aec6-5cda-4e8b-ac0e-a30a57fa94c9
POST /graphql 200 58.596 ms - 310
POST /graphql 200 15.526 ms - 682
Request id: 1d051f3a-7d80-464b-b50f-6d8e733d1940
<------------- Here I get the 502
Cancelled request 2e0a8e14-9880-46e7-8e51-ad528d55a81d
Cancelled request b9489e71-7fc5-4f1c-b30a-668aac4652f9
Cancelled request 249c529c-b9cb-4b48-a491-8e38a7ee50d8
Cancelled request a5be4a66-9d43-4e30-a92d-862b355399a0
Cancelled request 3721fe71-fe18-4389-812a-a90cc2f4f0f1
Cancelled request 71b74750-8078-471e-91b8-a8119e5db797
Cancelled request 34fb6b91-9fa5-4d68-92da-c267089f5910
Cancelled request 692770b1-61c3-49c2-8309-8e7be629dca1
Cancelled request 05790579-8290-4787-a7b7-82596ad24520
Cancelled request c8edcc39-30c7-4812-941c-a1899298acf7
Cancelled request 2ba9e715-ab7c-48ee-9d35-b5609179de6e
Cancelled request b34f4725-665f-4b27-b3e1-cefec20c2ade
Cancelled request 04bd3718-f6aa-4318-a469-fa3e17f54a20
Cancelled request 4aedc60c-269a-420c-b083-1ea8f2e3243e
Cancelled request 25be7334-43f9-4135-9537-36b0e36e698c
Cancelled request 47bc1f9f-55c7-4f31-9957-7f0ad4285314
Cancelled request bae3237c-efc8-4831-8260-6edbcedef28f
Cancelled request 54685ecb-4d34-4698-b956-d0602b74a2e4
Cancelled request 965f6ff2-da91-423c-a8e4-c2f4252f25fc
Cancelled request 95c77d5c-230d-4875-8b25-fc0673c8e595
Cancelled request 01658960-4627-42f8-a496-d29408a9579b
Cancelled request 38221ac3-47ed-42f2-a56e-31deacdbfd62
Cancelled request e73bec6b-744c-47bc-b001-0d914f03e976
Cancelled request 73fade75-a943-45df-8b21-f8c50a480170
Cancelled request 02688ad9-e947-415f-b70c-3cda16c50cf2
Cancelled request 5d7d26c2-8c69-4083-a2d3-f0e1ae23bd0f
Cancelled request f81a0258-085d-462f-9fcb-8a8b47918d04
...

The failed request that gets a 502 response in the browser never reach the node server backend.

I get a whole bunch of canceled request after the 502 occurs. These request IDs have been successfully served by the nodejs application at an earlier point.

The canceling of the request seem to indicate some kind of connection leak? Or maybe just a sideffect of having chrome developer tools open?

Anyway I never get any error response when bypassing the traefik instance.

As the oxy issue describes, if I just could get some other response than 502 for cancelled requests I could handle this better on the client side.

Output of `traefik version`: (What version of Traefik are you using?)

Get the problem with the docker release as well as my homebrew install

Homebrew traefik version:

Version:      dev
Codename:     cheddar
Go version:   go1.10
Built:        I don't remember exactly
OS/Arch:      darwin/amd64

Docker traefik version:

Version:      v1.5.2
Codename:     cancoillotte
Go version:   go1.9.4
Built:        2018-02-12_10:56:31AM
OS/Arch:      linux/amd64

What is your environment & configuration (arguments, toml, provider, platform, ...)?

debug = true

logLevel = "DEBUG"
defaultEntryPoints = ["http"]

[entryPoints]
  [entryPoints.ping]
  address = ":8082"

  [entryPoints.api]
  address = ":8080"

  [entryPoints.http]
  address = ":80"

[retry]

[ping]
entryPoint = "ping"

[api]
entryPoint = "api"
  [api.statistics]

[file]
[backends]
  [backends.bct]
    [backends.bct.servers]
      [backends.bct.servers.server0]
        # url = "http://docker.for.mac.host.internal:6543"
        url = "http://localhost:6543"


[frontends]
  [frontends.bct]
    entryPoints = ["http"]
    backend = "bct"

[docker]
  endpoint = "unix:///var/run/docker.sock"
  # domain = "docker.for.mac.host.internal"
  domain = "localhost"
  watch = true
  exposedbydefault = false

Docker Swarm: Support for real time event listening (connection drain support).

What does this PR do?

These changes provide a support for load balancer draining for Docker Swarm. Note, the containers and services should also support graceful shutdowns.

This change makes sure Traefik stops routing, almost instantly, traffic to containers that are not in the "running" state.

We have backwards compatibility for Docker Swarm managers that don't offer live swarm events, by polling every 15 seconds (it's the same functionality as the current "master" branch offers).

Motivation

We require a Docker Swarm load balancer that supports connection draining.

Related to #41 Fixes #3035

Additional information

These changes do not break backwards compatibility.

These changes do not affect Traefik setups that are configured to route traffic using the internal Docker Swarm load balancing (IPVS). Traefik does not use the Docker Swarm load balancing by default (does not matter if Traefik is running with swarm mode set to true or not).

Stress testing results

Results from some tests I did locally on my Swarm cluster, using the official Traefik Docker image from the date of the testing (15th of March 2018), versus the patched Traefik binary. The file names describe what is being tested.

https://gist.github.com/kristinn/e3c450b71aa3898f39fea20abe87bade
customerrors middleware: allow preserving status code and method
What does this PR do?

Extend customerrors middleware to support new behaviours:

preserveStatusCode: allow using status code returned by service serving error page.

preserveMethod: query the service serving the error page using the same HTTP method as the one that caused the original error.

Motivation

We've switched from NGINX Ingress controller to Traefik and that changed how OPTIONS requests are treated in the error (e.g. 503) cases. On the initial OPTIONS call, even if backend is unavailable, we want to serve Access-Control-Allow-Origin: * with 204 No Content status code.

For more context see kubernetes/ingress-nginx#2140

More

[x] Added/updated tests

[x] Added/updated documentation

Additional Notes

Config plugin bug with array of string

Welcome!

[X] Yes, I've searched similar issues on GitHub and didn't find any.
[X] Yes, I've searched similar issues on the Traefik community forum and didn't find any.

What did you do?

Ref: https://github.com/alexandreh2ag/traefik-ipfilter-basicauth/issues/3

I use my own plugin https://github.com/alexandreh2ag/traefik-ipfilter-basicauth/releases/tag/v1.0.3 with config like :

type BasicAuth struct {
	Users        []string `json:"users,omitempty"`
	UsersFile    string   `json:"usersFile,omitempty"`
	Realm        string   `json:"realm,omitempty"`
	RemoveHeader bool     `json:"removeHeader,omitempty"`
	HeaderField  string   `json:"headerField,omitempty"`
}

type IPWhiteList struct {
	SourceRange []string `json:"sourceRange,omitempty"`
}

// Config the plugin configuration.
type Config struct {
	BasicAuth   BasicAuth   `json:"basicAuth,omitempty"`
	IPWhiteList IPWhiteList `json:"ipWhiteList,omitempty"`
}

http:
  middlewares:
    auth:
      plugin:
        auth:
          basicAuth:
            realm: "Test"
            useAuthCustomHeader: true
            headerField: "X-Test-Authorization"
            users: ["test:test"]
          ipWhiteList:
            sourceRange:
              - "192.168.1.1/27"
              - "127.0.0.1"

I except a value of struct Config in func New (ctx context.Context, next http.Handler, config *Config, name string) (http.Handler, error):

Config:
    IPWhiteList:
        SourceRange: []string{"192.168.1.1/27", 127.0.0.1"}

What did you see instead?

I got a value of struct Config in func New (ctx context.Context, next http.Handler, config *Config, name string) (http.Handler, error):

Config:
    IPWhiteList:
        SourceRange: []string{"║24║192.168.1.1/27║127.0.0.1"}

I think this issue was introduce with #8885.

What version of Traefik are you using?

This issue occurred since v2.8.2 and my plugin work with v2.8.1 and lower.

What is your environment & configuration?

static config

accessLog: {}
api:
  dashboard: true
  insecure: true
entryPoints:
  traefik:
    address: 172.18.0.1:8080
  web:
    address: 172.18.0.1:80
experimental:
  localPlugins:
    auth:
      moduleName: github.com/alexandreh2ag/traefik-ipfilter-basicauth
global:
  checkNewVersion: false
  sendAnonymousUsage: false
log:
  level: DEBUG

providers:
  file:
    directory: ./config
    watch: true

dynamic config

http:
  middlewares:
    auth:
      plugin:
        auth:
          basicAuth:
            realm: "Test"
            useAuthCustomHeader: true
            headerField: "X-Test-Authorization"
            users: ["test:test"]
          ipWhiteList:
            sourceRange:
              - "192.168.1.1/27"
              - "127.0.0.1"

  routers:
    traefik_dashboard_http:
      entrypoints:
        - web
      middlewares:
        - auth@file
      rule: "Host(`localhost`)"
      service: api@internal

If applicable, please paste the log output in DEBUG level

Log provided by my code:

ERRO[2023-01-03T18:35:16+01:00] cannot parse CIDR whitelist [║24║192.168.1.1/27║127.0.0.1]: parsing CIDR trusted IPs <nil>: invalid CIDR address: ║24║192.168.1.1/27║127.0.0.1  routerName=traefik_dashboard_http@file entryPointName=web

Have a dedicated documentation page/section for the "default" resources mechanism (TLSOptions, TLSStore, ServersTransport...)
Welcome!

[X] Yes, I've searched similar issues on GitHub and didn't find any.

[X] Yes, I've searched similar issues on the Traefik community forum and didn't find any.

What did you expect to see?

It could be great to improve the documentation of the "default" resources mechanism by creating a dedicated section/page to emphasize it.

In v3 documentation (still in beta at the time of writing this issue), HTTP and TCP serversTransport resources will move exclusively to the dynamic configuration, and hence will be ruled by the 'default' mechanism. So there will be multiple reminder sections among the documentation that explain it with respect to the context of a specific resource, as of today for the TLSOptions resource: https://doc.traefik.io/traefik/https/tls/#tls-options.

While this is necessary, having this mechanism explained in one place, could be good to understand it once and for all.

Possible memory leak in v2.8.3

Welcome!

[X] Yes, I've searched similar issues on GitHub and didn't find any.
[X] Yes, I've searched similar issues on the Traefik community forum and didn't find any.

What did you do?

In k8s traefik v2.8.3. With 1500 QPS requests which are all 404. Causing traefik pod memory (container_memory_working_set_bytes) increase to 30GB in 6.5 hours. But RSS(container_memory_rss) remain the same

It seems like the massive log print causing pod memory increase.

What did you see instead?

Causing traefik pod memory (container_memory_working_set_bytes) increase to 30GB in 6.5 hours. But RSS(container_memory_rss) remain the same

Below is pprof alloc space, inuse space keeps low.

What version of Traefik are you using?

v2.8.3

What is your environment & configuration?


global:
  checkNewVersion: false    ## 周期性的检查是否有新版本发布
  sendAnonymousUsage: false ## 周期性的匿名发送使用统计信息
serversTransport:
  insecureSkipVerify: true  ## Traefik 忽略验证代理服务的 TLS 证书
api:
  insecure: true            ## 允许 HTTP 方式访问 API
  dashboard: true           ## 启用 Dashboard
  debug: true              ## 启用 Debug 调试模式
metrics:
  prometheus:               ## 配置 Prometheus 监控指标数据，并使用默认配置
    addRoutersLabels: true  ## 添加 routers metrics
    entryPoint: metrics     ## 指定 metrics 8082为metric监听地址
entryPoints:
  web:
    address: ":80"          ## 配置 80 端口，并设置入口名称为 web
    forwardedHeaders: 
      insecure: true        ## 信任所有的forward headers
  websecure:
    address: ":443"         ## 配置 443 端口，并设置入口名称为 websecure
    forwardedHeaders: 
      insecure: true
  traefik:
    address: ":8090"        ## 配置 8090 端口，并设置入口名称为 dashboard
  metrics:
    address: ":8082"        ## 配置 8082 端口，作为metrics收集入口
  tcp:                    
    address: ":8079"        ## 配置 8079 端口，作为tcp访问入口
providers:
  kubernetesCRD:            ## 启用 Kubernetes CRD 方式来配置路由规则
    #ingressClass: traefik-v2.8
    allowCrossNamespace: true   ##允许跨namespace
    allowEmptyServices: true    ##允许空endpoints的service
  kubernetesIngress:        ## 启动 Kubernetes Ingress 方式来配置路由规则
    ingressClass: ""
log:
  filePath: "/etc/traefik/logs/traefik.log"              ## 设置调试日志文件存储路径，如果为空则输出到控制台
  level: error              ## 设置调试日志级别
  format: "common"                ## 设置调试日志格式
accessLog:
  filePath: "/etc/traefik/logs/access.log"              ## 设置访问日志文件存储路径，如果为空则输出到控制台
  format: "json"                ## 设置访问调试日志格式
  bufferingSize: 1000          ## 设置访问日志缓存行数
  filters:
    #statusCodes: ["200"]   ## 设置只保留指定状态码范围内的访问日志
    #retryAttempts: true     ## 设置代理访问重试失败时，保留访问日志
    #minDuration: 20         ## 设置保留请求时间超过指定持续时间的访问日志
  fields:                   ## 设置访问日志中的字段是否保留（keep 保留、drop 不保留）
    defaultMode: keep       ## 设置默认保留访问日志字段
    names:                  ## 针对访问日志特别字段特别配置保留模式
      ClientUsername: drop
      StartUTC: drop        ## 禁用日志 timestamp 使用UTC
    headers:                ## 设置 Header 中字段是否保留
      defaultMode: keep     ## 设置默认保留 Header 中字段
      names:                ## 针对 Header 中特别字段特别配置保留模式
        User-Agent: redact  ## 可以针对指定agent
        Authorization: drop
        Content-Type: keep
        Cookie: keep

Add more configuration information here.

If applicable, please paste the log output in DEBUG level

No response

Traefik still tries to send spans even if '--tracing.jaeger=false'
Welcome!

[X] Yes, I've searched similar issues on GitHub and didn't find any.

[X] Yes, I've searched similar issues on the Traefik community forum and didn't find any.

What did you do?

Create a docker-compose.yml with just a traefik service. Add - --tracing.jaeger=false in the command section of traefik service. Now I get Tracing jaeger error: failed to flush Jaeger spans to server: write udp 127.0.0.1:34710-\u003e127.0.0.1:6831: write: connection refused every time I send a request.

The usecase is I need tracing only for production, not in local development.

What did you see instead?

Now I get Tracing jaeger error: failed to flush Jaeger spans to server: write udp 127.0.0.1:34710-\u003e127.0.0.1:6831: write: connection refused every time I send a request.

What version of Traefik are you using?

Version: 2.9.6 Codename: banon Go version: go1.19.4 Built: 2022-12-07T14:17:58Z OS/Arch: linux/amd64

What is your environment & configuration?

version: "3.9" services: traefik: image: traefik:latest ports: ["80:80"] command: --tracing.jaeger=false

If applicable, please paste the log output in DEBUG level

Tracing jaeger error: failed to flush Jaeger spans to server: write udp 127.0.0.1:60122->127.0.0.1:6831: write: connection refused
Remove observability for internal resources
What does this PR do?

This PR disables access logs, metrics, and tracing, for internal routers and services.

Motivation

Fixes #9170 Fixes #6861

More

[x] Added/updated tests

[ ] Added/updated documentation

Additional Notes

Log4Shell: a middleware plugin for Traefik which blocks JNDI attacks based on HTTP header values

Overview

Features

Supported Backends

Quickstart

Web UI

Documentation

Support

Download

Introductory Videos

Maintainers

Contributing

Release Cycle

Mailing Lists

Credits

Owner

Traefik Labs

Comments

v2.8.2 go panic

Welcome!

What did you do?

What did you see instead?

What version of Traefik are you using?

What is your environment & configuration?

If applicable, please paste the log output in DEBUG level

Add Support for Consul Connect

What does this PR do?

Motivation

More

Additional Notes

Specify backend servers' weight via annotation for kubernetes

What does this PR do?

Motivation

More

Additional Notes

enable custom plugins/middlewares for Traefik

Need URL rewrite to add trailing slash

New web ui

Support FastCGI protocol

#504 Initial support for Docker 1.12 Swarm Mode

ACME HTTP-01 challenge fails by timeout

Do you want to request a feature or report a bug?

What did you do?

What did you expect to see?

What did you see instead?

Possible problems / fixes

Output of traefik version: (What version of Traefik are you using?)

What is your environment & configuration (arguments, toml, provider, platform, ...)?

If applicable, please paste the log output in debug mode (--debug switch)

Sporadic 502 response only when running through traefik

Do you want to request a feature or report a bug?

What did you do?

Output of traefik version: (What version of Traefik are you using?)

What is your environment & configuration (arguments, toml, provider, platform, ...)?

Docker Swarm: Support for real time event listening (connection drain support).

What does this PR do?

Motivation

Additional information

Stress testing results

customerrors middleware: allow preserving status code and method

What does this PR do?

Motivation

More

Additional Notes

Config plugin bug with array of string

Welcome!

What did you do?

What did you see instead?

What version of Traefik are you using?

What is your environment & configuration?

If applicable, please paste the log output in DEBUG level

Have a dedicated documentation page/section for the "default" resources mechanism (TLSOptions, TLSStore, ServersTransport...)

Welcome!

What did you expect to see?

Possible memory leak in v2.8.3

Welcome!

What did you do?

What did you see instead?

What version of Traefik are you using?

What is your environment & configuration?

Output of `traefik version`: (What version of Traefik are you using?)

If applicable, please paste the log output in debug mode (`--debug` switch)

Output of `traefik version`: (What version of Traefik are you using?)