Automatically set GOMAXPROCS to match Linux container CPU quota.

automaxprocs GoDoc Build Status Coverage Status

Automatically set GOMAXPROCS to match Linux container CPU quota.

Installation

go get -u go.uber.org/automaxprocs

Quick Start

import _ "go.uber.org/automaxprocs"

func main() {
  // Your application logic here.
}

Development Status: Stable

All APIs are finalized, and no breaking changes will be made in the 1.x series of releases. Users of semver-aware dependency management systems should pin automaxprocs to ^1.

Contributing

We encourage and support an active, healthy community of contributors — including you! Details are in the contribution guide and the code of conduct. The automaxprocs maintainers keep an eye on issues and pull requests, but you can also report any negative conduct to [email protected]. That email list is a private, safe space; even the automaxprocs maintainers don't have access, so don't hesitate to hold us to a high standard.


Released under the MIT License.

Owner
Uber Go
Uber's open source software for Go development
Uber Go
Comments
  • Support cgroup2

    Support cgroup2

    CGroup-V2 has a different architecture which is unified. Since it is not yet supported in go, it makes sense to currently have the support. This commit adds the functionality to set GOMAXPROCS in cgroup2 enabled systems.

    Also there were lint errors because of build annotation comments which are fixed in a separate commit.

    This PR resolves https://github.com/uber-go/automaxprocs/issues/21.

  • Add benchmarks to README?

    Add benchmarks to README?

    Hi! I was wondering if it would possible to include some benchmarks in the README? We run some open-source software with cpu quotas and being able to link to some benchmarks in the README might go a long way to convincing other people to incorporate automaxprocs into their projects as well.

  • v1.5.0 I am still getting that the CPU Quota is not defined.

    v1.5.0 I am still getting that the CPU Quota is not defined.

    Originally posted by @ardan-bkennedy in https://github.com/uber-go/automaxprocs/issues/44#issuecomment-1089508688

    This change has not fixed the problem. I switched my code from "github.com/emadolsky/automaxprocs/maxprocs" to v1.5.0 and I am still getting that the CPU Quota is not defined.

    I noticed that in v1.5.0

    https://github.com/uber-go/automaxprocs/blob/v1.5.0/internal/runtime/cpu_quota_linux.go#L35

    func CPUQuotaToGOMAXPROCS(minValue int) (int, CPUQuotaStatus, error) {[Jason Lai, 5 years ago: • Import `automaxprocs/x/runtime`](https://sourcegraph.com/github.com/uber-go/automaxprocs/-/commit/572239b375b1ca6b76babeb8b0ee139dc35fd4d8)
    	cgroups, err := newQueryer()
    	if err != nil {
    		return -1, CPUQuotaUndefined, err
    	}
    
    	quota, defined, err := cgroups.CPUQuota()
    	if !defined || err != nil {
    		return -1, CPUQuotaUndefined, err
    	}
    
    	maxProcs := int(math.Floor(quota))
    	if minValue > 0 && maxProcs < minValue {
    		return minValue, CPUQuotaMinUsed, nil
    	}
    	return maxProcs, CPUQuotaUsed, nil
    }
    
    var (
    	_newCgroups2 = cg.NewCGroups2ForCurrentProcess
    	_newCgroups  = cg.NewCGroupsForCurrentProcess
    )
    
    func newQueryer() (queryer, error) {
    	cgroups, err := _newCgroups2()
    	if err == nil {
    		return cgroups, nil
    	}
    	if errors.Is(err, cg.ErrNotV2) {
    		return _newCgroups()
    	}
    	return nil, err
    }
    

    In the emadolsky repo, it's working. I don't know enough to tell you why.

    func CPUQuotaToGOMAXPROCS(minValue int) (int, CPUQuotaStatus, error) {
    	var quota float64
    	var defined bool
    	var err error
    
    	isV2, err := cg.IsCGroupV2()
    	if err != nil {
    		return -1, CPUQuotaUndefined, err
    	}
    
    	if isV2 {
    		quota, defined, err = cg.CPUQuotaV2()
    		if !defined || err != nil {
    			return -1, CPUQuotaUndefined, err
    		}
    	} else {
    		cgroups, err := cg.NewCGroupsForCurrentProcess()
    		if err != nil {
    			return -1, CPUQuotaUndefined, err
    		}
    
    		quota, defined, err = cgroups.CPUQuota()
    		if !defined || err != nil {
    			return -1, CPUQuotaUndefined, err
    		}
    	}
    
    	maxProcs := int(math.Floor(quota))
    	if minValue > 0 && maxProcs < minValue {
    		return minValue, CPUQuotaMinUsed, nil
    	}
    	return maxProcs, CPUQuotaUsed, nil
    }
    
  • maxprocs: CPU quota undefined

    maxprocs: CPU quota undefined

    This has not always been the case and I have tried to trace the code, but I don't know enough.

    The project in question is https://github.com/ardanlabs/service and I am using KIND.

    It seems to me /proc/self/cgroup is missing from the list of SuperOptions. Not sure why this is now happening.

    Any guidance to debug or understand this would be appreciated. I can walk someone to run the code.

  • Support child control cgroups in cgroupsv2

    Support child control cgroups in cgroupsv2

    The original v2 support code does not support child control groups. This commit fixes that.

    For how child control groups are organized see: https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/managing_monitoring_and_updating_the_kernel/using-cgroups-v2-to-control-distribution-of-cpu-time-for-applications_managing-monitoring-and-updating-the-kernel

  • Move lint tooling install to CI

    Move lint tooling install to CI

    Lint tools added to the main project's go.mod results in about an extra megabyte of vendored libs for all library consumers

    ❯ du -h vendor/honnef.co/go/tools
    4.0K    vendor/honnef.co/go/tools/arg
    8.0K    vendor/honnef.co/go/tools/cmd/staticcheck
    8.0K    vendor/honnef.co/go/tools/cmd
    12K     vendor/honnef.co/go/tools/config
    8.0K    vendor/honnef.co/go/tools/deprecated
    20K     vendor/honnef.co/go/tools/facts
    12K     vendor/honnef.co/go/tools/functions
    32K     vendor/honnef.co/go/tools/go/types/typeutil
    32K     vendor/honnef.co/go/tools/go/types
    32K     vendor/honnef.co/go/tools/go
    28K     vendor/honnef.co/go/tools/internal/cache
    4.0K    vendor/honnef.co/go/tools/internal/passes/buildssa
    4.0K    vendor/honnef.co/go/tools/internal/passes
    4.0K    vendor/honnef.co/go/tools/internal/renameio
    4.0K    vendor/honnef.co/go/tools/internal/sharedcheck
    40K     vendor/honnef.co/go/tools/internal
    12K     vendor/honnef.co/go/tools/lint/lintdsl
    4.0K    vendor/honnef.co/go/tools/lint/lintutil/format
    28K     vendor/honnef.co/go/tools/lint/lintutil
    88K     vendor/honnef.co/go/tools/lint
    8.0K    vendor/honnef.co/go/tools/loader
    12K     vendor/honnef.co/go/tools/printf
    72K     vendor/honnef.co/go/tools/simple
    328K    vendor/honnef.co/go/tools/ssa
    4.0K    vendor/honnef.co/go/tools/ssautil
    60K     vendor/honnef.co/go/tools/staticcheck/vrp
    216K    vendor/honnef.co/go/tools/staticcheck
    40K     vendor/honnef.co/go/tools/stylecheck
    72K     vendor/honnef.co/go/tools/unused
    12K     vendor/honnef.co/go/tools/version
    1004K   vendor/honnef.co/go/tools
    

    We can move to a more adhoc install process during CI to avoid this. This was tested by running inside a debian docker container, but will ensure CI still passes through Travis as well

  • Needs readme, with bonus screenshots

    Needs readme, with bonus screenshots

    People shouldn't land on our repos and be expected to write/run code to see what it does. Take a look at any of our other big OSS projects for a sense of what we're looking for.

    cc @prashantv not sure how this got past the internal review

  • Use floor(quota) rather than ceil(quota)

    Use floor(quota) rather than ceil(quota)

    Apologies, this was an oversight on my part back when we wrote the x/runtime precursor that I helped to review. I'd since been working on the (it turns out false) assumption that this had been floor all along. Recent investigation into pervasive low-grade throttling tho led me to this discovery.

    In short:

    • as far as the Go scheduler is concerned, there's no such thing as a fractional CPU
    • so let's say you have your quota set to N + p for some integer N and some 0.0 < p < 1.0
    • the only safe assumption then is that you're using that p value as a hedge for something like "systems effects" or "c libraries"
      • in that latter case, what you really might want to be able give maxprocs some value K of reserved CPUs for some parasite like a C library or sidecar; but that's probably more of an option for our Fx wrapper?
  • is it not default behavior after version 1.5?

    is it not default behavior after version 1.5?

    hi, I am curious that isn't it auto set GOMAXPROCS to the cpu core number?just as this StackOverflow answer says, from version 1.5, it is a default behavior

  • tools: Move to submodule

    tools: Move to submodule

    Move lint dependencies to a tools submodule so that these are not introduced as dependencies of the package.

    In other projects, we experimented with renaming tools.go to tools_test.go but that wasn't enough either. The tools submodule appears to be the safest approach.

    Supersedes #32 Refs uber-go/atomic#78

  • Fix cgroup sub system parsing with names containing colons

    Fix cgroup sub system parsing with names containing colons

    cgroup sub system names can contain colons which is also used as separator symbol, e.g.:

    12:cpu,cpuacct:/system.slice/containerd.service/kubepods-besteffort-podb41662f7_b03a_4c65_8ef9_6e4e55c3cf27.slice:cri-containerd:1753b7cbbf62734d812936961224d5bc0cf8f45214e0d5cdd1a781a053e7c48f
    
  • feature: set GOMAXPROCS env var during Set

    feature: set GOMAXPROCS env var during Set

    Currently, when do import _ "go.uber.org/automaxprocs", the GOMAXPROCS is set automatically for users. But it doesn't set up the GOMAXPROCS environment variable which means if the programs exec some children processes to do some work, the env var needs to be set up manually.

    For me, I think there are no side effects to setup the GOMAXPROCS environment variable also when we set up it in the program. So I raised this PR.

    Regrads.

  • What happens if cpu quota on k8s side is less than 1 core?

    What happens if cpu quota on k8s side is less than 1 core?

    Am i right that if you have a service with a cpu limit less than 1000m (eg 1 core) the go process would still think it has one core available and tries to utilize more than its limit to get throttled eventually. So my understanding here is that if you do use limits less than one core is a very unideal situation for the go-scheduler to work efficently:

    a.) 4 replicas with 500m cpu limit b.) 2 replicas with 1000m cpu limit

    In total both cases use the same amount of total cores (2) but case b.) would be more efficient as the go scheduler knows how much it can utilize?

    Sorry that i created a bug ticket for my simple question. But i think if my assumptions are correct, it would be good to make this clear in the README.

    Thanks for the awesome library :+1:

  • Fix mountpoint multiple match

    Fix mountpoint multiple match

    When use make test base on this case 8 5 0:7 /docker /sys/fs/cgroup/memory rw,nosuid,nodev,noexec,relatime shared:8 - cgroup cgroup rw,memory 9 5 0:8 /docker/memory.vmstat /sys/fs/cgroup/vmstat rw,nosuid,nodev,noexec,relatime shared:8 - cgroup cgroup rw,memory been failed.

    Because it's match multiple target,cannot uniquely match

  • no cgroup mountinfo in /proc/self/mountinfo if shareProcessNamespace=true

    no cgroup mountinfo in /proc/self/mountinfo if shareProcessNamespace=true

    when shareProcessNamespace=true, business container's pid is not equals to 1. There is no cgroup mountinfo in /proc/self/mountinfo, but it is in /proc/1/mountinfo.

An application written in Go to generate fractals like the Mandelbrot set and the Julia set.
An application written in Go to generate fractals like the Mandelbrot set and the Julia set.

Fractals An application written in Go to generate fractals like the Mandelbrot set and the Julia set. Screenshots Mandelbrot set Julia set Prerequisit

May 9, 2022
A fully Go userland with Linux bootloaders! u-root can create a one-binary root file system (initramfs) containing a busybox-like set of tools written in Go.

u-root Description u-root embodies four different projects. Go versions of many standard Linux tools, such as ls, cp, or shutdown. See cmds/core for m

Dec 29, 2022
AOE2 Definitive Edition Match Finder

AOE2 DE Match Finder AOE2 DE Match Finder is a Go library for fetching specific matches from the popular game Age of Empires 2 Definitive Edition. It

Jun 10, 2022
CPU feature identification for Go

cpuid Package cpuid provides information about the CPU running the current program. CPU features are detected on startup, and kept for fast access thr

Dec 29, 2022
gProfiler combines multiple sampling profilers to produce unified visualization of what your CPU
gProfiler combines multiple sampling profilers to produce unified visualization of what your CPU

gProfiler combines multiple sampling profilers to produce unified visualization of what your CPU is spending time on, displaying stack traces of your processes across native programs1 (includes Golang), Java and Python runtimes, and kernel routines.

Dec 27, 2022
Continuous profiling for analysis of CPU, memory usage over time, and down to the line number. Saving infrastructure cost, improving performance, and increasing reliability.
Continuous profiling for analysis of CPU, memory usage over time, and down to the line number. Saving infrastructure cost, improving performance, and increasing reliability.

Continuous profiling for analysis of CPU, memory usage over time, and down to the line number. Saving infrastructure cost, improving performance, and increasing reliability.

Jan 2, 2023
Automatically creates & tiles .tmx format maps from a world map interface
Automatically creates & tiles .tmx format maps from a world map interface

Autotile Create tiled maps for an arbitrarily large world space from a simple interface, then add larger objects randomly with simple rules (eg. place

Aug 19, 2022
A set of instrumentation features for collecting relevant tracing data as well as secure an application

Go Agent goagent provides a set of instrumentation features for collecting relevant tracing data as well as secure an application by blocking requests

Dec 21, 2021
redis-util business-friendly encapsulation of redis operations, such as the common cache set get operation

redis-util 方便业务使用的redis操作封装,比如常见的缓存set get操作, 一行代码搞定,不像开源库需要写好多行 使用方法

Oct 22, 2021
Simple utility to get/set the PWM duty cycle and to measure the RPM for a fan connected to the 4-pin header on the CM4IO.

cm4iofan Simple utility to get/set the PWM duty cycle and to measure the RPM for a fan connected to the 4-pin header on the CM4IO. Requirements Enable

Mar 31, 2022
Julia Set Generator Written In Golang
 Julia Set Generator Written In Golang

Julia Set Generator Written In Golang This is a simple (naive) Julia Set generator written in Golang. The project utilizes concurrent workers to speed

Nov 5, 2021
This Go package allows you to set handler functions that run when named events occur

This Go package allows you to set handler functions that run when named events occur

Feb 10, 2022
Golang 1.18+ Generics implementation of Set methods

Golang Generics: Set A golang 1.18+ implementation of Set using Go generics Installation $ go get -u github.com/chrispappas/golang-generics-set Quick

Oct 26, 2022
Perforator is a tool for recording performance metrics over subregions of a program using the Linux "perf" interface.

Perforator Perforator is a tool for recording performance metrics over subregions of a program (e.g., functions) using the Linux "perf" interface.

Dec 15, 2022
Package ethtool allows control of the Linux ethtool generic netlink interface.

ethtool Package ethtool allows control of the Linux ethtool generic netlink interface.

Dec 14, 2022
A Go library for the Linux Landlock sandboxing feature

Go Landlock library The Go Landlock library restricts the current processes' ability to use files, using Linux 5.13's Landlock feature. (Package docum

Dec 27, 2022
📋 cross-platform clipboard package that supports accessing text and image in Go (macOS/Linux/Windows/Android/iOS)

clipboard Cross platform (macOS/Linux/Windows/Android/iOS) clipboard package in Go import "golang.design/x/clipboard" Features Cross platform supports

Dec 24, 2022
Execute a binary from memory, without touching the disk. Linux only.
Execute a binary from memory, without touching the disk. Linux only.

Memit Execute a binary from memory, without touching the disk. Linux only. Available as both a Go module and a binary. Using the Go module The Command

Jan 5, 2023
Minict is a minimal container runtime written in Go.

Minict Minict is a minimal container runtime written in Go. It was made mainly for learning purposes and is intended to be as simple as possible.

Oct 31, 2022