Go benchmark harness.

autobench

autobench is a framework to compare the performance of Go 1.2 and Go 1.3.

usage

autobench downloads and builds the latest Go 1.1 and Go tip branches and runs a set of Go 1 benchmarks for comparison.

Useful targets are

make bench		# runs all benchmarks, _once_
make go1 		# runs bench/go1 benchmarks _once_
make runtime 	# runs bench/runtime benchmarks _once_
make http	 	# runs bench/http benchmarks _once_
make float	 	# runs bench/float benchmarks _once_
make extra		# runs extra benchmarks. 
make clean 		# removes any previous benchmark results
make update		# updates both branches to the latest revision, clears any benchmark results

You can optionally benchmark with gccgo instead of gc by either uncommenting the corresponding line in the Makefile or by setting TESTFLAGS to an appropriate value:

make TESTFLAGS=-compiler=gccgo bench

known issues

  • If you are benchmarking on Freebsd, you may need to use gmake.

contributing

Contributions and pull requests are always welcome. If you are submitting a pull request with benchmark data, please include the value of OLD and NEW at the top of the Makefile in the suffix of your file (follow the examples) so we can trace which revision this benchmark was taken from. If you want to include commentry in your benchmark, comments should start with a #.

 make > linux-386-go1.2.1-vs-go.1.3beta1.txt

licence

This package uses benchmark code from the Go project. Where otherwise unspecified this code is released into the public domain.

Comments
  • The benchmark doesn't work with an ARM VFP2 CPU.

    The benchmark doesn't work with an ARM VFP2 CPU.

    The Go 1.0 release doesn't support the VFP2 coprocessor (RPi).

    I did https://code.google.com/p/go/source/detail?r=bfad57cc7171 (Sep 17, 2012) vs the tip https://code.google.com/p/go/source/detail?r=672069179b80

    Looks great: https://github.com/capnm/autobench/blob/master/linux-arm-rpi-capnm.txt

    The runtime bench is missing symbols:

    /data4/my/go/autobench-capnm/work/go.103/bin/go test -test.run=XXX -test.bench=. bench/runtime
    # bench/runtime
    src/bench/runtime/export_arm_test.go:9: undefined: fadd64
    src/bench/runtime/export_arm_test.go:10: undefined: fsub64
    src/bench/runtime/export_arm_test.go:11: undefined: fmul64
    src/bench/runtime/export_arm_test.go:12: undefined: fdiv64
    src/bench/runtime/export_arm_test.go:13: undefined: f64to32
    src/bench/runtime/export_arm_test.go:14: undefined: f32to64
    src/bench/runtime/export_arm_test.go:15: undefined: fcmp64
    src/bench/runtime/export_arm_test.go:16: undefined: fintto64
    src/bench/runtime/export_arm_test.go:17: undefined: f64toint
    FAIL    bench/runtime [build failed]
    
  • export GOPATH

    export GOPATH

    /my/go/autobench/work/go.103/bin/go test -bench=. bench/go1 > /my/go/autobench/work/go1-103.txt
    can't load package: package bench/go1: import "bench/go1": cannot find package
    make: *** [/my/go/autobench/work/go1-103.txt] Error 1
    
    
    diff --git a/Makefile b/Makefile
    index 98d2dfc..c0aba79 100644
    --- a/Makefile
    +++ b/Makefile
    @@ -10,6 +10,7 @@ GOPATH=$(TOP)
     BENCHCMP=$(GO_TIP_ROOT)/misc/benchcmp
    
     unexport GOROOT
    +export GOPATH
    
    
  • Added neural net benchmark cases.

    Added neural net benchmark cases.

    This commit implements a feed-forward neural network and tests its performance for a variety of input sizes. This benchmark is a combination test. It tests numerical computation performance (in the kernel of the activator function) interface dereference (to get the specific activator in use) and parallel peformance (when computing predictions in parallel).

  • Benchmark

    Benchmark

    I ran all the tests, do you want me to merge them together or what is the preferred way? If I need to alter anything, tell me and I'll update the pull-request.

  • Add BenchmarkGoroutinesWaiting

    Add BenchmarkGoroutinesWaiting

    This benchmark tests the performance of the runtime when opening thousands of goroutines simultaneously and leaving them waiting on a channel read. It seems to expose a substantial performance regression in Go 1.1:

    benchmark                                old ns/op    new ns/op    delta
    BenchmarkGoroutinesWaiting                    3579       110215  +2979.49%
    

    I'll send a link to this to the mailing list.

  • Add some Mac OS X benchmarks

    Add some Mac OS X benchmarks

    Previously added a benchmark function, but now this just adds some benchmark data from an OS X environment.

    This benchmark tests the performance of the runtime when opening thousands of goroutines simultaneously, and leaving them open. It may effectively be testing the internal allocation routines, but there seems to be a substantial performance regression in 1.1.

  • ArchLinux compiler issues

    ArchLinux compiler issues

    Hi,

    Since Arch is using GCC 4.8.0 and the release tag of Go contains some undefined behaviour in /src/cmd/cc/funct.c, I am unable to bench on my system without first modifying the build. Perhaps you could add the steps for GCC 4.8+?

  • Darwin Benchmarks

    Darwin Benchmarks

    2.4 GHz Core 2 Duo, 8 GB 1067 MHz DDR3 OSX 10.8.3 (Build 12D78) MacBookPro7,1

    Great tool buddy, thanks for the work. I hope I appended the hash to the file in the correct manner.

  • Benchmark go 1.1.2 vs 1.2rc3 for linux/amd64

    Benchmark go 1.1.2 vs 1.2rc3 for linux/amd64

    Did the benchmark on my notebook with Ubuntu 13.10 running in a VM. Processor is Intel Core i5 @ 2.5 GHz dual core (and HT), with 4GB of RAM.

    Looks promising, thanks for the great work, keep it going!

    /home/sebastian/development/benchmark/autobench/work/go/misc/benchcmp /home/sebastian/development/benchmark/autobench/work/go1-go1.1.2.txt /home/sebastian/development/benchmark/autobench/work/go1-go1.2rc3.txt
    benchmark                         old ns/op    new ns/op    delta
    BenchmarkBinaryTree17            2147483647   2147483647  -23.26%
    BenchmarkFannkuch11              2147483647   2147483647   -9.20%
    BenchmarkFmtFprintfEmpty                113          104   -7.96%
    BenchmarkFmtFprintfString               321          259  -19.31%
    BenchmarkFmtFprintfInt                  210          215   +2.38%
    BenchmarkFmtFprintfIntInt               379          336  -11.35%
    BenchmarkFmtFprintfPrefixedInt          451          307  -31.93%
    BenchmarkFmtFprintfFloat                568          463  -18.49%
    BenchmarkFmtManyArgs                   1375         1293   -5.96%
    BenchmarkGobDecode                 13702056      9750442  -28.84%
    BenchmarkGobEncode                 15107187      5750027  -61.94%
    BenchmarkGzip                     531715193    508078252   -4.45%
    BenchmarkGunzip                   143795333    120807117  -15.99%
    BenchmarkHTTPClientServer             61145        56661   -7.33%
    BenchmarkJSONEncode                50428104     26565288  -47.32%
    BenchmarkJSONDecode               106237163     98844718   -6.96%
    BenchmarkMandelbrot200              5061852      5024612   -0.74%
    BenchmarkGoParse                    7017737      5584122  -20.43%
    BenchmarkRegexpMatchEasy0_32            137          149   +8.76%
    BenchmarkRegexpMatchEasy0_1K            363          381   +4.96%
    BenchmarkRegexpMatchEasy1_32            110          121  +10.00%
    BenchmarkRegexpMatchEasy1_1K            952         1037   +8.93%
    BenchmarkRegexpMatchMedium_32           203          212   +4.43%
    BenchmarkRegexpMatchMedium_1K         74599        73543   -1.42%
    BenchmarkRegexpMatchHard_32            3792         3596   -5.17%
    BenchmarkRegexpMatchHard_1K          120452       116962   -2.90%
    BenchmarkRevcomp                 1010231541    823446016  -18.49%
    BenchmarkTemplate                 143905860    113241955  -21.31%
    BenchmarkTimeParse                      512          470   -8.20%
    BenchmarkTimeFormat                     571          475  -16.81%
    
    benchmark                          old MB/s     new MB/s  speedup
    BenchmarkGobDecode                    56.02        78.72    1.41x
    BenchmarkGobEncode                    50.81       133.48    2.63x
    BenchmarkGzip                         36.49        38.19    1.05x
    BenchmarkGunzip                      134.95       160.63    1.19x
    BenchmarkJSONEncode                   38.48        73.05    1.90x
    BenchmarkJSONDecode                   18.27        19.63    1.07x
    BenchmarkGoParse                       8.25        10.37    1.26x
    BenchmarkRegexpMatchEasy0_32         232.68       214.43    0.92x
    BenchmarkRegexpMatchEasy0_1K        2814.15      2682.11    0.95x
    BenchmarkRegexpMatchEasy1_32         288.62       263.16    0.91x
    BenchmarkRegexpMatchEasy1_1K        1074.69       987.00    0.92x
    BenchmarkRegexpMatchMedium_32          4.91         4.70    0.96x
    BenchmarkRegexpMatchMedium_1K         13.73        13.92    1.01x
    BenchmarkRegexpMatchHard_32            8.44         8.90    1.05x
    BenchmarkRegexpMatchHard_1K            8.50         8.75    1.03x
    BenchmarkRevcomp                     251.59       308.66    1.23x
    BenchmarkTemplate                     13.48        17.14    1.27x
    /home/sebastian/development/benchmark/autobench/work/go/misc/benchcmp /home/sebastian/development/benchmark/autobench/work/runtime-go1.1.2.txt /home/sebastian/development/benchmark/autobench/work/runtime-go1.2rc3.txt
    benchmark                                old ns/op    new ns/op    delta
    BenchmarkAppend                                108          109   +0.93%
    BenchmarkAppend1Byte                           225          145  -35.56%
    BenchmarkAppend4Bytes                          307          138  -55.05%
    BenchmarkAppend8Bytes                          393          136  -65.39%
    BenchmarkAppend16Bytes                         551          243  -55.90%
    BenchmarkAppend32Bytes                         547          142  -74.04%
    BenchmarkAppendSpecialCase                      46           40  -12.26%
    BenchmarkSelectUncontended                     277          282   +1.81%
    BenchmarkSelectContended                       275          275   +0.00%
    BenchmarkSelectNonblock                        120          117   -2.50%
    BenchmarkChanUncontended                        70           72   +2.69%
    BenchmarkChanContended                          70           72   +2.12%
    BenchmarkChanSync                              164          157   -4.27%
    BenchmarkChanProdCons0                         166          165   -0.60%
    BenchmarkChanProdCons10                         97           98   +0.72%
    BenchmarkChanProdCons100                        74           73   -1.87%
    BenchmarkChanProdConsWork0                     804          823   +2.36%
    BenchmarkChanProdConsWork10                    745          750   +0.67%
    BenchmarkChanProdConsWork100                   727          717   -1.38%
    BenchmarkChanCreation                          196          188   -4.08%
    BenchmarkChanSem                                69           69   +0.00%
    BenchmarkCallClosure                             2            2   +1.89%
    BenchmarkCallClosure1                            3            3   +0.26%
    BenchmarkCallClosure2                           57           46  -19.76%
    BenchmarkCallClosure3                           57           47  -18.09%
    BenchmarkCallClosure4                           58           47  -17.84%
    BenchmarkComplex128DivNormal                    25           26   +1.17%
    BenchmarkComplex128DivNisNaN                    15           16   +0.63%
    BenchmarkComplex128DivDisNaN                    16           15   -6.21%
    BenchmarkComplex128DivNisInf                    11           11   +3.51%
    BenchmarkComplex128DivDisInf                    11           11   +0.85%
    BenchmarkConvT2ESmall                           11           10   -1.82%
    BenchmarkConvT2EUintptr                          0            0   +1.41%
    BenchmarkConvT2ELarge                           65           58   -9.54%
    BenchmarkConvT2ISmall                           13           13   -0.75%
    BenchmarkConvT2IUintptr                          1            1   +0.00%
    BenchmarkConvT2ILarge                           69           61  -11.65%
    BenchmarkConvI2E                                 4            4   +1.69%
    BenchmarkConvI2I                                17           18   +4.00%
    BenchmarkAssertE2T                              13           12   -5.93%
    BenchmarkAssertE2TLarge                         13           13   -0.73%
    BenchmarkAssertE2I                              20           20   +1.00%
    BenchmarkAssertI2T                              12           12   -2.36%
    BenchmarkAssertI2I                              19           18   -4.55%
    BenchmarkAssertI2E                               4            4   +0.65%
    BenchmarkAssertE2E                               0            0   +0.00%
    BenchmarkMalloc8                                52           42  -19.35%
    BenchmarkMalloc16                               57           48  -15.74%
    BenchmarkMallocTypeInfo8                        77           63  -18.13%
    BenchmarkMallocTypeInfo16                       80           68  -14.75%
    BenchmarkHashStringSpeed                        28           30   +7.77%
    BenchmarkHashInt32Speed                         17           20  +12.29%
    BenchmarkHashInt64Speed                         17           19   +9.04%
    BenchmarkHashStringArraySpeed                   87           91   +4.00%
    BenchmarkMegMap                                 19           23  +25.13%
    BenchmarkMegOneMap                              26           15  -43.23%
    BenchmarkMegEqMap                            81371        82264   +1.10%
    BenchmarkMegEmptyMap                             3            3   +8.79%
    BenchmarkSmallStrMap                            20           22   +9.57%
    BenchmarkMapStringKeysEight_16                  21           27  +25.93%
    BenchmarkMapStringKeysEight_32                  19           23  +23.16%
    BenchmarkMapStringKeysEight_64                  19           23  +20.94%
    BenchmarkMapStringKeysEight_1M                  19           23  +21.99%
    BenchmarkIntMap                                 13           20  +48.20%
    BenchmarkRepeatedLookupStrMapKey32              40           41   +4.49%
    BenchmarkRepeatedLookupStrMapKey1M          357805       351139   -1.86%
    BenchmarkNewEmptyMap                           135          135   +0.00%
    BenchmarkMemmove32                              24            6  -73.38%
    BenchmarkMemmove4K                             206          205   -0.49%
    BenchmarkMemmove64K                           3985         3845   -3.51%
    BenchmarkMemmove4M                          910392       907557   -0.31%
    BenchmarkMemmove64M                       17304313     17181375   -0.71%
    BenchmarkFinalizer                             179          180   +0.56%
    BenchmarkFinalizerRun                          792          768   -3.03%
    BenchmarkStackGrowth                           858          748  -12.82%
    BenchmarkStackGrowthDeep                    106290        96324   -9.38%
    BenchmarkCreateGoroutines                      122          124   +1.64%
    BenchmarkCreateGoroutinesParallel              122          123   +0.82%
    BenchmarkMatmult                                12           12   +0.81%
    BenchmarkIfaceCmp100                           249          239   -4.02%
    BenchmarkIfaceCmpNil100                        182          176   -3.30%
    BenchmarkDefer                                 104           77  -25.96%
    BenchmarkDefer10                                98           69  -29.59%
    BenchmarkDeferMany                             150          119  -20.67%
    BenchmarkCompareStringEqual                     11           10  -10.53%
    BenchmarkCompareStringIdentical                  4            4   -0.43%
    BenchmarkCompareStringSameLength                 9            9   -5.21%
    BenchmarkCompareStringDifferentLength            2            2   +0.49%
    BenchmarkCompareStringBigUnaligned           89513        89606   +0.10%
    BenchmarkCompareStringBig                    87135        84454   -3.08%
    
    benchmark                                 old MB/s     new MB/s  speedup
    BenchmarkMemmove32                         1330.93      5006.49    3.76x
    BenchmarkMemmove4K                        19839.13     19979.73    1.01x
    BenchmarkMemmove64K                       16445.23     17041.05    1.04x
    BenchmarkMemmove4M                         4607.14      4621.53    1.00x
    BenchmarkMemmove64M                        3878.16      3905.91    1.01x
    BenchmarkCompareStringBigUnaligned        11714.23     11702.09    1.00x
    BenchmarkCompareStringBig                 12033.93     12415.98    1.03x
    /home/sebastian/development/benchmark/autobench/work/go/misc/benchcmp /home/sebastian/development/benchmark/autobench/work/http-go1.1.2.txt /home/sebastian/development/benchmark/autobench/work/http-go1.2rc3.txt
    benchmark                                   old ns/op    new ns/op    delta
    BenchmarkHeaderWriteSubset                       1565         1123  -28.24%
    BenchmarkReadRequestChrome                       7733         6539  -15.44%
    BenchmarkReadRequestCurl                         4081         3547  -13.09%
    BenchmarkReadRequestApachebench                  4131         3614  -12.52%
    BenchmarkReadRequestSiege                        5951         4676  -21.42%
    BenchmarkReadRequestWrk                          2810         2519  -10.36%
    BenchmarkClientServer                           66308        60238   -9.15%
    BenchmarkClientServerParallel4                  62411        57224   -8.31%
    BenchmarkClientServerParallel64                 65226        61024   -6.44%
    BenchmarkServer                                148454       159640   +7.53%
    BenchmarkServerFakeConnNoKeepAlive              18121        13405  -26.03%
    BenchmarkServerFakeConnWithKeepAlive            15168        11276  -25.66%
    BenchmarkServerFakeConnWithKeepAliveLite         9862         7076  -28.25%
    BenchmarkServerHandlerTypeLen                   18129         9048  -50.09%
    BenchmarkServerHandlerNoLen                     11307         8016  -29.11%
    BenchmarkServerHandlerNoType                    11737         8344  -28.91%
    BenchmarkServerHandlerNoHeader                   9264         6255  -32.48%
    
    benchmark                                    old MB/s     new MB/s  speedup
    BenchmarkReadRequestChrome                      79.00        93.43    1.18x
    BenchmarkReadRequestCurl                        19.11        21.99    1.15x
    BenchmarkReadRequestApachebench                 19.85        22.69    1.14x
    BenchmarkReadRequestSiege                       25.37        32.29    1.27x
    BenchmarkReadRequestWrk                         14.23        15.88    1.12x
    /home/sebastian/development/benchmark/autobench/work/go/misc/benchcmp /home/sebastian/development/benchmark/autobench/work/floats-go1.1.2.txt /home/sebastian/development/benchmark/autobench/work/floats-go1.2rc3.txt
    benchmark                  old ns/op    new ns/op    delta
    BenchmarkMinSmall                 14           14   +4.20%
    BenchmarkMinMed                  741          785   +5.94%
    BenchmarkMinLarge              70823        74005   +4.49%
    BenchmarkMinHuge            11020214     10984005   -0.33%
    BenchmarkAddTwoSmall              23           24   +6.52%
    BenchmarkAddFourSmall             44           52  +17.63%
    BenchmarkAddTwoMed              1020         1024   +0.39%
    BenchmarkAddFourMed             3023         3058   +1.16%
    BenchmarkAddTwoLarge          106333       108031   +1.60%
    BenchmarkAddFourLarge         400389       403529   +0.78%
    BenchmarkAddTwoHuge         20725263     20506885   -1.05%
    BenchmarkAddFourHuge        61205672     62760577   +2.54%
    BenchmarkLogSumExpSmall          273          276   +1.10%
    BenchmarkLogSumExpMed          20815        20877   +0.30%
    BenchmarkLogSumExpLarge      2095056      2127040   +1.53%
    BenchmarkLogSumExpHuge     218645571    219865694   +0.56%
    BenchmarkDotSmall                 15           19  +21.38%
    BenchmarkDotMed                 1062         1085   +2.17%
    BenchmarkDotLarge             109494       110954   +1.33%
    BenchmarkDotHuge            19107550     17878878   -6.43%
    
  • Snappy error on arm / ubuntu

    Snappy error on arm / ubuntu

    Just trying to run the full suite on 13.04 arm and i get the following error when i run make.

    #snappy
    benchcmp: no repeated benchmarks
    make: *** [snappy] Error 1
    

    Also make float doesn't seem to do anything.

    make float
    make: *** No rule to make target `float'.  Stop.
    

    Any ideas on how to get this snappy segment of benchmarks working?

Related tags
:zap: Go web framework benchmark
:zap: Go web framework benchmark

go-web-framework-benchmark This benchmark suite aims to compare the performance of Go web frameworks. It is inspired by Go HTTP Router Benchmark but t

Dec 28, 2022
Key/Value database benchmark

USAGE: kvbench OPTIONS OVERVIEW kvbench is a simple benchmarking tool to evaluate the read performance of a key/value store while writes are being app

Sep 27, 2022
A rapid http(s) benchmark tool written in Go
A rapid http(s) benchmark tool written in Go

gonetx/httpit httpit is a rapid http(s) benchmark tool which on top of fasthttp. Also thanks to cobra and bubbletea. Installation Get binaries from re

Dec 23, 2022
Cache benchmark for Golang
Cache benchmark for Golang

Cache comparison benchmark for Go This benchmark compares cache algorithms using scrambled zipfian distribution (a few occur very often while many oth

Dec 26, 2022
Go http server benchmark
Go http server benchmark

go-http-server-benchmark The more connections, nbio cost the less memory, and performance the better than other frameworks. We can serve for 1000k or

Sep 1, 2022
Golang json encoders\decoders benchmark

Json encoder and decoder benchmark Test Encode for: Build-in encoder https://github.com/mailru/easyjson https://github.com/json-iterator/go Test Decod

Dec 3, 2021
Go Script Lang Benchmark

Go Script Lang Benchmark git clone github.com/akkuman/go_script_lang_benchmark c

Nov 9, 2022
Redis-benchmark - Simple get, mget and pipelined get benchmark.

redis-benchmark Simple get, mget and pipelined get benchmark. Usage git clone https://github.com/Ali-A-A/redis-benchmark.git cd ./redis-benchmark dock

Dec 31, 2021
Bxd redis benchmark - Redis benchmark tool for golang

使用 redis benchmark 工具, 测试 10 20 50 100 200 1k 5k 字节 value 大小,redis get set 性能。 r

Jan 22, 2022
Go-driver-benchmark - Driver benchmark with golang

We use ClickkHouse for time-series databases, and the driver's performance is ve

Sep 5, 2022
Benchmark - Benchmark of logr implementations

Benchmark of logr implementations Implementations a function (can bridge to non-

Nov 6, 2022
Harness Drone/CIE SonarQube Plugin with Quality Gateway
 Harness Drone/CIE SonarQube Plugin with Quality Gateway

Harness Drone/CIE SonarQube Plugin with Quality Gateway The plugin of Harness Drone/CIE to integrate with SonarQube (previously called Sonar), which i

Nov 16, 2022
firedrill is a malware simulation harness for evaluating your security controls
firedrill is a malware simulation harness for evaluating your security controls

firedrill ?? Malware simulation harness. Build native binaries for Windows, Linux and Mac simulating malicious behaviours. Test the effectiveness of y

Dec 22, 2022
Go HTTP request router and web framework benchmark

Go HTTP Router Benchmark This benchmark suite aims to compare the performance of HTTP request routers for Go by implementing the routing structure of

Dec 27, 2022
:zap: Go web framework benchmark
:zap: Go web framework benchmark

go-web-framework-benchmark This benchmark suite aims to compare the performance of Go web frameworks. It is inspired by Go HTTP Router Benchmark but t

Dec 28, 2022
Key/Value database benchmark

USAGE: kvbench OPTIONS OVERVIEW kvbench is a simple benchmarking tool to evaluate the read performance of a key/value store while writes are being app

Sep 27, 2022
A rapid http(s) benchmark tool written in Go
A rapid http(s) benchmark tool written in Go

gonetx/httpit httpit is a rapid http(s) benchmark tool which on top of fasthttp. Also thanks to cobra and bubbletea. Installation Get binaries from re

Dec 23, 2022
Cache benchmark for Golang
Cache benchmark for Golang

Cache comparison benchmark for Go This benchmark compares cache algorithms using scrambled zipfian distribution (a few occur very often while many oth

Dec 26, 2022
:zap: Go web framework benchmark
:zap: Go web framework benchmark

go-web-framework-benchmark This benchmark suite aims to compare the performance of Go web frameworks. It is inspired by Go HTTP Router Benchmark but t

Dec 31, 2022
A CLI to turn Go's benchmark output into pictures

benchdraw benchdraw allows you to make easy to read picture plots from data in Go's benchmark format, implemented in pure Go. Benchdraw does not try t

Nov 28, 2022