Some of the TODO would be good starter tasks:

Current set as of this writing:

$ git grep TODO

Makefile:# TODO: do something about cyclomatic complexity
fgrpc/grpcrunner.go:// TODO: refactor common parts between http and grpc runners
fgrpc/grpcrunner.go:            // TODO: option to use certs
fhttp/http.go:// Version is the fortio package version (TODO:auto gen/extract).
fhttp/http.go:                  Timeout: 3 * time.Second, // TODO: make configurable
fhttp/http.go:// TODO: refactor - unwiedly/ugly atm
fhttp/http.go:  // TODO: safer to start with -1 and fix ok for http 1.0
fhttp/http.go:          // TODO: need automated tests
fhttp/http.go:                  c.code = ParseDecimal(c.buffer[retcodeOffset : retcodeOffset+3]) //TODO do that only once...
fhttp/http.go:                  // TODO handle 100 Continue
fhttp/http.go:                  // TODO: keep track of list of newlines to efficiently search headers only there
fhttp/http.go:                                          // TODO: just consume the extra instead
fhttp/http.go:          // TODO: this easily lead to contention - use 'thread local'
fhttp/http.go:// TODO: switch to Duration.Round once switched to go 1.9
fhttp/httprunner.go:    // TODO 1. use std client automatically when https url
pingsrv.go:     // TODO doesn't work for ipv6 addrs etc
stats/stats.go:// TODO: consider using an interval search for the last N big buckets
stats/stats.go: // TODO potentially merge despite different offset/scale
stats/stats_test.go:    // TODO: fix the p51 (and p1...), should be 0 not 10
stats/stats_test.go:    tP := []float64{100.} // TODO: use 75 and fix bug
ui/static/js/fortio_chart.js:// TODO: object-ify
ui/static/js/fortio_chart.js:      // TODO may need updateChart() if we persist settings even the first time
ui/templates/browse.html:<p><!-- TODO: find a way to flush/not need two p to get the form visible! -->
ui/templates/main.html:{{end}}{{end}} <!-- 2 extra header lines, TODO: add a JS 'more headers' button -->
ui/uihandler.go:// TODO: auto map from (Http)RunnerOptions to form generation and/or accept
ui/uihandler.go:// TODO: unit tests, allow additional data sets.
ui/uihandler.go:        debugPath = ".." + debugpath // TODO: calculate actual path if not same number of directories

• adds option to run grpc load test from the UI
• #TODO add more options for configuring the grpc run
• this PR may need some refactoring, but sending this out for feedback and course-correction

@ldemailly can you review? w.r.t #146 thanks a lot for creating this tool! it has proven quite useful in quick evaluation of grpc services.

before switching to 1.9 let's measure that it's same or better (in term of fortio self qps - should be > 100k qps with the fast client, benchmark against the stdclient to see if it's gotten better)

cc @bochunz want to run / compare ?

More about the UI custom header bug

Custom header parsing must be performed before headers are consumed. Certain custom headers, like Content-Type, were inadvertently being ignored since HTTPOptions.AllHeaders() was being called to populate the template variables in the !JSONOnly block.

Screenshot of UI with payload field

We have been using fortio to measure Istio's latency. One of the problems with its long-tail latency issue is the benchmark itself. When we specify -c 100, 100 goroutines are created to send off 100 requests all at the same time (using tcpdump, we saw these requests all got to the server side within 100us). So on the server side, it sees these mini-bursts of requests over and over again. Due to Envoy worker threads being single threaded, this creates a significant queuing delay. In the real world, bursts of requests do happen, but we should not artificially create thundering herd problem to make Istio (or whatever is being tested) to have latency problem that is worse than what it really is.

A flag --request-jitter is added to add (+/-)10% jitter between requests of parallel clients. Here is the result measuring Istio latency with and without the flag:

• Without added jitter:

# ./fortio load -qps 1000 -c 100 -t 300s 10.97.103.115:8079
Fortio 1.3.2-pre running at 1000 queries per second, 4->4 procs, for 5m0s: 10.97.103.115:8079
16:33:26 I httprunner.go:82> Starting http test for 10.97.103.115:8079 with 100 threads at 1000.0 qps
16:33:26 W http_client.go:142> Assuming http:// on missing scheme for '10.97.103.115:8079'
Starting at 1000 qps with 100 thread(s) [gomax 4] for 5m0s : 3000 calls each (total 300000)
Ended after 5m0.013834356s : 300000 calls. qps=999.95
Sleep times : count 299900 avg 0.088003375 +/- 0.00315 min 0.059958014 max 0.098711726 sum 26392.2122
Aggregated Function Time : count 300000 avg 0.011201918 +/- 0.003032 min 0.001004862 max 0.038903617 sum 3360.57543
# range, mid point, percentile, count
>= 0.00100486 <= 0.002 , 0.00150243 , 0.01, 19
> 0.002 <= 0.003 , 0.0025 , 0.02, 33
> 0.003 <= 0.004 , 0.0035 , 0.04, 75
> 0.004 <= 0.005 , 0.0045 , 0.07, 96
> 0.005 <= 0.006 , 0.0055 , 0.27, 601
> 0.006 <= 0.007 , 0.0065 , 3.53, 9766
> 0.007 <= 0.008 , 0.0075 , 14.33, 32409
> 0.008 <= 0.009 , 0.0085 , 28.50, 42496
> 0.009 <= 0.01 , 0.0095 , 41.03, 37599
> 0.01 <= 0.011 , 0.0105 , 52.01, 32940
> 0.011 <= 0.012 , 0.0115 , 62.61, 31792
> 0.012 <= 0.014 , 0.013 , 81.53, 56771
> 0.014 <= 0.016 , 0.015 , 93.95, 37256
> 0.016 <= 0.018 , 0.017 , 97.99, 12126
> 0.018 <= 0.02 , 0.019 , 99.12, 3370
> 0.02 <= 0.025 , 0.0225 , 99.90, 2349
> 0.025 <= 0.03 , 0.0275 , 100.00, 300
> 0.03 <= 0.035 , 0.0325 , 100.00, 1
> 0.035 <= 0.0389036 , 0.0369518 , 100.00, 1
# target 50% 0.0108168
# target 75% 0.0133096
# target 90% 0.0153637
# target 99% 0.0197929
# target 99.9% 0.0250333
Sockets used: 100 (for perfect keepalive, would be 100)
Code 200 : 300000 (100.0 %)
Response Header Sizes : count 300000 avg 295.00439 +/- 0.06609 min 295 max 296 sum 88501316
Response Body/Total Sizes : count 300000 avg 295.00439 +/- 0.06609 min 295 max 296 sum 88501316
All done 300000 calls (plus 100 warmup) 11.202 ms avg, 1000.0 qps

• With the --request-jitter flag:

# ./fortio load -qps 1000 -c 100 -t 300s --request-jitter 10.97.103.115:8079
Fortio 1.3.2-pre running at 1000 queries per second, 4->4 procs, for 5m0s: 10.97.103.115:8079
16:28:15 I httprunner.go:82> Starting http test for 10.97.103.115:8079 with 100 threads at 1000.0 qps
16:28:15 W http_client.go:142> Assuming http:// on missing scheme for '10.97.103.115:8079'
Starting at 1000 qps with 100 thread(s) [gomax 4] for 5m0s : 3000 calls each (total 300000)
Ended after 5m0.010714621s : 300000 calls. qps=999.96
Sleep times : count 299900 avg 0.097020439 +/- 0.00854 min 0.054097274 max 0.119709755 sum 29096.4297
Aggregated Function Time : count 300000 avg 0.0027441949 +/- 0.002333 min 0.000662307 max 0.031994066 sum 823.258457
# range, mid point, percentile, count
>= 0.000662307 <= 0.001 , 0.000831154 , 7.27, 21823
> 0.001 <= 0.002 , 0.0015 , 54.81, 142595
> 0.002 <= 0.003 , 0.0025 , 73.09, 54853
> 0.003 <= 0.004 , 0.0035 , 80.92, 23490
> 0.004 <= 0.005 , 0.0045 , 86.02, 15301
> 0.005 <= 0.006 , 0.0055 , 90.61, 13770
> 0.006 <= 0.007 , 0.0065 , 93.94, 9987
> 0.007 <= 0.008 , 0.0075 , 96.14, 6611
> 0.008 <= 0.009 , 0.0085 , 97.51, 4090
> 0.009 <= 0.01 , 0.0095 , 98.32, 2426
> 0.01 <= 0.011 , 0.0105 , 98.80, 1450
> 0.011 <= 0.012 , 0.0115 , 99.12, 960
> 0.012 <= 0.014 , 0.013 , 99.55, 1281
> 0.014 <= 0.016 , 0.015 , 99.77, 676
> 0.016 <= 0.018 , 0.017 , 99.86, 266
> 0.018 <= 0.02 , 0.019 , 99.92, 168
> 0.02 <= 0.025 , 0.0225 , 99.98, 179
> 0.025 <= 0.03 , 0.0275 , 100.00, 62
> 0.03 <= 0.0319941 , 0.030997 , 100.00, 12
# target 50% 0.00189889
# target 75% 0.00324389
# target 90% 0.00586696
# target 99% 0.0116292
# target 99.9% 0.0194405
Sockets used: 100 (for perfect keepalive, would be 100)
Code 200 : 300000 (100.0 %)
Response Header Sizes : count 300000 avg 295.00006 +/- 0.007528 min 295 max 296 sum 88500017
Response Body/Total Sizes : count 300000 avg 295.00006 +/- 0.007528 min 295 max 296 sum 88500017
All done 300000 calls (plus 100 warmup) 2.744 ms avg, 1000.0 qps

The latency difference is pretty significant with or without jitter under the same test condition (1000 qps, 100 clients, 300 seconds). Avg latency went from 11.2ms to 2.7ms. Long-tail latency also improved significantly. This problem with request jitter has been discussed here as well: https://github.com/envoyproxy/envoy/issues/5536

this is a very rough first cut for #99 creating this to gather feedback

• refactor results operations to a results package. the logic can be shared between fortio_main and uihandler
• more assumptions outlined in comments on this PR

For people who don't already have go installed, it would be great if someone would create a brew recipe: https://docs.brew.sh/How-To-Open-a-Homebrew-Pull-Request.html

(apparently they don't want the main author to do it, but please update this issue if you contribute one so I can help make sure it works well)

• Switching build image to golang:1.17.8
• Changes golangci-lint installation due to golangci/golangci-lint#2374
• Building windows/mac binaries only when architecture is amd64 in fortio Dockerfile
• The docker Debian repo is now added based on the arch

An alternative that allows installing golang-ci-lint as before also on ARM64 is by bumping to golang:1.18. See the issue above.

cc @yselkowitz

Refers ARMOCP-293

Originally I was solving this issue filed here:https://github.com/istio/istio/issues/21289

While digging into the code, I found it was originated from fortio side. Since in the fortio jsonfilename aka ID() function sets the length limit to be 64.

Offline discussed with @mandarjog and xinnan, we can either use the unix epoch time format to get rid of this https://github.com/fortio/fortio/blob/master/periodic/periodic.go#L573 formatDate() func, or make the length limit to be larger.

While, after comparing with the unix time with the time generated from formatDate() func, I found that there is not so much decrease in length. Like unix time: 1582245603 is 10 chars. The time generated from formatDate(): 2020-02-20-114432 is 17 chars. While, the drawback of using unix time is that we have to do a conversion to know the exact date and time, which is inconvenience.

Therefore, I decide to only change the length limit.

fixes #181

@ldemailly this is a minor fix that I had on my local version as the escape logic failed a few of my perf test URLs. sending this in based on discussion on 181.

did i miss any documentation updates?

fortio version: docker package->istio/fortio:latest from 01/23/2018

18:11:12 F fortio_main.go:273> Unable to json serialize result: json: unsupported value: NaN panic: aborting... goroutine 1 [running]: istio.io/fortio/log.logPrintf(0x6, 0x931b21, 0x23, 0xc420205a50, 0x1, 0x1) /go/src/istio.io/fortio/log/logger.go:160 +0x28c istio.io/fortio/log.Fatalf(0x931b21, 0x23, 0xc420205a50, 0x1, 0x1) /go/src/istio.io/fortio/log/logger.go:208 +0x5c main.fortioLoad() /go/src/istio.io/fortio/fortio_main.go:273 +0xb5c main.main() /go/src/istio.io/fortio/fortio_main.go:144 +0x669

Now fortio does not support setting grpc metadata like '-H' flag in Http Load Generation and it would be better if it did. I'd be happy to do the work if needed.

• [x] adding optional ClientTrace and Context to jrpc
• [x] also removed deprecated jrpc functions: CallNoPayload -> Get; CallWithPayload -> Fetch
• [x] adding to fhttp std client
• [x] extend the logger api to be able to be used for Otel traces
• [x] demonstrate usage in load test: see https://github.com/fortio/fortiotel/blob/main/fortio_with_otel.go

fixes #650

From @wiardvanrij

Wiard van Rij

For jrpc in fortio (or maybe even other parts in fortio) - it would be nice to be able to instrument calls. Basically: https://go.dev/blog/http-tracing You wrap the ctx in a httptrace.WithClientTrace I think it would be possible to implement this without making fundamental changes by allowing someone to pass along a httptrace.ClientTrace object into the Destination struct here: https://github.com/fortio/fortio/blob/master/jrpc/jrpcClient.go#L73

fortio server

then in ui trigger test against http://localhost:8080/echo?size=200000 - eg http://localhost:8080/fortio/?url=http%3A%2F%2Flocalhost%3A8080%2Fecho%3Fsize%3D200000&qps=10&t=3s&load=Start shows

Code  -1 : 16 (66.7 %)
Code 200 : 8 (33.3 %)

and in the server logs the actual problem:

13:47:00 W http_client.go:960> [7] Buffer is too small for headers 120 + data 200000 - change -httpbufferkb flag to at least 196

these aren't bubbled up to the ui/visible

-----edited, was ---- Running fortio server and then running a load test against itself using { "url": "http://localhost:8080/?size=131072:50,65536:25,32768:25", "qps": "100", "t": "10s", "p": "50, 95, 99", "jitter": "on", "uniform": "on", "nocatchup": "on", "stdclient": "on" } works properly.

When I remove the stdclient and switch to the fast one about one third of the requests fail with 'broken pipe' while writing the response.