An HTTP proxy library for Go

Introduction

GoDoc Join the chat at https://gitter.im/elazarl/goproxy Status

Package goproxy provides a customizable HTTP proxy library for Go (golang),

It supports regular HTTP proxy, HTTPS through CONNECT, and "hijacking" HTTPS connection using "Man in the Middle" style attack.

The intent of the proxy is to be usable with reasonable amount of traffic, yet customizable and programmable.

The proxy itself is simply a net/http handler.

In order to use goproxy, one should set their browser to use goproxy as an HTTP proxy. Here is how you do that in Chrome and in Firefox.

For example, the URL you should use as proxy when running ./bin/basic is localhost:8080, as this is the default binding for the basic proxy.

Mailing List

New features will be discussed on the mailing list before their development.

Latest Stable Release

Get the latest goproxy from gopkg.in/elazarl/goproxy.v1.

Why not Fiddler2?

Fiddler is an excellent software with similar intent. However, Fiddler is not as customizable as goproxy intends to be. The main difference is, Fiddler is not intended to be used as a real proxy.

A possible use case that suits goproxy but not Fiddler, is gathering statistics on page load times for a certain website over a week. With goproxy you could ask all your users to set their proxy to a dedicated machine running a goproxy server. Fiddler is a GUI app not designed to be run like a server for multiple users.

A taste of goproxy

To get a taste of goproxy, a basic HTTP/HTTPS transparent proxy

package main

import (
    "github.com/elazarl/goproxy"
    "log"
    "net/http"
)

func main() {
    proxy := goproxy.NewProxyHttpServer()
    proxy.Verbose = true
    log.Fatal(http.ListenAndServe(":8080", proxy))
}

This line will add X-GoProxy: yxorPoG-X header to all requests sent through the proxy

proxy.OnRequest().DoFunc(
    func(r *http.Request,ctx *goproxy.ProxyCtx)(*http.Request,*http.Response) {
        r.Header.Set("X-GoProxy","yxorPoG-X")
        return r,nil
    })

DoFunc will process all incoming requests to the proxy. It will add a header to the request and return it. The proxy will send the modified request.

Note that we returned nil value as the response. Had we returned a response, goproxy would have discarded the request and sent the new response to the client.

In order to refuse connections to reddit at work time

proxy.OnRequest(goproxy.DstHostIs("www.reddit.com")).DoFunc(
    func(r *http.Request,ctx *goproxy.ProxyCtx)(*http.Request,*http.Response) {
        if h,_,_ := time.Now().Clock(); h >= 8 && h <= 17 {
            return r,goproxy.NewResponse(r,
                    goproxy.ContentTypeText,http.StatusForbidden,
                    "Don't waste your time!")
        }
        return r,nil
})

DstHostIs returns a ReqCondition, that is a function receiving a Request and returning a boolean. We will only process requests that match the condition. DstHostIs("www.reddit.com") will return a ReqCondition accepting only requests directed to "www.reddit.com".

DoFunc will receive a function that will preprocess the request. We can change the request, or return a response. If the time is between 8:00am and 17:00pm, we will reject the request, and return a precanned text response saying "do not waste your time".

See additional examples in the examples directory.

Type of handlers for manipulating connect/req/resp behavior

There are 3 kinds of useful handlers to manipulate the behavior, as follows:

// handler called after receiving HTTP CONNECT from the client, and before proxy establish connection 
// with destination host
httpsHandlers   []HttpsHandler
    
// handler called before proxy send HTTP request to destination host
reqHandlers     []ReqHandler 
    
// handler called after proxy receives HTTP Response from destination host, and before proxy forward 
// the Response to the client.
respHandlers    []RespHandler 

Depending on what you want to manipulate, the ways to add handlers to each handler list are:

// Add handlers to httpsHandlers 
proxy.OnRequest(Some ReqConditions).HandleConnect(YourHandlerFunc())

// Add handlers to reqHandlers
proxy.OnRequest(Some ReqConditions).Do(YourReqHandlerFunc())

// Add handlers to respHandlers
proxy.OnResponse(Some RespConditions).Do(YourRespHandlerFunc())

For example:

// This rejects the HTTPS request to *.reddit.com during HTTP CONNECT phase
proxy.OnRequest(goproxy.ReqHostMatches(regexp.MustCompile("reddit.*:443$"))).HandleConnect(goproxy.RejectConnect)

// This will NOT reject the HTTPS request with URL ending with gif, due to the fact that proxy 
// only got the URL.Hostname and URL.Port during the HTTP CONNECT phase if the scheme is HTTPS, which is
// quiet common these days.
proxy.OnRequest(goproxy.UrlMatches(regexp.MustCompile(`.*gif$`))).HandleConnect(goproxy.RejectConnect)

// The correct way to manipulate the HTTP request using URL.Path as condition is:
proxy.OnRequest(goproxy.UrlMatches(regexp.MustCompile(`.*gif$`))).Do(YourReqHandlerFunc())

What's New

  1. Ability to Hijack CONNECT requests. See the eavesdropper example
  2. Transparent proxy support for http/https including MITM certificate generation for TLS. See the transparent example.

License

I put the software temporarily under the Go-compatible BSD license. If this prevents someone from using the software, do let me know and I'll consider changing it.

At any rate, user feedback is very important for me, so I'll be delighted to know if you're using this package.

Beta Software

I've received positive feedback from a few people who use goproxy in production settings. I believe it is good enough for usage.

I'll try to keep reasonable backwards compatibility. In case of a major API change, I'll change the import path.

Owner
Comments
  • https mitm - Google

    https mitm - Google

    First of all thank you for putting together this project.

    I'm trying the eavesdropper example and I can't get Google to return search results in https. I'm using Firefox. Note that mitmproxy (http://mitmproxy.org/doc/howmitmproxy.html) is working fine.

  • [RFC] Implement SNI sniffing based on github.com/inconshreveable/go-vhost

    [RFC] Implement SNI sniffing based on github.com/inconshreveable/go-vhost

    Changes the behavior slightly for the "ConnectAccept" case, which will always send a "200 OK", but will shut down the connection if the remote end doesn't answer.

    The SniffSNI property on the ProxyHttpServer is a proposition, so we could handle both cases. For those who don't need SNI, we could skip the overhead, but those who enable it, would accept the altered "ConnectAccept".

    The code doesn't deal with this switch, but if the idea is accepted, I could rework it to make it a nicely switchable feature.

  • Ability to MITM with known keys and certificates

    Ability to MITM with known keys and certificates

    Hi, it looks like the ConnectMitm action always generates a new key/certificate for every request.

    I actually have our production keys/certificates that I'd like to use (as the reason we're doing this is for development)

    Would love to be able to load up the list of known keys/certificates for given domains.

  • Default embedded certificate cannot be parsed with Go ~ 1.8 (tip)

    Default embedded certificate cannot be parsed with Go ~ 1.8 (tip)

    Hello,

    Since this Go commit:

    crypto/x509: require a NULL parameters for RSA public keys. https://github.com/golang/go/commit/59aeac20c0412442848982a9287b4bab66c25682

    The certificate in certs.go fails to parse with:

    panic: Error parsing builtin CA x509: RSA key missing NULL parameters

    Maybe it is time to generate a new one.

  • http_load commands make goproxy use more and more memory

    http_load commands make goproxy use more and more memory

    Following http_load commands make goproxy use more and more memory each time when running:

    ./http_load -proxy 192.168.0.88:8080 -parallel 10 -seconds 1200 urls ./http_load -proxy 192.168.0.88:8080 -rate 10 -jitter -seconds 1200 urls

    where urls file consists of some 13 URL. Top reports:

    last pid: 3997; load averages: 0.70, 0.83, 0.71 up 0+23:24:29 18:49:30 43 processes: 1 running, 42 sleeping CPU: 0.8% user, 0.0% nice, 8.6% system, 0.4% interrupt, 90.2% idle Mem: 307M Active, 352M Inact, 294M Wired, 16M Cache, 154M Buf, 872K Free Swap: 4096M Total, 174M Used, 3922M Free, 4% Inuse

    PID USERNAME THR PRI NICE SIZE RES STATE TIME WCPU COMMAND 3688 root 30 29 0 412M 387M uwait 15:53 1.36% my_goproxy 3371 tsgan 1 20 0 86360K 1776K select 0:51 0.23% sshd 3687 root 6 20 0 26152K 5944K uwait 0:59 0.21% gcvis 3396 tsgan 1 20 0 21816K 1432K RUN 0:18 0.09% top

    gcvis reports: screen shot 2014-10-26 at 6 44 45 pm

    my_proxy is simple code that has: ... proxy := goproxy.NewProxyHttpServer() proxy.Verbose = true log.Fatal(http.ListenAndServe(":8080", proxy)) ...

  • https proxy example please

    https proxy example please

    I'm trying to learn to use goproxy through the example https://github.com/elazarl/goproxy/tree/master/examples/goproxy-httpdump.

    I found that it can only handle http requests, not https'. Could you provide an example that can handle https requests please? (As almost all web servers are moving away from http to https now, e.g., github, wikipedia, wordpress, etc).

    PS, by "handle https requests" I meant that none of the https requests are dumped by the current goproxy-httpdump program.

    Thanks

  • regretting after overflow makes no sense

    regretting after overflow makes no sense

    Hi,

    It seems image decode is giving this error.

    2014/10/07 15:38:50 http: panic serving 192.168.1.78:3446: regretting after overflow makes no sense goroutine 95 [running]: net/http.func011() /usr/local/go/src/pkg/net/http/server.go:1100 +0xb7 runtime.panic(0x6a7b20, 0xc2088004c0) /usr/local/go/src/pkg/runtime/panic.c:248 +0x18d github.com/elazarl/goproxy/regretable.(_RegretableReader).Regret(0xc2089ae500) /home/tsgan/go/src/github.com/elazarl/goproxy/regretable/regretreader.go:49 +0x70 github.com/elazarl/goproxy/ext/image.func001(0xc20809e240, 0xc2089ae370, 0x800baf538) /home/tsgan/go/src/github.com/elazarl/goproxy/ext/image/image.go:38 +0x20c github.com/elazarl/goproxy.FuncRespHandler.Handle(0xc2081ef610, 0xc20809e240, 0xc2089ae370, 0xc2081f0000) /home/tsgan/go/src/github.com/elazarl/goproxy/actions.go:35 +0x36 github.com/elazarl/goproxy.func016(0xc20809e240, 0xc2089ae370, 0x13) /home/tsgan/go/src/github.com/elazarl/goproxy/dispatcher.go:279 +0x23f github.com/elazarl/goproxy.FuncRespHandler.Handle(0xc2081b12e0, 0xc20809e240, 0xc2089ae370, 0x2) /home/tsgan/go/src/github.com/elazarl/goproxy/actions.go:35 +0x36 github.com/elazarl/goproxy.(_ProxyHttpServer).filterResponse(0xc208054280, 0xc20809e240, 0xc2089ae370, 0xc20809e240) /home/tsgan/go/src/github.com/elazarl/goproxy/proxy.go:69 +0xb3 github.com/elazarl/goproxy.(_ProxyHttpServer).ServeHTTP(0xc208054280, 0x800bb6d20, 0xc208998280, 0xc208027ba0) /home/tsgan/go/src/github.com/elazarl/goproxy/proxy.go:123 +0x6e7 net/http.serverHandler.ServeHTTP(0xc2080045a0, 0x800bb6d20, 0xc208998280, 0xc208027ba0) /usr/local/go/src/pkg/net/http/server.go:1673 +0x19f net/http.(_conn).serve(0xc208054580) /usr/local/go/src/pkg/net/http/server.go:1174 +0xa7e created by net/http.(*Server).Serve /usr/local/go/src/pkg/net/http/server.go:1721 +0x313

  • Add ability to accept direct non-proxy connections

    Add ability to accept direct non-proxy connections

    Hi,

    Have been building out a little service stubbing tool here based on your library (which is great by the way!): https://github.com/BenPhegan/fauxpi

    One thing that would be useful is to be able to accept direct as well as proxied connections to enable service stubbing directly for apps that do not support proxy use.

    A recent change https://github.com/elazarl/goproxy/commit/44e67200a91914114c155004d63afa8babc86f00 has prevented direct connections, which is a good fix for the problem it addresses.

    This pull request is to make this the default behaviour still, but enable the proxy to be set to respond to direct requests via setting an AcceptDirect bool on the proxy object.

    Very new to Golang, so not sure if there is a better way to do this or whether the tests (copied from the inverse) are correct. Happy to discuss further!

    Regards,

    Ben Phegan

  • Allow proxies over HTTPS connections

    Allow proxies over HTTPS connections

    Currently https.go has a cast to TCPConn, which is needed to close the read/write sides of the connections after all IO has been piped. This means that connections of type tls.Conn, which cannot be casted, will fail. This change removes the cast, closing both pipes when no more IO is available.

    This allows HTTPS proxies, e.g.

    proxy := goproxy.NewProxyHttpServer()
    http.ListenAndServeTLS(":8081", "cert.pem", "key.pem", proxy)
    
  • Why is ctx.Req.Body empty in OnResponse()?

    Why is ctx.Req.Body empty in OnResponse()?

    In OnResponse() I would like to view what the request body was. But ctx.Req.Body is empty, even when ctx.Req.ContentLength is not 0. What can be done?

  • What's mean of

    What's mean of "too many open files"?

    I find some log is :

    2014/11/18 20:03:06 http: Accept error: accept tcp [::]:80: too many open files; retrying in 5ms
    2014/11/18 20:03:06 http: Accept error: accept tcp [::]:80: too many open files; retrying in 10ms
    2014/11/18 20:03:06 http: Accept error: accept tcp [::]:80: too many open files; retrying in 20ms
    2014/11/18 20:03:06 http: Accept error: accept tcp [::]:80: too many open files; retrying in 40ms
    2014/11/18 20:03:06 http: Accept error: accept tcp [::]:80: too many open files; retrying in 80ms
    

    What conditioin/sense it happens?

    goproxy may support how many clients?

  • Facebook and Instagram apps fail to work through GoProxy

    Facebook and Instagram apps fail to work through GoProxy

    Hi,

    I've been using GoProxy for a while, and I've noticed that both the Facebook app and Instagram fail to work with GoProxy. In Facebook app, if you click on the Friend tab, it displays "Unable to connect".

    I have tested GoProxy in a basic mode by using GoProxy-Basic from the examples and but both Facebook and Instagram still don't work.

    I can't see any errors generated by GoProxy, so I am a bit lost as to what the apps can't do when running through GoProxy as a proxy server.

    Does anyone have any ideas?

  • HTTP error handling customization

    HTTP error handling customization

    This PR unifies HTTP and HTTPS error handling and makes it customizable by adding Proxy.HTTPErrorHandler.

    • HTTP code is moved to http.go file
    • HTTP code calls httpError if response in nil
    • httpError uses Proxy.HTTPErrorHandler if specified

    Fixes #51

  • add method for ProxyHttpServer object

    add method for ProxyHttpServer object

    I will execute Proxy.OnResponse().DoFunc many times.

    type ProxyHttpServer struct { respHandlers []RespHandler }

    so,I hope clear respHandlers chain please add method for ProxyHttpServer to clear respHandlers

    add method like this: func (proxy *ProxyHttpServer) ClearRespHandlers() { proxy.respHandlers = proxy.respHandlers[0:0] }

    thank you

  • how to print client IP addrs on my server?

    how to print client IP addrs on my server?

    Excuse me, how to print client IP addrs on my server?

    func main() { flag.Usage = func() { flag.PrintDefaults() } flag.Parse() if serverPort == "" { log.Fatalln("Error: please setting correct server port") } endProxy := goproxy.NewProxyHttpServer() endProxy.Verbose = true auth.ProxyBasic(endProxy, "my_realm", func(user, pwd string) bool { return user == username && pwd == password }) log.Println("serving proxy server port at", serverPort) http.ListenAndServe(":"+serverPort, endProxy) fmt.Println("The proxy server end.") }

  • IsLocalHost: support host:port

    IsLocalHost: support host:port

    Without this patch IsLocalHost does not work for URLs with port specified i.e. it works for http://localhost but does not work for http://localhost:80 or http://localhost:10000.

    Fixes #487

Related tags
An HTTP/HTTPS intercept proxy written in Go.
An HTTP/HTTPS intercept proxy written in Go.

Broxy Broxy is an open source intercept proxy written in Go. It makes use of goproxy as core proxy implementation and the interface is implemented wit

Nov 29, 2022
This is a simple single-host reverse proxy that intercept and save HTTP requests and responses
This is a simple single-host reverse proxy that intercept and save HTTP requests and responses

HTTP Telescope Debug HTTP requests using a reverse proxy. Description This is a simple single-host reverse proxy that intercept and save HTTP requests

Mar 20, 2022
Fast HTTP package for Go. Tuned for high performance. Zero memory allocations in hot paths. Up to 10x faster than net/http
Fast HTTP package for Go. Tuned for high performance. Zero memory allocations in hot paths. Up to 10x faster than net/http

fasthttp Fast HTTP implementation for Go. Currently fasthttp is successfully used by VertaMedia in a production serving up to 200K rps from more than

Jan 2, 2023
Speak HTTP like a local. (the simple, intuitive HTTP console, golang version)

http-gonsole This is the Go port of the http-console. Speak HTTP like a local Talking to an HTTP server with curl can be fun, but most of the time it'

Jul 14, 2021
Http client call for golang http api calls

httpclient-call-go This library is used to make http calls to different API services Install Package go get

Oct 7, 2022
fhttp is a fork of net/http that provides an array of features pertaining to the fingerprint of the golang http client.

fhttp The f stands for flex. fhttp is a fork of net/http that provides an array of features pertaining to the fingerprint of the golang http client. T

Jan 1, 2023
NATS HTTP Round Tripper - This is a Golang http.RoundTripper that uses NATS as a transport.

This is a Golang http.RoundTripper that uses NATS as a transport. Included is a http.RoundTripper for clients, a server that uses normal HTTP Handlers and any existing http handler mux and a Caddy Server transport.

Dec 6, 2022
Simple HTTP package that wraps net/http

Simple HTTP package that wraps net/http

Jan 17, 2022
Http-conection - A simple example of how to establish a HTTP connection using Golang

A simple example of how to establish a HTTP connection using Golang

Feb 1, 2022
Simple HTTP and REST client library for Go

Resty Simple HTTP and REST client library for Go (inspired by Ruby rest-client) Features section describes in detail about Resty capabilities Resty Co

Jan 1, 2023
A Go HTTP client library for creating and sending API requests
A Go HTTP client library for creating and sending API requests

Sling Sling is a Go HTTP client library for creating and sending API requests. Slings store HTTP Request properties to simplify sending requests and d

Jan 7, 2023
httpreq is an http request library written with Golang to make requests and handle responses easily.

httpreq is an http request library written with Golang to make requests and handle responses easily. Install go get github.com/binalyze/http

Feb 10, 2022
Cake is a lightweight HTTP client library for GO, inspired by Java Open-Feign.

Cake is a lightweight HTTP client library for GO, inspired by Java Open-Feign. Installation # With Go Modules, recommanded with go version > 1.16

Oct 6, 2022
Parcel - HTTP rendering and binding library for Go

parcel HTTP rendering/binding library for Go Getting Started Add to your project

Jan 1, 2022
Httpx - a fast and multi-purpose HTTP toolkit allow to run multiple probers using retryablehttp library
Httpx - a fast and multi-purpose HTTP toolkit allow to run multiple probers using retryablehttp library

httpx is a fast and multi-purpose HTTP toolkit allow to run multiple probers using retryablehttp library, it is designed to maintain the result reliability with increased threads.

Feb 3, 2022
Full-featured, plugin-driven, extensible HTTP client toolkit for Go

gentleman Full-featured, plugin-driven, middleware-oriented toolkit to easily create rich, versatile and composable HTTP clients in Go. gentleman embr

Dec 23, 2022
An enhanced http client for Golang
An enhanced http client for Golang

go-http-client An enhanced http client for Golang Documentation on go.dev ?? This package provides you a http client package for your http requests. Y

Dec 23, 2022
An enhanced HTTP client for Go
An enhanced HTTP client for Go

Heimdall Description Installation Usage Making a simple GET request Creating a hystrix-like circuit breaker Creating a hystrix-like circuit breaker wi

Jan 9, 2023
Enriches the standard go http client with retry functionality.

httpRetry Enriches the standard go http client with retry functionality using a wrapper around the Roundtripper interface. The advantage of this libra

Dec 10, 2022