Fork of Go stdlib's net/http that works with alternative TLS libraries like refraction-networking/utls.

github.com/ooni/oohttp

This repository contains a fork of Go's standard library net/http package including patches to allow using this HTTP code with github.com/refraction-networking/utls.

Motivation and maintenance

We created this package because it simplifies testing URLs using specific TLS Client Hello messages. We will continue to keep it up to date as long as it serves our goals.

Usage and limitations

To use this fork, replace

import "net/http"

with

import "github.com/ooni/oohttp"

Please, keep in mind the following limitations:

  1. This fork does not include a fork of pprof because such package depends on the stdlib's internal/profile package. If your code uses http/pprof, then you cannot switch to this fork.

  2. This fork's httptrace package is partly broken because there is no support for network events tracing, which requires the stdlib's internal/nettrace package. If your code depends on network events tracing, then you cannot switch to this fork.

  3. This fork tracks the latest stable version of Go by merging upstream changes into the main branch. This means that it may not be working with earlier versions of Go. For example, when writing this note we are at Go 1.16 and this package accordingly uses io.ReadAll. If you are compiling using Go 1.15, you should get build errors because io.ReadAll did not exist before Go 1.16.

Issue tracker

Please, report issues in the ooni/probe repository. Make sure you mention oohttp in the issue title.

Patches

We started from the src/net/http subtree at go1.16 and we applied patches to fork the codebase (#1, #2 and #3). Then, we introduced the http.TLSConn abstraction that allows using different TLS libraries (#4).

Every major change is documented by a pull request. We may push minor changes (e.g., updating docs) directly on the main branch.

Update procedure

(Adapted from refraction-networking/utls instructions.)

  1. run the following commands:
git remote add golang [email protected]:golang/go.git || git fetch golang
git branch -D golang-upstream golang-http-upstream
git checkout -b golang-upstream go1.16.7
git subtree split -P src/net/http/ -b golang-http-upstream
git checkout merged-main
git merge golang-http-upstream
git push merged-main
  1. make sure the codebase does not assume *net.Conn anywhere (every merge may introduce new changes);

  2. open a pull request on github and merge it with merge commit.

Owner
Open Observatory of Network Interference (OONI)
The Open Observatory of Network Interference (OONI) is a free software project that measures internet censorship around the world.
Open Observatory of Network Interference (OONI)
Comments
  • Update to go 1.19.4

    Update to go 1.19.4

    See https://github.com/ooni/probe/issues/2273

    Here's the diff with the dev branch (aka go1.19.4): https://github.com/ooni/oohttp/compare/2125b6b..82e690c

    Inline you can see the diff with respect to main (aka go1.18.9)

    We also include scripts to have more confidence in the merging process robustness.

  • chore: merge stable into main

    chore: merge stable into main

    We currently have two branches and two release trains. However, this is ~bad because it creates more maintenance burden. So, just reckon we're still using go1.18 and make sure main is basically stable with small changes.

    So, https://github.com/ooni/oohttp/compare/6a399fd..899c58a contains this diff:

    diff --git a/.github/workflows/go.yml b/.github/workflows/go.yml
    index 63678c6..dd75520 100644
    --- a/.github/workflows/go.yml
    +++ b/.github/workflows/go.yml
    @@ -2,9 +2,9 @@ name: Go
     
     on:
       push:
    -    branches: [ main, stable ]
    +    branches: [ main ]
       pull_request:
    -    branches: [ main, stable ]
    +    branches: [ main ]
     
     jobs:
     
    diff --git a/README.md b/README.md
    index 8e75a95..18348d8 100644
    --- a/README.md
    +++ b/README.md
    @@ -223,14 +223,14 @@ minor changes (e.g., updating docs) directly on the `main` branch.
     
     ```bash
     set -ex
    -git checkout stable
    +git checkout main
     git remote add golang [email protected]:golang/go.git || git fetch golang
    -git branch -D golang-upstream golang-http-upstream merged-stable || true
    +git branch -D golang-upstream golang-http-upstream merged-main || true
     git fetch golang
     git checkout -b golang-upstream go1.18.9
     git subtree split -P src/net/http/ -b golang-http-upstream
    -git checkout stable
    -git checkout -b merged-stable
    +git checkout main
    +git checkout -b merged-main
     git merge golang-http-upstream
    
    

    Part of https://github.com/ooni/probe/issues/2273

  • feat: implement better integration tests

    feat: implement better integration tests

    There's a need to write better integration tests for this repository.

    In particular:

    1. we need to have confidence that the JA3 signature produced when using a uTLS client is different than the default one (i.e., can we be confident that it's possible to replace TLS?)

    2. we need to have robust tests that do not depend onto external services but only use localhost, so they don't break often.

    Part of https://github.com/ooni/probe/issues/2273

  • Merge go1.18.5 into the stable v0.2.x train

    Merge go1.18.5 into the stable v0.2.x train

    This pull request merges go1.18.5 changes into the stable train, which corresponds to v0.2.x of this repository.

    Part of https://github.com/ooni/probe/issues/2223

  • chore: update submodules

    chore: update submodules

    Unfortunately ja3er.com is down, so the examples now are much less useful than they were before :-(.

    Part of https://github.com/ooni/probe/issues/2211

  • doc: document TLSClientFactory

    doc: document TLSClientFactory

    It seems TLSClientFactory works regardless of whether we're using a proxy or not.

    Therefore, using the TLSClientFactory field directly is the simplest way to use this library with uTLS.

    Hence, update the examples. The basic example now uses the TLSClientFactory. I've copied the original example to be a more advanced usage example.

    While there, also update the proxy example to use the per-transport TLSClientFactory rather than the global one.

    Also, make sure the README file is in sync with the recently implemented TLSClientFactory changes.

  • feat: add factory for TLS proxying with uTLS (v3)

    feat: add factory for TLS proxying with uTLS (v3)

    This commit build upon 475e58062b3ef43e1dacea9d5ecc39bfd4b71353 to allow users to optionally specify the TLSClientFactory to use on a per-Transport basis. If the transport TLSClientFactory isn't set, we'll default to using the global TLSClientFactory.

    While working on this diff, I realized that, for the common use case, a user doesn't need to override DialTLSContext and they can just set the global or per-transport TLSClientFactory in order to use their uTLS fingerprint of choice.

  • doc: show how to use the experimental tls factory

    doc: show how to use the experimental tls factory

    This commit adds a new example showing how one can use the experimental TLSClientFactory function to create an uTLS connection when using a SOCKS5 (or other) proxy.

    Users have requested this functionality.

    The new example seems to work for me.

    Compare:

    sbs@localhost:~$ go run . -utls | jq -r .ja3_hash
    0ffee3ba8e615ad22535e7f771690a28
    

    to:

    sbs@localhost:~$ go run . | jq -r .ja3_hash
    3fed133de60c35724739b913924b6c24
    

    Once users have confirmed this code works as intended, I'll update the README to advert this functionality.

  • feat: add factory for TLS proxying with uTLS (v2)

    feat: add factory for TLS proxying with uTLS (v2)

    This commit adds an experimental factory for create proxying TLS connections using uTLS rather than crypto/tls.

    A user has requested this functionality.

    For now, I'd like to avoid advertising it until I get confirmation that this interface is okay for the user who requested it.

    When it's confirmed it's okay, I'll change the README.

    The v1 implementation of this functionality was in commit d6f7e24250e479a949b8cd230186fe8a4eaed071. Yet, it turns out the TLSClientFactory ended up having the wrong return type (*tls.Conn). This return type is incorrect; the factory needs to return oohttp.TLSConn instead.

  • Merged from upstream: go1.17

    Merged from upstream: go1.17

    As part of merging, I made sure that *tls.Conn is not used anywhere:

    % git grep 'tls.Conn'
    cgi/child.go:           r.TLS = &tls.ConnectionState{HandshakeComplete: true}
    h2_bundle.go:   ConnectionState() tls.ConnectionState
    h2_bundle.go:// client. If c has a ConnectionState method like a *tls.Conn, the
    h2_bundle.go:           sc.tlsState = new(tls.ConnectionState)
    h2_bundle.go:   tlsState         *tls.ConnectionState        // shared by all handlers, like net/http
    h2_bundle.go:                   // TODO: add CloseWrite to crypto/tls.Conn first
    h2_bundle.go:   var tlsState *tls.ConnectionState // nil if not scheme https
    h2_bundle.go:   // If the returned net.Conn has a ConnectionState method like tls.Conn,
    h2_bundle.go:   tlsState  *tls.ConnectionState // nil only for specialized impls
    httptest/httptest.go:           req.TLS = &tls.ConnectionState{
    httptest/httptest_test.go:                              TLS: &tls.ConnectionState{
    httptrace/trace.go:     TLSHandshakeDone func(tls.ConnectionState, error)
    request.go:     TLS *tls.ConnectionState
    response.go:    TLS *tls.ConnectionState
    server.go:      tlsState *tls.ConnectionState
    server.go:              c.tlsState = new(tls.ConnectionState)
    server.go:              req.TLS = &tls.ConnectionState{}
    tlsconn.go:// TLSConn is the interface representing a *tls.Conn compatible
    tlsconn.go:// connection, which could possibly be different from a *tls.Conn
    tlsconn.go:     ConnectionState() tls.ConnectionState
    transport.go:                   trace.TLSHandshakeDone(tls.ConnectionState{}, err)
    transport.go:                                   trace.TLSHandshakeDone(tls.ConnectionState{}, err)
    transport.go:   tlsState  *tls.ConnectionState
    transport_test.go:              TLSHandshakeDone: func(s tls.ConnectionState, err error) {
    transport_test.go:              TLSHandshakeDone: func(s tls.ConnectionState, err error) {
    transport_test.go:                      TLSHandshakeDone: func(cfg tls.ConnectionState, err error) {
    
  • feat: implement tracing without composition

    feat: implement tracing without composition

    Tracing with composition is tricky for OONI because we sometimes have a DoH resolution (so two HTTP round trips) before a real HTTP round trip when we're using a DoH resolver.

    In such a case, tracing with composition complicates attributing the correct connection to the request that's using it. This happens because the whole chain of operations is using the same context.Context, so we end up attaching three traces to the context (one per round trip).

    Conversely, if traces do not compose, each round trip gets its own Trace and we should be able to bind connections and requests.

    Because ooni/oohttp is experimental and will always be experimental, I don't feel bad about adding this API for experimentation and for our own sake. The original API is still there, so it all feels ~fine.

    I will revert this commit if the experiment fails 😬.

    Reference issue: https://github.com/ooni/probe/issues/2220

Related tags
Native macOS networking for QEMU using vmnet.framework and socket networking.

qemu-vmnet Native macOS networking for QEMU using vmnet.framework and socket networking. Getting started TODO -netdev socket,id=net0,udp=:1234,localad

Jan 5, 2023
High-performance, non-blocking, event-driven, easy-to-use networking framework written in Go, support tls/http1.x/websocket.

High-performance, non-blocking, event-driven, easy-to-use networking framework written in Go, support tls/http1.x/websocket.

Jan 8, 2023
Go package to simulate bandwidth, latency and packet loss for net.PacketConn and net.Conn interfaces

lossy Go package to simulate bandwidth, latency and packet loss for net.PacketConn and net.Conn interfaces. Its main usage is to test robustness of ap

Oct 14, 2022
Fast HTTP package for Go. Tuned for high performance. Zero memory allocations in hot paths. Up to 10x faster than net/http
Fast HTTP package for Go. Tuned for high performance. Zero memory allocations in hot paths. Up to 10x faster than net/http

fasthttp Fast HTTP implementation for Go. Currently fasthttp is successfully used by VertaMedia in a production serving up to 200K rps from more than

Jan 5, 2023
Simple GUI to convert Charles headers to golang's default http client (net/http)

Charles-to-Go Simple GUI to convert Charles headers to golang's default http client (net/http) Usage Compile code to a binary, go build -ldflags -H=wi

Dec 14, 2021
go HTTP client that makes it plain simple to configure TLS, basic auth, retries on specific errors, keep-alive connections, logging, timeouts etc.

goat Goat, is an HTTP client built on top of a standard Go http package, that is extremely easy to configure; no googling required. The idea is simila

Jun 25, 2022
Transparent TLS and HTTP proxy serve and operate on all 65535 ports, with domain regex whitelist and rest api control

goshkan Transparent TLS and HTTP proxy serve & operating on all 65535 ports, with domain regex whitelist and rest api control tls and http on same por

Nov 5, 2022
High performance async-io(proactor) networking for Golang。golangのための高性能非同期io(proactor)ネットワーキング
High performance async-io(proactor) networking for Golang。golangのための高性能非同期io(proactor)ネットワーキング

gaio Introduction 中文介绍 For a typical golang network program, you would first conn := lis.Accept() to get a connection and go func(net.Conn) to start a

Dec 29, 2022
🚀 gnet is a high-performance, lightweight, non-blocking, event-driven networking framework written in pure Go./ gnet 是一个高性能、轻量级、非阻塞的事件驱动 Go 网络框架。
🚀 gnet is a high-performance, lightweight, non-blocking, event-driven networking framework written in pure Go./ gnet 是一个高性能、轻量级、非阻塞的事件驱动 Go 网络框架。

English | ???? 中文 ?? Introduction gnet is an event-driven networking framework that is fast and lightweight. It makes direct epoll and kqueue syscalls

Jan 2, 2023
Fast event-loop networking for Go
Fast event-loop networking for Go

evio is an event loop networking framework that is fast and small. It makes direct epoll and kqueue syscalls rather than using the standard Go net pac

Dec 31, 2022
Packiffer is a lightweight cross-platform networking toolkit that let you sniff/analyze/inject/filter packets.
Packiffer is a lightweight cross-platform networking toolkit that let you sniff/analyze/inject/filter packets.

Packiffer is a lightweight cross-platform networking toolkit that let you sniff/analyze/inject/filter packets.

Dec 19, 2022
A decentralized P2P networking stack written in Go.

noise noise is an opinionated, easy-to-use P2P network stack for decentralized applications, and cryptographic protocols written in Go. noise is made

Dec 29, 2022
Netpoll is a high-performance non-blocking I/O networking framework, which focused on RPC scenarios, developed by ByteDance.
Netpoll is a high-performance non-blocking I/O networking framework, which focused on RPC scenarios, developed by ByteDance.

Netpoll is a high-performance non-blocking I/O networking framework, which focused on RPC scenarios, developed by ByteDance. RPC is usually heavy on processing logic and therefore cannot handle I/O serially. But Go's standard library net designed blocking I/O API, so that the RPC framework can only follow the One Conn One Goroutine design.

Jan 2, 2023
🧪 Run common networking tests against your site.
🧪 Run common networking tests against your site.

dstp dstp, run common networking tests against your site. Usage Usage: dstp [OPTIONS] [ARGS]

Jan 3, 2023
Hybridnet is an open source container networking solution, integrated with Kubernetes and used officially by following well-known PaaS platforms

Hybridnet What is Hybridnet? Hybridnet is an open source container networking solution, integrated with Kubernetes and used officially by following we

Jan 4, 2023
Basic Got chat program using Ably for networking

Go Terminal Chat Basic Got chat program using Ably for networking. Taken from GopherCon UK 2021: Tom Camp - Creating a basic chat app. Setup Replace t

Nov 30, 2021
NAT puncher for Wireguard mesh networking.

natpunch-go This is a NAT hole punching tool designed for creating Wireguard mesh networks. It was inspired by Tailscale and informed by this example.

Dec 12, 2022
Zero Trust Network Communication Sentinel provides peer-to-peer, multi-protocol, automatic networking, cross-CDN and other features for network communication.
Zero Trust Network Communication Sentinel provides peer-to-peer, multi-protocol, automatic networking, cross-CDN and other features for network communication.

Thank you for your interest in ZASentinel ZASentinel helps organizations improve information security by providing a better and simpler way to protect

Nov 1, 2022