An Advanced HTTP Reverse Proxy with Dynamic Sharding Strategies

Weaver - A modern HTTP Proxy with Advanced features

Build Status Go Report Card Coverage Status GitHub Release

Description

Weaver is a Layer-7 Load Balancer with Dynamic Sharding Strategies. It is a modern HTTP reverse proxy with advanced features.

Features:

  • Sharding request based on headers/path/body fields
  • Emits Metrics on requests per route per backend
  • Dynamic configuring of different routes (No restarts!)
  • Is Fast
  • Supports multiple algorithms for sharding requests (consistent hashing, modulo, s2 etc)
  • Packaged as a single self contained binary
  • Logs on failures (Observability)

Installation

Build from source

  • Clone the repo:
git clone [email protected]:gojektech/weaver.git
  • Build to create weaver binary
make build

Binaries for various architectures

Download the binary for a release from: here

Architecture

Weaver uses etcd as a control plane to match the incoming requests against a particular route config and shard the traffic to different backends based on some sharding strategy.

Weaver can be configured for different routes matching different paths with various sharding strategies through a simple route config named ACL.

The various sharding strategies supported by weaver are:

  • Consistent hashing (hashring)
  • Simple lookup based
  • Modulo
  • Prefix lookup
  • S2 based

Deploying to Kubernetes

Currently we support deploying to kubernetes officially. You can check the doc here

Examples

We have examples defined to deploy it to kubernetes and using acls. Please checkout out examples

Configuration

Defining ACL's

Details on configuring weaver can be found here

Please note

As the famous saying goes, All Load balancers are proxies, but not every proxy is a load balancer, weaver currently does not support load balancing.

Contributing

If you'd like to contribute to the project, refer to the contributing documentation

License

Copyright 2018, GO-JEK Tech (http://gojek.tech)

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
Owner
Gojek
SuperApp from Southeast Asia
Gojek
Comments
  • go sum update

    go sum update

    Build failing due to. Updated by regenerating go.sum

    go: verifying github.com/coreos/[email protected]+incompatible: checksum mismatch
            downloaded: h1:v3H7yHgF+94suF7Xg6V7Haq6Anac3X6WosuKGTTJCGM=
            go.sum:     h1:WyMdmwfw02n2i3gTkzy+HBR2dsh1axjxn808iAe9PLI=
    
  • Sometimes tests are stuck for longer time.

    Sometimes tests are stuck for longer time.

    Sometimes when running tests, they are stuck for longer time, couldn't get them to complete.

    ╭─TheInsider ~/Documents/repos/gojektech/weaver ‹ruby-2.5.0› ‹weaver-cli*›
    ╰─➤  mk test
    cp weaver.conf.yaml.sample weaver.conf.yaml
    GO111MODULE=on go test ./...
    ok  	github.com/gojektech/weaver	(cached)
    ?   	github.com/gojektech/weaver/cmd/weaver	[no test files]
    ok  	github.com/gojektech/weaver/config	0.021s
    ^C{"level":"info","msg":"bootstrapping router using etcd on /weaver/acls/","time":"2019-03-19T07:53:25+05:30"}
    {"level":"info","msg":"creating router namespace on etcd using /weaver/acls/","time":"2019-03-19T07:53:25+05:30"}
    FAIL	github.com/gojektech/weaver/etcd	14.251s
    make: *** [Makefile:40: test] Error 1
    

    Here is the message, that I got when trying to reproduce it.

  • weaver as go library

    weaver as go library

    Hello, thanks for sharing this amazing project. Any option to release it as go library so we can integrate it a a more large application? I'm speaking something as https://github.com/devopsfaith/krakend

    please let me know Thanks

  • Helm charts to deploy to kubenetes

    Helm charts to deploy to kubenetes

    Pull request to close #1. It is still draft pull request.

    Currently: ETCD is optional, can deploy or ignore. configurations are mentioned in the chart itself via env variable. Currently these are not configured in chart, but hardcoded.

    TODO: - Have to extract env variables - Have to configure statsd, newrelic (work required on weaver) - All env keys should not be mandatory (work required on weaver) - Sentry should be optional (work required on weaver) - Helm chart to deploy statsd (in progress)

  • Full Documentation

    Full Documentation

    Required Topics include:

    • [x] Usage Guidelines
    • [x] Development Guidelines
    • [x] Contribution Guidelines
    • [x] Weaver Design Overview
    • [x] ACL Basics, Formats and Management
    • [x] Routing and Sharding (Working & available strategies)
  • Correct the spelling of Kubernets into Kubernetes

    Correct the spelling of Kubernets into Kubernetes

    I noticed that many parts of the markdown files contain misspelled "Kubernets". This pull request fixes all those occurrences in the markdown files to "Kubernetes".

  • #6 Can run tests in docker

    #6 Can run tests in docker

    Fix #6 Next, remove env variables from docker-compose and use hostname directly to simplify. This required little more clean up.

    @rShetty if it looks good, please merge.

  • update Dockerfile, remove mkdir command, WORKDIR  will create if it doen't exist

    update Dockerfile, remove mkdir command, WORKDIR will create if it doen't exist

    as documented here:

    If the WORKDIR doesn’t exist, it will be created even if it’s not used in any subsequent Dockerfile instruction.

    so there is no need RUN mkdir /weaver

  • deployment error

    deployment error

    When I am trying to deploy the weaver I am getting the error :

    version for aks I am using 1.19.0 helm version : version.BuildInfo{Version:"v3.3.4", GitCommit:"a61ce5633af99708171414353ed49547cf05013d", GitTreeState:"clean", GoVersion:"go1.14.9"}

    When I run the command I get the following error , Please help me.

    ubuntu@test:~/weaver$ helm upgrade --debug --install proxy-cluster ./deployment/weaver -f ./deployment/weaver/values-env.yaml history.go:53: [debug] getting history for release proxy-cluster upgrade.go:121: [debug] preparing upgrade for proxy-cluster Error: UPGRADE FAILED: error validating "": error validating data: [ValidationError(Deployment): unknown field "name" in io.k8s.api.apps.v1.Deployment, ValidationError(Deployment.spec): missing required field "selector" in io.k8s.api.apps.v1.DeploymentSpec] helm.go:94: [debug] error validating "": error validating data: [ValidationError(Deployment): unknown field "name" in io.k8s.api.apps.v1.Deployment, ValidationError(Deployment.spec): missing required field "selector" in io.k8s.api.apps.v1.DeploymentSpec] helm.sh/helm/v3/pkg/kube.scrubValidationError /home/circleci/helm.sh/helm/pkg/kube/client.go:566 helm.sh/helm/v3/pkg/kube.(*Client).Build /home/circleci/helm.sh/helm/pkg/kube/client.go:159 helm.sh/helm/v3/pkg/action.validateManifest /home/circleci/helm.sh/helm/pkg/action/upgrade.go:451 helm.sh/helm/v3/pkg/action.(*Upgrade).prepareUpgrade /home/circleci/helm.sh/helm/pkg/action/upgrade.go:233 helm.sh/helm/v3/pkg/action.(*Upgrade).Run /home/circleci/helm.sh/helm/pkg/action/upgrade.go:122 main.newUpgradeCmd.func2 /home/circleci/helm.sh/helm/cmd/helm/upgrade.go:154 github.com/spf13/cobra.(*Command).execute /go/pkg/mod/github.com/spf13/[email protected]/command.go:842 github.com/spf13/cobra.(*Command).ExecuteC /go/pkg/mod/github.com/spf13/[email protected]/command.go:950 github.com/spf13/cobra.(*Command).Execute /go/pkg/mod/github.com/spf13/[email protected]/command.go:887 main.main /home/circleci/helm.sh/helm/cmd/helm/helm.go:93 runtime.main /usr/local/go/src/runtime/proc.go:203 runtime.goexit /usr/local/go/src/runtime/asm_amd64.s:1373 UPGRADE FAILED main.newUpgradeCmd.func2 /home/circleci/helm.sh/helm/cmd/helm/upgrade.go:156 github.com/spf13/cobra.(*Command).execute /go/pkg/mod/github.com/spf13/[email protected]/command.go:842 github.com/spf13/cobra.(*Command).ExecuteC

  • Adding ACL List, Show Cli

    Adding ACL List, Show Cli

    Weaver CLI is ready for review. Commands will register them selves with CLI. In main we need to import the commands that we want them to be with the binary.

    Let's say we want to separate out acls commands, we can create weaverctl in cmd/weaverctl and import only acl commands. Binary produced from weaverctl will have only acls commands and will not have acl commands.

    View layers currently renders output to json, this can manipulated easily using a flag and rending to outputs like yaml, table and etc. This will be a enhancement. For now will stick to json output.

  • Command Line tool to manage ACLs

    Command Line tool to manage ACLs

    Description: Build an admin tool to add ACL's to the control plane (etcd)

    This tool will be a command line utility inside Weaver (weaverctl) which helps management (addition/deletion/updation) of all the ACL's

mt-multiserver-proxy is a reverse proxy designed for linking multiple Minetest servers together

mt-multiserver-proxy mt-multiserver-proxy is a reverse proxy designed for linking multiple Minetest servers together. It is the successor to multiserv

Nov 17, 2022
Tcp-proxy - A dead simple reverse proxy server.

tcp-proxy A proxy that forwords from a host to another. Building go build -ldflags="-X 'main.Version=$(git describe --tags $(git rev-list --tags --max

Jan 2, 2022
The Dual-Stack Dynamic DNS client, the world's first dynamic DNS client built for IPv6.

dsddns DsDDNS is the Dual-Stack Dynamic DNS client. A dynamic DNS client keeps your DNS records in sync with the IP addresses associated with your hom

Sep 27, 2022
Http-logging-proxy - A HTTP Logging Proxy For Golang

http-logging-proxy HTTP Logging Proxy Description This project builds a simple r

Aug 1, 2022
Open Source HTTP Reverse Proxy Cache and Time Series Dashboard Accelerator
Open Source HTTP Reverse Proxy Cache and Time Series Dashboard Accelerator

Trickster is an HTTP reverse proxy/cache for http applications and a dashboard query accelerator for time series databases. Learn more below, and chec

Jan 2, 2023
A standalone Web Server developed with the standard http library, suport reverse proxy & flexible configuration
A standalone Web Server developed with the standard http library, suport reverse proxy & flexible configuration

paddy 简介 paddy是一款单进程的独立运行的web server,基于golang的标准库net/http实现。 paddy提供以下功能: 直接配置http响应 目录文件服务器 proxy_pass代理 http反向代理 支持请求和响应插件 部署 编译 $ go build ./main/p

Oct 18, 2022
Goproxy - HTTP/HTTPS Forward and Reverse Proxy

Go HTTP(s) Forward/Reverse Proxy This is intended to provide the proxy for the goproxy frontend. It is currently a work in progress, and is not very s

Jan 4, 2022
Websockify-go - A reverse proxy that support tcp, http, https, and the most important, noVNC, which makes it a websockify

websockify-go | mproxy a reverse proxy that support tcp, http, https, and the mo

Aug 14, 2022
A load balancer supporting multiple LB strategies written in Go
A load balancer supporting multiple LB strategies written in Go

farely A load balancer supporting multiple LB strategies written in Go. Goal The goal of this project is purley educational, I started it as a brainst

Dec 21, 2022
Go HTTP tunnel is a reverse tunnel based on HTTP/2.

Go HTTP tunnel is a reverse tunnel based on HTTP/2. It enables you to share your localhost when you don't have a public IP.

Dec 28, 2022
A simple tool to convert socket5 proxy protocol to http proxy protocol

Socket5 to HTTP 这是一个超简单的 Socket5 代理转换成 HTTP 代理的小工具。 如何安装? Golang 用户 # Required Go 1.17+ go install github.com/mritd/s2h@master Docker 用户 docker pull m

Jan 2, 2023
Reverse cwmp proxy
Reverse cwmp proxy

cwmp-proxy Integration of the proxy will provide you the ability to place CPEs and ACS servers in different networks. What about if the devices are pl

Nov 15, 2022
A fast reverse proxy to help you expose a local server behind a NAT or firewall to the internet.
A fast reverse proxy to help you expose a local server behind a NAT or firewall to the internet.

frp README | 中文文档 What is frp? frp is a fast reverse proxy to help you expose a local server behind a NAT or firewall to the Internet. As of now, it s

Jan 5, 2023
Dead simple reverse proxy for all your containerized needss
Dead simple reverse proxy for all your containerized needss

Whats this ? Pawxi is yet another reverse proxy designed with simplicity in mind. Born out of a certain users frustration at the complexity of setting

Oct 17, 2022
Simple edge server / reverse proxy

reproxy Reproxy is simple edge HTTP(s) sever / reverse proxy supporting various providers (docker, static, file). One or more providers supply informa

Dec 29, 2022
gobetween - modern & minimalistic load balancer and reverse-proxy for the ☁️ Cloud era.
gobetween - modern & minimalistic load balancer and reverse-proxy for the ☁️ Cloud era.

gobetween - modern & minimalistic load balancer and reverse-proxy for the ☁️ Cloud era. Current status: Maintenance mode, accepting PRs. Currently in

Dec 25, 2022
Reverse Proxy for geth node
Reverse Proxy for geth node

geth-proxy Reverse Proxy for geth node gcr.io/moonrhythm-containers/geth-proxy Features Health check base on last synced block timestamp Merge websock

Jul 26, 2022
4chain is a simple、fast reverse proxy to help you expose a local server behind a NAT or firewall to the Internet.

4Chain What is 4chain? 4chain is a simple、fast reverse proxy to help you expose a local server behind a NAT or firewall to the Internet. Using the ssh

Nov 1, 2022