Fast and idiomatic client-driven REST APIs.

Vulcain: Use HTTP/2 Server Push to create fast and idiomatic client-driven REST APIs

Vulcain is a brand new protocol using HTTP/2 Server Push to create fast and idiomatic client-driven REST APIs.

An open source gateway server which you can put on top of any existing web API to instantly turn it into a Vulcain-compatible one is also provided!

It supports hypermedia APIs but also any "legacy" API by documenting its relations using OpenAPI.

Plant Tree PkgGoDev Build Status codecov Go Report Card

Vulcain Schema

Grab What You Need... Burn The REST!

The protocol has been published as an Internet Draft that is maintained in this repository.

A reference, production-grade, implementation gateway server is also available in this repository. It's free software (AGPL) written in Go. A Docker image is provided.

Introduction

Over the years, several formats have been created to fix performance bottlenecks impacting web APIs: over fetching, under fetching, the n+1 problem...

Current solutions for these problems (GraphQL, JSON:API's embedded resources and sparse fieldsets, ...) are smart network hacks for HTTP/1. But these hacks come with (too) many drawbacks when it comes to HTTP cache, logs and even security.

Fortunately, thanks to the new features introduced in HTTP/2, it's now possible to create true REST APIs fixing these problems with ease and class! Here comes Vulcain!

See also the comparison between Vulcain and GraphQL and other API formats.

Pushing Relations

Preload Schema

Considering the following resources:

/books

{
    "member": [
        "/books/1",
        "/books/2"
    ]
}

/books/1

{
    "title": "1984",
    "author": "/authors/1"
}

/books/2

{
    "title": "Homage to Catalonia",
    "author": "/authors/1"
}

/authors/1

{
    "givenName": "George",
    "familyName": "Orwell"
}

The Preload HTTP header introduced by Vulcain can be used to ask the server to immediately push resources related to the requested one using HTTP/2 Server Push:

GET /books/ HTTP/2
Preload: "/member/*/author"

In addition to /books, a Vulcain server will use HTTP/2 Server Push to push the /books/1, /books/2 and /authors/1 resources!

Example in JavaScript:

const bookResp = await fetch("/books/1", { headers: { Preload: `"/author"` } });
const bookJSON = await bookResp.json();

// Returns immediately, the resource has been pushed and is already in the push cache
const authorResp = await fetch(bookJSON.author);
// ...

Full example, including collections, see also use GraphQL as query language for Vulcain.

Thanks to HTTP/2 multiplexing, pushed responses will be sent in parallel.

When the client will follow the links and issue a new HTTP request (for instance using fetch()), the corresponding response will already be in cache, and will be used instantly!

For non-hypermedia APIs (when the identifier of the related resource is a simple string or int), use an OpenAPI specification to configure links between resources. Tip: the easiest way to create a hypermedia API is to use the API Platform framework (by the same author as Vulcain).

More than 90% of users have devices supporting HTTP/2. However, for the remaining 10%, and for cases where using HTTP/2 Server Push isn't allowed such as when resources are served by different authorities, Vulcain allows to gracefully fallback to preload links, which can be used together with the 103 status code.

Query Parameter

Alternatively to HTTP headers, the preload query parameter can be used:

Preload Query Schema

Filtering Resources

Fields Schema

The Fields HTTP header allows the client to ask the server to return only the specified fields of the requested resource, and of the preloaded related resources.

Multiple Fields HTTP headers can be passed. All fields matching at least one of these headers will be returned. Other fields of the resource will be omitted.

Considering the following resources:

/books/1

{
    "title": "1984",
    "genre": "novel",
    "author": "/authors/1"
}

/authors/1

{
    "givenName": "George",
    "familyName": "Orwell"
}

And the following HTTP request:

GET /books/1 HTTP/2
Preload: "/author"
Fields: "/author/familyName", "/genre"

A Vulcain server will return a response containing the following JSON document:

{
    "genre": "novel",
    "author": "/authors/1"
}

It will also push the following filtered /authors/1 resource:

{
    "familyName": "Orwell"
}

Query Parameter

Alternatively to HTTP headers, the fields query parameter can be used to filter resources:

Preload Query Schema

See Also

License and Copyright

tl;dr:

  • proprietary software can implement the Vulcain specification
  • proprietary software can be used behind the Vulcain Gateway Server without having to share their sources
  • modifications made to the Vulcain Gateway Server must be shared
  • alternatively, a commercial license is available for the Vulcain Gateway Server

The specification is available under the IETF copyright policy. The Vulcain specification can be implemented by any software, including proprietary software.

The Vulcain Gateway Server is licensed under AGPL-3.0. This license implies that if you modify the Vulcain Gateway Server, you must share those modifications. However, the AGPL-3.0 license applies only to the gateway server itself, not to software used behind the gateway.

For companies not wanting, or not able to use AGPL-3.0 licensed software, commercial licenses are also available. Contact us for more information.

Treeware

This package is Treeware. If you use it in production, then we ask that you buy the world a tree to thank us for our work. By contributing to the Treeware forest you’ll be creating employment for local families and restoring wildlife habitats.

Credits

Created by Kévin Dunglas. Sponsored by Les-Tilleuls.coop.

Some ideas and code used in Vulcain's reference implementation have been taken from Hades by Gabe Sullice, an HTTP/2 reverse proxy for JSON:API backend.

See also the prior arts.

Owner
Kévin Dunglas
Founder of @coopTilleuls / Creator of @api-platform, Mercure.rocks and Vulcain.rocks / @symfony Core Team
Kévin Dunglas
Comments
  • [Question] Relations preloaded within first 'main' request, making the gateway server requests very slow

    [Question] Relations preloaded within first 'main' request, making the gateway server requests very slow

    Hi,

    First of all, very interesting project!

    I'm implementing the Vulcain Gateway server in my application and it's performing slow. To be sure it's not the fault of my application I downloaded and ran the demo and this is also slow at preloading the relations. It looks like Vulcain is preloading every relation in the main request and slowing down the request when it's calling the 'conferences' endpoint:
    Schermafbeelding 2020-03-10 om 15 27 05 This is in contrast to the slide in the Symfonycon slides, where the preloading 'conferences' request takes about the same time (bit more than 4sec) it needs to request the 'conferences' endpoint without preloading: image

    I'm not sure if this is how Vulcain is supposed to work, but if it is I don't really see how Vulcain can be faster than GraphQL for example?

  • [Question] Can/should i use Nginx and/or Traefik with Vulcain? Where should be Varnish?

    [Question] Can/should i use Nginx and/or Traefik with Vulcain? Where should be Varnish?

    Hello, everybody! Cool idea, cool project!

    But i'm not understand how to modify my current scheme to use Vulcain.

    Now:

    (https, http/2)-> Traefik -(http)-> Nginx -(static or fastcgi)-> php-fpm (ApiPlatform)

    • Traefik is my ingress controller with auto routing to some microservices via docker labels. +http/2 endpoint.
    • Nginx handles static files and routes dynamic to php-fpm.
    1. How it will looks with Vulcain?
    2. Where should be Varnish?
  • Passing query params to sub-resources

    Passing query params to sub-resources

    There is a mention about using GraphQL as a query language for Vulcain here : https://github.com/dunglas/vulcain/blob/master/docs/graphql.md#using-graphql-as-query-language-for-vulcain

    I've been playing on a toyproject about Vulcain and I'm wondering if all GraphQL queries can be converted to a HTTP Request using Vulcain, and especially, deep arguments like the following :

    Given this query

    {
      persons(orderBy: name_ASC) {
        id
        name,
        starships(type: destroyer) {
          name,
          weapons(category: nuclear) {
            name
          }
        }
      }
    }
    

    I believe it may be translated to the following requests :

    GET /persons?orderBy=name
    Preload: /*/starships/*/weapons
    
    [
      {
        "id": "/persons/1",
        "name": "Person 1",
        "starships": "/starships?person=/persons/1&type=destroyer"
      } 
    ]
    
    GET /starships?person=/persons/1&type=destroyer
    
    [
      {
        "id": "/starships/1",
        "name": "Starship 1",
        "weapons": "/weapons?starship=/startships/1&category=nuclear"
      }
    ]
    
    GET /weapons?starship=/startships/1&category=nuclear
    
    [
      {
        "id": "/weapons/1",
        "name": "Weapon 1",
      }
    ]
    

    Now It's easy to change weapons(category: nuclear) to weapons(category: explosives) in GraphQL, but I'm not sure how to do it with Vulcain. Do you have any insight about this problem?

  • ci: bump go to 1.16, resolve golang-ci lint flakyness (#1)

    ci: bump go to 1.16, resolve golang-ci lint flakyness (#1)

    • Bumped Go to v1.16, updated modules with go mod tidy

    • Update ci.yml: golang-ci Actions job acting flaky when staging my PR. Linter is now pinned to the same go version the project is using at 1.16. Also configured it to only run against newly committed code when triggered by a pull_request. A merge into main from a PR will still trigger a full run of the golang-ci linter suite against the codebase.

  • spec: allow to select rel and media types

    spec: allow to select rel and media types

    • Allow to select specific media types as requested in #46 but also specific relation types and languages
    • Allow to select all links by passing an empty string (best used with the new rel parameter
    • Switch to the format defined in the structured headers I-D
    • Add an "Implementation Status" section (RFC6982
    • Add an "Acknowledgment" section

    @andrerom could you review this change? It should address the eZ Publish's use case (closes #46). @evert as discussed during API Days Interface, this change to the spec should allow to cover the use cases also covered by your Prefer-Push I-D. Would you mind to take a look?

    Thanks!

  • Why is it AGPL licensed?

    Why is it AGPL licensed?

    While AGPL is a valid license, it might be blocking the usage of Vulcain in enterprise environments (legal considerations). Is there any reason to not publish it under an even more less restrictive license as MIT?

  • Link to IETF

    Link to IETF

    Hi,

    I noticed that the link to IETF page (https://datatracker.ietf.org/doc/draft-dunglas-vulcain/) from README page is not working. I tried to find correct link on IETF website but could not find it.

    Is it because the request for this new standard is not registered yet by IETF?

    Laurent

  • Is it ok to use the name

    Is it ok to use the name "Vulcain" in an alternative implementation of the protocol?

    I am working on a (WIP) implementation of parts of the Vulcain protocol to be used in a server less (OpenWhisk) environment here: https://github.com/trieloff/helix-vulcain-filters

    Is it ok to use the term "Vulcain" in the project name?

  • Curren state

    Curren state

    Vulcain's approach is quite intriguing. However major browsers per my understanding have deprecated the h2 server push feature.

    Does this mean that vulcain in current state is unusable?

    I understand that it might make use of Early Hints feature in the future.

  •  feat!: make the library usable without the builtin reverse proxy

    feat!: make the library usable without the builtin reverse proxy

    • Make the library usable with any Go HTTP handler or reverse proxy, including Caddy server
    • Improve the documentation
    • Add an example
    • Switch from Logrus to Uber Zap
  • Will 'object' remain a permanent key for targeting nested content ?

    Will 'object' remain a permanent key for targeting nested content ?

    Hi Dunglas,

    I see examples and I have to say that the object attribute is a little weird for me..

    This key name is coming for your conception or from another architecture?

    IMHO, I'll prefer to use a key name named like content or value or more specific to Vulca(i)n outbreak or rash.

    Thanks for reading !

  • Response's body is percent encoded when parametrized queries

    Response's body is percent encoded when parametrized queries

    Hello,

    When I fetch a resource with a "fields" query parameter, the received fields are percent encoded. But when I vanilla fetch or header-based-fields fetch, the resource's fields are "normal".

    Front scenario A What I code:

    const eventResp = await fetch("https://localhost:8000/events/2?fields=\"\/subtitle\",\"name\"")
    console.log(await eventResp.json())
    

    What it outputs:

    Object { subtitle: "Internationaux%20de%20France%20de%20tennis%202019", name: "Roland%20Garros%202019" }

    ScenarioB What I code:

    const eventResp = await fetch("https://localhost:8000/events/2")
    console.log(await eventResp.json())
    

    What it outputs:

    Object { "@context": "/contexts/Event", "@id": "/events/2", "@type": "https://schema.org/Event", id: 2, name: "Roland Garros 2019", subtitle: "Internationaux de France de tennis 2019", heading: "120ème édition des Internationaux de France de tennis sont le plus grand tournoi de la saison de tennis sur terre battue.", startDate: "2019-05-26T00:00:00+00:00", endDate: "2019-06-09T00:00:00+00:00", visibility: "public", … }

    Back PHP 6.1 / APIP 3.0.3, served by Caddy 2.6

    localhost:8000
    
    log
    
    route {
        root * /app/public
        vulcain
        push
        php_fastcgi backend:9000
        encode zstd gzip
        file_server
    }
    

    Another proof/test with Postman (see screenshot here)

    1. Is it intended?
    2. What can I do to avoid this behavior?
    3. Do you need more information from my context?

    Thank you!

  • feat: send 103 Early Hints responses

    feat: send 103 Early Hints responses

    This makes Vulcain sends 103 Early Hints responses when server-push isn't supported (with preload links). The patch might be very naive but here we are, first contrib in Go. Please tell me if there is something else to be done.

  • Preload only working on localhost

    Preload only working on localhost

    When making API calls from http://localhost:19006 push seems to be working fine and it loads 400 requests in a few seconds however when I try via 192.168.1.181:19006 or on my iOS app making fetch requests it seems to be failing and loading them normally which takes over 30s. success failed

    I'm using the API Platform Docker for the backend. Any help would be appreciated

  • Unable to decode with compressed response from origin

    Unable to decode with compressed response from origin

    I was working in a Api Platform app with Caddy behind a IIS server. The IIS server manages HTTP/2 and stream the headers in lowercase to the Caddy server running HTTP/1.1. According to the RFC, the HTTP/2 headers should be case-senstive and with all headers in lowercase, and the HTTP/1.1 should be case-insensitive. The code that extracts the headers only verifies the "Field" header, in a case-sensitive manner, causing that the Fields header can not be readed correctly. I would try to test a patch, but I don't know Go, so I can't guarranty nothing.

  • Fields issue on preload entity

    Fields issue on preload entity

    Hi! I'm playing with Vulcain but i'm facing to an issue... I've 2 entities in an Api Platform project (greeting and author, author have many greetings, for relationship)

    If i fetch greetings endpoint:

    window.fetch('https://localhost/greetings', {
            headers: {
                preload: '"/hydra:member/*/author"',
                fields: '"/hydra:member/*/@id", "/hydra:member/*/name", "/hydra:member/*/author/@id"'
            }
        }).then(() => window.fetch('https://localhost/authors/1'))
    

    Of course authors/1 exists and it linked to all my greetings for more simplicity.

    So... In this case, with fields contains "/hydra:member/*/authors/@id" (it the same with firstname or anyother property) push doesn't work image

    But if instead "/hydra:member/*/authors/@id", i've "/hydra:member/*/authors" (without /@id) push works (but i get full object after, of course) image

    What am i missing....? Documentation seems clear about, i think this should work...

  • Link Varnish X-Key broken

    Link Varnish X-Key broken

    In the Cache-Documentaion there is a broken link to X-Key. https://github.com/dunglas/vulcain/blob/main/docs/cache.md

    Broken in Master-Branch https://github.com/varnish/varnish-modules/blob/master/docs/vmod_xkey.rst

    Branch 6.5 is available https://github.com/varnish/varnish-modules/blob/6.5/docs/vmod_xkey.rst

A Realtime API Gateway used with NATS to build REST, real time, and RPC APIs, where all your clients are synchronized seamlessly.
A Realtime API Gateway used with NATS to build REST, real time, and RPC APIs, where all your clients are synchronized seamlessly.

Realtime API Gateway Synchronize Your Clients Visit Resgate.io for guides, live demos, and resources. Resgate is a Go project implementing a realtime

Dec 31, 2022
Minimal and idiomatic WebSocket library for Go

websocket websocket is a minimal and idiomatic WebSocket library for Go. Install go get nhooyr.io/websocket Highlights Minimal and idiomatic API First

Dec 30, 2022
A simple Hasura client for query and metadata APIs

Hasura Client A Hasura client for schema APIs https://hasura.io/docs/latest/graphql/core/api-reference/index.html Usage Create client client := hasura

Oct 31, 2021
The Go client to access APIs on the Micro Platform

This is the Go client to access APIs on the Micro Platform

Nov 1, 2022
Fake server, Consumer Driven Contracts and help with testing performance from one configuration file with zero system dependencies and no coding whatsoever
Fake server, Consumer Driven Contracts and help with testing performance from one configuration file with zero system dependencies and no coding whatsoever

mockingjay server Mockingjay lets you define the contract between a consumer and producer and with just a configuration file you get: A fast to launch

Jan 6, 2023
webrpc is a schema-driven approach to writing backend services for modern Web apps and networks
webrpc is a schema-driven approach to writing backend services for modern Web apps and networks

webrpc is a schema-driven approach to writing backend servers for the Web. Write your server's api interface in a schema format of RIDL or JSON, and t

Jan 7, 2023
llb - It's a very simple but quick backend for proxy servers. Can be useful for fast redirection to predefined domain with zero memory allocation and fast response.

llb What the f--k it is? It's a very simple but quick backend for proxy servers. You can setup redirect to your main domain or just show HTTP/1.1 404

Sep 27, 2022
🚀 gnet is a high-performance, lightweight, non-blocking, event-driven networking framework written in pure Go./ gnet 是一个高性能、轻量级、非阻塞的事件驱动 Go 网络框架。
🚀 gnet is a high-performance, lightweight, non-blocking, event-driven networking framework written in pure Go./ gnet 是一个高性能、轻量级、非阻塞的事件驱动 Go 网络框架。

English | ???? 中文 ?? Introduction gnet is an event-driven networking framework that is fast and lightweight. It makes direct epoll and kqueue syscalls

Jan 2, 2023
Go driven rpc code generation tool for right now.
Go driven rpc code generation tool for right now.

Go driven rpc code generation tool for right now. 100% Go Describe services with Go interfaces (with structs, methods, comments, etc.) Generate server

Dec 20, 2022
High-performance, non-blocking, event-driven, easy-to-use networking framework written in Go, support tls/http1.x/websocket.

High-performance, non-blocking, event-driven, easy-to-use networking framework written in Go, support tls/http1.x/websocket.

Jan 8, 2023
A sample web API in GO (with GIn) under a domain driven architecture.

Golang Sample API Domain Driven Design Pattern 1. About This sample project presents a custom made domain driven API architecture in Golang using the

Jan 10, 2022
Event driven modular status-bar for dwm; written in Go & uses Unix sockets for signaling.

dwmstat A simple event-driven modular status-bar for dwm. It is written in Go & uses Unix sockets for signaling. The status bar is conceptualized as a

Dec 25, 2021
A yaml data-driven testing format together with golang testing library

Specified Yaml data-driven testing Specified is a yaml data format for data-driven testing. This enforces separation between feature being tested the

Jan 9, 2022
An event driven remote access trojan for experimental purposes.

erat An event driven remote access trojan for experimental purposes. This example is very simple and leverages ssh failed login events to trigger erat

Jan 16, 2022
Package event-driven makes it easy for you to drive events between services
Package event-driven makes it easy for you to drive events between services

Event-Driven Event-driven architecture is a software architecture and model for application design. With an event-driven system, the capture, communic

Apr 20, 2022
kcp is a prototype of a Kubernetes API server that is not a Kubernetes cluster - a place to create, update, and maintain Kube-like APis with controllers above or without clusters.
kcp is a prototype of a Kubernetes API server that is not a Kubernetes cluster - a place to create, update, and maintain Kube-like APis with controllers above or without clusters.

kcp is a minimal Kubernetes API server How minimal exactly? kcp doesn't know about Pods or Nodes, let alone Deployments, Services, LoadBalancers, etc.

Jan 6, 2023
iceportal-api is a Golang client implementation to interact with the REST API of iceportal.de when connected to the WiFi-Network offered in German ICE Trains.
iceportal-api is a Golang client implementation to interact with the REST API of iceportal.de when connected to the WiFi-Network offered in German ICE Trains.

iceportal-api is a Golang client implementation to interact with the REST API of iceportal.de when connected to the WiFi-Network offered in German ICE Trains.

Aug 20, 2022
Simple REST client library in Go

Simple REST client library in Go Context The goal was to make a minimal library that could be easily reused and expanded in other projects. It doesn't

Sep 21, 2022
Sample program of GCP pub/sub client with REST API

GCP pub/sub sample using REST API in Go GCP pub/sub publisher and subscriber sample programs. These use REST API and don't use pub/sub client library

Oct 12, 2021