Carbyne Stack serverless compute service for secure multiparty computation

Carbyne Stack Ephemeral Service

codecov Codacy Badge Known Vulnerabilities pre-commit Contributor Covenant

Ephemeral is a serverless compute service for secure multiparty computation based on Knative, Istio and Kubernetes.

DISCLAIMER: Carbyne Stack Ephemeral is alpha software. The software is not ready for production use. It has neither been developed nor tested for a specific use case. The underlying Secure Multiparty Computation protocols are currently used in a way that is not secure.

Ephemeral is composed of these components:

  • Ephemeral - The Knative user container that enables execution of MPC functions within a Carbyne Stack virtual cloud. Supports MP-SPDZ as the underlying MPC engine (see also Carbyne Stack Ephemeral SPDZ Base Image).

  • Discovery Service - Coordinates the execution of functions across Carbyne Stack virtual cloud providers.

  • Network Controller - Enables communication between MPC engines by configuring Istio to route incoming traffic from remote MPC engines to the Ephemeral Knative pods.

  • Client - A Java client that can be used to invoke Ephemeral functions.

  • Helm Chart - A Helm chart to deploy Ephemeral on a Kubernetes cluster.

Namesake

ephemeral (plural ephemerals): Something which lasts for a short period of time.

As of Wikipedia:

Ephemerality (from Greek ephemeros, literally "lasting only one day") is the concept of things being transitory, existing only briefly. Typically, the term ephemeral is used to describe objects found in nature, although it can describe a wide range of things, including human artifacts intentionally made to last for only a temporary period, in order to increase their perceived aesthetic value.

Authoring Ephemeral Functions

Ephemeral uses the MP-SPDZ library as the underlying MPC engine. That means you write Ephemeral functions using the Python dialect used by MP-SPDZ.

I/O

I/O is implemented in Ephemeral using socket functionality provided by MP-SPDZ. A function for adding two secret shared values from and writing the result back to Amphora looks like this:

# Open socket for I/O
listen(10000)
client_socket_id = regint()
acceptclientconnection(client_socket_id, 10000)

# Read all input data at once
v = sint.read_from_socket(client_socket_id, 2)
a = v[0]
b = v[1]

# Compute result
sum = a + b

# Pack result into array and write to socket
resp = Array(1, sint)
resp[0] = sum
sint.write_to_socket(client_socket_id, resp)

Known issues

Old Knative revisions must be deleted manually

Patching of Kubernetes Pods managed by Knative causes dangling old revisions when a new revision is created. While the new revision is activated and traffic is forwarded correctly, the old pods belonging to the previous revision are lying around and must be manually deleted. The following commands must be used:

kubectl get revisions
# Pick up the older revision that must deleted, e.g. <REVISION_NAME>
# And delete it manually.
kubectl delete revision <REVISION_NAME>

License

Carbyne Stack Ephemeral is open-sourced under the Apache License 2.0. See the LICENSE file for details.

3rd Party Licenses

For information on how license obligations for 3rd party OSS dependencies are fulfilled see the README file of the Carbyne Stack repository.

Contributing

Please see the Carbyne Stack Contributor's Guide.

Owner
Carbyne Stack
Cloud Native Secure Multiparty Computation
Carbyne Stack
Comments
  • update to new mp spdz version

    update to new mp spdz version

    requires https://github.com/carbynestack/ephemeral/pull/14

    Updates to MP-SPDZ v0.2.8

    corresponding cli part https://github.com/carbynestack/cli/pull/19

  • Draft: Feature/test/tuple streaming

    Draft: Feature/test/tuple streaming

    This is a test for fetching tuples from Castor (see #6) It kind of requires the Latest Version of Castor (the changes in Reservation handling carbynestack/castor#29) -> Otherwise it would at times have errors with relaying the reservation to the castor slave -> With the current Castor Master, the error was gone ^^

    This PR is not ready for production in any way but is rather intended as a first step for further work ^^
    --> Deliberatly not signed-off

    • It contains quite a lot of TODO comments
    • It also hardcodes some values

    How this works:

    • Before starting the MP-SPDZ execution, it will create Named Pipes/FIFOs
      (using the unix package, so on windows this would not work, though I don't think that's a show-stopper)
    • Will start 1 Gothread per TupleType and MP-SPDZ VM Thread (currently only tested with 1 thread!)
      • Each thread will write to 1 pipe
      • Starting with the MP-SPDZ Header
      • Then fetching X tuples at a time from castor and writing them
      • Until the context is closed

    Does NOT really do any proper error handling or logging (latter is just printlns instead of the logger)

    Open questions:

    1. Should the Prime/Mac value files be written to by ephemeral, or should they remain part of the base-image?
    2. Where do we get information about how many threads are needed for the computation?
    3. Which Tuple types are currently supported by CarbyneStack? Castor knows about GFP and GF2N tuples.

    Example with Millionaires Problem: Logs from apollo for this single computation can be found here (not inlined since they are quite big ^^) https://gist.github.com/kindlich/82ee7b34207a0efb025fb51b4d29e4e9

    Before

    java -jar cache/cli/cs.jar --config-file cache/cli/config.json castor get-telemetry 1
    Initializing ExecutorService
    bit_gfp
            available:      99640
            consumption/s:  0
    
    inputmask_gfp
            available:      299999
            consumption/s:  0
    
    inversetuple_gfp
            available:      99540
            consumption/s:  0
    
    multiplicationtriple_gfp
            available:      99360
            consumption/s:  0
    
    interval: 60000
    

    Run like normal (in my case with a helper-script)

    ./scripts/execute-ephemeral.sh ../carbyne/mpc-evaluation/mp-spdz/Programs/Source/cb-billionaires.mpc -i $SECRET_A -i $SECRET_A 
    d76d967c-da5d-4090-bd03-cb5fa560cf97
    

    After

    Initializing ExecutorService
    bit_gfp
            available:      99280
            consumption/s:  0
    
    inputmask_gfp
            available:      299999
            consumption/s:  0
    
    inversetuple_gfp
            available:      99540
            consumption/s:  0
    
    multiplicationtriple_gfp
            available:      99210
            consumption/s:  0
    
    interval: 60000
    
  • make it possible to select secrets by tags from amphora

    make it possible to select secrets by tags from amphora

    this change is using the existing /secret-shares-endpoint in amphora

    requires:

    • https://github.com/carbynestack/ephemeral/pull/14
    • https://github.com/carbynestack/ephemeral/pull/15

    corresponding cli update https://github.com/carbynestack/cli/pull/18

  • Ensure that client supports any number of virtual cloud providers

    Ensure that client supports any number of virtual cloud providers

    The ephemeral client uses Java Parallel Streams (JPS) to interact with the virtual cloud providers (VCPs). By default JPS allocates n threads on a n core machine. As the HTTP calls are blocking, this results in a timeout on a system with less cores than the number of VCPs as the distributed execution in the backend will only kick-off after all VCPs have received an invocation request.

  • enhance ephemeral to work with more than 2 players

    enhance ephemeral to work with more than 2 players

    To use more than 2 players, it's required to create new crypto-material with given number of players e.g. for 3 players. This also means that the base-image of ephemeral needs to be rebuild including the new crypto-material!

    The helm chart is using 2 players as a default, so the current deployment stays with 2 players.

    requires https://github.com/carbynestack/ephemeral/pull/19

  • Prefetching of Tuples from Castor

    Prefetching of Tuples from Castor

    To perform secure computations, Ephemeral has to fetch tuples from Castor, which are then consumed throughout the execution of the MPC program. However, Ephemeral is currently deployed with pre-generated tuples as streaming tuples from Castor to Ephemeral is not yet available. To increase security throughout the computation:

    • Ephemeral must use the Castor API to fetch tuples and stream them into the MPC program execution
    • Pre-generated tuples should be removed from the Ephemeral Docker Image
  • use go modules dependency management

    use go modules dependency management

  • Secure TLS-Connection for new MP-SPDZ Version

    Secure TLS-Connection for new MP-SPDZ Version

    In #15, the MP-SPDZ Version is changed to v0.2.8 which introduces TLS-secured communication between MP-SPDZ Clients (=ephemeral) and the MP-SPDZ Process.

    The way the PR adds this is really rudimentary and insecure (e.g. the option InsecureSkipVerify is set to bypass certificate validation).

    	tlsConfig := &tls.Config{
    		Certificates:       []tls.Certificate{cert},
    		InsecureSkipVerify: true,
    	}
    

    We will need to think about some points regarding the TLS connection:

    • Who will create certificates (and re-created them when expired)?
      Types of connections
      • Ephemeral <-> MP-SPDZ Process (for transmitting secret shares/results)
      • Cross-Cloud MP-SPDZ instances (for computation)
    • Will we continue using self-signed certificates or have a way to provide certifactes signed by CAs?
    • Can we provide certificate using e.g. K8s Secrets instead of baking them into the base-image (do some parts of our infrastructure like Istio/Knative already provide some functionality for this here?)
  • Upgrade helm-oci-chart-releaser to v0.3.0

    Upgrade helm-oci-chart-releaser to v0.3.0

    Required since Helm 3.7.0 broke former version 0.2.0. This was resolved with appany/helm-oci-chart-releaser#4

    Signed-off-by: Sebastian Becker [email protected]

  • upgrade apiextensions.k8s.io version to v1 from beta1

    upgrade apiextensions.k8s.io version to v1 from beta1

    The apiextensions.k8s.io/v1beta1 API version of CustomResourceDefinition is no longer served as of v1.22.

    Migrate manifests and API clients to use the apiextensions.k8s.io/v1 API version, available since v1.16.
    

    Signed-off-by: Veselin Vlasakiev [email protected]

  • Compare the MP-SPDZ-Prime when connecting as MP-SPDZ Client

    Compare the MP-SPDZ-Prime when connecting as MP-SPDZ Client

    When Player 0 connects as client to the MP-SPDZ process, it will receive the Prime number from MP-SPDZ.

    If Ephemeral knew the expected prime we could check for configuration mismatches (e.g. if the wrong base-image was used)

    See comments from this PR: https://github.com/carbynestack/ephemeral/pull/15#discussion_r844660968

  • Have another look at Discovery_test.go Test for players sending multiple messages in a row

    Have another look at Discovery_test.go Test for players sending multiple messages in a row

    The issue is about the test "[When] a single player sends 2 messages in a row, [It] doesn't create the second network" -> In there the test for originally 2 players sent 2 "Player 1 Ready" messages and expected a "PlayerReady" response.

    In #14 this is/was extended to send N times a "Player 1 Ready" message when N players are used. In any case, we were not sure about whether this test is flawed or not, hence this issue.

    Let's have another look at that test and fix it if necessary. -> Should N times Player1 Ready lead to "[All] PlayersReady"? -> We only want to test that sending multiple messages does not create a second network, is there another way than waiting for a playersReady event here?

    Those questions also came up in the PR, see links below: https://github.com/carbynestack/ephemeral/pull/14#discussion_r840373453 https://github.com/carbynestack/ephemeral/pull/14#discussion_r839315519

  • Support mulitple output objects per execution

    Support mulitple output objects per execution

    It looks like, Ephemeral currently supports only a single output object for MPC programs.

    https://github.com/carbynestack/ephemeral/blob/70795c1e688fdffef3761431c341f1302f0105b0/pkg/ephemeral/io/feeder.go#L140-L160

    That seems limiting in many scenarios.

Related tags
Node of the decentralized oracle network, bridging on and off-chain computation

Chainlink is middleware to simplify communication with blockchains. Here you'll find the Chainlink Golang node, currently in alpha. This initial imple

Dec 31, 2022
Compute and print message digest hash values from stdin.

Compute and print message digest hash values from stdin.

Jan 31, 2022
Service that calls uzma24/project1 service, takes input from .txt file and prints JSON output returned from the service.

Service that calls uzma24/project1 service, takes input from .txt file and prints JSON output returned from the service. Program can take large input files.

Feb 6, 2022
Nov 9, 2022
Serverless Lightning Address server

Addressless Put a Lightning Address "server" on a serverless host like Vercel. The code is heavily borrowed from satdress, minus the federated part. T

Aug 29, 2022
Diameter stack and Base Protocol (RFC 6733) for the Go programming language

Diameter Base Protocol Package go-diameter is an implementation of the Diameter Base Protocol RFC 6733 and a stack for the Go programming language. St

Dec 28, 2022
A RTP stack for Go

RTP/RTCP stack for Go This Go package implements a RTP/RTCP stack for Go. The package is a sub-package of the standard Go net package and uses standar

Dec 21, 2022
Go SIP Stack

GoSIPs Go SIP Stack (http://www.GoSIPs.org) The objective of GoSIPs is to develop a Golang stack interface and implementation to the Session Initiatio

Aug 31, 2022
A decentralized P2P networking stack written in Go.

noise noise is an opinionated, easy-to-use P2P network stack for decentralized applications, and cryptographic protocols written in Go. noise is made

Dec 29, 2022
The Dual-Stack Dynamic DNS client, the world's first dynamic DNS client built for IPv6.

dsddns DsDDNS is the Dual-Stack Dynamic DNS client. A dynamic DNS client keeps your DNS records in sync with the IP addresses associated with your hom

Sep 27, 2022
A modular is an opinionated, easy-to-use P2P network stack for decentralized applications written in Go.

xlibp2p xlibp2p is an opinionated, easy-to-use P2P network stack for decentralized applications written in Go. xlibp2p is made to be minimal, robust,

Nov 9, 2022
Technical specifications for the IPFS protocol stack
Technical specifications for the IPFS protocol stack

IPFS Specifications This repository contains the specs for the IPFS Protocol and associated subsystems. Understanding the meaning of the spec badges a

Jan 7, 2023
Xlibp2p: an opinionated, easy-to-use P2P network stack for decentralized applications written in Go

xlibp2p xlibp2p is an opinionated, easy-to-use P2P network stack for decentraliz

Nov 9, 2022
Fix Burp Suite's horrible TLS stack & spoof any browser fingerprint
Fix Burp Suite's horrible TLS stack & spoof any browser fingerprint

Awesome TLS This extension hijacks Burp's HTTP and TLS stack to make it more powerful and less prone to fingerprinting by all kinds of WAFs. It does t

Jan 2, 2023
A service registry and service discovery implemention for kitex based on etcd

kitex etcd Introduction kitexetcd is an implemention of service registry and service discovery for kitex based on etcd. Installation go get -u github.

Feb 18, 2022
A service to proxy requests to a given backend service.

Proxy Service A service to proxy requests to a given backend service. Go 1.17+ Clone git clone [email protected]:janu-cambrelen/proxy-service.git Run (L

Jan 5, 2022
A Crypto-Secure, Production-Grade Reliable-UDP Library for golang with FEC
 A Crypto-Secure, Production-Grade Reliable-UDP Library for golang with FEC

Introduction kcp-go is a Production-Grade Reliable-UDP library for golang. This library intents to provide a smooth, resilient, ordered, error-checked

Dec 28, 2022
A Stable & Secure Tunnel based on KCP with N:M multiplexing and FEC. Available for ARM, MIPS, 386 and AMD64。KCPプロトコルに基づく安全なトンネル。KCP 프로토콜을 기반으로 하는 보안 터널입니다。
A Stable & Secure Tunnel based on KCP with N:M multiplexing and FEC. Available for ARM, MIPS, 386 and AMD64。KCPプロトコルに基づく安全なトンネル。KCP 프로토콜을 기반으로 하는 보안 터널입니다。

Disclaimer: kcptun maintains a single website — github.com/xtaci/kcptun. Any websites other than github.com/xtaci/kcptun are not endorsed by xtaci. Re

Jan 9, 2023
A secure image proxy server

go-camo Contents About How it works Differences from Camo Installing pre-built binaries Building Running Running on Heroku Securing an installation Co

Jan 7, 2023