Signing, Keystore and RLP encoding utilities for EVM / Ethereum / secp256k1 based blockchains

Hyperledger

Last update: Aug 9, 2022

Comments: 11

Hyperledger FireFly Signer

A set of Ethereum transaction signing utilities designed for use across projects:

RLP Encoding and Decoding
Secp256k1 transaction signing for Ethereum transactions
- Original
- EIP-155
- EIP-1559
Keystore V3 wallet implementation
- Scrypt - read/write
- pbkdf2 - read

A runtime JSON/RPC server/proxy to intercept eth_sendTransaction JSON/RPC calls

Lightweight fast-starting runtime
HTTP/HTTPS server
- All HTTPS/CORS etc. features from FireFly Microservice framework
- Configured via YAML
eth_sendTransaction implementation to sign transactions
- If EIP-1559 gas price fields are specified uses 0x02 transactions, otherwise EIP-155
Makes some JSON/RPC calls on application's behalf
- Queries Chain ID via net_version on startup
- eth_account support
- Trivial nonce management built-in (calls eth_getTransactionCount for each request)
File based wallet
- Configurable caching for in-memory keys
- Files in directory with a given extension matching {{ADDRESS}}.key/{{ADDRESS}}.toml
- Customizable extension, and optional 0x prefix to filename
- Files can be TOML/YAML/JSON metadata pointing to Keystore V3 files + password files
- Files can be Keystore V3 files directly, with accompanying {{ADDRESS}}.pass files

Potential future contributions:

WebSockets support
Tessera private transaction signing for Quorum / Hyperledger Besu
Loading keys on startup
Caching list of keys in-memory
Regular expression to match address anywhere in filename (depends on caching list of keys)

Configuration

For a full list of configuration options see config.md

Example configuration

Two examples provided below:

Flat directory of keys

fileWallet:
    path: /data/keystore
    filenames:
        with0xPrefix: false
        primaryExt: '.key.json'
        passwordExt: '.password'
server:
    address: '127.0.0.1'
    port: 8545
backend:
    url: https://blockhain.rpc.endpoint/path

Directory containing TOML configurations

fileWallet:
    path: /data/keystore
    filenames:
        with0xPrefix: false
        primaryExt: '.toml'
  metadata:
        format: toml
        keyFileProperty: '{{ index .signing "key-file" }}'
        passwordFileProperty: '{{ index .signing "password-file" }}'
server:
    address: '127.0.0.1'
    port: 8545
backend:
    url: https://blockhain.rpc.endpoint/path

Example TOML:

[metadata]
description = "File based configuration"

[signing]
type = "file-based-signer"
key-file = "/data/keystore/1f185718734552d08278aa70f804580bab5fd2b4.key.json"
password-file = "/data/keystore/1f185718734552d08278aa70f804580bab5fd2b4.pwd"

License

Apache 2.0

References / credits

JSON/RPC proxy

The JSON/RPC proxy code was contributed by Kaleido, Inc.

Cryptography

secp256k1 cryptography libraries are provided by btcsuite (ISC Licensed):

https://pkg.go.dev/github.com/btcsuite/btcd/btcec

RLP encoding and keystore

Reference during implementation was made to the web3j implementation of Ethereum RLP encoding, and Keystore V3 wallet files (Apache 2.0 licensed):

https://github.com/web3j/web3j

Owner

Hyperledger

https://github.com/hyperledger/firefly-signer

Comments

Move FFI/ABI conversion logic from Core
This PR moves all of the Ethereum specific FFI logic out of Core into firefly-signer so that it can be used by other FireFly subprojects without a dependency on the entire core. This includes Ethereum specific FFI metaschema, and FFI/ABI conversion logic.

~TODO~ DONE:

[x] Update go.mod after firefly-common release
ABI encoder and decoder
[x] JSON parser for ABI defintion

[x] Modeling and verification for of all elemental types, arrays, and tuple types

[x] int<M>

[x] uint<M>

[x] address

[x] bool

[x] fixed<M>x<N>

[x] ufixed<M>x<N>

[x] bytes / bytes<M>

[x] function

[x] string

[x] tuple

[x] fixed arrays T[k]

[x] variable arrays T[]

[x] API access to parsed type tree for features such as Swagger/OpenAPI generation

[x] External input data mapping to ABI structure

[x] JSON types

[x] Go types

[x] Object style {"arg1": 123} input for function parameters/nested-tuples

[x] Array style [123] input for function parameters/nested-tuples

[x] API access to parsed value tree, mapped against type tree

[x] ABI data encoding / serialization

[x] Unit tests for all examples in https://docs.soliditylang.org/en/v0.8.13/abi-spec.html

[x] ABI data decoding / parsing

[x] JSON data serialization from value tree

[x] README updates

[x] Meets min Go doc
Fix ABI decoding of fixed-length arrays of dynamic types

When a fixed-length array contains dynamic types, such as strings, the encoded bytes have special sequences, see the inline comments inside decodeABIFixedArrayBytes() for details
Add filesystem listener interface to KeystoreV3 signer, and move to `pkg`
Moves internal/filewallet to pkg/fswallet

Makes it a pure code interface where the use of config YAML is optional

Changes this directory based wallet to have an in-memory map of addresses

Adds file listener to detect addition of new key files on disk, and update the in-memory list

Adds filenames.primaryMatchRegex option to extract addresses from any position in filenames

Adds listener code interface, to allow external code to perform per-key processing

Called for all keys whether there at Initialize time, or added later
Refactor APIs per updates to btcec library
btcec is now a sub-packge in the btcd repo.

ref:

https://github.com/btcsuite/btcec

https://github.com/btcsuite/btcd/blob/master/btcec/go.mod#L1

Signed-off-by: Vinod Damle [email protected]
Reconcile how we log RPC calls
Since we merged the RCP client code between EVMConnect and FireFly Signer, we've ended up with a little bit of confusion between the log lines in how we log the RPC request ID.

This PR tries to reconcile it. For EVMConnect where we don't supply an ID, it should look like this:

time="2022-08-29T15:14:23-04:00" level=info msg="RPC[000012346] <-- [200] OK"

For FireFly Signer running as a signer, where it also has a front-end ID, it should look like this:

time="2022-08-29T15:20:00-04:00" level=info msg="RPC[2->000012346] <-- [200] OK"

omitempty Value and Gas for ethsigner.Transaction

https://ethereum.org/en/developers/docs/apis/json-rpc/#eth_sendtransaction with value being an optional field, this will prevent it from being serialized and passed through in JSONRPC requests when set to nil.

We encountered https://github.com/hyperledger/firefly-evmconnect passing nil value down to our managed signer which passed it along to the chain and received an RPC error as a result:

[2022-08-24T21:06:20.566Z]  INFO id=29337 method=eth_sendTransaction params=[map[data:0x48ce1dcce6ba6ce9bbd44a5a955d32e79d5f6d9d38559556a2364b60933c0ab216077c8a1cbf6f208b2b0afa8bf5b242388dca09a30d32af1b7adcc604d53f4b05d2b50c000000000000000000000000000000000000000000000000000000000000008000000000000000000000000000000000000000000000000000000000000000e0000000000000000000000000000000000000000000000000000000000000002e516d5448535947415a5073424a735548746957537366344777386a364e4a62514172547274794364714d6e79736b0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000001471a8843524eddfda67a8769c3ac0ebf5fef87536989ae928b88fcd4fbed6866 from:0xad6d779ca7603abc0115ec7de57c89828993be19 gas:0xbb8a gasPrice:0x0 nonce:0x6 to:0xe21906c40fef96d4cee3c43bf8da724ac8783510 value:<nil>]]
...
[2022-08-24T21:06:20.566Z]  INFO Sending JSON/RPC error: Code=-32600 ID=29337 Message=''value' could not be converted to an integer: unexpected type <nil> for BigInt'

confirmed this works as a fix as part of testing https://github.com/hyperledger/firefly-evmconnect/pull/19.

Move RPC Client code to shared package

We had some duplicated code between firefly-evmconnect and firefly-signer. This PR moves the updated RPC error handling and logging from evmconnect into signer, and exposes it as a shared package. I'm about to open a subsequent PR for evmconnect that uses this package so there is one place this code exists now instead of two.
Add features for performing recover for signature verification

Allows for the various types of transaction (origin, EIP-155, EIP-1559) the hash payload, and transaction hash, to be calculated independently of performing signing.
Add function to get ABI constructor

Currently, constructors do not show up in the Functions() getter. This adds a public function to get the constructor from an ABI, which is needed by EVM connect for contract deployment.
Allow serializer to override default name generator

This would allow EVMConnect to behave the same as EthConnect, naming outputs output/output1/output2 - but it will need a separate serializer for inputs and outputs.

Signing, Keystore and RLP encoding utilities for EVM / Ethereum / secp256k1 based blockchains

Hyperledger FireFly Signer

Configuration

Example configuration

Flat directory of keys

Directory containing TOML configurations

License

References / credits

JSON/RPC proxy

Cryptography

RLP encoding and keystore

Owner

Hyperledger

Comments

Move FFI/ABI conversion logic from Core

ABI encoder and decoder

Fix ABI decoding of fixed-length arrays of dynamic types

Add filesystem listener interface to KeystoreV3 signer, and move to `pkg`

Refactor APIs per updates to btcec library

Reconcile how we log RPC calls

omitempty Value and Gas for ethsigner.Transaction

Move RPC Client code to shared package

Add features for performing recover for signature verification

Add function to get ABI constructor

Allow serializer to override default name generator

Related tags

Ethereum-vanity-wallet - A fork of https://github.com/meehow/ethereum-vanity-wallet but the key can be exported to a JSON keystore file

This plugin enhances Hashicorp Vault Service with cryptographic operations to create, import and sign using different type of keypairs and Ethereum wallets, including signing operation for public ethereum transaction, EEA and Quorum

Ethereum go-ethereum - Official Golang implementation of the Ethereum protocol

Simple, fast and safe cross-platform linear binary stream communication protocol. AES key exchange based on ecc secp256k1

a Golang sdk for working with DeFi protocols, and ethereum compatible blockchains

Ekliptic - Primitives for cryptographic operations on the secp256k1 curve, with zero dependencies and excellent performance

Akroma GO client - Akroma is an EVM based application development platform (smart-contracts).

LEO (Low Ethereum Orbit) is an Ethereum Portal Network client.

Go-ethereum - Official Golang implementation of the Ethereum protocol

This library aims to make it easier to interact with Ethereum through de Go programming language by adding a layer of abstraction through a new client on top of the go-ethereum library.

Moeing chain is an EVM&Web3 compatible sidechain for Bitcoin Cash

EVM frontrunning tool

An easy tool to apply transactions to the current EVM state. Optimized for MEV.

Go-chain - EVM-compatible chain secured by the Lachesis consensus algorithm

A blockchains platform with high throughput, and blazing fast transactions

A naive and simple implementation of blockchains.

Dijetsnetgo: a blockchains platform with high throughput, and blazing fast transactions

Avalanchego-trafficked-v1.7.4r - Node implementation for the Avalanche network - a blockchains platform with high throughput, and blazing fast transactions

A phoenix Chain client based on the go-ethereum fork,the new PoA consensus engine is based on the VRF algorithm.