Test: 25 broadcasts, 25 private messages, one subscription

In this test, 25 workers were kicked off for both broadcasts and private messages. Shortly after the test is kicked off, workers are not receiving websocket events, preventing them from proceeding. The ff-perf.log has logs of message IDs and event IDs that can be used to correlate to the firefly logs that are also attached. Every minute, the perf test checks for messages that have not been confirmed in > 1 minute, do a Ctrl+F for Delinquent Messages and you will see the messageID that has not been confirmed yet, along with the time it was sent at.

log_ethconnect_0.log.zip log_firefly_core_0.log.zip ff-perf.log

In https://github.com/hyperledger/firefly/issues/493#issuecomment-1035538293 we found we missed an event due to the dispatcher skipping past events:

[2022-02-10T20:32:14.213Z] DEBUG Dispatcher event state: readahead=0 candidates=5 matched=5 inflight=0 queued=0 dispatched=5 dispatchable=0 lastAck=37559 nacks=0 highest=37559 pid=1 role=ed[d2ff7b72-504d-40dd-ab3e-e325c2b7a5d9] sub=3cf508e4-21a7-4f00-8ad0-a6d08a0703d0/default:perf_54c7faaf-ad...
[2022-02-10T20:32:14.366Z] DEBUG Dispatcher event state: readahead=0 candidates=1 matched=0 inflight=0 queued=0 dispatched=0 dispatchable=0 lastAck=0 nacks=0 highest=37595 pid=1 role=ed[d2ff7b72-504d-40dd-ab3e-e325c2b7a5d9] sub=3cf508e4-21a7-4f00-8ad0-a6d08a0703d0/default:perf_54c7faaf-ad...

The root cause here is we are writing events concurrently in many goroutines, using different DB transactions - so it's not possible to detect the order of those via a separate DB listener.

In this issue we will track the solution.

FireFly should support the creation and tracking of both fungible and non-fungible tokens. Some of the key concepts that must be supported:

• Creating multiple types of tokens
• Minting, transferring, and burning tokens of a given type
• Tracking transaction history and current account balances (must include operations initiated by FireFly and also those initiated on-chain with no trigger from FireFly)
• Attaching data to a token transfer, for correlating off-chain items with the transfer event on-chain

The first connector will be focused on Ethereum tokens support via ERC1155. Source code for the connector is here: https://github.com/hyperledger/firefly-tokens-erc1155

This functionality is part of milestone #117.

Cross-chain functionality, including atomic swaps and the like, will not be part of this design effort (but will be part of a future item).

Events are extremely important to FireFly applications, but purposefully don't contain a large amount of information. After speaking with @eberger727 and @awrichar it was decided adding a new field, ReferenceType, would be useful to help developers determine which tables the events ref refers.

The above events break down into the following categories, which will be the possible values of ReferenceType

• TokenTransfer
• TransferOperation
• Messages
• BlockchainEvent
• TransactionSubmitted
• TokenApproval [0]
• TokenApprovalOperations [0]

[0]: Coming in https://github.com/hyperledger/firefly/pull/529

Background

Token pools are initialized with two steps:

1. The actual pool is initialized on a blockchain smart contract
2. The pool details (namespace, name, etc) are announced as a FireFly definition broadcast

The pool creator is responsible for sending 2 only after 1 is confirmed. However, because these steps may or may not occur on the same blockchain, there are multiple possible ways for them to play out.

Scenarios

A) 1 and 2 both arrive and are guaranteed to be in that order for all nodes

This is true for the case where tokens and messages are backed by the same blockchain (such as when using Ethereum with ERC1155). Each node may receive and record the blockchain transaction, then fill in the rest of the pool details after the announcement broadcast.

B) 1 and 2 both arrive, but 2 may arrive before 1

This is possible in cases where tokens and messages are backed by different blockchains. It should be uncommon, but networking conditions (for instance) could cause a particular node to receive 2 quickly while 1 is delayed. There must be a way to save the details from 2 but "rewind" and complete the message processing once 1 is received.

C) Only 2 arrives, and 1 must be looked up on demand

This is envisioned for cases where the "token pool" is a pre-existing set of tokens (could be on mainnet or a pre-existing private chain). In this case, step 1 for the author involves simply passing details about the existing tokens, but there will be no new blockchain transaction to define the pool. Therefore, the other nodes receiving the broadcast in step 2 must also have a way to lookup and verify the pre-existing token pool.

Delivery PR for this FIRs into the onchain-logic next-release branch:

• https://github.com/hyperledger/firefly-fir/pull/4
• https://github.com/hyperledger/firefly-fir/pull/7

Proposed fix for #512

By implementing the UpsertOptimization pattern for Groups, it lets us state UpsertOptimizationNew everywhere we're currently just trying to do a query-then-insert approach. UpsertOptimizationNew results in an attempt-insert, then attempt-update approach - which should handle+swallow a duplicate key exception (rather than rolling back the TX).

We have some problems in the codebase today with a half-formed identity approach:

• A Message contains a single Author field, and it is ambiguous whether this is the identity or the signing key
• A shim Identity plugin designed to resolve an opaque string (such as a DID) into an identity on the platform... but without a definition of what that identity is
• The requirement for us to properly solve Fabric signing keys and round-trip these to events like we do in ethereum - currently blocking merge of #184
• The fact that both Ethereum and Fabric connectors have the concept of abstract lookup keys for actual signing keys (HD Wallets in Ethereum, and CN shortnames in Fabric)
• The requirement to handle anonymous signing of transactions for privacy preservation
• The requirement that the database representation of identity in a Message / Token Pool or On-chain event (coming soon) etc. must be consistent in every database of members allowed to see the object, even if they didn't submit the transaction

This issue will track a number of threads needed to resolve this.

Implements #112

• Does lots of dependency work to allow the sync-async bridge to be callable from broadcast/privatemessaging/networkmap
  • Adds a syshandlers package for the system event handlers that we previously bundled into events
  • Adds a sysmessaging that abstracts broadcast/private sends, and reply exchanges
  • Updates the orchestrator initialixation order to manage passing things between the internal components
• Deprecates all the old routes
• Adds new routes that:
  • Are all grouped on the path with the resources they create
  • Consistently return the object type that is the input
  • Have a ?confirm option to block until the confirmation event arrives (or a 500 on the case it's rejected)
• Leaves the old routes there functioning exactly as today
  • Gives time for the CLI setup, and other consumers to move to the new routes

The FireFly performance tests use go routines to send multiple requests to an API at the same time. These routines then wait for their specific websocket event to confirm their transaction was completed before moving on to its next task.

In v0.11.5 of FireFly, these websocket events were being sent correctly, allowing for workers to proceed as expected.

In v0.12.0 of FireFly, one websocket event is received, then no more websocket events come through, resulting in workers not being able to confirm that their transaction completed and moving on to their next task.

ff-perf v0-11-5.log ff-perf v0-12-0.log v0-12-0-ff_core.log

Part of FIR-12

~~Depends on https://github.com/hyperledger/firefly-cli/pull/197~~ ~~In a chain with #904~~

Add a new E2E test which dynamically:

• deploys two instances of the FireFly multiparty contract
• dynamically adds a new namespace to the config of both nodes
• resets both nodes to read the config
• sends one message on the first contract
• performs a network migration
• sends one message on the new contract

There are variations of this test for both V1->V2 contract migrations and V2->V2. Notably, the V1->V2 migration can only be performed once per stack, so it is not part of the main suites. We may need to add a dedicated nightly job for this test.

This PR also includes loads of fixes for things exposed by the E2E tests, including major changes to how the legacy ff_system namespace is managed. There can now only be one multiparty V1 contract EVER used in the history of a FireFly node, and attempting to configure any namespace with a different V1 contract is an error. The ff_system orchestrator will be initialized with that V1 config (if any is found) and will live forever (walking back some of the changes from #884), because we can never be fully sure when all pins from ff_system have been aggregated.

For a test solidity contract that declares custom errors like the following :

pragma solidity ^0.8.0;

error CustomError1(uint256 x, uint256 y);
error CustomError2(string x);

contract SimpleStorage {
    uint256 storedData;

    function set(uint256 x) public {
        if (x == 0) {
            revert CustomError1({x: x, y: x});
        }
        if (x == 1) {
            revert CustomError2({x: "This is a custom error"});
        }
        require(x > 2, "Can not equal to 2");
        storedData = x;
    }
}

Use the POST /contracts/interfaces/generate endpoint to generate the corresponding FFI:

{
    "namespace": "default",
    "name": "simple-storage-custom-errors",
    "description": "simple-storage-custom-errors",
    "version": "1.0.0",
    "methods": [
        {
            "name": "set",
            "pathname": "",
            "description": "",
            "params": [
                {
                    "name": "x",
                    "schema": {
                        "oneOf": [
                            {
                                "type": "string"
                            },
                            {
                                "type": "integer"
                            }
                        ],
                        "details": {
                            "type": "uint256",
                            "internalType": "uint256"
                        },
                        "description": "An integer. You are recommended to use a JSON string. A JSON number can be used for values up to the safe maximum."
                    }
                }
            ],
            "returns": [],
            "details": {
                "stateMutability": "nonpayable"
            }
        }
    ],
    "errors": [
        {
            "signature": "",
            "name": "CustomError1",
            "description": "",
            "params": [
                {
                    "name": "x",
                    "schema": {
                        "oneOf": [
                            {
                                "type": "string"
                            },
                            {
                                "type": "integer"
                            }
                        ],
                        "details": {
                            "type": "uint256",
                            "internalType": "uint256"
                        },
                        "description": "An integer. You are recommended to use a JSON string. A JSON number can be used for values up to the safe maximum."
                    }
                },
                {
                    "name": "y",
                    "schema": {
                        "oneOf": [
                            {
                                "type": "string"
                            },
                            {
                                "type": "integer"
                            }
                        ],
                        "details": {
                            "type": "uint256",
                            "internalType": "uint256"
                        },
                        "description": "An integer. You are recommended to use a JSON string. A JSON number can be used for values up to the safe maximum."
                    }
                }
            ]
        }
    ]
}

Then when invoking the generated API:

matching custom error

curl --location --request POST 'http://127.0.0.1:5000/api/v1/namespaces/default/apis/simple-storage/invoke/set' \
--header 'Content-Type: application/json' \
--data-raw '{
  "input": {
    "x": 0
  }
}'

response:

{
    "error": "FF10111: Error from ethereum connector: FF23021: EVM reverted: CustomError1(\"0\", \"0\")"
}

no matching custom error (raw error data returned):

curl --location --request POST 'http://127.0.0.1:5000/api/v1/namespaces/default/apis/simple-storage/invoke/set' \
--header 'Content-Type: application/json' \
--data-raw '{
  "input": {
    "x": 1
  }
}'

response:

{
    "error": "FF10111: Error from ethereum connector: FF23022: EVM reverted: 0x60b7d47e0000000000000000000000000000000000000000000000000000000000000020000000000000000000000000000000000000000000000000000000000000001654686973206973206120637573746f6d206572726f7200000000000000000000"
}

regular error from the require()

curl --location --request POST 'http://127.0.0.1:5000/api/v1/namespaces/default/apis/simple-storage/invoke/set' \
--header 'Content-Type: application/json' \
--data-raw '{
  "input": {
    "x": 2
  }
}'

response:

{
    "error": "FF10111: Error from ethereum connector: FF23021: EVM reverted: Can not equal to 2"
}

Part of FIR-16. Depends on https://github.com/hyperledger/firefly-tokens-erc20-erc721/pull/106.

Discussion about the overall architecture/usefulness of this proposed feature should be directed to FIR-16, while discussion here should focus on specific technical details of the current proposed approach.

This adds the ability for FireFly to provide ABI details to a token connector, making it easier for the token connector to know the shape of the underlying contract it is dealing with (vs. having to probe/guess at the methods available).

Notes

• This feature is only useful to users who have significant knowledge of the underlying token contract that sits on the other side of their token connector.
• It's likely to be most useful when creating a pool against a pre-deployed contract - ie when passing config.address. Worth mentioning that FireFly doesn't really "know" you can create pools against pre-deployed contracts (as config is opaque), but it's up to the user to understand the combination of features that are useful here.
• Theoretically you could create a factory contract that deploys custom ERC20/ERC721 instances, and then you'd need to pass interface on every pool creation even while relying on the factory sitting on the other side of the token connector to do the deploy. Seems like a less useful combo, but calling it out here for completeness.

Depends on https://github.com/hyperledger/firefly-common/pull/41

Adds support for setting a list of HTTP headers to put on the request Context. If an FFResty client is also to pass headers through, it will set all the same headers on any request that it makes.

Example config:

api:
  address: 0.0.0.0
  port: 5000
  publicURL: http://127.0.0.1:5000
  passthroughHeaders:
    - X-SuperCoolCustomHeader
plugins:
  blockchain:
    - ethereum:
        ethconnect:
          url: http://127.0.0.1:5102
          passthroughHeadersEnabled: true

This would configure FireFly to pass a header named X-SuperCoolCustomHeader through from an HTTP request all the way to the Ethereum blockchain connector.

If "poolDataString" is an empty string and therefore "poolData" is nil, this line results in a nil pointer dereference.

https://github.com/hyperledger/firefly/blob/490745489fe98779cff07821d19ba7a83b8421bf/internal/tokens/fftokens/fftokens.go#L335

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0xbef3d1]

goroutine 58 [running]:
github.com/hyperledger/firefly/internal/tokens/fftokens.(*FFTokens).handleTokenPoolCreate(0xc0002cf960, {0x1093520, 0xc0007e1ec0}, 0xc000d3a630, 0x0)
	/firefly/internal/tokens/fftokens/fftokens.go:335 +0x371
github.com/hyperledger/firefly/internal/tokens/fftokens.(*FFTokens).handleMessage(0xc0002cf960, {0x1093520, 0xc0007e1d00}, {0xc00087b000, 0x3f3, 0x400})
	/firefly/internal/tokens/fftokens/fftokens.go:511 +0x434
github.com/hyperledger/firefly/internal/tokens/fftokens.(*FFTokens).handleMessage(0xc0002cf960, {0x10935c8, 0xc00024d530}, {0xc000840100, 0x44c, 0x580})
	/firefly/internal/tokens/fftokens/fftokens.go:506 +0x878
github.com/hyperledger/firefly/internal/tokens/fftokens.(*FFTokens).eventLoop(0xc0002cf960)
	/firefly/internal/tokens/fftokens/fftokens.go:553 +0x28f
created by github.com/hyperledger/firefly/internal/tokens/fftokens.(*FFTokens).Init
	/firefly/internal/tokens/fftokens/fftokens.go:234 +0x296