Note this is an interim change, and we should fully implement https://github.com/hyperledger/firefly/issues/1108 such that only a transition of the state to Error would result in a WebSocket update (not simply us writing an error to our local state from a single execution of the polling loop).

This PR:

1. Fixes the problem that we didn't propagate error updates at all
2. Introduces a more complete history list, that includes when we submitted the transaction, when we got a receipt etc.

Subscribers can use /transactions/{txid} to provide full detail about a transaction if required.

Delivered as part of closing https://github.com/hyperledger/firefly/issues/1108

Signed-off-by: Matthew Whitehead [email protected]

Adding a boolean flag in the Managed Transaction to indicate whether the transaction is pre-signed. Therefore, the connector can use that flag to call sendRawTransaction .

Adding endpoints for use within K8s for liveness / readiness probes, as well as for upstream connector services like tokens. See https://github.com/hyperledger/firefly-tokens-erc20-erc721/pull/85 where we noticed we are missing these endpoints that Ethconnect did have.

TODO

• [x] unit tests

... also removes the last vestige of a dependency on FF Core 🪶

Updates the reply processing to emit all updates - note that FF core will need to process a new header.type of TransactionUpdate (as well as TransactionFailure and TransactionSuccess).

I've merged #15 into this branch

This is quite a big change.

• Removes the Docker build from this package - this means the existing builds will stay for those using them
• Reinstates the FFCAPI definition here, which was recently moved to FF common, and now removed in https://github.com/hyperledger/firefly-common/pull/15
• Refactors the FFCAPI to be a local interface, rather than remote
  • This is because it's just too complex to having the potential of state mismatch in event state between the FFCAPI and the FFTM, and build a clean interface
• Removes the fftm and cmd directories
• Moves the internal/manager to be pkg/fftm for use by FFCAPI connectors
  • They will inject their implementation into the fftm.Manager to start their server

The aim of this PR is to add an address balance API call to the FFCAPI interface. This makes it possible for users of of the FFCAPI (such as FireFly Transaction Manager) to query and optionally surface via their own APIs.

Under this PR I have introduced a new REST API /gastoken/balances/{address} which calls AddressBalance() for the the configured blockchain connector. The use of gastoken in the API path is intended to be clear that it is unrelated to token balances.

Signed-off-by: Matthew Whitehead [email protected]

The initial commit adds a single metric, ff_transaction_submission_error_total, that allows the user to track the number of errors from transaction submissions.

Signed-off-by: Matthew Whitehead [email protected]

There was an unused receive channel on the WebSocket connection, which was updated in #36 - but the actual channel that was used to queue acks (the receiveChannel) that is shared between all connections on given topic, was not updated.

We saw a case in a performance run, where FireFly core skipped a batch of 3 events from EVMConnect.

EVMConnect believed they had been acknowledged and received, but FF Core never actually processed them.

What we found from the logs, is that FireFly Core had ended up in a situation where it was sending acknowledgements that were being interpreted by FFTM/EVMConnect as being for the next batch after the one that had just been processed.

This combined with a disconnect/reconnect of the websocket, where a batch was dropped. After the reconnect - EVMConnect didn't re-send the missed batch, because it believed it already to be acknowledged.

The problem is the "acknowledge last thing you sent" protocol, combined with the fact that the WS code doesn't close the go channels through reconnects (deliberately to support multiple WS connections/reconnects). It leaves a window where an ack can be misinterpretted from before a reconnect. There are attempts in the code to make this window very small - with the code by clearing out the go channel that passes ack payloads to the batch dispatcher, before delivering the batch. However, that does not eliminate the window completely.

So this PR make a breaking change to the protocol to be specific about the batch being ack'd:

• Change the payload for events passed by FFTM/EVMConnect to FF Core, from a top-level array, to an object structure like:

  { "batchNumber":  12345, "events": [ ... ] }
  

• Change the ack payload to have a batchNumber declaring which batch is being ack'd
• Moved "spurious" / extra ack processing from the WS side of the Go channel, to the batch manager side
• Extend the length of the go channel from 1 to 10 - because now the acks are unique, we could stall if we missed one
• If for any reason we get too many spurious acks that fill up the channel, then close the websocket rather than losing it (there really only ever should be one, after a reconnect)

FF Core and the Tokens connectors will need to be updated:

• To handle either the old EthConnect flat array payloads, or the new style FFTM/EVMConnect payloads with an object containing batchNumber and events
• Include the batchNumber in the corresponding ack

Note that the reply payloads that do not require acknowledgement come over the same pipe, and this is an area that could be overall made more consistent in the future.

http://masterminds.github.io/sprig/ is a common supplemental library for Go templating that has handy functions for ints and floats.

For oracle templates where we need to convert from gwei to wei, and /or convert from float to int, these utility functions can be very useful. This also provides us with a good starting point of examples in the event we ever need to write our own Go template functions such as converting from hex to bigints, etc.

Thanks @peterbroadhurst for doing the hard part of getting everything in place.

Similar to https://github.com/hyperledger/firefly/issues/189, it'd be great if FFTM had a Prometheus metrics manager and use https://gitlab.com/hfuss/mux-prometheus in the API server middleware so that all the APIs are automatically instrumented.

Then with a metrics manager, implementations of FFTM like EVMConnect can add their own custom metrics.

Currently if I submit a transaction with mistakes, evmconnect will continue to retry the transaction forever with no way to stop it. For example, I accidentally submitted a transaction with the wrong from address which had no funds to pay for gas. The transaction continually failed with no way to retry it. In instances like this, it would be good to have an endpoint that can be used to prevent retries from occurring.