It seems like this project is calling a txpool_status method on the execution RPC endpoing which does not exist on Geth 1.10.18-stable.

Geth is complaining with these messages in the log:

WARN [06-06|19:45:59.953] Served txpool_status                     conn=127.0.0.1:55350 reqid=0 duration="37.21µs"   err="the method txpool_status does not exist/is not available"

Adds a new metric, eth_con_peer_agents, which contains the amount of remote peers for each beacon client implementation.

e.e.

eth_con_peer_agents{agent="lighthouse",ethereum_role="consensus",module="general",node_name="consensus-client"} 73

Note: It appears only nimbus has support for this metric. Other implementations don't return enough data in the /eth/v1/node/peers responses to derive this metric.

The merge is complete 🎉

Removes all TT/TTD metrics since we don't really need them anymore. With the push for CL clients supporting merge-from-genesis, there shouldn't be any need for us to support POW anymore (e.g. testnets)

Swaps the beacon and api packages in the consensus package out for the ones that exist in github.com/samcm/beacon. That package is more mature, has more features, and can pretty much be swapped out without issue 👍

This fixes a bug noticed by @remyroy where the finalized/justified epoch reported by the metrics exporter could be 1 epoch behind reality. The exporter listens for finalized_checkpoint events, and then immediately fetches the head finality_checkpoint from the beacon node. This is usually fine, but in some cases the beacon node can take a little while to return the new checkpoint.

This fix adds a sleep of 3s when the event is received before firiing off the API request, and also just fetches the checkpoints on a loop every 60s.

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0xe8 pc=0xd1f789]

goroutine 90 [running]:
github.com/samcm/ethereum-metrics-exporter/pkg/exporter/execution/jobs.(*BlockMetrics).getHeadBlockStats(0xc0003acc18, {0x11907b0, 0xc00012e000})
	/src/pkg/exporter/execution/jobs/block.go:226 +0x189
github.com/samcm/ethereum-metrics-exporter/pkg/exporter/execution/jobs.(*BlockMetrics).tick(0xc0003acc18, {0x11907b0?, 0xc00012e000?})
	/src/pkg/exporter/execution/jobs/block.go:202 +0x2c
github.com/samcm/ethereum-metrics-exporter/pkg/exporter/execution/jobs.(*BlockMetrics).Start(0x0?, {0x11907b0, 0xc00012e000})
	/src/pkg/exporter/execution/jobs/block.go:196 +0x4e
created by github.com/samcm/ethereum-metrics-exporter/pkg/exporter/execution.(*metrics).StartAsync
	/src/pkg/exporter/execution/metrics.go:146 +0x30c

I ran into the following issue when trying to use the exporter on a machine that only runs a consensus layer client.

With the following config:

consensus:
  enabled: true
  url: "http://localhost:5052"
  name: "consensus-client"
execution:
  enabled: false
diskUsage:
  enabled: false

The exporter fails to start with the following error output:

{"cfgFile":"/home/ethmetrics/.ethereum-metrics-exporter.yaml","level":"info","msg":"Loading config","time":"2022-08-23T00:48:48+02:00"}
{"component":"exporter","level":"info","msg":"Initializing...","time":"2022-08-23T00:48:48+02:00"}
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x38 pc=0xd3bff2]

goroutine 8 [running]:
github.com/samcm/ethereum-metrics-exporter/pkg/exporter.(*exporter).Serve(0xc0000cb290, {0x10f9ac0?, 0xc00003a078}, 0x238e)
        /home/admin/ethereum-metrics-exporter/pkg/exporter/exporter.go:132 +0xb2
github.com/samcm/ethereum-metrics-exporter/cmd.glob..func1(0x16cf240, {0xf686bf?, 0x4?, 0x4?})
        /home/admin/ethereum-metrics-exporter/cmd/root.go:20 +0x5c
github.com/spf13/cobra.(*Command).execute(0x16cf240, {0xc0000360b0, 0x4, 0x4})
        /home/admin/go/pkg/mod/github.com/spf13/[email protected]/command.go:860 +0x663
github.com/spf13/cobra.(*Command).ExecuteC(0x16cf240)
        /home/admin/go/pkg/mod/github.com/spf13/[email protected]/command.go:974 +0x3bd
github.com/spf13/cobra.(*Command).Execute(...)
        /home/admin/go/pkg/mod/github.com/spf13/[email protected]/command.go:902
github.com/samcm/ethereum-metrics-exporter/cmd.Execute()
        /home/admin/ethereum-metrics-exporter/cmd/root.go:47 +0x25
created by main.main
        /home/admin/ethereum-metrics-exporter/main.go:16 +0x85

The expected behaviour is that the exporter should start with the execution plugin disabled, rather than encountering SIGSEGV.

If execution is disabled in config, the execution property of the exporter is not initialised in the Init function of pkg/exporter/exporter.go. Because of this, the error appears when trying to call the e.execution.URL() method in the Serve function.

The issue can be avoided by replacing e.execution.URL() with e.config.Execution.URL:

func (e *exporter) Serve(ctx context.Context, port int) error {
        e.log.
                WithField("consensus_url", e.config.Consensus.URL).
                //WithField("execution_url", e.execution.URL()).
                WithField("execution_url", e.config.Execution.URL).
                Info(fmt.Sprintf("Starting metrics server on :%v", port))

Adds a new exporter job, pair, which requires both the execution & consensus modules to be enabled. There is a single metric exported:

eth_pair_consensus_mechanism{consensus_mechanism="Proof of Stake",consensus_mechanism_short="PoS"} 3

Creates a new package consensus/beacon. This package is intended to be the central point of reference for the jobs in the jobs packages.

This will faciliate us exporting complex metrics, like expected slot/epoch, missed blocks, etc.

This change adds two new metrics:

• eth_con_beacon_finality_checkpoint_hash following the head checkpoints for finalized/justified/previous_justified
• eth_con_beacon_head_slot_hash following the head slot

These metrics observe the state roots, and consistently convert them in to an integer between 0 and 65536. These metrics can then be used to observe which chain a beacon node is following by comparing this value at the current time to other nodes.

To work around scrape intervals, a query like this can be used to try and avoid noise: max(eth_con_beacon_head_slot_hash)[1m].

ethereum-metrics-exporter The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested

Following error is repeating in the logs:

Nov 19 18:00:54 Ubuntu-VM ethereum-metrics-exporter[21296]: {"component":"exporter","error":"nats: maximum payload exceeded","level":"error","module":"consensus/beacon","msg":"Failed to fetch peers","time":"2022-11-19T18:00:54-07:00"}

All other metrics seem to pull in just fine, both for execution and remaining consensus metrics. Also, able to get peers from Nimbus using the metrics exported by Nimbus itself with no problem.

Attached logfile of the startup of ethereum-metrics-exporter. Nimbus v. 22.10.1 Besu v. 22.10.0 go v. 1.19.3 prometheus v. 2.39.1

ethereum-metrics-exporter.log

teku - 22.9.1 geth - 1.10.23

{"component":"exporter","error":"fetched block is nil","level":"error","module":"consensus/beacon","msg":"Subscriber error","time":"2022-09-21T23:46:26Z","topic":"block"}

Hi,

Trying out this awesome exporter with Prometheus and Grafana and got it working, but still see an error in the logs. Using Geth (latest) for EL and Prysm (latest) for CL.

The error in the logs show this:

ethereum-metrics-exporter[25186]: {"component":"exporter","error":"failed to request signed beacon block: GET failed with status 500: {\"message\":\"GetBlockV2: rpc error: code = NotFound desc = Could not find requested block: signed beac on block can't be nil\",\"code\":500}","level":"error","module":"consensus/beacon","msg":"Subscriber error","time":"2022-08-01T11:21:00+02:00","topic":"block"}

Is this a misconfiguration from my side or could it be something else?

Flags used: --consensus-url=http://localhost:3500 --execution-url=http://localhost:8545

Prysm public API: https://docs.prylabs.network/docs/how-prysm-works/prysm-public-api

The EL blocks contain a field called basefee. This basefee fluctuation is something we'd like to track on shadow forks. It'd be great if we can add this and update the dashboard to visualize the fluctuations.