Using latest backend and Postgres which Hypersync set to false causes the backend to restarted every x seconds. Last processed block is 39120. No exceptions spotted so far

part of log ` backend | I0508 10:10:03.886061 1 config.go:168] SnapshotBlockHeightPeriod: 1000 backend | I0508 10:10:03.886365 1 config.go:172] ArchivalMode: ON backend | I0508 10:10:03.886526 1 config.go:178] DisableSlowSync: OFF backend | I0508 10:10:03.886771 1 config.go:198] READ ONLY MODE backend | I0508 10:10:03.887080 1 config.go:206] IGNORING INBOUND INVS backend | I0508 10:10:03.887215 1 config.go:209] Max Inbound Peers: 125 backend | I0508 10:10:03.887326 1 config.go:210] Protocol listening on port 17000 backend | I0508 10:10:03.887417 1 config.go:216] Rate Limit Feerate: 0 backend | I0508 10:10:03.887529 1 config.go:217] Min Feerate: 1000 backend | I0508 10:10:03.903783 1 node.go:126] Looking for AddIPs: 0 backend | I0508 10:10:03.903805 1 node.go:131] Looking for DNSSeeds: 10 backend | badger 2022/05/08 10:10:22 INFO: All 0 tables opened in 0s backend | badger 2022/05/08 10:10:22 INFO: Discard stats nextEmptySlot: 0 backend | badger 2022/05/08 10:10:22 INFO: Set nextTxnTs to 78238 backend | badger 2022/05/08 10:10:22 INFO: Deleting empty file: /db/v-00000/badgerdb/000258.vlog backend | Migrations already up to date backend | I0508 10:10:24.588252 1 mempool.go:2262] Calling StartReadOnlyUtxoViewRegenerator... backend | I0508 10:10:24.588354 1 server.go:2304] Server.Start: Starting Server backend | I0508 10:10:24.589516 1 connection_manager.go:828] Full node socket initialized backend | I0508 10:10:24.590674 1 node.go:40] GlobalState BadgerDB Dir: /db/v-00000/badgerdb/global_state backend | I0508 10:10:24.590744 1 node.go:41] GlobalState BadgerDB ValueDir: /db/v-00000/badgerdb/global_state/badgerdb backend exited with code 137 backend | I0508 10:10:28.736972 1 config.go:151] Logging to directory /db/v-00000 backend | I0508 10:10:28.738364 1 config.go:152] Running node in MAINNET mode backend | I0508 10:10:28.738384 1 config.go:153] Data Directory: /db/v-00000

` Also see some SSLerrors with "https://api.blockchain.com/v3/exchange/tickers/CLOUT-USD

backend | E0508 10:09:20.123602 1 base.go:142] GetBlockchainDotComExchangeRate: Problem with HTTP request https://api.blockchain.com/v3/exchange/tickers/CLOUT-USD: Get "https://api.blockchain.com/v3/exchange/tickers/CLOUT-USD": net/http: TLS handshake timeout Problem with HTTP request https://api.blockchain.com/v3/exchange/tickers/CLOUT-USD: Get "https://api.blockchain.com/v3/exchange/tickers/CLOUT-USD": net/http: TLS handshake timeout

Not sure with exited with code 137 means.

I am proud to announce that the dev community has committed to moving to a zero-waste consensus mechanism, and that it is taking a major step in that direction with a new update.

Cryptocurrencies have gotten a lot of heat recently for being harmful to the environment.

This is due in large part to "proof of work," a consensus mechanism that burns CPU power to secure the network.

BitClout has relied on proof of work to get off the ground, but since launching the dev community has already invested in consensus mechanisms that allow BitClout to operate on orders of magnitude less power than most existing proof of work cryptocurrencies.

Now, in a show of solidarity with the environmental movement, and to take the first big step toward a zero-waste consensus mechanism, BitClout is deploying a new update that reduces its power consumption by a full order of magnitude.

BitClout is reducing the block reward, and thus the amount of environmental waste its consensus mechanism incurs, by a factor of ten.

This was a difficult change to make, and it took the input of several major node operators to deploy it safely.

We understand that miners in particular may not like this change; however, there is now consensus around the decision to push the limits of the BitClout chain toward conservation rather than allowing the depletion of what are ultimately finite environmental resources.

There is still more work to do to move to a zero-waste consensus mechanism without compromising on decentralization (e.g. full proof of stake).

But the dev community is committed to getting there sooner rather than later, and we hope this change shows just how committed we are.

When doing a fresh sync with the latest version of backend / core. Fresh node, fresh DB blocks are not processed correctly from block 98493.

Seems like some cache or array map index mismatch. Will check it out.

See error:

`I0205 11:21:58.422185 1 server.go:1147] Server._handleBlock: Received block ( 98492 / 101262 ) from Peer [ Remote Address: 34.123.41.111:17000 PeerID=1 ] I0205 11:21:58.661121 1 server.go:1147] Server._handleBlock: Received block ( 98493 / 101262 ) from Peer [ Remote Address: 34.123.41.111:17000 PeerID=1 ] panic: assignment to entry in nil map [recovered] panic: assignment to entry in nil map

goroutine 116 [running]: github.com/go-pg/pg/v10.(*Tx).RunInTransaction.func1() /root/go/pkg/mod/github.com/go-pg/pg/[email protected]/tx.go:91 +0x86 panic({0x10a30a0, 0x16fdd80}) /usr/lib/go/src/runtime/panic.go:1038 +0x215 github.com/deso-protocol/core/lib.(*Postgres).flushPosts(0xa33af3, 0xc0bac47f90, 0xc0009e1680) /deso/src/core/lib/postgres.go:1462 +0x94a github.com/deso-protocol/core/lib.(*Postgres).FlushView.func1(0x0) /deso/src/core/lib/postgres.go:1303 +0x7d github.com/go-pg/pg/v10.(*Tx).RunInTransaction(0xc0bb0334c0, {0x172c7e8, 0xc00003a290}, 0xc0b3b1d638) /root/go/pkg/mod/github.com/go-pg/pg/[email protected]/tx.go:95 +0xa5 github.com/go-pg/pg/v10.(*baseDB).RunInTransaction(0xc00003a290, {0x172c7e8, 0xc00003a290}, 0xc0b3b1d688) /root/go/pkg/mod/github.com/go-pg/pg/[email protected]/tx.go:74 +0x51 github.com/deso-protocol/core/lib.(*Postgres).FlushView(0xb1df40, 0xc000010040) /deso/src/core/lib/postgres.go:1296 +0x57 github.com/deso-protocol/core/lib.(*UtxoView).FlushToDb(0xc0009e1680) /deso/src/core/lib/block_view_flush.go:16 +0x34 /deso/src/core/lib/blockchain.go:1942 +0x1946 github.com/deso-protocol/core/lib.(*Server)._handleBlock(0xc0002103c0, 0xc0003eed80, 0xc0bcf05110) /deso/src/core/lib/server.go:1193 +0x3e5 github.com/deso-protocol/core/lib.(*Server)._handlePeerMessages(0xc0002103c0, 0xc1128a2fc0) /deso/src/core/lib/server.go:1494 +0x18d github.com/deso-protocol/core/lib.(*Server).messageHandler(0xc0002103c0) /deso/src/core/lib/server.go:1534 +0x19c created by github.com/deso-protocol/core/lib.(*Server).Start /deso/src/core/lib/server.go:1710 +0xb1`

Seems the issue is related to additional NFT royalties introduced after the hard fork. Error is related to this line https://github.com/deso-protocol/core/blob/main/lib/postgres.go#L1462

This transaction 3JuEUKubpQdqioRRtyP7JrNV5Fo1FQ1iJ9QQDpCri1C59b15asubezin block 98493 has additional NFT royalties

Might be a byte to string conversion issue. Seems like byte is expected but a string is used. Not sure. Need Go crash course

team,

the salomon bug has been fixed however i'm still seeing accounts that their price are not making sense.

https://github.com/bitclout/core/blob/2cbd77da61109303f3f961c227a414ab063a74d6/lib/constants.go#L78

It seems the bug is not only happening on buy option but sell too.

Please check https://bitclout.com/u/Petrh -> it seems the bug occurred in the sell option , different from salomon bug. https://bitcloutsignal.com/history/petrh

Also please look at https://bitclout.com/u/speakinglight -> this seems to be like the salomon bug https://bitcloutsignal.com/history/speakinglight

please look at this before we get more users onboarded, since this can be a significant challenge going forward if the creator coins are not always working correctly.

this might be a UI problem and not blockchain, if we go to signalclout we can see their real price.

petrh - should be around 555 usd speakinglight - should be around 268 usd

thanks in advance!

When running a fresh postgres sync with backend:latest the backend hangs and keeps restarting at block 46631.

Probably related to some recent changes. It´s the block with the first NFTs created it seems.

`I0226 09:05:44.548962 1 server.go:1147] Server._handleBlock: Received block ( 46631 / 107387 ) from Peer [ Remote Address : 35.232.92.5:17000 PeerID=1 ] panic: runtime error: invalid memory address or nil pointer dereference [recovered] panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x49 pc=0xb1b525]

goroutine 56 [running]: github.com/go-pg/pg/v10.(*Tx).RunInTransaction.func1() /root/go/pkg/mod/github.com/go-pg/pg/[email protected]/tx.go:91 +0x86 panic({0x10a56c0, 0x1e8a660}) /usr/lib/go/src/runtime/panic.go:1038 +0x215 github.com/deso-protocol/core/lib.(*Postgres).InsertTransactionsTx(0xc0016054b8, 0xc0016054c8, {0xc016be2000, 0xae4, 0xc0037c7c c0}, 0xc004e97780) /deso/src/core/lib/postgres.go:1092 +0x1b65 github.com/deso-protocol/core/lib.(*Postgres).UpsertBlockAndTransactions.func1(0x0) /deso/src/core/lib/postgres.go:1336 +0x98 github.com/go-pg/pg/v10.(*Tx).RunInTransaction(0xc0016eb280, {0x1730468, 0xc00003a280}, 0xc016cb4230) /root/go/pkg/mod/github.com/go-pg/pg/[email protected]/tx.go:95 +0xa5 github.com/go-pg/pg/v10.(*baseDB).RunInTransaction(0xc017fda798, {0x1730468, 0xc00003a280}, 0xc010586ea0) /root/go/pkg/mod/github.com/go-pg/pg/[email protected]/tx.go:74 +0x51 github.com/deso-protocol/core/lib.(*Postgres).UpsertBlockAndTransactions(0xc000168e10, 0xc010586ea0, 0xc017c22000) /deso/src/core/lib/postgres.go:1324 +0x5c github.com/deso-protocol/core/lib.(*Blockchain).ProcessBlock(0xc00082ebb0, 0xc010586ea0, 0xe5) /deso/src/core/lib/blockchain.go:1935 +0x191c github.com/deso-protocol/core/lib.(*Server)._handleBlock(0xc000fd80c0, 0xc001bb6d80, 0xc010586ea0) /deso/src/core/lib/server.go:1193 +0x3e5 github.com/deso-protocol/core/lib.(*Server)._handlePeerMessages(0xc000fd80c0, 0xc001bb6d80) /deso/src/core/lib/server.go:1494 +0x18d github.com/deso-protocol/core/lib.(*Server).messageHandler(0xc000fd80c0) /deso/src/core/lib/server.go:1534 +0x19c created by github.com/deso-protocol/core/lib.(*Server).Start /deso/src/core/lib/server.go:1710 +0xb1 `

Will check what is causing it.

During Postgres sync with the last Postgres Dao updates got this error:

server.go:1147] Server._handleBlock: Received block ( 98475 / 98627 ) from Peer [ Remote Address: 34.123.41.111:17000 PeerID=32 ] E0126 08:54:21.794215 1 server.go:1141] Server._handleBlock: Encountered an error processing block <Header: < 98475, 00000000000058e8268e3ca270ab453fdd41382623c5ca30516d6ddf95b7b764, 1 >, Signer Key: BC1YLh768bVj2R3QpSiduxcvn7ipxF3L3XHsabZYtCGtsinUnNrZvNN>. Disconnecting from peer [ Remote Address: 34.123.41.111:17000 PeerID=32 ]: Error while processing block: : ProcessBlock: Problem upserting block and transactions: ERROR #23502 null value in column "transfer_restriction_status" violates not-null constraint

Seem related to Dao coin insert/update at first glance. Related to DAOCoinTransferRestrictionStatus TransferRestrictionStatus pg:"dao_coin_transfer_restriction_status"

When this block hits in the sync all blocks after get status 26 in pg_blocks.

Looking for a solution or fix.

So transfer_restriction_status SMALLINT NOT NULL seems to get a NULL value and rejects that. type PGMetadataDAOCoin struct { tableName struct{}pg:"pg_metadata_dao_coins"`

TransactionHash           *BlockHash           `pg:",pk,type:bytea"`
ProfilePublicKey          []byte               `pg:",type:bytea"`
OperationType             DAOCoinOperationType `pg:",use_zero"`
CoinsToMintNanos          string
CoinsToBurnNanos          string
TransferRestrictionStatus `pg:",use_zero"`

}`

Might be related to pg:",use_zero" for that enum type.

Would transfer_restriction_status SMALLINT NOT NULL with DEFAULT 0 added solve the issue? Took me 2 days to sync to get to that point. Any ideas are welcome.

This PR from @tijno brings us to block 98477 #201 But still seems something goes wrong with the status field

As BitClout development becomes increasingly community-driven, I believe it is time for the creation of a repository to draft and track BitClout Improvement Proposals (abbreviated BCIPs).

Overall, the BitClout community is understanding of the need, early on, for the core development team to have very flexible control over the protocol. However, more and more of us in the development community are building real-world businesses on top of the BitClout protocol, and I believe it would be a benefit to everyone if such a BCIP system existed.

This proposal calls for the creation of a repo at github.com/bitclout/bcips, similar to the equivalent repo within Bitcoin, github.com/bitcoin/bips.

Individuals drafting proposals would simply fork the bitclout/bcips repo, create a folder and README for their proposal (bcip-001, bcip-002, etc), and submit a pull request back to the bitclout/bcips repo. It would be up to the core development team to edit and merge these proposals as they see fit.

These proposals should be limited in scope to the core BitClout protocol, not the backend, frontend, or any other project.

Thank you for your consideration.

Not sure where else to report this.

All third party nodes are down, and transaction-info on the main node is throwing 502s.

I am bitclout.com/u/smartalec

Postgres sync of latest build stops on block 99415 and keeps restarting. With fresh sync. Probably some nil value. Will debug

`I0305 08:07:47.297285 1 server.go:1147] Server._handleBlock: Received block ( 99415 / 109439 ) from Peer [ Remote Address: 34.123.41.111:17000 PeerID=2 ] panic: runtime error: invalid memory address or nil pointer dereference [recovered] panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0xb26d7e]

goroutine 64 [running]: github.com/go-pg/pg/v10.(*Tx).RunInTransaction.func1() /root/go/pkg/mod/github.com/go-pg/pg/[email protected]/tx.go:91 +0x86 panic({0x10b9580, 0x1eaa660}) /usr/lib/go/src/runtime/panic.go:1038 +0x215 github.com/deso-protocol/core/lib.(*Postgres).InsertTransactionsTx(0xc0017114b8,0xc0017114c8, {0xc0053de800, 0xb7, 0xc006142660}, 0xc009163540) /deso/src/core/lib/postgres.go:1119 +0x1c1e github.com/deso-protocol/core/lib.(*Postgres).UpsertBlockAndTransactions.func1(0x0) /deso/src/core/lib/postgres.go:1353 +0x98 github.com/go-pg/pg/v10.(*Tx).RunInTransaction(0xc005bf8580, {0x17493e8, 0xc00003a290}, 0xc0054e9f80) /root/go/pkg/mod/github.com/go-pg/pg/[email protected]/tx.go:95 +0xa5 github.com/go-pg/pg/v10.(*baseDB).RunInTransaction(0xc0053e3f68, {0x17493e8, 0xc00003a290}, 0xc001b0dc20) /root/go/pkg/mod/github.com/go-pg/pg/[email protected]/tx.go:74 +0x51 github.com/deso-protocol/core/lib.(*Postgres).UpsertBlockAndTransactions(0xc00540a2d0, 0xc001b0dc20, 0xc0056f6000) /deso/src/core/lib/postgres.go:1341 +0x5c`

Hey there, we have setup our node, setup GCP_BUCKET_NAME and GCP_CREDENTIALS_PATH, we're able to upload images to the GCP_BUCKET without issue,

So far so good, buut I had to update the Caddy file to be able to display images from our bucket to mitigate CSP blocking on our node (adding our bucket address to the img-src - see the Caddyfile attached) . After that, we can display images on our own node, but other nodes (starting with bitclout.com) are not loading images with error "(blocked:csp)". Is there a way how to enable images from our gcp bucket or what is general approach to this when you host your own node and bucket storage?

Post example: Our, working, node: https://node.flickapp.com/posts/8b2877038285a0e2ed59caa3228305f9e91ae0aee68a34ca2d2ac6a82a3530b3

Bitclout node, same post is not displaying image with - (blocked:csp) https://bitclout.com/posts/8b2877038285a0e2ed59caa3228305f9e91ae0aee68a34ca2d2ac6a82a3530b3

Any advice what we can setup to avoid image breakdown on other nodes?

Caddyfile.dev.txt

Thanks for any advice.

Core idea is we accept txns signed by non-owner (derived) keys in _verifySignatureDerived() which is called in _connectBasicTransfer() on signature verification. Client looks for all derived keys associated with the txn public key and checks if any of them matches the signature. With this approach, I'm not making any changes to how txns are sent. A derived key must first be authorized by sending a new TxnTypeAuthorizeDerivedKey transaction constructed in CreateAuthorizeDerivedKeyTxn(). When clients receive the Authorize txn, they connect it and modify the new PKIDToDerivedKeyEntry map in UtxoView. I also added a new prefix in the DB for Authorize entries. When we flush to the DB, we call _flushDerivedKeyEntryToDbWithTxn. Let me know if this makes sense!

Note: I haven't written tests yet, and last time I checked, there was no YT tutorial on how to add a new txn to BitClout. So this is probably buggy :)

Validates total input, spend amount, change amount, fees, and error result computed from chain.AddInputsAndChangeToTransaction.

@AeonSw4n Added tests to validate the output of AddInputsAndChangeToTransaction. Wrote this test as I was working through understanding how transactions, utxoviews, mining work in DeSo.