I'm running the latest mongo image from dockerhub

The sync_service can't connect.

Error:

Failed to ping MongoDB:  server selection error: server selection timeout, current topology: { Type: Unknown, Servers: [{ Addr: localhost:27017, Type: Unknown, State: Connected, Average RTT: 0, Last error: connection() : dial tcp 127.0.0.1:27017: connect: connection refused }, ] }

Using a test script executed with go run it connects fine. The docker-image being built seems to causing networking issues.

Test script:

package main
import (
	"context"
	"fmt"
	"log"
	"os"

	"go.mongodb.org/mongo-driver/mongo"
	"go.mongodb.org/mongo-driver/mongo/options"
)

// Establishes and returns a MongoDB client with associated URI MongoDbURI
func ConnectToMongo() {
	// Establish MongoDB client options and create client
	clientOptions := options.Client().ApplyURI("mongodb://localhost:27017")
	client, err := mongo.Connect(context.Background(), clientOptions)
	if err != nil {
		log.Fatalf("Failed establishing a connection with MongoDB: %v", err)
	}

	fmt.Println("Pinging")

	// Check MongoDB Connection and ensure data transmission
	err = client.Ping(context.Background(), nil)
	if err != nil {
		log.Fatalf("Failed to ping MongoDB:  %v", err)
	}

	fmt.Println("Successfully Connected to MongoDB.")
}

func main() {
	fmt.Println("Hello", os.Args[1])
	ConnectToMongo()
	os.Exit(42)
}

I came across the below info in https://www.deso.org/blog/state-of-decentralized-social-1-a-status-update-on-decentralized-social

How do I access the Deso posts through SQL? Should I look at https://github.com/deso-protocol/core or this repo? @maebeam @bluepartyhat

The mongodb dumper container halts in between with following message

I0125 12:59:17.844897       1 server.go:1147] Server._handleBlock: Received block ( 95432 / 98294 ) from Peer [ Remote Address: 34.123.41.111:17000 PeerID=1 ]
badger 2022/01/25 12:59:17 INFO: [2] [E] LOG Compact 5->6 (1, 13 -> 13 tables with 5 splits). [36877 . 36657 36662 36663 36658 36659 36214 36216 36218 36205 36208 36197 36198 36189 .] -> [37242 37262 37269 37251 37265 37268 37250 37260 37270 37254 37257 37252 37253 .], took 3.563s
badger 2022/01/25 12:59:18 INFO: [0] [E] LOG Compact 5->6 (1, 10 -> 10 tables with 4 splits). [36889 . 36244 36246 36237 36239 36235 36236 35982 35985 36125 36133 .] -> [37245 37266 37267 37241 37256 37246 37264 37273 37274 37249 .], took 4.279s
badger 2022/01/25 12:59:19 INFO: [3] [E] LOG Compact 5->6 (1, 12 -> 13 tables with 4 splits). [36863 . 36700 36702 36160 36166 36153 36240 36245 36249 36242 36248 36276 36284 .] -> [37247 37263 37271 37243 37255 37258 37244 37261 37272 37248 37259 37275 37276 .], took 5.133s
I0125 12:59:19.739431       1 server.go:1147] Server._handleBlock: Received block ( 95433 / 98294 ) from Peer [ Remote Address: 34.123.41.111:17000 PeerID=1 ]
I0125 12:59:19.830921       1 block_producer.go:327] Produced block with 1 txns with approx 0 total txns in mempool
badger 2022/01/25 12:59:21 INFO: [2] [E] LOG Compact 5->6 (1, 10 -> 11 tables with 4 splits). [36884 . 36411 36419 36415 36420 36422 36416 36417 36310 36314 36302 .] -> [37281 37285 37292 37294 37279 37284 37288 37277 37283 37278 37280 .], took 3.168s
badger 2022/01/25 12:59:21 INFO: [0] [E] LOG Compact 5->6 (1, 10 -> 11 tables with 4 splits). [36891 . 37249 36131 36354 36359 36366 36353 36357 36362 36356 36361 .] -> [37282 37289 37299 37286 37295 37300 37287 37293 37302 37290 37291 .], took 3.204s
badger 2022/01/25 12:59:22 INFO: [3] [E] LOG Compact 5->6 (1, 9 -> 10 tables with 3 splits). [36850 . 35948 35951 35958 35959 35949 35954 35956 35947 35952 .] -> [37296 37301 37306 37308 37297 37303 37305 37298 37304 37311 .], took 2.982s
badger 2022/01/25 12:59:24 INFO: [2] [E] LOG Compact 5->6 (1, 11 -> 11 tables with 4 splits). [36869 . 37275 37276 36286 36278 36285 36290 36279 36452 36458 36455 36462 .] -> [37310 37313 37312 37323 37324 37309 37318 37322 37307 37315 37316 .], took 3.337s
badger 2022/01/25 12:59:25 INFO: [0] [E] LOG Compact 5->6 (1, 12 -> 12 tables with 4 splits). [36881 . 37252 37253 36190 36180 36184 36185 36176 36261 36265 36267 36266 37281 .] -> [37314 37317 37319 37334 37336 37320 37326 37332 37333 37321 37325 37327 .], took 3.874s
badger 2022/01/25 12:59:27 INFO: [3] [E] LOG Compact 5->6 (1, 7 -> 8 tables with 3 splits). [36858 . 35961 35965 35968 36696 36699 36698 37247 .] -> [37328 37335 37342 37329 37337 37344 37330 37331 .], took 4.825s
badger 2022/01/25 12:59:27 INFO: [2] [E] LOG Compact 5->6 (1, 13 -> 13 tables with 5 splits). [36887 . 36547 36554 36491 36494 36495 36482 36490 36492 36478 36488 36489 36238 37245 .] -> [37343 37348 37353 37339 37345 37349 37341 37352 37356 37340 37346 37347 37338 .], took 3.259s
I0125 12:59:27.990608       1 server.go:1147] Server._handleBlock: Received block ( 95434 / 98294 ) from Peer [ Remote Address: 34.123.41.111:17000 PeerID=1 ]
badger 2022/01/25 12:59:28 INFO: [0] [E] LOG Compact 5->6 (1, 12 -> 11 tables with 4 splits). [36875 . 37315 37316 36469 36453 36460 36464 36456 36457 36065 36066 36058 37242 .] -> [37355 37359 37350 37354 37357 37358 37362 37363 37351 37360 37361 .], took 3.085s
I0125 12:59:29.846159       1 block_producer.go:327] Produced block with 1 txns with approx 0 total txns in mempool
badger 2022/01/25 12:59:30 INFO: [3] [E] LOG Compact 5->6 (1, 11 -> 12 tables with 4 splits). [36856 . 37311 35955 35950 35953 35960 35964 35967 35963 35966 35969 37328 .] -> [37364 37373 37377 37366 37375 37381 37383 37368 37379 37384 37365 37367 .], took 2.908s
badger 2022/01/25 12:59:30 INFO: [2] [E] LOG Compact 5->6 (1, 12 -> 12 tables with 4 splits). [36885 . 37280 36307 36308 36301 36545 36551 36553 36546 36555 36556 36544 37343 .] -> [37369 37370 37371 37376 37386 37387 37372 37380 37385 37374 37378 37382 .], took 2.538s
I0125 12:59:31.067953       1 server.go:1147] Server._handleBlock: Received block ( 95435 / 98294 ) from Peer [ Remote Address: 34.123.41.111:17000 PeerID=1 ]
badger 2022/01/25 12:59:31 INFO: [1] [E] LOG Compact 0->5 (5, 206 -> 393 tables with 5 splits). [36843 36844 36845 36846 36847 . 35596 35624 35786 35858 35861 35862 35865 35868 35872 35875 35878 35879 35883 35886 35889 35893 35899 35908 35912 35916 35920 35924 35929 35936 35938 35939 35944 35945 35539 35545 35552 35560 35573 35579 35590 35599 35614 35619 35622 35627 35636 35655 35668 35706 35731 35740 35766 35779 35793 35823 35832 35841 35848 35857 35867 35874 35880 35884 35888 35892 35903 35906 35921 35925 35550 35559 35562 35568 35577 35584 35587 35600 35602 35607 35612 35621 35623 35630 35634 35639 35656 35661 35669 35672 35676 35678 35685 35686 35688 35691 35700 35703 35707 35709 35716 35718 35725 35727 35735 35744 35750 35755 35759 35763 35770 35776 35782 35789 35798 35805 35811 35818 35824 35556 35570 35583 35595 35608 35620 35631 35638 35651 35658 35682 35693 35704 35713 35719 35728 35733 35739 35745 35752 35761 35767 35774 35778 35783 35790 35796 35804 35809 35816 35821 35834 35838 35844 35850 35856 35864 35871 35876 35881 35885 35890 35897 35910 35918 35923 35930 35937 35940 35942 35546 35554 35565 35581 35592 35605 35617 35629 35640 35648 35657 35665 35677 35687 35696 35710 35723 35730 35732 35742 35746 35751 35757 35760 35764 35768 35771 35775 35777 35781 35784 35787 35795 35799 35801 35814 35820 35828 35831 35836 35842 35845 35927 .] -> [36850 36856 36858 36863 36869 36875 36877 36881 36884 36885 36887 36889 36891 36896 36898 36902 36905 36909 36912 36917 36925 36935 36940 36943 36946 36948 36951 36953 36956 36962 36967 36972 36975 36976 36978 36981 36984 36985 36986 36990 36992 36993 36995 36999 37001 37003 37006 37009 37011 37014 37015 37016 37017 37018 37022 37025 37028 37032 37036 37040 37043 37048 37053 37057 37063 37068 37074 37079 37083 37087 37096 37101 37107 37114 37119 37125 37130 37134 37139 37143 37148 37153 37159 37163 37170 37174 37180 37186 37190 37195 37201 37205 37209 37214 37217 37221 37225 37228 37231 37233 37235 37238 37239 37240 36851 36859 36866 36872 36878 36886 36894 36900 36907 36915 36921 36937 36944 36952 36957 36965 36971 36979 36987 36997 37005 37020 37029 37035 37039 37044 37045 37049 37051 37052 37054 37056 37058 37061 37064 37065 37067 37070 37072 37073 37077 37078 37081 37084 37086 37088 37091 37092 37093 37095 37099 37100 37104 37105 37106 37108 37111 37113 37116 37118 37121 37122 37124 37126 37129 37132 37133 37135 37137 37141 37142 37144 37145 37150 37151 37152 37154 37157 37160 37162 37164 37166 37169 37171 37172 37173 37177 37179 37181 37183 37185 37188 37189 37192 37196 37198 37200 37202 37203 37204 37206 37207 37208 37210 37211 37212 37213 37215 37216 37218 37219 37220 37222 37223 37224 37226 37227 37229 37230 37232 37234 37236 37237 36848 36849 36852 36854 36855 36861 36862 36864 36865 36870 36871 36873 36874 36876 36879 36882 36888 36890 36892 36897 36899 36901 36903 36904 36906 36908 36913 36914 36916 36919 36920 36923 36924 36927 36928 36929 36930 36931 36932 36933 36936 36939 36947 36955 36959 36963 36968 36974 36982 36989 36996 37004 37010 37021 37027 37034 37041 37050 37060 37071 37080 37090 37098 37109 37117 37128 37140 37149 37158 37167 37176 37184 37191 37193 36853 36860 36868 36880 36893 36910 36918 36926 36934 36941 36945 36949 36954 36958 36961 36966 36969 36973 36977 36983 36988 36994 37000 37007 37012 37024 37031 37038 37046 37059 37066 37076 37085 37094 37103 37112 37120 37127 37136 37147 37156 37165 37178 37187 37194 36857 36867 36883 36895 36911 36922 36938 36942 36950 36960 36964 36970 36980 36991 36998 37002 37008 37013 37019 37023 37026 37030 37033 37037 37042 37047 37055 37062 37069 37075 37082 37089 37097 37102 37110 37115 37123 37131 37138 37146 37155 37161 37168 37175 37182 37197 37199 .], took 1m2.101s
I0125 12:59:33.103860       1 server.go:1147] Server._handleBlock: Received block ( 95436 / 98294 ) from Peer [ Remote Address: 34.123.41.111:17000 PeerID=1 ]
I0125 12:59:37.675211       1 server.go:1147] Server._handleBlock: Received block ( 95437 / 98294 ) from Peer [ Remote Address: 34.123.41.111:17000 PeerID=1 ]
badger 2022/01/25 12:59:39 INFO: [2] [E] LOG Compact 5->6 (1, 11 -> 11 tables with 4 splits). [36986 . 36328 36329 36668 36675 36676 36667 36672 36674 36227 36229 36212 .] -> [37397 37420 37421 37393 37407 37412 37394 37409 37415 37401 37403 .], took 7.529s
badger 2022/01/25 12:59:39 INFO: [1] [E] LOG Compact 5->6 (1, 10 -> 10 tables with 4 splits). [36999 . 36243 36251 36270 36272 36250 36252 36163 36170 36142 36217 .] -> [37392 37406 37422 37423 37389 37391 37395 37416 37419 37388 .], took 7.786s
I0125 12:59:39.850975       1 block_producer.go:327] Produced block with 1 txns with approx 0 total txns in mempool
badger 2022/01/25 12:59:40 INFO: [3] [E] LOG Compact 5->6 (1, 8 -> 10 tables with 3 splits). [36978 . 36082 36536 36542 36534 36541 36552 36537 36827 .] -> [37399 37411 37424 37425 37404 37417 37426 37432 37398 37402 .], took 9.167s
badger 2022/01/25 12:59:41 INFO: [0] [E] LOG Compact 5->6 (1, 9 -> 11 tables with 3 splits). [36984 . 36289 36300 36317 36292 36350 36360 36365 36351 36364 .] -> [37390 37405 37413 37414 37396 37418 37428 37434 37400 37408 37410 .], took 9.726s

The disk and memory look fine. So, don't think its resources issue

Getting the below Deadlock error

I1021 12:48:25.391863       1 server.go:1131] Server._handleBlock: Received block ( 10484 / 71060 ) from Peer [ Remote Address: 35.232.92.5:17000 Peer
ID=1 ]                                                                                                                                                
I1021 12:48:32.586066       1 block_producer.go:327] Produced block with 1 txns with approx 0 total txns in mempool                                   
I1021 12:48:42.593449       1 block_producer.go:327] Produced block with 1 txns with approx 0 total txns in mempool                                   
I1021 12:48:44.700017       1 server.go:1131] Server._handleBlock: Received block ( 10485 / 71060 ) from Peer [ Remote Address: 35.232.92.5:17000 Peer
ID=1 ]                                                                                                                                                
I1021 12:48:52.613061       1 block_producer.go:327] Produced block with 1 txns with approx 0 total txns in mempool                                   
I1021 12:49:02.693992       1 block_producer.go:327] Produced block with 1 txns with approx 0 total txns in mempool                                   
I1021 12:59:27.471689       1 connection_manager.go:574] Rejecting INBOUND peer (65.21.93.96:40864) due to already having an inbound connection from t
he same IP with limit_one_inbound_connection_per_ip set.                                                                                              
E1021 12:59:27.477029       1 peer.go:1060] ReadDeSoMessage: : ReadMessage: Could not read MsgType: read tcp 185.69.166.144:39742->34.123.41.111:17000
: use of closed network connection                                                                                                                    
E1021 12:59:27.477092       1 peer.go:919] Peer.inHandler: Can't read message from peer [ Remote Address: 34.123.41.111:17000, DISCONNECTED PeerID=2 ]
: ReadDeSoMessage: : ReadMessage: Could not read MsgType: read tcp 185.69.166.144:39742->34.123.41.111:17000: use of closed network connection        
E1021 12:59:27.477046       1 peer.go:1060] ReadDeSoMessage: : ReadMessage: Could not read MsgType: read tcp 185.69.166.144:33458->46.4.89.216:17000: 
use of closed network connection                                                                                                                      
E1021 12:59:27.477204       1 peer.go:919] Peer.inHandler: Can't read message from peer [ Remote Address: 46.4.89.216:17000, DISCONNECTED PeerID=4 ]: 
ReadDeSoMessage: : ReadMessage: Could not read MsgType: read tcp 185.69.166.144:33458->46.4.89.216:17000: use of closed network connection            
E1021 15:54:49.651852       1 peer.go:1060] ReadDeSoMessage: : ReadMessage: Could not read MsgType: read tcp 185.69.166.144:17000->65.21.93.96:37920: 
use of closed network connection                                                                                                                      
badger 2021/10/21 15:54:49 INFO: Block cache metrics: hit: 5385222 miss: 973539 keys-added: 282663 keys-updated: 44 keys-evicted: 216295 cost-added: 1
136598192 cost-evicted: 868163136 sets-dropped: 0 sets-rejected: 688614 gets-dropped: 1856 gets-kept: 6356672 gets-total: 6358761 hit-ratio: 0.85     
POTENTIAL DEADLOCK:                                                                                                                                   
Previous place where the lock was grabbed                                                                                                             
goroutine 887651 lock 0xc29cb1aacc                                                                                                                    
E1021 15:54:49.990728       1 peer.go:788] Peer.outHandler: Peer [ Remote Address: 35.232.92.5:17000 PeerID=1 ] took too long to response to reqest. E
xpected MsgType=BLOCK at time 2021-10-21 13:02:55.341119554 +0000 UTC m=+2917.552453170 but it is now time 2021-10-21 13:03:29.916084911 +0000 UTC m=+
2952.127418529                                                                                                                                        
E1021 15:55:01.989426       1 peer.go:919] Peer.inHandler: Can't read message from peer [ Remote Address: 65.21.93.96:37920, DISCONNECTED PeerID=7 ]: 
ReadDeSoMessage: : ReadMessage: Could not read MsgType: read tcp 185.69.166.144:17000->65.21.93.96:37920: use of closed network connection            
deso/src/core/lib/peer.go:156 lib.(*Peer).MaybeDequeueDeSoMessage ??? <<<<<                                                                           
deso/src/core/lib/peer.go:418 lib.(*Peer).StartDeSoMessageProcessor ???                                                                               

Have been trying to lock it again for more than 10m0s
goroutine 2657040 lock 0xc29cb1aacc                                                                                                                   
                                                                                                                                                      
Here is what goroutine 887651 doing now                                                                                                               
goroutine 887651 [semacquire]:                                                                                                                        
sync.runtime_SemacquireMutex(0x488b45, 0xcc, 0xc00003e030)                                                                                            
        /usr/lib/go/src/runtime/sema.go:71 +0x25                                                                                                      
sync.(*Mutex).lockSlow(0xc0001260c0)                                                                                                                  
        /usr/lib/go/src/sync/mutex.go:138 +0x165                                                                                                      
sync.(*Mutex).Lock(...)                                                                                                                               
        /usr/lib/go/src/sync/mutex.go:81                                                                                                              
github.com/sasha-s/go-deadlock.(*lockOrder).postUnlock(0xc0001260c0, {0xfade20, 0xc29cb1aacc})
        /deso/src/core/third_party/github.com/sasha-s/go-deadlock/deadlock.go:332 +0x50
github.com/sasha-s/go-deadlock.postUnlock(...)
        /deso/src/core/third_party/github.com/sasha-s/go-deadlock/deadlock.go:169
github.com/sasha-s/go-deadlock.(*RWMutex).Unlock(0xc29cb1aacc)
        /deso/src/core/third_party/github.com/sasha-s/go-deadlock/deadlock.go:131 +0x47
github.com/deso-protocol/core/lib.(*Peer).MaybeDequeueDeSoMessage(0xc29cb1a900)
        /deso/src/core/lib/peer.go:161 +0x142
github.com/deso-protocol/core/lib.(*Peer).StartDeSoMessageProcessor(0xc29cb1a900)
        /deso/src/core/lib/peer.go:418 +0x8b
created by github.com/deso-protocol/core/lib.(*Peer).Start
        /deso/src/core/lib/peer.go:1017 +0x16f
Other goroutines holding locks:
goroutine 2335 lock 0xc0007a088c
deso/src/core/lib/peer.go:156 lib.(*Peer).MaybeDequeueDeSoMessage ??? <<<<< 
deso/src/core/lib/peer.go:418 lib.(*Peer).StartDeSoMessageProcessor ???
goroutine 58 lock 0xc0001612d0
deso/src/core/lib/blockchain.go:1643 lib.(*Blockchain).ProcessBlock ??? <<<<<
deso/src/core/lib/server.go:1177 lib.(*Server)._handleBlock ???
deso/src/core/lib/server.go:1480 lib.(*Server)._handlePeerMessages ???
deso/src/core/lib/server.go:1520 lib.(*Server).messageHandler ???

goroutine 54 lock 0xc0005342e0
deso/src/core/lib/mempool.go:1965 lib.(*DeSoMempool).RegenerateReadOnlyView ??? <<<<<
deso/src/core/lib/mempool.go:1924 lib.(*DeSoMempool).StartReadOnlyUtxoViewRegenerator.func1 ???

goroutine 291 lock 0xc00016e344
deso/src/core/lib/connection_manager.go:206 lib.(*ConnectionManager).getRandomAddr ??? <<<<<
deso/src/core/lib/connection_manager.go:304 lib.(*ConnectionManager)._getOutboundConn ???
deso/src/core/lib/connection_manager.go:389 lib.(*ConnectionManager).ConnectPeer ???

goroutine 16245 lock 0xc0541f240c
deso/src/core/lib/peer.go:156 lib.(*Peer).MaybeDequeueDeSoMessage ??? <<<<< 
deso/src/core/lib/peer.go:418 lib.(*Peer).StartDeSoMessageProcessor ???



POTENTIAL DEADLOCK:
Previous place where the lock was grabbed
goroutine 16245 lock 0xc0541f240c
deso/src/core/lib/peer.go:156 lib.(*Peer).MaybeDequeueDeSoMessage ??? <<<<< 
deso/src/core/lib/peer.go:418 lib.(*Peer).StartDeSoMessageProcessor ???
Have been trying to lock it again for more than 10m0s
goroutine 2657086 lock 0xc0541f240c

This PR fixes the following bug:

ERROR: unable to select packages:
  go-1.16.5-r0:
    breaks: world[go=1.16.4-r0]

The core repo also uses go-1.16.5-r0. I think that's where the issue comes from.

it seems that the third_party directory has been removed from the main branch of core

Step 10/22 : COPY core/third_party/ core/third_party/ COPY failed: file not found in build context or excluded by .dockerignore: stat core/third_party/: file does not exist

Getting the below error when running docker build -t mongodb-dumper -f Dockerfile ..

go: finding module for package github.com/deso-protocol/core/migrate
../core/cmd/node.go:13:2: module github.com/deso-protocol/core@latest found (v1.2.1, replaced by ../core/), but does not contain package github.com/deso-protocol/core/migrate

I ran mongo-dumper thrice.

In all the three times, it gets stuck at around block 7000 and then stops.

First it got stuck at block 7818. The 2nd time it got stuck at block 7658.

Here is the screenshot, t

Getting this error Error: Invalid UTF-8 string in BSON document when I try to access the db by running
deso> db.data.find( ).limit(1)

How to ignore the error and still read the db?

Attaching the screenshot of the error

What is up with the transaction ids like AtKc7wqJuLy/VXGIR1MxVo46zOGrML+HwFDmiTv137Op instead of 3JuETz96LrotPgv9db7awdfENVW4p149HgEjQ33wtNxM7o9gLF8Q53?

Looking at this block, its first transaction should be 3JuETz96LrotPgv9db7awdfENVW4p149HgEjQ33wtNxM7o9gLF8Q53 but it appears as AtKc7wqJuLy/VXGIR1MxVo46zOGrML+HwFDmiTv137Op when I do the query:

db.data.findOne({ "BadgerKeyPrefix": "_PrefixBlockHashToBlock:0", "BlockHash": "000000000000bb022da16a8b0990995b05431cc4ddbd9977278cec5dea3d1e78"}, {"Txns" : { $slice: 1 } })

I've noticed when I try and use the badgerdb directory from my live running backend, it complains about a .mem file.

The only solution I've found is to copy -r the full directory somewhere else and delete that .mem file and then it works fine.

But I've noticed this mongodb-dumper seems to avoid this issue. Is there a trick to say ignore .mem file?

This PR adds a section on how to setup a MongoDB server in a separate docker container.

Why is it necessary? Initially, I thought the repo would setup a mongodb for me. This is not the case. When I tried to setup a mongodb server myself I ran into a connection issue. I looked for raised issues in the repo but they didn't solve my problem. In particular, the following closed issue [link] (https://github.com/bitclout/mongodb-dumper/issues/3) was misleading and I wasted a couple of hours. I hope this PR saves people some time.

I want the user interaction data like who is following whom or we can say how the network is interacting as I wish to do some network analysis on that data So if anyone can help me with this part ping me at Bitclout: @Phigex or just reply to this thread