The MongoDB driver for Go

Build Status GoDoc

The MongoDB driver for Go

This fork has had a few improvements by ourselves as well as several PR's merged from the original mgo repo that are currently awaiting review. Changes are mostly geared towards performance improvements and bug fixes, though a few new features have been added.

Further PR's (with tests) are welcome, but please maintain backwards compatibility.

Detailed documentation of the API is available at GoDoc.

A sub-package that implements the BSON specification is also included, and may be used independently of the driver.

Supported Versions

mgo is known to work well on (and has integration tests against) MongoDB v3.0, 3.2, 3.4 and 3.6.

MongoDB 4.0 is currently experimental - we would happily accept PRs to help improve support!

Changes

  • Fixes attempting to authenticate before every query (details)
  • Removes bulk update / delete batch size limitations (details)
  • Adds native support for time.Duration marshalling (details)
  • Reduce memory footprint / garbage collection pressure by reusing buffers (details, more)
  • Support majority read concerns (details)
  • Improved connection handling (details)
  • Hides SASL warnings (details)
  • Support for partial indexes (details)
  • Fixes timezone handling (details)
  • Integration tests run against MongoDB 3.2 & 3.4 releases (details, more, more)
  • Improved multi-document transaction performance (details, more, more)
  • Fixes cursor timeouts (details)
  • Support index hints and timeouts for count queries (details)
  • Don't panic when handling indexed int64 fields (details)
  • Supports dropping all indexes on a collection (details)
  • Annotates log entries/profiler output with optional appName on 3.4+ (details)
  • Support for read-only views in 3.4+ (details)
  • Support for collations in 3.4+ (details, more)
  • Provide BSON constants for convenience/sanity (details)
  • Consistently unmarshal time.Time values as UTC (details)
  • Enforces best practise coding guidelines (details)
  • GetBSON correctly handles structs with both fields and pointers (details)
  • Improved bson.Raw unmarshalling performance (details)
  • Minimise socket connection timeouts due to excessive locking (details)
  • Natively support X509 client authentication (details)
  • Gracefully recover from a temporarily unreachable server (details)
  • Use JSON tags when no explicit BSON are tags set (details)
  • Support $changeStream tailing on 3.6+ (details)
  • Fix deadlock in cluster synchronisation (details)
  • Implement maxIdleTimeout for pooled connections (details)
  • Connection pool waiting improvements (details)
  • Fixes BSON encoding for $in and friends (details)
  • Add BSON stream encoders (details)
  • Add integer map key support in the BSON encoder (details)
  • Support aggregation collations (details)
  • Support encoding of inline struct references (details)
  • Improved windows test harness (details)
  • Improved type and nil handling in the BSON codec (details, more)
  • Separated network read/write timeouts (details)
  • Expanded dial string configuration options (details)
  • Implement MongoTimestamp (details)
  • Support setting writeConcern for findAndModify operations (details)
  • Add ssl to the dial string options (details)

Thanks to

  • @aksentyev
  • @bachue
  • @bozaro
  • @BenLubar
  • @carldunham
  • @carter2000
  • @cedric-cordenier
  • @cezarsa
  • @DaytonG
  • @ddspog
  • @drichelson
  • @dvic
  • @eaglerayp
  • @feliixx
  • @fmpwizard
  • @gazoon
  • @gedge
  • @gnawux
  • @idy
  • @jameinel
  • @jefferickson
  • @johnlawsharrison
  • @KJTsanaktsidis
  • @larrycinnabar
  • @mapete94
  • @maxnoel
  • @mcspring
  • @Mei-Zhao
  • @peterdeka
  • @Reenjii
  • @roobre
  • @smoya
  • @steve-gray
  • @tbruyelle
  • @wgallagher
Comments
  • Add switch for `omitempty` tag

    Add switch for `omitempty` tag

    Currently we have bson.SetJSONTagFallback to fallback to json tag when no bson tag available, which is pretty useful when using protobuf with mgo, leave generated *.pb.go untouched and get a consistent field name between json & bson.

    With the default omitempty tag generated by protobuf, updated struct fields with empty values will not be encoded, leave the database untouched. Adding a switch for ignoring omitempty will gain another benefit that we can keep protobuf performant and make updating operations easier.

  • Add signaling support for connection pool waiting

    Add signaling support for connection pool waiting

    The current behaviour when the poolLimit is reached and a new connection is required is to poll every 100ms to see if there is now headroom to make a new connection. This adds tremendous latency to the limit-hit-path.

    This commit changes the checkout behaviour to watch on a condition variable for connections to become available, and the checkin behaviour to signal this variable. This should allow waiters to use connections immediately after they become available.

    A new parameter is also added to DialInfo, PoolTimeout, which is the maximum time that clients will wait for connection headroom to become available. By default this is unlimited.

    Full disclosure - I haven't yet actually tried this in our production environment yet. The only testing this has so far is that it passes the regression tests. I'm pull-requesting this early to get some feedback on the design of this change and to see if this would be useful for other people :). I intend to try and get this into production sometime in the next couple of weeks.

  • read tcp i/o timeout on Windows 10

    read tcp i/o timeout on Windows 10

    After some db calls, we have to restart our Go backend:

    Message: read tcp 127.0.0.1:50743->127.0.0.1:27017: i/o timeout

    MongoDB 3.4.10 Go 1.9.2 windows/amd64 github.com/globalsign/mgo 5be15cc6a44937b6937a9487e1633f8a5fb25e4a

    I think it was ok with: MongoDB 3.2.16 github.com/globalsign/mgo c4a7121bd9f6f7246d9f54b78bdd2eb4d18b4c76 Go 1.9.1 Alpine 3.6

  • MongoDB 3.6: implement the new wire protocol

    MongoDB 3.6: implement the new wire protocol

    Mongodb 3.6: implement the new wire protocol

    WIP - Do not merge

    Goal

    This PR is an attempt to implement the new wire protocol introduced in Mongodb 3.6. This protocol add a new message format: OP_MSG

    OP_MSG is a bi-directional wire protocol opcode introduced in MongoDB 3.6 with the goal of replacing most existing opcodes, merging their use into one extendable opcode.

    Tests currently run against MongoDB 3.6-rc3 (first 3.6 stable version should be released next month) I would like to have some feedbacks on this before going any further !

    What's in this PR

    this PR contains several commits, first 6 are small update to make test pass against last version of each major serie, and the last one is the new wire protocol implementation

    • commit c1fc70d3d748fb4dd0ebb851b1a8d9a908e053fa test against 3.6-rc3

    • commit a594e559227a1301f8aeecb5895cd00127cff502 update mongodb version use last minor version for each major serie

    • commit f6a8010534991d83d812e235a6fc91f5bb801699 test fixes on 3.2.17

    • commit 8242b9c0ab4762bbbb3e6865c91c7ed888a9ae7f fix X509 test fix TestAuthX509CredRDNConstruction test: need to create an user with {"username": subject}
      before trying to login

    • commit 3365de912f1ba8ebbe2372b8d8646c097d927105 Fix auth test on 3.6-rc3

      Make sure that "rs3/127.0.0.1/40031" is elected at primary. Create user before running 'addShard' command as it requires root access Also add a retry mechanism to make sure that shard are correctly added cf https://docs.mongodb.com/manual/tutorial/deploy-shard-cluster/

    • commit 17833465551a225e567eb2250c2759354260c96d re-enable TestViewWithCollation SERVER-31049 was fixed in 3.4.10, so re-enable it in tests

    • commit 6a815bbddf6a20ab1e1ddc981cebeb5045e59946 start implementing new wire protocol

    TODO

    • [x] fix last 9 failing Test:
      • [x] TestDirect
      • [x] TestBulkUpdateErrorUnordered
      • [x] TestAuthLoginLogout
      • [x] TestAuthLoginLogoutAll
      • [x] TestAuthLoginChangePassword
      • [x] TestAuthUpsertUser
      • [x] TestAuthAddUser
      • [x] TestAuthLoginCachingAcrossPoolWithLogout
      • [x] TestTopologySyncWithSlaveSeed
    • [x] add possibility to set writeConcern
    • [x] make sure that bulk op doesn't contain more than 100000 docs
    • [x] enable/disable OP_MSG with an param in connection URI
  • test killed after 10min on travis with docker mongo

    test killed after 10min on travis with docker mongo

    Hi,

    Every once in a while our mongo suite gets killed on TravicCI. We run go 1.10 and use docker for our test suites. Our Postgres and Neo4j test suites run just fine with this setup but with mgo and Mongo we're having these issues.

    Stacktrace information can be found below. Any idea why this is happening?

    +go test -v -race -coverprofile=coverage.out -covermode=atomic ./...
    === RUN   TestMongoSuiteWithoutCredentials
    2018/03/04 13:50:01 CREATING NEW POOL
    2018/03/04 13:50:01 POOL CREATED <nil>
    2018/03/04 13:50:01 RUNNING MONGO CONTAINER
    2018/03/04 13:50:11 MONGO CONTAINER CREATED <nil>
    2018/03/04 13:50:11 BEFORE testConnect
    2018/03/04 13:50:11 START DialWithTimeout
    2018/03/04 13:50:11 MONGO URL = mongodb://localhost:32768
    SIGQUIT: quit
    PC=0x474643 m=0 sigcode=0
    
    goroutine 31 [syscall]:
    runtime.notetsleepg(0x12be9e0, 0x37e09133e, 0x16)
    	/home/travis/.gimme/versions/go1.10.linux.amd64/src/runtime/lock_futex.go:227 +0x42 fp=0xc420052760 sp=0xc420052730 pc=0x422022
    runtime.timerproc(0x12be9c0)
    	/home/travis/.gimme/versions/go1.10.linux.amd64/src/runtime/time.go:261 +0x2f9 fp=0xc4200527d8 sp=0xc420052760 pc=0x461889
    runtime.goexit()
    	/home/travis/.gimme/versions/go1.10.linux.amd64/src/runtime/asm_amd64.s:2361 +0x1 fp=0xc4200527e0 sp=0xc4200527d8 pc=0x472bd1
    created by runtime.(*timersBucket).addtimerLocked
    	/home/travis/.gimme/versions/go1.10.linux.amd64/src/runtime/time.go:160 +0x107
    
    goroutine 1 [chan receive]:
    testing.(*T).Run(0xc42021c000, 0xd2d7a8, 0x20, 0xd41b80, 0xc4201e5c00)
    	/home/travis/.gimme/versions/go1.10.linux.amd64/src/testing/testing.go:825 +0x597
    testing.runTests.func1(0xc42021c000)
    	/home/travis/.gimme/versions/go1.10.linux.amd64/src/testing/testing.go:1063 +0xa5
    testing.tRunner(0xc42021c000, 0xc4201e5d48)
    	/home/travis/.gimme/versions/go1.10.linux.amd64/src/testing/testing.go:777 +0x16e
    testing.runTests(0xc4201378e0, 0x127b3e0, 0x1, 0x1, 0xc420160800)
    	/home/travis/.gimme/versions/go1.10.linux.amd64/src/testing/testing.go:1061 +0x4e2
    testing.(*M).Run(0xc420160800, 0x0)
    	/home/travis/.gimme/versions/go1.10.linux.amd64/src/testing/testing.go:978 +0x2ce
    main.main()
    	_testmain.go:90 +0x325
    
    goroutine 19 [syscall]:
    os/signal.signal_recv(0x472bd1)
    	/home/travis/.gimme/versions/go1.10.linux.amd64/src/runtime/sigqueue.go:139 +0xa6
    os/signal.loop()
    	/home/travis/.gimme/versions/go1.10.linux.amd64/src/os/signal/signal_unix.go:22 +0x30
    created by os/signal.init.0
    	/home/travis/.gimme/versions/go1.10.linux.amd64/src/os/signal/signal_unix.go:28 +0x4f
    
    goroutine 20 [semacquire]:
    sync.runtime_notifyListWait(0xc42023a6e8, 0xc400000000)
    	/home/travis/.gimme/versions/go1.10.linux.amd64/src/runtime/sema.go:510 +0x11a
    sync.(*Cond).Wait(0xc42023a6d8)
    	/home/travis/.gimme/versions/go1.10.linux.amd64/src/sync/cond.go:56 +0x8e
    github.com/globalsign/mgo.(*mongoCluster).AcquireSocket(0xc42023a6c0, 0x0, 0xc420240a01, 0x6fc23ac00, 0x6fc23ac00, 0x0, 0x0, 0x0, 0x1000, 0x1c5b320, ...)
    	/home/travis/gopath/src/github.com/globalsign/mgo/cluster.go:644 +0xff
    github.com/globalsign/mgo.(*Session).acquireSocket(0xc4202409c0, 0xb9e201, 0x0, 0x0, 0x0)
    	/home/travis/gopath/src/github.com/globalsign/mgo/session.go:4853 +0x271
    github.com/globalsign/mgo.(*Database).Run(0xc42017bc20, 0xc2ee40, 0xda60b0, 0x0, 0x0, 0x0, 0x0)
    	/home/travis/gopath/src/github.com/globalsign/mgo/session.go:799 +0x5e
    github.com/globalsign/mgo.(*Session).Run(0xc4202409c0, 0xc2ee40, 0xda60b0, 0x0, 0x0, 0xcf84e0, 0xc42023a6c0)
    	/home/travis/gopath/src/github.com/globalsign/mgo/session.go:2270 +0xba
    github.com/globalsign/mgo.(*Session).Ping(0xc4202409c0, 0xc42023a6c0, 0x6fc23ac00)
    	/home/travis/gopath/src/github.com/globalsign/mgo/session.go:2299 +0x5d
    github.com/globalsign/mgo.DialWithInfo(0xc4202c0000, 0x17, 0xc4202c0000, 0x0)
    	/home/travis/gopath/src/github.com/globalsign/mgo/session.go:563 +0x566
    github.com/globalsign/mgo.DialWithTimeout(0xc420026d20, 0x17, 0x6fc23ac00, 0x0, 0xc420167780, 0xc4200b0120)
    	/home/travis/gopath/src/github.com/globalsign/mgo/session.go:304 +0xc3
    mongo_test.(*suite).testConnect(0xc42017bf48, 0xc42021c0f0)
    	/home/travis/build/qdentity/qdentity/go/src/mongo/mongo_test.go:36 +0xc8
    mongo_test.TestMongoSuiteWithoutCredentials(0xc42021c0f0)
    	/home/travis/build/qdentity/qdentity/go/src/mongo/mongo_test.go:22 +0x187
    testing.tRunner(0xc42021c0f0, 0xd41b80)
    	/home/travis/.gimme/versions/go1.10.linux.amd64/src/testing/testing.go:777 +0x16e
    created by testing.(*T).Run
    	/home/travis/.gimme/versions/go1.10.linux.amd64/src/testing/testing.go:824 +0x565
    
    goroutine 30 [semacquire]:
    sync.runtime_notifyListWait(0xc42023a6e8, 0xc400000001)
    	/home/travis/.gimme/versions/go1.10.linux.amd64/src/runtime/sema.go:510 +0x11a
    sync.(*Cond).Wait(0xc42023a6d8)
    	/home/travis/.gimme/versions/go1.10.linux.amd64/src/sync/cond.go:56 +0x8e
    github.com/globalsign/mgo.(*mongoCluster).AcquireSocket(0xc42023a6c0, 0x1, 0xc420240b01, 0x2540be400, 0x2540be400, 0x0, 0x0, 0x0, 0x1000, 0xc420082700, ...)
    	/home/travis/gopath/src/github.com/globalsign/mgo/cluster.go:644 +0xff
    github.com/globalsign/mgo.(*Session).acquireSocket(0xc420240b60, 0xc5f001, 0x0, 0x0, 0x0)
    	/home/travis/gopath/src/github.com/globalsign/mgo/session.go:4853 +0x271
    github.com/globalsign/mgo.(*Database).Run(0xc4200779b8, 0xc5f0c0, 0xc42000d200, 0xc10ec0, 0xc420232630, 0x0, 0x0)
    	/home/travis/gopath/src/github.com/globalsign/mgo/session.go:799 +0x5e
    github.com/globalsign/mgo.(*Session).Run(0xc420240b60, 0xc5f0c0, 0xc42000d200, 0xc10ec0, 0xc420232630, 0x0, 0x1)
    	/home/travis/gopath/src/github.com/globalsign/mgo/session.go:2270 +0xba
    github.com/globalsign/mgo.(*mongoCluster).isMaster(0xc42023a6c0, 0xc4202c20f0, 0xc420232630, 0xc4202c20f0, 0x0)
    	/home/travis/gopath/src/github.com/globalsign/mgo/cluster.go:182 +0x258
    github.com/globalsign/mgo.(*mongoCluster).syncServer(0xc42023a6c0, 0xc4202c00e0, 0xd, 0xc42001ed20, 0xc4202c00e0, 0xc42023a6c0, 0xc440000000, 0x0)
    	/home/travis/gopath/src/github.com/globalsign/mgo/cluster.go:231 +0x434
    github.com/globalsign/mgo.(*mongoCluster).syncServersIteration.func1.1(0xc420292060, 0xc420026d2a, 0xd, 0xc420292070, 0xc420026d00, 0xc4202867b0, 0xc42023a6c0, 0xc4202867e0, 0xc420286810, 0x0, ...)
    	/home/travis/gopath/src/github.com/globalsign/mgo/cluster.go:553 +0x1fb
    created by github.com/globalsign/mgo.(*mongoCluster).syncServersIteration.func1
    	/home/travis/gopath/src/github.com/globalsign/mgo/cluster.go:525 +0x175
    
    goroutine 11 [semacquire]:
    sync.runtime_Semacquire(0xc42029206c)
    	/home/travis/.gimme/versions/go1.10.linux.amd64/src/runtime/sema.go:56 +0x39
    sync.(*WaitGroup).Wait(0xc420292060)
    	/home/travis/.gimme/versions/go1.10.linux.amd64/src/sync/waitgroup.go:129 +0xb3
    github.com/globalsign/mgo.(*mongoCluster).syncServersIteration(0xc42023a6c0, 0x0)
    	/home/travis/gopath/src/github.com/globalsign/mgo/cluster.go:582 +0x4c5
    github.com/globalsign/mgo.(*mongoCluster).syncServersLoop(0xc42023a6c0)
    	/home/travis/gopath/src/github.com/globalsign/mgo/cluster.go:390 +0x17c
    created by github.com/globalsign/mgo.newCluster
    	/home/travis/gopath/src/github.com/globalsign/mgo/cluster.go:81 +0x2e3
    
    goroutine 12 [sleep]:
    time.Sleep(0x37e11d600)
    	/home/travis/.gimme/versions/go1.10.linux.amd64/src/runtime/time.go:102 +0x146
    github.com/globalsign/mgo.(*mongoServer).pinger(0xc4202c00e0, 0x479801)
    	/home/travis/gopath/src/github.com/globalsign/mgo/server.go:314 +0x7ad
    created by github.com/globalsign/mgo.newServer
    	/home/travis/gopath/src/github.com/globalsign/mgo/server.go:89 +0x24b
    
    goroutine 34 [IO wait]:
    internal/poll.runtime_pollWait(0x7f50f3494f00, 0x72, 0x128aff0)
    	/home/travis/.gimme/versions/go1.10.linux.amd64/src/runtime/netpoll.go:173 +0x5e
    internal/poll.(*pollDesc).wait(0xc420234e18, 0x72, 0xda9f00, 0x128aff0, 0xffffffffffffffff)
    	/home/travis/.gimme/versions/go1.10.linux.amd64/src/internal/poll/fd_poll_runtime.go:85 +0xe5
    internal/poll.(*pollDesc).waitRead(0xc420234e18, 0xc420028800, 0x24, 0x24)
    	/home/travis/.gimme/versions/go1.10.linux.amd64/src/internal/poll/fd_poll_runtime.go:90 +0x4b
    internal/poll.(*FD).Read(0xc420234e00, 0xc420028840, 0x24, 0x24, 0x0, 0x0, 0x0)
    	/home/travis/.gimme/versions/go1.10.linux.amd64/src/internal/poll/fd_unix.go:157 +0x22a
    net.(*netFD).Read(0xc420234e00, 0xc420028840, 0x24, 0x24, 0x4ab9ed, 0xc420234e00, 0x0)
    	/home/travis/.gimme/versions/go1.10.linux.amd64/src/net/fd_unix.go:202 +0x66
    net.(*conn).Read(0xc42000e0c8, 0xc420028840, 0x24, 0x24, 0x0, 0xc4202c24b0, 0xc420062dc0)
    	/home/travis/.gimme/versions/go1.10.linux.amd64/src/net/net.go:176 +0x85
    github.com/globalsign/mgo.fill(0xdb3660, 0xc42000e0c8, 0xc420028840, 0x24, 0x24, 0x0, 0x11)
    	/home/travis/gopath/src/github.com/globalsign/mgo/socket.go:567 +0x64
    github.com/globalsign/mgo.(*mongoSocket).readLoop(0xc4202c24b0)
    	/home/travis/gopath/src/github.com/globalsign/mgo/socket.go:583 +0x15b
    created by github.com/globalsign/mgo.newSocket
    	/home/travis/gopath/src/github.com/globalsign/mgo/socket.go:197 +0x341
    
    rax    0xfffffffffffffffc
    rbx    0x12bb3a0
    rcx    0x474643
    rdx    0x0
    rdi    0x12be9e0
    rsi    0x0
    rbp    0xc4200526e8
    rsp    0xc420052698
    r8     0x0
    r9     0x0
    r10    0xc4200526d8
    r11    0x202
    r12    0xc420079c80
    r13    0x12bb3a0
    r14    0xc420001500
    r15    0x1a354620
    rip    0x474643
    rflags 0x202
    cs     0x33
    fs     0x0
    gs     0x0
    *** Test killed with quit: ran too long (10m0s).
    FAIL	mongo	600.006s
    
  • Data race for socket

    Data race for socket

    Hello! I got race condition randomly. Here is stack:

    WARNING: DATA RACE
    Read at 0x00c4201381b8 by goroutine 94:
      github.com/globalsign/mgo.(*Session).acquireSocket()
          /.../github.com/globalsign/mgo/session.go:4822 +0x768
      github.com/globalsign/mgo.(*Database).Run()
          /.../github.com/globalsign/mgo/session.go:799 +0x6d
      github.com/globalsign/mgo.(*Session).Run()
          /.../github.com/globalsign/mgo/session.go:2270 +0xc3
      github.com/globalsign/mgo.(*mongoCluster).isMaster()
          /.../github.com/globalsign/mgo/cluster.go:182 +0x27c
      github.com/globalsign/mgo.(*mongoCluster).syncServer()
          /.../github.com/globalsign/mgo/cluster.go:231 +0x457
      github.com/globalsign/mgo.(*mongoCluster).syncServersIteration.func1.1()
          /.../github.com/globalsign/mgo/cluster.go:553 +0x200
    
    Previous write at 0x00c4201381b8 by goroutine 33:
      github.com/globalsign/mgo.(*mongoSocket).kill()
          /.../github.com/globalsign/mgo/socket.go:351 +0x251
      github.com/globalsign/mgo.(*mongoSocket).resetNonce.func1()
          /.../github.com/globalsign/mgo/auth.go:123 +0x135
      github.com/globalsign/mgo.(*mongoSocket).readLoop()
          /.../github.com/globalsign/mgo/socket.go:628 +0x1101
    
    Goroutine 94 (running) created at:
      github.com/globalsign/mgo.(*mongoCluster).syncServersIteration.func1()
          /.../github.com/globalsign/mgo/cluster.go:525 +0x174
      github.com/globalsign/mgo.(*mongoCluster).syncServersIteration()
          /.../github.com/globalsign/mgo/cluster.go:580 +0x54b
      github.com/globalsign/mgo.(*mongoCluster).syncServersLoop()
          /.../github.com/globalsign/mgo/cluster.go:390 +0x18e
    
    Goroutine 33 (running) created at:
      github.com/globalsign/mgo.newSocket()
          /.../github.com/globalsign/mgo/socket.go:197 +0x392
      github.com/globalsign/mgo.(*mongoServer).Connect()
          /.../github.com/globalsign/mgo/server.go:183 +0x60c
      github.com/globalsign/mgo.(*mongoServer).AcquireSocket()
          /.../github.com/globalsign/mgo/server.go:129 +0x349
      github.com/globalsign/mgo.(*mongoCluster).syncServer()
          /.../github.com/globalsign/mgo/cluster.go:225 +0x2a5
      github.com/globalsign/mgo.(*mongoCluster).syncServersIteration.func1.1()
          /.../github.com/globalsign/mgo/cluster.go:553 +0x200
    ==================
    

    I can't reproduce it. But I believe, that we need to Lock mutex for s.masterSocket on /.../globalsign/mgo/session.go:4822 before read dead property. Am I right?

    Му OS is Ubuntu 16.04 and GO version is 1.9.2 linux/amd64.

    Thank you in advance!

  • Partial 4.0 Transaction support

    Partial 4.0 Transaction support

    Third time's a charm! I think we did it right this time.

    This time, I pulled the latest version of development and ported my changes into that. This contains the following changes:

    1. Partial 4.0 transaction support. This supports the basic Update, Upsert, Remove, and Insert support. Batch support is not included (but could probably easily be added - it's not a use case we needed at the moment).
    2. dbtest supports creating a replica set now. You have to explicitly enable that.
    3. A few other minor bugs and issues fixed.

    Documentation is included in the repo.

    All changes are, to the best of my knowledge, backwards compatible. All tests pass for me with the exception of some clustering tests that may be related to 4.0 but are not related to my changes to the best of my knowledge.

    We at The Home Depot are excited to be sharing these changes with the community, please let me know if there are further issues that need to be addressed.

  • Changestreams db and cluster level #251

    Changestreams db and cluster level #251

    Enhancing change streams in order to support DB level and cluster level changes notifications introduced in mongoDB 4.0 as per issue #251. I strongly ask review of the way i addressed the question as i'm not really sure it is not a workaround. If we go this way, i think it's better to make unexported all session and database methods that were added (Pipe(), NewIter()), except for the Watch() method in order to not pollute with meaningless methods. Tests are still missing and will be added once we agree on the way i implemented the functionality and Mongo 4.0 is added to the test suite.

  • cross operation on multi bulks result in additional records inserts

    cross operation on multi bulks result in additional records inserts

    say we have two bulk variables defined from different collections named "Bulk1" and "Bulk2". After running following code, collection "Bulk1" get 2 document inserted, but collection "Bulk2" also get 2 documents inserted! the reason for this is because actionPool defined in bulk.go is not safe in this situation. i just made a pull request to resolve this issue, hope it will get accepted, thanks

    var blk = DB().C("Bulk1").Bulk()
    var blk2 = DB().C("Bulk2").Bulk()
    for i := 0; i < 2; i++ {
    	blk.Insert(bson.M{})
    	if i==0{
    		blk.Run()
    	}
    }
    blk2.Insert(bson.M{})
    blk.Run()
    blk2.Run()`
    
  • Change session.DatabaseNames to use the `nameOnly` option (if available)

    Change session.DatabaseNames to use the `nameOnly` option (if available)

    listDatabases by default returns the list of database names as well as some stats about the database (size, etc.). To gather the stats mongod acquires a db level lock, which is very slow (and depending on other locks etc. may cause this to unnecessarily fail).

    Perf comparison:

    basic took 2.772775182s
    nameonly took 93.680166ms
    

    I have also tested that this additional option falls back to the previous behavior on mongo <3.6

    Mongo docs: https://docs.mongodb.com/manual/reference/command/listDatabases/

  • support MongoDB 4.0 startAtOperationTime option

    support MongoDB 4.0 startAtOperationTime option

    I wasn't sure how to write new tests for this. But this is an attempt to support the new option in MongoDB 4.0 to start the change stream from a specific Timestamp in the oplog.

  • BSON unmarshal unable to parse $date

    BSON unmarshal unable to parse $date

    We use the issue tracker to track bugs with mgo - if you have a usage question, it's best to try Stack Overflow :)

    Replace this text with your description, and please answer the questions below before submitting your issue to help us out. Thanks!


    What version of MongoDB are you using (mongod --version)?

    db version v6.0.1
    Build Info: {
        "version": "6.0.1",
        "gitVersion": "32f0f9c88dc44a2c8073a5bd47cf779d4bfdee6b",
        "modules": [],
        "allocator": "system",
        "environment": {
            "distarch": "aarch64",
            "target_arch": "aarch64"
        }
    }
    

    What version of Go are you using (go version)?

    go1.19
    

    What did you do?

    If possible, provide a recipe for reproducing the error. A runnable program is great and really helps!

    var tsMap map[string]interface{}
    	bsonData := []byte(`
    {"_id": {"$numberLong": "10004"},"order_date": {"$date": 1456012800000},"purchaser_id": {"$numberLong": "1003"},"quantity": 1,"product_id": {"$numberLong": "107"}}
    
    `)
    err := bson.UnmarshalJSON(bsonData, &tsMap)
    
    assert.NoError(p.T(), err)
    

    Instead of it working, it returned an error, see: err cannot parse date: "{\"$date\": 1456012800000}"

    Can you reproduce the issue on the latest development branch?

  • current

    current

    We use the issue tracker to track bugs with mgo - if you have a usage question, it's best to try Stack Overflow :)

    Replace this text with your description, and please answer the questions below before submitting your issue to help us out. Thanks!


    What version of MongoDB are you using (mongod --version)?

    <mongodb version here>
    

    What version of Go are you using (go version)?

    <go version here>
    

    What operating system and processor architecture are you using (go env)?

    <go env here>
    

    What did you do?

    If possible, provide a recipe for reproducing the error. A runnable program is great and really helps!

    Can you reproduce the issue on the latest development branch?

  • Panic With ObjectIdHex(s string)

    Panic With ObjectIdHex(s string)

    It is very danger with mgo/bson/bson.go.212,see code; if anyone else who wanted to change request args to mongoDB objectID, the code maybe like this:

    objID := bson.ID
    if IsObjectIdHex(req.ObjectID) {
        objID = ObjectIdHex(req.ObjectID)
    }else {
        // TODO do else logic
    }
    

    And Why not add like this:

    if objID,err := MaybeObjectIdHex(req.ObjectID);err != nil {
        // TODO add return logic
        return
    }
    
        // TODO do correct logic
    

    Any way, panic in runtime is very unfriendly for every programmer.

  • Fully automate dev setup with Gitpod

    Fully automate dev setup with Gitpod

    This commit implements a fully-automated development setup using Gitpod.io, an online IDE for GitLab, GitHub, and Bitbucket that enables Dev-Environments-As-Code. This makes it easy for anyone to get a ready-to-code workspace for any branch, issue or pull request almost instantly with a single click.

Go-driver-benchmark - Driver benchmark with golang

We use ClickkHouse for time-series databases, and the driver's performance is ve

Sep 5, 2022
A tool to run queries in defined frequency and expose the count as prometheus metrics. Supports MongoDB and SQL
A tool to run queries in defined frequency and expose the count as prometheus metrics. Supports MongoDB and SQL

query2metric A tool to run db queries in defined frequency and expose the count as prometheus metrics. Why ? Product metrics play an important role in

Jul 1, 2022
Bifrost ---- 面向生产环境的 MySQL 同步到Redis,MongoDB,ClickHouse,MySQL等服务的异构中间件
Bifrost ---- 面向生产环境的 MySQL 同步到Redis,MongoDB,ClickHouse,MySQL等服务的异构中间件

Bifrost ---- 面向生产环境的 MySQL 同步到Redis,ClickHouse等服务的异构中间件 English 漫威里的彩虹桥可以将 雷神 送到 阿斯加德 和 地球 而这个 Bifrost 可以将 你 MySQL 里的数据 全量 , 实时的同步到 : Redis MongoDB Cl

Dec 30, 2022
WAL-G is an archival restoration tool for PostgreSQL, MySQL/MariaDB, and MS SQL Server (beta for MongoDB and Redis).

WAL-G is an archival restoration tool for PostgreSQL, MySQL/MariaDB, and MS SQL Server (beta for MongoDB and Redis).

Jan 1, 2023
Simple instagram backend api built using golang and mongodb
Simple instagram backend api built using golang and mongodb

Insta_api using golang and mongodb Proper hashing algorithms have been implemented so that user password cannot be reverse engineered. Features: User

Apr 9, 2022
A truly Open Source MongoDB alternative

FerretDB FerretDB (previously MangoDB) was founded to become the de-facto open-source substitute to MongoDB. FerretDB is an open-source proxy, convert

Jan 2, 2023
An easy to use web authorization system integrated with MongoDB.

goth An easy to use web authorization system for MongoDB. Written 100% in Golang! Contributing Feel free to add an issue, and if you have a fix, creat

Feb 22, 2022
go mysql driver, support distributed transaction

Go-MySQL-Driver A MySQL-Driver for Go's database/sql package Features Requirements Installation Usage DSN (Data Source Name) Password Protocol Address

Jul 23, 2022
A fully-featured AWS Athena database driver (+ athenareader https://github.com/uber/athenadriver/tree/master/athenareader)
A fully-featured AWS Athena database driver (+ athenareader https://github.com/uber/athenadriver/tree/master/athenareader)

?? athenadriver - A fully-featured AWS Athena database driver for Go ?? athenareader - A moneywise command line utililty to query athena in command li

Jan 3, 2023
Apache H2 Database Go Driver

Apache H2 Database Go Driver This driver is VERY experimental state NOT use for production yet Introduction Apache H2 Database is a very-low footprint

Nov 27, 2022
sqlcomment is an ent driver that adds SQL comments following sqlcommenter specification.

sqlcomment sqlcomment is an ent driver that adds SQL comments following sqlcommenter specification. sqlcomment includes support for OpenTelemetry and

Nov 14, 2022
Attach hooks to any database/sql driver

sqlhooks Attach hooks to any database/sql driver. The purpose of sqlhooks is to provide a way to instrument your sql statements, making really easy to

Jan 6, 2023
Golang database driver for SQLite

go-sqlite Golang database driver for SQLite. Does not use cgo. This driver is based on pure-Go SQLite implementation (https://gitlab.com/cznic/sqlite)

Dec 30, 2022
BigQuery database/sql golang driver

BigQuery SQL Driver This library is compatible with Go 1.17+ Please refer to CHA

Dec 7, 2022
Otelsql - OpenTelemetry SQL database driver wrapper for Go
Otelsql - OpenTelemetry SQL database driver wrapper for Go

OpenTelemetry SQL database driver wrapper for Go Add a OpenTelemetry wrapper to

Dec 15, 2022
Google Cloud Spanner driver for Go's database/sql package.

go-sql-spanner Google Cloud Spanner driver for Go's database/sql package. This support is currently in the Preview release status. import _ "github.co

Dec 11, 2022
Qmgo - The Go driver for MongoDB. It‘s based on official mongo-go-driver but easier to use like Mgo.

Qmgo English | 简体中文 Qmgo is a Go driver for MongoDB . It is based on MongoDB official driver, but easier to use like mgo (such as the chain call). Qmg

Dec 28, 2022
Go-mongodb - Practice Go with MongoDB because why not

Practice Mongo DB with Go Because why not. Dependencies gin-gonic go mongodb dri

Jan 5, 2022
Mongo Go Models (mgm) is a fast and simple MongoDB ODM for Go (based on official Mongo Go Driver)
Mongo Go Models (mgm) is a fast and simple MongoDB ODM for Go (based on official Mongo Go Driver)

Mongo Go Models Important Note: We changed package name from github.com/Kamva/mgm/v3(uppercase Kamva) to github.com/kamva/mgm/v3(lowercase kamva) in v

Jan 2, 2023
The MongoDB driver for Go

The MongoDB driver for Go This fork has had a few improvements by ourselves as well as several PR's merged from the original mgo repo that are current

Jan 8, 2023