Golang Mongo Pagination For Package mongo-go-driver

Golang Mongo Pagination For Package mongo-go-driver

Build Go Report Card GoDoc Coverage Status

For all your simple query to aggregation pipeline this is simple and easy to use Pagination driver with information like Total, Page, PerPage, Prev, Next, TotalPage and your actual mongo result. View examples from here

🔈 🔈 For normal queries new feature have been added to directly pass struct and decode data without manual unmarshalling later. Only normal queries support this feature for now. Sort chaining is also added as new feature

Example api response of Normal Query click here.
Example api response of Aggregate Query click here.
View code used in this example from here

Install

$ go get -u -v github.com/gobeam/mongo-go-pagination

or with dep

$ dep ensure -add github.com/gobeam/mongo-go-pagination

For Aggregation Pipelines Query

package main

import (
	"context"
	"fmt"
	. "github.com/gobeam/mongo-go-pagination"
	"go.mongodb.org/mongo-driver/bson"
	"go.mongodb.org/mongo-driver/bson/primitive"
	"go.mongodb.org/mongo-driver/mongo"
	"go.mongodb.org/mongo-driver/mongo/options"
	"time"
)

type Product struct {
	Id       primitive.ObjectID `json:"_id" bson:"_id"`
	Name     string             `json:"name" bson:"name"`
	Quantity float64            `json:"qty" bson:"qty"`
	Price    float64            `json:"price" bson:"price"`
}

func main() {
	// Establishing mongo db connection
	ctx, _ := context.WithTimeout(context.Background(), 10*time.Second)
	client, err := mongo.Connect(ctx, options.Client().ApplyURI("mongodb://localhost:27017"))
	if err != nil {
		panic(err)
	}
	
	var limit int64 = 10
	var page int64 = 1
	collection := client.Database("myaggregate").Collection("stocks")

	//Example for Aggregation

	//match query
	match := bson.M{"$match": bson.M{"qty": bson.M{"$gt": 10}}}

	//group query
	projectQuery := bson.M{"$project": bson.M{"_id": 1, "qty": 1}}

    // set collation if required
    collation := options.Collation{
		Locale:    "en",
		CaseLevel: true,
	}

	// you can easily chain function and pass multiple query like here we are passing match
	// query and projection query as params in Aggregate function you cannot use filter with Aggregate
	// because you can pass filters directly through Aggregate param
	aggPaginatedData, err := New(collection).SetCollation(&collation).Context(ctx).Limit(limit).Page(page).Sort("price", -1).Aggregate(match, projectQuery)
	if err != nil {
		panic(err)
	}

	var aggProductList []Product
	for _, raw := range aggPaginatedData.Data {
		var product *Product
		if marshallErr := bson.Unmarshal(raw, &product); marshallErr == nil {
			aggProductList = append(aggProductList, *product)
		}

	}

	// print ProductList
	fmt.Printf("Aggregate Product List: %+v\n", aggProductList)

	// print pagination data
	fmt.Printf("Aggregate Pagination Data: %+v\n", aggPaginatedData.Data)
}

For Normal queries

func main() {
	// Establishing mongo db connection
	ctx, _ := context.WithTimeout(context.Background(), 10*time.Second)
	client, err := mongo.Connect(ctx, options.Client().ApplyURI("mongodb://localhost:27017"))
	if err != nil {
		panic(err)
	}

	// Example for Normal Find query
    	filter := bson.M{}
    	var limit int64 = 10
    	var page int64 = 1
    	collection := client.Database("myaggregate").Collection("stocks")
    	projection := bson.D{
    		{"name", 1},
    		{"qty", 1},
    	}
    	// Querying paginated data
    	// Sort and select are optional
        // Multiple Sort chaining is also allowed
        // If you want to do some complex sort like sort by score(weight) for full text search fields you can do it easily
        // sortValue := bson.M{
        //		"$meta" : "textScore",
        //	}
        // aggPaginatedData, err := paginate.New(collection).Context(ctx).Limit(limit).Page(page).Sort("score", sortValue)...
        var products []Product
    	paginatedData, err := New(collection).Context(ctx).Limit(limit).Page(page).Sort("price", -1).Select(projection).Filter(filter).Decode(&products).Find()
    	if err != nil {
    		panic(err)
    	}
    
    	// paginated data or paginatedData.Data will be nil because data is already decoded on through Decode function
    	// pagination info can be accessed in  paginatedData.Pagination
    	// print ProductList
    	fmt.Printf("Normal Find Data: %+v\n", products)
    
    	// print pagination data
    	fmt.Printf("Normal find pagination info: %+v\n", paginatedData.Pagination)
}
    

Notice:

paginatedData.data //it will be nil incase of  normal queries because data is already decoded on through Decode function

Running the tests

$ go test

Contributing

Pull requests are welcome. For major changes, please open an issue first to discuss what you would like to change.

Please make sure to update tests as appropriate.

Acknowledgments

  1. https://github.com/mongodb/mongo-go-driver

MIT License

Copyright (c) 2021
Owner
Roshan Ranabhat
Full Stack Developer (Golang, Node.js, PHP, React & Redux, Vue.js)
Roshan Ranabhat
Comments
  • Feature to sort first before limit

    Feature to sort first before limit

    Description

    • The default settings for Aggregate() are inconvenient. There are needs to sort and limit throughout total documents, but now limit first and then sorting within it.
    • Example code
    collection := client.Database("test_database").Collection("test_sort")
    // insert 20 documents in order
    for i := 1; i <= 20; i++ {
     	loc, _ := time.LoadLocation("Asia/Seoul")
     	now := primitive.NewDateTimeFromTime(time.Now().In(loc))
     	time.Sleep(100 * time.Millisecond)
     	newDoc := map[string]interface{}{
     		"name":       i,
     		"created_at": now,
     	}
     	if _, err := collection.InsertOne(context.Background(), newDoc); err != nil {
     		panic(err)
     	}
    }
    var queries []interface{}
    var docs []map[string]interface{}
    pqr := pg.New(collection).Limit(int64(10)).Page(int64(1)).Sort("created_at", -1) // sort desc
    paginatedData, err := pqr.Aggregate(queries...)
    if err != nil {
     	panic(err)
    }
    for _, raw := range paginatedData.Data {
     	doc := make(map[string]interface{})
     	if marshallErr := bson.Unmarshal(raw, &doc); marshallErr == nil {
     		docs = append(docs, doc)
     	}
    }
    for _, doc := range docs {
     	fmt.Printf("Name : %v CreatedAt : %v\n ", doc["name"], doc["created_at"])
    }
    
    • Expected result with 20 coming first, but 10.

    Suggestion

    • How about adding interface to sort first?
    type pagingQuery struct {
    	Collection  *mongo.Collection
    	...
    	SortFirst   bool // added
    }
    
    type PagingQuery interface {
    	// Find set the filter for query results.
    	Find() (paginatedData *PaginatedData, err error)
            ...
    	SetSortFirst(sortFirst bool) PagingQuery // added
    }
    
    // SetSortFirst is function to set sortFirst field
    func (paging *pagingQuery) SetSortFirst(sortFirst bool) PagingQuery { // added
    	paging.SortFirst = sortFirst
    	return paging
    }
    
    func (paging *pagingQuery) Aggregate(filters ...interface{}) (paginatedData *PaginatedData, err error) {
    	// checking if user added required params
    	if err := paging.validateQuery(false); err != nil {
    		return nil, err
    	}
    	if paging.FilterQuery != nil {
    		return nil, errors.New(FilterInAggregateError)
    	}
    
    	var aggregationFilter []bson.M
    	// combining user sent queries
    	for _, filter := range filters {
    		aggregationFilter = append(aggregationFilter, filter.(bson.M))
    	}
    	skip := getSkip(paging.PageCount, paging.LimitCount)
    	var facetData []bson.M
    	facetData = setFacetDataInOrder(paging.SortFirst, paging.SortFields, skip, paging.LimitCount) // replaced
    
    	//if paging.SortField != "" {
    	//	facetData = append(facetData, bson.M{"$sort": bson.M{paging.SortField: paging.SortValue}})
    	//}
    	// making facet aggregation pipeline for result and total document count
    	facet := bson.M{"$facet": bson.M{
    		"data":  facetData,
    		"total": []bson.M{{"$count": "count"}},
    	},
    ..........
    }
    
    // setFacetDataInOrder return facet data by pre-set sortFirst option
    func setFacetDataInOrder(sortFirst bool, sortFields bson.D, skip, limitCount int64) []bson.M { // added
    	var facetData []bson.M
    	if sortFirst {
    		if len(sortFields) > 0 {
    			facetData = append(facetData, bson.M{"$sort": sortFields})
    		}
    		facetData = append(facetData, bson.M{"$skip": skip})
    		facetData = append(facetData, bson.M{"$limit": limitCount})
    	} else {
    		facetData = append(facetData, bson.M{"$skip": skip})
    		facetData = append(facetData, bson.M{"$limit": limitCount})
    		if len(sortFields) > 0 {
    			facetData = append(facetData, bson.M{"$sort": sortFields})
    		}
    	}
    	return facetData
    }
    

    Example code after change

    var queries []interface{}
    var docs []map[string]interface{}
    pqr := pg.New(collection).Limit(int64(10)).Page(int64(1)).Sort("created_at", -1).SetSortFirst(true) // look here!
    paginatedData, err := pqr.Aggregate(queries...)
    if err != nil {
     	panic(err)
    }
    for _, raw := range paginatedData.Data {
     	doc := make(map[string]interface{})
     	if marshallErr := bson.Unmarshal(raw, &doc); marshallErr == nil {
     		docs = append(docs, doc)
     	}
    }
    for _, doc := range docs {
     	fmt.Printf("Name : %v CreatedAt : %v\n ", doc["name"], doc["created_at"])
    }
    
    • If SetSortFirst() is not called, the default value is false , so the existing code is not affected.
  • How to do multiple sort fields?

    How to do multiple sort fields?

    We can have a single sort by using .Sort("price", -1). But how to add another sort field?

    I tried .Sort("price", -1).Sort("_id", 1) didn't work.

    Thanks

  • Is there any way to sort by weight(score) for full text search fields?

    Is there any way to sort by weight(score) for full text search fields?

    Is your feature request related to a problem? Please describe. As I know right now we have only .Sort() method which accept 2 arguments: field name and sort by ASC, DESC but it would be nice to have a method to sort by score(weight) for full text search fields. The query might be something like: db.table.find({$text:{$search:"Test"}},{score:{$meta:"textScore"}}).sort({score:{$meta:"textScore"}})

    Describe the solution you'd like It would be nice to have a method like .sortByScore()

  • Page limit not applied for aggregation

    Page limit not applied for aggregation

    Hi,

    I have 3 entries in my db and I apply following aggregation query.

            matchStage := bson.M{"$match": bson.M{"user_id":  userId}}
    	lookup := bson.M{"$lookup": bson.M{"from": "com", "localField": "com_id", "foreignField": "_id", "as": "com"}}
    	unwindStage := bson.M{"$unwind": bson.M{"path": "$com", "preserveNullAndEmptyArrays": false}}
    	
            cur, err := mongopagination.New(cr.GetCollection()).
    		Page(1).
    		Limit(1).
    		Sort("role", -1).
    		Aggregate(matchStage)
    	if err != nil {
    		panic(err)
    	}
    

    Expected 3 pages with 1 result per page.

    Actual 3 pages with all 3 results per page.

    {
       "data":[
          {
             "com":{
                "id":"5f68d3dc9b8155bb8807d1e6",
                "name":"ceE5u25yH2",
                "info":{
                   "address":{
                      
                   }
                },
                "is_open":true,
                "is_active":true
             },
             "role":2
          },
          {
             "com":{
                "id":"5f68d3dc9b8155bb8807d1e8",
                "name":"3QmOnaG3uy",
                "info":{
                   "address":{
                      
                   }
                },
                "is_open":true,
                "is_active":true
             },
             "role":2
          },
          {
             "com":{
                "id":"5f68d3dc9b8155bb8807d1ea",
                "name":"RDn8LlZGeX",
                "info":{
                   "address":{
                      
                   }
                },
                "is_open":true,
                "is_active":true
             },
             "role":2
          }
       ],
       "pagination":{
          "total":3,
          "page":1,
          "perPage":1,
          "prev":0,
          "next":2,
          "totalPage":3
       }
    }
    

    Cause: File: pagingQuery.go Method: Aggregate

    facetData slice "$skip" (and "$limit" if sort is used) is overridden as slice 0 index value replaced.

            facetData := make([]bson.M, 1)
    	facetData[0] = bson.M{"$skip": skip}
    	facetData[0] = bson.M{"$limit": paging.LimitCount}
    	if paging.SortField != "" {
    		facetData[0] = bson.M{"$sort": bson.M{paging.SortField: paging.SortValue}}
    	}
    
  • Feature/set collation

    Feature/set collation

    Description

    Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.

    Fixes #30

    Type of change

    Please delete options that are not relevant.

    • [X] New feature (non-breaking change which adds functionality)
    • [X] This change requires a documentation update

    How Has This Been Tested?

    Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration

    Test method is included

    Checklist:

    • [X] My code follows the style guidelines of this project
    • [X] I have performed a self-review of my own code
    • [X] I have commented my code, particularly in hard-to-understand areas
    • [X] I have made corresponding changes to the documentation
    • [X] My changes generate no new warnings
    • [X] I have added tests that prove my fix is effective or that my feature works
    • [X] New and existing unit tests pass locally with my changes
  • Correct the offset calculation

    Correct the offset calculation

    The offset wasn't correct and the last result from the previous page was included as the first result in a subsequent page.

    Description

    Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.

    Fixes # (issue)

    Type of change

    Please delete options that are not relevant.

    • [X] Bug fix (non-breaking change which fixes an issue)

    How Has This Been Tested?

    Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration

    • Create records which are a certain number e.g. 22
    • send a request for page 1, limit 15. Note 15 records are correctly returned
    • send another request for page 2, limit 15 and note there are 8 records where there should be 7. Record number 15 is repeated in both result sets.
    • The correct offset for page 2, limit 15 is 16 so the calculation is limit-1 = 2 * offset 15 which is 15 + 1 which is 16.

    Checklist:

    • [X] My code follows the style guidelines of this project
    • [X] I have performed a self-review of my own code
    • [X] I have commented my code, particularly in hard-to-understand areas
    • [ ] I have made corresponding changes to the documentation
    • [X] My changes generate no new warnings
    • [ ] I have added tests that prove my fix is effective or that my feature works
    • [X] New and existing unit tests pass locally with my changes
    • [ ] Any dependent changes have been merged and published in downstream modules
  • getSkip decrements skip by one more, causing the pagination to pick up one less document

    getSkip decrements skip by one more, causing the pagination to pick up one less document

    Describe the bug A clear and concise description of what the bug is. "getSkip" function decrements skip by one more, causing the pagination to pick up one less document.

    To Reproduce Steps to reproduce the behavior: Create a collection with 2 documents;

    1. Call pagination function with page number 2 and limit 1.
    2. The function returns the first document instead of second document.

    Expected behavior The function should return second document in page 2.

    Additional context This happens in version v0.0.5 where the "getSkip" function decrements the "skip" value by one more after calculating the correct value.

    // getSkip return calculated skip value for query

    func getSkip(page, limit int64) int64 {
    	page--
    	skip := page * limit
    	skip--
    
    	if skip <= 0 {
    		skip = 0
    	}
    
    	return skip
    }
    
  • Skip calculation error

    Skip calculation error

    Describe the bug A clear and concise description of what the bug is.

    To Reproduce Steps to reproduce the behavior:

    1. Call function '...' mongopagination.New(models.RedPackCollection).Context(context.TODO()).Limit(int64(p.PageSize)).Page(int64(p.CurPage)).Filter()
    2. Pass value '...' p.PageSize=10 p.CurPage=2
    3. See error

    func getSkip(page, limit int64) int64 { page-- skip := page * limit skip-- / / produces an extra result

    if skip <= 0 {
    	skip = 0
    }
    
    return skip
    

    }

    Expected behavior A clear and concise description of what you expected to happen.

    I think we should delete him

    Screenshots If applicable, add screenshots to help explain your problem.

    Additional context Add any other context about the problem here.

  • decode to struct we want instead of bson.raw

    decode to struct we want instead of bson.raw

    in line, you use a decoder and pass variable in type of bson.raw, in the example you use decode again to convert data to struct we want ... in my opinion we can add func to app to get the struct and decode it in the first time ! what is your opinion about this ?

  • Hardcoded Sort on Aggregate function

    Hardcoded Sort on Aggregate function

    facet := bson.M{"$facet": bson.M{
    		"data": []bson.M{
    			{"$sort": bson.M{"createdAt": -1}},
    			{"$skip": skip},
    			{"$limit": paging.LimitCount},
    		},
    		"total": []bson.M{{"$count": "count"}},
    	},
    }
    

    It makes it impossible to have a sort pipeline when doing aggregation

  • Use aggregate with optional match stage

    Use aggregate with optional match stage

    Is your feature request related to a problem? Please describe. How can i use aggregate with optional match stage. For example, I have to use aggregate like this:

      var matchStage bson.M
      if accountId != "" {
          matchStage = bson.M{
                "$match": bson.M{ "_id": accountId, },
          }
      }
      aggPaginatedData, err := New(collection).Context(ctx).Limit(limit).Page(page).Sort("created_at",sort).Aggregate(matchStage)
    

    Here, if there is value for accountId, then the aggregate works fine but if the accountId == "" then i got an error says A pipeline stage specification object must contain exactly one field How can we deal with that scenario? Thanks

Go MySQL Driver is a MySQL driver for Go's (golang) database/sql package

Go-MySQL-Driver A MySQL-Driver for Go's database/sql package Features Requirements Installation Usage DSN (Data Source Name) Password Protocol Address

Jan 4, 2023
Go driver for PostgreSQL over SSH. This driver can connect to postgres on a server via SSH using the local ssh-agent, password, or private-key.

pqssh Go driver for PostgreSQL over SSH. This driver can connect to postgres on a server via SSH using the local ssh-agent, password, or private-key.

Nov 6, 2022
Firebird RDBMS sql driver for Go (golang)

firebirdsql (Go firebird sql driver) Firebird RDBMS http://firebirdsql.org SQL driver for Go Requirements Firebird 2.5 or higher Golang 1.13 or higher

Dec 20, 2022
Lightweight Golang driver for ArangoDB

Arangolite Arangolite is a lightweight ArangoDB driver for Go. It focuses on pure AQL querying. See AranGO for a more ORM-like experience. IMPORTANT:

Sep 26, 2022
Golang driver for ClickHouse

ClickHouse Golang SQL database driver for Yandex ClickHouse Key features Uses native ClickHouse tcp client-server protocol Compatibility with database

Jan 8, 2023
Golang MySQL driver

Install go get github.com/vczyh/go-mysql-driver Usage import _ "github.com/vczyh

Jan 27, 2022
Mirror of Apache Calcite - Avatica Go SQL Driver

Apache Avatica/Phoenix SQL Driver Apache Calcite's Avatica Go is a Go database/sql driver for the Avatica server. Avatica is a sub-project of Apache C

Nov 3, 2022
Microsoft ActiveX Object DataBase driver for go that using exp/sql

go-adodb Microsoft ADODB driver conforming to the built-in database/sql interface Installation This package can be installed with the go get command:

Dec 30, 2022
Microsoft SQL server driver written in go language

A pure Go MSSQL driver for Go's database/sql package Install Requires Go 1.8 or above. Install with go get github.com/denisenkom/go-mssqldb . Connecti

Dec 26, 2022
Oracle driver for Go using database/sql

go-oci8 Description Golang Oracle database driver conforming to the Go database/sql interface Installation Install Oracle full client or Instant Clien

Dec 30, 2022
sqlite3 driver for go using database/sql

go-sqlite3 Latest stable version is v1.14 or later not v2. NOTE: The increase to v2 was an accident. There were no major changes or features. Descript

Jan 8, 2023
GO DRiver for ORacle DB

Go DRiver for ORacle godror is a package which is a database/sql/driver.Driver for connecting to Oracle DB, using Anthony Tuininga's excellent OCI wra

Jan 5, 2023
Go Sql Server database driver.

gofreetds Go FreeTDS wrapper. Native Sql Server database driver. Features: can be used as database/sql driver handles calling stored procedures handle

Dec 16, 2022
PostgreSQL driver and toolkit for Go

pgx - PostgreSQL Driver and Toolkit pgx is a pure Go driver and toolkit for PostgreSQL. pgx aims to be low-level, fast, and performant, while also ena

Jan 4, 2023
Pure Go Postgres driver for database/sql

pq - A pure Go postgres driver for Go's database/sql package Install go get github.com/lib/pq Features SSL Handles bad connections for database/sql S

Jan 2, 2023
Go language driver for RethinkDB
Go language driver for RethinkDB

RethinkDB-go - RethinkDB Driver for Go Go driver for RethinkDB Current version: v6.2.1 (RethinkDB v2.4) Please note that this version of the driver on

Dec 24, 2022
goriak - Go language driver for Riak KV
goriak - Go language driver for Riak KV

goriak Current version: v3.2.1. Riak KV version: 2.0 or higher, the latest version of Riak KV is always recommended. What is goriak? goriak is a wrapp

Nov 22, 2022
The MongoDB driver for Go

The MongoDB driver for Go This fork has had a few improvements by ourselves as well as several PR's merged from the original mgo repo that are current

Jan 8, 2023
The Go driver for MongoDB
The Go driver for MongoDB

MongoDB Go Driver The MongoDB supported driver for Go. Requirements Installation Usage Bugs / Feature Reporting Testing / Development Continuous Integ

Dec 31, 2022