Description

Closes #1593.

We cannot enable authorization to run both FerretDB and MongoDB with PLAIN authorization mechanism because it is not available for MongoDB. Implemented stub was tested locally with FerretDB. To test it locally you need:

• checkout into this branch;
• execute bin/task run
• execute in another terminal docker compose exec mongodb mongosh 'mongodb://username:[email protected]:27018/test?tls=true&tlsCAFile=/etc/certs/rootCA.pem&heartbeatFrequencyMS=300000&authMechanism=PLAIN'

In the FerretDB output check for the saslStart string. FerretDB will respond with ok, MongoDB will respond with MechanismUnavailable error.

Readiness checklist

• [ ] I added tests for new functionality or bugfixes.
• [x] I ran task all, and it passed.
• [x] I added/updated comments for both exported and unexported top-level declarations (functions, types, etc).
• [ ] I checked comments rendering with task godocs.
• [x] I ensured that the title is good enough for the changelog.
• [x] (for maintainers only) I set Reviewers (@FerretDB/core), Assignee, Labels, Project and project's Sprint fields.
• [x] I marked all done items in this checklist.

Description

Closes #1310.

Readiness checklist

• [ ] I added tests for new functionality or bugfixes.
• [ ] I ran task all, and it passed.
• [ ] I added/updated comments for both exported and unexported top-level declarations (functions, types, etc).
• [ ] I checked comments rendering with task godocs.
• [x] I ensured that the title is good enough for the changelog.
• [x] (for maintainers only) I set Reviewers (@FerretDB/core), Assignee, Labels, Project and project's Sprint fields.
• [x] I marked all done items in this checklist.

Description

This PR closes #773.

Readiness checklist

• [x] I added tests for new functionality or bugfixes.
• [x] I ran task all, and it passed.
• [x] I added/updated comments for both exported and unexported top-level declarations (functions, types, etc).
• [ ] I checked comments rendering with task godocs.
• [ ] (for maintainers only) I set Reviewers (@FerretDB/core), Assignee, Labels, Project and project's Sprint fields.
• [ ] I marked all done items in this checklist.

Description

This PR closes #557 .

Run linters in all modules except tools (., ./integration)

Readiness checklist

• [ ] I added tests for new functionality or bugfixes.
• [ ] I ran task all, and it passed.
• [ ] I added/updated comments for both exported and unexported top-level declarations (functions, types, etc).
• [ ] I checked comments rendering with task godocs.
• [ ] (for maintainers only) I set Reviewers (@FerretDB/core), Assignee, Labels, Project and project's Sprint fields.
• [ ] I marked all done items in this checklist.

Description

This PR closes #848.

Readiness checklist

• [x] I added tests for new functionality or bugfixes.
• [x] I ran task all, and it passed.
• [x] I added/updated comments for both exported and unexported top-level declarations (functions, types, etc).
• [ ] I checked comments rendering with task godocs.
• [x] (for maintainers only) I set Reviewers (@FerretDB/core), Assignee, Labels, Project and project's Sprint fields.
• [x] I marked all done items in this checklist.

Description

This is analysis PR for #2, we won't merge it. Select queries of the form {_id: <value>}. PR scope is {_id: <value>} where <value> is ObjectID.

Tigris

TODO: How will we ensure that Tigris compares values the same way as MongoDB?

Not in scope

Other cases when <value> of types are not in scope:

• Strings
  • string comparison difference between MongoDB and PostgreSQL i.e.
  • case sensitive
  • zero values
  • collation
  • encoding checks
  • etc.
• Binary data
  • len
  • sub-type
  • byte by byte
• Numeric
• Dates and Timestamps
  • ordering
• Non-existent Fields (NaN, Inf)
• Objects
• Arrays
  • arrays are restricted to be in _id

Links

The pushdown term. MongoDB restrictions on _id. MongoDB Comparison/Sort Order.

Description

Closes #640.

Readiness checklist

• [ ] I added tests for new functionality or bugfixes.
• [ ] I ran task all, and it passed.
• [ ] I added/updated comments for both exported and unexported top-level declarations (functions, types, etc).
• [ ] I checked comments rendering with task godocs.
• [x] I ensured that the title is good enough for the changelog.
• [x] (for maintainers only) I set Reviewers (@FerretDB/core), Assignee, Labels, Project and project's Sprint fields.
• [x] I marked all done items in this checklist.

Description

Closes #1151.

Readiness checklist

• [x] I added tests for new functionality or bugfixes.
• [ ] I ran task all, and it passed.
• [x] I added/updated comments for both exported and unexported top-level declarations (functions, types, etc).
• [ ] I checked comments rendering with task godocs.
• [x] I ensured that the title is good enough for the changelog.
• [x] (for maintainers only) I set Reviewers (@FerretDB/core), Assignee, Labels, Project and project's Sprint fields.
• [x] I marked all done items in this checklist.

Description

Closes #1295.

• test-integration-pg runs on unix socket, -target-unix-socket is already specified in Taskfile.yml.
• We don't support unix socket on windows, windows always uses TCP.

Readiness checklist

• [ ] I added tests for new functionality or bugfixes.
• [x] I ran task all, and it passed.
• [x] I added/updated comments for both exported and unexported top-level declarations (functions, types, etc).
• [ ] I checked comments rendering with task godocs.
• [x] I ensured that the title is good enough for the changelog.
• [x] (for maintainers only) I set Reviewers (@FerretDB/core), Assignee, Labels, Project and project's Sprint fields.
• [x] I marked all done items in this checklist.

Had a quick shot at the issue mentioned in #61 - It's not complete but I thought I'd just put it up so you could have a look and point me in the right direction in terms of how I'm approaching it! Still learning a bit about pgx, postgres and mongo.

Are there any examples so far I could look at to begin getting a feel for how to go abouts this? And also how about the testing.

Sorry if it's a bit lame, thanks though!

Final Design Deicsion

• [x] Retry parameters maxRetries, delay are constants
• [x] Context should only be checked in inTransaction (no need to check it in InTransactioRetry)
• [x] ctxutil.Sleep(ctx, delay) should be used for delays between attempts
• [x] transactionConflictError should wrap the original error
• [ ] ~lazyerror should be used for errors that are not expected / couldn't be processed~ inTransaction already wraps everything in lazyError.

Initital Proposal (draft, might not compile)

Implement InTransactionRetry function that would use InTransaction with the given retry in case the transaction failed.

Details: In principle, we need to retry the transaction only if there is a pgerrcode.UniqueViolation, pgerrcode.DuplicateObject, or a similar error.

We can define a special error to be returned in such cases. For example, an error could be something like:

// errTransactionConflict is returned when one of the queries in the transaction returned an error because
// of an unexpected conflict. The caller could retry such a transaction.
var errTransactionConflict = fmt.Errorf("transaction can't continue due to a conflict")

And then InTransactionRetry would be something like:

func (pgPool *Pool) InTransactionRetry(ctx context.Context, f func(pgx.Tx) error, maxRetries int, delay time.Duration) (err error) {
	for retry := 0; retry < maxRetries; retry++ {

		// TODO: Maybe check if context is done  

		err := pgPool.InTransaction(ctx, f)

		switch {
		case err == nil:
			return nil
		case errors.Is(err, errTransactionConflict):
			time.Sleep(delay)
		default:
			return err
		}
	}
}

With that, the places where we check for pg errors could be like:

	switch pgErr.Code {
	case pgerrcode.UniqueViolation, pgerrcode.DuplicateObject:
		// https://www.postgresql.org/message-id/CA+TgmoZAdYVtwBfp1FL2sMZbiHCWT4UPrzRLNnX1Nb30Ku3-gg@mail.gmail.com
		return errTransactionConflict
	default:
		return lazyerrors.Error(err)
	}

Definition of done

• [x] Implement InTransactionRetry
• [x] Modify at least one handler (for example, msg_insert) to use InTransactionRetry in case of a pg conflict for CreateCollectionIfNotExist
• [x] Find places where we should use InTransactionRetry and create issue for them

What should be done?

When an empty document bson.D{} is inserted through collection.InsertOne() or collection.InsertMany(), the driver itself sets a new unique _id for such a document.

However, if such a document is inserted through db.RunCommand(), a unique _id is not set.

In this case, FerretDB behaves differently than MongoDB: MongoDB inserts a doc and assigns an _id, and FerretDB returns an error.

We need to fix it.

Where?

See the issue mentioned in the skipped InsertEmpty test - https://github.com/FerretDB/FerretDB/pull/1673/files. Remove "skipped" and run the test to see the difference between MongoDB and FerretDB.

Definition of Done

• compatibility test unskipped.

Description

Closes #735. Closes #1447.

Readiness checklist

• [ ] I added unit tests for new functionality or bug fixes.
• [ ] I added integration tests for new functionality or bug fixes.
• [ ] I added compatibility tests for new functionality or bug fixes.
• [x] I made spot refactorings.
• [x] I updated user documentation.
• [x] I ran task all, and it passed.
• [x] I added/updated comments for both exported and unexported top-level declarations (functions, types, etc).
• [x] I checked comments rendering with task godocs.
• [x] I ensured that the title is good enough for the changelog.
• [x] (for maintainers only) I set Reviewers (@FerretDB/core), Assignee, Labels, Project and project's Sprint fields.
• [x] I marked all done items in this checklist.

What should be done?

It should be possible to configure FerretDB to validate the client's certificates against the given CA certificate and reject connections without valid certificates. See https://www.mongodb.com/docs/manual/tutorial/configure-ssl/#set-up-mongod-and-mongos-with-client-certificate-validation. Standard go library should be enough for that

Where?

https://github.com/FerretDB/FerretDB/blob/main/internal/clientconn/listener.go

Definition of Done

• unit, integration, or compatibility test should added (at least one of them should be possible to do);
• spot refactorings done;
• user documentation updated (see #1706).

What should be done?

Our documentation should provide more information than just --help flag.

Most importantly:

• the --mode flag requires some explanation in user documentation, not only in CONTRIBUTING.md;
• TLS flags should be explained.

Related: https://ferretdb.slack.com/archives/C049W6E01E0/p1672187174464469