Run a real Postgres database locally on Linux, OSX or Windows as part of another Go application or test

Godoc Coverage Status Build Status Go Report Card

embedded-postgres

Run a real Postgres database locally on Linux, OSX or Windows as part of another Go application or test.

When testing this provides a higher level of confidence than using any in memory alternative. It also requires no other external dependencies outside of the Go build ecosystem.

Heavily inspired by Java projects zonkyio/embedded-postgres and opentable/otj-pg-embedded and reliant on the great work being done by zonkyio/embedded-postgres-binaries in order to fetch precompiled binaries from Maven.

Installation

embedded-postgres uses Go modules and as such can be referenced by release version for use as a library. Use the following to add the latest release to your project.

go get -u github.com/fergusstrange/embedded-postgres

How to use

This library aims to require as little configuration as possible, favouring overridable defaults

Configuration Default Value
Username postgres
Password postgres
Database postgres
Version 12.1.0
RuntimePath $USER_HOME/.embedded-postgres-go/extracted
Port 5432
StartTimeout 15 Seconds

A single Postgres instance can be created, started and stopped as follows

postgres := embeddedpostgres.NewDatabase()
err := postgres.Start()

// Do test logic

err := postgres.Stop()

or with created with custom configuration

logger := &bytes.Byffer{}
postgres := NewDatabase(DefaultConfig().
            Username("beer").
            Password("wine").
            Database("gin").
            Version(V12).
            RuntimePath("/tmp").
            Port(9876).
            StartTimeout(45 * time.Second).
            Logger(logger))
err := postgres.Start()

// Do test logic

err := postgres.Stop()

It should be noted that if postgres.Stop() is not called then the child Postgres process will not be released and the caller will block.

Examples

There are a number of realistic representations of how to use this library in examples.

Credits

Contributing

View the contributing guide.

Comments
  • Cannot find binary in archive

    Cannot find binary in archive

    Hey all, stumbled upon this repository and decided to give it a try.

    Unfortunately the following snippet does not seem to work me.

    pg := embeddedpostgres.NewDatabase()
    
    if err := pg.Start(); err != nil {
      t.Error(err)
      return
    }
    
    defer pg.Stop()
    

    This gives me the error error fetching postgres: cannot find binary in archive retrieved from https://repo1.maven.org/maven2/io/zonky/test/postgres/embedded-postgres-binaries-linux-amd64/12.8.0/embedded-postgres-binaries-linux-amd64-12.8.0.jar

    go version gives me: go version go1.17.1 linux/amd64 and I'm running Ubuntu 20.04.01 with kernel 5.8.0-23-generic.

    Am I doing something wrong?

  • Add a BinariesPath configuration option, for prefetching Postgres binaries before the test

    Add a BinariesPath configuration option, for prefetching Postgres binaries before the test

    Downloading and Un-archiving the postgres binaries is slow and flaky in some CI systems. For example - in my case, our CI system starts a fresh sandbox for every test it runs, and as such caching the downloaded postgres file is impossible. Even when it is possible to cache the downloaded file, it's still might be slow due to the time it takes to unarchive it.

    The change in this PR does the following: It gives an option to specify a binaries path, where the binaries will be unarchived to, and will not get deleted between tests. If the BinariesPath/bin directory exists, the system will assume files were pre-downloaded, and will not try to fetch/unarchive them.

    The PR is split into 4 commits:

    1. Refactor of the usage of the folders, in order to not "recalculate" the data/runtime directories over and over again (makes code simpler)
    2. Separates the variable holding the runtimePath from the one holding the binaryExtractLocation (as those can now be different).
    3. Adds the BinariesPath to the configuration, and adds the logic to reuse unarchived binaries (as well as adds new tests)
    4. Updates README (documentation)

    This change should be fully backward compatible if BinariesPath is not specified.

    This also helps with #35 (@mbfr).

  • Fixed CVE-2020-16845 by upgrading xz

    Fixed CVE-2020-16845 by upgrading xz

    Upgraded xz to fix this CVE https://github.com/ulikunitz/xz/security/advisories/GHSA-25xm-hr59-7c27

    https://deps.dev/go/github.com%2Ffergusstrange%2Fembedded-postgres

  • unable to extract postgres archive: xz: data is truncated or corrupt

    unable to extract postgres archive: xz: data is truncated or corrupt

    When using embedded-postgres in a docker container we sometimes see the following error:

    Failed to start embedded postgres: unable to extract postgres archive: xz: data is truncated or corrupt Perhaps there's a timing error in the code that fetches the jar file from maven and extracts the embedded .tgz file containing the postgres binaries.

  • Fix goroutine and connection leak on database connect

    Fix goroutine and connection leak on database connect

    Closes #58

    I also added a goroutine leak detector to prevent unclosed connections in the future and one exception for goroutine leak detector, since I did not find how to fix it.

  • Support Intel and Arm MAC

    Support Intel and Arm MAC

    Note: this issue is created to isolate from the other issues that per affected on Linux and windows.

    Look into support for better logging on new Mac ARM, with rosetta emulation support by default.

    Longer term support for Mac ARM when the packages are available on Maven.

  • cleanup deleted my project

    cleanup deleted my project

    The cleanup just deleted my whole project because I set RuntimePath to ./ and apparently the Stop method deletes the entire directory? Luckily version control, but I lost all my certs and an hour or so of work. Cool project, but this was a pretty big surprise and a major foot-gun.

  • Add runAsUser functionality

    Add runAsUser functionality

    By supplying runAsUser it is possible to run initdb as non-root user (which otherwise fails) even if the calling process runs as root.

    In addition, flush logger on failures to get more detailed errors.

  • Unable to run in official docker golang:1.15

    Unable to run in official docker golang:1.15

    unable to init database using: /root/.embedded-postgres-go/extracted/bin/initdb -A password -U postgres -D /root/.embedded-postgres-go/extracted/data --pwfile=/root/.embedded-postgres-go/extracted/pwfile

  • Add locale config option

    Add locale config option

    This makes the locale of the database created configurable. E.g. for consistent testing where some things depend on the database locale, this is helpful. Or also on broken systems, where the default locale is an invalid string.

    I also bumped the versions to the latest point releases.

  • Return an error instead of deleting data on upgrade

    Return an error instead of deleting data on upgrade

    When upgrading the version of PostgreSQL, all data was deleted. This seems like undesired user behavior because there's not a clear warning message that everything will be reset.

    https://github.com/fergusstrange/embedded-postgres/blob/7bf3aab2dca3964165ded64585e2516791c88876/embedded_postgres.go#L112-L118

    I think returning an error due to a data version mismatch PostgreSQL won't start is more expected, and causes less risk of users just losing all of their data.

  • Run as a different user (non-root)

    Run as a different user (non-root)

    Hi,

    First off, thanks for an awesome project! We're using embedded-postgres over at coder/coder as a tool to quickly get up-and-running and it's been great.

    We'd love to be able to launch embedded-postgres as a different (non-root) user to further improve the experience for users running Docker on macOS.

    I'll spare you the full details as to why we'd need this functionality (for the curious https://github.com/coder/coder/issues/5263), but TL;DR, we'd like to avoid extra steps where users have to change Docker socket permissions which has side-effects outside the container. The easiest way to achieve this is to run as root. Being able to run embedded-postgres as a different user would allow us to keep control within the main process running in the container.

    I saw that there was already a PR for this functionality, https://github.com/fergusstrange/embedded-postgres/pull/81, but I thought it would be worthwhile to open an issue for it to show our support for the feature.

    Ours is just one use-case though, and I understand if you still feel it's not a great fit for your project. Cheers.

  • Can't change `postgres.conf` values

    Can't change `postgres.conf` values

    I want to override in the MAX_CONNECTIONS variable which is located in RuntimePath/postgresql.conf. The problem is, this value needs to be overridden prior to the DB process starting in order to take effect. postgres.Start() clears RuntimePath directory, which doesn't allow a custom postgresql.conf to be injected. There's also no restart mechanism that would utilize the same postgresql.conf file without wiping RuntimePath.

  • Patch for m1 macs: Use native binaries for 14.2 and above

    Patch for m1 macs: Use native binaries for 14.2 and above

    Since arm64 binaries exist for postgres >= 14.2, use those. This helps avoid some architecture-specific issues when using AMD architecture binaries on ARM platforms.

    See https://github.com/fergusstrange/embedded-postgres/issues/86

  • Own the postgres process

    Own the postgres process

    Currently pg_ctl is used to start postgres, and we're seeing an issue where if code using this lib gets hard killed, the postgres process lives on. By running postgres directly it will die when the supervisor process dies.

    Maybe there's a way to do this using pg_ctl? That would be great since then we wouldn't have to poll the port to check if the server is listening.

  • Unable to init database

    Unable to init database

    unable to init database using '/var/folders/70/rld0rwh92bg781zkxt1x6szh0000gn/T/TestProcessEvmEvent_IRIsExist1738323574/001/bin/initdb -A password -U postgres -D /var/folders/70/rld0rwh92bg781zkxt1x6szh0000gn/T/TestProcessEvmEvent_IRIsExist1738323574/001/data --pwfile=/var/folders/70/rld0rwh92bg781zkxt1x6szh0000gn/T/TestProcessEvmEvent_IRIsExist1738323574/001/pwfile': exit status 1

    System Specs:

    Apple M1 Pro MacOS Ventura 13.0

    using github.com/fergusstrange/embedded-postgres v1.19.0

    Only way i've been able to reconcile this is by restarting my computer.

Ruby on Rails like test fixtures for Go. Write tests against a real database

testfixtures Warning: this package will wipe the database data before loading the fixtures! It is supposed to be used on a test database. Please, doub

Jan 8, 2023
Flugel Test Documentation for steps to run and test the automatio
Flugel Test Documentation for steps to run and test the automatio

Flugel Test Documentation Documentation for steps to run and test the automation #Test-01 1 - Local Test Using Terratest (End To End) 1- By runing " t

Nov 13, 2022
This is a simple test application that sends fake video data from one pion instance to another

Pion test app for BWE This is a simple test application that sends fake video data from one pion instance to another. It is a modified version of the

Jun 8, 2022
Cloud Spanner load generator to load test your application and pre-warm the database before launch

GCSB GCSB Quickstart Create a test table Load data into table Run a load test Operations Load Single table load Multiple table load Loading into inter

Nov 30, 2022
go-test-trace is like go test but it also generates distributed traces.
go-test-trace is like go test but it also generates distributed traces.

go-test-trace go-test-trace is like go test but it also generates distributed traces. Generated traces are exported in OTLP to a OpenTelemetry collect

Jan 5, 2023
Test-assignment - Test assignment with golang
Test-assignment - Test assignment with golang

test-assignment We have a two steam of data and we need to save it in the map: I

Jan 19, 2022
This repository includes consumer driven contract test for provider, unit test and counter api.

This repository includes consumer driven contract test for provider, unit test and counter api.

Feb 1, 2022
Sql mock driver for golang to test database interactions

Sql driver mock for Golang sqlmock is a mock library implementing sql/driver. Which has one and only purpose - to simulate any sql driver behavior in

Dec 31, 2022
Plow is a high-performance HTTP benchmarking tool with real-time web UI and terminal displaying
Plow is a high-performance HTTP benchmarking tool with real-time web UI and terminal displaying

Plow is a HTTP(S) benchmarking tool, written in Golang. It uses excellent fasthttp instead of Go's default net/http due to its lightning fast performance.

Jan 9, 2023
Load generator for measuring overhead generated by EDRs and other logging tools on Linux

Simple load generator for stress-testing EDR software The purpose of this tool is to measure CPU overhead incurred by having active or passive securit

Nov 9, 2022
manage your mocks / run mockgen more quickly / mocks up-to-date checking
manage your mocks / run mockgen more quickly / mocks up-to-date checking

gomockhandler If you find any bugs or have feature requests, please feel free to create an issue. gomockhandler is handler of golang/mock, as the name

Dec 30, 2022
How we can run unit tests in parallel mode with failpoint injection taking effect and without injection race

This is a simple demo to show how we can run unit tests in parallel mode with failpoint injection taking effect and without injection race. The basic

Oct 31, 2021
Ritchie CLI is an open-source tool that allows to create, store and share any kind of automation, executing them through command lines, to run operations or start workflows ⚙️ 🖥 💡
Ritchie CLI is an open-source tool that allows to create, store and share any kind of automation, executing them through command lines, to run operations or start workflows ⚙️ 🖥 💡

Table of contents 1. About 2. Getting Started i. Installation ii. Initialize rit locally iii. Add your first formulas repository iv. Run the Hello Wor

Dec 29, 2022
End to end functional test and automation framework
End to end functional test and automation framework

Declarative end to end functional testing (endly) This library is compatible with Go 1.12+ Please refer to CHANGELOG.md if you encounter breaking chan

Jan 6, 2023
Test your code without writing mocks with ephemeral Docker containers 📦 Setup popular services with just a couple lines of code ⏱️ No bash, no yaml, only code 💻

Gnomock – tests without mocks ??️ Spin up entire dependency stack ?? Setup initial dependency state – easily! ?? Test against actual, close to product

Dec 29, 2022
go-carpet - show test coverage in terminal for Go source files
go-carpet - show test coverage in terminal for Go source files

go-carpet - show test coverage for Go source files To view the test coverage in the terminal, just run go-carpet. It works outside of the GOPATH direc

Jan 8, 2023
http integration test framework

go-hit hit is an http integration test framework written in golang. It is designed to be flexible as possible, but to keep a simple to use interface f

Dec 29, 2022
Go testing in the browser. Integrates with `go test`. Write behavioral tests in Go.
Go testing in the browser. Integrates with `go test`. Write behavioral tests in Go.

GoConvey is awesome Go testing Welcome to GoConvey, a yummy Go testing tool for gophers. Works with go test. Use it in the terminal or browser accordi

Dec 30, 2022
A Go test assertion library for verifying that two representations of JSON are semantically equal
A Go test assertion library for verifying that two representations of JSON are semantically equal

jsonassert is a Go test assertion library for verifying that two representations of JSON are semantically equal. Usage Create a new *jsonassert.Assert

Jan 4, 2023