WAL-G is an archival restoration tool for PostgreSQL, MySQL/MariaDB, and MS SQL Server (beta for MongoDB and Redis).

Last update: Jan 1, 2023

Comments: 16

WAL-G

This documentation is also available at wal-g.readthedocs.io

WAL-G is an archival restoration tool for PostgreSQL, MySQL/MariaDB, and MS SQL Server (beta for MongoDB and Redis).

WAL-G is the successor of WAL-E with a number of key differences. WAL-G uses LZ4, LZMA, or Brotli compression, multiple processors, and non-exclusive base backups for Postgres. More information on the original design and implementation of WAL-G can be found on the Citus Data blog post "Introducing WAL-G by Citus: Faster Disaster Recovery for Postgres".

Table of Contents

Installation
Configuration
Usage
Databases
Development
Authors
License
Acknowledgments
Chat

Installation

A precompiled binary for Linux AMD 64 of the latest version of WAL-G can be obtained under the Releases tab.

To decompress the binary, use:

tar -zxvf wal-g.linux-amd64.tar.gz
mv wal-g /usr/local/bin/

For other incompatible systems, please consult the Development section for more information.

Configuration

Storage

To configure where WAL-G stores backups, please consult the Storages section.

Compression

WALG_COMPRESSION_METHOD

To configure the compression method used for backups. Possible options are: lz4, lzma, brotli. The default method is lz4. LZ4 is the fastest method, but the compression ratio is bad. LZMA is way much slower. However, it compresses backups about 6 times better than LZ4. Brotli is a good trade-off between speed and compression ratio, which is about 3 times better than LZ4.

Encryption

YC_CSE_KMS_KEY_ID

To configure Yandex Cloud KMS key for client-side encryption and decryption. By default, no encryption is used.

YC_SERVICE_ACCOUNT_KEY_FILE

To configure the name of a file containing private key of Yandex Cloud Service Account. If not set a token from the metadata service (http://169.254.169.254) will be used to make API calls to Yandex Cloud KMS.

WALG_LIBSODIUM_KEY

To configure encryption and decryption with libsodium. WAL-G uses an algorithm that only requires a secret key.

WALG_LIBSODIUM_KEY_PATH

Similar to WALG_LIBSODIUM_KEY, but value is the path to the key on file system. The file content will be trimmed from whitespace characters.

WALG_GPG_KEY_ID (alternative form WALE_GPG_KEY_ID) ⚠️ DEPRECATED

To configure GPG key for encryption and decryption. By default, no encryption is used. Public keyring is cached in the file "/.walg_key_cache".

WALG_PGP_KEY

To configure encryption and decryption with OpenPGP standard. You can join multiline key using \n symbols into one line (mostly used in case of daemontools and envdir). Set private key value when you need to execute wal-fetch or backup-fetch command. Set public key value when you need to execute wal-push or backup-push command. Keep in mind that the private key also contains the public key.

WALG_PGP_KEY_PATH

Similar to WALG_PGP_KEY, but value is the path to the key on file system.

WALG_PGP_KEY_PASSPHRASE

If your private key is encrypted with a passphrase, you should set passphrase for decrypt.

Database-specific options

More options are available for the chosen database. See it in Databases

Usage

WAL-G currently supports these commands for all type of databases:

`backup-list`

Lists names and creation time of available backups.

--pretty flag prints list in a table

--json flag prints list in JSON format, pretty-printed if combined with --pretty

--detail flag prints extra backup details, pretty-printed if combined with --pretty, json-encoded if combined with --json

`delete`

Is used to delete backups and WALs before them. By default, delete will perform a dry run. If you want to execute deletion, you have to add --confirm flag at the end of the command. Backups marked as permanent will not be deleted.

delete can operate in four modes: retain, before, everything and target.

retain [FULL|FIND_FULL] %number% [--after %name|time%]

if FULL is specified, keep %number% full backups and everything in the middle. If with --after flag is used keep $number$ the most recent backups and backups made after %name|time% (including).

before [FIND_FULL] %name%

If FIND_FULL is specified, WAL-G will calculate minimum backup needed to keep all deltas alive. If FIND_FULL is not specified, and call can produce orphaned deltas, the call will fail with the list.

everything [FORCE]

target [FIND_FULL] %name% | --target-user-data %data% will delete the backup specified by name or user data.

(Only in Postgres) By default, if delta backup is provided as the target, WAL-G will also delete all the dependant delta backups. If FIND_FULL is specified, WAL-G will delete all backups with the same base backup as the target.

Examples

everything all backups will be deleted (if there are no permanent backups)

everything FORCE all backups, include permanent, will be deleted

retain 5 will fail if 5th is delta

retain FULL 5 will keep 5 full backups and all deltas of them

retain FIND_FULL 5 will find necessary full for 5th and keep everything after it

retain 5 --after 2019-12-12T12:12:12 keep 5 most recent backups and backups made after 2019-12-12 12:12:12

before base_000010000123123123 will fail if base_000010000123123123 is delta

before FIND_FULL base_000010000123123123 will keep everything after base of base_000010000123123123

target base_0000000100000000000000C9 delete the base backup and all dependant delta backups

target --target-user-data "{ \"x\": [3], \"y\": 4 }" delete backup specified by user data

target base_0000000100000000000000C9_D_0000000100000000000000C4 delete delta backup and all dependant delta backups

target FIND_FULL base_0000000100000000000000C9_D_0000000100000000000000C4 delete delta backup and all delta backups with the same base backup

More commands are available for the chosen database engine. See it in Databases

Databases

Development

Installing

It is specified for your type of database.

Testing

WAL-G relies heavily on unit tests. These tests do not require S3 configuration as the upload/download parts are tested using mocked objects. Unit tests can be run using

make unittest

For more information on testing, please consult test, testtools and unittest section in Makefile.

WAL-G will perform a round-trip compression/decompression test that generates a directory for data (e.g., data...), compressed files (e.g., compressed), and extracted files (e.g., extracted). These directories will only get cleaned up if the files in the original data directory match the files in the extracted one.

Test coverage can be obtained using:

make coverage

This command generates coverage.out file and opens HTML representation of the coverage.

Development on Windows

Information about installing and usage

Authors

See also the list of contributors who participated in this project.

License

This project is licensed under the Apache License, Version 2.0, but the lzo support is licensed under GPL 3.0+. Please refer to the LICENSE.md file for more details.

Acknowledgments

WAL-G would not have happened without the support of Citus Data

WAL-G came into existence as a result of the collaboration between a summer engineering intern at Citus, Katie Li, and Daniel Farina, the original author of WAL-E, who currently serves as a principal engineer on the Citus Cloud team. Citus Data also has an open-source extension to Postgres that distributes database queries horizontally to deliver scale and performance.

WAL-G development is supported by Yandex Cloud

Chat

We have a Slack group and Telegram chat to discuss WAL-G usage and development. To join PostgreSQL slack, use invite app.

Owner

https://github.com/wal-g/wal-g

Comments

wal-g hangs on push for gcs
Hello, I have an issue with WAL-G v0.2.9 hanging on pushes to GCS sometimes.

Today, I hopped on my box when disk alerts started going off. I took a look at my wal directory, and I saw a ton of transaction logs that shouldn't be there.

I took a look at ps, and saw that wal-g had been running for over 24 hours on a single wal-push operation

postgres 3253 0.0 0.0 19700 3208 ? S May07 0:00 /bin/bash /usr/local/bin/wal-g-wrapper.sh wal-push pg_wal/000000090000CF82000000D2 postgres 3254 0.1 0.0 3328960 34344 ? Sl May07 1:04 /usr/local/bin/wal-g wal-push pg_wal/000000090000CF82000000D2

To recover, I killed the process. WAL-G fired up again, ran the same command, but this time was successful. This has happened multiple times on multiple servers.

Out of memory error on deployment with millions of files in PGDATA

WAL-G version: 0.2.15 release binary

PostgreSQL version: 9.5

Operating system/version: CentOS 7.8

Hi,

I'm evaluating WAL-G as a replacement for my companys current backup system WAL-E. We have a schema-level style of sharding our tenants, where each of our database servers hosts 4000 customer schemas, spread over 16 databases. This has worked great for scalability, but presents a challenge for many backup systems - since this layout results in a lot of files in postgresql data directory.

Output of ls -1RL /var/lib/pgsql/9.5/data/ | wc -l; 6602805

So above 6 million files, total size 77gb. Around 5 years ago we deployed WAL-E since it was one of the few backup systems that could handle so many files without a problem. However, since WAL-E is no longer maintained we're looking for alternatives. When testing WAL-G, I'm running out of memory when performing a full-backup. It seems to be some kind of go internal memory error, since the memory of the server is not fully utilized. See graphs;

Screenshot from 2020-08-24 12-28-14

Server specs:

12gb of ram
shared_buffers set to 3gb
1800 hugepages (~ 3.6 gb to fit shared_buffers )
vm.overcommit = 2 (not allowed to overcommit)
7.8 gb regular memory limit
CentOS 7
postgres 9.5
wal-g 0.2.15

Chunk of the error log;

INFO: 2020/08/24 11:49:19.354150 Calling pg_stop_backup()
fatal error: runtime: out of memory

runtime stack:
runtime.throw(0x193bfd8, 0x16)
        /home/travis/.gimme/versions/go1.13.9.linux.amd64/src/runtime/panic.go:774 +0x72
runtime.sysMap(0xc110000000, 0x10000000, 0x271a7f8)
        /home/travis/.gimme/versions/go1.13.9.linux.amd64/src/runtime/mem_linux.go:169 +0xc5
runtime.(*mheap).sysAlloc(0x27017c0, 0xfbe0000, 0x150b986, 0x288007575f7)
        /home/travis/.gimme/versions/go1.13.9.linux.amd64/src/runtime/malloc.go:701 +0x1cd
runtime.(*mheap).grow(0x27017c0, 0x7df0, 0xffffffff)
        /home/travis/.gimme/versions/go1.13.9.linux.amd64/src/runtime/mheap.go:1255 +0xa3
runtime.(*mheap).allocSpanLocked(0x27017c0, 0x7df0, 0x271a808, 0x0)
        /home/travis/.gimme/versions/go1.13.9.linux.amd64/src/runtime/mheap.go:1170 +0x266
runtime.(*mheap).alloc_m(0x27017c0, 0x7df0, 0x7f20f7020100, 0x0)
        /home/travis/.gimme/versions/go1.13.9.linux.amd64/src/runtime/mheap.go:1022 +0xc2
runtime.(*mheap).alloc.func1()
        /home/travis/.gimme/versions/go1.13.9.linux.amd64/src/runtime/mheap.go:1093 +0x4c
runtime.(*mheap).alloc(0x27017c0, 0x7df0, 0x7f2109010100, 0x7f20f7027ff8)
        /home/travis/.gimme/versions/go1.13.9.linux.amd64/src/runtime/mheap.go:1092 +0x8a
runtime.largeAlloc(0xfbdfd80, 0x9e0001, 0x7f20f7027ff8)
        /home/travis/.gimme/versions/go1.13.9.linux.amd64/src/runtime/malloc.go:1138 +0x97
runtime.mallocgc.func1()
        /home/travis/.gimme/versions/go1.13.9.linux.amd64/src/runtime/malloc.go:1033 +0x46
runtime.systemstack(0x0)
        /home/travis/.gimme/versions/go1.13.9.linux.amd64/src/runtime/asm_amd64.s:370 +0x66
runtime.mstart()
        /home/travis/.gimme/versions/go1.13.9.linux.amd64/src/runtime/proc.go:1146

Attatched the full log output when performing the backup, including the error as well; walg_log.txt

Target datastore is a local minio S3 instance. Tried WALG_UPLOAD_DISK_CONCURRENCY set to 1 and 4, same result. I see no kernel level OOM-killer logs in syslog, it appears to fail internally.

archive file has wrong size

Hi, we are currently testing wal-g with a small database that writes an entry every minute. When we try to restore the DB, we sometimes get this error:

2018-02-10 05:48:59.478 EST [25962] FATAL:  archive file "000000010000000000000028" has wrong size: 8388608 instead of 16777216

If we start over (rm -rf the data dir, backup-fetch, then recovery), we sometimes manage to fully restore the db, sometimes we get another similar error:

2018-02-10 06:01:29.671 EST [27419] FATAL:  archive file "000000010000000000000027" has wrong size: 8388608 instead of 16777216

The relevant part of the postgresql.conf:

wal_level = logical
archive_mode = on
archive_command = 'envdir /etc/wal-g.d/env /usr/local/bin/wal-g wal-push %p'
archive_timeout = 60

The recovery.conf file when we try to restore the db on a secondary cluster:

restore_command = 'envdir /etc/wal-g.d/env /usr/local/bin/wal-g wal-fetch "%f" "%p"'

We are not using GPG and we only declare basic environment variables:

ls -l /etc/wal-g.d/env/
total 12
-rwxr-x--- 1 root postgres 13 Feb  9 04:32 AWS_REGION
-rwxr-x--- 1 root postgres 16 Feb  9 04:53 PGHOST
-rwxr-x--- 1 root postgres 39 Feb  9 04:32 WALE_S3_PREFIX

backup-fetch fails with "Interpret: copy failed"

Two servers failed to boot up and restore their databases last night. These two servers were booting in different AWS Regions, restoring from the same S3 backup. They both failed on exactly the same file, which indicates that we pushed a corrupt backup in some way.

The backup-push from the DB master had been run by hand, and there were no errors in the log output.

Master Server Details:

Ubuntu 14.04
Postgres 9.6.6
WAL-G 0.1.7

DB Size: ~1.8TB

Server 1 Failure

Notice: /Stage[main]/Wal_g::Db_restore/Exec[wal_g::db_restore]/returns: /base/16400/187655160
Notice: /Stage[main]/Wal_g::Db_restore/Exec[wal_g::db_restore]/returns: /base/16400/187655162
Notice: /Stage[main]/Wal_g::Db_restore/Exec[wal_g::db_restore]/returns: /base/16400/187655164
Notice: /Stage[main]/Wal_g::Db_restore/Exec[wal_g::db_restore]/returns: 2018/04/05 10:56:19 unexpected EOF
Notice: /Stage[main]/Wal_g::Db_restore/Exec[wal_g::db_restore]/returns: Interpret: copy failed
Notice: /Stage[main]/Wal_g::Db_restore/Exec[wal_g::db_restore]/returns: github.com/wal-g/wal-g.(*FileTarInterpreter).Interpret
Notice: /Stage[main]/Wal_g::Db_restore/Exec[wal_g::db_restore]/returns: 	/home/travis/gopath/src/github.com/wal-g/wal-g/tar.go:86
Notice: /Stage[main]/Wal_g::Db_restore/Exec[wal_g::db_restore]/returns: github.com/wal-g/wal-g.extractOne
Notice: /Stage[main]/Wal_g::Db_restore/Exec[wal_g::db_restore]/returns: 	/home/travis/gopath/src/github.com/wal-g/wal-g/extract.go:51
Notice: /Stage[main]/Wal_g::Db_restore/Exec[wal_g::db_restore]/returns: github.com/wal-g/wal-g.ExtractAll.func2.3
Notice: /Stage[main]/Wal_g::Db_restore/Exec[wal_g::db_restore]/returns: 	/home/travis/gopath/src/github.com/wal-g/wal-g/extract.go:156
Notice: /Stage[main]/Wal_g::Db_restore/Exec[wal_g::db_restore]/returns: runtime.goexit
Notice: /Stage[main]/Wal_g::Db_restore/Exec[wal_g::db_restore]/returns: 	/home/travis/.gimme/versions/go1.8.7.linux.amd64/src/runtime/asm_amd64.s:2197
Notice: /Stage[main]/Wal_g::Db_restore/Exec[wal_g::db_restore]/returns: extractOne: Interpret failed
Notice: /Stage[main]/Wal_g::Db_restore/Exec[wal_g::db_restore]/returns: github.com/wal-g/wal-g.extractOne
Notice: /Stage[main]/Wal_g::Db_restore/Exec[wal_g::db_restore]/returns: 	/home/travis/gopath/src/github.com/wal-g/wal-g/extract.go:53
Notice: /Stage[main]/Wal_g::Db_restore/Exec[wal_g::db_restore]/returns: github.com/wal-g/wal-g.ExtractAll.func2.3
Notice: /Stage[main]/Wal_g::Db_restore/Exec[wal_g::db_restore]/returns: 	/home/travis/gopath/src/github.com/wal-g/wal-g/extract.go:156
Notice: /Stage[main]/Wal_g::Db_restore/Exec[wal_g::db_restore]/returns: runtime.goexit
Notice: /Stage[main]/Wal_g::Db_restore/Exec[wal_g::db_restore]/returns: 	/home/travis/.gimme/versions/go1.8.7.linux.amd64/src/runtime/asm_amd64.s:2197
Info: Class[Wal_g::Db_restore]: Unscheduling all events on Class[Wal_g::Db_restore]

Server 2 Failure

Notice: /Stage[main]/Wal_g::Db_restore/Exec[wal_g::db_restore]/returns: /base/16400/187655162
Notice: /Stage[main]/Wal_g::Db_restore/Exec[wal_g::db_restore]/returns: /base/16400/187655164
Notice: /Stage[main]/Wal_g::Db_restore/Exec[wal_g::db_restore]/returns: 2018/04/05 06:56:25 unexpected EOF
Notice: /Stage[main]/Wal_g::Db_restore/Exec[wal_g::db_restore]/returns: Interpret: copy failed
Notice: /Stage[main]/Wal_g::Db_restore/Exec[wal_g::db_restore]/returns: github.com/wal-g/wal-g.(*FileTarInterpreter).Interpret
Notice: /Stage[main]/Wal_g::Db_restore/Exec[wal_g::db_restore]/returns: 	/home/travis/gopath/src/github.com/wal-g/wal-g/tar.go:86
Notice: /Stage[main]/Wal_g::Db_restore/Exec[wal_g::db_restore]/returns: github.com/wal-g/wal-g.extractOne
Notice: /Stage[main]/Wal_g::Db_restore/Exec[wal_g::db_restore]/returns: 	/home/travis/gopath/src/github.com/wal-g/wal-g/extract.go:51
Notice: /Stage[main]/Wal_g::Db_restore/Exec[wal_g::db_restore]/returns: github.com/wal-g/wal-g.ExtractAll.func2.3
Notice: /Stage[main]/Wal_g::Db_restore/Exec[wal_g::db_restore]/returns: 	/home/travis/gopath/src/github.com/wal-g/wal-g/extract.go:156
Notice: /Stage[main]/Wal_g::Db_restore/Exec[wal_g::db_restore]/returns: runtime.goexit
Notice: /Stage[main]/Wal_g::Db_restore/Exec[wal_g::db_restore]/returns: 	/home/travis/.gimme/versions/go1.8.7.linux.amd64/src/runtime/asm_amd64.s:2197
Notice: /Stage[main]/Wal_g::Db_restore/Exec[wal_g::db_restore]/returns: extractOne: Interpret failed
Notice: /Stage[main]/Wal_g::Db_restore/Exec[wal_g::db_restore]/returns: github.com/wal-g/wal-g.extractOne
Notice: /Stage[main]/Wal_g::Db_restore/Exec[wal_g::db_restore]/returns: 	/home/travis/gopath/src/github.com/wal-g/wal-g/extract.go:53
Notice: /Stage[main]/Wal_g::Db_restore/Exec[wal_g::db_restore]/returns: github.com/wal-g/wal-g.ExtractAll.func2.3
Notice: /Stage[main]/Wal_g::Db_restore/Exec[wal_g::db_restore]/returns: 	/home/travis/gopath/src/github.com/wal-g/wal-g/extract.go:156
Notice: /Stage[main]/Wal_g::Db_restore/Exec[wal_g::db_restore]/returns: runtime.goexit
Notice: /Stage[main]/Wal_g::Db_restore/Exec[wal_g::db_restore]/returns: 	/home/travis/.gimme/versions/go1.8.7.linux.amd64/src/runtime/asm_amd64.s:2197
Info: Class[Wal_g::Db_restore]: Unscheduling all events on Class[Wal_g::Db_restore]
STDERR> Error: /Stage[main]/Wal_g::Db_restore/Exec[wal_g::db_restore]/returns: change from 'notrun' to ['0'] failed: '/usr/local/bin/wal-g-restore.sh' returned 1 instead of one of [0]

[ERROR] Failed to fetch backup: Expect pg_control archive, but not found
Hello, I try use wal-g for backup=>restore (wal-g backup-push => wal-g backup-fetch) via S3-compatible service. Backup passed without any errors, but restore stuck on: ERROR: 2020/12/16 10:26:33.016402 Failed to fetch backup: Expect pg_control archive, but not found. But:

ryzand@nice:~$ s3cmd ls s3://postgres-database-backups-wal-g/basebackups_005/base_00000001000106090000003D/tar_partitions/ | grep control 2020-12-15 21:16 506 s3://postgres-database-backups-wal-g/basebackups_005/base_00000001000106090000003D/tar_partitions/pg_control.tar.lz4

Log with DEBUG=DEVEL: https://pastebin.com/SyC78CA2

ENV: wal-g 0.2.19 OS - linux (try 5.4.0 and 5.9.14) S3 - own s3-compatible service DB - postgresql 12

P.S. it's pretty works with 0.1.x version (I try wal-g 0.1.17).

backup-fetch misses a couple of part_* files and hangs at the end leading to corrupt pgdata

Database name

PostgreSQL 14.4 wal-g version:

wal-g version v2.0.0    1eb88a5 2022.05.20_10:45:35     PostgreSQL

Issue description

Incomplete restore leads to corrupt pgdata directory.

Describe your problem

wal-g backup-fetch seems to ignore (or miss?) a couple of part_*.tar.lz4 files. E.g. from files 1-67 it misses 2-5,7,8,9. The missing files are different every time we try.

Also at the end (that is, after the last file extracted) the process hangs indefinitely.

Please provide steps to reproduce

envdir "/run/etc/wal-e.d/env" ./strace -s 128 -v -f -T -o log/walg.strace wal-g backup-fetch

We operate our own S3 on premise. The restore runs in a pod on Kubernetes.

Whats interesting: if we install the Minio client mc and download the files with it, everything can be downloaded. So it seems the S3 is not the culprit here.

Please add config and wal-g stdout/stderr logs for debug purpose

wal-g output w/o DEVEL:

# envdir "/run/etc/wal-e.d/env" ./strace -s 128 -v -f -T -o log/walg.strace wal-g backup-fetch /home/postgres/pgdata/tmp/restore LATEST                               INFO: 2022/08/19 11:28:09.703976 Selecting the latest backup...
INFO: 2022/08/19 11:28:09.776681 LATEST backup is: 'base_00000014000000A000000036'                                                                          
INFO: 2022/08/19 11:28:09.929715 AO files metadata was not found. Skipping the AO segments unpacking.                                                                                                                                                                                                                   
INFO: 2022/08/19 11:28:22.467885 Finished extraction of part_009.tar.lz4                                                                                    
INFO: 2022/08/19 11:28:28.828507 Finished extraction of part_003.tar.lz4                                                                                    
INFO: 2022/08/19 11:28:31.151378 Finished extraction of part_006.tar.lz4                                                                                                                                                                                                                                                
INFO: 2022/08/19 11:28:33.939739 Finished extraction of part_011.tar.lz4                                                                                                                                                                                                                                                
INFO: 2022/08/19 11:28:40.641426 Finished extraction of part_012.tar.lz4                                                                                    
INFO: 2022/08/19 11:28:45.938125 Finished extraction of part_013.tar.lz4                                                                                    
INFO: 2022/08/19 11:28:57.101820 Finished extraction of part_015.tar.lz4                                                                                    
INFO: 2022/08/19 11:29:01.312001 Finished extraction of part_001.tar.lz4                                                                                    
INFO: 2022/08/19 11:29:01.675549 Finished extraction of part_014.tar.lz4                                                                                    
INFO: 2022/08/19 11:29:02.746148 Finished extraction of part_016.tar.lz4                                                                                    
INFO: 2022/08/19 11:29:12.603862 Finished extraction of part_017.tar.lz4                                                                                    
INFO: 2022/08/19 11:29:16.324988 Finished extraction of part_018.tar.lz4                                                                                    
INFO: 2022/08/19 11:29:22.657113 Finished extraction of part_019.tar.lz4                                                                                    
INFO: 2022/08/19 11:29:24.696857 Finished extraction of part_020.tar.lz4                                                                                    
INFO: 2022/08/19 11:29:35.063565 Finished extraction of part_021.tar.lz4                                                                                    
INFO: 2022/08/19 11:29:37.435743 Finished extraction of part_022.tar.lz4                                                                                    
INFO: 2022/08/19 11:29:43.176691 Finished extraction of part_023.tar.lz4                                                                                    
INFO: 2022/08/19 11:29:47.625606 Finished extraction of part_024.tar.lz4                                                                                    
INFO: 2022/08/19 11:30:14.058504 Finished extraction of part_026.tar.lz4                                                                                    
INFO: 2022/08/19 11:30:31.390589 Finished extraction of part_029.tar.lz4                                                                                    
INFO: 2022/08/19 11:30:45.917514 Finished extraction of part_030.tar.lz4                                                                                    
INFO: 2022/08/19 11:31:12.932315 Finished extraction of part_031.tar.lz4                                                                                    
INFO: 2022/08/19 11:31:24.620205 Finished extraction of part_032.tar.lz4                                                                                    
INFO: 2022/08/19 11:31:39.158482 Finished extraction of part_033.tar.lz4                                                                                    
INFO: 2022/08/19 11:31:49.923850 Finished extraction of part_034.tar.lz4                                                                                    
INFO: 2022/08/19 11:32:02.087891 Finished extraction of part_035.tar.lz4                                                                                    
INFO: 2022/08/19 11:32:39.218819 Finished extraction of part_036.tar.lz4                                                                                    
INFO: 2022/08/19 11:33:01.869772 Finished extraction of part_037.tar.lz4                                                                                    
INFO: 2022/08/19 11:33:28.000880 Finished extraction of part_038.tar.lz4                                                                                    
INFO: 2022/08/19 11:33:47.561752 Finished extraction of part_039.tar.lz4                                                                                    
INFO: 2022/08/19 11:34:04.776766 Finished extraction of part_040.tar.lz4                                                                                    
INFO: 2022/08/19 11:34:23.333978 Finished extraction of part_041.tar.lz4                                                                                    
INFO: 2022/08/19 11:34:39.226808 Finished extraction of part_042.tar.lz4                                                                                    
INFO: 2022/08/19 11:35:07.478093 Finished extraction of part_043.tar.lz4                                                                                    
INFO: 2022/08/19 11:35:28.517997 Finished extraction of part_044.tar.lz4                                                                                    
INFO: 2022/08/19 11:35:53.070435 Finished extraction of part_045.tar.lz4                                                                                    
INFO: 2022/08/19 11:36:10.519795 Finished extraction of part_046.tar.lz4                                                                                    
INFO: 2022/08/19 11:36:19.153122 Finished extraction of part_047.tar.lz4                                                                                    
INFO: 2022/08/19 11:36:42.117938 Finished extraction of part_048.tar.lz4                                                                                    
INFO: 2022/08/19 11:37:09.240082 Finished extraction of part_049.tar.lz4                                                                                    
INFO: 2022/08/19 11:37:32.350660 Finished extraction of part_050.tar.lz4                                                                                    
INFO: 2022/08/19 11:37:46.413640 Finished extraction of part_051.tar.lz4                                                                                    
INFO: 2022/08/19 11:38:07.614133 Finished extraction of part_052.tar.lz4                                                                                    
INFO: 2022/08/19 11:38:15.210508 Finished extraction of part_053.tar.lz4                                                                                    
INFO: 2022/08/19 11:38:35.472436 Finished extraction of part_054.tar.lz4                                                                                    
INFO: 2022/08/19 11:38:50.261030 Finished extraction of part_055.tar.lz4                                                                                    
INFO: 2022/08/19 11:39:04.117539 Finished extraction of part_056.tar.lz4                                                                                    
INFO: 2022/08/19 11:39:16.478275 Finished extraction of part_057.tar.lz4                                                                                    
INFO: 2022/08/19 11:39:33.624068 Finished extraction of part_058.tar.lz4                                                                                    
INFO: 2022/08/19 11:39:42.833370 Finished extraction of part_059.tar.lz4                                                                                    
INFO: 2022/08/19 11:39:52.400737 Finished extraction of part_060.tar.lz4                                                                                    
INFO: 2022/08/19 11:40:01.136057 Finished extraction of part_061.tar.lz4                                                                                    
INFO: 2022/08/19 11:40:09.087877 Finished extraction of part_062.tar.lz4                                                                                    
INFO: 2022/08/19 11:40:34.432220 Finished extraction of part_063.tar.lz4                                                                                    
INFO: 2022/08/19 11:41:01.976348 Finished extraction of part_064.tar.lz4                                                                                    
INFO: 2022/08/19 11:41:02.239310 Finished extraction of part_065.tar.lz4                                                                                    
INFO: 2022/08/19 11:41:02.285595 Finished extraction of part_067.tar.lz4 
.. # hangs here

DEVEL Logs are here strace output after last stdout output are here

Can't restore due to missing archive, but archive never existed in first place

We are using WAL-G on 4 different database clusters. We're having issues restoring the database from this one cluster after an upgrade to PG10. This archive that it's looking for doesn't exist in S3.

The WAL log it's looking for is 000000010000D6E5000000C1 and it's true, it doesn't exist.

In fact, that whole series doesn't exist. I have no files that start with 000000010000D6 but I do have a ton with 000000010000D7

I'm wondering if something with the timeline is getting messed up during backups or during restore somehow, but I'm not sure where to look first.

2018-09-22 13:57:51.587 GMT [2278] LOG:  starting point-in-time recovery to 2018-09-22 07:00:00+00
BUCKET: bucketname
SERVER: 
2018/09/22 08:57:52 Archive '%s' does not exist 000000010000D6E5000000C1
github.com/wal-g/wal-g.DownloadAndDecompressWALFile
	/home/travis/gopath/src/github.com/wal-g/wal-g/commands.go:590
github.com/wal-g/wal-g.HandleWALFetch
	/home/travis/gopath/src/github.com/wal-g/wal-g/commands.go:515
main.main
	/home/travis/gopath/src/github.com/wal-g/wal-g/cmd/wal-g/main.go:131
runtime.main
	/home/travis/.gimme/versions/go1.10.4.linux.amd64/src/runtime/proc.go:198
runtime.goexit
	/home/travis/.gimme/versions/go1.10.4.linux.amd64/src/runtime/asm_amd64.s:2361
2018-09-22 13:57:52.014 GMT [2278] LOG:  invalid checkpoint record
2018-09-22 13:57:52.014 GMT [2278] FATAL:  could not locate required checkpoint record
2018-09-22 13:57:52.014 GMT [2278] HINT:  If you are not restoring from a backup, try removing the file "/var/lib/postgresql/10/main/backup_label".```

wal-g delete fails on WAL-E backups

Database name

PostgreSQL 12

Issue description

Describe your problem

wal-g delete retain N fails if any backups are found from WAL-E

Please provide steps to reproduce

wal-e backup-push /var/lib/postgresql/12/main
wal-e backup-push /var/lib/postgresql/12/main
wal-e backup-push /var/lib/postgresql/12/main
wal-e backup-push /var/lib/postgresql/12/main
wal-g delete retain 3

Please add config and wal-g stdout/stderr logs for debug purpose

{
  "PGHOST": "/var/run/postgresql",
  "PGUSER": "postgres",
  "WALG_COMPRESSION_METHOD": "brotli",
  "WALG_DOWNLOAD_CONCURRENCY": 2,
  "WALG_FILE_PREFIX": "/mnt/pg_backup",
  "WALG_PREFETCH_DIR": "/var/lib/postgresql/prefetch",
  "WALG_PREVENT_WAL_OVERWRITE": true,
  "WALG_UPLOAD_CONCURRENCY": 1
}

If you can, provide logs

INFO: 2022/06/10 16:31:16.361203 retrieving permanent objects
ERROR: 2022/06/10 16:31:16.374858 failed to fetch backup meta for backup base_000000020000016B0000005E_00000040 with error failed to unmarshal metadata: object '/mnt/pg_backup/basebackups_005/base_000000020000016B0000005E_00000040/metadata.json' not found in storage
github.com/wal-g/wal-g/pkg/storages/storage.NewObjectNotFoundError
        /home/runner/work/wal-g/wal-g/pkg/storages/storage/errors.go:15
github.com/wal-g/wal-g/pkg/storages/fs.(*Folder).ReadObject
        /home/runner/work/wal-g/wal-g/pkg/storages/fs/folder.go:100
github.com/wal-g/wal-g/internal.(*StorageReaderMaker).Reader
        /home/runner/work/wal-g/wal-g/internal/storage_reader_maker.go:31
github.com/wal-g/wal-g/internal.FetchDto
        /home/runner/work/wal-g/wal-g/internal/backup.go:86
github.com/wal-g/wal-g/internal.(*Backup).FetchMetadata
        /home/runner/work/wal-g/wal-g/internal/backup.go:72
github.com/wal-g/wal-g/internal/databases/postgres.(*Backup).FetchMeta
        /home/runner/work/wal-g/wal-g/internal/databases/postgres/backup.go:177
github.com/wal-g/wal-g/internal/databases/postgres.GetPermanentBackupsAndWals
        /home/runner/work/wal-g/wal-g/internal/databases/postgres/delete_util.go:23
github.com/wal-g/wal-g/cmd/pg.runDeleteRetain
        /home/runner/work/wal-g/wal-g/cmd/pg/delete.go:79
github.com/spf13/cobra.(*Command).execute
        /home/runner/work/wal-g/wal-g/vendor/github.com/spf13/cobra/command.go:860
github.com/spf13/cobra.(*Command).ExecuteC
        /home/runner/work/wal-g/wal-g/vendor/github.com/spf13/cobra/command.go:974
github.com/spf13/cobra.(*Command).Execute
        /home/runner/work/wal-g/wal-g/vendor/github.com/spf13/cobra/command.go:902
github.com/wal-g/wal-g/cmd/pg.Execute
        /home/runner/work/wal-g/wal-g/cmd/pg/pg.go:45
main.main
        /home/runner/work/wal-g/wal-g/main/pg/main.go:8
runtime.main
        /opt/hostedtoolcache/go/1.17.10/x64/src/runtime/proc.go:255
runtime.goexit
        /opt/hostedtoolcache/go/1.17.10/x64/src/runtime/asm_amd64.s:1581, ignoring...
ERROR: 2022/06/10 16:31:16.376487 failed to fetch backup meta for backup base_000000020000016B0000007A_00000040 with error failed to unmarshal metadata: object '/mnt/pg_backup/basebackups_005/base_000000020000016B0000007A_00000040/metadata.json' not found in storage
github.com/wal-g/wal-g/pkg/storages/storage.NewObjectNotFoundError
        /home/runner/work/wal-g/wal-g/pkg/storages/storage/errors.go:15
github.com/wal-g/wal-g/pkg/storages/fs.(*Folder).ReadObject
        /home/runner/work/wal-g/wal-g/pkg/storages/fs/folder.go:100
github.com/wal-g/wal-g/internal.(*StorageReaderMaker).Reader
        /home/runner/work/wal-g/wal-g/internal/storage_reader_maker.go:31
github.com/wal-g/wal-g/internal.FetchDto
        /home/runner/work/wal-g/wal-g/internal/backup.go:86
github.com/wal-g/wal-g/internal.(*Backup).FetchMetadata
        /home/runner/work/wal-g/wal-g/internal/backup.go:72
github.com/wal-g/wal-g/internal/databases/postgres.(*Backup).FetchMeta
        /home/runner/work/wal-g/wal-g/internal/databases/postgres/backup.go:177
github.com/wal-g/wal-g/internal/databases/postgres.GetPermanentBackupsAndWals
        /home/runner/work/wal-g/wal-g/internal/databases/postgres/delete_util.go:23
github.com/wal-g/wal-g/cmd/pg.runDeleteRetain
        /home/runner/work/wal-g/wal-g/cmd/pg/delete.go:79
github.com/spf13/cobra.(*Command).execute
        /home/runner/work/wal-g/wal-g/vendor/github.com/spf13/cobra/command.go:860
github.com/spf13/cobra.(*Command).ExecuteC
        /home/runner/work/wal-g/wal-g/vendor/github.com/spf13/cobra/command.go:974
github.com/spf13/cobra.(*Command).Execute
        /home/runner/work/wal-g/wal-g/vendor/github.com/spf13/cobra/command.go:902
github.com/wal-g/wal-g/cmd/pg.Execute
        /home/runner/work/wal-g/wal-g/cmd/pg/pg.go:45
main.main
        /home/runner/work/wal-g/wal-g/main/pg/main.go:8
runtime.main
        /opt/hostedtoolcache/go/1.17.10/x64/src/runtime/proc.go:255
runtime.goexit
        /opt/hostedtoolcache/go/1.17.10/x64/src/runtime/asm_amd64.s:1581, ignoring...
ERROR: 2022/06/10 16:31:16.377954 failed to fetch backup meta for backup base_000000020000016B00000080_00000040 with error failed to unmarshal metadata: object '/mnt/pg_backup/basebackups_005/base_000000020000016B00000080_00000040/metadata.json' not found in storage
github.com/wal-g/wal-g/pkg/storages/storage.NewObjectNotFoundError
        /home/runner/work/wal-g/wal-g/pkg/storages/storage/errors.go:15
github.com/wal-g/wal-g/pkg/storages/fs.(*Folder).ReadObject
        /home/runner/work/wal-g/wal-g/pkg/storages/fs/folder.go:100
github.com/wal-g/wal-g/internal.(*StorageReaderMaker).Reader
        /home/runner/work/wal-g/wal-g/internal/storage_reader_maker.go:31
github.com/wal-g/wal-g/internal.FetchDto
        /home/runner/work/wal-g/wal-g/internal/backup.go:86
github.com/wal-g/wal-g/internal.(*Backup).FetchMetadata
        /home/runner/work/wal-g/wal-g/internal/backup.go:72
github.com/wal-g/wal-g/internal/databases/postgres.(*Backup).FetchMeta
        /home/runner/work/wal-g/wal-g/internal/databases/postgres/backup.go:177
github.com/wal-g/wal-g/internal/databases/postgres.GetPermanentBackupsAndWals
        /home/runner/work/wal-g/wal-g/internal/databases/postgres/delete_util.go:23
github.com/wal-g/wal-g/cmd/pg.runDeleteRetain
        /home/runner/work/wal-g/wal-g/cmd/pg/delete.go:79
github.com/spf13/cobra.(*Command).execute
        /home/runner/work/wal-g/wal-g/vendor/github.com/spf13/cobra/command.go:860
github.com/spf13/cobra.(*Command).ExecuteC
        /home/runner/work/wal-g/wal-g/vendor/github.com/spf13/cobra/command.go:974
github.com/spf13/cobra.(*Command).Execute
        /home/runner/work/wal-g/wal-g/vendor/github.com/spf13/cobra/command.go:902
github.com/wal-g/wal-g/cmd/pg.Execute
        /home/runner/work/wal-g/wal-g/cmd/pg/pg.go:45
main.main
        /home/runner/work/wal-g/wal-g/main/pg/main.go:8
runtime.main
        /opt/hostedtoolcache/go/1.17.10/x64/src/runtime/proc.go:255
runtime.goexit
        /opt/hostedtoolcache/go/1.17.10/x64/src/runtime/asm_amd64.s:1581, ignoring...
ERROR: 2022/06/10 16:31:16.379419 failed to fetch backup meta for backup base_000000020000016B00000082_00000040 with error failed to unmarshal metadata: object '/mnt/pg_backup/basebackups_005/base_000000020000016B00000082_00000040/metadata.json' not found in storage
github.com/wal-g/wal-g/pkg/storages/storage.NewObjectNotFoundError
        /home/runner/work/wal-g/wal-g/pkg/storages/storage/errors.go:15
github.com/wal-g/wal-g/pkg/storages/fs.(*Folder).ReadObject
        /home/runner/work/wal-g/wal-g/pkg/storages/fs/folder.go:100
github.com/wal-g/wal-g/internal.(*StorageReaderMaker).Reader
        /home/runner/work/wal-g/wal-g/internal/storage_reader_maker.go:31
github.com/wal-g/wal-g/internal.FetchDto
        /home/runner/work/wal-g/wal-g/internal/backup.go:86
github.com/wal-g/wal-g/internal.(*Backup).FetchMetadata
        /home/runner/work/wal-g/wal-g/internal/backup.go:72
github.com/wal-g/wal-g/internal/databases/postgres.(*Backup).FetchMeta
        /home/runner/work/wal-g/wal-g/internal/databases/postgres/backup.go:177
github.com/wal-g/wal-g/internal/databases/postgres.GetPermanentBackupsAndWals
        /home/runner/work/wal-g/wal-g/internal/databases/postgres/delete_util.go:23
github.com/wal-g/wal-g/cmd/pg.runDeleteRetain
        /home/runner/work/wal-g/wal-g/cmd/pg/delete.go:79
github.com/spf13/cobra.(*Command).execute
        /home/runner/work/wal-g/wal-g/vendor/github.com/spf13/cobra/command.go:860
github.com/spf13/cobra.(*Command).ExecuteC
        /home/runner/work/wal-g/wal-g/vendor/github.com/spf13/cobra/command.go:974
github.com/spf13/cobra.(*Command).Execute
        /home/runner/work/wal-g/wal-g/vendor/github.com/spf13/cobra/command.go:902
github.com/wal-g/wal-g/cmd/pg.Execute
        /home/runner/work/wal-g/wal-g/cmd/pg/pg.go:45
main.main
        /home/runner/work/wal-g/wal-g/main/pg/main.go:8
runtime.main
        /opt/hostedtoolcache/go/1.17.10/x64/src/runtime/proc.go:255
runtime.goexit
        /opt/hostedtoolcache/go/1.17.10/x64/src/runtime/asm_amd64.s:1581, ignoring...
ERROR: 2022/06/10 16:31:16.381343 object '/mnt/pg_backup/basebackups_005/base_000000020000016B0000005E_backup_stop_sentinel.json' not found in storage
github.com/wal-g/wal-g/pkg/storages/storage.NewObjectNotFoundError
        /home/runner/work/wal-g/wal-g/pkg/storages/storage/errors.go:15
github.com/wal-g/wal-g/pkg/storages/fs.(*Folder).ReadObject
        /home/runner/work/wal-g/wal-g/pkg/storages/fs/folder.go:100
github.com/wal-g/wal-g/internal.(*StorageReaderMaker).Reader
        /home/runner/work/wal-g/wal-g/internal/storage_reader_maker.go:31
github.com/wal-g/wal-g/internal.FetchDto
        /home/runner/work/wal-g/wal-g/internal/backup.go:86
github.com/wal-g/wal-g/internal.(*Backup).FetchSentinel
        /home/runner/work/wal-g/wal-g/internal/backup.go:67
github.com/wal-g/wal-g/internal/databases/postgres.(*Backup).GetSentinel
        /home/runner/work/wal-g/wal-g/internal/databases/postgres/backup.go:86
github.com/wal-g/wal-g/internal/databases/postgres.getIncrementInfo
        /home/runner/work/wal-g/wal-g/internal/databases/postgres/delete.go:197
github.com/wal-g/wal-g/internal/databases/postgres.makeBackupObjects
        /home/runner/work/wal-g/wal-g/internal/databases/postgres/delete.go:109
github.com/wal-g/wal-g/internal/databases/postgres.NewDeleteHandler
        /home/runner/work/wal-g/wal-g/internal/databases/postgres/delete.go:45
github.com/wal-g/wal-g/cmd/pg.runDeleteRetain
        /home/runner/work/wal-g/wal-g/cmd/pg/delete.go:81
github.com/spf13/cobra.(*Command).execute
        /home/runner/work/wal-g/wal-g/vendor/github.com/spf13/cobra/command.go:860
github.com/spf13/cobra.(*Command).ExecuteC
        /home/runner/work/wal-g/wal-g/vendor/github.com/spf13/cobra/command.go:974
github.com/spf13/cobra.(*Command).Execute
        /home/runner/work/wal-g/wal-g/vendor/github.com/spf13/cobra/command.go:902
github.com/wal-g/wal-g/cmd/pg.Execute
        /home/runner/work/wal-g/wal-g/cmd/pg/pg.go:45
main.main
        /home/runner/work/wal-g/wal-g/main/pg/main.go:8
runtime.main
        /opt/hostedtoolcache/go/1.17.10/x64/src/runtime/proc.go:255
runtime.goexit
        /opt/hostedtoolcache/go/1.17.10/x64/src/runtime/asm_amd64.s:1581

Unable to create a full basebackup with backup_push

I've been using wal-e to successfully create backups of a GitLab CE server to an S3 clone. I just tried to switch to wal-g, and while I managed to get it to talk to the S3 clone, it doesn't seem to upload a full backup.

Here the command I am running:

sudo -u gitlab-psql -s /bin/sh -c "envdir /etc/wal-g.d/env wal-g backup-push /var/opt/gitlab/postgresql/data"

And below its output.

It looks like part 1 is never finished, part 2 is omitted altogether, and only part 3 and part 4 get processed correctly, resulting in a tiny incomplete basebackups_005 folder that contains only 219kb of data.

What am I doing wrong?

BUCKET: gitlab.*****.com
SERVER: wal-g
Walking ...
Starting part 1 ...

/PG_VERSION
/base
/base/1
[...]
/base/16385/PG_VERSION
/base/16385/pg_filenode.map
/base/16385/pg_internal.init
/global
/global/1136
[...]
/global/pg_filenode.map
/global/pg_internal.init
/pg_clog
/pg_clog/0000
/pg_commit_ts
/pg_dynshmem
/pg_hba.conf
/pg_ident.conf
/pg_logical
/pg_logical/mappings
/pg_logical/snapshots
/pg_multixact
/pg_multixact/members
/pg_multixact/members/0000
/pg_multixact/offsets
/pg_multixact/offsets/0000
/pg_notify
/pg_replslot
/pg_serial
/pg_snapshots
/pg_stat
/pg_stat_tmp
/pg_subtrans
/pg_tblspc
/pg_twophase
/pg_xlog
/postgresql.auto.conf
/postgresql.conf
/runtime.conf
/server.crt
/server.key
Starting part 3 ...
/global/pg_control
Finished writing part 3.
Starting part 4 ...
backup_label
tablespace_map
Finished writing part 4.
Uploaded 4 compressed tar Files.

Some LZOP archives don't work

I'm still taking this apart but it can be related to #22.

I have this backup that has hundreds of archives, but one of those archives causes a systematic crash, whereas lzop seems fine with it. There's not much to do but for me to go through it with a fine-tooth comb, but, FYI.

cc @x4m

DecompressLzo: write to pipe failed

Versions

CentOS 7.3 wal-g v0.1.2 wal-e 1.0.3 (creator of source basebackup)

Problem

Two attempts to backup-fetch a ~1TB basebackup have resulted in wal-g failing with the following stack trace:

base/16417/12983_vm
base/16417/27620292
base/16417/10323582
base/16417/10324516
base/16417/33825612_fsm
2017/08/29 20:07:43 DecompressLzo: write to pipe failed
github.com/wal-g/wal-g.DecompressLzo
        /home/travis/gopath/src/github.com/wal-g/wal-g/decompress.go:126
github.com/wal-g/wal-g.tarHandler
        /home/travis/gopath/src/github.com/wal-g/wal-g/extract.go:66
github.com/wal-g/wal-g.ExtractAll.func2.2
        /home/travis/gopath/src/github.com/wal-g/wal-g/extract.go:138
runtime.goexit
        /home/travis/.gimme/versions/go1.8.3.linux.amd64/src/runtime/asm_amd64.s:2197
ExtractAll: lzo decompress failed
github.com/wal-g/wal-g.tarHandler
        /home/travis/gopath/src/github.com/wal-g/wal-g/extract.go:68
github.com/wal-g/wal-g.ExtractAll.func2.2
        /home/travis/gopath/src/github.com/wal-g/wal-g/extract.go:138
runtime.goexit
        /home/travis/.gimme/versions/go1.8.3.linux.amd64/src/runtime/asm_amd64.s:2197

In both cases, wal-g appeared to be near the end of the restore (over 1TB of data was written to the restore directory) and failed with the same trace. After inspecting the restore and attempting to start postgres, I can confirm that the restore is indeed incomplete.

The basebackup was taken with wal-e 1.0.3, which was also able to restore the same backup without any issues.

how to use obs or oss replace aws s3?

Database name

greenplum 6.12

Issue description

Describe your problem

example is aws s3, but we use aliyun oss or huawei cloud obs, then how to change the config file:

PGDATABASE: "postgres"
WALE_S3_PREFIX: "s3://some/s3/prefix/"
WALG_NETWORK_RATE_LIMIT: 8388608
PGPASSFILE: "/home/gpadmin/.pgpass"
WALG_GP_LOGS_DIR: "/var/log/greenplum"
AWS_ACCESS_KEY_ID: "aws_access_key_id"
WALG_UPLOAD_CONCURRENCY: 5
WALG_PGP_KEY_PATH: "/path/to/PGP_KEY"
WALG_DOWNLOAD_CONCURRENCY: 5
WALE_GPG_KEY_ID: "gpg_key_id"
WALG_DISK_RATE_LIMIT: 167772160
PGUSER: "gpadmin"
GOMAXPROCS: 6
PGHOST: "localhost"
AWS_ENDPOINT: "https://s3-endpoint.host.name"
AWS_SECRET_ACCESS_KEY: "aws_secret_access_key"
WALG_COMPRESSION_METHOD: "brotli"

Please provide steps to reproduce

// it can really help

Build wal-g on CentOS 7 and CentOS 8 Stream

OS:

cat /etc/redhat-release 
CentOS Stream release 8

Install golang git brotli-devel lzo-devel cmake

yum -y install  epel-release
yum -y install golang git brotli-devel lzo-devel cmake

GOPATH

mkdir /root/go
export GOPATH=/root/go

Clone

git clone https://github.com/wal-g/wal-g.git
cd wal-g/

go get

go get github.com/wal-g/wal-g
go: github.com/wal-g/wal-g: invalid github.com import path "github.com/wal-g"

go get -v github.com/wal-g/wal-g
go: github.com/wal-g/wal-g: invalid github.com import path "github.com/wal-g"

go install

go install github.com/wal-g/[email protected]
go: github.com/wal-g/[email protected]: github.com/wal-g/[email protected]: invalid version: module contains a go.mod file, so module path must match major version ("github.com/wal-g/wal-g/v2")

go install github.com/wal-g/wal-g
no required module provides package github.com/wal-g/wal-g; to add it:
	go get github.com/wal-g/wal-g

Implementation of the new binlog-server command

Database name

mysql

Pull request description

wal-g runs MySQL implementation, and when a slave is connected to it, downloads binlogs from the repository and gives them over the replication protocol.

Currently requires the use of the library version provided in this pull-request: https://github.com/go-mysql-org/go-mysql/pull/759
Add sorting by creation time when --detailed flag is not specified.
Pull request description

[Issue] (https://github.com/wal-g/wal-g/issues/1108) Added GenericMetaInteractors for:

Mongo

Redis

Fdb

SqlServers

Interactors are used for fetching backup StartTime to sort backups by creation time. Added sorting by creation time in DefaultHandleBackupList when --detailed flag is not specified.
AWS config file only supported via undocumented environment variable
Database name

PostgreSQL

Issue description

Using an AWS config file to provide a profile requires setting the environment variable AWS_SDK_LOAD_CONFIG=1 which is not documented in wal-g.

Describe your problem

Ideally either the session should be created to use the config file by default, or the wal-g documentation should mention this environment variable. The wal-g s3 docs currently say

WAL-G determines AWS credentials like other AWS tools.

The linked to docs are for the aws cli. These docs do not mention the environment variable as this is specific to the go aws sdk, and the cli is not implemented in go. Setting AWS_CONFIG_FILE as these docs describe does not result in the config file being loaded. The config file is different than the credentials file and is necessary when specifying some fields such as external_id.

I believe session creation similar to the following would make the statement about loading credentials like other tools more accurate.

session.NewSessionWithOptions{session.Options{ SharedConfigState: session.SharedConfigEnable, }}

Please provide steps to reproduce

You can probably reproduce this with a simpler setup, This is how our environment was setup when we ran into this.

This example needs an ec2 instance, 2 iam roles (instance profile, restore role) and an s3 bucket.

The ec2 instance has one iam role as it's instance profile.

The instance iam role has permission to assume the restore iam role.

The restore iam role has permission to list and read from the s3 bucket.

The s3 bucket allows the restore role to list and read from it.

Create an aws config file with a profile describing the restore role. Make sure the user running wal-g has perms to read this file (chmod 0644).

[profile default] credential_source = Ec2InstanceMetadata [profile walg] credential_source = Ec2InstanceMetadata session_name = mgp-dev_inventory-replica-db_1 role_arn = arn:aws:iam::######:role/postgres-restore external_id = <redacted> duration_seconds = 3600 region=us-east-2

Set the following env vars

AWS_CONFIG_FILE=<config file location> AWS_PROFILE=walg AWS_REGION=us-east-2 WALG_S3_PREFIX=<bucket/path>

Run aws s3 ls s3://<bucket/path>/basebackups_005/. This will successfully list the bucket contents. Run wal-g backup-list and the result will be an access denied error.

Set AWS_SDK_LOAD_CONFIG=1 and re-run the wal-g command, and it will work.

WAL-G is an archival restoration tool for PostgreSQL, MySQL/MariaDB, and MS SQL Server (beta for MongoDB and Redis).

WAL-G

Installation

Configuration

Storage

Compression

Encryption

Database-specific options

Usage

backup-list

delete

Examples

Databases

PostgreSQL

MySQL/MariaDB

SQLServer

Mongo [Beta]

FoundationDB [Work in progress]

Redis [Beta]

Development

Installing

Testing

Development on Windows

Authors

License

Acknowledgments

Chat

Owner

Comments

wal-g hangs on push for gcs

Out of memory error on deployment with millions of files in PGDATA

archive file has wrong size

backup-fetch fails with "Interpret: copy failed"

Server 1 Failure

Server 2 Failure

[ERROR] Failed to fetch backup: Expect pg_control archive, but not found

backup-fetch misses a couple of part_* files and hangs at the end leading to corrupt pgdata

Database name

Issue description

Describe your problem

Please provide steps to reproduce

Please add config and wal-g stdout/stderr logs for debug purpose

Can't restore due to missing archive, but archive never existed in first place

wal-g delete fails on WAL-E backups

Database name

Issue description

Describe your problem

Please provide steps to reproduce

Please add config and wal-g stdout/stderr logs for debug purpose

Unable to create a full basebackup with backup_push

Some LZOP archives don't work

DecompressLzo: write to pipe failed

Versions

Problem

how to use obs or oss replace aws s3?

Database name

Issue description

Describe your problem

Please provide steps to reproduce

Build wal-g on CentOS 7 and CentOS 8 Stream

Implementation of the new binlog-server command

Database name

Pull request description

Add sorting by creation time when --detailed flag is not specified.

Pull request description

AWS config file only supported via undocumented environment variable

Database name

Issue description

Describe your problem

Please provide steps to reproduce

Related tags

Support MySQL or MariaDB for gopsql/psql and gopsql/db

Redis-shake is a tool for synchronizing data between two redis databases. Redis-shake是一个用于在两个redis之间同步数据的工具，满足用户非常灵活的同步、迁移需求。

Devcloud-go provides a sql-driver for mysql which named devspore driver and a redis client which named devspore client,

A tool to run queries in defined frequency and expose the count as prometheus metrics. Supports MongoDB and SQL

Bifrost ---- 面向生产环境的 MySQL 同步到Redis,MongoDB,ClickHouse,MySQL等服务的异构中间件

Go-Postgresql-Query-Builder - A query builder for Postgresql in Go

Interactive client for PostgreSQL and MySQL

Interactive terminal user interface and CLI for database connections. MySQL, PostgreSQL. More to come.

Dumpling is a fast, easy-to-use tool written by Go for dumping data from the database(MySQL, TiDB...) to local/cloud(S3, GCP...) in multifarious formats(SQL, CSV...).

`backup-list`

`delete`