Currently, litestream replicas a database to cold storage (file or s3). It would be nice to provide a way to replicate streams of WAL frames to other live nodes to serve as read replicas.

This a network replication endpoint (probably http) as well as figuring out how to apply WAL writes to a live database.

I am not too familiar with how cross-compilation works in this case, but I am wondering if it'd be too much work?

If a build VM is needed then I am willing to provide one :)

Hi!

First of all, thank you for a great piece of software.

It might be a terrible to try to run litestream and SQLite on Cloud Run. so feel free to close this issue 👍

Problem

I'm trying to run get the litestream+s6 example working on Google Cloud Run (from https://github.com/benbjohnson/litestream-s6-example). If I run the container on Cloud Run, I get the error below

s3: restoring snapshot 9e3296261a575b63/00000000 to /tmp/db.tmp
s3: restoring wal files: generation=9e3296261a575b63 index=[00000000,00000001]
s3: downloaded wal 9e3296261a575b63/00000001 elapsed=278.371883ms
s3: downloaded wal 9e3296261a575b63/00000000 elapsed=355.99923ms
cannot apply wal: disk I/O error: invalid argument
[cont-init.d] 00-litestream: exited 1.

If I run the container locally, it works well

...
s3: downloaded wal 9e3296261a575b63/00000000 elapsed=832.108ms
s3: applied wal 9e3296261a575b63/00000000 elapsed=11.0846ms
s3: applied wal 9e3296261a575b63/00000001 elapsed=10.6193ms
s3: renaming database from temporary location
[cont-init.d] 00-litestream: exited 0.

Any idea what might be the problem?

I'm guessing this might be due to Cloud Run's in memory file system, but I don't know how to fix it (https://cloud.google.com/appengine/docs/standard/go/using-temp-files).

I'm testing out a litestream setup on k8s, where a python app container and litestream container share the db via a local volume. The db is created via an init container that just runs CREATE TABLE before either of them start.

Here is what init looks like:

litestream v0.3.2
initialized db: /db/hello.sqlite3
replicating to: name="s3" type="s3" bucket="..." path="hello.sqlite3" region=""
litestream /db/hello.sqlite3: init: cannot determine last wal position, clearing generation (primary wal header: EOF)
litestream /db/hello.sqlite3: sync: new generation "b7852795bc0c7d42", no generation exists

Afterwards, I get no logs until I run a query through the app (read or write). Then, I get a steady stream of litestream /db/hello.sqlite3: sync error: cannot verify wal state: stat /db/hello.sqlite3-wal: no such file or directory until I shut down litestream. I've tried issuing the wal pragma from the app's connection and enabling autocommit mode, but neither seem to make a difference.

Similar to https://github.com/benbjohnson/litestream/issues/58, I'm wondering if I'm doing something wrong or whether this is a litestream bug. If you think it's the latter, I can try to work on a simpler repro (I haven't tried it locally yet).

Howdy! I've used Litestream with great success for a personal project and was trying to work it into a new project. I have Litestream running in Docker, reading a WAL-mode DB in a volume shared with another container. However, the first time I tried turning it on, it began spamming the following error:

/db/db.db(s3): monitor error: cannot determine replica position: no snapshot available: generation=60bedb457196f69d
/db/db.db(s3): snapshot written 60bedb457196f69d/00000003
/db/db.db(s3): monitor error: cannot determine replica position: no snapshot available: generation=60bedb457196f69d
/db/db.db(s3): snapshot written 60bedb457196f69d/00000003
/db/db.db(s3): monitor error: cannot determine replica position: no snapshot available: generation=60bedb457196f69d
/db/db.db(s3): snapshot written 60bedb457196f69d/00000003

The DB file it's replicating had already existed for some time before I tried adding Litestream to the mix. Restarting Litestream did not fix it, though it did create a new snapshot dir on S3.

The weird thing is that it seemed to be writing successfully to the bucket, though it pretty quickly dumped ~9mb of data despite the DB being ~250kb, and only two or three small transactions taking place while I had it running. I had it hooked up to a free-tier Backblaze bucket and it hit the daily transaction limit within ten minutes as well (2500 S3 API calls).

I'm not really sure what's going on here -- it can definitely read the DB file, it can authenticate with Backblaze. Googling the error message didn't give me any leads either. Any chance someone can parse this issue better than I?

Thanks again for this software!

I'm using litestream with a service that has infrequent database writes (like a handful of times per day).

I noticed my AWS dashboard reporting many PUTs and revisited the documentation and realized that litestream replicates the WAL every 10s.

A nice to have feature would be if litestream skips the WAL replication if no local changes have occurred since last sync.

An even nicer feature for my scenario would be a "sync only on write" mode where instead of replicating every N seconds, it replicates immediately after each change to the WAL, but otherwise does not replicate.

Low priority since my understanding is that even 300k S3 PUTs is only ~$1.50/month, but it'd be cool if litestream could run completely in the free tier for infrequent write scenearios.

Hi !

First of all, thank you for you great project. I am really exited to start new projects with SQLite. I tried your last commits (d6ece0b82644f8963fff1d20087bb07a6a2d99c4) on master in order to upload generations into Google Cloud Storage.

Everything worked correctly until I tried to import a ~10MB CSV dataset into SQLite.

yann@xps:~/Projects/Perso/poc-litestream$ sqlite3 test.db
SQLite version 3.34.1 2021-01-20 14:10:07
Enter ".help" for usage hints.
sqlite> .mode csv
sqlite> .import ../Shakespeare_data.csv shakespear
_________________________________________
yann@xps:~/Projects/Perso/poc-litestream$ litestream replicate -config ~/Golang/bin/litestream.yml
litestream (development build)
initialized db: /home/yann/Projects/Perso/poc-litestream/test.db
replicating to: name="s3" type="s3" bucket="test-litestream-regional" path="db" region="europe-west1" endpoint="https://storage.googleapis.com" sync-interval=10s
/home/yann/Projects/Perso/poc-litestream/test.db(s3): monitor error: InvalidArgument: Invalid argument.
        status code: 400, request id: , host id:
/home/yann/Projects/Perso/poc-litestream/test.db(s3): monitor error: InvalidArgument: Invalid argument.
        status code: 400, request id: , host id:
_________________________________________
yann@xps:~/Projects/Perso/poc-litestream$ tree -ah
.
├── [9.3M]  test.db
├── [4.0K]  .test.db-litestream
│   ├── [  17]  generation
│   └── [4.0K]  generations
│       └── [4.0K]  310573930e9594aa
│           └── [4.0K]  wal
│               ├── [197K]  00000006.wal
│               ├── [9.4M]  00000007.wal
│               └── [4.1K]  00000008.wal
├── [ 32K]  test.db-shm
└── [9.4M]  test.db-wal

4 directories, 7 files

Here is my configuration:

dbs:
 - path: /home/yann/Projects/Perso/poc-litestream/test.db
   replicas:
    - type: s3
      bucket: test-litestream-regional
      path: db
      endpoint: https://storage.googleapis.com
      forcePathStyle: true
      region: europe-west1

yann@xps:~/Projects/Perso/poc-litestream$ env | grep AWS
AWS_SECRET_ACCESS_KEY=<<REDACTED>>
AWS_ACCESS_KEY_ID=<<REDACTED>>

I wonder if this issue is GCP related of it is more general. I'm pretty sure the error is raised here: https://github.com/benbjohnson/litestream/blob/main/s3/s3.go#L818-L824

If I can do anything to help, I would love to ! Have a great day and Thank you !

Can we cache some query and save some $ ?

| Opération | sept. 2021 | |----------------------|------------| | Total cost ($) | 6.37 | | ListBucket ($) | 6.12 | | CreateVolume-Gp2 ($) | 0.21 | | StandardStorage ($) | 0.02 | | PutObject ($) | 0.01 |

Let us know.

Does this work with https://github.com/resilar/sqleet and, if so, any recommendations on how to make it work best? I've found sqleet to be the best for encrypted sqlite.

sqlite is the de-facto on-device database on Android [0], and it's also popular in iOS apps.

litestream might a be really useful and simple library for synchronizing data from a mobile app to a server, without having to change the app's persistence layer.

There's a couple of proprietary libraries [1] [2] in this space that I know of, but they do more than sync, and require using the library as data layer.

Since litestream is already cross-compiled for aarch64, the binary should already work out of a box. Due to Android's sandboxing, it's not usually possible for one app to access the database of another. Litestream would have to run as part of the app the database belongs to. Maybe a library could be created that wraps the existing executable, and invokes it at appropriate times, e.g. using the SyncAdapter [3]

[0] https://developer.android.com/training/data-storage/sqlite [1] https://objectbox.io [2] https://realm.io [3] https://developer.android.com/training/sync-adapters

Just an idea.

Bumps github.com/aws/aws-sdk-go from 1.42.40 to 1.42.44.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Some may wonder way there is few traffic in the project lately.

There is a new fantastic tool from benbjohnson: LiteFS

It replicates SQLite via a FUSE filesystem.

Someone with commit permissions: Maybe leave a small note about this in the README?

when starting litestream I saw this message in the log:

litestream v0.3.8
initialized db: /data/db.sqlite3
replicating to: name="s3" type="s3" bucket="xxx" path="lb-pipeline-prod/db.sqlite3" region="fra1" endpoint="https://fra1.digitaloceanspaces.com" sync-interval=1s
/data/db.sqlite3: init: cannot determine last wal position, clearing generation; primary wal header: EOF
/data/db.sqlite3: sync: new generation "c51b0ab65d5a9c1f", no generation exists


/data/db.sqlite3(s3): monitor error: MultipartUpload: upload multipart failed
        upload id: 2~AVDf9oLvoUjwWcYb5So7CZmoZnpUguF
caused by: TotalPartsExceeded: exceeded total allowed configured MaxUploadParts (10000). Adjust PartSize to fit in this limit

litestream snapshots / litestream generations does not reveal anything under new generation. Apparently the new generation creation has failed.

Is there any config I could set to tune the multipart upload?

my config is:

access-key-id: xxx
secret-access-key: xxx

dbs:
  - path: /data/db.sqlite3
    replicas:
      - url: s3://xxx.fra1.digitaloceanspaces.com/lb-pipeline-prod/db.sqlite3
        retention: 1h
        retention-check-interval: 20m

Hi, there! I'm trying to recover a database by using the command litestream restore -o /tmp/db /path/to/replica.db. However, I'm getting the error database not found in config: /path/to/replica.db. Also, I tried it without specifying the -o parameter, with same error. I can only recover the database without specifying the replica parameter.

Am I making a mistake specifying the command?

Thanks in advance!

“ It runs as a background process and safely replicates changes incrementally to another file or S3.”

but there is no any example how to replicate from one file db to another file db

To get a copy of the database before replication starts, you need to write a shell script like this: https://github.com/benbjohnson/litestream-docker-example/blob/8f4d71055049ccd3cc52f499539e70601bb10d48/scripts/run.sh#L4-L10

Could replicate get a flag like -restore-if-db-not-exists that would restore the database if it doesn't exist prior to starting replication?

This seems like a common use case and it would be nice to be able to cover it with a single command instead of adding an additional script to handle it.

I couldn't find an existing issue asking this - is there a way to consume this programmatically? On some app platforms, one can point to a code base and have it automatically built and deployed. In this case, you can't use litestream, but if you're writing a Go-based app then you could import it and do a restore on app startup, and run periodic backups in a goroutine.

As a workaround, one could build a docker image that had litestream in it, but it can be more attractive to use it in-process in the example above.

Apologies in advance if this is already addressed elsewhere.