I was testing the gui version (2.0.8) in a vm, trying to restore with 64 threads some 20+ GB of stuff, 30k folders, 77k files (linux kernel, photos, videos + duplicates of those -- all the general file sizes), and a few minutes after starting restore i got an error about rate limit exceeded, and the restore stopped.

"error failed to find the chuck : googleapi: error 403: user rate limit exceeded, userRateLimitExceeded."

From what i remembered this error was fixed in 2.0.6, right?

While running the initial backup from MacOS and Linux computers running 2.0.10 to Azure storage, the process eventually fails with a message like this:

Failed to upload the chunk f041b71f8ae1e436ea2e6ce647c0491426a3b146bd2538308cf186fcea5d3e48: Put https://[my storage account].blob.core.windows.net/[my storage container]/chunks/f0/41b71f8ae1e436ea2e6ce647c0491426a3b146bd2538308cf186fcea5d3e48: read tcp 192.168.1.145:59117->52.239.153.4:443: read: connection timed out Incomplete snapshot saved to /mnt/USBHD/.duplicacy/incomplete

Whenever this happens I just restart the backup process again. Eventually, it finishes. The problem is that these are the initial backups so they are huge and instead of it just running for a day, it's taking forever because it fails after an hour or two and doesn't get started up again until I notice it.

I am not sure if this happens on Windows or not because I haven't tried from a Windows PC yet but I assume it does. I am also not sure if this only happens when writing to Azure storage or if it affects other clients too.

'duplicacy copy -from wasabi -to local -threads 16' frequently crashes with this message:

2017-09-20 19:27:10.262 ERROR UPLOAD_CHUNK Failed to decrypt the chunk da7f61e4ba27912563f37d1a1849479dac940cd85e60dc5b8068528df8eb889e: cipher: message authentication failed

Sometimes it crashes after less than 100 chunks, sometimes it crashes after more than 1000, but it will reliably crash. It's a different chunk every time.

The same behaviour occurs with both the s3 and s3c providers. v2.0.9 is affected. So is HEAD (ae44bf722696e1a33e311ffe6729eebf23fca755). Linux and FreeBSD are both impacted.

Currently I'm using https://github.com/restic/restic for backups, and it's just great. But I'm also looking at Duplicacy as a secondary backup tool.

However, it seems to me that Duplicacy isn't tailored for "ad-hoc" backups, in the sense that I can't just initalize the remote storage and then start backing up any folder I want to it the same way I do with restic.

For example, with restic I initialize the remote storage with restic -r sftp://HOST/backup init (for the SFTP backend), and then I can start backing up whatever files and folders I want to using e.g. restic -r sftp://HOST/backup backup ~/work /any/other/folder. Restic will store a cache on my local machine, but that's it - there's no need to initialize anything locally. I can be anywhere in the filesystem and call restic telling it to back up any path I want in the filesystem.

But with Duplicacy it seems I have to decide a local folder where I want the "repository" to be stored, and then I can run backups in/for that local folder/repository using the backup command (which doesn't seem to take files/folders as argument), but without being able to specify which folders I want to back up this and that backup run. I suppose that I can use excludes/includes to control which files and folders under this local repository are effectively backed up, but that seems a bit messy to me. I would rather just say what data I want to back up, when I'm backing up.

So is it possible to be more "ad-hoc" with Duplicacy, specifying data to be backed up when you actually back up? Without includes/excludes, can I specify what folders to back up on each run? Can i skip initializing a local repository entirely, considering I have no need for it (I'm happy to specify the target/remote repository, as well as what to back up, as well as the password on each backup run - the latter can be dealt with in restic by environment variables or a password file)?

I suppose I can initialize the local repository at / but that's not a folder my non-privileged user have access to, so it seems rather silly to have to do that. Also, initializing one local repository for each folder I want to back up is pretty silly as well, because I usually just back up the folders I have had some changes in that day (one day it's the Economy folder, the other day it's some development projects, another day some other project, etc).

EDIT: Some may wonder why I don't just back everything up every time, and it's pretty simple; If I only changed a couple of files in one place, it makes most sense to me to

The issue #148 seems to be a similar discussion, albeit not entirely the same one.

As per https://github.com/gilbertchen/duplicacy/issues/163#issuecomment-331242178

[...] listing (eventually) 65k dirs over SFTP, with a full roundtrip for each one, would just take forever (it's already annoying enough within the LAN). Single level nesting would result in ~1k files per dir with a 1TB repo with the default chunk size which hardly seems overwhelming - and would cut down dirlist requests to 256, or about 5 minutes, which is far more reasonable.

Quick and informal testing (with a tiny repo) shows that backup, copy, prune, list and check keep working as expected, i.e. single-nested (new) chunks can coexist alongside preexisting double-nested chunks.

I am not sure if this is the right place but please help, I can't figure this out. How do I get from

pkg install wget pkg install go go get -u github.com/gilbertchen/duplicacy/... go build -v /root/go/src/github.com/gilbertchen/duplicacy/duplicacy/duplicacy_main.go chmod +x /root/go/bin/duplicacy

To running duplicacy.

I am running this on a FreenNAS ver-11.1-U5 jail.

----edit----

Ok got it running by downloading the prebuilt exe. Here are the steps.

pkg install wget Needed to download a file on my system. You may have it installed already.
pkg install go. Not sure if I needed it but I got it anyway. mkdir /root/go/ mkdir /root/go/bin/ cd /root/go/bin/ wget https://github.com/gilbertchen/duplicacy/releases/download/v2.1.0/duplicacy_freebsd_x64_2.1.0 mv duplicacy_freebsd_x64_2.1.0 duplicacy Changes name to duplicacy. The .0 at the end of the original file was preventing it from running not sure why. chmod +x duplicacy Makes the file executable. ./duplicacy Runs the file in this directory.
/root/go/bin/duplicacy Runs the file in any other directory.

This has been mentioned a few times on the duplicacy website, so i though i could add it here as well.

I would mention the issues, but there's no search in there :oh-darn:.

In case @gilbertchen is interested, i have a testing discourse running on my own pc here, with some interesting plugins, and it only took me 30 minutes to get it up and running along with LetsEncrypt certificate and https only.

My pc is not always on though, so you may or may not need to check the website from time to time 😵, however the discourse meta is always available and has a lot helpful folks (including, of course the discourse creators -- they're really nice!).

I have also seen @tophee hanging around on meta, and i also assume he knows discourse better than i do.

If @gilbertchen wants, i'd be willing to have a go at configuring, at least for testing, a discourse instance (and eventually try and see how sso works with duplicacy's own website).

Let me preface this by saying I hope I got all my math right. Corrections are welcome. I'm attaching the python script I used for calculations and a table of those results.

tl;dr: b2_list_file_names is inefficient to check for existing chunks. Instead use b2_download_file_by_name with a header "Range: bytes=0-0".

Duplicacy calls b2_list_file_names prior to every upload to check if a given chunk has already been uploaded. As referenced in #30, the B2 pricing means around 10 GB can be uploaded for free each day, and then $.001/GB. However, the list query only returns up to 1000 entries each call so multiple calls per chunk must be made once more than 1000 chunks have been uploaded. 1000 chunks works out to around only 4 GB. By the time I reach 1 TB of data (around 263000 chunks) it's going to be 263 calls to b2_list_file_names for each new chunk. Ignoring the free data and transactions (negligible given the total data) it's going to cost almost $140 to make the 35 million calls while uploading. 10 TB would require 2.5 million chunks, 3.5 billion total calls, cost $13,750, and require another 2622 list calls just to upload another new chunk. That's over a penny every 4 MB.

Now, there is the b2_get_file_info call, but that requires passing the fileId which is normally obtained from b2_list_file_names. Duplicacy could save the fileId in the snapshots files, but that would require an architecture change.

Alternatively, Duplicacy could call b2_download_file_by_name and look at the response code (200 vs 400) and headers. A GET or HEAD adds the entire payload size to your download quota, but including a Range header only adds the requested Range size. Setting a "Range: bytes=0-0" is only a single byte transferred. This reduces the cost by an order of magnitude and also is guaranteed to require a single call, rather than multiple calls. This reduces the cost to upload 10 TB from $13,750 to $1.05. The additional single byte (GET or HEAD) equals around 2 billion calls for a penny. Conveniently, the response headers include the relevant info also returned by b2_get_file_info. It looks like the only real difference is a lack of the bucket id.

On the more hacky side, b2_hide_file and b2_delete_file_version are each unbilled calls. These could be combined to test for the existence of a file. I can see an issue if the process dies before unhiding the chunk and it looks like Duplicacy uses B2 hidden files for fossils. I think this option is more trouble than it's worth.

More generally, B2 chunks are all stored in the "chunks" prefix. If these were instead broken up by nested prefixes taken from the hash, the b2_list_file_names could be made more stable by using the relevant prefix when calling. In other words, chunk/badcoffee12345 would instead be stored at chunk/ba/dc/of/fe/e1/23/45. Each prefix, and therefore each call to b2_list_file_names, would hold no more than 256 files. 10 TB would now cost $10.49. Switching to b2_download_by_name would be a tenth further, but wouldn't benefit any further than using the download call with all chunks in the same directory.

Changing the file structure would break old backups, but perhaps it could be implemented as new storage target.

I also sent a message to Backblaze to ask about new API calls that could be used to check for file existence. Specifically:

• b2_get_file_info_by_name
• b2_get_file_id_by_name I received a response that it's been added as a feature request.

b2-calc.txt b2-calc.py.txt

Using dfdbfed on Ubuntu 16.04.4.

When pruning, the Wasabi backend consistently fails with "Failed to fossilize chunk". It reports a 404 Not Found, however the chunk is present on the storage backend.

Executing the same command again repeats the process, except failing on fossilizing a different chunk. Each time fails on a different chunk, with the previous chunks remaining orphaned on storage.

Swapping back to the s3 backend and the prune completed without errors.

Attached is a full log showing repeated consecutive invocations failing on different chunks, plus the final invocation with the s3 backend which was successful on first attempt.

Here's an excerpt from one of the failed runs:

# duplicacy -d -log prune -keep 1:1 -keep 5:7 -keep 15:30 -keep 30:180 -keep 0:360 | tee -a /var/log/duplicacy.log
2018-07-11 21:20:03.673 INFO STORAGE_SET Storage set to wasabi://[email protected]/redacted
2018-07-11 21:20:03.673 DEBUG PASSWORD_ENV_VAR Reading the environment variable DUPLICACY_WASABI_KEY
2018-07-11 21:20:03.673 DEBUG PASSWORD_PREFERENCE Reading wasabi_key from preferences
2018-07-11 21:20:03.673 DEBUG PASSWORD_ENV_VAR Reading the environment variable DUPLICACY_WASABI_SECRET
2018-07-11 21:20:03.673 DEBUG PASSWORD_PREFERENCE Reading wasabi_secret from preferences
2018-07-11 21:20:03.673 DEBUG PASSWORD_ENV_VAR Reading the environment variable DUPLICACY_WASABI_KEY
2018-07-11 21:20:03.674 DEBUG PASSWORD_PREFERENCE Reading wasabi_key from preferences
2018-07-11 21:20:03.674 DEBUG PASSWORD_ENV_VAR Reading the environment variable DUPLICACY_WASABI_SECRET
2018-07-11 21:20:03.674 DEBUG PASSWORD_PREFERENCE Reading wasabi_secret from preferences
2018-07-11 21:20:03.674 DEBUG PASSWORD_ENV_VAR Reading the environment variable DUPLICACY_PASSWORD
2018-07-11 21:20:03.674 DEBUG PASSWORD_PREFERENCE Reading password from preferences
2018-07-11 21:20:03.982 TRACE CONFIG_ITERATIONS Using 16384 iterations for key derivation
2018-07-11 21:20:04.053 DEBUG STORAGE_NESTING Chunk read levels: [1], write level: 1
2018-07-11 21:20:04.067 INFO CONFIG_INFO Compression level: 100
2018-07-11 21:20:04.067 INFO CONFIG_INFO Average chunk size: 4194304
2018-07-11 21:20:04.067 INFO CONFIG_INFO Maximum chunk size: 16777216
2018-07-11 21:20:04.067 INFO CONFIG_INFO Minimum chunk size: 1048576
2018-07-11 21:20:04.067 INFO CONFIG_INFO Chunk seed: 50a4d4a8aaa118dbbceed243e071d24fd799d32ba31efe8f033a391fe1d2f1c7
2018-07-11 21:20:04.067 DEBUG PASSWORD_ENV_VAR Reading the environment variable DUPLICACY_PASSWORD
2018-07-11 21:20:04.067 DEBUG PASSWORD_PREFERENCE Reading password from preferences
2018-07-11 21:20:04.067 DEBUG DELETE_PARAMETERS id: backup, revisions: [], tags: [], retentions: [1:1 5:7 15:30 30:180 0:360], exhaustive: false, exclusive: false, dryrun: false, deleteOnly: false, collectOnl
y: false
2018-07-11 21:20:04.068 INFO RETENTION_POLICY Keep 1 snapshot every 1 day(s) if older than 1 day(s)
2018-07-11 21:20:04.068 TRACE SNAPSHOT_LIST_IDS Listing all snapshot ids
2018-07-11 21:20:04.113 TRACE SNAPSHOT_LIST_REVISIONS Listing revisions for snapshot backup
2018-07-11 21:20:04.198 DEBUG DOWNLOAD_FILE_CACHE Loaded file snapshots/backup/1 from the snapshot cache
2018-07-11 21:20:04.239 DEBUG DOWNLOAD_FILE_CACHE Loaded file snapshots/backup/2 from the snapshot cache
2018-07-11 21:20:04.285 DEBUG DOWNLOAD_FILE_CACHE Loaded file snapshots/backup/3 from the snapshot cache
2018-07-11 21:20:04.326 DEBUG DOWNLOAD_FILE_CACHE Loaded file snapshots/backup/4 from the snapshot cache
2018-07-11 21:20:04.367 DEBUG DOWNLOAD_FILE_CACHE Loaded file snapshots/backup/5 from the snapshot cache
2018-07-11 21:20:04.409 DEBUG DOWNLOAD_FILE_CACHE Loaded file snapshots/backup/6 from the snapshot cache
2018-07-11 21:20:04.410 DEBUG SNAPSHOT_DELETE Snapshot backup at revision 2 to be deleted - older than 1 days, less than 1 days from previous
2018-07-11 21:20:04.410 INFO SNAPSHOT_DELETE Deleting snapshot backup at revision 2
2018-07-11 21:20:04.424 DEBUG CHUNK_CACHE Chunk 1b3bdb3229746baa88efe5c48cf6189fbe572e863d63ab56fb2679ab580b579a has been loaded from the snapshot cache
2018-07-11 21:20:04.444 DEBUG DOWNLOAD_FETCH Fetching chunk 1ce42d419b9ecaef78f4760bd5180443084fa7eca8e8c6eba174ea3a8ba1fe6f
2018-07-11 21:20:04.445 DEBUG CHUNK_CACHE Chunk 1ce42d419b9ecaef78f4760bd5180443084fa7eca8e8c6eba174ea3a8ba1fe6f has been loaded from the snapshot cache
2018-07-11 21:20:04.447 DEBUG DOWNLOAD_FETCH Fetching chunk 7c3f4844cd94b75c0b969257775b7bc2fb1a132c9d93446a350ff088f99d2b2c
2018-07-11 21:20:04.452 DEBUG CHUNK_CACHE Chunk 7c3f4844cd94b75c0b969257775b7bc2fb1a132c9d93446a350ff088f99d2b2c has been loaded from the snapshot cache
2018-07-11 21:20:04.572 DEBUG DOWNLOAD_FETCH Fetching chunk 54ea3d5845e7687fef56a098199a8db2950c1c7d1796754dcae341b82d83b5f2
2018-07-11 21:20:04.577 DEBUG CHUNK_CACHE Chunk 54ea3d5845e7687fef56a098199a8db2950c1c7d1796754dcae341b82d83b5f2 has been loaded from the snapshot cache
2018-07-11 21:20:04.579 DEBUG DOWNLOAD_FETCH Fetching chunk a14efa66b126ee24aa13ea39960eca7a4b710a128d898c9ddffce5d57f3b3a2d
2018-07-11 21:20:04.580 DEBUG CHUNK_CACHE Chunk a14efa66b126ee24aa13ea39960eca7a4b710a128d898c9ddffce5d57f3b3a2d has been loaded from the snapshot cache
2018-07-11 21:20:04.684 DEBUG DOWNLOAD_FETCH Fetching chunk 54ea3d5845e7687fef56a098199a8db2950c1c7d1796754dcae341b82d83b5f2
2018-07-11 21:20:04.689 DEBUG CHUNK_CACHE Chunk 54ea3d5845e7687fef56a098199a8db2950c1c7d1796754dcae341b82d83b5f2 has been loaded from the snapshot cache
2018-07-11 21:20:04.689 DEBUG DOWNLOAD_FETCH Fetching chunk 2bceac3d48bef096404bfe4a7c4645701603b9738198fdae0c867d6d4ea6a3ff
2018-07-11 21:20:04.690 DEBUG CHUNK_CACHE Chunk 2bceac3d48bef096404bfe4a7c4645701603b9738198fdae0c867d6d4ea6a3ff has been loaded from the snapshot cache
2018-07-11 21:20:04.794 DEBUG DOWNLOAD_FETCH Fetching chunk 54ea3d5845e7687fef56a098199a8db2950c1c7d1796754dcae341b82d83b5f2
2018-07-11 21:20:04.799 DEBUG CHUNK_CACHE Chunk 54ea3d5845e7687fef56a098199a8db2950c1c7d1796754dcae341b82d83b5f2 has been loaded from the snapshot cache
2018-07-11 21:20:04.800 DEBUG DOWNLOAD_FETCH Fetching chunk ccf9a9c0085d1cf160f6437c71a47f3b6e6560d501ec049d232038a44ff6f6d8
2018-07-11 21:20:04.800 DEBUG CHUNK_CACHE Chunk ccf9a9c0085d1cf160f6437c71a47f3b6e6560d501ec049d232038a44ff6f6d8 has been loaded from the snapshot cache
2018-07-11 21:20:05.192 ERROR CHUNK_DELETE Failed to fossilize the chunk 515b226b6f533991a99884759bd428e46a1d36aa66aca71897490e95ded33a70: 404 Not Found

prune.log

Running a backup on Windows 10, duplicacy 1.1.6 is giving the following error many times for different paths:

Failed to list subdirectory: 
open C:\Users\martin/.duplicacy\shadow\\Users\martin/data/identities/personal/project/mithril-boilerplate/node_modules/watchify/node_modules/browserify/node_modules/insert-module-globals/node_modules/lexical-scope/node_modules/astw/node_modules/esprima-fb/examples: 
Access is denied.

Is this a long path issue?

Note: In an Admin shell, backup executed using C:\Users\martin>duplicacy backup -stats -vss

I was originally just concerned with MacOS resource forks, but was reminded of the Backup Bouncer test suite.

Macs currently split non-data fork data to AppleDouble encoded "._filename" files when saving to non-Mac systems.

These forks can be listed and displayed with the xattr command.

The easiest way to create a file with a resource fork is dragging a URL or text selection to the desktop to create a "webloc" or "textClipping" file.

https://github.com/gilbertchen/duplicacy/blob/0a794e6feae91cf0e6511b77a5bf160999c1838e/src/duplicacy_storage.go#L165-L211

Just something I stumbled across trying to understand how duplicacy verifies the host it is connecting to.

1. This code is not fully compatible with the standard definition of the SSH known_hosts file syntax [1]. Perhaps it's not meant to be, but then the name of the file is misleading. While the code appears to be able to account for markers, it does not seem to be able to account for the brackets around the host name, in case a non-standard port is provided. Also if the standard port :22 is used, it is not enforced that it is omitted in the config file.

2. I'd argue that duplicacy should not automatically connect to hosts for which it has never seen a fingerprint before, without explicit user consent. Right now, a MITM could be happening during the first use of duplicacy, leading to exposed data.

3. Maybe consider taking into account ~/.ssh/known_hosts as well in addition to .duplicacy/known_hosts?

[1] https://man.openbsd.org/sshd#SSH_KNOWN_HOSTS_FILE_FORMAT

I was trying to understand the encryption code in duplicacy and its handling of the many keys stored in the config file when I discovered something unexpected here:

https://github.com/gilbertchen/duplicacy/blob/f2d6de3fff7567740e86cb49801b0698f085cc59/src/duplicacy_chunk.go#L216-L218

For some reason this code is using the 'derivation key', which for things like snapshots is just a plaintext file path, as the secret key for Blake2b, and then digesting the encryptionKey to get the combined key. I would have expected the opposite, using the encryptionKey as the secret key for Blake2b, and then digesting the path. Indeed the wiki page on encryption here https://github.com/gilbertchen/duplicacy/wiki/Encryption states:

"The snapshot is encrypted by AES-GCM too, using an encrypt key that is the HMAC-SHA256 of the file path with the File Key as the secret key."

Which is precisely the opposite of what the code actually does. Worse, it looks like this may have led to issues in the past, specifically this commit https://github.com/gilbertchen/duplicacy/commit/d330f61d251f12c24cdd38b77d143cbb716913da - which would never have been an issue if the construction wasn't backwards.

I am not sure how exploitable this is, I hope it isn't, but it's a pretty big code smell in the middle of the encryption code.

This work is inspired by !628, but it's a more complete implementation.

I'm using cgofuse so it works on all platforms, including Windows.

There are two commands:

• mount <MOUNT_POINT>: must be in a repository directory, uses preferences information to load snapshot info.
• mount-storage <STORAGE_URL> <MOUNT_POINT>: mounts a storage directly without the need for a repository. note that it may need a repository under some circumstances anyway, so there's a -repository option for that.

Snapshot/revision information is loaded only when you try to browse the containing folder. By default it'll create a base tree organizing snapshot revisions by date using the format /%YYYY/%MM/%DD/%HH%mm.%REV. For instance, let's say you have this output from duplicacy list:

$ duplicacy list
Snapshot Users revision 1 created at 2022-10-19 20:09 -hash
Snapshot Users revision 2 created at 2022-10-19 21:38
Snapshot Users revision 3 created at 2022-10-20 10:38
Snapshot Users revision 4 created at 2022-11-14 13:44

The following base tree will be created:

2022/
    10/
        19/
            2009.1/
            2138.2/
        20/
            1038.3/
    11/
        14/
            1344.4/

If you want all revisions to be on the same directory, you can use the -flat parameter. Revision and snapshot directories have an extra empty directory to prevent Windows explorer from prematurely triggering the downloading of data from the storage.

Folders and files will display their saved attributes, with the caveat that everything that's not a folder is shown as a regular file.

File reading is efficiently implemented by only downloading the chunks needed for the specific OS read request and caching them. The cache is a 2Q LRU cache by golang-lru with size 20 and keyed by the chunk hash.

I've tested using a repository with these characteristics:

• 10.2 GB
• 70151 Files
• 10467 Folders
• 4 revisions
• sftp storage on a remote server

It takes a couple of seconds to create the base tree and a few seconds for every revision dir that is loaded for the first time, but otherwise everything works as expected. Memory use hovered around 700MB, even tarring a whole revision dir and piping it to /dev/null.

Hello,

I'm currently doing benchmarks for deduplication backup tools, including duplicacy. I decided to write a script that would:

• Install the backup programs
• Prepare the source server
• Prepare local targets / remote targets
• Run backup and restore benchmarks
• Use public available data (linux kernel sources as git repo) and checkout various git tags to simulate user changes in the dataset

The idea of the script would be to have reproductible results, the only changing factor being the machine specs & network link between sources and targets.

So far, I've run two sets of benchmarks, each done locally and remotely. You can find the results at https://github.com/deajan/backup-bench

I'd love you to review the recipe I used for duplicacy, and perhaps guide me on what parameters to use to get maximum performance. Any remarks / ideas / PRs are welcome.

I've also made a comparaison table of some features of the backup solutions I'm benchmarking. I still miss some informations for some of the backup programs. Would you mind having a look at the comparaison table and fill the question marks related to the features of duplicacy ? Also, if duplicacy has an interesting feature I didn't list, I'll be happy to extend the comparaison.

PS: I'm trying to be as unbiased as possible when it comes to those benchmarks, please forgive me if I didn't treat your program with the parameters it deserves.

Also, I've created the same issue in every git repo of the backup tools I'm testing, so every author / team / community member can judge / improve the instructions for better benchmarking.

As discussed on the forum: Duplicacy and btrfs snapshots and quesiton 2 of ... Have 3 Questions.

I would like to be able to do something akin to btrfs send | duplicacy. That way I can use the deduplication, compression, and encryption mechanisms of duplicacy. This is similar to btrbk, which effectly does btrfs send | zstd | gpg | ssh, but better.

I would also like to suggest duplicacy default to $HOME for the .duplicacy folder, if there is none in the present directory. And/or, add flags like -backup-dir and -snapshot-id to specify.

Added support for SharePoint document libraries to the ODB backend. Used the same format as for Team Drives in GCD, so odb://DRIVEID@path/to/storage, where DRIVEID is in the format "b!xxxxxxx". If not using @ in the odb:// storage specification the behaviour should be exactly the same as before, so there are no changes to existing setups.

No to minimal changes needed for the GUI as pathspec can go into the existing field (filtering may need to be adjusted).

This pull request is built on top of custom_odb_creds PR - not because they have dependencies on each other (they don't), but because they modify the same files so merge would have been non-trivial.