Open Source Continuous File Synchronization

Syncthing


Latest Linux & Cross Build Latest Windows Build Latest Mac Build MPLv2 License CII Best Practices Go Report Card

Goals

Syncthing is a continuous file synchronization program. It synchronizes files between two or more computers. We strive to fulfill the goals below. The goals are listed in order of importance, the most important one being the first. This is the summary version of the goal list - for more commentary, see the full Goals document.

Syncthing should be:

  1. Safe From Data Loss

    Protecting the user's data is paramount. We take every reasonable precaution to avoid corrupting the user's files.

  2. Secure Against Attackers

    Again, protecting the user's data is paramount. Regardless of our other goals we must never allow the user's data to be susceptible to eavesdropping or modification by unauthorized parties.

  3. Easy to Use

    Syncthing should be approachable, understandable and inclusive.

  4. Automatic

    User interaction should be required only when absolutely necessary.

  5. Universally Available

    Syncthing should run on every common computer. We are mindful that the latest technology is not always available to any given individual.

  6. For Individuals

    Syncthing is primarily about empowering the individual user with safe, secure and easy to use file synchronization.

  7. Everything Else

    There are many things we care about that don't make it on to the list. It is fine to optimize for these values, as long as they are not in conflict with the stated goals above.

Getting Started

Take a look at the getting started guide.

There are a few examples for keeping Syncthing running in the background on your system in the etc directory. There are also several GUI implementations for Windows, Mac and Linux.

Docker

To run Syncthing in Docker, see the Docker README.

Vote on features/bugs

We'd like to encourage you to vote on issues that matter to you. This helps the team understand what are the biggest pain points for our users, and could potentially influence what is being worked on next.

Getting in Touch

The first and best point of contact is the Forum. There is also an IRC channel, #syncthing on freenode (with a web client), for talking directly to developers and users. If you've found something that is clearly a bug, feel free to report it in the GitHub issue tracker.

Building

Building Syncthing from source is easy, and there's a guide that describes it for both Unix and Windows systems.

Signed Releases

As of v0.10.15 and onwards release binaries are GPG signed with the key D26E6ED000654A3E, available from https://syncthing.net/security.html and most key servers.

There is also a built in automatic upgrade mechanism (disabled in some distribution channels) which uses a compiled in ECDSA signature. macOS binaries are also properly code signed.

Documentation

Please see the Syncthing documentation site.

All code is licensed under the MPLv2 License.

Comments
  • Filesystem Notification - Continuation

    Filesystem Notification - Continuation

    This is the continuation of plouj's PR #2807 on a branch in the syncthing repo for ease of maintenance. Refer to that PR for general information what this is about and commits previous to https://github.com/syncthing/syncthing/commit/1d424d84609a40d2a11ac8e06cb87aac6b130629.

  • Support for file encryption (e.g. non-trusted servers)

    Support for file encryption (e.g. non-trusted servers)

    So I have had a look at BitTorrent sync, syncthing and alternatives and what I always wondered about was the possibility to not only sync between resources I own and trust, but also external resources/servers which I do NOT trust with my data, up to a certain extent.

    One way to do this is using ecryptfs or encfs, but this has many obvious downsides: it is not an interoperable solution (only works on Linux), the files are actually stored in encrypted form on the disk (even if the resource is trusted and this is not necessary, for instance because of the file system being encrypted already), etc.

    What I propose is somehow configuring nodes which are only sent the files in an encrypted format, with all file contents (and potentially file/directory names as well; or even permissions) being encrypted. This way, if I want to store my private files on a fast server in a datacenter to access them from anywhere, I could do this with syncthing without essentially giving up ownership of those files. I could also prevent that particular sync node from being allowed/able to make any changes to the files without me noticing.

    I realize that this requires a LOT of additional effort, but it would be a killer feature that seems to not be available in any other "private cloud" solution so far. What are your thoughts on this feature?

    EDIT: BitTorrent sync mentions a feature like this in their API docs: "Encryption secret API users can generate folder secrets with encrypted peer support. Encryption secrets are read-only. They make Sync data encrypted on the receiver’s side. Recipients can sync files, but they can’t see file content, and they can’t modify the files. Encryption secrets come in handy if you need to sync to an untrusted location." (from http://www.bittorrent.com/intl/de/sync/developers/api)

  • all: Store pending devices and folders in database (fixes #7178)

    all: Store pending devices and folders in database (fixes #7178)

    Purpose

    As discussed in #5758 and mentioned on the forum, storing the pending (offered from remote but not yet added locally) devices and folders in the XML configuration is not a nice and scalable design. Instead, the information should live in the database, properly structured, and made available over dedicated API endpoints.

    This is also ground work and practice to finally approach an acceptable implementation of the prototype in #5758, extending the concept to other devices we have heard about in ClusterConfig messages. That is deliberately saved for another PR though.

    Testing

    All code paths have undergone extensive manual testing within an existing setup of four instances (one compiled from this PR). Especially regarding the clean-up of pending entries upon config changes, both online through the POST /rest/system/config REST API and offline by modifying the XML while Syncthing is not running. Notifications on the GUI look as before, API endpoints have been verified with curl.

    ~~Unit tests would be rather complicated and were discussed as not strictly needed considering how low the importance of the handled information is.~~ Unit tests included, with a few basic test cases.

    Documentation

    The envisioned API for this stuff is described in https://github.com/syncthing/docs/pull/498.

    The commit messages contain much of the rationale for each change, so I'd be happy to keep them intact instead of squash-merging in a single commit. If desired, I can rebase to clean out obvious fixup commits and end up with a nice patch series of self-contained changes.

    Breaking Changes For The User

    Existing <pendingDevice> / <pendingFolder> entries in the XML config are not carried over to the database. The corresponding notifications will disappear until the next connection attempt with the offering device. It has been discussed that the low importance of this information does not warrant a separate logic for config --> database info migrations. One possibility to minimize disruption would be to just keep the XML entries intact for some releases and hold back on commit 8ae24612397fea6182cfb3d6dcce592c78c6df5d until then.

  • Option to not sync mtime

    Option to not sync mtime

    Trying to sync my android device and my NAS(armv5) both using syncthing v0.10.0:

    On the ntfs mount of the NAS the file /my/directory/.syncthing.example.test is created then I get:

    puller: final: chtimes /my/directory/.syncthing.example.test: invalid argument

    File /my/directory/example.test is not created. Syncthing continues with the next file but the same error happens for all the files in the directory.

  • GUI display of global changes

    GUI display of global changes

    Purpose

    This PR adds a button to the device section of the GUI that the user can click on to see all filesystem changes that originated/occurred on that machine. That is to say, not changed copied from the network. This should help people discover more easily what computer made a certain change that was propagated in larger P2P swarms.

    Screenshots

    image

  • Folder isn't making progress

    Folder isn't making progress

    All nodes are on v0.10.2 GUI says

    9:35:58: Folder "default" isn't making progress - check logs for possible root cause. Pausing puller for 1m0s.
    9:36:41: Folder "PIA" isn't making progress - check logs for possible root cause. Pausing puller for 1m0s.
    

    In the log I see the "hash mismatch" info but the file is not changed during pull. It was probably the whole night in that state but i don't have more logs, I should probably increase the size... logs from the pulling node: http://alex-graf.de/public/st/no-progress.tar.gz

    Restarting also does not fix it

  • gui: add advance config port mapping to gui

    gui: add advance config port mapping to gui

    Purpose

    https://github.com/syncthing/syncthing/issues/4824

    Gives the user the opportunity to display a link to web gui remote machines. The user can choose the port and set if the link will display.

    Testing

    I just check to see if the link was shown depending on the setting.

    Screenshots

    Screen Shot 2020-09-29 at 11 13 53 PM Screen Shot 2020-09-29 at 11 34 37 PM

    Documentation

    https://github.com/syncthing/docs/pull/573

    Authorship

    Your name and email will be added automatically to the AUTHORS file based on the commit metadata.

  • lib/connections: Add KCP support (fixes #804)

    lib/connections: Add KCP support (fixes #804)

    This is mostly for benchmarks and testing.

    Seems to connect, and not crash immediately, which is good news.

    Known issues:

    1. No timeouts as part of the protocol, so we usually end up with a dead connection for 5-7.5 minutes until it times out. This could be addressed by always using the new connection that we get, as long as the priority is right
    2. Has some weirdness around closing connections. See https://github.com/xtaci/kcp-go/issues/18

    The diff is massive due to deps, so I suggest you pull the PR locally and diff inside the lib and cmd/syncthing dirs.

  • Memory and CPU usage is prohibitively high

    Memory and CPU usage is prohibitively high

    I've been testing syncthing across 3 machines, a laptop with 8GB of RAM and two NAS-style servers with 1GB and 2GB of RAM. My repositories have the following sizes:

    • 19 items, 536 KiB
    • 83471 items, 16.2 GiB
    • 181482 items, 387 GiB

    To sync these three repositories syncthing 0.9.0 uses a bit over 700MB of RAM and while syncing continuously pegs the CPU at 150% on all nodes.

    While the CPU usage I could manage during the initial sync the memory usage is simply too high. A typical NAS server like the two I have holds >4TB of storage. At the current level of usage that would require ~8GB of memory just for syncthing.

    Without looking at the code I assume an index is being kept in memory for all the repository contents. 700MB is 2.6kb per item on disk, which seems way too high. The index should only really need to store filename, parent, permissions, size and timestamp. On average (depends on filename sizes) that should only be 50-60 bytes per item which would only be 13MB. Moving that index to disk would also make a lot more sense.

    I assume the CPU usage is hashing lots of files. There I assume using librsync might be a better option.

  • /rest/system/status returns empty response during startup

    /rest/system/status returns empty response during startup

    Syncthing version 0.12.20 (although it's been around since at least the beginning of February).

    During startup, the /rest/system/status endpoint can return an empty response (seems to happen more on some devices than others).

    The headers returned are:

      Access-Control-Allow-Origin: *
      Pragma: no-cache
      X-Syncthing-Id: IDIDIID-IDIDID-IDIDID-IDIDID-IDIDID-IDIDID-IDIDID-IDIDID
      X-Syncthing-Version: v0.12.20
      Cache-Control: no-store, no-cache, max-age=0
      Date: Sat, 19 Mar 2016 22:40:58 GMT
      Content-Length: 0
      Content-Type: application/json; charset=utf-8
      Expires: Sat, 19 Mar 2016 22:40:58 GMT
    

    Of particular interest is the Content-Length: 0. I had a quick look through the code but couldn't see why it might be doing this. I fetch the /rest/system/version resource milliseconds afterwards and that returns the correct data.

    Thanks!

  • Use protocol buffers instead of custom XDR

    Use protocol buffers instead of custom XDR

    Not done, not for merging, just for review and discussion.

    This changes our serialization code to use protocol buffers instead of our custom code. Todo/done:

    • [X] Protocol structs should be protobuf
    • [X] Database structs should be protobuf
    • [x] Local discovery structs should be protobuf by humans?
    • [x] Clean up bitfields vs booleans (as per Audrius stuff)
    • [x] Remove currently unused fields as these can be added painlessly in the future

    Changes I think we should not do:

    • The global discovery protocol. No need, might be nice to have human readable, no real reason to cause the disturbance.
    • The relay protocol, as far as it concerns speaking to a relay. No reason to break it currently and leaving it unchanged lets us reap the benefits of the deployed infrastructure.

    Some changes that were necessary but unexpected or non-obvious:

    • ~~The protobuf serializer uses a Size() method which conflicted with our Size fields on BlockInfo. I've renamed it to Length.~~
    • I've added a Size field instead of CachedSize to FileInfo and let this be serialized everywhere. It means we get it for free in the TruncatedFileInfo instead of having to extract it from the block list.
    • ~~The TruncatedFileInfo type is "officialized" and moved into the protocol package, but it's still mostly used by the database. I'm not sure of this, maybe it can be undone and moved back to db as we have some protobufs stuff there as well (but didn't when I first made this change)...~~
    • The version vector type Vector []Counter definition doesn't work with protobuf so I had to make it a struct which resulted in a lot of tedious changes everywhere, but it's conceptually the same.
    • I added FileName() and FileLength() methods to the interface that covers both FileInfo and FileInfoTruncated so we can get the name or size of the interface without having to type assert to the underlying type.
    • I threw out the conversion from 0.12 db:s. We'll need full rescan for this change, but on the other hand it ought to be the last time ever.
  • Unknown Device after suspend

    Unknown Device after suspend

    After waking up my laptop from suspension, the web UI will fail to identify this device (XPS) and show it as "unknown device," but also show the correct information of this device in the Remote Devices tab. There are no errors in the JS console.

    image

    This issue can always be fixed using a systemctl restart syncthing@username reboot, but always show up after waking from suspension.

    image

    Include required information

    Please be sure to include at least:

    Syncthing Version: syncthing v1.22.1 "Fermium Flea" (go1.19.3 linux-amd64) syncthing@archlinux 2022-11-03 07:35:59 UTC [noupgrade] Browser Version: Google Chrome 108.0.5359.124 OS: image

  • Container start-time volume check

    Container start-time volume check

    Purpose

    Opt-in to check if mapped volume contains configuration file. If mapped volume is mounted from remote host, it may be temporarily unavailable during boot time. Which the optional NOCREATE environment variable entrypoint script checks if config.xml is available; if not - container exits for Docker to handle restarting periodically until configuration appears (volume mounts).

    Testing

    Tested on local setup: container keeps restarted by Docker until mounted volume with data (config/config.xml) becomes available.

    Documentation

    Related paragraph added to README.

  • build(deps): bump github.com/shirou/gopsutil/v3 from 3.22.11 to 3.22.12

    build(deps): bump github.com/shirou/gopsutil/v3 from 3.22.11 to 3.22.12

    Bumps github.com/shirou/gopsutil/v3 from 3.22.11 to 3.22.12.

    Release notes

    Sourced from github.com/shirou/gopsutil/v3's releases.

    v3.22.12

    Important notice

    v3.22.11 was reverted because some issue by #1384, and retracted by #1386.

    What's Changed

    cpu

    disk

    host

    load

    Other Changes

    New Contributors

    Full Changelog: https://github.com/shirou/gopsutil/compare/v3.22.10...v3.22.12

    Commits

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
  • Incorrect local state when using `!` negated ignore patterns combined with parent folder ignored

    Incorrect local state when using `!` negated ignore patterns combined with parent folder ignored

    I believe this is likely responsible for the long-term issue I've been experiencing since basically forever which leads to non-matching local and remote folder states despite using the exact same ignore patterns on both sides.

    1. Share an empty folder between Device 1 (D1) and Device 2 (D2).
    2. Set the folder to "Send & Receive" on both sides.
    3. Add the following ignore patterns to the folder on both sides.
      !*.txt
      /
      
    4. The folders on both sides should now look as follows. image image
    5. Create the following hierarchy in the folder on D1.
      /folder
      /folder/file.txt
      
    6. Scan the folder on D1. Wait for the changes to sync to D2.
    7. The final state looks as follows. image image
    8. As you can see, the folder state on D1 is incorrect. Because file.txt is present inside /folder, the folder should be included in the local state as well, however in reality only the file is. Once this happens, neither rescanning the folder nor restarting Syncthing repairs the broken local state.

    There is a caveat though. The steps above aren't complete. I've just managed to reproduce this in my test environment and I'm attaching logs with model,db enabled below, however in my testing I was repeatedly creating and removing folders and files in order to find the culprit. While doing so, I eventually managed to reach the state as seen above, but I can't reproduce it at will directly yet.

    The logs should contain the relevant information though. They aren't very large either, as the whole testing took less than 10 minutes or so.

    syncthing1.log syncthing2.log

    For the record, I think the actual synchronisation still functions fine despite the incorrect local states. However, I've encountered problems with Receive Only folders being stuck in a permanent "Out of Sync" state which I believe is likely related to this very bug. I still haven't been able to reproduce that one in my test environment though.

  • gui: add xattr filter editor (fixes #8660)

    gui: add xattr filter editor (fixes #8660)

    Purpose

    Added a GUI element in the advanced folder configuration panel to be able to manipulate the extended attribute filter settings.

    I intended to keep it as simple as possible, no automatic additions or other magic behind the screens. Mostly because I expect that the users who will use this will know what they are doing. But also because I'm not a fan myself of automatically added rules/lines when manipulating settings, unnecessary complexity. However, some information will still be shown when appropriate;

    • Default behaviour if a wild-card/any is not present as last element
    • A reminder in case that only deny-rules exist while the default in that case is deny - suggesting the addition of a permit-any rule in the end.

    Otherwise empty rules are being cleaned up on saving the configuration and new entries are being pushed before a wild-card/any rule, if that one is present as last element.

    Any further logic/validation is not present, on purpose.

    Testing

    • Manipulate a folder's xattr filter using the new part in the GUI -> check result in the config file.
    • Manipulate the default folder configuration -> add a new folder -> verify that the settings are correct for the newly added folder (and default settings).

    Screenshots

    Screenshot 2022-12-28 at 11 51 46 Screenshot 2022-12-28 at 11 51 54 Screenshot 2022-12-28 at 11 52 03

    Documentation

    Supporting documentation is prepared as well, https://github.com/syncthing/docs/pull/779

  • Ignore patterns combining `/**/` and `{a,b}` alternatives not matching as expected

    Ignore patterns combining `/**/` and `{a,b}` alternatives not matching as expected

    For reference: https://forum.syncthing.net/t/confusing-behaviour-when-trying-to-ignore-direct-and-indirect-child-files-involving-curly-bracketed-alternatives/19536

    1. Create the following folder structure
    Documents/document.txt
    Documents/Photos/document.txt
    
    1. Add Documents/**/document.txt to ignore patterns. Both files have been ignored.
    2. Replace the patterns with Documents/**/{document.txt}. Only Documents/Photos/document.txt has been ignored.
    3. Replace the patterns with Documents{/**/}{document.txt}. Now both files have been ignored again.

    The behaviour doesn't seem consistent because a) both files should be ignored in 3), and b) there should be no difference between 3) and 4) while in reality the two patterns behave differently.

    For the record, tested under Windows 10 and Syncthing v1.22.2.

Magma is an open-source software platform that gives network operators an open, flexible and extendable mobile core network solution.
Magma is an open-source software platform that gives network operators an open, flexible and extendable mobile core network solution.

Connecting the Next Billion People Magma is an open-source software platform that gives network operators an open, flexible and extendable mobile core

Dec 31, 2022
Open Source HTTP Reverse Proxy Cache and Time Series Dashboard Accelerator
Open Source HTTP Reverse Proxy Cache and Time Series Dashboard Accelerator

Trickster is an HTTP reverse proxy/cache for http applications and a dashboard query accelerator for time series databases. Learn more below, and chec

Jan 2, 2023
Open source 5G core network base on 3GPP R15
Open source 5G core network base on 3GPP R15

What is free5GC The free5GC is an open-source project for 5th generation (5G) mobile core networks. The ultimate goal of this project is to implement

Jan 4, 2023
Squzy - is a high-performance open-source monitoring, incident and alert system written in Golang with Bazel and love.

Squzy - opensource monitoring, incident and alerting system About Squzy - is a high-performance open-source monitoring and alerting system written in

Dec 12, 2022
Project Kebe is the open-source Snap Store implementation.

Introduction Kebe intends to be a full replacement for the Snap Store. Quickstart Once you have an environment setup (for instance using https://githu

Nov 9, 2022
CDN for Open Source, Non-commercial CDN management
CDN for Open Source, Non-commercial CDN management

CDN Control Official Website: https://cluckcdn.buzz Documentation (Traditional Chinese): https://cluckcdn.buzz/docs/ 简体中文 README: README_CN.md Please

Dec 20, 2021
Headscale - An open source, self-hosted implementation of the Tailscale control server

Headscale - An open source, self-hosted implementation of the Tailscale control server

Dec 29, 2022
Apache Traffic Control is an Open Source implementation of a Content Delivery Network

Apache Traffic Control Apache Traffic Control is an Open Source implementation of a Content Delivery Network. Documentation Intro CDN Basics Traffic C

Jan 6, 2023
mysshw - a free and open source ssh cli client soft.

mysshw install go version <= 1.16.* use go get go get -u github.com/cnphpbb/mysshw go version >= 1.17.* use go install go install github.com/cnphpbb/

Dec 16, 2021
gRelay is an open source project written in Go that provides the circuit break pattern with a relay idea behind.
gRelay is an open source project written in Go that provides the circuit break pattern with a relay idea behind.

gRELAY gRelay is an open source project written in Go that provides: Circuit Break ✔️ Circuit Break + Relay ✔️ Concurrecny Safe ✔️ Getting start Insta

Sep 30, 2022
An open source Pusher server implementation compatible with Pusher client libraries written in Go

Try browsing the code on Sourcegraph! IPÊ An open source Pusher server implementation compatible with Pusher client libraries written in Go. Why I wro

Aug 27, 2022
Hybridnet is an open source container networking solution, integrated with Kubernetes and used officially by following well-known PaaS platforms

Hybridnet What is Hybridnet? Hybridnet is an open source container networking solution, integrated with Kubernetes and used officially by following we

Jan 4, 2023
Openp2p - an open source, free, and lightweight P2P sharing network
Openp2p - an open source, free, and lightweight P2P sharing network

It is an open source, free, and lightweight P2P sharing network. As long as any device joins in, you can access them anywhere

Dec 31, 2022
Scout is a standalone open source software solution for DIY video security.
Scout is a standalone open source software solution for DIY video security.

scout Scout is a standalone open source software solution for DIY video security. https://www.jonoton-innovation.com Features No monthly fees! Easy In

Oct 25, 2022
NebulaChat - Open source mtproto server written in golang
 NebulaChat - Open source mtproto server written in golang

NebulaChat - Open source mtproto server written in golang open source mtproto server implemented in golang with compatible telegram client. Introduce

Jan 5, 2023
Open-source platform to request any SSP like Bidswitch or Xandr.

The project goal is to provide an unique program to contact every SSP without know the differences between all of them.

Apr 7, 2022
Kasen - Yet-another open-source CMS for scanlators

Kasen Oh no, a yet-another open-source CMS for scanlators. Anyways... The back-e

Dec 27, 2022