The lightweight, distributed relational database built on SQLite

Circle CI appveyor GoDoc Go Report Card Release Docker Google Group

rqlite is a lightweight, distributed relational database, which uses SQLite as its storage engine. Forming a cluster is very straightforward, it gracefully handles leader elections, and tolerates failures of machines, including the leader. rqlite is available for Linux, macOS, and Microsoft Windows.

Check out the rqlite FAQ.

Why?

rqlite gives you the functionality of a rock solid, fault-tolerant, replicated relational database, but with very easy installation, deployment, and operation. With it you've got a lightweight and reliable distributed relational data store. Think etcd or Consul, but with relational data modelling also available.

You could use rqlite as part of a larger system, as a central store for some critical relational data, without having to run larger, more complex distributed databases.

Finally, if you're interested in understanding how distributed systems actually work, rqlite is a good example to study. Much thought has gone into its design and implementation, with clear separation between the various components, including storage, distributed consensus, and API.

How?

rqlite uses Raft to achieve consensus across all the instances of the SQLite databases, ensuring that every change made to the system is made to a quorum of SQLite databases, or none at all. You can learn more about the design here.

Key features

Quick Start

Detailed documentation is available. You may also wish to check out the rqlite Google Group.

The quickest way to get running on macOS and Linux is to download a pre-built release binary. You can find these binaries on the Github releases page. If you prefer Windows you can download the latest build here. Once installed, you can start a single rqlite node like so:

rqlited -node-id 1 ~/node.1

Setting -node-id isn't strictly necessary at this time, but highly recommended. It makes cluster management much clearer.

This single node automatically becomes the leader. You can pass -h to rqlited to list all configuration options.

Docker

Alternatively you can pull the latest release via docker pull rqlite/rqlite.

Forming a cluster

While not strictly necessary to run rqlite, running multiple nodes means you'll have a fault-tolerant cluster. Start two more nodes, allowing the cluster to tolerate failure of a single node, like so:

rqlited -node-id 2 -http-addr localhost:4003 -raft-addr localhost:4004 -join http://localhost:4001 ~/node.2
rqlited -node-id 3 -http-addr localhost:4005 -raft-addr localhost:4006 -join http://localhost:4001 ~/node.3

This demonstration shows all 3 nodes running on the same host. In reality you probably wouldn't do this, and then you wouldn't need to select different -http-addr and -raft-addr ports for each rqlite node.

With just these few steps you've now got a fault-tolerant, distributed relational database. For full details on creating and managing real clusters, including running read-only nodes, check out this documentation.

Cluster Discovery

There is also a rqlite Discovery Service, allowing nodes to automatically connect and form a cluster. This can be much more convenient, allowing clusters to be dynamically created. Check out the documentation for more details.

Inserting records

Let's insert some records via the rqlite CLI, using standard SQLite commands. Once inserted, these records will be replicated across the cluster, in a durable and fault-tolerant manner. Your 3-node cluster can suffer the failure of a single node without any loss of functionality or data.

$ rqlite
127.0.0.1:4001> CREATE TABLE foo (id INTEGER NOT NULL PRIMARY KEY, name TEXT)
0 row affected (0.000668 sec)
127.0.0.1:4001> .schema
+-----------------------------------------------------------------------------+
| sql                                                                         |
+-----------------------------------------------------------------------------+
| CREATE TABLE foo (id INTEGER NOT NULL PRIMARY KEY, name TEXT)               |
+-----------------------------------------------------------------------------+
127.0.0.1:4001> INSERT INTO foo(name) VALUES("fiona")
1 row affected (0.000080 sec)
127.0.0.1:4001> SELECT * FROM foo
+----+-------+
| id | name  |
+----+-------+
| 1  | fiona |
+----+-------+

Data API

rqlite has a rich HTTP API, allowing full control over writing to, and querying from, rqlite. Check out the documentation for full details. There are also client libraries available.

Performance

rqlite replicates SQLite for fault-tolerance. It does not replicate it for performance. In fact performance is reduced somewhat due to the network round-trips.

Depending on your machine (particularly its IO performance) and network, individual INSERT performance could be anything from 10 operations per second to more than 200 operations per second. However, by using the bulk API, transactions, or both, throughput will increase significantly, often by 2 orders of magnitude. This speed-up is due to the way Raft and SQLite work. So for high throughput, execute as many operations as possible within a single transaction.

In-memory databases

By default rqlite uses an in-memory SQLite database to maximise performance. In this mode no actual SQLite file is created and the entire database is stored in memory. If you wish rqlite to use an actual file-based SQLite database, pass -on-disk to rqlite on start-up.

Does using an in-memory database put my data at risk?

No.

Since the Raft log is the authoritative store for all data, and it is written to disk by each node, an in-memory database can be fully recreated on start-up. Using an in-memory database does not put your data at risk.

Limitations

  • Only SQL statements that are deterministic are safe to use with rqlite, because statements are committed to the Raft log before they are sent to each node. In other words, rqlite performs statement-based replication. For example, the following statement could result in a different SQLite database under each node:
INSERT INTO foo (n) VALUES(random());
  • Technically this is not supported, but you can directly read the SQLite under any node at anytime, assuming you run in "on-disk" mode. However there is no guarantee that the SQLite file reflects all the changes that have taken place on the cluster unless you are sure the host node itself has received and applied all changes.
  • In case it isn't obvious, rqlite does not replicate any changes made directly to any underlying SQLite file, when run in "on disk" mode. If you change the SQLite file directly, you will cause rqlite to fail. Only modify the database via the HTTP API.
  • SQLite dot-commands such as .schema or .tables are not directly supported by the API, but the rqlite CLI supports some very similar functionality. This is because those commands are features of the sqlite3 command, not SQLite itself.

Status and Diagnostics

You can learn how to check status and diagnostics here.

Backup and restore

Learn how to hot backup your rqlite cluster here. You can also load data directly from a SQLite dump file.

Security

You can learn about securing access, and restricting users' access, to rqlite here.

Google Group

There is a Google Group dedicated to discussion of rqlite.

Pronunciation?

How do I pronounce rqlite? For what it's worth I try to pronounce it "ree-qwell-lite". But it seems most people, including me, often pronouce it "R Q lite".

Owner
rqlite
The lightweight, distributed relational database built on SQLite.
rqlite
Comments
  • rqlite response with

    rqlite response with "database or disk is full" on insert or create index

    I started rqlite using default config as cluster on server ( 3 nodes) , the server is 8 cores , 32Gb memory . right now still 15G memory available (free -mh) , but can not insert any more rows into db , with error message "database or disk is full" . I don't know why , how to deal with this ? here is the status result:

    {"build":{"branch":"master","build_time":"2021-08-05T21:24:10-0400","commit":"7103d425c8a50a24ffa81812d85c45d5fc26b15d7","version":"v6.1.0"},"cluster":{"addr":"127.0.0.1:4002","api_addr":"localhost:4001","https":"false","timeout":"10s"},"http":{"addr":"127.0.0.1:4001","auth":"disabled"},"node":{"start_time":"2021-08-17T14:52:54.366931301+08:00","uptime":"26h53m16.453137467s"},"runtime":{"GOARCH":"amd64","GOMAXPROCS":8,"GOOS":"linux","num_cpu":8,"num_goroutine":24,"version":"go1.15"},"store":{"addr":"127.0.0.1:4002","apply_timeout":"10s","db_applied_index":135308,"db_conf":{"Memory":true},"dir":"/root/node.1","dir_size":1220944362,"election_timeout":"1s","fsm_index":135308,"heartbeat_timeout":"1s","leader":{"addr":"127.0.0.1:4002","node_id":"1"},"node_id":"1","nodes":[{"id":"1","addr":"127.0.0.1:4002","suffrage":"Voter"},{"id":"2","addr":"127.0.0.1:4004","suffrage":"Voter"},{"id":"3","addr":"127.0.0.1:4006","suffrage":"Voter"}],"raft":{"applied_index":135308,"commit_index":135308,"fsm_pending":0,"last_contact":0,"last_log_index":135308,"last_log_term":3,"last_snapshot_index":133380,"last_snapshot_term":3,"latest_configuration":"[{Suffrage:Voter ID:1 Address:127.0.0.1:4002} {Suffrage:Voter ID:2 Address:127.0.0.1:4004} {Suffrage:Voter ID:3 Address:127.0.0.1:4006}]","latest_configuration_index":0,"log_size":807632896,"num_peers":2,"protocol_version":3,"protocol_version_max":3,"protocol_version_min":0,"snapshot_version_max":1,"snapshot_version_min":0,"state":"Leader","term":3},"request_marshaler":{"compression_batch":5,"compression_size":150,"force_compression":false},"snapshot_interval":30000000000,"snapshot_threshold":8192,"sqlite3":{"compile_options":["COMPILER=gcc-7.5.0","DEFAULT_WAL_SYNCHRONOUS=1","ENABLE_DBSTAT_VTAB","ENABLE_FTS3","ENABLE_FTS3_PARENTHESIS","ENABLE_JSON1","ENABLE_RTREE","ENABLE_UPDATE_DELETE_LIMIT","OMIT_DEPRECATED","OMIT_SHARED_CACHE","SYSTEM_MALLOC","THREADSAFE=1"],"conn_pool_stats":{"ro":{"max_open_connections":0,"open_connections":1,"in_use":0,"idle":1,"wait_count":0,"wait_duration":0,"max_idle_closed":0,"max_idle_time_closed":0,"max_lifetime_closed":0},"rw":{"max_open_connections":1,"open_connections":1,"in_use":0,"idle":1,"wait_count":0,"wait_duration":0,"max_idle_closed":0,"max_idle_time_closed":0,"max_lifetime_closed":0}},"db_size":1073741824,"path":":memory:","ro_dsn":"file:/aODMOaApCNMdLmBlgfKq?mode=ro\u0026vfs=memdb\u0026_txlock=deferred","rw_dsn":"file:/aODMOaApCNMdLmBlgfKq?mode=rw\u0026vfs=memdb\u0026_txlock=immediate","version":"3.36.0"},"trailing_logs":10240}}
    

    image image

  • Transactions?

    Transactions?

    Hi,

    I'm interested in some sort of transaction functionality beyond what rqlite currently provides today. In particular, I'd like to be able to get the results of a query in the middle of a transaction, as opposed to just "process this set of queries", the primitive that rqlite provides today.

    We could do transactions as the leader, applying them to the sqlite database and accumulating the queries to apply until the transaction was committed, but I don't think that would actually guarantee correctness (someone else who isn't in the transaction could take e.g. a primary key ID that was allocated in the transaction, since we're not committing things to the raft logs as they're added to the tx).

    Do you have any ideas on how we'd go about doing this, or if it's even possible?

    Thanks

  • cannot run quries to cluster members - only the leader responds

    cannot run quries to cluster members - only the leader responds

    after a while (a couple of hours/days) our rqlite clusters (V5.11.1) reach a point where we can only query the leader:

    for example running the query using any member other than the leader returns:

    bash-4.4$ /rqlite -H rqlite-0.rqlite
    Welcome to the rqlite CLI. Enter ".help" for usage hints.
    Connected to version v5.11.1
    rqlite-0.rqlite:4001> select * from rules
    ERR! server responded with 503 Service Unavailable: not leader
    
    rqlite-0.rqlite:4001> select * from  issues
    ERR! server responded with 503 Service Unavailable: not leader
    
    

    the status response from one of the followers is :

    
    curl http://rqlite-0.rqlite:4001/status?pretty
    
    
    {
        "build": {
            "branch": "master",
            "build_time": "2021-04-13T10:24:42-0400",
            "commit": "927611c82c72056a99e20cde3279fac7fdf51484",
            "version": "v5.11.1"
        },
        "http": {
            "addr": "10.233.122.246:4001",
            "auth": "disabled",
            "redirect": ""
        },
        "node": {
            "start_time": "2021-04-22T19:32:14.932309505Z",
            "uptime": "16h38m28.56345708s"
        },
        "runtime": {
            "GOARCH": "amd64",
            "GOMAXPROCS": 8,
            "GOOS": "linux",
            "num_cpu": 8,
            "num_goroutine": 17,
            "version": "go1.15.7"
        },
        "store": {
            "addr": "10.233.122.246:4002",
            "apply_timeout": "10s",
            "db_conf": {
                "DSN": "",
                "Memory": true
            },
            "dir": "/node",
            "dir_size": 14560570,
            "election_timeout": "5s",
            "heartbeat_timeout": "4s",
            "leader": {
                "addr": "10.233.92.133:4002",
                "node_id": "rqlite-4"
            },
            "metadata": {
                "rqlite-0": {
                    "api_addr": "rqlite-0.rqlite:4001",
                    "api_proto": "http"
                },
                "rqlite-1": {
                    "api_addr": "rqlite-1.rqlite:4001",
                    "api_proto": "http"
                },
                "rqlite-2": {
                    "api_addr": "rqlite-2.rqlite:4001",
                    "api_proto": "http"
                }
            },
            "node_id": "rqlite-0",
            "nodes": [
                {
                    "id": "rqlite-0",
                    "addr": "10.233.122.246:4002"
                },
                {
                    "id": "rqlite-1",
                    "addr": "10.233.89.103:4002"
                },
                {
                    "id": "rqlite-2",
                    "addr": "10.233.67.182:4002"
                },
                {
                    "id": "rqlite-3",
                    "addr": "10.233.100.143:4002"
                },
                {
                    "id": "rqlite-4",
                    "addr": "10.233.92.133:4002"
                }
            ],
            "raft": {
                "applied_index": 496115,
                "commit_index": 496115,
                "fsm_pending": 0,
                "last_contact": "21.858677ms",
                "last_log_index": 496115,
                "last_log_term": 28,
                "last_snapshot_index": 494286,
                "last_snapshot_term": 22,
                "latest_configuration": "[{Suffrage:Voter ID:rqlite-3 Address:10.233.100.143:4002} {Suffrage:Voter ID:rqlite-4 Address:10.233.92.133:4002} {Suffrage:Voter ID:rqlite-0 Address:10.233.122.246:4002} {Suffrage:Voter ID:rqlite-2 Address:10.233.67.182:4002} {Suffrage:Voter ID:rqlite-1 Address:10.233.89.103:4002}]",
                "latest_configuration_index": 0,
                "log_size": 8388608,
                "num_peers": 4,
                "protocol_version": 3,
                "protocol_version_max": 3,
                "protocol_version_min": 0,
                "snapshot_version_max": 1,
                "snapshot_version_min": 0,
                "state": "Follower",
                "term": 28
            },
            "request_marshaler": {
                "compression_batch": 5,
                "compression_size": 150,
                "force_compression": false
            },
            "snapshot_interval": 30000000000,
            "snapshot_threshold": 4096,
            "sqlite3": {
                "db_size": 22736896,
                "dsn": "",
                "fk_constraints": "disabled",
                "path": ":memory:",
                "version": "3.34.0"
            },
            "trailing_logs": 5120
        }
    } 
    
    

    please notice that in the status metadata there are only 3 nodes and not all 5 members appear under:

    .store.metadata
    

    they do however appear under:

    .store.nodes
    

    the leader actually does not appear in .store.metadata.

    Can yo please assist?

    Thanks in advance, L

  • Resuming instance from different network address with existing raft.db triggers unnecessary election and fails to join

    Resuming instance from different network address with existing raft.db triggers unnecessary election and fails to join

    What version are you running? Tested 5.9.0 and 6.0.0 on OSX. Was also having the same issues with the rqlite/rqlite:6.0.0 docker image.

    What did you do? Started 3 rqlited's locally on different ports to form a 3 proc cluster, say node-ids rqlite-0, rqlite-1 and rqlite-2.

    My intent was to remove rqlite-0 running on port 4001+4002 and re-add it on another port, say 4007+4008 to simulate a different network address like one may find when pods migrate around in Kubernetes.

    What did you expect to happen?

    When rejoining the cluster by either using the network addresses inside the existing raft log or by using the new -join params provided it would connect to the other members and catch its log up to the latest point and continue along its merry way at the new network addresses without needing to replicate the entire raft log.

    What happened instead?

    No matter what I tried it would try to trigger a new vote that the other nodes would reject because there was already a healthy leader.

    The only way to get it to rejoin was by wiping out the raft.db for that process and having it join fresh.

    Please include the Status, Nodes, and Expvar output from each node (or at least the Leader!)

    Start 3 rqlited's. Have the 2nd and 3rd join the first to form the cluster.

    $ rqlited -node-id=rqlite-0 -http-addr=localhost:4001 -raft-addr=localhost:4002 0
    $ rqlited -node-id=rqlite-1 -http-addr=localhost:4003 -raft-addr=localhost:4004 -join localhost:4001 1
    $ rqlited -node-id=rqlite-2 -http-addr=localhost:4005 -raft-addr=localhost:4006 -join localhost:4001 2
    

    Check cluster state:

    $ curl "http://localhost:4003/nodes?pretty"
    {
        "rqlite-0": {
            "api_addr": "http://localhost:4001",
            "addr": "127.0.0.1:4002",
            "reachable": true,
            "leader": true
        },
        "rqlite-1": {
            "api_addr": "http://localhost:4003",
            "addr": "127.0.0.1:4004",
            "reachable": true,
            "leader": false
        },
        "rqlite-2": {
            "api_addr": "http://localhost:4005",
            "addr": "127.0.0.1:4006",
            "reachable": true,
            "leader": false
        }
    }
    

    Kill rqlite-0 and check to see that the leader has moved.

    $ curl "http://localhost:4003/nodes?pretty"
    {
        "rqlite-0": {
            "addr": "127.0.0.1:4002",
            "reachable": false,
            "leader": false
        },
        "rqlite-1": {
            "api_addr": "http://localhost:4003",
            "addr": "127.0.0.1:4004",
            "reachable": true,
            "leader": false
        },
        "rqlite-2": {
            "api_addr": "http://localhost:4005",
            "addr": "127.0.0.1:4006",
            "reachable": true,
            "leader": true
        }
    }
    

    Remove rqlite-0 from the cluster

    $ MASTER=$(curl -s "http://localhost:4001/nodes" "http://localhost:4003/nodes" "http://localhost:4005/nodes" | jq -r '.[] | select(.leader==true) | .api_addr' | head -n 1)
    $ curl -XDELETE -Lv $MASTER/remove -d "{\"id\": \"rqlite-0\"}"
    

    Check cluster state:

    $ curl "http://localhost:4003/nodes?pretty"
    {
      "rqlite-1": {
        "api_addr": "http://localhost:4003",
        "addr": "127.0.0.1:4004",
        "reachable": true,
        "leader": false
      },
      "rqlite-2": {
        "api_addr": "http://localhost:4005",
        "addr": "127.0.0.1:4006",
        "reachable": true,
        "leader": true
      }
    }
    

    Restart rqlite-0 and you'll see that it tries to trigger a vote forever

    [rqlited] 2021/06/16 10:11:48 rqlited starting, version 6, commit unknown, branch unknown
    [rqlited] 2021/06/16 10:11:48 go1.16.3, target architecture is amd64, operating system target is darwin
    [rqlited] 2021/06/16 10:11:48 launch command: rqlited -node-id=rqlite-0 -http-addr=localhost:4001 -raft-addr=localhost:4002 0
    [mux] 2021/06/16 10:11:48 received handler registration request for header 1
    [mux] 2021/06/16 10:11:48 received handler registration request for header 2
    [cluster] 2021/06/16 10:11:48 service listening on 127.0.0.1:4002
    [mux] 2021/06/16 10:11:48 mux serving on 127.0.0.1:4002, advertising 127.0.0.1:4002
    [rqlited] 2021/06/16 10:11:48 preexisting node state detected in /Users/brentaylor/rqlite-data/0
    [rqlited] 2021/06/16 10:11:48 no join addresses set
    [store] 2021/06/16 10:11:48 opening store with node ID rqlite-0
    [store] 2021/06/16 10:11:48 ensuring directory at /Users/brentaylor/rqlite-data/0 exists
    [store] 2021/06/16 10:11:48 0 preexisting snapshots present
    [store] 2021/06/16 10:11:48 first log index: 1, last log index: 4, last command log index: 0:
    2021-06-16T10:11:48.141-0700 [INFO]  raft: initial configuration: index=4 servers="[{Suffrage:Voter ID:rqlite-0 Address:127.0.0.1:4002} {Suffrage:Voter ID:rqlite-1 Address:127.0.0.1:4004} {Suffrage:Voter ID:rqlite-2 Address:127.0.0.1:4006}]"
    [store] 2021/06/16 10:11:48 no cluster bootstrap requested
    2021-06-16T10:11:48.141-0700 [INFO]  raft: entering follower state: follower="Node at 127.0.0.1:4002 [Follower]" leader=
    2021-06-16T10:11:50.046-0700 [WARN]  raft: heartbeat timeout reached, starting election: last-leader=
    2021-06-16T10:11:50.046-0700 [INFO]  raft: entering candidate state: node="Node at 127.0.0.1:4002 [Candidate]" term=3
    2021-06-16T10:11:50.239-0700 [INFO]  raft: entering follower state: follower="Node at 127.0.0.1:4002 [Follower]" leader=
    2021-06-16T10:11:52.016-0700 [WARN]  raft: heartbeat timeout reached, starting election: last-leader=
    2021-06-16T10:11:52.016-0700 [INFO]  raft: entering candidate state: node="Node at 127.0.0.1:4002 [Candidate]" term=5
    2021-06-16T10:11:53.850-0700 [WARN]  raft: Election timeout reached, restarting election
    2021-06-16T10:11:53.850-0700 [INFO]  raft: entering candidate state: node="Node at 127.0.0.1:4002 [Candidate]" term=6
    2021-06-16T10:11:55.282-0700 [WARN]  raft: Election timeout reached, restarting election
    2021-06-16T10:11:55.282-0700 [INFO]  raft: entering candidate state: node="Node at 127.0.0.1:4002 [Candidate]" term=7
    ... etc etc
    [rqlited] 2021/06/16 10:13:48 leader did not appear within timeout: timeout expired
    

    And the other nodes will reject the request for a vote since a leader is already present and healthy

    2021-06-16T10:13:15.361-0700 [WARN]  raft: rejecting vote request since we have a leader: from=127.0.0.1:4002 leader=127.0.0.1:4006
    

    Kill all the rqlited's and delete the raft state on disk. Restart the cluster fresh. Once cluster is formed kill rqlite-0, remove the node and start it on new ports and tell it to join the others:

    $ MASTER=$(curl -s "http://localhost:4001/nodes" "http://localhost:4003/nodes" "http://localhost:4005/nodes" | jq -r '.[] | select(.leader==true) | .api_addr' | head -n 1)
    $ curl -XDELETE -Lv $MASTER/remove -d "{\"id\": \"rqlite-0\"}"
    rqlited -node-id=rqlite-0 -http-addr=localhost:4007 -raft-addr=localhost:4008 -join localhost:4003,localhost:4005 0
    

    The same thing happens as before where rqlite-0 triggers a vote and the others reject. Eventually rqlite-0 fails. The other nodes never consider it joined.

    $ curl "http://localhost:4003/nodes?pretty"
    {
      "rqlite-1": {
        "api_addr": "http://localhost:4003",
        "addr": "127.0.0.1:4004",
        "reachable": true,
        "leader": true
      },
      "rqlite-2": {
        "api_addr": "http://localhost:4005",
        "addr": "127.0.0.1:4006",
        "reachable": true,
        "leader": false
      }
    }
    

    If we remove rqlite-0's state it can rejoin at the new address without issue and it just replicates the raft log from the other members. This does not require the node to be removed, it gets replaced properly.

    $ rm -rf 0
    rqlited -node-id=rqlite-0 -http-addr=localhost:4007 -raft-addr=localhost:4008 -join localhost:4003,localhost:4005 0
    [rqlited] 2021/06/16 10:28:04 rqlited starting, version 6, commit unknown, branch unknown
    [rqlited] 2021/06/16 10:28:04 go1.16.3, target architecture is amd64, operating system target is darwin
    [rqlited] 2021/06/16 10:28:04 launch command: rqlited -node-id=rqlite-0 -http-addr=localhost:4007 -raft-addr=localhost:4008 -join localhost:4003,localhost:4005 0
    [mux] 2021/06/16 10:28:04 received handler registration request for header 1
    [mux] 2021/06/16 10:28:04 received handler registration request for header 2
    [cluster] 2021/06/16 10:28:04 service listening on 127.0.0.1:4008
    [mux] 2021/06/16 10:28:04 mux serving on 127.0.0.1:4008, advertising 127.0.0.1:4008
    [rqlited] 2021/06/16 10:28:04 no preexisting node state detected in /Users/brentaylor/rqlite-data/0, node may be bootstrapping
    [rqlited] 2021/06/16 10:28:04 join addresses specified, node is not bootstrapping
    [store] 2021/06/16 10:28:04 opening store with node ID rqlite-0
    [store] 2021/06/16 10:28:04 ensuring directory at /Users/brentaylor/rqlite-data/0 exists
    [store] 2021/06/16 10:28:04 0 preexisting snapshots present
    [store] 2021/06/16 10:28:04 first log index: 0, last log index: 0, last command log index: 0:
    2021-06-16T10:28:04.534-0700 [INFO]  raft: initial configuration: index=0 servers=[]
    [store] 2021/06/16 10:28:04 no cluster bootstrap requested
    [rqlited] 2021/06/16 10:28:04 join addresses are: [localhost:4003 localhost:4005]
    2021-06-16T10:28:04.534-0700 [INFO]  raft: entering follower state: follower="Node at 127.0.0.1:4008 [Follower]" leader=
    [rqlited] 2021/06/16 10:28:04 successfully joined cluster at http://localhost:4003/join
    2021-06-16T10:28:04.675-0700 [WARN]  raft: failed to get previous log: previous-index=7 last-index=0 error="log not found"
    [store] 2021/06/16 10:28:04 waiting for up to 2m0s for application of initial logs
    [rqlited] 2021/06/16 10:28:04 store has reached consensus
    [http] 2021/06/16 10:28:04 service listening on 127.0.0.1:4007
    [rqlited] 2021/06/16 10:28:04 node is ready
    
  • Problems quering from other  nodes

    Problems quering from other nodes

    Hi. I'm having problems with the cluster connection, im following the steps in the documentation, and when I connect directly to the leader through rqlite cmd and execute queries everything is ok, but when i do the same thing from other nodes, the answer to all the queries is: ERR! unexpected end of JSON input

    I thinks this is related to the recent changes made. because i use curl to make a query in a non-leader node and this is what i get:

    curl -G 'localhost:4003/db/query?pretty&timings' --data-urlencode 'q=.tables' Moved Permanently.

    btw, this is my configuration :

    rqlited ./node.1 rqlited -http localhost:4003 -raft localhost:4004 -join http://localhost:4001 ./node.2 rqlited -http localhost:4005 -raft localhost:4006 -join http://localhost:4001 ./node.3

    This is my first step on the leader node

    $ rqlite 127.0.0.1:4001> CREATE TABLE foo (id INTEGER NOT NULL PRIMARY KEY, name TEXT) 0 row affected (0.000668 sec) 127.0.0.1:4001> .schema +-----------------------------------------------------------------------------+ | sql | +-----------------------------------------------------------------------------+ | CREATE TABLE foo (id INTEGER NOT NULL PRIMARY KEY, name TEXT) | +-----------------------------------------------------------------------------+ 127.0.0.1:4001> INSERT INTO foo(name) VALUES("fiona") 1 row affected (0.000080 sec) 127.0.0.1:4001> SELECT * FROM foo +----+-------+ | id | name | +----+-------+ | 1 | fiona | +----+-------+

    When i try to do the same on a non-leader node, this is the answer

    $ rqlite -H localhost -p 4003 localhost:4003> .schema ERR! unexpected end of JSON input

    I tried to make a curl request and this is what i get $ curl -G 'localhost:4003/db/query?pretty&timings' --data-urlencode 'q=select * from foo' Moved Permanently.

    This behavior was not present on the previous releases. I hope you can help me with this.

  • rqlite nodes crashine with no apparent reason

    rqlite nodes crashine with no apparent reason

    What version are you running? verision 6.0.1

    What did you do? ran a cluster of 5 rqlite nodes on k8s

    What did you expect to happen? rqlite nodes should be stable

    What happened instead? nodes crash with the following log:

    2021-07-21T18:04:09.076Z [WARN]  raft: heartbeat timeout reached, starting election: last-leader=10.244.250.158:4002
    2021-07-21T18:04:09.076Z [INFO]  raft: entering candidate state: node="Node at 10.244.242.39:4002 [Candidate]" term=3
    2021-07-21T18:04:09.082Z [INFO]  raft: election won: tally=2
    2021-07-21T18:04:09.082Z [INFO]  raft: entering leader state: leader="Node at 10.244.242.39:4002 [Leader]"
    2021-07-21T18:04:09.082Z [INFO]  raft: added peer, starting replication: peer=rqlite-0
    2021-07-21T18:04:09.082Z [INFO]  raft: added peer, starting replication: peer=rqlite-4
    2021-07-21T18:04:09.083Z [WARN]  raft: appendEntries rejected, sending older logs: peer="{Voter rqlite-0 10.244.250.158:4002}" next=1
    2021-07-21T18:04:09.084Z [INFO]  raft: pipelining replication: peer="{Voter rqlite-4 10.244.42.143:4002}"
    2021-07-21T18:04:09.352Z [INFO]  raft: pipelining replication: peer="{Voter rqlite-0 10.244.250.158:4002}"
    [rqlited] 2021/07/21 18:04:10 http: panic serving 10.244.193.211:50948: interface conversion: interface {} is nil, not *store.fsmExecuteResponse
    goroutine 2256 [running]:
    net/http.(*conn).serve.func1(0xc000250460)
    	/usr/local/go/src/net/http/server.go:1824 +0x153
    panic(0xbb7fa0, 0xc00037e000)
    	/usr/local/go/src/runtime/panic.go:971 +0x499
    github.com/rqlite/rqlite/store.(*Store).execute(0xc000240000, 0xc000ab80c0, 0x416c38, 0x48, 0xc1d520, 0xc00022a101, 0xc000226050)
    	/home/theog/go/src/rqlitesource/src/github.com/rqlite/rqlite/store/store.go:593 +0x4a9
    github.com/rqlite/rqlite/store.(*Store).Execute(0xc000240000, 0xc000ab80c0, 0x26000, 0xc000010008, 0x1, 0x1, 0x0)
    	/home/theog/go/src/rqlitesource/src/github.com/rqlite/rqlite/store/store.go:538 +0x78
    github.com/rqlite/rqlite/http.(*Service).handleExecute(0xc00016c0f0, 0xd6e0b0, 0xc000ab0000, 0xc000456100)
    	/home/theog/go/src/rqlitesource/src/github.com/rqlite/rqlite/http/service.go:740 +0x41c
    github.com/rqlite/rqlite/http.(*Service).ServeHTTP(0xc00016c0f0, 0xd6e0b0, 0xc000ab0000, 0xc000456100)
    	/home/theog/go/src/rqlitesource/src/github.com/rqlite/rqlite/http/service.go:262 +0x651
    net/http.serverHandler.ServeHTTP(0xc000ab01c0, 0xd6e0b0, 0xc000ab0000, 0xc000456100)
    	/usr/local/go/src/net/http/server.go:2887 +0xa3
    net/http.(*conn).serve(0xc000250460, 0xd6f660, 0xc000411100)
    	/usr/local/go/src/net/http/server.go:1952 +0x8cd
    created by net/http.(*Server).Serve
    	/usr/local/go/src/net/http/server.go:3013 +0x39b
    panic: interface conversion: driver.Stmt is nil, not *sqlite3.SQLiteStmt
    
    goroutine 37 [running]:
    github.com/rqlite/go-sqlite3.(*SQLiteConn).exec(0xc000218240, 0xd6f5f0, 0xc0000c8060, 0xc000a6c000, 0x1e1c6, 0x11628f8, 0x0, 0x0, 0xc00043b960, 0x47781b, ...)
    	/home/theog/go/pkg/mod/github.com/rqlite/[email protected]/sqlite3.go:803 +0x78e
    github.com/rqlite/go-sqlite3.(*SQLiteConn).Exec(0xc000218240, 0xc000a6c000, 0x1e1c6, 0x0, 0x0, 0x0, 0x0, 0x0, 0x1, 0x0)
    	/home/theog/go/pkg/mod/github.com/rqlite/[email protected]/sqlite3.go:792 +0x138
    github.com/rqlite/rqlite/db.(*DB).Execute.func1(0xc00043baf0, 0xc9a400, 0xc0002005a0, 0xc0005b6c30, 0x1501, 0x0, 0x0)
    	/home/theog/go/src/rqlitesource/src/github.com/rqlite/rqlite/db/db.go:321 +0x2b4
    github.com/rqlite/rqlite/db.(*DB).Execute(0xc0002005a0, 0xc0005b6c30, 0xc000420101, 0x0, 0x0, 0x0, 0x0, 0xc00043bc18)
    	/home/theog/go/src/rqlitesource/src/github.com/rqlite/rqlite/db/db.go:356 +0xa5
    github.com/rqlite/rqlite/store.(*Store).Apply(0xc000240000, 0xc000490398, 0x0, 0x0)
    	/home/theog/go/src/rqlitesource/src/github.com/rqlite/rqlite/store/store.go:940 +0x3c7
    github.com/hashicorp/raft.(*Raft).runFSM.func1(0xc00058a5c0)
    	/home/theog/go/pkg/mod/github.com/hashicorp/[email protected]/fsm.go:90 +0x2c6
    github.com/hashicorp/raft.(*Raft).runFSM.func2(0xc000376400, 0x1, 0x40)
    	/home/theog/go/pkg/mod/github.com/hashicorp/[email protected]/fsm.go:113 +0x78
    github.com/hashicorp/raft.(*Raft).runFSM(0xc000272000)
    	/home/theog/go/pkg/mod/github.com/hashicorp/[email protected]/fsm.go:216 +0x392
    github.com/hashicorp/raft.(*raftState).goFunc.func1(0xc000272000, 0xc000204280)
    	/home/theog/go/pkg/mod/github.com/hashicorp/[email protected]/state.go:146 +0x55
    created by github.com/hashicorp/raft.(*raftState).goFunc
    	/home/theog/go/pkg/mod/github.com/hashicorp/[email protected]/state.go:144 +0x66
    
    
  • "clustering failure: error reading DNS configuration: invalid character '\'' " in kube stateufll set

    What version are you running?

    [rqlited] 2022/02/03 22:12:15 rqlited starting, version v7.2.0, commit 73681a69f692f2a75b9d1e570261ac3e4a73631e, branch master, compiler gc

    What did you do?

    I have applied the headless service and the statefull set

    apiVersion: v1
    kind: Service
    metadata:
      name: rqlite-svc
    spec:
      clusterIP: None 
      selector:
        app: rqlite
      ports:
        - protocol: TCP
          port: 4001
          targetPort: 4001
    
    apiVersion: apps/v1
    kind: StatefulSet
    metadata:
      name: rqlite
    spec:
      selector:
        matchLabels:
          app: rqlite # has to match .spec.template.metadata.labels
      serviceName: "rqlite"
      replicas: 3 # by default is 1
      template:
        metadata:
          labels:
            app: rqlite # has to match .spec.selector.matchLabels
        spec:
          terminationGracePeriodSeconds: 10
          containers:
          - name: rqlite
            image: rqlite/rqlite
            args: ["-disco-mode=dns","-disco-config='{\"name\":\"rqlite-svc\"}'","-bootstrap-expect 3"]
            ports:
            - containerPort: 4001
              name: rqlite
            volumeMounts:
            - name: rqlite-file
              mountPath: /rqlite/file
      volumeClaimTemplates:
      - metadata:
          name: rqlite-file
        spec:
          accessModes: [ "ReadWriteOnce" ]
          storageClassName: "standard"
          resources:
            requests:
              storage: 1Gi
    

    I get this error in the pod log after CrashLoopBackOff

    kubectl logs rqlite-0
    
                _ _ _
               | (_) |
      _ __ __ _| |_| |_ ___
     | '__/ _  | | | __/ _ \   The lightweight, distributed
     | | | (_| | | | ||  __/   relational database.
     |_|  \__, |_|_|\__\___|
             | |               www.rqlite.io
             |_|
    
    [rqlited] 2022/02/03 22:12:15 rqlited starting, version v7.2.0, commit 73681a69f692f2a75b9d1e570261ac3e4a73631e, branch master, compiler gc
    [rqlited] 2022/02/03 22:12:15 go1.17, target architecture is amd64, operating system target is linux
    [rqlited] 2022/02/03 22:12:15 launch command: /bin/rqlited -http-addr 0.0.0.0:4001 -raft-addr 0.0.0.0:4002 -http-adv-addr 10.244.1.3:4001 -raft-adv-addr 10.244.1.3:4002 -disco-mode=dns -disco-config='{"name":"rqlite-svc"}' -bootstrap-expect 3 /rqlite/file/data
    [rqlited] 2022/02/03 22:12:15 Raft TCP mux Listener registered with 1
    [rqlited] 2022/02/03 22:12:15 preexisting node state detected in /rqlite/file/data
    [store] 2022/02/03 22:12:15 opening store with node ID 10.244.1.3:4002
    [store] 2022/02/03 22:12:15 configured for an in-memory database
    [store] 2022/02/03 22:12:15 ensuring directory for Raft exists at /rqlite/file/data
    [mux] 2022/02/03 22:12:15 mux serving on [::]:4002, advertising 10.244.1.3:4002
    [store] 2022/02/03 22:12:15 0 preexisting snapshots present
    [store] 2022/02/03 22:12:15 first log index: 0, last log index: 0, last command log index: 0:
    [store] 2022/02/03 22:12:15 created in-memory database at open
    2022-02-03T22:12:15.338Z [INFO]  raft: initial configuration: index=0 servers=[]
    [cluster] 2022/02/03 22:12:15 service listening on 10.244.1.3:4002
    [rqlited] 2022/02/03 22:12:15 Cluster TCP mux Listener registered with 2
    [http] 2022/02/03 22:12:15 service listening on [::]:4001
    [rqlited] 2022/02/03 22:12:15 discovery mode: dns
    2022-02-03T22:12:15.338Z [INFO]  raft: entering follower state: follower="Node at 10.244.1.3:4002 [Follower]" leader=
    [rqlited] 2022/02/03 22:12:15 clustering failure: error reading DNS configuration: invalid character '\'' looking for beginning of value
    

    What did you expect to happen?

    The Pod should start

    What happened instead?

    The pod ends up in CrashLoopBackOff

    Please include the Status, Nodes, and Expvar output from each node (or at least the Leader!)

    See https://github.com/rqlite/rqlite/blob/master/DOC/DIAGNOSTICS.md

  • Full connection and transaction control

    Full connection and transaction control

    This change sees the addition of a new endpoint /db/connections.

    A POST to this endpoint returns a new location, that is backed by a dedicated connection to the database. On this connection transactions are supported, meaning commands such as BEGIN and COMMIT are supported. These connections can be configured to abort any transaction that has been without activity for a period of time, or close altogether for the same reason.

  • Remove all cluster aspect from rqlite for single instance purpose

    Remove all cluster aspect from rqlite for single instance purpose

    Greetings everybody!

    I realized that I didn't need any of the distributed aspect of rqlite so I decided to try to remove/implement the features that I talked about with @otoolep.

    You can follow the latest exchanges about our discussion : https://github.com/rqlite/rqlite/pull/493

    I will need some more guidance from other people who know the code better than me.

    I hope we could modularize rqlite so that both can be provided as solution.

    Cheers

  • DB always locked when running rqlite, even for reads.

    DB always locked when running rqlite, even for reads.

    I have a case where I prefer to read from the local sqlite db file directly with my own sqlite jdbc driver. I want to do this mainly for speed, and because I'm using an ORM framework that makes reading and constructing write statements for rql easier.

    But whenever RQL is running, it completely locks the db.sqlite file from reading.

    Here's the relevant part of the stack trace error I'm getting:

    Caused by: java.sql.SQLException: [SQLITE_BUSY]  The database file is locked (database is locked)
        at org.sqlite.DB.newSQLException(DB.java:383)
        at org.sqlite.DB.newSQLException(DB.java:387)
        at org.sqlite.DB.throwex(DB.java:374)
        at org.sqlite.NativeDB.prepare(Native Method)
        at org.sqlite.DB.prepare(DB.java:123)
        at org.sqlite.PrepStmt.<init>(PrepStmt.java:42)
    
  • I deployed rqlite by docker compose but it returned peer error

    I deployed rqlite by docker compose but it returned peer error

    Hi, when I deploy rqlite and a service that call rqlite, but I received an error like this

    tried all peers unsuccessfully. here are the results: peer #0: http://0.0.0.0:4001/status failed due to Get "http://0.0.0.0:4001/status": dial tcp 0.0.0.0:4001: connect: connection refused

    Here how I write the docker compose file:

    rqlite:
        image: rqlite/rqlite:5.9.0
        container_name: rqlite
        ports:
          - 4001:4001
          - 4002:4002
        restart: always
        logging:
          driver: "json-file"
          options:
            max-file: "10"
            max-size: 20m
        healthcheck:
          test: ["CMD", "curl", "http://localhost:4001/status"]
    

    And the connection uri I used is: http://rqlite:4001

    Have anyone met that issue and how to fix it?

  • Write throughput when globally distributed

    Write throughput when globally distributed

    Using this as a place to further discuss the idea of having an option to allow rqlite cache write transactions for ~100ms in order to decrease the amount of round trips per transaction (which could significantly improve write throughout).

    Are there any other thoughts since the last discussion about this?

    If we come up with a solution I’d be willing to fund the development.

  • linux/mips64 and linux/riscv64 are not supported

    linux/mips64 and linux/riscv64 are not supported

    What version are you running? 6.6.0 What did you do?

    linux/mips64 and linux/riscv64 Stderr: # github.com/boltdb/bolt ../../../pkg/mod/github.com/boltdb/[email protected]/bolt_unix.go:62:14: [maxMapSize]byte used as value ../../../pkg/mod/github.com/boltdb/[email protected]/bolt_unix.go:62:15: undefined: maxMapSize ../../../pkg/mod/github.com/boltdb/[email protected]/bucket.go:135:15: undefined: brokenUnaligned ../../../pkg/mod/github.com/boltdb/[email protected]/db.go:101:12: [maxMapSize]byte used as value ../../../pkg/mod/github.com/boltdb/[email protected]/db.go:101:13: undefined: maxMapSize ../../../pkg/mod/github.com/boltdb/[email protected]/db.go:317:12: undefined: maxMapSize ../../../pkg/mod/github.com/boltdb/[email protected]/db.go:335:10: undefined: maxMapSize ../../../pkg/mod/github.com/boltdb/[email protected]/db.go:336:8: undefined: maxMapSize ../../../pkg/mod/github.com/boltdb/[email protected]/freelist.go:169:18: [maxAllocSize]pgid used as value ../../../pkg/mod/github.com/boltdb/[email protected]/freelist.go:169:19: undefined: maxAllocSize ../../../pkg/mod/github.com/boltdb/[email protected]/freelist.go:169:18: too many errors

A distributed MySQL binlog storage system built on Raft
A distributed MySQL binlog storage system built on Raft

What is kingbus? 中文 Kingbus is a distributed MySQL binlog store based on raft. Kingbus can act as a slave to the real master and as a master to the sl

May 12, 2022
JuiceFS is a distributed POSIX file system built on top of Redis and S3.
JuiceFS is a distributed POSIX file system built on top of Redis and S3.

JuiceFS is a high-performance POSIX file system released under GNU Affero General Public License v3.0. It is specially optimized for the cloud-native

May 11, 2022
A distributed systems library for Kubernetes deployments built on top of spindle and Cloud Spanner.

hedge A library built on top of spindle and Cloud Spanner that provides rudimentary distributed computing facilities to Kubernetes deployments. Featur

Jan 4, 2022
A distributed locking library built on top of Cloud Spanner and TrueTime.

A distributed locking library built on top of Cloud Spanner and TrueTime.

May 6, 2022
Distributed lock manager. Warning: very hard to use it properly. Not because it's broken, but because distributed systems are hard. If in doubt, do not use this.

What Dlock is a distributed lock manager [1]. It is designed after flock utility but for multiple machines. When client disconnects, all his locks are

Dec 24, 2019
Distributed reliable key-value store for the most critical data of a distributed system

etcd Note: The main branch may be in an unstable or even broken state during development. For stable versions, see releases. etcd is a distributed rel

May 8, 2022
CockroachDB - the open source, cloud-native distributed SQL database.
CockroachDB - the open source, cloud-native distributed SQL database.

CockroachDB is a cloud-native distributed SQL database designed to build, scale, and manage modern, data-intensive applications. What is CockroachDB?

May 14, 2022
A linearizability distributed database by raft and wisckey.

AlfheimDB A linearizability distributed database by raft and wisckey, which supports redis client. Build This project build by mage, you will need ins

Apr 27, 2022
Distributed disk storage database based on Raft and Redis protocol.
Distributed disk storage database based on Raft and Redis protocol.

IceFireDB Distributed disk storage system based on Raft and RESP protocol. High performance Distributed consistency Reliable LSM disk storage Cold and

May 6, 2022
Lightweight, fault-tolerant message streams.
Lightweight, fault-tolerant message streams.

Liftbridge provides lightweight, fault-tolerant message streams by implementing a durable stream augmentation for the NATS messaging system. It extend

May 11, 2022
May 11, 2022
Kafka implemented in Golang with built-in coordination (No ZK dep, single binary install, Cloud Native)

Jocko Kafka/distributed commit log service in Go. Goals of this project: Implement Kafka in Go Protocol compatible with Kafka so Kafka clients and ser

May 7, 2022
distributed data sync with operational transformation/transforms

DOT The DOT project is a blend of operational transformation, CmRDT, persistent/immutable datastructures and reactive stream processing. This is an im

Apr 5, 2022
High performance, distributed and low latency publish-subscribe platform.
High performance, distributed and low latency publish-subscribe platform.

Emitter: Distributed Publish-Subscribe Platform Emitter is a distributed, scalable and fault-tolerant publish-subscribe platform built with MQTT proto

May 14, 2022
Fast, efficient, and scalable distributed map/reduce system, DAG execution, in memory or on disk, written in pure Go, runs standalone or distributedly.

Gleam Gleam is a high performance and efficient distributed execution system, and also simple, generic, flexible and easy to customize. Gleam is built

May 8, 2022
Go Micro is a framework for distributed systems development

Go Micro Go Micro is a framework for distributed systems development. Overview Go Micro provides the core requirements for distributed systems develop

May 15, 2022
Simplified distributed locking implementation using Redis

redislock Simplified distributed locking implementation using Redis. For more information, please see examples. Examples import ( "fmt" "time"

May 9, 2022
A distributed lock service in Go using etcd

locker A distributed lock service client for etcd. What? Why? A distributed lock service is somewhat self-explanatory. Locking (mutexes) as a service

Dec 2, 2021