[jw@cn06 immudb]$ sudo make VERBOSE=1 -j1 all go build -v -ldflags '-s -X "github.com/codenotary/immudb/cmd/version.Version=0.9.2" -X "github.com/codenotary/immudb/cmd/version.Commit=b9ebfe19487a500a35a8e3ffa1158c195663844c" -X "github.com/codenotary/immudb/cmd/version.BuiltBy=" -X "github.com/codenotary/immudb/cmd/version.BuiltAt=1618902229"' ./cmd/immudb go: github.com/gizak/termui/[email protected]: reading github.com/gizak/termui/go.mod at revision v3.1.0: unknown revision v3.1.0 make: *** [Makefile:58: immudb] Error 1 [jw@cn06 immudb]$

It seems anyone can use rawsafeset command to set what ever value to what ever key, so if the server is compromised and the hacker can change values easily.

This seems contradicting the claim that :

You can only add records, but never change or delete records.

Is there anything I missed?

I'm attempting to create a large number of databases. However, when I've created about 128, then things just stop working.

I'm using 0.7.0, and going through the rest gateway.

Is there any known limit to the number of databases that can be supported? I'm hoping to create thousands for my usecase.

There is currently no rule or protocol to manage meaningful data in the communication between immudb and clients. This task aim to setup a first iteration to provide a structured value with 2 property: timestamp and payload setup by clients and managed by immudb.

Like other databases(ex mongodb) our strategy will be to setup structured values client side. This will be done inside our drivers(SDKs) or immugw. This give us 3 advantages: 1) immudb will remain simple. It will deal only with a raw bytestream as value. By this fact we will don't have to modify too much things inside the core 2) performance don't will decrease, memory usage will not increase. We don't have to open value to save timestamp serverside. In case somebody start to send us huge messages this will be a great advantage. 3) timestamp will be tamperproof because it's inserted inside value and it will be used to produce the hash for the merkletree

Main task To deal with raw structure server side we would like to use google.protobuf.Any as type of value inside KeyValue message

message KeyValue {
	bytes key = 1;
	bytes value = 2;
}

will be

message KeyValue {
	bytes key = 1;
	google.protobuf.Any value = 2;
}

In another value.proto file we define the message structure

message StructuredValue { uint64 timestamp = 1 bytes payload = 2; }

After that you will have to do a small refactor inside immudb. The inner type of any is []Byte, so you shouldn't have any problems because the old value is of the same type.

Client side Client side the implementation will be based on following pseudo code:

sv = new StructuredValue
sv.timestamp = getOsTimestamp
sv.payload = payloadByteArray
Any.pack(sv)
safesendObject = {
  "kv": {
    "key": "string",
    "value": sv
  },
  "rootIndex": {
    "index": "..."
  }
}
client.safeSend(safesendObject)

Please add a test that proofs that an element can be correctly deserialized serverside. This is needed because we will need to deserialize messages in future tasks(auth, crypto Signatures, audit ecc... )

Implementation in Immugw Immugw is an intelligent gateway in front of immudb. It make as a proxy but it simplify safeSet and safeGet calls providing autonomously the root index and, after retrieving data, it make all the verification needed in order to ensure that immudb is not been tampered. It can ran on the same machine in which the client is running, in a 3th machine between the client and immudb or just in the same machine of immudb(with a lower security). We need to provide the timestamp generation inside immugw for safeset, set and batch set. The client which is consuming immugw safeSet endpoints is completly unaware of the generation. In the safeget, get, scan and history methods the structured value has to be returned exploded in json. Please provide unit tests and api tests on safeGet and safeSet methods.

Implementation in immu client Same approch has to be done for immu client for method get, set, scan and history. Theese methods has to return timestamp in set scan and history and setup the structured value in set.

Implementation in JS-SDK, Python-SDK, .Net-SDK and JAVA-SDK Please provide the same approch for safeSet and Get methods in our SDKs. Provide also integration tests

Useful tips.

To rebuild schema.pb.go and schema.pb.gw.go:

make build/codegen

If you like to test easily immugw and immud take a look at this repo -> https://github.com/mmeloni/immudbdemo

Please use Conventional Commits -> feat(pkg/subPackage...):{commit message}

What would you like to be added or enhanced

User should be able to perform deletion as in SQL standard

Why is this needed immugorm

Additional context

It's not a standard delete as we are dealing with immutability, the data should be in db but it's not being retrieved with simple read queries - also some work on indexing is needed.

closes #730 Examples from the response of the /metrics endpoint:

• [x] DB sizes

# HELP immudb_db_size_bytes Database size in bytes.
# TYPE immudb_db_size_bytes gauge
immudb_db_size_bytes{db="cnlcdata"} 61044
immudb_db_size_bytes{db="cnlcinternalledger"} 20600
immudb_db_size_bytes{db="cnlcsyslogledger"} 2156
immudb_db_size_bytes{db="defaultdb"} 20612
immudb_db_size_bytes{db="lcledgeronekkgeln"} 107992
immudb_db_size_bytes{db="systemdb"} 11158

⚠️ The db sizes are NOT the actual size that is occupied on the disk. I tried to do it, initially, but it only worked on Linux (using Fstat from golang.org/x/sys package). I already spent to much time trying to find out how to do it in a cross-platform way, so i resorted to the same approach that is on master which uses os.FileInfo.Size().

• [x] Number of entries

# HELP immudb_number_of_stored_entries Number of key-value entries currently stored by the database.
# TYPE immudb_number_of_stored_entries gauge
immudb_number_of_stored_entries{db="cnlcdata"} 34
immudb_number_of_stored_entries{db="cnlcinternalledger"} 0
immudb_number_of_stored_entries{db="cnlcsyslogledger"} 0
immudb_number_of_stored_entries{db="defaultdb"} 0
immudb_number_of_stored_entries{db="lcledgeronekkgeln"} 111
immudb_number_of_stored_entries{db="systemdb"} 11

⚠️ The number of entries are actually the latest transaction ID of each database, as i am not aware of any other (better) way of "counting" all entries.

• [x] Update immuadmin stats command to correctly handle the new metrics format above (both for the default graphical mode and the --text mode). ⚠️ Instead of the defaultdb, the stats command now shows data for the database which has the most entries (i.e. it's assuming that that one is the one which would be of most interest for the user). In the future we should enhance it to show stats for the specified db - e.g. with a new flag --db.

• [x] I've also added CORS to metrics endpoints in a separate commit c15b64042101db0eaa3996a14ba7b320eb405f51 (i closed the separate PR #757 that i created for that)

This issue contains more details.

ℹ️ /initz is intended to be used for startup probes, /readyz for readiness probes and /livez for liveness probes.

~There is an open point though:~

• ~❓ what checks should be done in the code which is executed for GET initz, /readyz and livez (of immudb) in order to determine if immudb is healthy or not?~

UPDATE: right now these endpoints will always return 200 OK.

ℹ️ For auditor's GET /initz, /readyz and /livez (which are implemented identically right now), a Health command is issued to immudb and if there's no error and response contains Status true, then 200 OK is responded, otherwise 503 Service Unavailable.

Dear developers,

Where can i find more info about embedding immudb in a Go project? Besides some issues mention using the "embedded" package, no other info on website and github repo.

It should not be possible to edit the DB files. We need a lock on them.

We also need a security mechanism to detect that data is not lost. Check with mmap. Also be aware of NFS.

This PR aims to remove the current limitation of UPSERT ("upsert is only supported in tables without secondary indexes")

This implementation leverage new metadata record in the underlying store. Meaning this change preserves backward-compatibility.

What happened I want to run immudb as a service on macOS 11.2.3, I am following official docs It is failing when I run immudb service install in terminal. Below is the log:

xeuser@AE_C02Z94A6LVCG ~ % sudo immudb service install 
Error: exec: "groupadd": executable file not found in $PATH
Usage:
  immudb service [install uninstall start stop restart status] [flags]

Examples:
Install the immutable database
sudo ./immudb service install    -  Installs the daemon
sudo ./immudb service stop       -  Stops the daemon
sudo ./immudb service start      -  Starts initialized daemon
sudo ./immudb service restart    -  Restarts daemon
sudo ./immudb service uninstall  -  Removes daemon and its setup
Uninstall immudb after 20 second
sudo ./immudb service install --time 20 immudb

Flags:
  -h, --help       help for service
  -t, --time int   number of seconds to wait before stopping | restarting the service

Global Flags:
      --config string   config file (default path are configs or $HOME. Default filename is immudb.toml)

exec: "groupadd": executable file not found in $PATH

As far as I understand it is trying to use linux command groupadd but that doesnot exists in mac due to security reasons.

What you expected to happen It was expected to be installed without issues, it could perhaps use currently logged in user instead of trying to run groupadd

How to reproduce it (as minimally and precisely as possible) Run sudo immudb service install in macOS after setting immudb path in .bashrc or .zshrc

Environment

xeuser@AE_C02Z94A6LVCG ~ % immudb version
immudb 1.0.0
Commit  : fb5398fe7b7d69dd352685d477fbd01a86b64568
Built by: [email protected]
Built at: Fri, 21 May 2021 16:29:30 IST

This PR introduces optimizations in bulk insertions by following an optimistic concurrent approach. KV pairs to be inserted are first distributed among corresponding nodes and insertion takes place at the same time, finally the new list of nodes is built up.

Bumps github.com/jaswdr/faker from 1.4.3 to 1.16.0.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

This PR introduces support for read-only transactions.

The benefit of this type of transactions is that MVCC validations are disabled for this type of transactions. It means, read-only transactions should be faster and lighter than read-write ones.

Note: This PR may be reviewed after PR #1470

This PR is aimed to improve performance by reusing snapshots.

Transaction creation can be parametrized with conditions under which snapshots can be reused.

MVCC validations are in place, it means, if a read conflict is detected, it will be raised and the transaction may need to re-started.

Note: This PR may be reviewed after PR #1468