EliasDB a graph-based database.

EliasDB

EliasDB is a graph-based database which aims to provide a lightweight solution for projects which want to store their data as a graph.

Code coverage Go Report Card Go Doc

Features

  • Build on top of a custom key-value store which supports transactions and memory-only storage.
  • Data is stored in nodes (key-value objects) which are connected via edges.
  • Stored graphs can be separated via partitions.
  • Stored graphs support cascading deletions - delete one node and all its "children".
  • All stored data is indexed and can be quickly searched via a full text phrase search.
  • EliasDB has a GraphQL interface which can be used to store and retrieve data.
  • For more complex queries EliasDB has an own query language called EQL with an sql-like syntax.
  • Written in Go from scratch. Only uses gorilla/websocket to support websockets for GraphQL subscriptions.
  • The database can be embedded or used as a standalone application.
  • When used as a standalone application it comes with an internal HTTPS webserver which provides user management, a REST API and a basic file server.
  • When used as an embedded database it supports transactions with rollbacks, iteration of data and rule based consistency management.

Getting Started (standalone application)

You can download a pre-compiled package for Windows (win64) or Linux (amd64) here.

Extract it and execute the executable with:

eliasdb server

The executable should automatically create 3 subfolders and a configuration file. It should start an HTTPS server on port 9090. To see a terminal point your webbrowser to:

https://localhost:9090/db/term.html

After accepting the self-signed certificate from the server you should see a web terminal. EliasDB can be stopped with a simple CTRL+C or by overwriting the content in eliasdb.lck with a single character.

Getting Started (docker image)

You can pull the latest docker image of EliasDB from Dockerhub:

docker pull krotik/eliasdb

Create an empty directory, change into it and run the following to start the server:

docker run --user $(id -u):$(id -g) -v $PWD:/data -p 9090:9090 krotik/eliasdb server

This exposes port 9090 from the container on the local machine. All runtime related files are written to the current directory as the current user/group.

Connect to the running server with a console by running:

docker run --rm --network="host" -it -v $PWD:/data --user $(id -u):$(id -g) -v $PWD:/data krotik/eliasdb console

Tutorial:

To get an idea of what EliasDB is about have a look at the tutorial. This tutorial will cover the basics of EQL and show how data is organized.

There is a separate tutorial on using ELiasDB with GraphQL.

REST API:

The terminal uses a REST API to communicate with the backend. The REST API can be browsed using a dynamically generated swagger.json definition (https://localhost:9090/db/swagger.json). You can browse the API of EliasDB's latest version here.

Clustering:

EliasDB supports to be run in a cluster by joining multiple instances of EliasDB together. You can read more about it here.

Command line options

The main EliasDB executable has two main tools:

Usage of ./eliasdb <tool>

EliasDB graph based database

Available commands:

    console   EliasDB server console
    server    Start EliasDB server

The most important one is server which starts the database server. The server has several options:

Usage of ./eliasdb server [options]

  -export string
    	Export the current database to a zip file
  -help
    	Show this help message
  -import string
    	Import a database from a zip file
  -no-serv
    	Do not start the server after initialization

Once the server is started the console tool can be used to interact with the server. The options of the console tool are:

Usage of ./eliasdb console [options]

  -exec string
    	Execute a single line and exit
  -file string
    	Read commands from a file and exit
  -help
    	Show this help message
  -host string
    	Host of the EliasDB server (default "localhost")
  -port string
    	Port of the EliasDB server (default "9090")

On the console type 'q' to exit and 'help' to get an overview of available commands:

Command Description
export  Exports the last output.
find    Do a full-text search of the database.
help    Display descriptions for all available commands.
info    Returns general database information.
part    Displays or sets the current partition.
ver     Displays server version information.

It is also possible to directly run EQL and GraphQL queries on the console. Use the arrow keys to cycle through the command history.

Configuration

EliasDB uses a single configuration file called eliasdb.config.json. After starting EliasDB for the first time it should create a default configuration file. Available configurations are:

Configuration Option Description
ClusterConfigFile Cluster configuration file.
ClusterLogHistory File which is used to store the console history.
ClusterStateInfoFile File which is used to store the cluster state.
CookieMaxAgeSeconds Lifetime for cookies used by EliasDB.
EnableAccessControl Flag if access control for EliasDB should be enabled. This provides user authentication and authorization features.
EnableCluster Flag if EliasDB clustering support should be enabled. EXPERIMENTAL!
EnableClusterTerminal Flag if the cluster terminal file /web/db/cluster.html should be created.
EnableReadOnly Flag if the datastore should be open read-only.
EnableWebFolder Flag if the files in the webfolder /web should be served up by the webserver. If false only the REST API is accessible.
EnableWebTerminal Flag if the web terminal file /web/db/term.html should be created.
HTTPSCertificate Name of the webserver certificate which should be used. A new one is created if it does not exist.
HTTPSHost Hostname the webserver should listen to. This host is also used in the dynamically generated swagger definition.
HTTPSKey Name of the webserver private key which should be used. A new one is created if it does not exist.
HTTPSPort Port on which the webserver should listen on.
LocationAccessDB File which is used to store access control information. This file can be edited while the server is running and changes will be picked up immediately.
LocationDatastore Directory for datastore files.
LocationHTTPS Directory for the webserver's SSL related files.
LocationUserDB File which is used to store (hashed) user passwords.
LocationWebFolder Directory of the webserver's webfolder.
LockFile Lockfile for the webserver which will be watched duing runtime. Replacing the content of this file with a single character will shutdown the webserver gracefully.
MemoryOnlyStorage Flag if the datastore should only be kept in memory.
ResultCacheMaxAgeSeconds EQL queries create result sets which are cached. The value describes the amount of time in seconds a result is kept in the cache.
ResultCacheMaxSize EQL queries create result sets which are cached. The value describes the number of results which can be kept in the cache.

Note: It is not (and will never be) possible to access the REST API via HTTP.

Enabling Access Control

It is possible to enforce access control by enabling the EnableAccessControl configuration option. When started with enabled access control EliasDB will only allow known users to connect. Users must authenticate with a password before connecting to the web interface or the REST API. On the first start with the flag enabled the following users are created by default:

Username Default Password Groups Description
elias elias admin/public Default Admin
johndoe doe public Default unprivileged user

Users can be managed from the console. Please do either delete the default users or change their password after starting EliasDB.

Users are organized in groups and permissions are assigned to groups. Permissions are given to endpoints of the REST API. The following permissions are available:

Type Allowed HTTP Request Type Description
Create Post Creating new data
Read Get Read data
Update Put Modify existing data
Delete Delete Delete data

The default group permissions are:

Group Path Permissions
admin /db/* CRUD
public / -R--
/css/* -R--
/db/* -R--
/img/* -R--
/js/* -R--
/vendor/* -R--

Building EliasDB

To build EliasDB from source you need to have Go installed (go >= 1.12):

  • Create a directory, change into it and run:
git clone https://devt.de/krotik/eliasdb/ .
  • You can build EliasDB's executable with:
go build cli/eliasdb.go

Building EliasDB as Docker image

EliasDB can be build as a secure and compact Docker image.

  • Create a directory, change into it and run:
git clone https://devt.de/krotik/eliasdb/ .
  • You can now build the Docker image with:
docker build --tag krotik/eliasdb .

Example Applications

  • Chat - A simple chat application showing user /management and subscriptions.
  • Data-mining - A more complex application which uses the cluster feature of EliasDB and GraphQL for data queries.

Further Reading

  • A design document which describes the different components of the graph database. Link
  • A reference for the EliasDB query language EQL. Link
  • A reference for the EliasDB's support for GraphQL. Link
  • A quick overview of what you can do when you embed EliasDB in your own Go project. Link

License

EliasDB source code is available under the Mozilla Public License.

Owner
Comments
  • tutorial demo -

    tutorial demo - "Get Line fails on Term (browser) but not in Terminal CLI

    this code: https://github.com/krotik/eliasdb/tree/master/examples/tutorial

    env

    am on mac if it helps :) golang 1.6.3

    I am running this from dist directory BTW.

    Terminal

    it works ๐Ÿ‘

    >>>get Line
    โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”
    โ”‚Line Key โ”‚Line Name               โ”‚
    โ”‚1:n:key  โ”‚1:n:name                โ”‚
    โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ค
    โ”‚10       โ”‚Piccadilly Line         โ”‚
    โ”‚13       โ”‚Docklands Light Railway โ”‚
    โ”‚12       โ”‚Waterloo & City Line    โ”‚
    โ”‚11       โ”‚Victoria Line           โ”‚
    โ”‚8        โ”‚Metropolitan Line       โ”‚
    โ”‚1        โ”‚Bakerloo Line           โ”‚
    โ”‚2        โ”‚Central Line            โ”‚
    โ”‚3        โ”‚Circle Line             โ”‚
    โ”‚9        โ”‚Northern Line           โ”‚
    โ”‚5        โ”‚East London Line        โ”‚
    โ”‚7        โ”‚Jubilee Line            โ”‚
    โ”‚6        โ”‚Hammersmith & City Line โ”‚
    โ”‚4        โ”‚District Line           โ”‚
    โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜
    >>>
    
    find liverpool
    Partition main
    Kind      Station
    โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”
    โ”‚has_rail โ”‚key โ”‚kind    โ”‚latitude โ”‚longitude โ”‚name             โ”‚zone โ”‚
    โ”œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”ค
    โ”‚true     โ”‚156 โ”‚Station โ”‚51.5178  โ”‚-0.0823   โ”‚Liverpool Street โ”‚1    โ”‚
    โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”˜
    
    
    

    Term browser

    same query fails ๐Ÿ‘Ž

    Screenshot 2021-05-17 at 13 07 04

  • EQL: count func with a conditional?

    EQL: count func with a conditional?

    Hi Krotik

    Don't really know if this is supported right now, but I'd rally need to add a condition to the @count function in a conditional statement. For instance, with the dataset provided in the tutorial, let's try to get the stations that are in 2 or more lines whose names begin with D.

    I was thinking in something like:

    get Station where @count(:Connection::Line where Name beginswith D) > 1

    Is there any possibility for this to be added to the EQL language?

    Tahnks!

  • Question : why comparaison operators don't work on string ?

    Question : why comparaison operators don't work on string ?

    Hello, I'm digging into eliasdb for few days and I'm stuck with a 'query where clause' with strings.
    I want to query nodes with a date string greater than a given string :
    get message where date > 2017-03-21 for example. But I get the error Value of operand is not a number.

    Does this mean that it is not possible to make strings comparaison in EQLย ?
    Thanks in advance for your response.

  • Scalability in p2p network

    Scalability in p2p network

    Hi I'm using a p2p network that has an immutable log for each user (secure scuttlebutt). It's a kappa architecture / event sourced p2p database. What I want to do is make it CQRS by not using the database for querying, but instead run through the log inserting messages into eliasdb. That way eliasdb can be the read side only via graphql or EQL.

    I'm guessing scaling should be no problem because even though ssb comprises a global network each user generally only stores their data, their friends, and friends of friends. Blobs are stored separately to the log which is json so the log isn't actually that big. I have a huge log stored locally after setting hops=6 to get a lot of data! If it works on my log then it will be fine! It's 1.3 gig, but a normal log would only be 200-300 meg of json messages.

    contact messages are used to declare follow, unfollow, or block. I threw them all with directed edges into a graphology graph the other day and there are about 390,000 edges between users. I forget how many nodes. So that's what a large graph would look like. What do you think? Would eliasdb handle that scale, and would it handle the type of queries I would need for a social network?

  • Invalid paths

    Invalid paths

    I wanted to try this db with the memory example but it fails to run because it expects paths to be "devt.de/..." instead of "github.com/krotik/eliasdb/...".

    You should update the project's structure so it can be used as library.

  • Traversal Example

    Traversal Example

    Hello Matthias, first off, I wanted to compliment you on the project you have created. To help new users understand the concepts it be nice if you could provide an traveral example for the tutorial_data. E.g. Show all Stations for Line something

    Best regards Thomas

  • Return key?

    Return key?

    Well first off all, I like it very much. The storage engine is nice and well documented. I like how it's follows physical storage model, but dynamic size allocation. I wish I could get it for python lol. Anyway trying to see on how I can get a key if I supply just a data. The swagger looks, like it doesn't return anything but 200 when you store. Is there special sequence provider I overlooked? Thirdly, partitions are nice, but I liked how OrientDb linked them to the types(classes) so if you store Line db knows which partition it goes to. Makes for explicit configuration, but more fluid application coding. Just a suggestion. Fourthly, do you hang on Gitter or any other channels to talk about this project other then issues section?

  • godoc not allowing cross searching

    godoc not allowing cross searching

    you are using a different go docs ? I was hoping to check how many other projects are using eliasdb, but i cant do that with your project.

    https://godoc.org/?q=github.com%2Fkrotik%2Feliasdb

  • Race condition in lockfile.go:219 after DiskGraphStorage.Close()

    Race condition in lockfile.go:219 after DiskGraphStorage.Close()

    First of all thanks for the nice piece of software. While POC-ing it in our software (https://github.com/nuts-foundation/) I came across a race condition (or rather, go test -race found it) in common's lockfile.go:219 (devt.de/krotik/[email protected]/lockutil/lockfile.go:219) on this line:

    lf.running = false
    

    It's triggered by DiskGraphStorage.Close().

    You can reproduce it by running the following test with the -race flag:

    import (
    	"devt.de/krotik/eliasdb/graph"
    	"devt.de/krotik/eliasdb/graph/data"
    	"devt.de/krotik/eliasdb/graph/graphstorage"
    	"github.com/stretchr/testify/assert"
    	"os"
    	"testing"
    )
    
    func TestRacy(t *testing.T) {
    	// Cleanup afterwards
    	defer os.RemoveAll("racy-testdb")
    
    	gs, err := graphstorage.NewDiskGraphStorage("racy-testdb", false)
    	assert.NoError(t, err)
    	// Taken from https://github.com/krotik/eliasdb/blob/master/embedding.md
    	gm := graph.NewGraphManager(gs)
    	node1 := data.NewGraphNode()
    	node1.SetAttr("key", "123")
    	node1.SetAttr("kind", "mynode")
    	node1.SetAttr("name", "Node1")
    	node1.SetAttr("text", "The first stored node")
    	gm.StoreNode("main", node1)
    
    	// This causes race condition
    	assert.NoError(t, gs.Close())
    }
    

    The output is as follows:

    ==================
    WARNING: DATA RACE
    Write at 0x00c000116568 by goroutine 19:
      devt.de/krotik/common/lockutil.(*LockFile).Finish()
          <redacted>/go/pkg/mod/devt.de/krotik/[email protected]/lockutil/lockfile.go:219 +0x556
      devt.de/krotik/eliasdb/storage.(*ByteDiskStorageManager).Close()
          <redacted>/go/pkg/mod/devt.de/krotik/[email protected]/storage/diskstoragemanager.go:627 +0x4f0
      devt.de/krotik/eliasdb/storage.(*CachedDiskStorageManager).Close()
          <redacted>/go/pkg/mod/devt.de/krotik/[email protected]/storage/cacheddiskstoragemanager.go:229 +0x67
      devt.de/krotik/eliasdb/graph/graphstorage.(*DiskGraphStorage).Close()
          <redacted>/go/pkg/mod/devt.de/krotik/[email protected]/graph/graphstorage/diskgraphstorage.go:202 +0x1e1
      github.com/nuts-foundation/nuts-network/pkg/nextgen/distdoc/racy.TestRacy()
          <redacted>/workspace/nuts-network/pkg/nextgen/distdoc/racy/eliasdb_test.go:28 +0x2e1
      testing.tRunner()
          /usr/local/go/src/testing/testing.go:1127 +0x202
    
    Previous read at 0x00c000116568 by goroutine 23:
      devt.de/krotik/common/lockutil.(*LockFile).watch()
          <redacted>/go/pkg/mod/devt.de/krotik/[email protected]/lockutil/lockfile.go:82 +0x124
    
    Goroutine 19 (running) created at:
      testing.(*T).Run()
          /usr/local/go/src/testing/testing.go:1178 +0x796
      testing.runTests.func1()
          /usr/local/go/src/testing/testing.go:1449 +0xa6
      testing.tRunner()
          /usr/local/go/src/testing/testing.go:1127 +0x202
      testing.runTests()
          /usr/local/go/src/testing/testing.go:1447 +0x5aa
      testing.(*M).Run()
          /usr/local/go/src/testing/testing.go:1357 +0x4eb
      main.main()
          _testmain.go:43 +0x236
    
    Goroutine 23 (running) created at:
      devt.de/krotik/common/lockutil.(*LockFile).Start()
          <redacted>/go/pkg/mod/devt.de/krotik/[email protected]/lockutil/lockfile.go:185 +0xc5
      devt.de/krotik/eliasdb/storage.initByteDiskStorageManager()
          <redacted>/go/pkg/mod/devt.de/krotik/[email protected]/storage/diskstoragemanager.go:655 +0x1933
      devt.de/krotik/eliasdb/storage.NewByteDiskStorageManager()
          <redacted>/go/pkg/mod/devt.de/krotik/[email protected]/storage/diskstoragemanager.go:235 +0x370
      devt.de/krotik/eliasdb/storage.NewDiskStorageManager()
          <redacted>/go/pkg/mod/devt.de/krotik/[email protected]/storage/diskstoragemanager.go:101 +0x278
      devt.de/krotik/eliasdb/graph/graphstorage.(*DiskGraphStorage).StorageManager()
          <redacted>/go/pkg/mod/devt.de/krotik/[email protected]/graph/graphstorage/diskgraphstorage.go:149 +0x1b4
      devt.de/krotik/eliasdb/graph.(*Manager).getNodeStorageHTree()
          <redacted>/go/pkg/mod/devt.de/krotik/[email protected]/graph/helpers.go:213 +0x421
      devt.de/krotik/eliasdb/graph.(*Manager).storeOrUpdateNode()
          <redacted>/go/pkg/mod/devt.de/krotik/[email protected]/graph/graphmanager_nodes.go:216 +0x1ae
      devt.de/krotik/eliasdb/graph.(*Manager).StoreNode()
          <redacted>/go/pkg/mod/devt.de/krotik/[email protected]/graph/graphmanager_nodes.go:187 +0x2c9
      github.com/nuts-foundation/nuts-network/pkg/nextgen/distdoc/racy.TestRacy()
          <redacted>/workspace/nuts-network/pkg/nextgen/distdoc/racy/eliasdb_test.go:25 +0x285
      testing.tRunner()
          /usr/local/go/src/testing/testing.go:1127 +0x202
    ==================
    
  • Question: how to actually set up an eliasdb cluster?

    Question: how to actually set up an eliasdb cluster?

    I can see in the source code that there is some support for running eliasdb as a cluster. However, how should this be set up? It seems the CLi has no support for this.

  • Index and query nested attributes?

    Index and query nested attributes?

    Very cool project! Just saw it today for the first time.

    What are your thoughts on supporting nested JSON structures rather than just primitives within nodes? This is a requirement for the project I'm working on, and I'm willing to help if it's a feature you'd care to add.

    For example, earlier today after working through the tutorial, I tried to store the JSON structure below:

    > store
    {
       "nodes": [
          {
             "key":"mytest",
             "kind":"Test",
             "int":42,
             "float":3.1415926,
             "str":"foo bar",
             "nested":{
                "nested_int":12,
                "nested_float":1.234,
                "nested_str":"time flies like an arrow"
             }
          }
       ]
    }
    

    The first try, I got this result:

    SyntaxError: JSON.parse: expected property name or '}' at line 1 column 4 of the JSON data
    

    After validating the JSON, I suspected that might be masking a different error, so I put all of the JSON on a single line and tried again:

    GraphError: Could not write graph information (gob: type not registered for interface: map[string]interface {}) 
    

    So I added the appropriate gob.Register call, and rebuilt. After doing this, the store operation succeeded, but I was of course unable to query based on nested values. With this feature, I would expect the node to be returned by all the following queries:

    > get Test where nested.nested_int = 12
    > get Test where nested.nested_float > 1.0
    > get Test where nested.nested_str beginswith "time"
    > index Test nested.nested_int value 12
    > index Test nested.nested_str word "flies"
    
  • Could not take ownership of lockfile

    Could not take ownership of lockfile

    When benchmarking for a disk storage graph library I got this. The directory is made right before the test. A test graph is made if it does not exist in the storage already. .

    panic: Could not take ownership of lockfile unitTestGraph/cagesenseAlarm.nodeidx: Could not write lockfile - read result after writing: 1647273004919593180(expected: 1647273007075243120)

    Stacktrace: goroutine 56 [running]: devt.de/krotik/eliasdb/storage.initByteDiskStorageManager(0xc0008a9980) /home/torbjorn/go/pkg/mod/devt.de/krotik/[email protected]/storage/diskstoragemanager.go:657 +0xca8 devt.de/krotik/eliasdb/storage.NewByteDiskStorageManager({0xc00098f5c0, 0x7819e9}, 0x0, 0x0, 0x0, 0x0) /home/torbjorn/go/pkg/mod/devt.de/krotik/[email protected]/storage/diskstoragemanager.go:235 +0x1ef devt.de/krotik/eliasdb/storage.NewDiskStorageManager(...) /home/torbjorn/go/pkg/mod/devt.de/krotik/[email protected]/storage/diskstoragemanager.go:101 devt.de/krotik/eliasdb/graph/graphstorage.(*DiskGraphStorage).StorageManager(0xc000b7a030, {0xc0001b43a8, 0x16}, 0x1) /home/torbjorn/go/pkg/mod/devt.de/krotik/[email protected]/graph/graphstorage/diskgraphstorage.go:149 +0x125 devt.de/krotik/eliasdb/graph.(*Manager).getIndexHTree(0xc00087c280, {0x77fff7, 0x9}, {0xc00041a158, 0x5}, 0x0, {0x77e7f4, 0x4}, {0x77f747, 0x8}) /home/torbjorn/go/pkg/mod/devt.de/krotik/[email protected]/graph/helpers.go:316 +0x18d devt.de/krotik/eliasdb/graph.(*Manager).getNodeIndexHTree(...) /home/torbjorn/go/pkg/mod/devt.de/krotik/[email protected]/graph/helpers.go:283 devt.de/krotik/eliasdb/graph.(*Manager).storeOrUpdateNode(0xc00087c280, {0x77fff7, 0x9}, {0x84d370, 0xc000996590}, 0x2) /home/torbjorn/go/pkg/mod/devt.de/krotik/[email protected]/graph/graphmanager_nodes.go:243 +0xe9 devt.de/krotik/eliasdb/graph.(*Manager).StoreNode(0xc00087c280, {0x77fff7, 0x9}, {0x84d370, 0xc000996590}) /home/torbjorn/go/pkg/mod/devt.de/krotik/[email protected]/graph/graphmanager_nodes.go:200 +0x17f

  • Better GraphQL Integration

    Better GraphQL Integration

    Hi @krotik!

    EliasDB looks very promising to become a one-stop GraphQL-DB solution.

    However, I think there are some points that could increase the current experience tremendously:

    1. Better GraphQL API

    For this, you might want to look at how Dgraph handles their GraphQL API.

    The API should hide as much backend/query knowledge from the frontend-user as possible. Currently, when using GraphQL, the frontend developer has to have domain knowledge on eliasdb specific syntax like traversal arguments.

    Most clients do not care about such power. They want to do the following:

    • Filter by fields
    • Order by fields
    • Paginate

    Thats pretty much all IMHO. More complex operations could of course be still available, but maybe optional.

    Example GraphQL query:

    query {
      queryDatasets(filter: {owner: {in: ["Alice", "Bob"]}}) { // get all datasets whos owner is either alice or bob
        owner
        name
        images(filter: {extension: {eq: "jpg"} }, first: 1) { // retrieve only the first jpg image of these datasets
          filename
          thumbnail
          metadata(order: {asc: key}) { // order metadata alphabetically by their keys
             key
             value
          }
        }
      }
    }
    
    

    The above query is 100% human readable and the client does not need to care about how the underlying DB traverses the graph.

    2. Interface/Union Support

    Interfaces and Unions are defined on the GraphQL standard. They are important as they reduce query complexity and boilerplate.

    Example Query:

    query {
      queryFoldersOrFile {
        name
        ... on File {
           extension
        }
        ... on Folder {
            childs {
                __typename
            }
        }
      }
    }
    

    3. Bonus Points: Create Database From GraphQL Schema:

    Again, this example comes from how dgraph handle things.

    Given a GraphQL type schema, eliasdb could bootstrap its whole CRUD-API:

    interface UUID {
       id: String! @id //ids are automatically searchable
    }
    
    interface Timestamped {
      createdAt: DateTime! @default(add: "now") //default values for different CRUD Actions
      updatedAt: DateTime! @default(add: "now", update: "now")
    }
    
    interface Ownable {
      owner: String!
    }
    
    type Dataset implements UUID & Timestamped & Ownable {
       name: String! @index // build indices for search
       images: [Image!]! @hasInverse(field: dataset, cascadeDelete: true) // type image has an inverse edge to Dataset, delete all images when a dataset is deleted
    }
    
    type Image implements UUID & Timestamped & Ownable {
       filename: String! @index
       extension: String! @index 
       dataset: Dataset! // inverse edge to dataset type required
    }
    

    Using the above Info you could completely bootstrap and layout an EliasDB instance and generate a GraphQL CRUD API as described as in 1.

    Why not use Dgraph after all? Good Question. Here are the answers:

    • Partially Licensed / not Open-Source
    • Discontinued / Development has stalled
    • Linux Only
    • huuuuge Memory requirements
    • Missing a lot of features that will not be developed because its discontinued.

    But, Dgraph has made a lot of things right and was very close in becoming the GraphQL-DB no-code solution. So we could borrow a lot of concepts to make EliasDB better in that regard.

    Happy to hear your thoughts about this!

  • Query not returning expected result

    Query not returning expected result

    I have a graph where 4 nodes (T1 through T4) are connected like this:

    [T1] <-(E2)- [T2] <-(E3)- [T3] -(E4)-> [T4]
    

    The goal of my query is:

    Given a T1, find the T4 items.

    The kind for T1, T2 and T3 are known but the type for T4 is not. For some reason the follwing query does not work (traversing from the left to right: T1 to T2 to T3 to T4).

    get T1 where key = "%s"
        traverse in:E2:out:T2
            traverse in:E3:out:T3
                traverse out:E4:in:
                end
            end
        end
    

    Traversing the nodes and edges in reverse works fine (Right to left).

    get T4 where key = "%s"
        traverse in:E4:out:T3
            traverse out:E3:in:T2
                traverse out:E2:in:
                end
            end
        end
    show 4:n:key
    

    If I instead start on a node for T2, traverse to T1 and check the contstraint, followed by traversing T2 to T3 to T4 works fine.

    get T2
        traverse out:E2:in:T1 where key = "%s"
        end 
        traverse in:E3:out:T3
            traverse out:E4:in:
            end
        end
    

    Alternative, if a use full wildcards for the second and the third traversal, T4 is returned (plus some extra data...)

    get T1 where key = "%s"
        traverse in:E2:out:T2
            traverse :::
                traverse :::
                end
            end
        end
    show 4:n:key
    

    I can't see why the first query doesn't work. Given that traversing in the reverse works, suggests that the graph is correct. Any idea what the issue is?

  • How well does Eliasdb scale?

    How well does Eliasdb scale?

    Hello,

    I am in search of a good Graph DB in Go that can be used as part of a huge P2P project in which there could be a large number of nodes dynamically joining and leaving at various times.

    Can you please tell me how well Eliasdb scales? Thanks in advance

  • Crashing the server while checking out the tutorial

    Crashing the server while checking out the tutorial

    When I run the tutorial and enter get line (instead of get Line) I get a list with 1 column of the "Line" keys.

    When I then run get Line I don't get any result and the server shows a lot of panics with "Could not take ownership of lockfile db/mainLine.nodes: Could not write lockfile...".

    I also wonder what get line would actually mean in that case?

Native GraphQL Database with graph backend
Native GraphQL Database with graph backend

The Only Native GraphQL Database With A Graph Backend. Dgraph is a horizontally scalable and distributed GraphQL database with a graph backend. It pro

Jan 4, 2023
An open-source graph database
An open-source graph database

Cayley is an open-source database for Linked Data. It is inspired by the graph database behind Google's Knowledge Graph (formerly Freebase). Documenta

Dec 31, 2022
Hard Disk Database based on a former database

Hard Disk Database based on a former database

Nov 1, 2021
Owl is a db manager platform,committed to standardizing the data, index in the database and operations to the database, to avoid risks and failures.

Owl is a db manager platform,committed to standardizing the data, index in the database and operations to the database, to avoid risks and failures. capabilities which owl provides include Process approvalใ€sql Auditใ€sql execute and execute as crontabใ€data backup and recover .

Nov 9, 2022
Simple key value database that use json files to store the database

KValDB Simple key value database that use json files to store the database, the key and the respective value. This simple database have two gRPC metho

Nov 13, 2021
Beerus-DB: a database operation framework, currently only supports Mysql, Use [go-sql-driver/mysql] to do database connection and basic operations

Beerus-DB ยท Beerus-DB is a database operation framework, currently only supports Mysql, Use [go-sql-driver/mysql] to do database connection and basic

Oct 29, 2022
Lightweight RESTful database engine based on stack data structures
Lightweight RESTful database engine based on stack data structures

piladb [pee-lah-dee-bee]. pila means stack or battery in Spanish. piladb is a lightweight RESTful database engine based on stack data structures. Crea

Nov 27, 2022
Time Series Database based on Cassandra with Prometheus remote read/write support

SquirrelDB SquirrelDB is a scalable high-available timeseries database (TSDB) compatible with Prometheus remote storage. SquirrelDB store data in Cass

Oct 20, 2022
rosedb is an embedded and fast k-v database based on LSM + WAL
rosedb is an embedded and fast k-v database based on LSM + WAL

A simple k-v database in pure Golang, supports string, list, hash, set, sorted set.

Dec 30, 2022
Rk-db - Enterprise level database bootstrapper with YAML based on rk-entry in Golang

rk-db Enterprise level database bootstrapper with YAML in golang. This belongs to rk-boot family. We suggest use this lib from rk-boot. Database Statu

Nov 16, 2022
An embedded key/value database for Go.

bbolt bbolt is a fork of Ben Johnson's Bolt key/value store. The purpose of this fork is to provide the Go community with an active maintenance and de

Jan 1, 2023
BuntDB is an embeddable, in-memory key/value database for Go with custom indexing and geospatial support
BuntDB is an embeddable, in-memory key/value database for Go with custom indexing and geospatial support

BuntDB is a low-level, in-memory, key/value store in pure Go. It persists to disk, is ACID compliant, and uses locking for multiple readers and a sing

Dec 30, 2022
CockroachDB - the open source, cloud-native distributed SQL database.
CockroachDB - the open source, cloud-native distributed SQL database.

CockroachDB is a cloud-native SQL database for building global, scalable cloud services that survive disasters. What is CockroachDB? Docs Quickstart C

Jan 2, 2023
ACID key-value database.

Coffer Simply ACID* key-value database. At the medium or even low latency it tries to provide greater throughput without losing the ACID properties of

Dec 7, 2022
A decentralized, trusted, high performance, SQL database with blockchain features
A decentralized, trusted, high performance, SQL database with blockchain features

ไธญๆ–‡็ฎ€ไป‹ CovenantSQL(CQL) is a Byzantine Fault Tolerant relational database built on SQLite: ServerLess: Free, High Availabile, Auto Sync Database Service

Jan 3, 2023
LevelDB key/value database in Go.

This is an implementation of the LevelDB key/value database in the Go programming language. Installation go get github.com/syndtr/goleveldb/leveldb R

Jan 1, 2023
immudb - worldโ€™s fastest immutable database
immudb - worldโ€™s fastest immutable database

immudb Note: The master branch is the joint point for all ongoing development efforts. Thus it may be in an unstable state and should not be used in p

Jan 4, 2023
A high performance NoSQL Database Server powered by Go
A high performance NoSQL Database Server powered by Go

LedisDB Ledisdb is a high-performance NoSQL database library and server written in Go. It's similar to Redis but store data in disk. It supports many

Dec 26, 2022
The Prometheus monitoring system and time series database.

Prometheus Visit prometheus.io for the full documentation, examples and guides. Prometheus, a Cloud Native Computing Foundation project, is a systems

Dec 31, 2022