Go framework to simplify CRUD of structured data using Graph operations

gocrud

Go framework to simplify creating, reading, updating, and deleting arbitrary depth structured data — to make building REST services fast and easy.

wercker status GoDoc

Releases

Note that the master branch always refers to latest version of Gocrud, which would contain breaking changes. To use stable version of Gocrud APIs, please use the packages released via gopkg.in.

Gocrud version Install instructions Godoc Source
v1 (stable) go get -v gopkg.in/manishrjain/gocrud.v1/... godoc source
master (dev) go get -v github.com/manishrjain/gocrud/... godoc source

Questions / Support

I primarily use IRC on freenode network. Channel is #gocrud. I'm mrjn on the network. #gocrud on freenode

I also hang out at the gophers.slack.com, at #gocrud channel. Although, Slack doesn't have a linux client which is what my workstation runs, and hence, I prefer IRC over Slack. You can get an invitation to join Slack via this link: Gopher Slack Signup. You can also direct message me, my user id is @manishrjain.

Why?

Courtesy: Monish, co-founder karma.wiki

Having built over 3 different startup backends, I think a lot of time is wasted figuring out and coding CRUD for data structures. In addition, the choice of database has to be made up front, which causes a lot of headache for startup founders. Gocrud was written with the aim to make CRUD easy, and provide the flexibility to switch out both the underlying storage and search engines at any stage of development.

Data stores

Datastore Driver Available Status
LevelDB Yes Ready
MySQL Yes Needs to implement Iterate func
PostgreSQL Yes Needs to implement Iterate func
Cassandra Yes Ready
MongoDB Yes Needs to implement Iterate func
Google Datastore Yes Needs to implement Iterate func
RethinkDB Yes Needs to implement Iterate func
Amazon DynamoDB No Needs work
Datastore usage shows how to use and initialize various datastores. One can add support for more by implementing this interface:
type Store interface {
  Init(args ...string)
  Commit(its []*x.Instruction) error
  IsNew(subject string) bool
  GetEntity(subject string) ([]x.Instruction, error)
  Iterate(fromId string, num int, ch chan x.Entity) (int, error)
}

Search engines

Search Engine Drive Available
Elastic Search Yes
Solr No

Can be added by implementing these interfaces:

type Engine interface {
	Init(args ...string)
	Update(x.Doc) error
	NewQuery(kind string) Query
}

type Query interface {
	Limit(num int) Query
	Order(field string) Query
	Run() ([]x.Doc, error)
  // and few others
}

Framework

This framework is built to follow these principles:

  1. Versioning: Keep track of all edits to the data, including deletion operations.
  2. Authorship: Be able to track who edited (/deleted) what.
  3. Retention: On deletion, only mark it as deleted. Never actually delete any data.

The framework makes it easy to have Parent-Child relationships, quite common in today’s CRUD operations. For e.g.

- Posts created by User (User -> Post)
- Comments on Posts (Post -> Comment)
- Likes on Posts (Post -> Like)
- Likes on Comments (Comment -> Like)

And be able to traverse these relationships and retrieve all of the children, grandchildren etc. For e.g. (User -> Post -> [(Comment -> Like), Like])

The framework does this by utilizing Graph operations, but without using a Graph database. This means the framework can be used to quickly build a Go backend to serve arbitrarily complex data, while still using your database of choice. See example usage

Dependency management

Users who import Gocrud into their packages are responsible to organize and maintain all of their dependencies to ensure code compatibility and build reproducibility. Gocrud makes no direct use of dependency management tools like Godep.

Performance considerations

For the example, this is what gets stored in the database:

mysql> select * from instructions;
+------------+--------------+-----------+--------------------------------------+-----------+---------------------+---------+----+
| subject_id | subject_type | predicate | object                               | object_id | nano_ts             | source  | id |
+------------+--------------+-----------+--------------------------------------+-----------+---------------------+---------+----+
| uid_oNM    | User         | Post      | NULL                                 | wClGp     | 1435408916326573229 | uid_oNM |  1 |
| wClGp      | Post         | body      | "You can search for cat videos here" |           | 1435408916326573229 | uid_oNM |  2 |
| wClGp      | Post         | tags      | ["search","cat","videos"]            |           | 1435408916326573229 | uid_oNM |  3 |
| wClGp      | Post         | url       | "www.google.com"                     |           | 1435408916326573229 | uid_oNM |  4 |
| wClGp      | Post         | Like      | NULL                                 | kStx9     | 1435408916341828408 | uid_qB3 |  5 |
| kStx9      | Like         | thumb     | 1                                    |           | 1435408916341828408 | uid_qB3 |  6 |
| wClGp      | Post         | Comment   | NULL                                 | 8f78r     | 1435408916341828408 | uid_qB3 |  7 |
| 8f78r      | Comment      | body      | "Comment by on the post"             |           | 1435408916341828408 | uid_qB3 |  8 |
| wClGp      | Post         | Like      | NULL                                 | Gyd7G     | 1435408916352622582 | uid_a30 |  9 |
| Gyd7G      | Like         | thumb     | 1                                    |           | 1435408916352622582 | uid_a30 | 10 |
| 8f78r      | Comment      | Like      | NULL                                 | q2IKK     | 1435408916357443075 | uid_I5u | 11 |
| q2IKK      | Like         | thumb     | 1                                    |           | 1435408916357443075 | uid_I5u | 12 |
| 8f78r      | Comment      | Comment   | NULL                                 | g8llL     | 1435408916357443075 | uid_I5u | 13 |
| g8llL      | Comment      | body      | "Comment xv on comment"              |           | 1435408916357443075 | uid_I5u | 14 |
| q2IKK      | Like         | Comment   | NULL                                 | oaztb     | 1435408916368908590 | uid_SPX | 15 |
| oaztb      | Comment      | body      | "Comment kL on Like"                 |           | 1435408916368908590 | uid_SPX | 16 |
| 8f78r      | Comment      | censored  | true                                 |           | 1435408916377065650 | uid_D2g | 17 |
| kStx9      | Like         | _delete_  | true                                 |           | 1435408916384422689 | uid_2a5 | 18 |
+------------+--------------+-----------+--------------------------------------+-----------+---------------------+---------+----+
18 rows in set (0.00 sec)

The writes are in constant time, where each (entity,predicate) constitutes one row. As the properties per entity grow, more rows need to be read (1 row = 1 edge/predicate) to get the entity, it's predicates and it's children. This however, shouldn't be much of a concern for any standard data, which has limited number of predicates/properties per entity. Gocrud in addition, retrieves all children in parallel via goroutines, instead of retrieving them one by one.

Property value filtering, sorting, full and partial text matching are now being made available via various search engines. Gocrud provides a search interface, which provides the most common search functionality right out of the box. Thus, there's a clear distinction between data store and search right from the beginning.

Reserved keywords

The following predicates are reserved by the framework, and shouldn't be used by the caller. Currently, this guideline isn't being hardly enforced by the framework.

Predicate Meaning
_parent_ Stores an edge from child -> parent entity.
_delete_ Marks a particular entity as deleted.

Contact

Feel free to contact me at my Twitter handle @manishrjain for any discussions related to this framework. Also, feel free to send pull requests, they're welcome!

Owner
Manish R Jain
Founder & CEO: Dgraph Labs. https://dgraph.io. We're hiring!
Manish R Jain
Comments
  • Cayley  - graphos database

    Cayley - graphos database

    @manishrjain do you know Cayley ?

    What you think about integrated Gocrud with Cayley? Is it possible? If you have interest can I help you?

    regards, Joao

  • Copied social to a test as a starting point.

    Copied social to a test as a starting point.

    Added a basic end to end test using social as the starting point.

    You can test it with other engines by supplying GOCRUD_STORE_TYPE environment variable. For example:

    GOCRUD_STORE_TYPE=mongodb go test ./... -v
    
  • Allow elasticsearch configuration

    Allow elasticsearch configuration

    Problem: There is no way to configure elasticsearch (or memsearch)

    Usecase: Index versions are taking too much space and I don't need old versions but I don't know exact versions to delete. I want to specify TTL of indexes so old indexes are removed periodically.

    Possible solution:

    • Allow passing elasticsearch options to the search driver (?)
    • Expose function to configure the driver(?)
    • Something else (?)
  • Creation ts

    Creation ts

    Hey @manishrjain! Thought I'd make this pull request, since the creator/creation_ts fields were really useful in our search indexer, I thought it'd be good to have them in all queries. What do you think? Also, while making this change I believe I caught a bug in query.go:299, where data["source"] is written to outside the timestamp comparison.

  • Updated the `math/rand` source to ensure unique values each run.

    Updated the `math/rand` source to ensure unique values each run.

    The math/rand API uses a fixed seed will produce the same output on every run. Which meant that the random directory names were the same each run causing odd errors..

    See the example at https://golang.org/pkg/math/rand/

    Thanks to @broady for his help in the gopher slack!

  • Added mongodb store.

    Added mongodb store.

    New mongo store which i built as close to the pattern you had in the other stores.

    As it turns out go run example/social/social.go -store=mongodb is effectively an integration test of sorts, probably just need to pull that out into a test package and convert it into a bunch of discrete tests.

    Cheers

  • Added RethinkDB store

    Added RethinkDB store

    I saw somebody mention on a reddit thread that they would like to see a RethinkDB store so I thought I would give it a go. It is based on the cassandra store and the example "test" seems to pass. Some db setup instructions are also included.

    Let me know if you have any questions about my implementation. Also great job, this library looks really cool and it was really easy to extend!

  • Project status?

    Project status?

    Just curious about the status of this project. Are any of the core developers using it or is it mostly on ice?

    I think it seems like a really nice project and I haven't found anything similar to it (besides ORMs of course, but that's not really the same thing).

  • update Google Cloud API client import paths and more

    update Google Cloud API client import paths and more

    The Google Cloud API client libraries for Go are making some breaking changes:

    • The import paths are changing from google.golang.org/cloud/... to cloud.google.com/go/.... For example, if your code imports the BigQuery client it currently reads import "google.golang.org/cloud/bigquery" It should be changed to import "cloud.google.com/go/bigquery"
    • Client options are also moving, from google.golang.org/cloud to google.golang.org/api/option. Two have also been renamed:
      • WithBaseGRPC is now WithGRPCConn
      • WithBaseHTTP is now WithHTTPClient
    • The cloud.WithContext and cloud.NewContext methods are gone, as are the deprecated pubsub and container functions that required them. Use the Client methods of these packages instead.

    You should make these changes before September 12, 2016, when the packages at google.golang.org/cloud will go away.

  • update Google Cloud API client import paths and more

    update Google Cloud API client import paths and more

    The Google Cloud API client libraries for Go are making some breaking changes:

    • The import paths are changing from google.golang.org/cloud/... to cloud.google.com/go/.... For example, if your code imports the BigQuery client it currently reads import "google.golang.org/cloud/bigquery" It should be changed to import "cloud.google.com/go/bigquery"
    • Client options are also moving, from google.golang.org/cloud to google.golang.org/api/option. Two have also been renamed:
      • WithBaseGRPC is now WithGRPCConn
      • WithBaseHTTP is now WithHTTPClient
    • The cloud.WithContext and cloud.NewContext methods are gone, as are the deprecated pubsub and container functions that required them. Use the Client methods of these packages instead.

    You should make these changes before September 12, 2016, when the packages at google.golang.org/cloud will go away.

  • Move drivers out of the main Gocrud project

    Move drivers out of the main Gocrud project

    Drivers bring in tons of dependencies on the main core project. At any point in time, a single client only depends on one or a couple of drivers, so there's no point in bringing in all these deps.

    Gocrud core project should just focus on the main principles of crud, and the drivers should be community driven.

  • Suggestion on adding memcached/in-memory db semantically for queries ?

    Suggestion on adding memcached/in-memory db semantically for queries ?

    Hi,

    I was thinking maybe gocrud should abstract away an easy way for having a a cache for queries, i.e some simple code/line saying enable cache in my app ? Do you think there is a way to create a key from the "execution plan" (i.e identify the incoming query with some params) so one doesn't have to query the ordinary backend to re-create the json but ask the cache first ?

    I havent really made my mind up on how to do it, just that it would probably improve things.

  • Getting inconsistent json data back

    Getting inconsistent json data back

    Hello!

    Thanks for this wonderful work with the graph data model, really clever!

    I have tried to a couchbase (of course more databases ;)) , you can see it here https://github.com/lundin/gocrud/blob/master/drivers/couchbase/couchbase.go

    But the problem is the inconsistent json response i get from different runs with the same query. Sometimes i get the full response/fields back and sometimes something is lacking.

    First run missing some fields on post 2:

    { ' "Post": [ ' ' { ' ' ' "id": "hJ8BDrBuKo", ' ' ' "kind": "Post", ' ' ' "ts_millis": 0 ' ' }, ' ' { ' ' ' "body": "You can search for cat videos here", ' ' ' "id": "CNnDRX9CLW", ' ' ' "kind": "Post", ' ' ' "source": "991", ' ' ' "ts_millis": 1443789962726 ' ' } ' ], ' "id": "991", ' "kind": "User", ' "ts_millis": 0

    }

    Same query another try get full response from post2:

    { ' "Post": [ ' ' { ' ' ' "body": "You can search for cat videos here", ' ' ' "id": "CNnDRX9CLW", ' ' ' "kind": "Post", ' ' ' "source": "991", ' ' ' "tags": [ ' ' ' ' "search", ' ' ' ' "cat", ' ' ' ' "videos" ' ' ' ], ' ' ' "ts_millis": 1443789962726, ' ' ' "url": "www.google.com" ' ' }, ' ' { ' ' ' "body": "This is my second post!", ' ' ' "id": "hJ8BDrBuKo", ' ' ' "kind": "Post", ' ' ' "source": "991", ' ' ' "ts_millis": 1443789962763, ' ' ' "url": "www.google.com" ' ' } ' ], ' "id": "991", ' "kind": "User", ' "ts_millis": 0

    }

Structured Data Templates

Structured Data Templates Structured data templates are a templating engine that takes a simplified set of input parameters and transforms them into a

May 12, 2022
SliceX provides functional operations on Go slices using Go 1.18 type parameters.

SliceX provides functional operations on Go slices using Go 1.18 type parameters.

Nov 6, 2021
Graph algorithms and data structures
Graph algorithms and data structures

Your basic graph Golang library of basic graph algorithms Topological ordering, image by David Eppstein, CC0 1.0. This library offers efficient and we

Jan 2, 2023
Graph algorithms and data structures
Graph algorithms and data structures

Your basic graph Golang library of basic graph algorithms Topological ordering, image by David Eppstein, CC0 1.0. This library offers efficient and we

Jan 25, 2021
A Go implementation of a radix tree, that uses binary searches to speed up insert, retrieve and delete operations on dense trees

radixs A Go implementation of a radix tree, that uses binary searches to speed up insert, retrieve and delete operations on dense trees. This implemen

Feb 14, 2022
Graph algorithms written in Go

Graph Algorithms in Go This repository contains implementations of various graph algorithms written in Go. I’ve written them to learn about these algo

Dec 26, 2022
Some algorithms in go: maxflow(min-cuts or graph-cuts), edit-distance.

Algorithms In this repository, some algorithms are implemented in go language. GoDoc link: ed maxflow About Max-flow problem: A flow network is repres

Sep 8, 2022
dagger is a fast, concurrency safe, mutable, in-memory directed graph library with zero dependencies
dagger is a fast, concurrency safe, mutable, in-memory directed graph library with zero dependencies

dagger is a blazing fast, concurrency safe, mutable, in-memory directed graph implementation with zero dependencies

Dec 19, 2022
graph package by golang

graph-go sample package main import ( "fmt" "github.com/Iovesophy/graph-go" ) func main() { samplePlace := []graph.Node{ {ID: 1, Element: "plac

Oct 24, 2021
A Connected Graph Generator tool that construct graphs of some given size

graph graph is a Connected Graph Generator tool that construct graphs of some given size. Notice that it generates all possible connected, undirected

Nov 5, 2021
Fast Raft framework using the Redis protocol for Go
Fast Raft framework using the Redis protocol for Go

This project has been archived. Please check out Uhaha for a fitter, happier, more productive Raft framework. Finn is a fast and simple framework for

Oct 10, 2022
A simple Set data structure implementation in Go (Golang) using LinkedHashMap.

Set Set is a simple Set data structure implementation in Go (Golang) using LinkedHashMap. This library allow you to get a set of int64 or string witho

Sep 26, 2022
Some data structures and algorithms using golang

Some data structures and algorithms using golang

Aug 13, 2022
A tree like tool help you to explore data structures in your redis server
 A tree like tool help you to explore data structures in your redis server

Redis-view is a tree like tool help you explore data structures in your redis server

Mar 17, 2022
Bitset data structure

Your basic bit Set data structure for positive numbers A bit array, or bit set, is an efficient set data structure. It consists of an array that compa

Dec 29, 2022
Probabilistic set data structure
Probabilistic set data structure

Your basic Bloom filter Golang probabilistic set data structure A Bloom filter is a fast and space-efficient probabilistic data structure used to test

Dec 15, 2022
Probabilistic data structures for processing continuous, unbounded streams.

Boom Filters Boom Filters are probabilistic data structures for processing continuous, unbounded streams. This includes Stable Bloom Filters, Scalable

Dec 30, 2022
Data structure and algorithm library for go, designed to provide functions similar to C++ STL

GoSTL English | 简体中文 Introduction GoSTL is a data structure and algorithm library for go, designed to provide functions similar to C++ STL, but more p

Dec 26, 2022