Go has excellent built-in support for web development, therefore it makes a lot of sense to develop a tiedot web control panel using Go.

Proposed features:

• Overview of server performance and databases
• Document management
• Index management
• Query executor

Based on github.com/dgrijalva/jwt-go implementation. I was wondering if a generic verification system can be achieved through a Authorization header?

The API should just define which kind of tokens tiedot expect before it will execute the query.

Generating the tokens itself is done by a separate server following the API specification. The goal of this is that webpages can interact with the database directly by presenting a JWT token in the "Authorization" header using ajax or a websocket.

Not sure if the API can be generic enough to be useful for most security implementations?

(https://github.com/gertcuykens/tiedot)

github.com/HouzuoGuo/tiedot/data/hashtable.go:43: constant 3735928559 overflows int

I am compiling tiedot from ARM based Linux Raspbian on my Raspberry Pi and I got the above message. Go version 1.3.

my laptop is i7 2.8G, 8G memory(3G for virtualbox), host OS win7, client OS linux mint 3.2.0-23-generic #36-Ubuntu SMP.

this is my test log:

$ tiedot -mode=bench
2013/08/14 12:36:01 GOMAXPROCS is set to 8
2013/08/14 12:36:13 File /tmp/tiedot_bench/data has grown 134217728 bytes
2013/08/14 12:36:15 File /tmp/tiedot_bench/a,b,c1662212681 has grown 67108864 bytes
2013/08/14 12:36:17 File /tmp/tiedot_bench/a,b,c1662212681 has grown 67108864 bytes
2013/08/14 12:36:19 File /tmp/tiedot_bench/c,d2035745710 has grown 67108864 bytes
2013/08/14 12:36:22 File /tmp/tiedot_bench/c,d2035745710 has grown 67108864 bytes
insert 400000: 746327 ns/iter, 1339 iter/sec
read 400000: 9181 ns/iter, 108916 iter/sec
lookup 400000: 32909 ns/iter, 30386 iter/sec
update 400000: 1775335 ns/iter, 563 iter/sec
delete 400000: 1009520 ns/iter, 990 iter/sec

is this normal? seems very poor.. and CPU seems not reach 100%, is this caused by some I/O bandwidth issue?

Hi, When I try to build my app for 32 bit machine from 64 bit machine,I am getting this error

F:\GoPATH\src\github.com\HouzuoGuo\tiedot\data\collection64.go:6: COL_FILE_GROWTH redeclared in this block previous declaration at F:\GoPATH\src\github.com\HouzuoGuo\tiedot\data\collection32.go:6 F:\GoPATH\src\github.com\HouzuoGuo\tiedot\data\hash64.go:6: HT_FILE_GROWTH redeclared in this block previous declaration at F:\GoPATH\src\github.com\HouzuoGuo\tiedot\data\hash32.go:6 F:\GoPATH\src\github.com\HouzuoGuo\tiedot\data\hash64.go:7: HASH_BITS redeclared in this block previous declaration at F:\GoPATH\src\github.com\HouzuoGuo\tiedot\data\hash32.go:7 F:\GoPATH\src\github.com\HouzuoGuo\tiedot\data\hash64.go:8: INITIAL_BUCKETS redeclared in this block previous declaration at F:\GoPATH\src\github.com\HouzuoGuo\tiedot\data\hash32.go:8 F:\GoPATH\src\github.com\HouzuoGuo\tiedot\data\hash64.go:12: HashKey redeclared in this block previous declaration at F:\GoPATH\src\github.com\HouzuoGuo\tiedot\data\hash32.go:12 F:\GoPATH\src\github.com\HouzuoGuo\tiedot\data\hash64.go:15: constant 3735928559 overflows int

Could u give me a solution for this?

Let's say I insert this:

_, err = App.Usergroups.Insert(map[string]interface{}{
    "id":         1,
    "users":      []int{1, 2, 3, 4},
})

How could I query the values in that slice?

If that's not possible, let's say I'm dumping the value in there comma-separated instead, like so:

_, err = App.Usergroups.Insert(map[string]interface{}{
    "id":         1,
    "users":    "1,2,3,4,5",
})

Could I do a LIKE-ish check on that field? Or can you only check the whole value in a Query?

I use Tiedot as embedded database on a raspberrypi (go version 1.8). After a while, I got this panic : panic: runtime error: slice bounds out of range

goroutine 217 [running]: github.com/jeromelesaux/photo/vendor/github.com/HouzuoGuo/tiedot/data.OpenDataFile(0x10b32cc0, 0x2f, 0x800000, 0x10b32cf0, 0x6096f0, 0x10bfa360) /home/jls/godev/src/github.com/jeromelesaux/photo/vendor/github.com/HouzuoGuo/tiedot/data/file.go:68 +0x2cc github.com/jeromelesaux/photo/vendor/github.com/HouzuoGuo/tiedot/data.OpenHashTable(0x10b32cc0, 0x2f, 0x4, 0x10b32cc0, 0x2f) /home/jls/godev/src/github.com/jeromelesaux/photo/vendor/github.com/HouzuoGuo/tiedot/data/hashtable.go:38 +0x8c github.com/jeromelesaux/photo/vendor/github.com/HouzuoGuo/tiedot/db.(*Col).load(0x10b7cf00, 0x10b7cf00, 0x0) /home/jls/godev/src/github.com/jeromelesaux/photo/vendor/github.com/HouzuoGuo/tiedot/db/col.go:76 +0x6b4 github.com/jeromelesaux/photo/vendor/github.com/HouzuoGuo/tiedot/db.OpenCol(0x10cd0ec0, 0x10bff9f2, 0x11, 0x13a08908, 0x0, 0x0) /home/jls/godev/src/github.com/jeromelesaux/photo/vendor/github.com/HouzuoGuo/tiedot/db/col.go:35 +0x68 github.com/jeromelesaux/photo/vendor/github.com/HouzuoGuo/tiedot/db.(*DB).load(0x10cd0ec0, 0x10cd0ec0, 0x1e0c561) /home/jls/godev/src/github.com/jeromelesaux/photo/vendor/github.com/HouzuoGuo/tiedot/db/db.go:78 +0x31c github.com/jeromelesaux/photo/vendor/github.com/HouzuoGuo/tiedot/db.OpenDB(0x109ec2a0, 0x11, 0x60c7c0, 0x13bfa240, 0x0) /home/jls/godev/src/github.com/jeromelesaux/photo/vendor/github.com/HouzuoGuo/tiedot/db/db.go:39 +0xcc

I use the tiedot version 3.4.0

How can I fix it ?

best regards

Jerome

You can create a file called data-config.json within the database directory. You only need to supply the values that differ from the defaults that were set before. When the file is not there one will be created for you.

The following can be set in the json

DocMaxRoom
DocHeader
Padding
ColFileGrowth
EntrySize
BucketHeader
PerBucket
HTFileGrowth
HashBits
InitialBuckets

@davesters Could it be there is a bug that you can not use a id field in the json document?

When I look at the code i can see that the json.id is been deleted before showing the document?

For example try saving

{
 "id":"test"
}

$ mount

tmpfs on /tmp type tmpfs (rw,nosuid,noatime,size=1024M,mode=1777)
/dev/sda4 on /home type btrfs (rw,noatime,ssd,discard,compress,subvol=@home)

$ go build && ./tiedot -mode=example

2014/06/04 22:14:17 GOMAXPROCS is set to 4
I have a collection called Feeds
I have a collection called Votes
map[name:Go 1.2 is released url:golang.org]
unexpected fault address 0x7fdad4006fff
fatal error: fault
[signal 0x7 code=0x2 addr=0x7fdad4006fff pc=0x4c1b7e]

goroutine 1 [running]:
runtime.throw(0x939bab)
    /usr/local/go/src/pkg/runtime/panic.c:464 +0x69 fp=0x7fdb455aca18
runtime.sigpanic()
    /usr/local/go/src/pkg/runtime/os_linux.c:229 +0x1ed fp=0x7fdb455aca30
github.com/HouzuoGuo/tiedot/data.OpenDataFile(0xc210063f40, 0x1e, 0x2000000, 0xc2100a08c0, 0x0, ...)
    /home/cs/gocode/src/github.com/HouzuoGuo/tiedot/data/file.go:38 +0x23e fp=0x7fdb455acac8
github.com/HouzuoGuo/tiedot/data.OpenHashTable(0xc210063f40, 0x1e, 0xc210063f60, 0x0, 0x0)
    /home/cs/gocode/src/github.com/HouzuoGuo/tiedot/data/hashtable.go:42 +0x96 fp=0x7fdb455acb08
github.com/HouzuoGuo/tiedot/db.(*Col).Index(0xc21001f870, 0xc21001eb60, 0x1, 0x1, 0x0, ...)
    /home/cs/gocode/src/github.com/HouzuoGuo/tiedot/db/col.go:154 +0x448 fp=0x7fdb455acc38
main.embeddedExample()
    /home/cs/gocode/src/github.com/HouzuoGuo/tiedot/example.go:97 +0x8bc fp=0x7fdb455ace48
main.main()
    /home/cs/gocode/src/github.com/HouzuoGuo/tiedot/main.go:72 +0x710 fp=0x7fdb455acf48
runtime.main()
    /usr/local/go/src/pkg/runtime/proc.c:220 +0x11f fp=0x7fdb455acfa0
runtime.goexit()
    /usr/local/go/src/pkg/runtime/proc.c:1394 fp=0x7fdb455acfa8

goroutine 4 [select]:
github.com/HouzuoGuo/tiedot/db.func·002()
    /home/cs/gocode/src/github.com/HouzuoGuo/tiedot/db/db.go:89 +0x2bb
created by github.com/HouzuoGuo/tiedot/db.(*DB).load
    /home/cs/gocode/src/github.com/HouzuoGuo/tiedot/db/db.go:101 +0x478

$ git log |head -n5

commit 41094b715b814c7108d2f714f4a470fa70061432
Author: HouzuoGuo <[email protected]>
Date:   Sat May 31 15:03:18 2014 +0200

    update readme

$ go version

go version go1.2.2 linux/amd64

OS=Ubuntu 13.10

$ du -sh /tmp/MyDatabase ;#After the "crash"

512M    /tmp/MyDatabase

If I change the code to create the database in ~/tmp/MyDatabase and comment out the defer of the DB, test works and du of the DB gives: $ du -sh ~/tmp/MyDatabase

56K /home/cs/tmp/MyDatabase

Something is strange here!?

The combo of GO and Tiedot looks promising. Keep up the good work!

Regards Claes

(I have /tmp on a ramdisk to spare my SSD-disk)

When using go get loveoneanother.at/tiedot, the cmd just finished with message: package loveoneanother.at/tiedot: unrecognized import path "loveoneanother.at/ti edot"

And the code could not be downloaded, so what's wrong?

Could be a feature request if it's not already in place and I just can't find it in the docs, but is there (or could there be) something like a "fuzzy" search for documents?

For example, I might need to find all docs that have a "title" field that merely contains the word "Foo" without matching the entire "title" field. Records like {"title": "Foofighters in London", ".....} would match when queried with [{"has": "foo", "in": ["title"]}]

Or is there a way of doing this already that I haven't found?

Thank you for this database. I have not done much with it yet (just played a little) but willing to use it in production.

From the docs, I suppose it does not support replication/sync, as of now. Do you have any plan for that so that it can be used alongside browser based offline database like pouchdb?

HouzuoGuo,

By incremented IDs, I mean ID 1 for first record, 2 for second record, 3 for third and so forth.

I am pretty interested in tiedot, played around it, I find it suitable for my project. One thing I am pretty annoyed of, is, it uses generated IDs. I looked into code and found that rand.Int() is used to generate new ID.

To me, it would be awesome if there won't be any issues with incremented IDs. I tried modifying Insert function little to achieve that, and achieved.

My question is that, are there any potential issues that will occur doing so?

In #157 from how I can interpret it, added json config feature to set how much space tiedot shall use when creating the database, yet there's barely any hint of where the documentation is nor any examples/guidelines on how to use it beyond godocs.

It works well when I check it yesterday, it has stores 51GB data. today it grows up to 61GB, it return nothing by http api, also by golang script. when I use Scrub to fix , panic as below

panic: runtime error: makeslice: len out of range

goroutine 1 [running]: github.com/HouzuoGuo/tiedot/data.(*Collection).Read(0xc00005e400, 0x17c0abd, 0x0, 0x0, 0x0) /root/go/src/github.com/HouzuoGuo/tiedot/data/collection.go:47 +0x11f github.com/HouzuoGuo/tiedot/data.(*Partition).ForEachDoc(0xc000060600, 0x0, 0x14a, 0xc00009de00, 0x1) /root/go/src/github.com/HouzuoGuo/tiedot/data/partition.go:133 +0xf1 github.com/HouzuoGuo/tiedot/db.(*Col).forEachDoc(0xc0000843c0, 0xc00009de00, 0x510d00) /root/go/src/github.com/HouzuoGuo/tiedot/db/col.go:116 +0x137 github.com/HouzuoGuo/tiedot/db.(*DB).Scrub(0xc0000602d0, 0x510dff, 0xd, 0x0, 0x0) /root/go/src/github.com/HouzuoGuo/tiedot/db/db.go:212 +0x581 main.main() /root/go/src/examples/main/dbtest.go:34 +0x14e