Hi,

I'm running the public alpha release of the system on my laptop, and I can access the web interface, sign up, log in and create manual entries into the system.

When I go to register a new device via either the Desktop Client or the Chrome Plugin, I can see a successful login in the application logs, however I get the following error:

I don't know if I'm missing an Admin account here, or whether I've just failed to setup my account properly, but I thought I'd ask here in case it's staring me straight in the face!

Full output from the logs when trying to sign up:

cxdb@fdc3a6498a75:/usr/share/connector_db/bin$ ./connectordb run /usr/share/connector_db/mydatabase/db/ --join
INFO[0000] Running ConnectorDB v0.3.0a1 at http://localhost:8000 (:8000) 
INFO[0026] User 'proffalken' Joined - 136.411023ms       addr=10.222.0.1:50102 op=JOIN uri=/join
INFO[0060] 0.07 queries/s                                active=0 authfails=0 errors=0 inserts=0 rest=4 web=1
WARN[0408] Can't access this resource.                   addr=10.222.0.1:50864 code=404 dev=proffalken/user op=ReadDevice ref=93f25ba4-72ae-40ae-557b-98a6eb6345b0 uri=/api/v1/crud/proffalken/laptop
INFO[0408] 1.169345ms                                    addr=10.222.0.1:50864 dev=proffalken/user op=ReadDevice uri=/api/v1/crud/proffalken/laptop
WARN[0408] illegal base64 data at input byte 8           addr=10.222.0.1:50864 code=403 dev=proffalken/user op=CreateDevice ref=26cced8f-d4d5-4d82-5ff9-60ef656d104e uri=/api/v1/crud/proffalken/laptop

So, I just gave it a shot, and I think it's working.

• You can build it locally with sudo docker build -t heedy .
• The created image can be executed with sudo docker run --rm -it -p 1324:1324 heedy
• To have a persistent database, you can use something like sudo docker run --rm -it -p 1324:1324 -v "$PWD/tmp:/root/.local/share" heedy , which allows heedy to save its data in a tmp folder in the current directory.

I have chosen the alpine base image as it is known for its small size. It is 30.9MB on my machine, which seems reasonable. For it to work, I also needed to build the backend with the golang:alpine image. I haven't tested the api or any other functionality like that yet, but it seems to be working so far.

You can potentially speed up build times by excluding files and folders with a .dockerignore. Maybe you could also only copy folders as necessary into the build stages. I have tried to keep complexity to a minimum here. Especially if builds are usually done online once, and after you just pull the completed image, I'm not sure if the added complexity would be worth it here.

Feedback is appreciated, and I can change things as needed.

Fixes #341

It looks like auth by user/password combo is broken. I looked at the code, and it seems that there are two major issues going on:

• [x] CreateUser does not actually create a "user" device (let's make it lowercase u)
• [x] Reading a device that does not exist does not give an error DNE

Work needs to be done on the user database connector so it can work with postgres and add time series chunks along with it.

At the same time we should push the API to a deeper nested folder or to a sibbling folder since it doesn't really go along with the new database.

Do we have indicies for our database as far as point insertion? I can imagine wanting to get the earliest point and the most recent, and getting a number of points after a certain time, but we can easily do all that on dates in sql. What I'm not sure about is why you'd want to randomly get the 1000th point. You can easily fetch points x at a time by starting at a particular time, limiting to x then querying for any > the date of the last one you chose.

My question comes partly around thinking about a hbase backend in the future which would have a few extra features such as:

• intelligent sharding
• simple horizontal scalability
• easy to insert out of order points
• data deletion

Currently we have issues with out of order inserts, I see a way of fixing this, but it'll require some logic:

To insert, we append to the bytea field (I don't know if out of order stuff will be an issue though), hopefully not as the clients should sort it themselves when the get it).

UPDATE timebatchtable SET blobfield = blobfield || ? WHERE ..."

If we're inserting a non-current key, we extract the chunk it falls into add it in and push it all back.

UPDATE timebatchtable WHERE StreamId = ? SET EndIndex = EndIndex + 1

This can all be done in a transaction to avoid errors as it chugs along.

Can we add a "MimeType" field to Stream? the type field is the jsonschema, but it would be nice to have the option of adding mimetypes for common types that can be searched.

We need a benchmark for our backend services through the REST API.

What we know

• Etsy is happy with ~200ms queries on their website
• RTT to the connectordb.com is roughly 53ms
• Pulling the AskReddit homepage json takes roughly .8 seconds
• 3G latency is roughly 300ms
• Various other latencies

What we want:

• 10000 users on our service
• Each user has 5 devices and ten streams each submitting a point every 30 seconds (16000/sec)
• Every REST query goes through the system within 200ms (plus RTT)
• 20% of our userbase is doing queries at any one time
• The length of a data query is 1 month of data (we may want to set a hard limit on this to avoid DOS attacks)

This issue will track progress towards version 0.4, and the planned features. It is expected that the new version will be released some time in 2020 (depending on how much free time I have to work on it).

0.4 is a planned ground-up rewrite of ConnectorDB, with a focus on extensibility and simplicity. The highlights are planned to be a serious plugin system and easy-to-use single-file bundle. It will also be a chance for me to fix the bad design choices I made in previous versions (append-only streams!).

For the moment I am keeping the source code private due to its highly experimental nature, but it will be released in a 0.4 branch once the fundamentals are decided.

• [x] Simplicitly - drop the executable, run it, and you immediately get a working database. In particular, the following are planned:
  • [x] Single File Bundle - The executable contains all assets needed to run
  • [x] No Dependencies - By default, CDB shouldn't need any external dependencies (0.3 needed gnatsd and redis).
  • [x] Setup - Rather than modifying a config file, CDB should present a web UI to set it up.
  • [x] Setting File Overwrite & Settings Page - Settings should automatically overwrite the config file
  • [ ] 0.3 Importer - Since the database will be incompatible with 0.3, it needs to include a built-in import system to ease migration.
• [x] Sources - The user/device/stream system was OK, but it wasn't very good at distinguishing services that access the database and actual devices with streams. It was also limited to stream data, when in reality sources can be aggregations, or even other data formats. Now, streams are going to be just one of many possible sources, with the ability to add/modify sources through plugins. Apps are going to access the database through "connections" which are a separate logical entity.
  • [x] Streams - Same as old CDB streams, except they are no longer batched in the backend - the goal is simplicitly and maintainability
    • [x] Pipescript? - This probably won't make the initial release, but I have been meaning to give Pipescript a serious upgrade for over 2 years now.
      • [ ] Plugin PipeScript functions
    • [ ] Remote Streams - streams where data is not kept in CDB, but rather comes from a remote source, or is generated on demand (mediated by a plugin). Might not make initial release.
    • [x] Aggregate Streams - streams made up of multiple streams merged together, with an optional transform.
  • [ ] Groups - The ability to group together streams is useful, since streams are part of a logical object, such as a room or a person
  • [x] Permissions - Sources will be shareable, with scope-based access.
• [x] Bugs/Issues
  • [x] Modifying stream data. Currently streams are append-only. This is stupid.
• [x] Plugin System - this is really the underlying reason CDB needs a rewrite. The old version required any modification of functionality to include a full recompile, the frontend was one huge monolithic bundle that discouraged idle experimentation. This became a serious hindrance once I started trying to do automated analysis. Analysis programs need to integrate with the UI - needing a separate web app to do anything was not good. The same is true for integrations. Modifying the core code to add fitbit support or something is not the correct way to go about things!
  • [x] Stacked assets (plugins can override default assets)
  • [x] Stacked config (plugins can modify config)
  • [x] Modular UI - this requires a full UI rewrite to do correctly, but the basic app skeleton is functional, with a separate js module for each page.
  • [x] Plugin Executables - plugins can have exectutable portions, with special support for Python. These executables will have API integration, allowing them to have real power to extend CDB
    • [x] REST overrides
    • [x] Python Support - plugins will get special python support
      • [x] Plugin Python library
      • [x] Plugin system uses an installed python and sets up necessary env
  • [ ] Plugin Repository - there should be a repository online where people can submit plugins they write, and CDB should be capable of auto-installing and updating them from a UI.
• [x] Vue UI - this is mainly due to the plugin system requiring a rewrite of the frontend, but vue allowed simple page splitting, and I think it is simpler for plugin writers.
  • [x] Base theme
  • [x] Setup
  • [x] Settings Page
  • [x] Analysis
  • [x] User Page
  • [x] Stream Page
  • [ ] Dashboard Page
  • [x] Actions Page (ie: downlinks)
• [ ] Docs - Docs are maybe even more important than features. Everything, especially the plugin system needs to be documented, and examples need to be given.
• [ ] Modify Android, Desktop & Chrome Apps for new version.

Adding a log stream to the user device which is updated when something belonging to the user is changed would be very useful. To do this efficiently, the user struct would ideally have a link to the user device and log stream by id so that we don't need to do any clever searching each time something is updated

Better logging would be awesome, @dkumor says to look at gnatsd.

Personally I'd like to also see some kind of automatic structured logging, prefixes, colors and logging levels if they're not in there.

We definitely need fine grained control of logging for security so we can't overrun the server with old logs, a log rotation system could be useful.

@dkumor

• [x] streamdb
• [x] rest
• [x] timebatch

@josephlewis42

• [x] connectordb
• [x] web interface
• [x] dbutil

It'd be nice if the dkumor/heedy docker image on dockerhub would be a multi-arch image. This can be achieved by building with Docker Buildx.

I can try to do the required changes to build instructions, but I didn't find where the images are pushed, I just saw the build in the Makefile - have the recent image tags been built and pushed manually?

Reasoning: Chances are high that people want to run heedy on eg. Rasphberry pi (my old pi 3b+ has armv7l) and according to the readme docker is a supported installation method and the GitHub releases actually do have arm binaries.

Edit: I gave it a try, but it didn't work out of the box - might require some tinkering

$ DOCKER_BUILD_KIT=1 DOCKER_CLI_EXPERIMENTAL=enabled docker buildx build -t "mammuth/heedy:0.5.1" --build-arg VERSION="0.5.1" --platform linux/amd64,linux/arm64 --push .

error: failed to solve: process "/dev/.buildkit_qemu_emulator /bin/sh -c apt-get update && apt-get install -y rsync" did not complete successfully: exit code: 100

Bumps minimist from 1.2.5 to 1.2.6.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Bumps nanoid from 3.1.23 to 3.3.1.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Bumps minimist from 1.2.5 to 1.2.6.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Bumps lodash from 4.17.15 to 4.17.21.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Bumps ajv from 6.10.2 to 6.12.6.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.