Bitemporal - Toy Bitemporal Database. Learning by doing

bitemporal

Building intuition about bitemporal databases by building one for myself.

TODO:

  • API v1 done
    • Find
    • List
    • Put
    • Delete
  • History API?
  • Document new intuition about mutations + the 2D time graph
    • Valid time management as a custom "version rule"?
    • "Domain time"?
    • Explore geographical map idea. 2D of data + transaction time => 3 dimensions?
  • Separate "db" and "storage" models. first pass was blending XTDB APIs with Snodgrass style records and things are getting muddled.
  • Consider Datomic accumulate and retract event style
  • Consider immutable "storage" style
  • Should data read and write APIs return tx time and valid time context at all. maybe that is relegated to "history" APIs only
  • SQL backed impl
  • Visualization
Comments
  • Implement Wasm DB

    Implement Wasm DB

    Fully functional memory.DB accessible in JavaScript 🪄

    pushed early to verify syscall/js import works

    • UPDATE: had to fix build directives and explicitly test go build with right GOOS and GOARCH env vars for wasm files

    Consider creating common helper to parse the []js.Value args

  • set up memory/wasm. initial scaffolding

    set up memory/wasm. initial scaffolding

    Provides a WebAssembly adapter for memory.DB.

    main/ contains the registrations for db.go to WebAssembly. This is built into a gitignored asserts/ dir.

    make test-wasm starts the test-server/ server that serves the .wasm files for testing.

  • optionally write VersionedKV history from tests to files

    optionally write VersionedKV history from tests to files

    serves 2 purposes

    • snapshot/golden comparison against future test changes
    • serve as input data for charting. will help debug SQL work and also be the first foray into visualization. unfortunately go-echarts does not seem to have support for arbitrary rectangle graphs. going to need to make that external to this repo.

    blows up repo size but i think being able to notice diffs if test behavior drifts is nice. consider running CI tests with the option on and enforcing no diffs

  • Export handling of options. Disallow specifying start and end valid times in the future

    Export handling of options. Disallow specifying start and end valid times in the future

    • export ReadOpt, WriteOpt handlers. split user specified options from defaults
    • disallow specifying start and end valid times in the future
    • establish convention for updated_at and deleted_at timestamps to convey information from base table changes to state tables
    • fix TestDelete to actually use parameterized values
  • Read from separate bitemporal state table

    Read from separate bitemporal state table

    This is the most practical implementation due to complications around primary keys on the original table. Management of the state table and triggers is not implemented right now.

    Triggers will be able to replicate all writes. When valid times are not specified, trigger should have a sane default behavior. We will need to identify relevant version records on the state table and and update (and add new ones) as needed.

  • implement Get for SQL DB

    implement Get for SQL DB

    • implement Get
    • provide generic ScanToVersionedKVs and ScanToMaps helpers

    Test improvements (pattern to be applied to other test changes)

    • update TestGet to be parameterized by values appropriate for DB implementation type.
    • update TestGet to take a dbFn that returns both the seeded db and a close function
    • clean up test DB set up
  • add tokei LOC badge

    add tokei LOC badge

    The badge is kind of cute and I like the idea of this staying simple. i'd be curious mainly about non-test lines of Go.

    Just having fun adding random badges haha. See https://github.com/XAMPPRocky/tokei

  • SQL.DB interface and Select() implementation

    SQL.DB interface and Select() implementation

    sql implements a SQL-backed, SQL-queryable, bitemporal database. This implements the key-value oriented interface of bitempura.DB and provides SQL querying.

    There is a new interface in sql with 1 major addition

    // Select executes a SQL query (as of optional valid and transaction times).
    Select(query squirrel.SelectBuilder, opts ...bt.ReadOpt) (*sql.Rows, error)
    

    Example time traveling SQL query

    s := squirrel.Select("type", "SUM(balance)").
        From("balances").
        GroupBy("type").
        OrderBy("SUM(balance) DESC")
    rows, err := db.Select(s, bt.AsOfValidTime(t3), bt.AsOfTransactionTime(t2))
    

    Current explorations optimizing for easiness and speed. squirrel for query building and sqlite for test db.

    Status

    • No support for setting up SQL tables
    • All SQL DB functions are unimplemented
    • Decide if scoping the sql.DB around a single table makes sense
    • Interface changes?
      • Still need to decide what to do about versioning information + public interfaces
      • Consider just promoting a separate interface. KV interface and SQL interface should feel different
  • add missing comments for exported types

    add missing comments for exported types

    ~might remove golint on build but in general I do like the enforcement about comments on exported types which doesn't seem on in golangci-lint by default and too lazy to find it.~

    don't want to include or install it during build. ignore for now

  • Rename module and types. Add

    Rename module and types. Add "Key-Value" qualifier

    • rename to something less generic
    • update language to "key-value" database so that it more accurately represents where it is. Retract some of the "toy" language for now
    • [x] update to key-value style.
      • [x] rename id to key.
      • [x] remove Attributes type and replace with a type Value interface{} type?
        • maybe try go generics in a separate cache project(!)
      • [x] rename Find/Put to Get/Set
    • [x] decide what to do about Document... I guess it should just become "VersionedValue" and be the only place that these times are externalized.
      • [x] Add model to README
      • consider a version of the interfaces that opts in to listing version information?
  • Revisit (or kill) SQL implementation

    Revisit (or kill) SQL implementation

    I ended up getting busy and switched off this project after I got the visualization working in bitempura-viz...

    I was originally intending to take the SQL implementation to completion and set it up using a SQL trigger based approach (which I had successfully taken to production before as a transaction time state table; not bitemporal). However, the practical viability of this sort of approach is unclear. I will probably remove the current stubs for now until I ever fill out the SQL implementation in full.

This repository is where I'm learning to write a CLI using Go, while learning Go, and experimenting with Docker containers and APIs.

CLI Project This repository contains a CLI project that I've been working on for a while. It's a simple project that I've been utilizing to learn Go,

Dec 12, 2021
A toy project to play music (aha, Nose in fact) through Prometheus metrics

tiplay A toy project to play music (aha, mostly Noise in fact) through Prometheu

Jan 12, 2022
Tcpdump-webhook - Toy Sidecar Injection with Mutating Webhook

tcpdump-webhook A simple demonstration of Kubernetes Mutating Webhooks. Injects

Feb 8, 2022
Learning about containers and how they work by creating them the hard way
Learning about containers and how they work by creating them the hard way

Containers the hard way: Gocker: A mini Docker written in Go It is a set of Linux's operating system primitives that provide the illusion of a contain

Jan 7, 2023
I'd like to share random apps in the spare times. Thus, I'm going to try learning some concepts of Go and as much as I can I try to clarify each line.

go-samples I'd like to share random apps in the spare times. Thus, I'm going to try learning some concepts of Go and as much as I can I try to clarify

Mar 16, 2022
Golang-samples - Help someone need some practices when learning golang

GO Language Samples This project is to help someone need some practices when lea

Jan 11, 2022
Vilicus is an open source tool that orchestrates security scans of container images(docker/oci) and centralizes all results into a database for further analysis and metrics.
Vilicus is an open source tool that orchestrates security scans of container images(docker/oci) and centralizes all results into a database for further analysis and metrics.

Vilicus Table of Contents Overview How does it work? Architecture Development Run deployment manually Usage Example of analysis Overview Vilicus is an

Dec 6, 2022
StaticBackend is a simple backend server API handling user mgmt, database, storage and real-time component
StaticBackend is a simple backend server API handling user mgmt, database, storage and real-time component

StaticBackend is a simple backend that handles user management, database, file storage, forms, and real-time experiences via channel/topic-based communication for web and mobile applications.

Jan 7, 2023
The Oracle Database Operator for Kubernetes (a.k.a. OraOperator) helps developers, DBAs, DevOps and GitOps teams reduce the time and complexity of deploying and managing Oracle Databases

The Oracle Database Operator for Kubernetes (a.k.a. OraOperator) helps developers, DBAs, DevOps and GitOps teams reduce the time and complexity of deploying and managing Oracle Databases. It eliminates the dependency on a human operator or administrator for the majority of database operations.

Dec 14, 2022
A tiny self-contained pasting service with a built-in database.

A tiny self-contained pasting service with a built-in database.

Dec 18, 2021
Golang CRUD using database PostgreSQL, adding some fremework like mux and pq.

Golang CRUD with PostgreSQL Table of contents ?? General info Technologies Blog Setup General info GOPOST or Go-Post is a Golang REST API made to show

Nov 27, 2021
A simple operator to interact with a Postgres database.

Built following this blog. Kubebuilder commands use to set up the project mkdir second-operator cd second-operator kubebuilder init --domain arshsharm

Nov 27, 2021
CelloDB is an easy-to-use database system implemented in Go

CelloDB CelloDB is the new, easy way to create and query databases. Table of Contents Table of Contents Features More Coming Soon! Features ❌ Querying

Dec 30, 2021
Backend Project using Go with Dgraph Database and Chi as HTTP Service

Drawflow-Requests For init DGraph Database docker run --rm -it -p 8080:8080 -v ~/dgraph:/dgraph dgraph/standalone:v20.03.0 docker run --rm -it -p 8080

Dec 11, 2021
In-memory database package

Go FRM Module In-memory database package Usage Install module in your project: go get github.com/xpartacvs/go-frm Use the packages: package main impo

Dec 25, 2021
A seed repository that contains a Go project that accepts input via a REST API and saves data to an Oracle database.

rest-oracle-go-seed A seed repository that contains a Go project that accepts input via a REST API and saves data to an Oracle database. Why Oracle? T

Apr 18, 2022
Extypes - Extra data types useful for database

ExTypes Extra data types useful for database JSON Object JSON Object is useful f

Jan 27, 2022
A simple program to automatically backup a database using git. Err handling by Sentry, Reporting by Betteruptime. Made with 🩸 , 😓 & 😭

backup What is this? A Simple program to automatically backup a database using git. Err handling by Sentry, Uses heartbeats by Betteruptime Made with

Nov 4, 2022
Kalasa is a NoSQL database and provides more data structures for ease of use.
Kalasa is a NoSQL database and provides more data structures for ease of use.

Kalasa Kalasa is a NoSQL database and provides more data structures for ease of use. 简体中文 | English 特 性 嵌入的存储引擎 数据可以加密存储 可以自定义实现存储加密器 即使数据文件被拷贝,也保证存储数

Dec 31, 2022