Community built data connectors and processors for Spice.ai

data-components-contrib

build License: MIT Discord Subreddit subscribers Follow on Twitter

Community built data connectors and processors for Spice.ai

The vision for data-components-contrib is a community-driven library of components for streaming and processing time series data for use in the Spice.ai runtime. Spice.ai provides a general interface that anyone can implement to create data connectors and data processors. This enables both authors and consumers of Spice.ai pods, to use the same data sources for training and inferencing ML models. Join us in helping make AI easy for developers.

See CONTRIBUTING.md on how you can contribute a component.

Data Component Concepts

Dataspace

A dataspace is a specification on how the Spice.ai runtime and AI engine loads, processes and interacts with data from a single source. A dataspace may contain a single data connector and data processor. There may be multiple dataspace definitions within a pod. The fields specified in the union of dataspaces are used as inputs to the neural networks that Spice.ai trains.

Data Connector

A data connector is a reuseable component that contains logic to fetch or ingest data from an external source.

Learn more at Data Connectors

Data Processor

A data processor is a reusable component, composable with a data connector that contains logic to process raw connector data into observations and state Spice.ai can use.

Learn more at Data Processors

Owner
Spice.ai
Powerful and easy-to-use time series AI designed for developers.
Spice.ai
https://github.com/spiceai/data-components-contrib
Comments
  • Fix benchmark to us data

    Fix benchmark to us data

    The benchmark function was not consistent with the test: parameter was a fileProcessor while tests use byte array. Also the benchmark was running empty.

    Now the benchmark runs with the biggest data (COINBASE) and all are consistently asking for byte array.

    There used to be 2 ways for reading the CSV: os.ReadFile and FileProcessor. I think the csv_test should only test it's associated packages, also the code is more shorter and clearer. We can add pipeline tests latter

  • Migrate to Arrow Go

    Migrate to Arrow Go

    What

    Replace the State and Observations structures with Arrow Table & Record. Update Data Processor interface to return Table (if appropriate) otherwise Record.

    E.g. now:

    type DataProcessor interface {
	Init(params map[string]string) error
	OnData(data []byte) ([]byte, error)
	GetObservations() ([]observations.Observation, error)
	GetState(fields *[]string) ([]*state.State, error)
}

    After migration:

    type DataProcessor interface {
	Init(params map[string]string) error
	OnData(data []byte) ([]byte, error)
	GetTable() (arrow.Table, error)
        Close() error
}

    Why

    Step along to migrate the underlying data foundation to Arrow for performance and scale.

  • JSON Processor: Parse integers for categorical data

    JSON Processor: Parse integers for categorical data

    If the data defined in categories: is a number, the runtime should process it as if it were a string value. Currently, it will fail on deserialization because it does not match.

  • Automatically convert bool tag value into string tag

    Automatically convert bool tag value into string tag

    E.g.

    tags:
  selectors:
    - private
  values:
    - private

    {
     "private": true
}

    Creates the observation

    {
    "tags": "private"
}
  • Azure Blob connector

    Azure Blob connector

    Similar to the file connector, but fetch an Azure blob instead of a file.

    Should support both individual blobs and containers of blobs.

    Initial iteration supports SAS authenticated blobs or credentials passed by SPICE_ prefixed environment variables.

  • JsonProcessor: Support JSON Path as a field value selector

    JsonProcessor: Support JSON Path as a field value selector

    Currently the Json Processor only supports a flat structure, mapping from measurement/category to json object top level field. Supporting a JSON Path query in the selector would allow it to map into nested objects.

Related tags
Frameworks and Toolkits data-components-contrib
Provide open, community driven reusable components for building distributed applications

Components Contrib The purpose of Components Contrib is to provide open, community driven reusable components for building distributed applications. T

Nov 28, 2021
A simple blog framework built with GO. Uses HTML files and a JSON dict to give you more control over your content.

Go-Blog A simple template based blog framework. Instructions Built for GO version: 1 See the Documentation or Getting Started pages in the wiki. Notes

Sep 10, 2022
Goal is a toolkit for high productivity web development in Go language in the spirit of Revel Framework that is built around the concept of code generation.

Goal Goal is a set of tools for high productivity web development in Go language. Goal, being mostly inspired by Revel Framework and its discussions,

Sep 27, 2021
Flamingo Framework and Core Library. Flamingo is a go based framework for pluggable web projects. It is used to build scalable and maintainable (web)applications.
 Flamingo Framework and Core Library. Flamingo is a go based framework for pluggable web projects. It is used to build scalable and maintainable (web)applications.

Flamingo Framework Flamingo is a web framework based on Go. It is designed to build pluggable and maintainable web projects. It is production ready, f

Jan 5, 2023
A lightweight and fast http router from outer space

Alien Alien is a lightweight http router( multiplexer) for Go( Golang ), made for humans who don't like magic. Documentation docs Features fast ( see

Nov 13, 2022
:link: Generate HTML and CSS together, on the fly
 :link: Generate HTML and CSS together, on the fly

On The Fly Package for generating HTML and CSS together, on the fly. Can also be used for generating HTML, XML or CSS (or templates). HTML and CSS can

Oct 12, 2022
A small and evil REST framework for Go

go-rest A small and evil REST framework for Go Reflection, Go structs, and JSON marshalling FTW! go get github.com/ungerik/go-rest import "github.com/

Dec 6, 2022
Fast and Reliable Golang Web Framework
 Fast and Reliable Golang Web Framework

Gramework The Good Framework Gramework long-term testing stand metrics screenshot made with Gramework Stats Dashboard and metrics middleware What is i

Dec 18, 2022
Mango is a modular web-application framework for Go, inspired by Rack, and PEP333.

Mango Mango is a modular web-application framework for Go, inspired by Rack and PEP333. Note: Not actively maintained. Overview Mango is most of all a

Nov 17, 2022
Simple HTTP and REST client library for Go

Resty Simple HTTP and REST client library for Go (inspired by Ruby rest-client) Features section describes in detail about Resty capabilities Resty Co

Jan 9, 2023
This is only a mirror and Moved to https://gitea.com/lunny/tango

Tango 简体中文 Package tango is a micro & pluggable web framework for Go. Current version: v0.5.0 Version History Getting Started To install Tango: go get

Nov 18, 2022
An idiomatic Go REST API starter kit (boilerplate) following the SOLID principles and Clean Architecture

Go RESTful API Starter Kit (Boilerplate) This starter kit is designed to get you up and running with a project structure optimized for developing REST

Jan 3, 2023
Golang CTF framework and exploit development module

Golang CTF framework and exploit development module

Dec 18, 2022
Spice.ai is an open source, portable runtime for training and using deep learning on time series data.
 Spice.ai is an open source, portable runtime for training and using deep learning on time series data.

Spice.ai Spice.ai is an open source, portable runtime for training and using deep learning on time series data. ⚠️ DEVELOPER PREVIEW ONLY Spice.ai is

Dec 15, 2022
Community edition nuclei templates, a simple tool that allows you to organize all the Nuclei templates offered by the community in one place

cent Community edition nuclei templates, a simple tool that allows you to organize all the Nuclei templates offered by the community in one place Inst

Jan 9, 2023
This is a community upload of a community project.

This is a community upload of a community project. The amount of people who worked on it is innumerous, and hard to keep track of. But id like to contribute this to the efforts of Ando, Fists Team, the French Team, Mai's Team and the many wondeful members of the MHFZ community who gave their time and energy to help us. No matter the relations, these files will remain public and open source, free for all to use and modify.

Nov 6, 2022
Payment abstraction library - one interface for multiple payment processors ( inspired by Ruby's ActiveMerchant )

Sleet Payment abstraction library - interact with different Payment Service Providers (PsP) with one unified interface. Installation go get github.com

Dec 28, 2022
Finite-state machine with processors

FSM Finite-state machine with processors. Builder Register state processors type state1Processor struct { // clients } func NewState1Processor(..

Jan 7, 2022
Airbyte is the go-sdk/cdk to help build connectors quickly in go. This package abstracts away much of the "protocol" away from the user and lets them focus on biz logic. It focuses on developer efficiency and tries to be strongly typed as much as possible to help dev's move fast without mistakes

Airbyte - Golang SDK/CDK This package aims to help developers build connectors (sources/destinations) really fast in Go. The focus of this package is

Dec 18, 2022
Google Go Client and Connectors for Redis

Go-Redis Go Clients and Connectors for Redis. The initial release provides the interface and implementation supporting the (~) full set of current Red

Oct 25, 2022