Airbyte is the go-sdk/cdk to help build connectors quickly in go. This package abstracts away much of the "protocol" away from the user and lets them focus on biz logic. It focuses on developer efficiency and tries to be strongly typed as much as possible to help dev's move fast without mistakes

Airbyte - Golang SDK/CDK

This package aims to help developers build connectors (sources/destinations) really fast in Go. The focus of this package is developer efficiency. It focusses on letting developers focus more on connector business logic instead of airbyte protocol knowledge.

Installation

go get github.com/bitstrapped/airbyte

Docs

Usage

By Example

  1. The fastest way to get started it to look at the full example in examples/httpsource or the Example in the godoc

Detailed Usage

  1. Define a source by implementing the Source interface.
// Source is the only interface you need to define to create your source!
type Source interface {
	// Spec returns the input "form" spec needed for your source
	Spec(logTracker LogTracker) (*ConnectorSpecification, error)
	// Check verifies the source - usually verify creds/connection etc.
	Check(srcCfgPath string, logTracker LogTracker) error
	// Discover returns the schema of the data you want to sync
	Discover(srcConfigPath string, logTracker LogTracker) (*Catalog, error)
	// Read will read the actual data from your source and use tracker.Record(), tracker.State() and tracker.Log() to sync data with airbyte/destinations
	// MessageTracker is thread-safe and so it is completely find to spin off goroutines to sync your data (just don't forget your waitgroups :))
	// returning an error from this will cancel the sync and returning a nil from this will successfully end the sync
	Read(sourceCfgPath string, prevStatePath string, configuredCat *ConfiguredCatalog,
		tracker MessageTracker) error
}
  1. Inside of main, pass your source into the sourcerunner
func main() {
	fsrc := filesource.NewFileSource("foobar.txt")
	runner := airbyte.NewSourceRunner(fsrc)
	err := runner.Start()
	if err != nil {
		log.Fatal(err)
	}
}
  1. Write a dockerfile (sample below)
FROM golang:1.17-buster as build

WORKDIR /base
ADD . /base/
RUN go build -o /base/app .


LABEL io.airbyte.version=0.0.1
LABEL io.airbyte.name=airbyte/source

ENTRYPOINT ["/base/app"]
  1. Push to your docker repository and profit!

Contributors

Similar Resources

Simple no frills AWS S3 Golang Library using REST with V4 Signing (without AWS Go SDK)

simples3 : Simple no frills AWS S3 Library using REST with V4 Signing Overview SimpleS3 is a golang library for uploading and deleting objects on S3 b

Nov 4, 2022

Package githubv4 is a client library for accessing GitHub GraphQL API v4 (https://developer.github.com/v4/).

githubv4 Package githubv4 is a client library for accessing GitHub GraphQL API v4 (https://docs.github.com/en/graphql). If you're looking for a client

Dec 26, 2022

A go sdk for baidu netdisk open platform 百度网盘开放平台 Go SDK

Pan Go Sdk 该代码库为百度网盘开放平台Go语言的SDK

Nov 22, 2022

Nextengine-sdk-go: the NextEngine SDK for the Go programming language

NextEngine SDK for Go nextengine-sdk-go is the NextEngine SDK for the Go programming language. Getting Started Install go get github.com/takaaki-s/nex

Dec 7, 2021

Commercetools-go-sdk is fork of original commercetools-go-sdk

commercetools-go-sdk The Commercetools Go SDK is automatically generated based on the official API specifications of Commercetools. It should therefor

Dec 13, 2021

Sdk-go - Go version of the Synapse SDK

synapsesdk-go Synapse Protocol's Go SDK. Currently in super duper alpha, do not

Jan 7, 2022

Redash-go-sdk - An SDK for the programmatic management of Redash, in Go

Redash-go-sdk - An SDK for the programmatic management of Redash, in Go

Redash Go SDK An SDK for the programmatic management of Redash. The main compone

Dec 13, 2022

Go package providing opinionated tools and methods for working with the `aws-sdk-go/service/cloudfront` package.

go-aws-cloudfront Go package providing opinionated tools and methods for working with the aws-sdk-go/service/cloudfront package. Documentation Tools $

Feb 2, 2022

Prismplus - Prism+ lets you multicast your rtmp stream to multiple destinations

Prismplus - Prism+ lets you multicast your rtmp stream to multiple destinations

prism+ Use at your own risk! It has worked for us.. but very much alpha quality!

Nov 9, 2022
Comments
  • Nested structs and schema inference

    Nested structs and schema inference

    Thanks for the Go CDK, love it!

    In this PR there is a small commit to handle nested structs, that require the nested properties in the prop spec.

    Additionally there is a piggy back using Daniel G. Taylors schema inference, that I have edited to work with airbytes implementation of json schema. For me this completely eliminates the need to define the schema twice. The Go struct handles it all.

    I've tested this with the BigQuery struct destination and get desired results.

    Hope you like it, I'd be happy to demo it to the community and hopefully drive some more traffic to the Go CDK!

  • Inject io.Writer to the SourceRunner instead of using an internal io.WriteCloser

    Inject io.Writer to the SourceRunner instead of using an internal io.WriteCloser

    Leave the close and flush details to the user. This way it easily use specific writers and be used in different contextes without changing any internals.

Aws-cdk-go-examples - Example projects using the AWS CDK by Golang

aws-cdk-go-examples Example projects using the AWS CDK by Golang Useful commands

Nov 24, 2022
A compute service that lets you run code without provisioning or managing servers

AWS Lambda for Go Lambda is a compute service that lets you run code without provisioning or managing servers. Lambda runs your code on a high-availab

Oct 13, 2021
The Fabric Token SDK is a set of API and services that lets developers create token-based distributed application on Hyperledger Fabric.

The Fabric Token SDK is a set of API and services that let developers create token-based distributed application on Hyperledger Fabric.

Dec 14, 2022
Empty an S3 bucket as fast as possible 💨

s3-destroyer Iteratively calls ListObjects, add objects keys to a buffer and calls DeleteObject in goroutines. Usage -access-key string s3 a

Dec 13, 2021
Go SDK for working with the Nightfall Developer Platform

Nightfall Go SDK nightfall-go-sdk is a Go client library for accessing the Nightfall API. It allows you to add functionality to your applications to s

Jun 20, 2022
Create an ESK cluster using CDK and Golang.

Welcome to your CDK Go project! This is a blank project for Go development with CDK. NOTICE: Go support is still in Developer Preview. This implies th

Feb 14, 2022
Http apis with AWS CDK for Go

Http apis with AWS CDK for Go Blog: https://harshq.medium.com/building-apps-with-aws-sdk-for-golang-api-gateway-and-lambda-b254858b1d71 Useful command

Jan 4, 2022
Terraform CDK aws Provider

Terraform CDK aws Provider

Jul 29, 2022
Sample CDK projects developed by AWS Professional Services (ProServe)

AWS CDK Examples - ProServe This repository contains a set of CDK example projects which have been developed by AWS Professional Services (ProServe).

Dec 27, 2022
Sample application managed with AWS CDK v2 (TypeScript).

honyakutter-ts This is an application that translates text entered in Japanese into English and tweets it, defined using AWS CDK v2 (TypeScript) and u

Dec 7, 2021