71 Resources
Golang etl-pipeline Libraries
Substation is a cloud native toolkit for building modular ingest, transform, and load (ITL) data pipelines
Substation Substation is a cloud native data pipeline toolkit. What is Substation? Substation is a modular ingest, transform, load (ITL) application f
GoBatch is a batch processing framework in Go like Spring Batch in Java
GoBatch English|中文 GoBatch is a batch processing framework in Go like Spring Batch in Java. If you are familiar with Spring Batch, you will find GoBat
WIP - Pin - local pipeline project with Docker Golang API.
pin 🔥 WIP - Local pipeline project with Docker Golang API. 🌐 Installation Download latest release You can download latest release from here Install
An experimental tektoncd/pipeline resolver
vegetable resolvers A set of experimental tektoncd/pipeline resolvers. The idea of this is to experiment with what could be achieved with a Resolver.
💧 Visual Data Preparation (VDP) is an open-source tool to seamlessly integrate Vision AI with the modern data stack
Website | Community | Blog Get Early Access Visual Data Preparation (VDP) is an open-source tool to streamline the end-to-end visual data processing p
✨Super-fast and easy data transforming with any interfaces
BELT Super-fast and easy data transforming with any interfaces Just connect Belt to your Factory Methods Transform data Map Filter Assistance Pipe App
Git-based DevOps PaaS: Project, Pipeline, Kubernetes, ServiceMesh, MutilCloud
gitctl 一体化 DevOps 平台 从代码到应用的一体化编排，应用全生命周期管理，多云托管。 gitctl 会有哪些功能？ git 代码托管 projec
Conduit - Data Integration for Production Data Stores
Conduit Data Integration for Production Data Stores. 💫 Overview Conduit is a da
Ordered-concurrently a library for parallel processing with ordered output in Go
Ordered-concurrently a library for parallel processing with ordered output in Go. Process work concurrently / in parallel and returns output in a channel in the order of input. It is useful in concurrently / parallelly processing items in a queue, and get output in the order provided by the queue.
Parapipe - paralleling pipeline
Parapipe - paralleling pipeline The library provides a zero-dependency non-blocking buffered FIFO-pipeline for structuring the code and vertically sca
Sentiment Analysis Pipeline + API written in Golang (currently processing Twitter tweets).
Go Sentiment Analysis Components Config: config module based in JSON (enter twitter credentials for use) Controllers: handle the API db call/logic for
Sentiment Analysis Pipeline + API written in Golang (currently processing Twitter tweets).
Go Sentiment Analysis Components Config: config module based in JSON (enter twitter credentials for use) Controllers: handle the API db call/logic for
Package flow provides support for basic FBP / pipelines
Flow - FBP / pipelines / workers pool Package flow provides support for very basic FBP / pipelines. It helps to structure multistage processing as a s
A sample for okteto pipelines with terraform
Okteto Pipeline with Terraform (PubSub) This sample covers a producer/consumer a
repo de teste para executar á pipeline do rancher
pipeline-example-go This is a sample golang project to demonstrate the integration with rancher pipeline. Building go build -o ./bin/hello-server Runn
Build the deployment system around GitHub in minutes.
Gitploy Gitploy helps your team build the deployment system around GitHub in minutes. Documentation | Community | Installation Guide What is Gitploy?
Demo app showing an end-to-end CI pipeline with Github Actions, goreleaser and ko
Motivation 💪 I created this repository for learning and education purposes to help everyone to level up their (open source) projects with good docume
mongodb helper functions, document and pipeline builder
MongoUtils Mongodb helper functions, document and pipeline builder. Helpers ParseObjectID Parse object id from string. ParseObjectID(id string) *primi
Go spider: A crawler of vertical communities achieved by GOLANG
go_spider A crawler of vertical communities achieved by GOLANG. Latest stable Release: Version 1.2 (Sep 23, 2014). QQ群号：337344607 Features Concurrent
Just a playground with some interesting concepts like pipelines aka middleware, handleFuncs, request validations etc. Check it out.
Pipeline a.k.a middleware in Go Just a playground with some interesting concepts like pipelines aka middleware, handleFuncs, request validations etc.
Simple HTTP server written in golang
Simple HTTP server written in golang Simple webserver in golang, to demonstrate basic functionalities like e.g. sending back some request header info,
This POC is built with the goal to collect events/logs from the host systems such as Kubernetes, Docker, VMs, etc. A buffering layer is added to buffer events from the collector
What is does This POC is build with the goal to collect events/logs from the host systems such as Kubernetes, docker, VMs etc. A buffering layer is ad
A simple Go app and GitHub workflow that shows how to use GitHub Actions to test, build and deploy a Go app to Docker Hub
go-pipeline-demo A repository containing a simple Go app and GitHub workflow that shows how to use GitHub Actions to test, build and deploy a Go app t
🤖 DroneCI plugin to skip pipelines based on files changes
DroneCI Skip Pipeline 🤖 DroneCI plugin to skip pipelines based on files changes Motivations This DroneCI plugin enables you skip (or short-circuit) a
xyr is a very lightweight, simple and powerful data ETL platform that helps you to query available data sources using SQL.
xyr [WIP] xyr is a very lightweight, simple and powerful data ETL platform that helps you to query available data sources using SQL. Supported Drivers
The open source, end-to-end computer vision platform. Label, build, train, tune, deploy and automate in a unified platform that runs on any cloud and on-premises.
End-to-end computer vision platform Label, build, train, tune, deploy and automate in a unified platform that runs on any cloud and on-premises. onepa
Dataflow is a Kubernetes-native platform for executing large parallel data-processing pipelines.
Dataflow Summary Dataflow is a Kubernetes-native platform for executing large parallel data-processing pipelines. Each pipeline is specified as a Kube
Ethereum event tracker
Eth-event-tracker Tracker of Ethereum events. Example package main import ( "context" "encoding/binary" "flag" "fmt" "os" "os/signal" "syscall
Secret management toolchain
Harp TL;DR. Why harp? Use cases How does it work? Like a Data pipeline but for secret Immutable transformation What can I do? FAQ License Homebrew ins
A sample golang project to demonstrate the integration with rancher pipeline
pipeline-example-go This is a sample golang project to demonstrate the integration with rancher pipeline. Building go build -o ./bin/hello-server Runn
tfa is a 2fa cli tool that aims to help you to generate 2fa code on CI/CD pipelines.
tfa tfa is 2fa cli tool that aim to help you to generate 2fa code on CI/CD pipelines. You can provide secret with stdin or flag. Install brew install
Zeus - A Devops Automation Tool
With this tool we are trying generalize and minimize devops reperating task while trying to encourage shared responsibility model acorss devloper teams.
go-stash is a high performance, free and open source server-side data processing pipeline that ingests data from Kafka, processes it, and then sends it to ElasticSearch.
go-stash is a high performance, free and open source server-side data processing pipeline that ingests data from Kafka, processes it, and then sends it to ElasticSearch.
depstat is a dependency analyzer for Go modules enabled projects.
depstat is a dependency analyzer for Go modules enabled projects. It runs as part of the Kubernetes CI pipeline to help evaluate dependency updates to Kubernetes.
A simple CLI tool to use the _simulate API of elasticsearch to quickly test pipelines
elasticsearch-pipeline-tester A simple CLI tool to use the _simulate API of elasticsearch to quickly test pipelines usage: pipelinetester [flags] p
go-stash is a high performance, free and open source server-side data processing pipeline that ingests data from Kafka, processes it, and then sends it to ElasticSearch.
English | 简体中文 go-stash go-stash is a high performance, free and open source server-side data processing pipeline that ingests data from Kafka, proces
Code for the Go pipeline blog post
Concurrent and Parallel Pipelines in Go This repository contains the code that goes along with a blog post about concurrent and parallel pipelines in
原生mongodb查询表单式转golang bson
Pipeline to BSON A simple CLI app to convert MongoDB pipeline command in JSON format to Golang BSON structs. How it works? simply by running the app:
churro is a cloud-native Extract-Transform-Load (ETL) application designed to build, scale, and manage data pipeline applications.
Churro - ETL for Kubernetes churro is a cloud-native Extract-Transform-Load (ETL) application designed to build, scale, and manage data pipeline appli
Optimus is an easy-to-use, reliable, and performant workflow orchestrator for data transformation, data modeling, pipelines, and data quality management.
Optimus Optimus is an easy-to-use, reliable, and performant workflow orchestrator for data transformation, data modeling, pipelines, and data quality
Machine is a library for creating data workflows.
Machine is a library for creating data workflows. These workflows can be either very concise or quite complex, even allowing for cycles for flows that need retry or self healing mechanisms.
Pack a Go workflow/function as a Unix-style pipeline command
tpack Pack a Go workflow/function as a Unix-style pipeline command. Wiki In Unix-like computer operating systems, a pipeline is a mechanism for inter-
Devtron is an open source software delivery workflow for kubernetes written in go.
Devtron is an open source software delivery workflow for kubernetes written in go.
Baker is a high performance, composable and extendable data-processing pipeline for the big data era
Baker is a high performance, composable and extendable data-processing pipeline for the big data era. It shines at converting, processing, extracting or storing records (structured data), applying whatever transformation between input and output through easy-to-write filters.
A go library for easy configure and run command chains. Such like pipelining in unix shells.
go-command-chain A go library for easy configure and run command chains. Such like pipelining in unix shells. Example cat log_file.txt | grep error |
A library to help you create pipelines in Golang
pipeline Pipeline is a go library that helps you build pipelines without worrying about channel management and concurrency. It contains common fan-in
httpx is a fast and multi-purpose HTTP toolkit allows to run multiple probers using retryablehttp library, it is designed to maintain the result reliability with increased threads.
Features • Installation • Usage • Running httpx • Notes • Join Discord httpx is a fast and multi-purpose HTTP toolkit allow to run multiple probers us
a go daemon that syncs MongoDB to Elasticsearch in realtime. you know, for search.
monstache a go daemon that syncs mongodb to elasticsearch in realtime Version 6 This version of monstache is designed for MongoDB 3.6+ and Elasticsear
Robust, flexible and resource-efficient pipelines using Go and the commandline
Robust, flexible and resource-efficient pipelines using Go and the commandline Project links: Documentation & Main Website | Issue Tracker | Chat Why
Powerful workflow engine and end-to-end pipeline solutions implemented with native Kubernetes resources. https://cyclone.dev
Cyclone Cyclone is a powerful workflow engine and end-to-end pipeline solution implemented with native Kubernetes resources, with no extra dependencie
A serverless cluster computing system for the Go programming language
Bigslice Bigslice is a serverless cluster data processing system for Go. Bigslice exposes composable API that lets the user express data processing ta
A distributed, fault-tolerant pipeline for observability data
Table of Contents What Is Veneur? Use Case See Also Status Features Vendor And Backend Agnostic Modern Metrics Format (Or Others!) Global Aggregation
VectorSQL is a free analytics DBMS for IoT & Big Data, compatible with ClickHouse.
NOTICE: This project have moved to fuse-query VectorSQL is a free analytics DBMS for IoT & Big Data, compatible with ClickHouse. Features High Perform
[爬虫框架 (golang)] An awesome Go concurrent Crawler(spider) framework. The crawler is flexible and modular. It can be expanded to an Individualized crawler easily or you can use the default crawl components only.
go_spider A crawler of vertical communities achieved by GOLANG. Latest stable Release: Version 1.2 (Sep 23, 2014). QQ群号：337344607 Features Concurrent
A serverless cluster computing system for the Go programming language
Bigslice Bigslice is a serverless cluster data processing system for Go. Bigslice exposes composable API that lets the user express data processing ta
A lightweight stream processing library for Go
go-streams A lightweight stream processing library for Go. go-streams provides a simple and concise DSL to build data pipelines. Wiki In computing, a
A rapid http(s) benchmark tool written in Go
gonetx/httpit httpit is a rapid http(s) benchmark tool which on top of fasthttp. Also thanks to cobra and bubbletea. Installation Get binaries from re
A Go native tabular data extraction package. Currently supports .xls, .xlsx, .csv, .tsv formats.
grate A Go native tabular data extraction package. Currently supports .xls, .xlsx, .csv, .tsv formats. Why? Grate focuses on speed and stability first
Declarative streaming ETL for mundane tasks, written in Go
Benthos is a high performance and resilient stream processor, able to connect various sources and sinks in a range of brokering patterns and perform hydration, enrichments, transformations and filters on payloads.
Package strit introduces a new type of string iterator, along with a number of iterator constructors, wrappers and combinators.
strit Package strit (STRing ITerator) assists in development of string processing pipelines by providing a simple iteration model that allows for easy
csvplus extends the standard Go encoding/csv package with fluent interface, lazy stream operations, indices and joins.
csvplus Package csvplus extends the standard Go encoding/csv package with fluent interface, lazy stream processing operations, indices and joins. The
[爬虫框架 (golang)] An awesome Go concurrent Crawler(spider) framework. The crawler is flexible and modular. It can be expanded to an Individualized crawler easily or you can use the default crawl components only.
go_spider A crawler of vertical communities achieved by GOLANG. Latest stable Release: Version 1.2 (Sep 23, 2014). QQ群号：337344607 Features Concurrent
go.pipeline is a utility library that imitates unix pipeline. It simplifies chaining unix commands (and other stuff) in Go.
go.pipeline go.pipeline is a utility library that imitates unix pipeline. It simplifies chaining unix commands (and other stuff) in Go. Installation g
Build powerful pipelines in any programming language.
Gaia is an open source automation platform which makes it easy and fun to build powerful pipelines in any programming language. Based on HashiCorp's g
omniparser: a native Golang ETL streaming parser and transform library for CSV, JSON, XML, EDI, text, etc.
omniparser Omniparser is a native Golang ETL parser that ingests input data of various formats (CSV, txt, fixed length/width, XML, EDI/X12/EDIFACT, JS
An n:m message multiplexer written in Go
What is Gollum? Gollum is an n:m multiplexer that gathers messages from different sources and broadcasts them to a set of destinations. Gollum origina
Declarative streaming ETL for mundane tasks, written in Go
Benthos is a high performance and resilient stream processor, able to connect various sources and sinks in a range of brokering patterns and perform h
An n:m message multiplexer written in Go
What is Gollum? Gollum is an n:m multiplexer that gathers messages from different sources and broadcasts them to a set of destinations. Gollum origina
A library for performing data pipeline / ETL tasks in Go.
Ratchet A library for performing data pipeline / ETL tasks in Go. The Go programming language's simplicity, execution speed, and concurrency support m
Apache AVRO for go
avro The purpose of this package is to facilitate use of AVRO with go strong typing. Features github.com/khezen/avro Marshal/Unmarshal AVRO schema git
Pipelines using goroutines
pipeline This package provides a simplistic implementation of Go pipelines as outlined in Go Concurrency Patterns: Pipelines and cancellation. Docs Go