60 Resources
Golang bioinformatics-pipeline Libraries
Substation is a cloud native toolkit for building modular ingest, transform, and load (ITL) data pipelines
Substation Substation is a cloud native data pipeline toolkit. What is Substation? Substation is a modular ingest, transform, load (ITL) application f
WIP - Pin - local pipeline project with Docker Golang API.
pin 🔥 WIP - Local pipeline project with Docker Golang API. 🌐 Installation Download latest release You can download latest release from here Install
An experimental tektoncd/pipeline resolver
vegetable resolvers A set of experimental tektoncd/pipeline resolvers. The idea of this is to experiment with what could be achieved with a Resolver.
💧 Visual Data Preparation (VDP) is an open-source tool to seamlessly integrate Vision AI with the modern data stack
Website | Community | Blog Get Early Access Visual Data Preparation (VDP) is an open-source tool to streamline the end-to-end visual data processing p
Git-based DevOps PaaS: Project, Pipeline, Kubernetes, ServiceMesh, MutilCloud
gitctl 一体化 DevOps 平台 从代码到应用的一体化编排，应用全生命周期管理，多云托管。 gitctl 会有哪些功能？ git 代码托管 projec
Conduit - Data Integration for Production Data Stores
Conduit Data Integration for Production Data Stores. 💫 Overview Conduit is a da
Ordered-concurrently a library for parallel processing with ordered output in Go
Ordered-concurrently a library for parallel processing with ordered output in Go. Process work concurrently / in parallel and returns output in a channel in the order of input. It is useful in concurrently / parallelly processing items in a queue, and get output in the order provided by the queue.
Parapipe - paralleling pipeline
Parapipe - paralleling pipeline The library provides a zero-dependency non-blocking buffered FIFO-pipeline for structuring the code and vertically sca
Csvtk - A cross-platform, efficient and practical CSV/TSV toolkit in Golang
csvtk - a cross-platform, efficient and practical CSV/TSV toolkit Documents: http://bioinf.shenwei.me/csvtk ( Usage and Tutorial). 中文介绍 Source code: h
Sentiment Analysis Pipeline + API written in Golang (currently processing Twitter tweets).
Go Sentiment Analysis Components Config: config module based in JSON (enter twitter credentials for use) Controllers: handle the API db call/logic for
Sentiment Analysis Pipeline + API written in Golang (currently processing Twitter tweets).
Go Sentiment Analysis Components Config: config module based in JSON (enter twitter credentials for use) Controllers: handle the API db call/logic for
Package flow provides support for basic FBP / pipelines
Flow - FBP / pipelines / workers pool Package flow provides support for very basic FBP / pipelines. It helps to structure multistage processing as a s
A sample for okteto pipelines with terraform
Okteto Pipeline with Terraform (PubSub) This sample covers a producer/consumer a
repo de teste para executar á pipeline do rancher
pipeline-example-go This is a sample golang project to demonstrate the integration with rancher pipeline. Building go build -o ./bin/hello-server Runn
Build the deployment system around GitHub in minutes.
Gitploy Gitploy helps your team build the deployment system around GitHub in minutes. Documentation | Community | Installation Guide What is Gitploy?
Demo app showing an end-to-end CI pipeline with Github Actions, goreleaser and ko
Motivation 💪 I created this repository for learning and education purposes to help everyone to level up their (open source) projects with good docume
mongodb helper functions, document and pipeline builder
MongoUtils Mongodb helper functions, document and pipeline builder. Helpers ParseObjectID Parse object id from string. ParseObjectID(id string) *primi
Go spider: A crawler of vertical communities achieved by GOLANG
go_spider A crawler of vertical communities achieved by GOLANG. Latest stable Release: Version 1.2 (Sep 23, 2014). QQ群号：337344607 Features Concurrent
Just a playground with some interesting concepts like pipelines aka middleware, handleFuncs, request validations etc. Check it out.
Pipeline a.k.a middleware in Go Just a playground with some interesting concepts like pipelines aka middleware, handleFuncs, request validations etc.
A Go package for engineering organisms.
(Poly)merase Poly is a Go package for engineering organisms. Fast: Poly is fast and scalable. Modern: Poly tackles issues that other libraries and uti
Simple HTTP server written in golang
Simple HTTP server written in golang Simple webserver in golang, to demonstrate basic functionalities like e.g. sending back some request header info,
This POC is built with the goal to collect events/logs from the host systems such as Kubernetes, Docker, VMs, etc. A buffering layer is added to buffer events from the collector
What is does This POC is build with the goal to collect events/logs from the host systems such as Kubernetes, docker, VMs etc. A buffering layer is ad
A simple Go app and GitHub workflow that shows how to use GitHub Actions to test, build and deploy a Go app to Docker Hub
go-pipeline-demo A repository containing a simple Go app and GitHub workflow that shows how to use GitHub Actions to test, build and deploy a Go app t
🤖 DroneCI plugin to skip pipelines based on files changes
DroneCI Skip Pipeline 🤖 DroneCI plugin to skip pipelines based on files changes Motivations This DroneCI plugin enables you skip (or short-circuit) a
Dataflow is a Kubernetes-native platform for executing large parallel data-processing pipelines.
Dataflow Summary Dataflow is a Kubernetes-native platform for executing large parallel data-processing pipelines. Each pipeline is specified as a Kube
Secret management toolchain
Harp TL;DR. Why harp? Use cases How does it work? Like a Data pipeline but for secret Immutable transformation What can I do? FAQ License Homebrew ins
A sample golang project to demonstrate the integration with rancher pipeline
pipeline-example-go This is a sample golang project to demonstrate the integration with rancher pipeline. Building go build -o ./bin/hello-server Runn
tfa is a 2fa cli tool that aims to help you to generate 2fa code on CI/CD pipelines.
tfa tfa is 2fa cli tool that aim to help you to generate 2fa code on CI/CD pipelines. You can provide secret with stdin or flag. Install brew install
Zeus - A Devops Automation Tool
With this tool we are trying generalize and minimize devops reperating task while trying to encourage shared responsibility model acorss devloper teams.
go-stash is a high performance, free and open source server-side data processing pipeline that ingests data from Kafka, processes it, and then sends it to ElasticSearch.
go-stash is a high performance, free and open source server-side data processing pipeline that ingests data from Kafka, processes it, and then sends it to ElasticSearch.
depstat is a dependency analyzer for Go modules enabled projects.
depstat is a dependency analyzer for Go modules enabled projects. It runs as part of the Kubernetes CI pipeline to help evaluate dependency updates to Kubernetes.
A simple CLI tool to use the _simulate API of elasticsearch to quickly test pipelines
elasticsearch-pipeline-tester A simple CLI tool to use the _simulate API of elasticsearch to quickly test pipelines usage: pipelinetester [flags] p
go-stash is a high performance, free and open source server-side data processing pipeline that ingests data from Kafka, processes it, and then sends it to ElasticSearch.
English | 简体中文 go-stash go-stash is a high performance, free and open source server-side data processing pipeline that ingests data from Kafka, proces
Code for the Go pipeline blog post
Concurrent and Parallel Pipelines in Go This repository contains the code that goes along with a blog post about concurrent and parallel pipelines in
原生mongodb查询表单式转golang bson
Pipeline to BSON A simple CLI app to convert MongoDB pipeline command in JSON format to Golang BSON structs. How it works? simply by running the app:
churro is a cloud-native Extract-Transform-Load (ETL) application designed to build, scale, and manage data pipeline applications.
Churro - ETL for Kubernetes churro is a cloud-native Extract-Transform-Load (ETL) application designed to build, scale, and manage data pipeline appli
Machine is a library for creating data workflows.
Machine is a library for creating data workflows. These workflows can be either very concise or quite complex, even allowing for cycles for flows that need retry or self healing mechanisms.
Pack a Go workflow/function as a Unix-style pipeline command
tpack Pack a Go workflow/function as a Unix-style pipeline command. Wiki In Unix-like computer operating systems, a pipeline is a mechanism for inter-
Devtron is an open source software delivery workflow for kubernetes written in go.
Devtron is an open source software delivery workflow for kubernetes written in go.
Baker is a high performance, composable and extendable data-processing pipeline for the big data era
Baker is a high performance, composable and extendable data-processing pipeline for the big data era. It shines at converting, processing, extracting or storing records (structured data), applying whatever transformation between input and output through easy-to-write filters.
A go library for easy configure and run command chains. Such like pipelining in unix shells.
go-command-chain A go library for easy configure and run command chains. Such like pipelining in unix shells. Example cat log_file.txt | grep error |
A library to help you create pipelines in Golang
pipeline Pipeline is a go library that helps you build pipelines without worrying about channel management and concurrency. It contains common fan-in
httpx is a fast and multi-purpose HTTP toolkit allows to run multiple probers using retryablehttp library, it is designed to maintain the result reliability with increased threads.
Features • Installation • Usage • Running httpx • Notes • Join Discord httpx is a fast and multi-purpose HTTP toolkit allow to run multiple probers us
Robust, flexible and resource-efficient pipelines using Go and the commandline
Robust, flexible and resource-efficient pipelines using Go and the commandline Project links: Documentation & Main Website | Issue Tracker | Chat Why
Powerful workflow engine and end-to-end pipeline solutions implemented with native Kubernetes resources. https://cyclone.dev
Cyclone Cyclone is a powerful workflow engine and end-to-end pipeline solution implemented with native Kubernetes resources, with no extra dependencie
A distributed, fault-tolerant pipeline for observability data
Table of Contents What Is Veneur? Use Case See Also Status Features Vendor And Backend Agnostic Modern Metrics Format (Or Others!) Global Aggregation
VectorSQL is a free analytics DBMS for IoT & Big Data, compatible with ClickHouse.
NOTICE: This project have moved to fuse-query VectorSQL is a free analytics DBMS for IoT & Big Data, compatible with ClickHouse. Features High Perform
[爬虫框架 (golang)] An awesome Go concurrent Crawler(spider) framework. The crawler is flexible and modular. It can be expanded to an Individualized crawler easily or you can use the default crawl components only.
go_spider A crawler of vertical communities achieved by GOLANG. Latest stable Release: Version 1.2 (Sep 23, 2014). QQ群号：337344607 Features Concurrent
A lightweight stream processing library for Go
go-streams A lightweight stream processing library for Go. go-streams provides a simple and concise DSL to build data pipelines. Wiki In computing, a
A rapid http(s) benchmark tool written in Go
gonetx/httpit httpit is a rapid http(s) benchmark tool which on top of fasthttp. Also thanks to cobra and bubbletea. Installation Get binaries from re
Package strit introduces a new type of string iterator, along with a number of iterator constructors, wrappers and combinators.
strit Package strit (STRing ITerator) assists in development of string processing pipelines by providing a simple iteration model that allows for easy
csvplus extends the standard Go encoding/csv package with fluent interface, lazy stream operations, indices and joins.
csvplus Package csvplus extends the standard Go encoding/csv package with fluent interface, lazy stream processing operations, indices and joins. The
[爬虫框架 (golang)] An awesome Go concurrent Crawler(spider) framework. The crawler is flexible and modular. It can be expanded to an Individualized crawler easily or you can use the default crawl components only.
go_spider A crawler of vertical communities achieved by GOLANG. Latest stable Release: Version 1.2 (Sep 23, 2014). QQ群号：337344607 Features Concurrent
go.pipeline is a utility library that imitates unix pipeline. It simplifies chaining unix commands (and other stuff) in Go.
go.pipeline go.pipeline is a utility library that imitates unix pipeline. It simplifies chaining unix commands (and other stuff) in Go. Installation g
biogo is a bioinformatics library for Go
bíogo Installation $ go get github.com/biogo/biogo/... Overview bíogo is a bioinformatics library for the Go language. Getting help Help or simil
Build powerful pipelines in any programming language.
Gaia is an open source automation platform which makes it easy and fun to build powerful pipelines in any programming language. Based on HashiCorp's g
An n:m message multiplexer written in Go
What is Gollum? Gollum is an n:m multiplexer that gathers messages from different sources and broadcasts them to a set of destinations. Gollum origina
An n:m message multiplexer written in Go
What is Gollum? Gollum is an n:m multiplexer that gathers messages from different sources and broadcasts them to a set of destinations. Gollum origina
A library for performing data pipeline / ETL tasks in Go.
Ratchet A library for performing data pipeline / ETL tasks in Go. The Go programming language's simplicity, execution speed, and concurrency support m
Pipelines using goroutines
pipeline This package provides a simplistic implementation of Go pipelines as outlined in Go Concurrency Patterns: Pipelines and cancellation. Docs Go