Go iter tools (for iterating , mapping, filtering, reducing streams -represented as channels-)

go_iter

Go iter tools (for iterating , mapping, filtering, reducing streams -represented as channels-)

  • to install : go get -u github.com/serge-hulne/go_iter
  • Documentation : See in the Documentation directory.

Defines:

  • Filter
  • Map
  • Reduce
  • Range
  • Take.

Can easily be extended/generalized to all collection types.

See examples for more information:

Partial example (code snippet) :

In the example, data coming from an input channel are mapped/filtered, using Map(), to an output channel. Map.is one of the functions provided by go_iter.

%v\n", person.(Person)) } ">
	//... User defined type
	type Person struct {
		Name string
		Age  int
	}
	
	//... Send list of "Persons in a channel"
	input_channel := input()

	for item := range input_channel {
		fmt.Printf("%v, %v\n", item.(Person).Name, item.(Person).Age)
	}

	fmt.Println("- - -")
	
	//... Refresh channel
	input_channel = input()

	cb := func(c1, c2 Chan) Chan {
		for person := range c1 {
			//... Some fitering action here:
			c2 <- filtered_item
		}
		return c2
	}

	electors := Map(input_channel, cb)

	for person := range electors {
		fmt.Printf("Elector --> %v\n", person.(Person))
	}
Similar Resources

Package iter provides generic, lazy iterators, functions for producing them from primitive types, as well as functions and methods for transforming and consuming them.

iter Package iter provides generic, lazy iterators, functions for producing them from primitive types, as well as functions and methods for transformi

Dec 16, 2022

Snowflake grafana datasource plugin allows Snowflake data to be visually represented in Grafana dashboards.

Snowflake grafana datasource plugin allows Snowflake data to be visually represented in Grafana dashboards.

Snowflake Grafana Data Source With the Snowflake plugin, you can visualize your Snowflake data in Grafana and build awesome chart. Get started with th

Dec 29, 2022

Find out which organisations are best represented in a repository

Find out which organisations are best represented in a repository

Prop-rep A golang command line tool to show you which organisations are contribu

Sep 30, 2022

tasq is a simple HTTP-based task queue. Each task is represented as a string

tasq tasq is a simple HTTP-based task queue. Each task is represented as a string (it could be anything). Tasks are pushed to the queue via an HTTP en

Nov 3, 2022

tools for working with streams of data

streamtools 4/1/2015 Development for streamtools has waned as our attention has turned towards developing a language paradigm that embraces blocking,

Nov 18, 2022

Go package for mapping values to and from space-filling curves, such as Hilbert and Peano curves.

Go package for mapping values to and from space-filling curves, such as Hilbert and Peano curves.

Hilbert Go package for mapping values to and from space-filling curves, such as Hilbert and Peano curves. Documentation available here This is not an

Dec 23, 2022

Type safe SQL builder with code generation and automatic query result data mapping

Type safe SQL builder with code generation and automatic query result data mapping

Jet Jet is a complete solution for efficient and high performance database access, consisting of type-safe SQL builder with code generation and automa

Jan 6, 2023

Golang binary decoder for mapping data into the structure

binstruct Golang binary decoder to structure Install go get -u github.com/ghostiam/binstruct Examples ZIP decoder PNG decoder Use For struct From file

Dec 17, 2022

csvutil provides fast and idiomatic mapping between CSV and Go (golang) values.

csvutil provides fast and idiomatic mapping between CSV and Go (golang) values.

csvutil Package csvutil provides fast and idiomatic mapping between CSV and Go (golang) values. This package does not provide a CSV parser itself, it

Jan 6, 2023

sigurlx a web application attack surface mapping tool.

sigurlx a web application attack surface mapping tool, it does ...:

Jul 24, 2021

7 days golang programs from scratch (web framework Gee, distributed cache GeeCache, object relational mapping ORM framework GeeORM, rpc framework GeeRPC etc) 7天用Go动手写/从零实现系列

7 days golang programs from scratch README 中文版本 7天用Go从零实现系列 7天能写什么呢?类似 gin 的 web 框架?类似 groupcache 的分布式缓存?或者一个简单的 Python 解释器?希望这个仓库能给你答案

Jan 5, 2023

A web application attack surface mapping tool. It takes in a list of urls then performs numerous probes

sigurlscann3r A web application attack surface mapping tool. It takes in a list of urls then performs numerous probes Resources Features Installation

Sep 24, 2022

Pointer was developed for massive hunting and mapping Cobalt Strike servers exposed on the internet.

Pointer was developed for massive hunting and mapping Cobalt Strike servers exposed on the internet.

Description The Pointer was developed for hunting and mapping Cobalt Strike servers exposed to the Internet. The tool includes the complete methodolog

Nov 23, 2022

Go-odm, a Golang Object Document Mapping for MongoDB.

Go-odm, a Golang Object Document Mapping for MongoDB.

A project of SENROK Open Source Go ODM Go-odm, a Golang Object Document Mapping for MongoDB. Table of contents Features Installation Get started Docum

Nov 4, 2022

Program to generate ruins using the Numenera Ruin Mapping Engine

Ruin Generator This is my attempt to build a program to generate ruins for Numenera using the rules from the Jade Colossus splatbook. The output only

Nov 7, 2021

A system written in Golang to help ops team to automate the process of mapping Vault groups to LDAP Groups.

A system written in Golang to help ops team to automate the process of mapping Vault groups to LDAP Groups. This utility automatically adds LDAP Groups' members to the corresponding Vault Groups.

Nov 12, 2021

Schemable - Schemable provides basic struct mapping against a database, using the squirrel package

Schemable Schemable provides basic struct mapping against a database, using the

Oct 17, 2022

Esdump is a migration CLI written in Go for migrating index mapping and data from one elasticsearch to another.

esdump Introduction esdump is a migration CLI written in Go for migrating index mapping and data from one elasticsearch to another. Compatibility Elas

Jul 23, 2022

A fast, easy-of-use and dependency free custom mapping from .csv data into Golang structs

csvparser This package provides a fast and easy-of-use custom mapping from .csv data into Golang structs. Index Pre-requisites Installation Examples C

Nov 14, 2022
Comments
  • Integrate with `samber/lo`

    Integrate with `samber/lo`

    I've stumbled upon your repo from this post. It's great to have this lazy iterators for efficient chaining of operations in go!

    Since https://github.com/samber/lo is the biggest functional programming library for go, have you considered integrating this into it?

  • If the parent function returns early, you end up with blocked orphaned go-routines

    If the parent function returns early, you end up with blocked orphaned go-routines

    If the parent function returns early, you end up with blocked orphaned go-routines

    In each case, you use a go-routine to populate the channel you are ranging over, and as you said, that blocks until the next element is read using the range function. Isn't this problematic if you do error handling in the for loop? Even though it won't block the main thread, wouldn't the go-routine block and stay around forever as long as the main thread runs? If this is a server, then as long as the server is alive? If it serves many requests, you might just keep adding more and more go-routines!

    Example:

    for item := range Range(10) {
    	if item == 2 {
    		return
    	}
    }
    

    The previous example would return without reading the rest of the ints in the channel and thus leave the orphaned go-routine around.

    Full example file that demonstrates the orphaned go-routines getting created:

    package main
    
    import (
    	"fmt"
    	"runtime"
    
    	"github.com/lucemhealth/go-common/lucemtelemetry"
    )
    
    func main() {
    	lucemtelemetry.ConfigureZerologLogger()
    
    	for i := 0; i < 10; i++ {
    		fmt.Printf("request number: %d\n", i)
    		fmt.Printf("starting number of routines: %d\n", runtime.NumGoroutine())
    		doForLoop()
    		fmt.Printf("ending number of routines: %d\n", runtime.NumGoroutine())
    		fmt.Println()
    	}
    }
    
    func doForLoop() {
    	for item := range Range(10) {
    		if item == 2 {
    			return
    		}
    	}
    }
    
    func Range(nmax int) chan int {
    	out := make(chan int)
    	go func() {
    		defer close(out)
    		for index := 0; index < nmax; index++ {
    			out <- index
    		}
    	}()
    	return out
    }
    
  • Hard to compose iterators together

    Hard to compose iterators together

    I have some performance concerns regarding the use of channels as the only way to propagate, but there's not a nice way for you to address that without, like, rewriting the whole thing. So I'll complain about something that's straightforward enough:

    currently, your worker interface takes two channels and returns nil. Instead, a worker should be expected to take a context and a receive-only channel, returning a new context and a new receive-only channel. This change would ensure that the responsibility for each channel is clear, error propagation on a pipeline is controlled at each step by the propagator, and the implementor is far less likely to leave a thread hanging in the background, still running.

    Unfortunately, I'd still have the doesn't-know-how-to-share-memory issue, so I'd still be using native loops and/or chains of reader/writer interfaces for same-thread iteration.

Package iter provides generic, lazy iterators, functions for producing them from primitive types, as well as functions and methods for transforming and consuming them.

iter Package iter provides generic, lazy iterators, functions for producing them from primitive types, as well as functions and methods for transformi

Dec 16, 2022
Go package for mapping values to and from space-filling curves, such as Hilbert and Peano curves.
Go package for mapping values to and from space-filling curves, such as Hilbert and Peano curves.

Hilbert Go package for mapping values to and from space-filling curves, such as Hilbert and Peano curves. Documentation available here This is not an

Dec 23, 2022
Probabilistic data structures for processing continuous, unbounded streams.

Boom Filters Boom Filters are probabilistic data structures for processing continuous, unbounded streams. This includes Stable Bloom Filters, Scalable

Dec 30, 2022
Juniper is an extension to the Go standard library using generics, including containers, iterators, and streams.

Juniper Juniper is a library of extensions to the Go standard library using generics, including containers, iterators, and streams. container/tree con

Dec 25, 2022
PartyGateRDF - PartyGate tools for linked-data scientists
PartyGateRDF - PartyGate tools for linked-data scientists

PartyGateRDF - PartyGate tools for linked-data scientists Introduction This is a data set containing references to information published by the media

Dec 31, 2022
siusiu (suite-suite harmonics) a suite used to manage the suite, designed to free penetration testing engineers from learning and using various security tools, reducing the time and effort spent by penetration testing engineers on installing tools, remembering how to use tools.
siusiu (suite-suite harmonics) a suite used to manage the suite, designed to free penetration testing engineers from learning and using various security tools, reducing the time and effort spent by penetration testing engineers on installing tools, remembering how to use tools.

siusiu (suite-suite harmonics) a suite used to manage the suite, designed to free penetration testing engineers from learning and using various security tools, reducing the time and effort spent by penetration testing engineers on installing tools, remembering how to use tools.

Dec 12, 2022
Iter tools for Go(Golang)

Iter tools for Go(Golang) The stable version has been moved to : https://github.com/serge-hulne/go_iter Defines: Filter Map Reduce Range Take on colle

Sep 26, 2022
JSONL graph tools - Graph is represented as JSONL of nodes and edges.

JSONL graph tools - Graph is represented as JSONL of nodes and edges.

Sep 27, 2022
M3U generator for Stirr, optimized for Channels' custom channels.
M3U generator for Stirr, optimized for Channels' custom channels.

Stirr for Channels This simple Docker image will generate an M3U playlist and EPG optimized for use in Channels and expose them over HTTP. Channels su

Oct 7, 2022
Reducing Malloc/Free traffic to cgo

CGOAlloc Reducing Malloc/Free traffic to cgo Why? Cgo overhead is a little higher than many are comfortable with (at the time of this writing, a simpl

Dec 24, 2022