Using golang to produce data to kinesis data stream

Using golang to produce data to kinesis data stream

What is this

The idea behind this repo was to quickly determine how easy it would be to add a serverless function to consume an API and add the data to a kinesis data stream.

To build

  1. Install golang: https://golang.org/doc/install
  2. Compile the program: GOOS=linux go build dataConsumer.go
  3. Zip up the program: zip function.zip dataConsumer

To Deploy

  1. Install terraform: https://learn.hashicorp.com/tutorials/terraform/install-cli
  2. Navigate to infra folder: cd infra
  3. Apply the infrastructure: terraform apply --auto-approve

Whats next?

Up next would be to either consume in a lambda function and perform some sort of ETL work prior to storing in to a database.
The other option is to send it to AWS GLUE to perform the ETL to continue learning the data ingestion then storing in to a database.

Owner
Mike Sherman
Full Stack polyglot, AWS Solutions Architect, and Terraform certified engineer
Mike Sherman
https://github.com/msherman/lambda-kinesis-golang-producer
Similar Resources

Send slack notifications using Github action

Slack notification This is a simple Slack notification action which runs using a Bot token. Example Action A simple example on how to use this action:

Aug 9, 2021

pubsub controller using kafka and base on sarama. Easy controll flow for actions streamming, event driven.

Psub helper for create system using kafka to streaming and events driven base. Install go get github.com/teng231/psub have 3 env variables for config

Sep 26, 2022

Insta API using Go and Mongodb

Insta API using Go and Mongodb

Insta-API HTTP JSON API that can be used to perform basic tasks of Creating User, Creating Post, Getting User & Post details through their Unique ID.

May 1, 2022

A quick introduction to how Apache Kafka works and differs from other messaging systems using an example application.

A quick introduction to how Apache Kafka works and differs from other messaging systems using an example application.

Apache Kafka in 6 minutes A quick introduction to how Apache Kafka works and differs from other messaging systems using an example application. In thi

Oct 27, 2021

Go library to build event driven applications easily using AMQP as a backend.

event Go library to build event driven applications easily using AMQP as a backend. Publisher package main import ( "github.com/creekorful/event" "

Dec 5, 2021

Producer x Consumer example using Kafka library and Go.

Go - Kafka - Example Apache Kafka Commands First of all, run the docker-compose docker-compose up, than run docker exec -it kafka_kafka_1 bash Topics

Dec 8, 2021

A basic pub-sub project using NATS

NATS Simple Pub-Sub Mechanism This is a basic pub-sub project using NATS. There is one publisher who publishes a "Hello World" message to a subject ca

Dec 13, 2021

Testing Apache Kafka using Go.

Apache Kafka Go Testing Apache Kafka using Go. Instructions Provision the single node Kafka cluster using Docker: docker-compose -p apache-kafka-go up

Dec 17, 2021

A simple visits tracker built using Go and Redis

visits-tracker A simple visits tracker built using Go and Redis Let me note that this is a really bad idea if you are not backing up Redis since it li

Jan 13, 2022
Related tags
Messaging lambda-kinesis-golang-producer
Build platforms that flexibly mix SQL, batch, and stream processing paradigms

Overview Gazette makes it easy to build platforms that flexibly mix SQL, batch, and millisecond-latency streaming processing paradigms. It enables tea

Dec 12, 2022
API for sending sms through the connected modem using golang

SMS sender API API for making SMS sending from modem, by sending post request to route Request for Send SMS url method 127.0.0.1:8000/api/v1/send-mode

Nov 5, 2021
Golang Restful API Messaging using GORM ORM (MySQL) Gorilla Mux

Golang Restful API Messaging using GORM ORM (MySQL) Gorilla Mux Getting Started Folder Structure This is my folder structure under my $GOPATH or $HOME

Dec 14, 2021
It's client library written in Golang for interacting with Linkedin Cruise Control using its HTTP API.

go-cruise-control It's client library (written in Golang) for interacting with Linkedin Cruise Control using its HTTP API. Supported Cruise Control ve

Jan 10, 2022
ZenQ - A low-latency thread-safe queue in golang implemented using a lock-free ringbuffer

ZenQ A low-latency thread-safe queue in golang implemented using a lock-free ringbuffer Features Much faster than native channels in both SPSC (single

Jan 8, 2023
⚡ HTTP/2 Apple Push Notification Service (APNs) push provider for Go — Send push notifications to iOS, tvOS, Safari and OSX apps, using the APNs HTTP/2 protocol.

APNS/2 APNS/2 is a go package designed for simple, flexible and fast Apple Push Notifications on iOS, OSX and Safari using the new HTTP/2 Push provide

Jan 1, 2023
Sending line notifications using a binary, docker or Drone CI.
 Sending line notifications using a binary, docker or Drone CI.

drone-line Sending line notifications using a binary, docker or Drone CI. Register Line BOT API Trial Please refer to LINE Business Center. Feature Se

Sep 27, 2022
:incoming_envelope: A fast Message/Event Hub using publish/subscribe pattern with support for topics like* rabbitMQ exchanges for Go applications

Hub ?? A fast enough Event Hub for go applications using publish/subscribe with support patterns on topics like rabbitMQ exchanges. Table of Contents

Dec 17, 2022
Discrete-event simulation in Go using goroutines

SimGo SimGo is a discrete event simulation framework for Go. It is similar to SimPy and aims to be easy to set up and use. Processes are defined as si

Sep 6, 2022
Scalable package delivery logistics simulator built using SingleStore and Vectorized Redpanda
 Scalable package delivery logistics simulator built using SingleStore and Vectorized Redpanda

Reference Architecture using SingleStore and Redpanda for global logistics ?? INFO: For the story behind this code (and epic dashboards), check out th

Oct 29, 2022