WIP - Pin - local pipeline project with Docker Golang API.



pin 🔥 GitHub Actions status | sdras/awesome-actions

WIP - Local pipeline project with Docker Golang API.

pingif

🌐 Installation

Download latest release

You can download latest release from here

Install with cloning

Clone the pin

git clone https://github.com/muhammedikinci/pin

Download packages

go mod download

Build executable

go build -o pin ./cmd/cli/.

Or you can run directly

go run ./cmd/cli/. apply -n "test" -f ./testdata/test.yaml

⚙️ Configuration

Sample yaml file

workflow:
  - run

logsWithTime: true

run:
  image: golang:alpine3.15
  copyFiles: true
  soloExecution: true
  script:
    - go mod download
    - go run .
    - ls
  port:
    - 8082:8080

You can create separate jobs like the run stage and if you want to run these jobs in the pipeline you must add its name to workflow.

copyFiles

default: false

If you want to copy all projects filed to the docker container, you must set this configuration to true

soloExecution

default: false

When you add multiple commands to the script field, commands are running in the container as a shell script. If soloExecution is set to true each command works in a different shell script.

soloExecution => false

# shell#1
cd cmd
ls

soloExecution => true

# shell#1
cd cmd
# shell#2
ls

If you want to see all files in the cmd folder you must set soloExecution to false or you can use this:

# shell#1
cd cmd && ls

logsWithTime

default: false

logsWithTime => true

⚉ 2022/05/08 11:36:30 Image is available
⚉ 2022/05/08 11:36:30 Start creating container
⚉ 2022/05/08 11:36:33 Starting the container
⚉ 2022/05/08 11:36:35 Execute command: ls -a

logsWithTime => false

⚉ Image is available
⚉ Start creating container
⚉ Starting the container
⚉ Execute command: ls -a

port

default: empty mapping

You can use this feature for port forwarding from container to your machine with multiple mapping

  port:
    - 8082:8080
    - 8083:8080

copyIgnore

default: empty mapping

You can use this feature to ignore copying the specific files in your project to the container.

Sample configuration yaml

run:
  image: node:current-alpine3.15
  copyFiles: true
  soloExecution: true
  port:
    - 8080:8080
  copyIgnore:
    - server.js
    - props
    - README.md
    - helper/.*/.py

Actual folder structure in project

index.js
server.js
README.md
helper:
    - test.py
    - mock
        test2.py
    - api:
        index.js
    - props:
        index.js

Folder structure in container

index.js
helper:
    - mock (empty)
    - api:
        index.js

parallel

default: false

If you want to run parallel job, you must add parallel field and the stage must be in workflow(position doesn't matter)

workflow:
  - testStage
  - parallelJob
  - run

...

parallelJob:
  image: node:current-alpine3.15
  copyFiles: true
  soloExecution: true
  parallel: true
  script:
    - ls -a

Tests

go test ./...

Checklist

  • Implement web interface
  • Support concurrent jobs Issue#9
  • Add working with remote docker deamon support
  • Change image pulling logs (get only status logs)Issue#1
  • Add custom ignore configuration to copyFiles for project files (like gitignore) Issue#7
  • Add shared artifacts support between different jobs
  • Add timestamp to container names Issue#2
  • Create small pieces with extracting codes from runner struct and write unit test:
  • Add port expose support Issue#6
  • Support long living containers
  • Add concurrency between jobs and graceful shutdown with global context Issue#8

Contact

Muhammed İkinci - [email protected]

Owner
Muhammed İKİNCİ
Software Developer
Muhammed İKİNCİ
Comments
  • Support concurrent jobs

    Support concurrent jobs

    I will add the parallel field to the stage for concurrent support.

    parallelJob:
      image: node:current-alpine3.15
      copyFiles: true
      soloExecution: true
      parallel: true
      script:
        - ls -a
    
  • Add custom ignore configuration to copyFiles for project files (like gitignore)

    Add custom ignore configuration to copyFiles for project files (like gitignore)

    I want to add file and folder ignore support with Golang regexp like the below example;

    sample configuration yaml

    run:
      image: node:current-alpine3.15
      copyFiles: true
      soloExecution: true
      port:
        - 8080:8080
      copyIgnore:
        - server.js
        - props
        - README.md
        - helper/.*/.py
    

    actual folder structure in project

    index.js
    server.js
    README.md
    helper:
        - test.py
        - mock
            test2.py
        - api:
            index.js
        - props:
            index.js
    

    expected folder structure in container

    index.js
    helper:
        - mock (empty)
        - api:
            index.js
    
  • Add port expose support

    Add port expose support

    I want to add port expose support like below;

    workflow:
      - run
    
    run:
      image: node:current-alpine3.15
      copyFiles: true
      soloExecution: true
      port:
        - 8082:8080
        - 8083:8080
    
  • Create image_manager, interfaces package. Write unit tests for image_manager with mockgen

    Create image_manager, interfaces package. Write unit tests for image_manager with mockgen

    I want to make codes loosely coupled and creation by micro pieces. Therefore, I will create interfaces and add some unit tests with mockgen. In this issue, I am starting with image_manager.

  • Add timestamp to container names

    Add timestamp to container names

    In pin, container names are created from stage names. This can block the creation new container while docker has another container with the same name as the stage.

    Error response from daemon: Conflict. The container name "/run" is already in use by container "{container_id}". You have to remove (or rename) that container to be able to reuse that name.
    
  • Change image pulling logs (get only status logs)

    Change image pulling logs (get only status logs)

    If the image that you used in a job doesn't exist in your system, the pin will pull the image and print the result like below. Print is not good at all. I will change the log to just getting the status field in the JSON.

    {"status":"Pulling from library/node","id":"current-alpine3.15"}
    {"status":"Already exists","progressDetail":{},"id":"df9b9388f04a"}
    {"status":"Pulling fs layer","progressDetail":{},"id":"75d830d4694c"}
    {"status":"Pulling fs layer","progressDetail":{},"id":"8f5e371a64cc"}
    {"status":"Pulling fs layer","progressDetail":{},"id":"0c741fedcc2f"}
    {"status":"Downloading","progressDetail":{"current":450,"total":450},"progress":"[==================================================\u003e]     450B/450B","id":"0c741fedcc2f"}
    {"status":"Verifying Checksum","progressDetail":{},"id":"0c741fedcc2f"}
    {"status":"Download complete","progressDetail":{},"id":"0c741fedcc2f"}
    {"status":"Downloading","progressDetail":{"current":23928,"total":2345696},"progress":"[\u003e                                                  ]  23.93kB/2.346MB","id":"8f5e371a64cc"}
    {"status":"Downloading","progressDetail":{"current":474350,"total":46627276},"progress":"[\u003e                                                  ]  474.4kB/46.63MB","id":"75d830d4694c"}
    {"status":"Downloading","progressDetail":{"current":261361,"total":2345696},"progress":"[=====\u003e                                             ]  261.4kB/2.346MB","id":"8f5e371a64cc"}
    {"status":"Downloading","progressDetail":{"current":818417,"total":2345696},"progress":"[=================\u003e                                 ]  818.4kB/2.346MB","id":"8f5e371a64cc"}
    {"status":"Downloading","progressDetail":{"current":952702,"total":46627276},"progress":"[=\u003e                                                 ]  952.7kB/46.63MB","id":"75d830d4694c"}
    
WIP: project fabrication tool

fab fabricate a new project from a template... in a fabulous way :-) setup The first time on a machine, fab can be initialized via # on unix os's this

Jan 25, 2022
Local Storage is one of HwameiStor components. It will provision the local LVM volume.
Local Storage is one of HwameiStor components. It will provision the local LVM volume.

Local Storage Module English | Simplified_Chinese Introduction Local Storage is one of modules of HwameiStor which is a cloud native local storage sys

Aug 6, 2022
repo de teste para executar á pipeline do rancher

pipeline-example-go This is a sample golang project to demonstrate the integration with rancher pipeline. Building go build -o ./bin/hello-server Runn

Dec 19, 2021
Kubernetes IN Docker - local clusters for testing Kubernetes
Kubernetes IN Docker - local clusters for testing Kubernetes

kind is a tool for running local Kubernetes clusters using Docker container "nodes".

Jan 5, 2023
Kubernetes IN Docker - local clusters for testing Kubernetes
Kubernetes IN Docker - local clusters for testing Kubernetes

Please see Our Documentation for more in-depth installation etc. kind is a tool for running local Kubernetes clusters using Docker container "nodes".

Feb 14, 2022
(WIP) Extremely simple unixway GitHub webhook listener for push event

(WIP) puffy Puffy is an extremely simple unixway GitHub webhook listener and handler for push events Todo Add payload signature validation (WIP) Depen

Oct 15, 2022
[WIP] Cheap, portable and secure NAS based on the Raspberry Pi Zero - with encryption, backups, and more

PortaDisk - Affordable Raspberry Pi Portable & Secure NAS Project Project Status: Early work in progress. web-unlock is still not ready for production

Nov 23, 2022
WIP Go implementation of CSIRAC as a virtual machine.

CSIRAC A work-in-progress Go implementation of CSIRAC, as described in The Last of the First - CSIRAC: Australia's First Computer. The intention is to

Jan 4, 2022
[WIP] Basic Echo CRUD template (no pagination)

echo-crud-template [WIP] Basic Echo CRUD template (no pagination) Overview Based on https://github.com/xesina/golang-echo-realworld-example-app. Echo

Jan 11, 2022
Explore Docker registries and manipulate Docker images!
Explore Docker registries and manipulate Docker images!

L/S tags Utility and API to manipulate (analyze, synchronize and aggregate) images across different Docker registries. Example invocation $ lstags alp

Nov 25, 2022
Docker-based remote code runner / 基于 Docker 的远程代码运行器
Docker-based remote code runner / 基于 Docker 的远程代码运行器

Docker-based remote code runner / 基于 Docker 的远程代码运行器

Nov 9, 2022
ecsk is a CLI tool to interactively use frequently used functions of docker command in Amazon ECS. (docker run, exec, cp, logs, stop)
ecsk is a CLI tool to interactively use frequently used functions of docker command in Amazon ECS. (docker run, exec, cp, logs, stop)

English / 日本語 ecsk ECS + Task = ecsk ?? ecsk is a CLI tool to interactively use frequently used functions of docker command in Amazon ECS. (docker run

Dec 13, 2022
Hassle-free minimal CI/CD for git repositories with docker or docker-compose projects.
Hassle-free minimal CI/CD for git repositories with docker or docker-compose projects.

GIT-PIPE Hassle-free minimal CI/CD for git repos for docker-based projects. Features: zero configuration for repos by default automatic encrypted back

Sep 23, 2022
Tool to convert docker-compose files to set of simple docker commands

docker-decompose Tool to convert docker-compose files to set of simple docker commands. Install Use go get to install the latest version of the librar

Apr 12, 2022
Go-http-server-docker - Simple sample server using docker and go

go-http-server-docker Simple sample webserver using docker and go.

Jan 8, 2022
Docker-hub-rate-limit - Show pulling rate status of Docker-hub

Docker-Hub Pull Rate Status This tool shows current status of docker hub pull ra

Jan 28, 2022
Dotnet-appsettings-env - Convert .NET appsettings.json file to Kubernetes, Docker and Docker-Compose environment variables

dotnet-appsettings-env Convert .NET appsettings.json file to Kubernetes, Docker

Dec 30, 2022
Dotnet-appsettings-env - Convert .NET appsettings.json file to Kubernetes, Docker and Docker-Compose environment variables

dotnet-appsettings-env Convert .NET appsettings.json file to Kubernetes, Docker

Feb 16, 2022
A best practices Go source project with unit-test and integration test, also use skaffold & helm to automate CI & CD at local to optimize development cycle

Dependencies Docker Go 1.17 MySQL 8.0.25 Bootstrap Run chmod +x start.sh if start.sh script does not have privileged to run Run ./start.sh --bootstrap

Apr 4, 2022