darkroom - An image proxy with changeable storage backends and image processing engines with focus on speed and resiliency.

Darkroom - Yet Another Image Proxy

build status Coverage Status Docs latest GoDoc Go Report Card GolangCI GitHub Release Mentioned in Awesome Go

Introduction

Darkroom combines the storage backend and the image processor and acts as an Image Proxy on your image source.
You may implement your own Storage and Processor interfaces to gain custom functionality while still keeping other Darkroom Server functionality.
The native implementations focus on speed and resiliency.

Features

Darkroom supports several image operations which are documented here.

Installation

go get -u github.com/gojek/darkroom

Other ways to run can be found here.

Metrics Support

Darkroom supports Prometheus and StatsD for tracking and monitoring metrics. You need to specify the metrics system by adding an environment variable, METRICS_SYSTEM=prometheus/statsd

Prometheus

The application exposes the metrics at "http://<application_url>/metrics" endpoint. Since it's a pull based system, Prometheus server that is set up from docker-compose scrapes metrics from the application endpoint and its configuration can be changed in prometheus.yml.

StatsD

In order to use StatsD as your metrics system, you also need to add the following env variables,

METRICS_STATSD_STATSDADDR=hostname:port
METRICS_STATSD_PREFIX=client-prefix
METRICS_STATSD_SAMPLERATE=sample-rate
METRICS_STATSD_FLUSHBYTES=flushbytes

These are used to set up the StatsD client.

Grafana

Darkroom currently supports grafana provisioning for Prometheus based metrics.

Grafana is preconfigured with dashboards and Prometheus as the default data source:

Visualization of Darkroom metrics(prometheus) on Grafana:

Contributing Guide

Read our contributing guide to learn about our development process, how to propose bugfixes and improvements, and how to build and test your changes to Darkroom.

License

Darkroom is MIT licensed.

Owner
Comments
  • Add Documentation to the project

    Add Documentation to the project

    What would you like to be added

    I suggest adding separate documentation than what we have in README right now. It shall be generated from the source in /docs folder and hosted on GitHub pages on every CI release job.

    Why is this needed

    The code has good GoDoc coverage but there should be better documentation for the project showing how to use it. Users might not be familiar with all the features that Darkroom provides.

    Comments

    If you've other suggestions on how the docs should be built other than GH Pages, feel free to add them. Eg: ReadTheDocs, etc.

  • Cleanup linter error

    Cleanup linter error

    It will be good to clean up these linter errors before it is causing problems. Also by following some simplification and code practice, it will make the code easier to maintain

    What should be cleaned up or changed

    from make lint

    pkg/config/config.go:86:15: exported func Source returns unexported type *config.source, which can be annoying to use
    
    pkg/processor/native/utils_test.go:162:41: exported func NewMockImage returns unexported type *native.mockImage, which can be annoying to use
    
    pkg/service/manipulator.go:166:52: exported func NewManipulator returns unexported type *service.manipulator, which can be annoying to use
    
    pkg/processor/native/encoder.go:79:1: comment on exported method Encoders.Options should be of the form "Options ..."
    
    

    from golangci-lint

    pkg/processor/native/processor_test.go:21:2: `badImage` is unused (structcheck)
    	badImage      image.Image
    	^
    pkg/processor/native/processor_test.go:181:11: ineffectual assignment to `err` (ineffassign)
    		img, _, err := s.processor.Decode(file)
    		        ^
    internal/handler/ping.go:12:3: S1023: redundant `return` statement (gosimple)
    		return
    		^
    pkg/server/api.go:63:2: S1005: '_ = <-ch' can be simplified to '<-ch' (gosimple)
    	_ = <-sig
    	^
    pkg/service/manipulator_test.go:132:27: S1019: should use make([]byte, 10) instead (gosimple)
    		ImageData: make([]byte, 10, 10),
    

    To Cleanup makefile

    -  We don't need to show the makefile command, can be achieved by using silent @
    
    Provide any links for context
    • https://github.com/golangci/golangci-lint

    Do anyone want to pick this up or Can I pick this up?

  • Add Orientation Auto-fix Support

    Add Orientation Auto-fix Support

    Currently, darkroom doesn't read the image exif data and when an image is rotated, darkroom doesn't fix the orientation of it. I am suggesting that darkroom should check the orientation of the image based on the exif data and return as a straight image, given the auto=compress parameter being sent.

    Some example on how imgix handles this:

    • http://gojek-bike.imgix.net/go_jek/image_uploader_service/image/d3bc45f5-c6cf-4f25-9875-3e918bb2996b_Mochammad_arif_nugraha.jpg?auto=compress vs
    • http://gojek-bike.imgix.net/go_jek/image_uploader_service/image/d3bc45f5-c6cf-4f25-9875-3e918bb2996b_Mochammad_arif_nugraha.jpg
  • Adds new image operations - flip and rotate

    Adds new image operations - flip and rotate

    Stumbled upon this yesterday. I have worked on something like this before, but that was very basic. This pull requests adds code to flip and rotate images. :)

  • Rearranged cache headers

    Rearranged cache headers

    Cache headers are not set predictably. The cache headers are not being returned when deployed on an EC2.

    https://stackoverflow.com/questions/43202919/http-headers-not-returned-on-ec2

  • Failing Deployment

    Failing Deployment

    Which jobs are failing:

    • The deploy part in the master branch

    Since when has it been failing:

    • Since add documentation commit https://github.com/gojek/darkroom/commit/21c61f2d1e3f292ff4750f7e8483e83c5c19cd13

    • This is the job https://travis-ci.org/gojek/darkroom/builds/577348025?utm_source=github_status&utm_medium=notification

    On deploying application stage

    Reason for failure:

    • Git exit with non-zero error code
    • It happens because the runner doesn't have push access to the repository
    nothing to commit, working tree clean
    remote: Invalid username or password.
    fatal: Authentication failed for 'https://[email protected]/gojek/darkroom.git/'
    Error: Git push failed
    error Command failed with exit code 1.
    info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
    

    Anything else we need to know:

    • Is it really necessary to push to publish-gh-pages branch?
    • If it is, then I would recommend giving the runner push access using a new ssh credentials, we can use similar to this gitlab documentation https://docs.gitlab.com/ee/ci/ssh_keys/
  • Support webp based on request headers

    Support webp based on request headers

    This is a WIP feature which will add few changes in darkroom's behaviour.

    What would I like to be added

    • Returns image as WebP whenever possible on any kind of image format being requested
    • Returns image as PNG (for now) whenever a webP image is being requested but the client doesn't support webP yet

    Why is this needed

    Considering a requirement inside Gojek where we always want to serve images which are called by browsers or webview in our mobile apps as webp, I would like to add webP support on darkroom which uses the encoder that @adalberht added earlier.

    Comments

    • Link to the webP resource: https://developers.google.com/speed/webp/
    • The case I want to cover with this PR is accessing a webP image, eg: https://5cb8463c1978353e1deec5ea--confident-swirles-8a7305.netlify.com/gojek_logo.5c92f997.webp from chrome will yield the actual image vs accessing above URL from safari (I tried using Version 11.1.2 (13605.3.8)) which will not give the actual image being shown.
    • The logic for this feature will only get applied if the caller gives auto=format as an extra param (I have not added this yet)
    • This is a WIP, so I am expecting early comments about the design choice first
  • Broken docker images in release

    Broken docker images in release

    Environment
    Installation method: Docker
    Darkroom version: >= v0.0.5
    Operating system: Ubuntu x86_64
    Go version ('go version' output): go version go1.12.9 linux/amd64
    
    Steps to reproduce
    docker pull gojektech/darkroom:v0.0.9
    docker run -p 3000:3000 --env-file <path-to-darkroom.env> gojektech/darkroom:v0.0.9
    
    Observed result
    standard_init_linux.go:211: exec user process caused "no such file or directory"
    
    Expected result

    The server should run without errors.

    Comments

    This has most probably resulted from CGO_ENABLED build pipeline.

  • [Request] Add support for fit=scale

    [Request] Add support for fit=scale

    What would you like to be added

    fit=scale support Scales the image to fit the constraining dimensions exactly. The resulting image will fill the dimensions, and will not maintain the aspect ratio of the input image.

    Why is this needed

    To get resized images with fit type scale.

    Comments
  • Image load issue when crop only with width

    Image load issue when crop only with width

    Steps to reproduce

    Open this image and you'll find that image not loaded at all.

    .../darkroom/gofresh/v2/images/uploads/1c6561c7-87f8-4fa0-a480-f9386dbbed28_Shuffle-Subscription-In-Final.jpg?w=686&fit=crop&auto=compress

    Observed result
    Expected result

    Old version support this query param

    .../uploads/c0c65eee-9c64-49e9-af21-6e2378ec96fa.jpg?w=686&fit=crop&auto=compress

    This query param include width, crop and compress and calculate height based on image ratio and width.

  • Add default parameters support

    Add default parameters support


    name: Default Parameters Support about: Adding support for having a default parameters which will always by applied without being passed by the caller labels: enhancement


    What would you like to be added

    Apply default parameters on all download requests based on environment configuration.

    Why is this needed

    Imgix has it and on some of our Imgix sources, we set auto=compress to be always applied by default on some buckets.

    Comments
    • Ref: https://blog.imgix.com/2017/01/24/default-parameters
  • Bump github.com/aws/aws-sdk-go from 1.27.0 to 1.33.0

    Bump github.com/aws/aws-sdk-go from 1.27.0 to 1.33.0

    Bumps github.com/aws/aws-sdk-go from 1.27.0 to 1.33.0.

    Changelog

    Sourced from github.com/aws/aws-sdk-go's changelog.

    Release v1.33.0 (2020-07-01)

    Service Client Updates

    • service/appsync: Updates service API and documentation
    • service/chime: Updates service API and documentation
      • This release supports third party emergency call routing configuration for Amazon Chime Voice Connectors.
    • service/codebuild: Updates service API and documentation
      • Support build status config in project source
    • service/imagebuilder: Updates service API and documentation
    • service/rds: Updates service API
      • This release adds the exceptions KMSKeyNotAccessibleFault and InvalidDBClusterStateFault to the Amazon RDS ModifyDBInstance API.
    • service/securityhub: Updates service API and documentation

    SDK Features

    • service/s3/s3crypto: Introduces EncryptionClientV2 and DecryptionClientV2 encryption and decryption clients which support a new key wrapping algorithm kms+context. (#3403)
      • DecryptionClientV2 maintains the ability to decrypt objects encrypted using the EncryptionClient.
      • Please see s3crypto documentation for migration details.

    Release v1.32.13 (2020-06-30)

    Service Client Updates

    • service/codeguru-reviewer: Updates service API and documentation
    • service/comprehendmedical: Updates service API
    • service/ec2: Updates service API and documentation
      • Added support for tag-on-create for CreateVpc, CreateEgressOnlyInternetGateway, CreateSecurityGroup, CreateSubnet, CreateNetworkInterface, CreateNetworkAcl, CreateDhcpOptions and CreateInternetGateway. You can now specify tags when creating any of these resources. For more information about tagging, see AWS Tagging Strategies.
    • service/ecr: Updates service API and documentation
      • Add a new parameter (ImageDigest) and a new exception (ImageDigestDoesNotMatchException) to PutImage API to support pushing image by digest.
    • service/rds: Updates service documentation
      • Documentation updates for rds

    Release v1.32.12 (2020-06-29)

    Service Client Updates

    • service/autoscaling: Updates service documentation and examples
      • Documentation updates for Amazon EC2 Auto Scaling.
    • service/codeguruprofiler: Updates service API, documentation, and paginators
    • service/codestar-connections: Updates service API, documentation, and paginators
    • service/ec2: Updates service API, documentation, and paginators
      • Virtual Private Cloud (VPC) customers can now create and manage their own Prefix Lists to simplify VPC configurations.

    Release v1.32.11 (2020-06-26)

    Service Client Updates

    • service/cloudformation: Updates service API and documentation
      • ListStackInstances and DescribeStackInstance now return a new StackInstanceStatus object that contains DetailedStatus values: a disambiguation of the more generic Status value. ListStackInstances output can now be filtered on DetailedStatus using the new Filters parameter.
    • service/cognito-idp: Updates service API

    ... (truncated)

    Commits

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    • @dependabot use these labels will set the current labels as the default for future PRs for this repo and language
    • @dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language
    • @dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language
    • @dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

    You can disable automated security fix PRs for this repo from the Security Alerts page.

  • Add Helm Deployments

    Add Helm Deployments

    What would you like to be added

    helm charts

    Why is this needed

    Having helm charts will help in 1 command to startup darkroom, enabling more users to try, onboard, and improve darkroom. The helm chart will enable easy local development, integration, and onboarding for darkroom.

    Comments

    Just need to use the golang binary in the Dockerfile, and run the server with some pre-configured application, the buckets can be interfacing, or created using some gsutil commands. This will help darkroom to be used by more users.

  • Cut down on Processor interface

    Cut down on Processor interface

    What should be cleaned up or changed

    Currently, the Processor is getting bulky as we add more operations to it, this creates a problem for implementing new Processor variants as they have to implement all the methods, even if they are out of scope for that processor.

    The Processor should have an array of operations that it supports, each Operation can be implemented independently and then the Processor can be composed by selecting these Operations. An RFC document can be prepared to have more clarity on what needs to be changed to achieve this.

    Provide any links for context

    Ref: https://github.com/gojek/darkroom/blob/9186130ab03110cf029216ee606558673ff3a000/pkg/processor/interface.go#L5-L39

  • Automate cluster management of darkroom instances

    Automate cluster management of darkroom instances

    What would you like to be added

    Add cluster management and portal to manage Darkroom deployments on Kubernetes.

    Why is this needed

    Darkroom should be super easy to deploy and manage, this can be done in many ways, but one of the faster ones would be to introduce a Kubernetes Operator that manages the lifecycle of a darkroom app instance. This can be backed by a management UI that can abstract away the underlying details and make cluster operation very efficient.

    Comments

    I did a PoC for this which can be found here. darkroom gui

  • Support GIF formatted file whenever darkroom's parameter is given

    Support GIF formatted file whenever darkroom's parameter is given

    What would you like to be added

    There are few cases where we want to serve GIF to our customers, but it will not work if we enable some of the parameters we support, such as: auto=compress

    Why is this needed

    Currently, the behaviour is to return non successful response, leading to the client to not be able to show anything. At the very least, darkroom can just return the original file in case non jpeg/png file is being accessed with darkroom's supported parameters.

Storage and image processing server written in Go
Storage and image processing server written in Go

Mort An S3-compatible image processing server written in Go. Still in active development. Features HTTP server Resize, Rotate, SmartCrop Convert (JPEG

Jan 7, 2023
Image - This repository holds supplementary Go image librariesThis repository holds supplementary Go image libraries

Go Images This repository holds supplementary Go image libraries. Download/Insta

Jan 5, 2022
Image processing library and rendering toolkit for Go.

blend Image processing library and rendering toolkit for Go. (WIP) Installation: This library is compatible with Go1. go get github.com/phrozen/blend

Nov 11, 2022
A lightning fast image processing and resizing library for Go

govips A lightning fast image processing and resizing library for Go This package wraps the core functionality of libvips image processing library by

Jan 8, 2023
Image processing algorithms in pure Go
Image processing algorithms in pure Go

bild A collection of parallel image processing algorithms in pure Go. The aim of this project is simplicity in use and development over absolute high

Jan 6, 2023
Go package for fast high-level image processing powered by libvips C library

bimg Small Go package for fast high-level image processing using libvips via C bindings, providing a simple programmatic API. bimg was designed to be

Jan 2, 2023
Fast, simple, scalable, Docker-ready HTTP microservice for high-level image processing

imaginary Fast HTTP microservice written in Go for high-level image processing backed by bimg and libvips. imaginary can be used as private or public

Jan 3, 2023
Imaging is a simple image processing package for Go
Imaging is a simple image processing package for Go

Imaging Package imaging provides basic image processing functions (resize, rotate, crop, brightness/contrast adjustments, etc.). All the image process

Dec 30, 2022
A library for basic image processing in Go.
A library for basic image processing in Go.

Imaging Package imaging provides basic image processing functions (resize, rotate, crop, brightness/contrast adjustments, etc.). All the image process

Nov 26, 2021
A library for basic image processing in Go.
A library for basic image processing in Go.

Imaging Package imaging provides basic image processing functions (resize, rotate, crop, brightness/contrast adjustments, etc.). All the image process

Nov 26, 2021
Asset storage and on-the-fly image resize

air - Asset & Image Resize Asset storage and on-the-fly image resize powered by libvips. Uploading an asset $ http -f POST http://127.0.0.1:1323/uploa

Feb 5, 2022
An API which allows you to upload an image and responds with the same image, stripped of EXIF data

strip-metadata This is an API which allows you to upload an image and responds with the same image, stripped of EXIF data. How to run You need to have

Nov 25, 2021
Easily customizable Social image (or Open graph image) generator

fancycard Easily customizable Social image (or Open graph image) generator Built with Go, Gin, GoQuery and Chromedp Build & Run Simply, Clone this rep

Jan 14, 2022
Imgpreview - Tiny image previews for HTML while the original image is loading
Imgpreview - Tiny image previews for HTML while the original image is loading

imgpreview This is a Go program that generates tiny blurry previews for images t

May 22, 2022
Go package captcha implements generation and verification of image and audio CAPTCHAs.
Go package captcha implements generation and verification of image and audio CAPTCHAs.

Package captcha ⚠️ Warning: this captcha can be broken by advanced OCR captcha breaking algorithms. import "github.com/dchest/captcha" Package captch

Dec 30, 2022
Go package for decoding and encoding TARGA image format

tga tga is a Go package for decoding and encoding TARGA image format. It supports RLE and raw TARGA images with 8/15/16/24/32 bits per pixel, monochro

Sep 26, 2022
Image resizing in pure Go and SIMD

rez Package rez provides image resizing in pure Go and SIMD. Download: go get github.com/bamiaux/rez Full documentation at http://godoc.org/github.com

Dec 11, 2022
A Go program that takes an image, uses pigo to detect a face, and creates a gif that zooms in on the face
A Go program that takes an image, uses pigo to detect a face, and creates a gif that zooms in on the face

ok-zoomer Every GIF is a gift. How it works face detection with esimov/pigo color quantization / dithering with esimov/colorquant image resizing with

Nov 27, 2022
An extensive, fast, and accurate command-line image dithering tool.
An extensive, fast, and accurate command-line image dithering tool.

didder is an extensive, fast, and accurate command-line image dithering tool. It is designed to work well for both power users as well as pipeline scripting. It is backed by my dithering library, and is unique in its correctness and variety of dithering algorithms.

Dec 31, 2022