Mahi is an all-in-one HTTP service for file uploading, processing, serving, and storage.

Mahi Go Report Card GoDoc

Mahi is an all-in-one HTTP service for file uploading, processing, serving, and storage. Mahi supports chunked, resumable, and concurrent uploads. Mahi uses Libvips behind the scenes making it extremely fast and memory efficient.

Mahi currently supports any s3 compatible storage, which includes (AWS s3, DO Spaces, Wasabi, Backblaze B2). The specific storage engine can be set when creating an application.

Mahi supports different databases for storing file meta-data and analytics. Currently, the 2 supported databased are PostgreSQL and BoltDB. The database of choice can be provided via the config file.

Features

Install

Libvips must be installed on your machine.

Ubuntu

sudo apt install libvips libvips-dev libvips-tools

MacOS

brew install vips

For other systems check out instructions here.

Installing mahid server.

go get -u github.com/threeaccents/mahi/...

This will install the mahid command in your $GOPATH/bin folder.

Usage

mahid -config=/path/to/config.toml

If no config is passed Mahi will look for a mahi.toml file in the current directory.

Applications

Mahi has the concept of applications. Each application houses specific files and the storage engine for those files. This makes Mahi extremely flexible to use for different projects. If on one project you decide to use s3 as your storage engine and another DO Spaces, Mahi easily handles it for you.

Applications can be created via our Web API.

Uploads

Files are uploaded to Mahi via multipart/form-data requests. Along with passing in the file data, you must also provide the application_id. Mahi will handle processing and storing the file blob in the application's storage engine along with storing the file meta-data in the database. To view an example upload response check out the Web API

Large File Uploads

When dealing with large files, it is best to split the file into small chunks and upload each chunk separately. Mahi easily handles chunked uploads storing each chunk and then re-building the whole file. Once the whole file is re-built Mahi uploads the file to the application's storage engine. To view an example upload response check out the Web API

Other benefits of chunking up files are the ability to resume uploads and uploading multiple chunks concurrently. Mahi handles both scenarios for you with ease.

File Transformations (More Coming Soon)

Mahi supports file transformations via URL query params. Currently, the supported operations are:

  • Resize (width, height) ?width=100&height=100
  • Smart Crop ?crop=true
  • Flip ?flip=true
  • Flop ?flop=true
  • Zoom ?zoom=2
  • Black and White ?bw=true
  • Quality(JPEG), Compression(PNG) ?quality=100 ?compression=10
  • Format conversion format is based on the file extension. To transform a png to webp, just use the .webp extension.

All queries can be used together. For example, to resize the width, make the image black and white, and change the format to webp the params would look like this:

https://yourdomain.com/myimage.webp?width=100&bw=true

Stats

Mahi currently tracks these stats for both specific applications and the service as a hole:

  • Transformations: Total transformations
  • Unique Transformations: Unique transformations per file.
  • Bandwidth: Bytes served.
  • Storage: Bytes stored.
  • File Count: Total files.

These stats can be retrieved via our Web API.

Config

Mahi's is configured via a toml file. Here are toml config examples. Configuration options include:

  • db_engine:string(default: bolt) The main database for mahi. Valid options are postgres and bolt. This is not to be confused with the storage engine. Storage engine is set per application via the Web API.
  • http
    • port:int(default: 4200) the port to run mahi on.
    • https:boolean(default: false) configures server to accept https requests.
    • ssl_cert_path:string path to ssl certificate. Only required if https is set to true.
    • ssl_key_path:string path to ssl key. Only required if https is set to true.
  • security
    • auth_token:string token for authenticating requests
    • aes_key:string key for use with AES-256 encryption. This is used to encrypt storage secrets.
  • upload
    • chunk_upload_dir:string(default: ./data/chunks) directory for storing chunks while an upload is happening. Once an upload is completed, the chunks are deleted.
    • full_file_dir:string(default: ./data/files) full_files are temp files used while building chunks or downloading files from the storage engine. These temp files are removed once the request is completed.
    • max_chunk_size:int64(default: 10MB) max size of a file chunk in bytes.
    • max_file_size_upload:int64(default: 50MB) max size of a file for a regular upload in bytes.
    • max_transform_file_size:int64(default: 50MB) max size of a file that can be transformed in bytes.
  • bolt(only used if db_engine is set to bolt
    • dir:string(default: ./data/mahi/mahi.db) directory for bolt db file.
  • postgresql(only used if db_engine is set to postgres)
    • database:string(default: mahi) name of database.
    • host:string(default: localhost) host of database.
    • port:int(default: 5432) port of database.
    • user:string(default: mahi) username of database.
    • password:string(default: ) password of database.
    • max_conns:int(default: 10 connections per CPU) maximum connections for database pool.

Postgres

To use Postgres the necessary data tables must be created. SQL files are located in the migrations folder. In the future, Mahi will come with a migrate command that will automatically handle creating the necessary tables for you. For now, you have 2 options. Install tern, cd into the migrations folder, and run tern migrate. The second option is just to copy and paste the SQL provided directly in a GUI or command line instance of Postgres.

Owner
Rodrigo Lessa
code, surf, build soil, and chill.
Rodrigo Lessa
Comments
  • Bump lodash from 4.17.15 to 4.17.20 in /cmd/mahiadmin/static

    Bump lodash from 4.17.15 to 4.17.20 in /cmd/mahiadmin/static

    Bumps lodash from 4.17.15 to 4.17.20.

    Commits
    Maintainer changes

    This version was pushed to npm by bnjmnt4n, a new releaser for lodash since your current version.


    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    • @dependabot use these labels will set the current labels as the default for future PRs for this repo and language
    • @dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language
    • @dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language
    • @dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

    You can disable automated security fix PRs for this repo from the Security Alerts page.

  • update usage

    update usage

    {"level":"error","service":"mahi","error":"start and end times cannot be the same","time":"2020-06-02T20:44:28-06:00","message":"failed to update usage"}

  • Bump lodash from 4.17.20 to 4.17.21 in /cmd/mahiadmin/static

    Bump lodash from 4.17.20 to 4.17.21 in /cmd/mahiadmin/static

    Bumps lodash from 4.17.20 to 4.17.21.

    Commits
    • f299b52 Bump to v4.17.21
    • c4847eb Improve performance of toNumber, trim and trimEnd on large input strings
    • 3469357 Prevent command injection through _.template's variable option
    • See full diff in compare view

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    • @dependabot use these labels will set the current labels as the default for future PRs for this repo and language
    • @dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language
    • @dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language
    • @dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

    You can disable automated security fix PRs for this repo from the Security Alerts page.

  • Bump ws from 5.2.2 to 5.2.3 in /cmd/mahiadmin/static

    Bump ws from 5.2.2 to 5.2.3 in /cmd/mahiadmin/static

    Bumps ws from 5.2.2 to 5.2.3.

    Release notes

    Sourced from ws's releases.

    5.2.3

    Bug fixes

    • Backported 00c425ec to the 5.x release line (76d47c14).
    Commits

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    • @dependabot use these labels will set the current labels as the default for future PRs for this repo and language
    • @dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language
    • @dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language
    • @dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

    You can disable automated security fix PRs for this repo from the Security Alerts page.

  • Bump y18n from 4.0.0 to 4.0.3 in /cmd/mahiadmin/static

    Bump y18n from 4.0.0 to 4.0.3 in /cmd/mahiadmin/static

    Bumps y18n from 4.0.0 to 4.0.3.

    Changelog

    Sourced from y18n's changelog.

    4.0.3 (2021-04-07)

    Bug Fixes

    • release: 4.x.x should not enforce Node 10 (#126) (1e21a53)

    4.0.1 (2020-11-30)

    Bug Fixes

    Commits
    Maintainer changes

    This version was pushed to npm by oss-bot, a new releaser for y18n since your current version.


    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    • @dependabot use these labels will set the current labels as the default for future PRs for this repo and language
    • @dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language
    • @dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language
    • @dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

    You can disable automated security fix PRs for this repo from the Security Alerts page.

  • Bump hosted-git-info from 2.8.8 to 2.8.9 in /cmd/mahiadmin/static

    Bump hosted-git-info from 2.8.8 to 2.8.9 in /cmd/mahiadmin/static

    Bumps hosted-git-info from 2.8.8 to 2.8.9.

    Changelog

    Sourced from hosted-git-info's changelog.

    2.8.9 (2021-04-07)

    Bug Fixes

    Commits
    Maintainer changes

    This version was pushed to npm by nlf, a new releaser for hosted-git-info since your current version.


    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    • @dependabot use these labels will set the current labels as the default for future PRs for this repo and language
    • @dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language
    • @dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language
    • @dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

    You can disable automated security fix PRs for this repo from the Security Alerts page.

A http service to verify request and bounce them according to decisions made by CrowdSec.

traefik-crowdsec-bouncer A http service to verify request and bounce them according to decisions made by CrowdSec. Description This repository aim to

Dec 21, 2022
A HTTP mole service
A HTTP mole service

httpmole provides a HTTP mock server that will act as a mole among your services, telling you everything http clients send to it and responding them whatever you want it to respond. Just like an actual mole.

Jul 27, 2022
HTTP/2 Apple Push Notification service (APNs) provider for Go with token-based connection

APNs Provider HTTP/2 Apple Push Notification service (APNs) provider for Go with token-based connection Example: key, err := apns.AuthKeyFromFile("Aut

Dec 29, 2022
A dead simple, stupid, http service.

A dead simple, stupid, http service implemented in a complicated way just for the sake of following Go design patterns and scalability. Useful for learning and testing basic kubernetes networking. Made on an insomniac night.

Sep 2, 2022
A Concurrent HTTP Static file server using golang .

A Concurrent HTTP static server using Golang. Serve Static files like HTML,CSS,Js,Images,Videos ,ect. using HTTP. It is Concurrent and Highly Scalable.Try now!

Dec 19, 2021
Simple, lightweight and faster response (JSON, JSONP, XML, YAML, HTML, File) rendering package for Go

Package renderer Simple, lightweight and faster response (JSON, JSONP, XML, YAML, HTML, File) rendering package for Go Installation Install the packag

Dec 13, 2022
Link converter service converts URLs to deeplinks or deeplinks to URLs.

Link converter Link converter service converts URLs to deeplinks or deeplinks to URLs. The service responds to the incoming request and first checks w

Dec 23, 2021
Composable chains of nested http.Handler instances.

chain go get github.com/codemodus/chain Package chain aids the composition of nested http.Handler instances. Nesting functions is a simple concept. I

Sep 27, 2022
Go http.Hander based middleware stack with context sharing

wrap Package wrap creates a fast and flexible middleware stack for http.Handlers. Features small; core is only 13 LOC based on http.Handler interface;

Apr 5, 2022
Minimalist net/http middleware for golang

interpose Interpose is a minimalist net/http middleware framework for golang. It uses http.Handler as its core unit of functionality, minimizing compl

Sep 27, 2022
Add interceptors to GO http.Client

mediary Add interceptors to http.Client and you will be able to Dump request and/or response to a Log Alter your requests before they are sent or resp

Nov 17, 2022
Lightweight Middleware for net/http

MuxChain MuxChain is a small package designed to complement net/http for specifying chains of handlers. With it, you can succinctly compose layers of

Dec 10, 2022
Idiomatic HTTP Middleware for Golang

Negroni Notice: This is the library formerly known as github.com/codegangsta/negroni -- Github will automatically redirect requests to this repository

Jan 2, 2023
A tiny http middleware for Golang with added handlers for common needs.

rye A simple library to support http services. Currently, rye provides a middleware handler which can be used to chain http handlers together while pr

Jan 4, 2023
A collection of useful middleware for Go HTTP services & web applications 🛃

gorilla/handlers Package handlers is a collection of handlers (aka "HTTP middleware") for use with Go's net/http package (or any framework supporting

Dec 31, 2022
Simple middleware to rate-limit HTTP requests.

Tollbooth This is a generic middleware to rate-limit HTTP requests. NOTE 1: This library is considered finished. NOTE 2: Major version changes are bac

Dec 28, 2022
OpenID Connect (OIDC) http middleware for Go

Go OpenID Connect (OIDC) HTTP Middleware Introduction This is a middleware for http to make it easy to use OpenID Connect. Currently Supported framewo

Jan 1, 2023
Hex - Expectations for HTTP handlers

Hex is a simple wrapper that extends httptest.Server with an expectation syntax, allowing you to create mock APIs using a simple and expressive DSL:

Aug 9, 2021
Statigz serves pre-compressed embedded files with http in Go

statigz statigz serves pre-compressed embedded files with http in Go 1.16 and later. Why? Since version 1.16 Go provides standard way to embed static

Dec 24, 2022