Self-hosted web app for encoding files to a target format using distributed computing.

GitHub all releases Docker Pulls Docker Pulls Docker Image Size (tag) Docker Image Size (tag) GitHub go.mod Go version (Controller) GitHub go.mod Go version (Runner) Go Report Card Controller CI/CD Runner CI/CD

What is Encodarr?

Encodarr is a self-hosted web application that encodes video files to a target format using distributed computing to spread the workload across many machines.

Why use Encodarr?

Easy to Setup

Encodarr bypasses the need to share media across the network by instead transmitting the file to be operated on to the Runners. This means that Encodarr is much easier to setup than other solutions.

Cross-platform

Both the Controller and Runner can be cross-compiled for any system supported by the Go toolchain including Raspberry Pis and M1 Macs.

Dependencies

The container images come with the dependencies installed, so if you go that route, all you need is Docker.

If you are not using containers, you will need mediainfo and FFmpeg in the PATH variable.

Installing

Container images are available from Docker Hub(Controller, Runner) and the GitHub Container Registry(Controller, Runner).

Pre-built executables are available via the Releases page.

Usage

Docker/Docker Compose

The latest Controller version can be run using the following docker run command:

docker run -d \
  --name Encodarr-Controller \
  -e TZ=Europe/London \
  -v <path to data>:/config:rw \
  -v <path to media>:/media:rw \
  -p 8123:8123 \
  --restart unless-stopped \
  --user 1000:1000 \
  brenekh/encodarr-controller:latest

or if you prefer Docker Compose:

version: "2.2"
services:
  encodarr-controller:
    image: brenekh/encodarr-controller:latest
    container_name: Encodarr-Controller
    environment:
      - TZ=Europe/London
    volumes:
      - <path to controller data>:/config:rw
      - <path to media>:/media:rw
    ports:
      - 8123:8123
    restart: unless-stopped
    user: "1000:1000"

The latest Runner version is similar.

Docker run:

docker run -d \
  --name Encodarr-Runner \
  -v <path to runner data>:/config:rw \
  -e TZ=Europe/London
  -e "ENCODARR_RUNNER_NAME=Runner 1" \
  -e ENCODARR_RUNNER_CONTROLLER_IP=<Controller IP> \
  -e ENCODARR_RUNNER_CONTROLLER_PORT=8123 \
  --restart unless-stopped \
  brenekh/encodarr-runner:latest

Docker Compose:

version: "2.2"
services:
  encodarr-runner:
    image: brenekh/encodarr-runner:latest
    container_name: Encodarr-Runner
    volumes:
      - <path to runner data>:/config:rw
    environment:
      - TZ=Europe/London
      - ENCODARR_RUNNER_NAME=Runner 1
      - ENCODARR_RUNNER_CONTROLLER_IP=<Controller IP>
      - ENCODARR_RUNNER_CONTROLLER_PORT=8123
    restart: unless-stopped

The Controller path to media and the corresponding mount inside the container can be anything you want and you can have as many as you want. For example, /mnt/disk/tv:/tv:rw and /mnt/disk/movies:/movies:rw.

In addition, the paths to data that are mounted to /config in the container should be separate folders, ideally with full paths(/home/user/docker instead of ~/docker).

Startup Configuration

Startup values configured either through environment variables, or command line arguments. All of the command line variants expect a value after a space (--port 8123).

Controller

ENCODARR_PORT, --port sets the port for the HTTP web server. For containers, it is recommended to modify the external port (8124:8123) instead of setting this value. (default: 8123)

ENCODARR_CONFIG_DIR, --config-dir sets the directory that the configuration files are saved to. This includes the log file. In a container, this is pre-set to /config. (default: <platform user config directory>/encodarr/controller/config)

Runner

ENCODARR_CONFIG_DIR, --config-dir sets the directory that the configuration files are saved to. This includes the log file. In a container, this is pre-set to /config. (default: <platform user config directory>/encodarr/runner-<time of runner startup>/config)

ENCODARR_TEMP_DIR, --temp-dir sets the directory that the media files are saved to when they are being worked on. If you want to protect your flash memory(SSDs and SD Cards) from excessive reads and writes, you can set this to be on another storage medium. (default: <platform user temp directory>)

ENCODARR_LOG_LEVEL, --log-level sets the level of output from the logging system to both the log file and the terminal output. Possible values are: trace, debug, info, warn (or warning, they are identical), error, critical. (default: info)

ENCODARR_RUNNER_NAME, --name sets the name to be shown in the Web UI when referring to this runner. (default: <machine hostname>-<random number>)

ENCODARR_RUNNER_CONTROLLER_IP, --controller-ip sets the IP for connecting to the Controller. (default: localhost)

ENCODARR_RUNNER_CONTROLLER_PORT, --controller-port sets the port for connecting to the Controller. (default: 8123)

Contributing

I am currently looking for someone to verify the Mac OS binaries. They theoretically should just work, but I can't test it because I don't have access to a Mac. If you do verify please let me know in the Discussions page, but if they don't please open a new issue.

If you want to contribute to this project, head over to CONTRIBUTING.md to get started.

Code of Conduct

This project holds all maintainers, contributors, and participants to the standards outlined by the Contributor Covenant, a copy of which can be found in CODE_OF_CONDUCT.md.

Future Plans

  • Instead of configuring with dropdowns and checkboxes, use a plugin system

  • Massive frontend overhaul

  • Scheduled working times

Owner
Brenek Harrison
Go/Python/Kotlin fanboy. Interested in all things development.
Brenek Harrison
Comments
  • Update to 0.3.0 make encodarr to not working.

    Update to 0.3.0 make encodarr to not working.

    So, doing update of encodarr runner + controller just make it not working.

    Controller terminal show absolutely no errors.

    Webapp get black for no reasons and runner show error and not encoding.

    (Controller running on a arm64 raspebrry pi 4 on debian 10)

    bug

    EDIT : weirdly, it's only the runners section that make the webapps goes dark. i can access to /settings /histories and all, but going to /running make him dark just after i click on it..

    Also, my libraries settings (path, codec and all) get blank/empty Did i have like, to reset the config folder something like that ? emptylib

  • 0.3.0 Controller report SQL logic error when add a new library

    0.3.0 Controller report SQL logic error when add a new library

    On Encodarr 0.3.0 on amr64 on Raspberry pi 4, on a fresh install with a new added library and path (to avoid bug when path mask is empty.

    To Reproduce Steps to reproduce the behavior:

    1. Encodarr on rpi 4, debug mode logs
    2. Delete data.db and data.db.backup
    3. launch encodarr, add librairy
    4. See bunch of errors at each files

    Expected behavior Scan the library properly

    Errors :

    SQL logic error : ON CONFLICT clauses does not match any PRIMARY KEY or UNIQUE constraint (1) and "SQL logic error: table files has no column named metadata (1)" when scan for a library

    at the end, before the controller crash and close

  • Release 0.2.0

    Release 0.2.0

    Enhancements:

    • Runner is now written in Go

    • Cache docker layers in GitHub Actions builds

    • CI/CD embeds version info using -ldflags

    • Controller and Runner now have --help and --version flags

    Bug fixes:

    • Running cards don't show the video encoding symbol (#75)

    • The Controller still deletes the original file even if the Runner reports that it failed (#81)

    Documentation:

    • Added Go version badge for Runner

    • Added Cross-platform as a reason for choosing Encodarr

    • Added TZ env var to Docker and Docker Compose commands

    • Rewrite Startup Configuration section for the Runner to match the options available with the Go Runner

    • Remove Go Runner from Future Plans section

  • Bump node-forge from 1.2.1 to 1.3.0 in /frontend

    Bump node-forge from 1.2.1 to 1.3.0 in /frontend

    Bumps node-forge from 1.2.1 to 1.3.0.

    Changelog

    Sourced from node-forge's changelog.

    1.3.0 - 2022-03-17

    Security

    • Three RSA PKCS#1 v1.5 signature verification issues were reported by Moosa Yahyazadeh ([email protected]).
    • HIGH: Leniency in checking digestAlgorithm structure can lead to signature forgery.
    • HIGH: Failing to check tailing garbage bytes can lead to signature forgery.
    • MEDIUM: Leniency in checking type octet.
      • DigestInfo is not properly checked for proper ASN.1 structure. This can lead to successful verification with signatures that contain invalid structures but a valid digest.
      • CVE ID: CVE-2022-24773
      • GHSA ID: GHSA-2r2c-g63r-vccr

    Fixed

    • [asn1] Add fallback to pretty print invalid UTF8 data.
    • [asn1] fromDer is now more strict and will default to ensuring all input bytes are parsed or throw an error. A new option parseAllBytes can disable this behavior.
      • NOTE: The previous behavior is being changed since it can lead to security issues with crafted inputs. It is possible that code doing custom DER parsing may need to adapt to this new behavior and optional flag.
    • [rsa] Add and use a validator to check for proper structure of parsed ASN.1 RSASSA-PKCS-v1_5 DigestInfo data. Additionally check that the hash algorithm identifier is a known value from RFC 8017 PKCS1-v1-5DigestAlgorithms. An invalid DigestInfo or algorithm identifier will now throw an error.
      • NOTE: The previous lenient behavior is being changed to be more strict since it could lead to security issues with crafted inputs. It is possible that code may have to handle the errors from these stricter checks.

    ... (truncated)

    Commits

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    • @dependabot use these labels will set the current labels as the default for future PRs for this repo and language
    • @dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language
    • @dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language
    • @dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

    You can disable automated security fix PRs for this repo from the Security Alerts page.

  • Can't create config folder at first launch, even with all permission

    Can't create config folder at first launch, even with all permission

    For some reason, encodarr-controller can't launch without "sudo" before, because it can't create a folder into .config for.. whatever reason. Trying everything :

    admin@nullmoon:~/Encodarr $ ls -l encodarr-controller
    -rwxr--r-- 1 admin 1007 16457113 Apr 23 23:48 encodarr-controller
    admin@nullmoon:~/Encodarr $ ./encodarr-controller
    mkdir /home/admin/.config/encodarr/controller/config: permission denied
    admin@nullmoon:~/Encodarr $ chmod 777 encodarr-controller
    admin@nullmoon:~/Encodarr $ ./encodarr-controller
    mkdir /home/admin/.config/encodarr/controller/config: permission denied
    admin@nullmoon:~/Encodarr $ sudo chmod 777 encodarr-controller
    admin@nullmoon:~/Encodarr $ ./encodarr-controller
    mkdir /home/admin/.config/encodarr/controller/config: permission denied
    

    Running on Debian 10 with RPI 4 on arm64

  • chore: update to go 1.17

    chore: update to go 1.17

    • update docker images to use go 1.17
    • improve docker images with suggestions from hadolint
    • update GitHub workflow files to use go 1.17
    • update .gitignore to ignore go specific files instead of python

    Closes https://github.com/BrenekH/encodarr/issues/130

  • Release 0.3.1

    Release 0.3.1

    Notable fixes:

    • Front-end now sends an empty list when the path masks field is empty (#111)
    • The files table is recreated with the correct column name during migration (#112)
    • Don't truncate the backup database unless a new backup is being performed

    Enhancements:

    • Update go.mod to Go 1.17.
  • Add encoding device (#101)

    Add encoding device (#101)

    • Add HWDevice as a jobParameter

    • Add failing test for HWDevice param

    • genFFmpegCmd handle HWDevice param

    • Add hw_device to http test string

    • Send HWDevice if hardware encoding is enabled

    • Show hw device in library card

    • Add hw device in create library modal

    • HWDevice field in edit library modal

    • Add warning text for hardware encoding

    • Deploy frontend changes to Controller

    • Fix failing test

  • Hardware encoding (#99)

    Hardware encoding (#99)

    • Backend support for hardware encoding

    • Show hardware codec info on LibraryCard

    • Hardware Codec support in EditLibrary modal

    • Only show codec if using hardware

    • Hardware encoding support for CreateLibrary modal

    • Change frontend version

    • Add new frontend to controller

    • Log ffmpegCodec instead of mapped codec

    • Add new hardware-related params to tests

  • Runner shuts down instead of requesting a new job when it has timed out.

    Runner shuts down instead of requesting a new job when it has timed out.

    This isn't an issue in Docker as long as restart: unless-stopped is set, but it isn't a particularly good idea to crash when requesting a new job is a perfectly acceptable action to take.

  • Some track indicies fail to parse because they are 64-bit

    Some track indicies fail to parse because they are 64-bit

    The simple fix is to change strconv.Atoi to strconv.ParseInt("", 10, 64) in controller/library/mediainfo/mediainfo.go, and changing the FileMetadata struct to use int64 instead of int.

  • [Tracker] Controller Unit Tests

    [Tracker] Controller Unit Tests

    Purpose

    This issue is for tracking the progress of writing unit tests for the Controller, which is mostly taking place on the controller-unit-tests branch.

    Contributions

    I'm happy to take pull requests which helps further this issue along, but be aware that I may request changes to better fit the project as I would any other PR.

    Some important things to keep in mind:

    • Dependencies must be mocked (dependencies in this case simply means anything that is not easily controllable in a test)
      • Rewriting existing Controller code to support the mocking of dependencies is perfectly acceptable and in fact highly encouraged.
    • Tests should be independent of one another. This means that shared structs should not be used and instead a new one should be allocated for each test.

    Coverage

    Total Coverage: 68.7%

    • [x] jobhealth
      • [x] job_health.go - 100.0%
      • [x] mock.go - 80.0%
      • [x] structs.go - 0.0%
    • [ ] library
      • [x] commanddecider
        • [x] cmd_decider.go - 97.4%
      • [x] cache.go - 96.3%
      • [ ] file_discovery.go - 45.8%
      • [ ] manager.go - 0.0%
    • [ ] settings
      • [x] mock.go - 92.9%
      • [ ] settings.go - 46.2%
    • [ ] sqlite
      • [ ] database.go - 0.0%
      • [ ] file_cache.go - 0.0%
      • [ ] health_checker - 14.8%
      • [ ] library_manager - 0.0%
      • [ ] runner_comm.go - 0.0%
      • [ ] user_interfacer.go - 0.0%
    • [x] mocks.go - 100.0%
    • [ ] run.go - 96.7%
    • [ ] structs.go - 0.0%
    • [x] util.go - 100.0%
  • Standardize Linux install using Makefiles

    Standardize Linux install using Makefiles

    Things like man-pages and Systemd services need to be copied to the appropriate locations, and Makefiles are a great way to do that without making it super distro-dependant(making distro packages will be fairly easy).

  • Allow pausing of a Library

    Allow pausing of a Library

    Sometimes it may be useful to pause a Library from dispatching.

    For example, one could be copying over files that need to be renamed before being encoded/transcoded.

  • Temporal Job Runner might help with the workflows / activities

    Temporal Job Runner might help with the workflows / activities

    Is your feature request related to a problem? Please describe.

    I noticed many issues related to Orchestration, and so think that perhaps a workflow runner is needed to track it all

    Describe the solution you'd like

    https://github.com/temporalio/temporal/ i think is pretty awesome. Its from the same team that build Uber Cadence, but much easier to use and looks quite solid.

    https://docs.temporal.io/docs/server-quick-install

    They have a golang SDK and samples.

    If you want really decoupled you can even use a DSL like this sample: https://github.com/temporalio/samples-go/tree/master/dsl

    Describe alternatives you've considered

    Cadence, NATS, etc

    Additional context

    If you hate it its cool as i know dependencies are tough to decide on. I noticed that Temporal is very decoupled from whatever docker project you use it with.

  • Controller sending the same job to multiple Runners

    Controller sending the same job to multiple Runners

    I don't think this is even a race condition causing an issue. Somehow the exact same uuid with the exact same file is being sent to two different Runners.

LiveKit - Open source, distributed video/audio rooms over WebRTC

LiveKit is an open source project that provides scalable, multi-user conferencing over WebRTC. It's designed to give you everything you need to build real time video/audio capabilities in your applications.

Jan 9, 2023
Frf-media-download - This tool downloads all the files you have uploaded to FreeFeed

FreeFeed Media Downloader This tool downloads all the files you have uploaded to

Jan 29, 2022
[WIP] a very simple, tiny and intuitive ffmpeg wrapper with a cli interface for inspecting & transforming media files supported by the original ffmpeg software

About a very simple, tiny and intuitive ffmpeg wrapper with a cli interface for inspecting & transforming media files supported by the original ffmpeg

Oct 21, 2022
Live on-demand transcoding in go using ffmpeg. Also with NVIDIA GPU hardware acceleration.

Go live HTTP on-demand transcoding Transcoding is expensive and resource consuming operation on CPU and GPU. For big companies with thousands of custo

Dec 16, 2022
Plays videos using Prometheus and Grafana, e.g. Bad Apple.
Plays videos using Prometheus and Grafana, e.g. Bad Apple.

prometheus_video_renderer Plays videos using Prometheus and Grafana, e.g. Bad Apple. Modes Currently 3 different modes are supported. Bitmap The bitma

Nov 30, 2022
A small program in Go that efficiently compresses videos using ffmpeg.

discordcompressor A small program in Go that efficiently compresses videos using ffmpeg. Dependencies FFmpeg including FFprobe Usage discordcompressor

Dec 18, 2022
Synthetic media is a realistic transformation of audio and video using artificial intelligence.

Synthetic media is a realistic transformation of audio and video using artificial intelligence.

Nov 20, 2021
golang function that download a video from youtube, and convert it to a mp3 file using ffmpeg

echedwnmp3 echedwnmp3 is a function that download a video from youtube, and convert it to a mp3 file using ffmpeg example package main import(echedwn

Dec 7, 2021
lmmp3 is a little golang library that download a video from youtube, and convert it to a mp3 file using ffmpeg

lmmp3 lmmp3 is a function that download a video from youtube, and convert it to a mp3 file using ffmpeg You need to have installed ffmpeg in your syst

Aug 12, 2022
GoatCounter is an open source web analytics platform available as a hosted service or self-hosted app

GoatCounter is an open source web analytics platform available as a hosted service (free for non-commercial use) or self-hosted app. It aims to offer easy to use and meaningful privacy-friendly web analytics as an alternative to Google Analytics or Matomo.

Dec 29, 2022
mass-binding-target is a command line tool for generating binding target list by search plot files from disk.

mass-binding-target mass-binding-target is a command line tool for generating binding target list by search plot files from disk. Build Go 1.13 or new

Nov 5, 2021
Myretail-target-case-study - Case study assessment for Target.com

myRetail This project contains two solutions to the Target myRetail case study. The prompt is copied over to PROMPT.md for convenience, but the TLDR i

Jan 1, 2022
EggContractor is a self-hosted contract monitoring web app + CLI client for Egg

EggContractor is a self-hosted contract monitoring web app + CLI client for Egg, Inc.. It allows you to easily monitor all your contract progress, as well as peeking into prospective coops you may want to join.

Nov 23, 2022
Bigmachine is a library for self-managing serverless computing in Go

Bigmachine Bigmachine is a toolkit for building self-managing serverless applications in Go. Bigmachine provides an API that lets a driver process for

Nov 15, 2022
Distributed, lock-free, self-hosted health checks and status pages
Distributed, lock-free, self-hosted health checks and status pages

Checkup is distributed, lock-free, self-hosted health checks and status pages, written in Go. It features an elegant, minimalistic CLI and an idiomati

Dec 30, 2022
High performance, self-hosted newsletter and mailing list manager with a modern dashboard. Single binary app.
High performance, self-hosted newsletter and mailing list manager with a modern dashboard. Single binary app.

listmonk is a standalone, self-hosted, newsletter and mailing list manager. It is fast, feature-rich, and packed into a single binary. It uses a Postg

Dec 30, 2022
listmonk is a standalone high performance, self-hosted newsletter and mailing list manager with a modern dashboard. Single binary app.
listmonk is a standalone high performance, self-hosted newsletter and mailing list manager with a modern dashboard. Single binary app.

listmonk is a standalone, self-hosted, newsletter and mailing list manager. It is fast, feature-rich, and packed into a single binary. It uses a PostgreSQL database as its data store.

Jan 1, 2023
go-eexcel implements encoding and decoding of XLSX like encoding/json

go-eexcel go-eexcel implements encoding and decoding of XLSX like encoding/json Usage func ExampleMarshal() { type st struct { Name string `eexce

Dec 9, 2021
Self-hosted music streaming server 🎶 with RESTful API and Web interface
Self-hosted music streaming server 🎶 with RESTful API and Web interface

Self-hosted music streaming server ?? with RESTful API and Web interface. Think of it as your very own Spotify!

Dec 27, 2022