Self-hosted video-hosting website and video archival manager for Niconico, Bilibili, and Youtube

Horahora

Self-hosted Video-hosting Website and Video Archival Manager for Niconico, Bilibili, and Youtube

Note: the above images are designs, rather than screenshots of the current frontend. The current frontend is essentially a worse version of the above. PRs are welcome 😉 ( 😭 )

Horahora is a microservice-based video hosting website with additional functionality for content archival from Niconico, Bilibili, and Youtube. Users can upload their own content, or schedule categories of content from other websites to be archived (e.g. a given channel on Niconico, a tag on Youtube, or a playlist from Bilibili). Content archived from other websites will be accessible in the same manner as user-uploaded videos, and will be organized under the same metadata (author, tags) associated with the original video.

https://discord.gg/vfwfpctJRZ

Local Use Instructions

  1. Install docker and docker-compose
  2. (Optional) If you don't want videos to be stored locally, modify secrets.env.template, adding the relevant values for your use case.
    • ORIGIN_FQDN: this will be the public URL of your Backblaze bucket WITH NO TRAILING SLASH. E.g. for me it's: https://f002.backblazeb2.com/file/otomads for backblaze, or https://horahora-dev-otomads.s3-us-west-1.amazonaws.com for s3.
    • STORAGE_BACKEND: 'b2' or 's3' (depending on which you want to use)
    • STORAGE_API_ID: the API ID for your Backblaze account if using backblaze, otherwise blank
    • STORAGE_API_KEY: The API key for your Backblaze account, otherwise blank
    • BUCKET_NAME: the storage bucket name for b2 or s3

If you want to use S3, you need to include your aws credentials and config in $HOME/.aws. The config and credentials will be mounted into the relevant services at runtime. See https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html for more information.

  1. sudo make up
  2. Visit localhost:8082 (or if it doesn't work initially, try to wait a minute)
    • there's currently a bug that causes the frontend on initial setup to return bad responses. If you encounter this, sudo docker-compose stop followed by sudo make up should fix the issue.
    • if it never works, check the container logs, and/or bug me on discord
    • you'll need to login as admin/horahora to view videos that have been encoded. There's an approval workflow which prevents unapproved videos from being viewed by regular users.
    • there's a delay between videos being downloaded/uploaded and being visible, as they need to be transcoded for DASH
  3. If everything comes up correctly, once you're logged in, visit the archival requests tab, and add a new category of content to be archived. If everything works, videos will start to be downloaded, and will be made available after a delay.

Contributing

Contributions are always welcome (and quite needed atm). If you'd like to contribute, and either aren't sure where to start, or lack familiarity with the relevant components of the project, please send me a message on Discord, and I'll help you out as best I can.

Horahora

Designs

Designs are listed here: https://github.com/horahoradev/horahora-designs

Task Roadmap

Missing features are tracked using Trello.

Our Trello board is: https://trello.com/b/Rm5TPR4Q/horahora

Note: this is currently outdated

Backup Restoration

Backup_service writes psql dumps of the three databases (userservice, videoservice, scheduler) to backblaze. To restore, place the three latest dumps in the sql dir, docker-compose up, run migrations, then run restore.sh from within the container.

Comments
  • Is it possible to transfer a copy of it to docker-hub?

    Is it possible to transfer a copy of it to docker-hub?

    This project is great and will save me a lot of trouble in saving the video, thanks for developing and sharing it.

    I tried to install it on unriad using docker compose, but it took 2 hours of searching trying... I finally got stuck at "sudo make up"

    It was too hard for me, who doesn't have much programming knowledge, and I couldn't find any detailed information about horahora.

    I don't know what to do next, it would be nice to find it in docker-hub.

    image

  • Remove Tailwindcss

    Remove Tailwindcss

    What I did was remove Tailwind from package.json, and consequently update the package-lock.json. Also, remove any imports in scss files, but I kept autoprefixer and postcss intact as it was not explicitly suggested to me to remove those two, although I'm pretty sure both came along with Tailwind.

  • refactor(webapp): archive requests page

    refactor(webapp): archive requests page

    Split /account/archive-requests into 4 pages:

    • /account/archives - Root page with links to the others.
    • /account/archives/requests - Requests page.
    • /account/archives/events - Events page.
    • /account/archives/downloads - Downloads page.
  • refactor(webapp): page layout

    refactor(webapp): page layout

    Lots of things:

    • sticky footer
    • redone account nav button
    • all pages share the same layout component (and therefore global state between client transitions)
  • refactor: webapp ci

    refactor: webapp ci

    • added .dockerignore to webapp so it wouldn't dump the local build artifacts and node_modules folder into the image
    • changed dockerfile so it would actually install with yarn and the package cache wouldn't bust on every source code change
  • react_app container crashes with `/bin/sh: craco: not found`

    react_app container crashes with `/bin/sh: craco: not found`

    Just trying to get started. After sudo make up docker logs are showing react_app_1 crashing as craco can't be found. Log looks like:

    react_app_1     | yarn run v1.22.19
    react_app_1     | $ craco start
    react_app_1     | /bin/sh: craco: not found
    react_app_1     | error Command failed with exit code 127.
    react_app_1     | info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
    

    This seems like a problem in one of the images you rely on but if you can't reproduce it let me know what more details I can give.

  • refactor(webapp): video post page

    refactor(webapp): video post page

    Parent PR: https://github.com/horahoradev/horahora/pull/82 SNH48《夜蝶》正式版MV 李艺彤黄婷婷大胆突破! Split it into several components to reduce the amount of prop drilling. The video page basically fetches the details to pass into the article while the comment list component within the video post article deals with the comments for that article.

  • Can't clone the repo

    Can't clone the repo

    First it starts with these errors:

    error: invalid path 'front_api/routes/GET_comments_:id.go'
    fatal: unable to checkout working tree
    warning: Clone succeeded, but checkout failed.
    You can inspect what was checked out with 'git status'
    and retry with 'git restore --source=HEAD :/'
    
    

    When trying to run git restore --source=HEAD :/:

    $ git restore --source=HEAD :/
    error: invalid path 'front_api/routes/GET_comments_:id.go'
    error: invalid path 'front_api/routes/GET_tag_:tag.go'
    error: invalid path 'front_api/routes/GET_users_:id.go'
    error: invalid path 'front_api/routes/GET_videos_:id.go'
    error: invalid path 'front_api/routes/POST_approve_:id.go'
    error: invalid path 'front_api/routes/POST_rate_:id.go'
    

    Looks like OS doesn't like colons in the path.

    Running on Windows 7.

  • Login Redirection Bug

    Login Redirection Bug

    There's an extremely nasty bug in which the client is continuously redirected to the login page, even on successful login. It does not occur, ever, if using inspect element (lmao), so I've been using that as a workaround. I will fix this...

  • Bump @xmldom/xmldom from 0.7.5 to 0.7.6 in /webapp

    Bump @xmldom/xmldom from 0.7.5 to 0.7.6 in /webapp

    Bumps @xmldom/xmldom from 0.7.5 to 0.7.6.

    Release notes

    Sourced from @​xmldom/xmldom's releases.

    0.7.6

    Commits

    Fixed

    Thank you, @​jftanner, @​Supraja9726 for your contributions

    Changelog

    Sourced from @​xmldom/xmldom's changelog.

    0.7.6

    Fixed

    Thank you, @​jftanner, @​Supraja9726 for your contributions

    0.8.3

    Fixed

    Thank you, @​Supraja9726 for your contributions

    0.9.0-beta.2

    Fixed

    Thank you, @​Supraja9726 for your contributions

    0.8.3

    Fixed

    Thank you, @​Supraja9726 for your contributions

    0.9.0-beta.1

    Fixed

    Only use HTML rules if mimeType matches [#338](https://github.com/xmldom/xmldom/issues/338), fixes [#203](https://github.com/xmldom/xmldom/issues/203)

    In the living specs for parsing XML and HTML, that this library is trying to implement, there is a distinction between the different types of documents being parsed: There are quite some rules that are different for parsing, constructing and serializing XML vs HTML documents.

    So far xmldom was always "detecting" whether "the HTML rules should be applied" by looking at the current namespace. So from the first time an the HTML default namespace (http://www.w3.org/1999/xhtml) was found, every node was treated as being part of an HTML document. This misconception is the root cause for quite some reported bugs.

    BREAKING CHANGE: HTML rules are no longer applied just because of the namespace, but require the mimeType argument passed to DOMParser.parseFromString(source, mimeType) to match 'text/html'. Doing so implies all rules for handling casing for tag and attribute names when parsing, creation of nodes and searching nodes.

    BREAKING CHANGE: Correct the return type of DOMParser.parseFromString to Document | undefined. In case of parsing errors it was always possible that "the returned Document" has not been created. In case you are using Typescript you now need to handle those cases.

    BREAKING CHANGE: The instance property DOMParser.options is no longer available, instead use the individual readonly property per option (assign, domHandler, errorHandler, normalizeLineEndings, locator, xmlns). Those also provides the default value if the option was not passed. The 'locator' option is now just a boolean (default remains true).

    BREAKING CHANGE: The following methods no longer allow a (non spec compliant) boolean argument to toggle "HTML rules":

    • XMLSerializer.serializeToString
    • Node.toString

    ... (truncated)

    Commits

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    • @dependabot use these labels will set the current labels as the default for future PRs for this repo and language
    • @dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language
    • @dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language
    • @dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

    You can disable automated security fix PRs for this repo from the Security Alerts page.

  • refactor: development compose config

    refactor: development compose config

    Changed development docker compose setup:

    • dev images are built off dev dockerfiles where applicable
    • dev images bind source volumes where applicable
    • added .dockerignore where applicable
    • golang images run in watcher mode using air, thus allowing live editing of the code.
    • runtime versions and their underlying images are passed as args to the images, thus ensuring uniform runtime versions across all images.

    New environment variables:

    • GOLANG_VERSION
    • GOLANG_AIR_VERSION
    • PYTHON_VERSION
    • POSTGRES_VERSION
    • FLYWAY_VERSION
    • ACTIVEMQ_VERSION
    • NODEJS_VERSION
    • UBUNTU_IMAGE
    • GOLANG_IMAGE
    • PYTHON_IMAGE
    • POSTGRES_IMAGE
    • ACTIVEMQ_IMAGE
    • NODEJS_IMAGE

    Compose file:

    • rewritten and renamed as per latest compose spec
    • containers have restart: unless-stopped directive so they will not restart by themselves when stopped
  • Bump @xmldom/xmldom from 0.7.5 to 0.7.8 in /webapp

    Bump @xmldom/xmldom from 0.7.5 to 0.7.8 in /webapp

    Bumps @xmldom/xmldom from 0.7.5 to 0.7.8.

    Release notes

    Sourced from @​xmldom/xmldom's releases.

    0.7.8

    Commits

    Fixed

    Thank you, @​fengxinming, for your contributions

    0.7.7

    Commits

    Fixed

    • Security: Prevent inserting DOM nodes when they are not well-formed CVE-2022-39353 In case such a DOM would be created, the part that is not well-formed will be transformed into text nodes, in which xml specific characters like < and > are encoded accordingly. In the upcoming version 0.9.0 those text nodes will no longer be added and an error will be thrown instead. This change can break your code, if you relied on this behavior, e.g. multiple root elements in the past. We consider it more important to align with the specs that we want to be aligned with, considering the potential security issues that might derive from people not being aware of the difference in behavior. Related Spec: https://dom.spec.whatwg.org/#concept-node-ensure-pre-insertion-validity

    Thank you, @​frumioj, @​cjbarth, @​markgollnick for your contributions

    0.7.6

    Commits

    Fixed

    Thank you, @​jftanner, @​Supraja9726 for your contributions

    Changelog

    Sourced from @​xmldom/xmldom's changelog.

    0.7.8

    Fixed

    Thank you, @​fengxinming, for your contributions

    0.9.0-beta.4

    Fixed

    • Security: Prevent inserting DOM nodes when they are not well-formed CVE-2022-39353 In case such a DOM would be created, the part that is not well-formed will be transformed into text nodes, in which xml specific characters like < and > are encoded accordingly. In the upcoming version 0.9.0 those text nodes will no longer be added and an error will be thrown instead. This change can break your code, if you relied on this behavior, e.g. multiple root elements in the past. We consider it more important to align with the specs that we want to be aligned with, considering the potential security issues that might derive from people not being aware of the difference in behavior. Related Spec: https://dom.spec.whatwg.org/#concept-node-ensure-pre-insertion-validity

    Chore

    Thank you, @​XhmikosR, @​awwright, @​frumioj, @​cjbarth, @​markgollnick for your contributions

    0.8.4

    Fixed

    • Security: Prevent inserting DOM nodes when they are not well-formed CVE-2022-39353 In case such a DOM would be created, the part that is not well-formed will be transformed into text nodes, in which xml specific characters like < and > are encoded accordingly. In the upcoming version 0.9.0 those text nodes will no longer be added and an error will be thrown instead. This change can break your code, if you relied on this behavior, e.g. multiple root elements in the past. We consider it more important to align with the specs that we want to be aligned with, considering the potential security issues that might derive from people not being aware of the difference in behavior. Related Spec: https://dom.spec.whatwg.org/#concept-node-ensure-pre-insertion-validity

    Thank you, @​frumioj, @​cjbarth, @​markgollnick for your contributions

    0.7.7

    Fixed

    • Security: Prevent inserting DOM nodes when they are not well-formed CVE-2022-39353 In case such a DOM would be created, the part that is not well-formed will be transformed into text nodes, in which xml specific characters like < and > are encoded accordingly. In the upcoming version 0.9.0 those text nodes will no longer be added and an error will be thrown instead. This change can break your code, if you relied on this behavior, e.g. multiple root elements in the past. We consider it more important to align with the specs that we want to be aligned with, considering the potential security issues that might derive from people not being aware of the difference in behavior. Related Spec: https://dom.spec.whatwg.org/#concept-node-ensure-pre-insertion-validity

    ... (truncated)

    Commits

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    • @dependabot use these labels will set the current labels as the default for future PRs for this repo and language
    • @dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language
    • @dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language
    • @dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

    You can disable automated security fix PRs for this repo from the Security Alerts page.

  • Document the purpose of each open port

    Document the purpose of each open port

    This is important for those who run this service along others (potential port conflicts). In particular, what needs to be known if any of these need to be actually exposed. Is the webapp required to listen on port 3000 outside the container, for example?

  • Bug: UserID 0 Has All Videos

    Bug: UserID 0 Has All Videos

    This is a fundamental design issue from how I've handled user queries. If the userID is 0, it won't be registered as a user query, and won't query for that user's videos. Not sure what I'll need to do here.

  • Undownloadable Video Observability and Retries

    Undownloadable Video Observability and Retries

    Currently, if a video can't be downloaded, it's marked as undownloadable, and will never be scheduled for download again. This is problematic in the context of transient/config-related errors, e.g. supplying the wrong credentials for S3 uploads.

    We should add some form of eventual retry, and better observability around videos that couldn't be downloaded.

listmonk is a standalone high performance, self-hosted newsletter and mailing list manager with a modern dashboard. Single binary app.
listmonk is a standalone high performance, self-hosted newsletter and mailing list manager with a modern dashboard. Single binary app.

listmonk is a standalone, self-hosted, newsletter and mailing list manager. It is fast, feature-rich, and packed into a single binary. It uses a PostgreSQL database as its data store.

Jan 1, 2023
listmonk is a standalone, self-hosted, newsletter and mailing list manager
listmonk is a standalone, self-hosted, newsletter and mailing list manager

listmonk is a standalone, self-hosted, newsletter and mailing list manager. It is fast, feature-rich, and packed into a single binary. It uses a Postg

Aug 15, 2022
This is my personal website, hosted on GitHub Pages

astrophena.name This is my personal website, hosted on GitHub Pages. Serving locally You need the latest Go and Node.js installed.

Nov 16, 2022
Remark42 is a self-hosted, lightweight, and simple comment engine
Remark42 is a self-hosted, lightweight, and simple comment engine

Remark42 is a self-hosted, lightweight, and simple (yet functional) comment engine, which doesn't spy on users. It can be embedded into blogs, articles or any other place where readers add comments.

Dec 28, 2022
Hammond is a self hosted vehicle management system to track fuel and other expenses related to all of your vehicles.
Hammond is a self hosted vehicle management system to track fuel and other expenses related to all of your vehicles.

Hammond is a self hosted vehicle management system to track fuel and other expenses related to all of your vehicles. It supports multiple users sharing multiple vehicles. It is the logical successor to Clarkson which has not been updated for quite some time now.

Jan 2, 2023
Focalboard is an open source, self-hosted alternative to Trello, Notion, and Asana.
Focalboard is an open source, self-hosted alternative to Trello, Notion, and Asana.

Focalboard Like what you see? ?? Give us a GitHub Star! ⭐ Focalboard is an open source, self-hosted alternative to Trello, Notion, and Asana. It helps

Jan 9, 2023
Mouthful is a self-hosted alternative to Disqus.

Mouthful is a lightweight commenting server written in GO and Preact. It's a self hosted alternative

Dec 20, 2022
longurl is a self-hosted short url service.

longurl Introduction longurl is a self-hosted short url service. Try It Out curl -H "Content-Type:application/json" -X POST --data '{"url": "https://g

Oct 24, 2022
🖖🏻 A self-hosted Quora like web application written in Go
🖖🏻 A self-hosted Quora like web application written in Go

Guora ???? A self-hosted Quora like web application written in Go 基于 Golang 类似知乎的私有部署问答应用 包含问答、评论、点赞、管理后台等功能 Quick Start (Docker Deploy) $ docker-comp

Dec 27, 2022
A self-hosted golang application that listens for Terraform pull request events via webhooks.
A self-hosted golang application that listens for Terraform pull request events via webhooks.

Atlantis Terraform Pull Request Automation Resources What is Atlantis? What does it do? Why should you use it? Stargazers over time Resources How to g

Dec 9, 2021
Hosting of the "Programming DIY / Программирование - это просто!" blog

This repository contains materials of the personal blog "Программирование - это просто / Programming DIY". How to use the blog engine A blog post abou

Jul 15, 2022
Server and bookmarklet to download files via youtube-dl directly from your browser
Server and bookmarklet to download files via youtube-dl directly from your browser

gropple A web service and bookmarklet to download videos with a single click. Pre-requisites a passing familiarity with the command line youtube-dl (p

Nov 12, 2022
:construction: Closed. A website and user system (Gin/Backbone).

A website and user system starter. Implemented with gin and Backbone. Gowall is port of Drywall Go Node.js Repository here Drywall Site Gowall Drywall

Jul 6, 2020
导航网站生成器(Navigation website generator)
导航网站生成器(Navigation website generator)

gena 导航网站生成器 | English Document 安装 一键生成(推荐) 从 gena-template 自动生成并自动部署到 GitHub Pages 源码安装 go1.16 required go get -u github.com/x1ah/gena/cmd/gena > gen

Nov 20, 2022
Collection of tools to interact with Intigriti website
Collection of tools to interact with Intigriti website

Small tool, written in Go, that constantly monitors Activity feed on https://app.intigriti.com Dashboard page and sends Slack/Discord notifications on

Dec 23, 2022
🔥 Hugo website builder, Hugo themes & Hugo CMS. No code, build with widgets!
🔥 Hugo website builder, Hugo themes & Hugo CMS. No code, build with widgets!

Wowchemy: the website builder for Hugo Join 750,000+ Sites. No Code. Easily Create Future-Proof Websites ✏️ ?? ?? ?? 1. Create any kind of website

Jan 9, 2023
Vela plugin designed for generating a static documentation website with Hugo.

Vela plugin designed for generating a static documentation website with Hugo.

Jul 22, 2022
API from AnonURL website, an anonymous URL shortener

API from AnonURL website, an anonymous URL shortener

Jan 23, 2022
Okta Golang Gin & Okta-Hosted Login Page Example

Okta Golang Gin & Okta-Hosted Login Page Example This example shows you how to use the Okta JWT verifier library to login a user to a Golang Gin appli

Oct 31, 2022