aqua

aqua is a simple file uploading and sharing server for personal use. It is built to be easy to set up and host on your own server, for example to use it in combination with uploading tools like ShareX.

It is the successor/rework of my previous project lightf, but this time without trying weird things out and building a complete product instead.

The diagram above shows the general structure of the system. It is heavily based on abstracting interfaces for things like the file system or databases. There is also the Storage Handler which does all the management work in combining the different data sources (i.e. the physical files and their metadata) together. Every X (configurable!) minutes, the Cleanup Scheduler will command the Storage Handler to delete all expired files. This will also happen when the server starts.

Installation

There are as always multiple ways to install the server. The recommended way is to use Docker Compose or deploy it to Kubernetes, but we start with the manual way.

Also, if you don't want the server to serve the files themselves, then you can disable it via the environment variable described in section Configuration. That could be the case if you already have a Nginx installed and want it to serve the files instead of aqua. That way, you have full customization options.

Manually

This is only for those people, that are still living in the 90s or are not comfortable with a Docker installation.

1. First download the specific binary from the releases and put it somewhere (e.g. with wget on Linux). Also, add execution permission with chmod +x aq-linux-amd64.
2. Do the configuration you want (e.g. setting environment variables and creating an auth.yml). See Configuration
3. Execute the downloaded binary and don't forget to open port 8765 on your machine/server.

For further instructions like creating a service that can easily be started with service aqua start, please refer to other pages (there are a bunch that explain this) - I won't.

Docker (Compose)

Before following the steps, make sure you have Docker installed. And if you want to use the preconfigured docker-compose file, install Docker Compose as well.

Before you can start anything, you need the basis configuration:

1. Download and edit the auth.yml with your custom auth tokens and settings.
2. Download and rename the .env.dist to .env and edit it as well.

For a quick start with Docker, you can use the following command:

docker run --rm                    \
  --env-file ./.env                \
  -v ./auth.yml:/etc/aqua/auth.yml \
  -v ./files:/var/lib/aqua/        \
  -p 8765:8765                     \
  ghcr.io/superioz/aqua:latest

If you are on Windows, you might need to add MSYS_NO_PATHCONV=1 so that the paths are correctly parsed. Also, to refer to the current directory, use "$(pwd)" instead of ./, because that sometimes makes problems in Windows as well.

For Docker Compose it's easier:

docker-compose up

If you want to build the image yourself instead, you can of course git clone [email protected]:Superioz/aqua.git and then execute docker-compose up --build.

Kubernetes

Make sure to have a Kubernetes cluster running somewhere (either on bare-metal or on GCP, AWS or some other provider) and check beforehand if pods inside the cluster can access ghcr.io, which is the GitHub Docker registry, which we need for our aqua server image. Of course, you also need kubectl to be installed on your system.

1. Clone the repository or copy at least the /kubernetes folder into some directory
2. Create a namespace called aqua with kubectl create ns aqua
3. Apply the configuration files from the /kubernetes folder with kubectl apply -k ./kubernetes/. The -k flag is part of the newer versions of kubectl and allows Kustomize to work.
4. Either create an Ingress or connect to the cluster via kubectl port-forward svc/aqua-server 8765
5. Done. Finito. You should now be able to access the server.

Configuration

Environment Variables

For examplary usage of the environment variables, have a look at the .env.dist file in the root directory of the repository.

Tokens

Inside the auth.yml file you can configure which tokens are valid and for what file types they can be used for. An example file could look like this:

validTokens:
  - token: 71a4c056ab9b0fb965063344cd6616bc
    fileTypes:
      - image/png
      - image/jpeg

Each token is an element of the validTokens list and can be generated however you want. It is not even necessary for it to be hexadecimal, it can be any kind of string you want ( e.g. my_favorite_password or something like X_!]6rk[wC]%7aN^fB>#PzuFd{L,ugsq).

But if you want to keep it simple, you can use the aq CLI tool to generate it directly like so:

user@host:~$ aq generate --length 32
L1dLUm12!Lb%7Nz1ep4h5Vo+Fn531&EU

After adding the token to the list you may want to restrict what files can be uploaded with that token. That can be done with the fileTypes field. If you leave it empty, all file types are possible, otherwise only the configured ones.

Normally we would accept every possible MIME type, but as they behave completely different sometimes and we want to keep it simple, we only support the following ones:

application/gzip
application/json
application/pdf
application/vnd.rar
application/x-7z-compressed
application/zip
audio/mpeg
audio/ogg
audio/opus
audio/webm
image/gif
image/jpeg
image/png
image/svg+xml
text/csv
text/plain
video/mp4
video/mpeg
video/webm

Metrics

We also expose Prometheus metrics to the port :8766, if the specific environment variable is not set to false. To scrape these metrics simply make sure that they are enabled and that you add them to the Prometheus scrape targets.

If you are in a Kubernetes environment instead, a normal Prometheus deployment should automatically pick up the metrics, because we add specific annotations to the pods. If that does not work, or you are using the Prometheus Operator, you might want to have a look at ServiceMonitors somewhere at the Prometheus Operator documentation.

Here is a list of all Prometheus metrics:

CLI Tool

Installing the CLI tool is fairly simple. Just download the respective binary from releases, rename it and put it somewhere in your $PATH. Don't forget to add execution permissions chmod +x aq.

Now to upload a local file, you just have to execute the following command:

aq upload --host https://my-domain.com:8765 --token my_token local_file.png

To upload multiple files at the same time, just do it like this:

aq upload --host https://my-domain.com:8765 --token my_token local_file1.png local_file2.txt [...]

ShareX

As mentioned in the first section, you can easily configure ShareX to use aqua as server. For that you can copy the contents below to a file with the .sxcu ending like aqua.sxcu. Edit it now to your own needs, i.e. replacing the your-domain.com with the address of your installation and maybe adjusting the expiration, -1 means that the files won't expire.

After that you can import it to your custom upload goals in the ShareX UI.

{
  "Version": "12.4.0",
  "DestinationType": "ImageUploader, TextUploader",
  "RequestMethod": "POST",
  "RequestURL": "https://your-domain.com/upload",
  "Headers": {
    "Authorization": "Bearer your-generated-token-here"
  },
  "Body": "MultipartFormData",
  "Arguments": {
    "metadata": "{ \"expiration\": 3600 }"
  },
  "FileFormName": "file",
  "URL": "https://your-domain.com/$json:fileName$"
}

More information can be found here: ShareX Custom Uploader Guide.