Docker-compose files for running full Storj network locally

docker-compose based Storj environment

storj-up is a swiss-army tool to create / customize Storj clusters with the help of docker-compose (not just storagenode but all satellite and edge services).

This is useful for Storj development and not for Storage Node operator.

Getting started

Install the tool:

go install storj.io/storj-up@latest

Go an empty directory an initialize docker-compose:

storj-up init

Start the cluster:

docker-compose up -d
docker-compose ps

You can check the generated credentials with:

storj-up credentials

You can set the required environment variables with eval $(storj-up credentials -e) (at least on Linux/OSX)

Or you can update the credentials of local rclone setup with storj-up credentials -w

More features

There are dedicated subcommands to modify the docker-compose easily. The generic form of these commands:

storj-up 
    
     
     

     
    
   

Here selector can be either a service (like storagenode) or a name of a service group. (like edge). To find out all the groups, please use storj-up services

Owner
Storj
Storj Labs is Building the Decentralized Future
Storj
Comments
  • How to use the storj-up binary inside of a container to retrieve the credentials?

    How to use the storj-up binary inside of a container to retrieve the credentials?

    With the v1 release storj-up binary is present inside the containers, however, the storj-up credentials command doesn't seem to work.

    That's the only command that the integration tests need.

    The error is the following

    docker compose exec satellite-api storj-up credentials -e
    #Server is not yet available. Retry in 1 sec... rpc: dial tcp 127.0.0.1:7777: connect: connection refused
    #Server is not yet available. Retry in 1 sec... rpc: dial tcp 127.0.0.1:7777: connect: connection refused
    #Server is not yet available. Retry in 1 sec... rpc: dial tcp 127.0.0.1:7777: connect: connection refused
    #Server is not yet available. Retry in 1 sec... rpc: dial tcp 127.0.0.1:7777: connect: connection refused
    #Server is not yet available. Retry in 1 sec... rpc: dial tcp 127.0.0.1:7777: connect: connection refused
    #Server is not yet available. Retry in 1 sec... rpc: dial tcp 127.0.0.1:7777: connect: connection refused
    #Server is not yet available. Retry in 1 sec... rpc: dial tcp 127.0.0.1:7777: connect: connection refused
    #Server is not yet available. Retry in 1 sec... rpc: dial tcp 127.0.0.1:7777: connect: connection refused
    #Server is not yet available. Retry in 1 sec... rpc: dial tcp 127.0.0.1:7777: connect: connection refused
    #Server is not yet available. Retry in 1 sec... rpc: dial tcp 127.0.0.1:7777: connect: connection refused
    #Server is not yet available. Retry in 1 sec... rpc: dial tcp 127.0.0.1:7777: connect: connection refused
    

    If I'm not mistaken that's because inside a container the other containers run in the docker subnetwork and they are not mapped to the container localhost.

    I'm looking for a workaround or to know if the implementation can be changed in a way that the command work when it runs inside of a container.

  • storj-up v2.0

    storj-up v2.0

    This is a full refactor of storj up it can be used to generate multiple runtime descriptions. This is a feature branch that is supposed to be merged. it has a lot of comments and 1 small feature is missing. Does not support custom config file configuration. The biggest challenge with this patch is to get reviewers since its such a huge change.

    Acceptance Criteria:

    • fix any issues that arise from code reviews or testing
    • get this patch merged
    • test with storjscan
    • resolve merge/ build conflicts.

    https://review.dev.storj.io/c/storj/up/+/8629

  • storj-up create user with older creation date

    storj-up create user with older creation date

    For invoice generation we insert fake usage into the database. The problem we insert with an old timestampe in order to be able to create the invoice for the previous month like we would in production. This only works if we manipulate the creation date of the user. It would be great it storj-up could create the user with an timestamp 90 days back. That would give us enough room to insert fake data.

    storj-up credentials --email [email protected] https://github.com/storj/up/blob/23e2923cc3d95ff54142c65eaa201d76910ad8af/pkg/console.go#L102

  • arm64 images

    arm64 images

    When running the Storj network on Mac M1s, storj-up attempts to run amd64 images which only works when you have the qemu emulation tools installed. It would be nice if we had arm64 images for storj-up to remove some layers of virtualization and simplify debugging.

  • Update `local-ws` to support storagenode and multinode dashboards

    Update `local-ws` to support storagenode and multinode dashboards

    Currently using storj-up local-ws command will add the required mounts for front end development of the satellite UI. This command should be expanded to allow for specifying which front end you would like to mount, so that it can also include the storagenode and multinode dashboards.

  • go generate stringer error

    go generate stringer error

    running a go generate gives the error

    stringer: internal error: package "fmt" without types was imported from "storj.io/storj-up/pkg/common"

    and does not update the key_string.go file with any changes.

  • storj-up generate invoice

    storj-up generate invoice

    Similar to https://github.com/storj/up/issues/17

    All we need is some fake usage for the previous month in order to be able to run the invoice command. The fake data doesn't need to be fancy. It can be a single tally result with 720 TBh of used space. -> 1 TB stored for a month.

  • How to register an access for the edge services

    How to register an access for the edge services

    I'd like to use storj-up to test the Edge Uplink API.

    Specifically, I tried to use it for testing the method config.RegisterAccess, but it fails, despite that I imagined that it was not going to work because I don't know how to provide a valid certificate.

    Do you know if it's possible to use it for testing the mentioned method?

  • Create tickets for missing features that are used by storj-sim

    Create tickets for missing features that are used by storj-sim

    Currently we are using storj-up and storj-sim side-by-side, however only one of them should be sufficient. Let's figure out the things we need to drop storj-sim as a tool.

    First, we need to figure out which features are still missing in storj-up. Especially around scenarios involving the tests in scripts.

    Acceptance Criteria:

    • Create tickets for each of the missing features in storj/up

    Things to make work:

    • [ ] #61
    • [ ] #62
    • [ ] #63
      • (maybe a testplanet like thing instead?) -- or maybe it's not needed
    • [ ] #64

    Things to replace storj-sim in github.com/storj/storj:

    • [ ] #65
    • [ ] #66
    • [ ] #67
    • [ ] #68
    • [ ] #69

    Things to replace storj-sim in github.com/storj/gateway-(st/mt):

    • [ ] #70
    • [ ] #71
    • [ ] #72

    Things to replace in uplink:

    • [ ] #73
  • Start using version tags

    Start using version tags

    I started to use storj-up a week ago for uplink-rust CI; see https://github.com/storj-thirdparty/uplink-rust/pull/28

    It worked perfectly until I merged https://github.com/storj-thirdparty/uplink-rust/pull/28 into the main branch and realized that those commits didn't pass the CI, but the CI passed finely with the PR.

    I was digging into it and I realized that the CI of storj-up doesn't pass either with the same issue 429 too many requests. In the case of uplink-rust, the error is returned when running thestorj-up credentials` command.

    If we would like to use storj-up in CIs, we have to start to consider tagging the repository with version tags or keeping the main branch always green and backward compatible, although I believe that the second is harder.

    For now, I solved the issue using a specific commit, however, it isn't ideal https://github.com/storj-thirdparty/uplink-rust/pull/32

  • cmd/testdata: Adding some new commands

    cmd/testdata: Adding some new commands

    Adding two commands; a command to generate storage & command to simulate a test situation (create a bucket, some usage, delete bucket, recreate bucket) *using testdata project-usage & testdata billing-deleted-bucket, commands respectively with storj-up

  • Verify storj-up native works cross platform.

    Verify storj-up native works cross platform.

    Currently most of the development on storj-up has been on linux / mac. Ensure that all the features work cross-platform and require minimal other tools to be installed.

Related tags
Template project to get started with a simple API skeleton in Go and Docker

A template project to create a Docker image for a Go application. The example application exposes an HTTP endpoint through a REST API packaged as a static binary.

Apr 4, 2022
Utility to compare files/directories with output in html. (like unix diff command)

#godiff A File/Directory diff-like comparison tool with HTML output. This program can be use to compare files and directories for differences. When co

Nov 4, 2022
ROFL eXtension Debugger - for League of Legends replay files

ROFLXD - ROFL eXtension Debugger This is an incomplete project. The original goal of this project was to convert GLR replays to a form that was playab

Nov 27, 2022
Docker-compose files for running full Storj network locally

docker-compose based Storj environment storj-up is a swiss-army tool to create / customize Storj clusters with the help of docker-compose (not just st

Nov 16, 2022
Compose Switch is a replacement to the Compose V1 docker-compose (python) executable

Compose Switch Compose Switch is a replacement to the Compose V1 docker-compose (python) executable. It translates the command line into Compose V2 do

Jan 8, 2023
Mutagen Compose is a modified version of Docker Compose that offers automated integration with Mutagen.

Mutagen Compose Mutagen Compose is a (minimally) modified version of Docker Compose that offers automated integration with Mutagen. This allows you to

Dec 22, 2022
Tool to convert docker-compose files to set of simple docker commands

docker-decompose Tool to convert docker-compose files to set of simple docker commands. Install Use go get to install the latest version of the librar

Apr 12, 2022
Kubedock is a minimal implementation of the docker api that will orchestrate containers on a Kubernetes cluster, rather than running containers locally.

Kubedock Kubedock is an minimal implementation of the docker api that will orchestrate containers on a kubernetes cluster, rather than running contain

Nov 11, 2022
Storj is building a decentralized cloud storage network
Storj is building a decentralized cloud storage network

Ongoing Storj v3 development. Decentralized cloud object storage that is affordable, easy to use, private, and secure.

Jan 8, 2023
Easily run your Compose application to the cloud with compose-cli

This CLI tool makes it easy to run Docker containers and Docker Compose applications in the cloud using either Amazon Elastic Container Service (ECS) or Microsoft Azure Container Instances (ACI) using the Docker commands you already know.

Jan 8, 2023
Hassle-free minimal CI/CD for git repositories with docker or docker-compose projects.
Hassle-free minimal CI/CD for git repositories with docker or docker-compose projects.

GIT-PIPE Hassle-free minimal CI/CD for git repos for docker-based projects. Features: zero configuration for repos by default automatic encrypted back

Sep 23, 2022
Dotnet-appsettings-env - Convert .NET appsettings.json file to Kubernetes, Docker and Docker-Compose environment variables

dotnet-appsettings-env Convert .NET appsettings.json file to Kubernetes, Docker

Dec 30, 2022
Dotnet-appsettings-env - Convert .NET appsettings.json file to Kubernetes, Docker and Docker-Compose environment variables

dotnet-appsettings-env Convert .NET appsettings.json file to Kubernetes, Docker

Feb 16, 2022
Mesos Framework to use docker-compose files.
Mesos Framework to use docker-compose files.

mesos-compose Mesos Framework to use docker-compose files. Requirements Apache Mesos min 1.6.0 Mesos with SSL and Authentication is optional Redis Dat

Dec 14, 2022
Temporal Server docker-compose files

Temporal Server docker-compose files This repository provides docker-compose fil

Dec 24, 2021
PoC for running AWS services(kinesis, dynamodb, lambdas) locally with Localstack

hotdog-localstack-PoC PoC for running AWS services(kinesis, dynamodb, lambdas) locally with Localstack alias awslocal="aws --endpoint-url=http://local

Dec 3, 2022
Uploads data to the Storj ecosystem and tries to download induvidual pieces from different servers to measure latency

Storj longtail This application uploads data to the Storj ecosystem and tries to download induvidual pieces from different servers to measure latency.

Oct 28, 2021
Sample multi docker compose environment setup

Instructions This is a demonstration of a Multi Docker Compose. The purpose of this repositoy is ongoing research on "Docker compose" architecture des

Oct 21, 2022
Example used to try a compose application with Docker Dev Environments

compose-dev-env Example used to try a Compose application with Docker Dev Environments. This example is based on the nginx-golang-mysql sample of awes

Dec 29, 2022
contaiNERD CTL - Docker-compatible CLI for containerd, with support for Compose, Rootless, eStargz, OCIcrypt, IPFS, ...

contaiNERD CTL - Docker-compatible CLI for containerd, with support for Compose, Rootless, eStargz, OCIcrypt, IPFS, ...

Jan 4, 2023