This looks very well designed and handles many edge cases others ignore - great work

i was wondering is we could also support minio and google cloud storage .. I have not texted it yet but it would be useful to know if you have ?

This is a new issue but continues on from our discussion in another issue I closed.

The proposal was to add Redis as a global counter system .

Here is my Use case.

i have clients ( browsers ) that store files and do mutations on those files

I have a server running in google cloud run , that talks to google storage / s3

The server is using s3 as it’s global store. Each web client has its own file store that persists offline ( using indexeddb ).

So we have an interesting situation with a file instance existing outside s3.

The notification system is what I am really interested in so that when a file on s3 changes we can notify all servers and clients. I think gose can do that now ?

when the client or server gets this event it should then request the file so it can get up to date.

when it asks for the file however it would be much more performant if it could get a binary byte array of the file difference . Yeah this is rather hard

So regarding Redis , I think that the file change events and the file change diff could be stored on this global message bus. So essentially the notification system uses this distributed message bus and can store the binary diff of the file .

Now I know that offline clients will not really work in this architecture , in the same way that git merges don’t always work because someone else changes the same part of the file but I think it’s possible to make that crazy aspect by using be tie clocks as well as a Chronos style clock system. But that’s more of a layer thing .

The main thing is to see if you think it’s possible to store the file diff in the message bus.

As i think about it it would mean that the Clients and Servers don’t even talk to s3 for sending file mutations . They would calculate the binary diff and send that into the message bus.

Importantly creation of new files would be send directly to S3.

I know this is probably out of scope for what Gose was created but I figured it’s worth explaining to see if you or others are interested in this use case .

https://flurryhead.medium.com/vector-clock-applications-965677624b94

When trying to upload a file to a minio bucket i get Upload failed: [object Object] in gose and this in the minio traces

version: '3.7'
services:
  minio:
    image: minio/minio:RELEASE.2022-06-03T01-40-53Z.fips
    command: server /mnt/data --console-address ":9001"
    ports:
      - 8200:9000 # API
      - 8201:9001 # Webinterface
    environment:
      MINIO_ROOT_USER: "<redacted>"
      MINIO_ROOT_PASSWORD: "<redacted>"
      MINIO_SERVER_URL: "<redacted>"
      MINIO_SITE_REGION: "s3"
    volumes:
      - minio-data:/mnt/data
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
      interval: 30s
      timeout: 20s
      retries: 3
  gose:
    image: ghcr.io/stv0g/gose:v0.4.0
    ports:
      - 8203:8080
    command: -config /config.yml
    volumes:
      - ./config.yml:/config.yml
    #environment:
      #GOSE_LISTEN: ":8080"
      #GOSE_BASE_URL: "<redacted>"
      #GOSE_BUCKET: "gose-uploads"
      #GOSE_ENDPOINT: "minio:9000"
      #GOSE_REGION: "s3"
      #GOSE_PATH_STYLE: "true"
      #GOSE_NO_SSL: "true"
      #GOSE_ACCESS_KEY: "<redacted>" # MINIO_ROOT_USER
      #GOSE_SECRET_KEY: "<redacted>" # MINIO_ROOT_PASSWORD
      #GOSE_S3_MAX_UPLOAD_SIZE: "50GB"
      #GOSE_S3_PART_SIZE: "5MB"
      #GOSE_S3_EXPIRATION_DEFAULT_CLASS: "1week"
    depends_on: [ minio ]

volumes:
  minio-data:

listen: ":8080"

base_url: <redacted>

max_upload_size: 50GB
part_size: 5MB

servers:
- bucket: gose-uploads

  endpoint: minio:9000
  region: s3

  path_style: true
  no_ssl: true

  access_key: "<redacted>" # MINIO_ROOT_USER
  secret_key: "<redacted>" # MINIO_ROOT_PASSWORD

  create_bucket: true

  expiration:
  - id: 1day
    title: 1 day
    days: 1
  - id: 1week
    title: 1 week
    days: 7
  - id: 1month
    title: 1 month
    days: 31
  - id: 1year
    title: 1 year
    days: 365

passing the config settings as environment variables seems to not work as well, settings like the bucket and max_upload/part_size arent getting set for me thats why i'm using the config

So, I fixed the final issue with the final checksum mismatch and tested it with the included docker-compose.yml file.

Let me know if works for you :)

got yet another issue i'm afraid, since i have my domain routed through cloudflare gose now thinks i'm using cloudflare s3 and not minio

Originally posted by @LeagueRaINi in https://github.com/stv0g/gose/issues/23#issuecomment-1152749492

So either i'm doing something terrible wrong or gose is not compatible with the latest version of MinIO.

Gose is throwing me this error on startup

gose_1   | 2022/06/02 22:31:33 Failed to setup servers: failed to set bucket gose-uploads's CORS rules: MalformedXML: The XML you provided was not well-formed or did not validate against our published schema.
gose_1   |      status code: 400, request id: 16F4EE6811CC0219, host id:

and MinIO complains about

minio_1  | API: SYSTEM()
minio_1  | Time: 22:31:33 UTC 06/02/2022
minio_1  | DeploymentID: 6fd95624-9ba0-4420-9c8e-e97c46253cf9
minio_1  | Error: expected element type <CreateBucketConfiguration> but have <CORSConfiguration> (xml.UnmarshalError)
minio_1  |        6: internal/logger/logger.go:278:logger.LogIf()
minio_1  |        5: cmd/handler-utils.go:55:cmd.parseLocationConstraint()
minio_1  |        4: cmd/auth-handler.go:344:cmd.checkRequestAuthTypeCredential()
minio_1  |        3: cmd/auth-handler.go:296:cmd.checkRequestAuthType()
minio_1  |        2: cmd/bucket-handlers.go:731:cmd.objectAPIHandlers.PutBucketHandler()
minio_1  |        1: net/http/server.go:2047:http.HandlerFunc.ServeHTTP()

I try to upload a file with connected minio s3 bucket.

The file is visible in the bucket but the UI shows the following error. If you try it again with the same file it works fine.

"Upload failed: Failed API request: failed to tag object"

docker-compose.yml following:

version: "3.3"
services:
  gose:
    image: ghcr.io/stv0g/gose:v0.4.0
    build:
      context: .
    expose:
     - "8080"
    environment:
      GOSE_LISTEN: ":8080"
      GOSE_BASE_URL: "https://dropme.domain"
      GOSE_BUCKET: "dropme"
      GOSE_ENDPOINT: "s3.eu-central-2.wasabisys.com"
      GOSE_REGION: "eu-central-2"
      GOSE_PATH_STYLE: "true"
      GOSE_NO_SSL: "false"
      GOSE_ACCESS_KEY: "***********" # changeme!
      GOSE_SECRET_KEY: "************" # changeme!
      GOSE_MAX_UPLOAD_SIZE: "50GB"
      GOSE_PART_SIZE: "16MB"
      GOSE_SETUP_CORS: "false"
      GOSE_SETUP_BUCKET: "false"
      GOSE_SETUP_LIFECYCLE: "false"
    networks:
      - web
    labels:
      - "traefik.docker.network=web"
      - "traefik.enable=true"
      - "traefik.frontend.rule=Host:dropme.domain"
      - "traefik.port=8080"

networks:
 web:
    external: true

Is there a way to enable a page that displays the name of the file and a download button instead of directly hotlinking the file? This would be a nice feature!

Error while building the Dockerfile provided in any of the versions: master, v0.6.0, v0.5.0 and v0.4.0

[...] 
[frontend-builder 6/7] COPY frontend/ .                                                                        0.1s
ERROR [frontend-builder 7/7] RUN npm run build                                                                 0.7s

 > [frontend-builder 7/7] RUN npm run build:
#11 0.628 
#11 0.628 > [email protected] build
#11 0.628 > webpack --config webpack.prod.js
#11 0.628 
#11 0.634 sh: 1: webpack: not found

executor failed running [/bin/sh -c npm run build]: exit code: 127

The dockerfile is untouched for this error above and it can be fixed by adding before the npm install: RUN npm i -g webpack webpack-cli webpack-merge

But then the error changes to:

 ERROR [frontend-builder 8/8] RUN npm run build                                                                 0.9s
                                                                                                              
 [frontend-builder 8/8] RUN npm run build:                                                                            
#12 0.571                                                                                                               
#12 0.571 > [email protected] build                                                                                            
#12 0.571 > webpack --config webpack.prod.js                                                                            
#12 0.571                                                                                                               
#12 0.693 [webpack-cli] Failed to load '/app/webpack.prod.js' config
#12 0.778 [webpack-cli] Error: Cannot find module 'webpack-merge'
#12 0.778 Require stack:
#12 0.778 - /app/webpack.prod.js
#12 0.778 - /usr/local/lib/node_modules/webpack-cli/lib/webpack-cli.js
#12 0.778 - /usr/local/lib/node_modules/webpack-cli/lib/bootstrap.js
#12 0.778 - /usr/local/lib/node_modules/webpack-cli/bin/cli.js
#12 0.778 - /usr/local/lib/node_modules/webpack/bin/webpack.js
#12 0.778     at Function.Module._resolveFilename (node:internal/modules/cjs/loader:933:15)
#12 0.778     at Function.Module._load (node:internal/modules/cjs/loader:778:27)
#12 0.778     at Module.require (node:internal/modules/cjs/loader:999:19)
#12 0.778     at require (node:internal/modules/cjs/helpers:102:18)
#12 0.778     at Object.<anonymous> (/app/webpack.prod.js:1:19)
#12 0.778     at Module._compile (node:internal/modules/cjs/loader:1099:14)
#12 0.778     at Object.Module._extensions..js (node:internal/modules/cjs/loader:1153:10)
#12 0.778     at Module.load (node:internal/modules/cjs/loader:975:32)
#12 0.778     at Function.Module._load (node:internal/modules/cjs/loader:822:12)
#12 0.778     at Module.require (node:internal/modules/cjs/loader:999:19) {
#12 0.778   code: 'MODULE_NOT_FOUND',
#12 0.778   requireStack: [
#12 0.778     '/app/webpack.prod.js',
#12 0.778     '/usr/local/lib/node_modules/webpack-cli/lib/webpack-cli.js',
#12 0.778     '/usr/local/lib/node_modules/webpack-cli/lib/bootstrap.js',
#12 0.778     '/usr/local/lib/node_modules/webpack-cli/bin/cli.js',
#12 0.778     '/usr/local/lib/node_modules/webpack/bin/webpack.js'
#12 0.778   ]
#12 0.778 }

executor failed running [/bin/sh -c npm run build]: exit code: 2

Development builds succeed by removing ENV NODE_ENV=production and removing --production from RUN npm install --production

We could add support for server-side encryption as supported by S3 SSE-C.

However this currently has the disadvantage that a user could not simple use the shortened URL anymore to download it via CURL/Wget, as SSE-C requires custom HTTP Headers for download.

This could be possibly worked around by a custom Javascript landing page which passes the headers via fetch() and then streams the response.

However, in this case we could already implement full end-to-end encryption which would make server-side encryption obsolete.

We currently use S3 bucket lifecycle rules to define a set of rentention classes from which the user could choose. Each uploaded file gets a tag assign which is matched by one of the life-cycle rules.

Despite not having set a url shortener and not being shown the setting in the gose ui it defaults to enabled leading to Upload failed: Failed API request: shortened URL requested but nut supported

Some S3 providers have especially both a separate endpoint and a separate public address, as well as allowing you to use a CDN What I mean by bucket.dev endpoint and CDN public.bucket.dev

We currently use S3 object tags to store some metadata information about each upload.

This works well for most use-cases, however has the disadvantage that the object tags are not easily modifiable. Or flat out not supported by some cloud hosters (Wasabi).

Some planned features require that we can modify upload metadata (#10, #11). Hence, I've thought about moving the metadata to a dedicated S3 object. There is still the disadvantage, that S3 does not really support atomic operations on objects so we still have a hard time tracking download numbers (e.g. any sort of counter). But I already have an idea of implementing some sort of local or distributed (Redis) caching layer in front of those metadata objects which could solve this.

docker-compose.yml

services:
  minio:
    container_name: Minio
    image: minio/minio
    command: server /mnt/data --console-address ":9001"
    ports:
      - 9000:9000 # API
      - 9001:9001 # Webinterface
    environment:
      MINIO_ROOT_USER: "minio-admin" # changeme!
      MINIO_ROOT_PASSWORD: "minio-pass" # changeme!
      MINIO_SERVER_URL: "http://localhost:9000"
      MINIO_SITE_REGION: "s3"
    volumes:
      - minio-data:/mnt/data
    healthcheck:
      test:
        [
          "CMD",
          "curl",
          "-f",
          "http://localhost:9000/minio/health/ready"
        ]
      interval: 30s
      timeout: 20s
      retries: 3

  gose:
    image: ghcr.io/stv0g/gose:v0.6.0
    ports:
      - 8080:8080
    env_file: .env
    depends_on:
      - minio

.env

# Listen address and port of Gose
GOSE_LISTEN= ":8080"
# Base URL at which Gose is accessible
GOSE_BASE_URL= "http://localhost:8080"
# Name of S3 bucket
GOSE_BUCKET= "gose-uploads"
# Hostname:Port of S3 server
GOSE_ENDPOINT= "minio:9000"
# Region of S3 server
GOSE_REGION= "s3"
# Prepend bucket name to path
GOSE_PATH_STYLE= "true"
# Disable SSL encryption for S3
GOSE_NO_SSL= "true"
# S3 Access Key
GOSE_ACCESS_KEY= "minio-admin"
# S3 Secret Key 
GOSE_SECRET_KEY= "minio-pass"
# Create S3 bucket if do not exists
GOSE_SETUP_BUCKET= "true"
# Setup S3 bucket CORS rules
GOSE_SETUP_CORS="true"
# Setup S3 bucket lifecycle rules
GOSE_SETUP_LIFECYCLE="true"
# Maximum upload size
GOSE_MAX_UPLOAD_SIZE= "50GB"
# Part-size for multi-part uploads
GOSE_PART_SIZE= "16MB"
# Number of days after which incomplete uploads are cleaned-up (set to 0 to disable)
GOSE_SETUP_ABORT_INCOMPLETE_UPLOADS=31

I have the idea on using S3 as a storage provider for IPFS. Given the fact the main IPFS client is implemented in Go, we could incorporate it into Gose and announce uploaded files to the IPFS DHT.

However for this to work we would need to make sure that IPFS can serve files from S3. In addition we would need to check if we can work around hashing the file itself in Gose without re-downloading it for hashing.

Related: https://github.com/ipfs/go-ipfs/blob/master/docs/datastores.md