Describe the bug with a clear and concise description of what the bug is.

Me and my users see their timeline being stuck after a few hours

What's your GoToSocial Version?

GoToSocial 0.5.1-SNAPSHOT git-ccd5b34

GoToSocial Arch

amd64 Docker

Browser version

Google Chrome 106.0.5249.119

What happened?

My timeline got stuck after a few hours

What you expected to happen?

Having new posts appear in my timeline, even after a few hours

How to reproduce it?

1. Have a server running for a long time
2. Check your timeline one day
3. Profit

Anything else we need to know?

I like pineapple pizza

Having a few issues with pleroma <-> GtS interactivity in relation to searches and follows, as discussed with @tsmethurst in the matrix room.

When I search for @[email protected] from my pleroma instance, the account is returned but it is the @[email protected] format (the same is seen when searching for my GtS account with the same formatting). There are no errors in this scenario, just the incorrect nickname is returned (visually). I can follow this returned account from pleroma.

However, when I attempt to follow my pleroma account from my GtS account it fails.

At the GtS end I see the following error:

[GIN] 2021/07/21 - 16:59:05 | 204 |     467.249µs |    192.168.1.83 | OPTIONS  "/api/v1/accounts/01FB4N70SVQPHBE9F766DG8ND1/follow"
time="2021-07-21T16:59:05+01:00" level=error msg="target account wasn't set on context" asType=Follow func=Create
[GIN] 2021/07/21 - 16:59:05 | 200 |   51.900197ms |    192.168.1.83 | POST     "/api/v1/accounts/01FB4N70SVQPHBE9F766DG8ND1/follow"
[GIN] 2021/07/21 - 16:59:06 | 200 |    7.007773ms |    192.168.1.83 | GET      "/users/jon/main-key"
[GIN] 2021/07/21 - 16:59:06 | 200 |    7.330689ms |    192.168.1.83 | GET      "/users/jon/main-key"
time="2021-07-21T16:59:06+01:00" level=error msg="batch deliver had at least one failure: POST request to https://social.nipponalba.scot/users/jk/inbox failed (400): 400 Bad Request"

At the pleroma end I see the following errors: Could not validate against known public keys: {:error, :error} Signature validation error for: https://gts.kelbie.scot/users/jon, make sure you are forwarding the HTTP Host header!

Pleroma version is: 2.3.0

trying to follow @[email protected] from @[email protected]:

Apr 14 20:02:20 social gotosocial[28943]: time=2022-04-14T20:02:20Z level=info msg=authentication not passed for public key owner https://glitch.social/actor; signature value was 'keyId="https://glitch.social/actor#main-key",algorithm="rsa-sha256",headers="(request-target) host date accept",signature="I3v2a1AuO0hVB3fib83VZARhUL3BO9HXbdvtHaq8L2vqu5gYWUy1z64Y7JNJ2CX1WQauZ+J+49fx3eK5BnU3O+bEJo1B6r4VDW7mLVzRemzHvQOLMgySLXtncn16P2PriimqQ1Ejn/q1B8O//POcs0n7Ot7z2pc4Uu3BMTY9eek7u6zv1hfXEoMavROwdyvJ2vBTxc6+1If185OVIJeVkOMivtufYRNkl1w29EaQVVsB0aRvL+oHm16+a5cCHV7MHnuEtyvOk0Ognt1n4/LtRPfp/VKpDKznzqmrGRA45UdzQncB8DpbB6OEQtl+Cd+kigbIQ6Ja+9NzAZyFR86Abw=="' func=AuthenticateFederatedRequest
Apr 14 20:02:20 social gotosocial[28943]: time=2022-04-14T20:02:20Z level=info msg=not authorized func=UsersGETHandler url=/users/me
Apr 14 20:02:20 social gotosocial[28943]: time=2022-04-14T20:02:20Z level=info msg=[8.08787ms] Unauthorized: wrote 54 bytes clientIP=192.168.17.10 latency=8.08787ms method=GET path=/users/me statusCode=401 userAgent=http.rb/4.4.1 (Mastodon/3.4.6; +https://glitch.social/)
Apr 14 20:02:20 websrv2-hel1 /usr/local/bin/gotosocial[28944]: time=2022-04-14T20:02:20Z level=error msg=batch deliver had at least one failure: POST request to https://glitch.social/users/meena/inbox failed (401): 401 Unauthorized 
Apr 14 20:02:20 social gotosocial[28943]: time=2022-04-14T20:02:20Z level=error msg=batch deliver had at least one failure: POST request to https://glitch.social/users/meena/inbox failed (401): 401 Unauthorized

similar failures occur when i try to follow an account on a newer codebase, with the federation fixes:

Apr 14 21:09:58 social gotosocial[28943]: time=2022-04-14T21:09:58Z level=debug msg=entering NewID func=NewID newID={"@context":"https://www.w3.org/ns/activitystreams","actor":"https://soc.eena.me/users/me","id":"https://soc.eena.me/users/me/follow/012X9043BMGPWHXEY6W3QZ6VWR","object":"https://cathode.church/users/meena","to":"https://cathode.church/users/meena","type":"Follow"}
Apr 14 21:09:58 social gotosocial[28943]: time=2022-04-14T21:09:58Z level=debug msg=entering Create create={"@context":"https://www.w3.org/ns/activitystreams","actor":"https://soc.eena.me/users/me","id":"https://soc.eena.me/users/me/follow/012X9043BMGPWHXEY6W3QZ6VWR","object":"https://cathode.church/users/meena","to":"https://cathode.church/users/meena","type":"Follow"} func=Create
Apr 14 21:09:58 social gotosocial[28943]: time=2022-04-14T21:09:58Z level=debug msg=entering Get func=Get id=https://soc.eena.me/users/me
Apr 14 21:09:58 social gotosocial[28943]: time=2022-04-14T21:09:58Z level=info msg=[22.09331ms] OK: wrote 176 bytes clientIP=192.168.17.10 latency=22.09331ms method=POST path=/api/v1/accounts/01FXX82TB2C3H026FKG18T9ZBY/follow statusCode=200 userAgent=Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:100.0) Gecko/20100101 Firefox/100.0
Apr 14 21:09:59 social gotosocial[28943]: time=2022-04-14T21:09:59Z level=info msg=[2.991909ms] OK: wrote 554 bytes clientIP=192.168.17.10 latency=2.991909ms method=GET path=/users/me/main-key statusCode=200 userAgent=http.rb/5.0.4 (Mastodon/3.5.1+glitch+cathode; +https://cathode.church/)
Apr 14 21:10:00 social gotosocial[28943]: time=2022-04-14T21:10:00Z level=debug msg=performing GET to https://cathode.church/actor#main-key func=Dereference
Apr 14 21:10:00 social gotosocial[28943]: time=2022-04-14T21:10:00Z level=info msg=authentication not passed for public key owner https://cathode.church/actor; signature value was 'keyId="https://cathode.church/actor#main-key",algorithm="rsa-sha256",headers="(request-target) host date accept",signature="TnNY5jJVteujED5P5Zy7oNvJUeiKqahpM5+B+3rnLSWQEzSwR+zh5asanc3oSWqtYcHMfUa38PVeYS4LnEKpEXvarYYOyFMQ2sEaJfMevODyCOv8+Z0py2e2USdu3gf+lT808mAESaE2OLIK5lG/EH19Cp4wJVWK60/P+p2TzCgYmCBW4hd78PXCNIvyg1LDAiCqpH90YQSFVM7nbzdwI8w2T62ckLX9jCk5zgvcFikD5F3OPl3ELo2e9fZCkOHZ580NJXy2d4G9J2cDNWBnu9AQFFWBI5UAUBqWVkSfG/YyZs0qN9vyANmlFpqMcUDnvR+XsZ1Ir4oLhdjBm2a/mg=="' func=AuthenticateFederatedRequest
Apr 14 21:10:00 social gotosocial[28943]: time=2022-04-14T21:10:00Z level=info msg=not authorized func=UsersGETHandler url=/users/me
Apr 14 21:10:00 social gotosocial[28943]: time=2022-04-14T21:10:00Z level=info msg=[287.51058ms] Unauthorized: wrote 54 bytes clientIP=192.168.17.10 latency=287.51058ms method=GET path=/users/me statusCode=401 userAgent=http.rb/5.0.4 (Mastodon/3.5.1+glitch+cathode; +https://cathode.church/)
Apr 14 21:10:01 websrv2-hel1 /usr/local/bin/gotosocial[28944]: time=2022-04-14T21:10:01Z level=error msg=batch deliver had at least one failure: POST request to https://cathode.church/users/meena/inbox failed (401): 401 Unauthorized 
Apr 14 21:10:01 social gotosocial[28943]: time=2022-04-14T21:10:01Z level=error msg=batch deliver had at least one failure: POST request to https://cathode.church/users/meena/inbox failed (401): 401 Unauthorized

MDomain soc.eena.me auto

<VirtualHost *:80>
  ServerName soc.eena.me
  DocumentRoot "/var/empty"
</VirtualHost>

<VirtualHost *:443>
  SSLEngine On
  ServerName soc.eena.me
  RewriteEngine on
  RewriteCond %{HTTP:Connection} Upgrade [NC]
  RewriteCond %{HTTP:Upgrade} websocket [NC]
  RewriteRule / wss://social:8000//$1 [P,L]
  ProxyPassReverse / wss://social:8000/
  ProxyPass / http://social:8080/
  ProxyPassReverse / http://social:8080/
  RequestHeader set "X-Forwarded-Proto" expr=https

</VirtualHost>

log-level: "debug"
application-name: "Meena's Social"
host: "soc.eena.me"
account-domain: "soc.eena.me"
protocol: "https"
bind-address: "[::]"
port: 8080
trusted-proxies:
  - "192.162.17.1/24"
db-type: "sqlite"
db-address: "/var/db/gotosocial/db/sqlite.db"
web-template-base-dir: "/usr/local/www/gotosocial/template/"
web-asset-base-dir: "/usr/local/www/gotosocial/assets/"
accounts-registration-open: false
accounts-approval-required: true
accounts-reason-required: true
media-image-max-size: 2097152
media-video-max-size: 10485760
media-description-min-chars: 0
media-description-max-chars: 500
storage-backend: "local"
storage-local-base-path: "/var/db/gotosocial/storage"
statuses-max-chars: 5000
statuses-cw-max-chars: 100
statuses-poll-max-options: 6
statuses-poll-option-max-chars: 50
statuses-media-max-files: 6
letsencrypt-enabled: false
oidc-enabled: false
smtp-host: ""
syslog-enabled: true
syslog-protocol: ""
syslog-address: ""

my gotosocial version is: gotosocial version 0.2.3 d350087 2022-04-13T20:40:15Z [go1.18] (in fact it's https://github.com/superseriousbusiness/gotosocial/pull/449) running in 13.0-RELEASE-p11 on amd64

It's now a normal Go test and does not need a precompiled GtS binary nor go run as it did before.

But the most important change is it now being independent from the declaration order of Configuration fields, as we aren't comparing a JSON dump anymore but test for specific fields instead.

Describe the bug with a clear and concise description of what the bug is.

My media cache, on a single user instance, filled my VPS's system disk. I have media-remote-cache-days set to 3, but there were > 30,000 media items with ages much older than that.

What's your GoToSocial Version?

0.5.2 git-c31f219

GoToSocial Arch

x86_64 binary install

Browser version

No response

What happened?

It's possible I'm misunderstanding the caching settings, but my VPS ran out of disk yesterday. I had media-remote-cache-days set to 3, but my storage usage was > 5GB. Looking at the database, I could see> 30,000 items in media_attachments where remote_url was NULL and all *_at dates were more than 3 days old.

To get things back up and running, I wrote a perl script to remove files where remote_url was NULL and file_updated_at was more than 3 days ago and to set cached=0 for those items. That removed > 3GB of cached data and, so far as I can tell, GoToSocial is happy to re-acquire them if needed.

The logs do show that the prune function is running and removing files but, for some reason, it's not removing anything like as much as I'd expect.

For now, I've set media-remote-cache-days to zero and am sticking with the perl script (which GitHub is refusing to let me attach at the moment).

What you expected to happen?

Media shouldn't have remained cached.

How to reproduce it?

No response

Anything else we need to know?

No response

Since I'd really like to post some videos, I'm working on implementing this right now. I've created this issue as grounds for discussion on the implementation and feature set.

IMHO integrating with ffmpeg is required in some way as there are no standalone go libraries to work with various video formats.

Open Topics:

• [ ] How to interact with video data (executing ffmpeg/ffprobe vs binding to the library)
• [ ] Reencode them for optimizations?
• [ ] Strip metadata?

For small (and single-user) instances, it would alleviate the maintenance burden to allow running an instance using SQLite as RDBMS.

Would you consider it something possible for this project?

Describe the bug with a clear and concise description of what the bug is.

The profile screen has UX issues. See:

What's your GoToSocial Version?

v0.6.0-rc1

GoToSocial Arch

amd64

Browser version

No response

What happened?

Went to profile, saw screen as above

What you expected to happen?

I expected to see a profile screen that allowed me to upload avatar and banner images.

How to reproduce it?

Go to https://instance/user.

Anything else we need to know?

No response

First off, I don't usually submit issues, so apologies in advances for any breaches in etiquette.

My knowledge of Go is very limited, however, there seems to be some problem with the autocert part of the SSL initialization. I tried to get the certs myself and adjust the code accordingly, but there is something that I'm missing in the install/setup.

So is there a good way to troubleshoot the process that autocert goes through? Or any recommendations for inserting the certs manually?

Fixes https://github.com/superseriousbusiness/gotosocial/issues/1112

This is my first contribution to GoToSocial, let me know if I did anything wrong. And thanks!

Embed both assets and templates in the executable to simplify packaging and installation/updates even further. It still tries to read files from disk (for ease of customization), but falls back to embedded files.

TODO

The functionality as described is working. For a meaningful change work is still to be done:

• [x] implement hybrid FS
• [x] use hybrid FS for remaining asset types (default avatars)
• [x] change builds to not bundle assets
• [x] add cli to extract assets for customization (analogous to gitea embedded)
• [x] adapt documentation
• [ ] add e2e tests with different configurations (eg check landing page response on GTS_WEB_TEMPLATE_BASE_DIR=/dev/null ./gotosocial testrig start)

Description

If this is a code change, please include a summary of what you've coded, and link to the issue(s) it closes/implements.

If this is a documentation change, please briefly describe what you've changed and why.

This PR updates some of our documentation to acknowledge our successful NLnet funding application :)

Checklist

Please put an x inside each checkbox to indicate that you've read and followed it: [ ] -> [x]

If this is a documentation change, only the first checkbox must be filled (you can delete the others if you want).

• [x] I/we have read the GoToSocial contribution guidelines.

Description

If this is a code change, please include a summary of what you've coded, and link to the issue(s) it closes/implements.

If this is a documentation change, please briefly describe what you've changed and why.

Current blockquote margin + padding values were causing issues on small screens. This PR updates these values to use horizontal space a bit better!

Before:

After:

Checklist

Please put an x inside each checkbox to indicate that you've read and followed it: [ ] -> [x]

If this is a documentation change, only the first checkbox must be filled (you can delete the others if you want).

• [x] I/we have read the GoToSocial contribution guidelines.
• [ ] I/we have discussed the proposed changes already, either in an issue on the repository, or in the Matrix chat.
• [x] I/we have performed a self-review of added code.
• [x] I/we have written code that is legible and maintainable by others.
• [ ] I/we have commented the added code, particularly in hard-to-understand areas.
• [ ] I/we have made any necessary changes to documentation.
• [ ] I/we have added tests that cover new code.
• [ ] I/we have run tests and they pass locally with the changes.
• [x] I/we have run go fmt ./... and golangci-lint run.

Description

It was interesting to note that since presigned urls are used buckets dont need to be exposed publically. this was an interesting change compared to other mastodon specific s3 bucket guides hence documented here for correct directions.

Checklist

• [x] I/we have read the GoToSocial contribution guidelines.
• [x] I/we have discussed the proposed changes already, either in an issue on the repository, or in the Matrix chat.

Right now we have an http middleware which aborts incoming http requests that don't have any user-agent set on them, returning code 418 I'm A Teapot, in order to force http callers to at least provide some kind of identification in order to use the API (though this identification is ofc not reliable since it can be trivially spoofed).

However, since user-agent is not actually a required header, and only a 'should' (https://www.rfc-editor.org/rfc/rfc7231#section-5.5.3), then we ought to make this behavior configurable by the instance admin (see https://github.com/superseriousbusiness/gotosocial/issues/1292) to let them choose whether empty user agents get the teapot treatment.

Connected to this, we should expand the user-agent middleware to also allow admins to provide a list of regular expressions that will be evaluated against the incoming user-agent header string. This will be useful in filtering out unwanted scraping from bots with a predictable user-agent, which do not respect robots.txt or robots meta tags.

Could also be used by admins who want to completely limit interaction with other fedi softwares that use predictable user-agent strings.

Config key could be something like advanced-user-agent-filters, with the value as an array/slice of regex strings. The default value would, i guess, replicate the existing behavior (so just one entry, which matches empty strings)

Description

When uploading a custom emoji with transparency in png format, the static version of the emoji will lose transparency info (and actually will turn into black).

Example

Original file at https://example.ltd/fileserver/XXX/emoji/original/XXX.png looks like:

Processed static file at https://example.ltd/fileserver/XXX/emoji/static/XXX.png looks like:

Related code

I have done an experiment that confirms the code in internal/media/png-stripper.go will cause this issue.
However, limited by my ability, I cannot figure out how to fix the bug in that file currently.

And I am curious that, since png file can never be an animated image, why do we need to decode the png file and encode that back like the code below?
But I haven't deeply dived to the whole project yet, so I am not sure if there is something thoughtless in my idea. https://github.com/superseriousbusiness/gotosocial/blob/941893a774c83802afdc4cc76e1d30c59b6c5585/internal/media/image.go#L99-L125

Related issue(s) & PR(s)

https://github.com/superseriousbusiness/gotosocial/issues/425 https://github.com/superseriousbusiness/gotosocial/pull/487

Version

v0.6.0