Commit graph

1265 commits

Author SHA1 Message Date
tobi db59ca36b1
[chore] Remove unused "env" module () 2024-08-25 15:37:20 +00:00
tobi 2bd31ab710
[chore] Tiny tweak to ApprovedByURI () 2024-08-25 15:44:08 +02:00
tobi d3887bf6cc
[bugfix] Let prometheus client do its own compression handling () 2024-08-25 13:25:32 +02:00
tobi bef0dfc66c
[bugfix] Carry-over ApprovedByURI to avoid marking already-approved remote statuses as pending approval ()
* [bugfix] Carry-over pendingApproval + approvedByURI to avoid pending already-approved remote statuses

* don't carry over pending_approval, not necessary

* comment
2024-08-25 12:18:39 +02:00
tobi da3fa2d4a2
[chore/bugfix] Fix missing insertion of preapproved interaction requests () 2024-08-24 12:17:55 +00:00
tobi f23f04e0b1
[feature] Interaction requests client api + settings panel ()
* [feature] Interaction requests client api + settings panel

* test accept / reject

* fmt

* don't pin rejected interaction

* use single db model for interaction accept, reject, and request

* swaggor

* env sharting

* append errors

* remove ErrNoEntries checks

* change intReqID to reqID

* rename "pend" to "request"

* markIntsPending -> mark interactionsPending

* use log instead of returning error when rejecting interaction

* empty migration

* jolly renaming

* make interactionURI unique again

* swag grr

* remove unnecessary locks

* invalidate as last step
2024-08-24 11:49:37 +02:00
kim 8e5a72ac5c
[performance] ffmpeg ffprobe wrapper improvements ()
* use a single instance of wazero runtime and compiled modules

* remove test output 🤦

* undo process-{media,emoji} changes

* update test runner to include wazero compilation cache

* sign drone.yml

---------

Co-authored-by: tobi <tobi.smethurst@protonmail.com>
2024-08-23 17:15:35 +02:00
tobi 53fccb8af8
[feature] Use local_only field, deprecate federated field ()
* [feature] Use `local_only` field, deprecate `federated` field

* use `deprecated` comment for form.Federated

* nolint
2024-08-22 19:47:10 +02:00
kim 889d4756ea
[performance] use native Go code to probe JPEGs ()
* use native Go code to probe JPEGs

* add note about copying from github.com/disintegration/imaging

* add more code comments
2024-08-19 13:38:10 +00:00
kim e1154453bb
[chore] update default http client timeout to 30s () 2024-08-19 14:56:43 +02:00
kim ae14aa004d
[bugfix] permit unspecified orientation data () 2024-08-16 14:10:03 +02:00
kim 9cd27b412d
[security] harden account update logic ()
* on account update, ensure that public key has not changed

* change expected error message

* also support the case of changing account keys when expired (not waiting for handshake)

* tweak account update hardening logic, add tests for updating account with pubkey expired

* add check for whether incoming data was via federator, accepting keys if so

* use freshest window for federated account updates + comment about it
2024-08-13 15:37:09 +00:00
kim 5212a1057e
[bugfix] relax missing preferred_username, instead using webfingered username ()
* support no preferred_username, instead using webfingered username

* add tests for the new preferred_username behaviour
2024-08-13 09:01:50 +00:00
kim 4cb3e4d3e6
[bugfix] incorrect AP serialize function used serializing worker data () 2024-08-12 18:23:24 +02:00
kim 865b3aeaac
[bugfix] updated pinned counts on status delete ()
* include pinned status when incrementing / decrementing status counts

* remove the pinned increment on status creation

* code comments

* microoptimize decr
2024-08-11 11:23:36 +02:00
kim f77005128a
[performance] move thumbnail generation to go code where possible ()
* wrap thumbnailing code to handle generation natively where possible

* more code comments!

* add even more code comments!

* add code comments about blurhash generation

* maintain image rotation if contained in exif data

* move rotation before resizing

* ensure pix_fmt actually selected by ffprobe, check for alpha layer with gifs

* use linear instead of nearest-neighbour for resizing

* work with image "orientation" instead of "rotation". use default 75% quality for both webp and jpeg generation

* add header to new file

* use thumb extension when getting thumb mime type

* update test models and tests with new media processing

* add suggested code comments

* add note about thumbnail filter count reducing memory usage
2024-08-08 17:12:13 +00:00
John Winston 94c615d417
[feature] Add db-postgres-connection-string option ()
* handle db-url

* lint and add doc

* add more doc

* fix config test

* return error

* change name from db-url to db-postgres-connection-string
2024-08-08 14:00:19 +02:00
tobi b19cfee7ae
[feature] Use gifv type for short soundless mp4 videos () 2024-08-08 08:12:16 +00:00
Leah Neukirchen 4697271cef
[bugfix] send back Sec-Websocket-Protocol header for streaming WebSocket ()
* [bugfix] send back Sec-Websocket-Protocol header for streaming WebSocket

Chrome expects the selected Sec-Websocket-Protocol to be sent back
on the WebSocket upgrade request (RFC6455 1.9).

* fiddle a bit to avoid getting headers multiple times

* add some explanatory notes

---------

Co-authored-by: tobi <tobi.smethurst@protonmail.com>
2024-08-06 11:52:27 +02:00
kim b85a9983d0
[bugfix] fix emoji recaching operations ()
* add test for emoji update image

* update emoji recache to set the instance account id

* don't refresh emoji if only not cached. in that case literally just recache

* code comment

* rename + move a few things

* add some more code comments, and rename some functions to make logic a bit clearer

* remove unnecessary nil check (the value can be nil)

* comment wording

* remove test data output

* handle the case of caching an emoji which has been refreshed then uncached

* allow overwriting on testrig storage as we do now on regular storage

* fix emoji category ID not getting updated

---------

Co-authored-by: tobi <tobi.smethurst@protonmail.com>
2024-08-03 19:05:38 +02:00
tobi fa59c3713c
[chore] Add media-ffmpeg-pool-size config var () 2024-08-03 16:40:26 +02:00
Daenney 9b50151f17
[feature] Beef up our AI opt-outs ()
* [chore] Synchronise our robots.txt with upstream

* [feature] Add headers to escape AI crawlers

This adds 2 headers that a number of AI crawlers respect to signal that
content should not be included in their datasets.
2024-08-02 18:22:39 +02:00
kim e5e996b28a
[bugfix] close files before error return ()
* close files before error return

* use defer statements

* shuffle around some defers
2024-08-02 15:11:24 +01:00
kim 0f734a2410
[chore] move PopulateAccountStats() nil check often performed into function itself ()
* move PopulateAccountStats() nil check often performed into function itself

* fix test to take in mind we don't repopulate account stats if not-nil
2024-08-02 13:15:11 +01:00
kim 94e87610c4
[chore] add back exif-terminator and use only for jpeg,png,webp ()
* add back exif-terminator and use only for jpeg,png,webp

* fix arguments passed to terminateExif()

* pull in latest exif-terminator

* fix test

* update processed img

---------

Co-authored-by: tobi <tobi.smethurst@protonmail.com>
2024-08-02 12:46:41 +01:00
tobi 7b5917d6ae
[feature] Allow import of following and blocks via CSV ()
* [feature] Import follows + blocks via settings panel

* test import follows
2024-08-02 12:41:46 +01:00
tobi 697261da53
[chore] Take account of rotation data when calculating full size image dimensions ()
* [chore] Take account of rotation data when calculating full size image dimensions

* boobies
2024-07-31 18:43:39 +00:00
Vyr Cossont fd837776e2
[feature] Implement Mastodon-compatible roles ()
* Implement Mastodon-compatible roles

- `Account.role` should only be available through verify_credentials for checking current user's permissions
- `Account.role` now carries a Mastodon-compatible permissions bitmap and a marker for whether it should be shown to the public
- `Account.roles` added for *public* display roles (undocumented but stable since Mastodon 4.1)
- Web template now uses only public display roles (no user-visible change here, we already special-cased the `user` role)

* Handle verify_credentials case for default role

* Update JSON exact-match tests

* Address review comments

* Add blocks bit to admin permissions bitmap
2024-07-31 09:26:09 -07:00
tobi 38f041cea1
[feature] Allow users to export data via the settings panel ()
* [feature] Allow users to export data via the settings panel

* rename/move some stuff
2024-07-31 15:03:34 +01:00
CDN 43519324b3
[feature] Object store custom URL (S3) ()
* tweaks

* boobs

* fix variable name + typo

---------

Co-authored-by: tobi <tobi.smethurst@protonmail.com>
2024-07-31 13:44:18 +01:00
kim 0b4a951b48
[performance] limit ffprobe read time to at most 1s after start of file () 2024-07-30 16:18:29 +02:00
kim 87cff71af9
[feature] persist worker queues to db ()
* persist queued worker tasks to database on shutdown, fill worker queues from database on startup

* ensure the tasks are sorted by creation time before pushing them

* add migration to insert WorkerTask{} into database, add test for worker task persistence

* add test for recovering worker queues from database

* quick tweak

* whoops we ended up with double cleaner job scheduling

* insert each task separately, because bun is throwing some reflection error??

* add specific checking of cancelled worker contexts

* add http request signing to deliveries recovered from database

* add test for outgoing public key ID being correctly set on delivery

* replace select with Queue.PopCtx()

* get rid of loop now we don't use it

* remove field now we don't use it

* ensure that signing func is set

* header values weren't being copied over 🤦

* use ptr for httpclient.Request in delivery

* move worker queue filling to later in server init process

* fix rebase issues

* make logging less shouty

* use slices.Delete() instead of copying / reslicing

* have database return tasks in ascending order instead of sorting them

* add a 1 minute timeout to persisting worker queues
2024-07-30 13:58:31 +02:00
tobi 42932f9820
[chore] replace UniqueStrings with Deduplicate () 2024-07-30 12:20:06 +02:00
kim 47c26818d6
[chore] add some more slice related utility functions + remove duplicated functions () 2024-07-30 11:29:32 +02:00
Vyr Cossont a237e2b295
[feature] Implement following hashtags ()
* Implement followed tags API

* Insert statuses with followed tags into home timelines

* Test following and unfollowing tags

* Correct Swagger path params

* Trim conversation caches

* Migration for followed_tags table

* Followed tag caches and DB implementation

* Lint and tests

* Add missing tag info endpoint, reorganize tag API

* Unwrap boosts when timelining based on tags

* Apply visibility filters to tag followers

* Address review comments
2024-07-29 19:26:31 +01:00
kim 368c97f0f8
[bugfix] take into account rotation when generating thumbnail ()
* take into account rotation when generating thumbnail, simplify ffprobe output to show only fields we need

* only show rotation side data

* remove unnecessary comment

* fix code comments

* remove debug logging
2024-07-28 21:10:41 +02:00
kim 58f8082795
[bugfix] moves file rename to earlier in media pipeline so ffmpeg calls ALWAYS have extension () 2024-07-28 10:31:49 +02:00
tobi 87ee64afa0
[chore] log delivery worker stop/start at debug level () 2024-07-27 13:43:18 +02:00
kim ecc114fc00
[chore] replaces nested ifs with switch cases, removes defer 'onFail()' being passed to funcs () 2024-07-27 11:09:02 +02:00
kim d9e70b942f
ensure ffmpeg muxer gets set correctly by setting input file extension before clean () 2024-07-26 16:30:24 +01:00
tobi ecfea10e35
[bugfix] Use punycode for host part of resource query param when doing webfinger requests ()
* [bugfix] use punycode when webfingering

* account for punycode when checking if final URI matches expected

* hmm

* fix test
2024-07-26 13:11:07 +02:00
tobi 8ab2b19a94
[feature] Federate interaction policies + Accepts; enforce policies ()
* [feature] Federate interaction policies + Accepts; enforce policies

* use Acceptable type

* fix index

* remove appendIRIStrs

* add GetAccept federatingdb function

* lock on object IRI
2024-07-26 12:04:28 +02:00
tobi c9b6220fef
[chore] Add interaction filter to complement existing visibility filter ()
* [chore] Add interaction filter to complement existing visibility filter

* pass in ptr to visibility and interaction filters to Processor{} to ensure shared

* use int constants for for match type, cache db calls in filterctx

* function name typo 😇

---------

Co-authored-by: kim <grufwub@gmail.com>
2024-07-24 12:27:42 +01:00
kim 63fc9b6c3e
[chore] renames the GTS caches to DB caches ()
* renames the `GTS` caches to `DB` caches, as it better references what they are

* change remaining Caches.GTS uses to Caches.DB
2024-07-24 10:41:43 +02:00
Vyr Cossont db0a47126e
[bugfix] Serialize empty conversation account list as empty list, not null () 2024-07-24 10:40:01 +02:00
tobi 6533531cf1
[chore] Add some log lines to recent migrations warning not to interrupt ()
* [chore] Add some log lines to recent migrations warning not to interrupt

* arse
2024-07-23 23:06:20 +02:00
Vyr Cossont 86a59db711
Remove content and related fields from boosts ()
These duplicate the content of the target and aren't necessary for anything.

- Stops copying some fields from target when boosting or processing a remote boost
- Adds a migration to null out existing duplicate data
- Updates tests
2024-07-23 20:51:57 +01:00
Vyr Cossont 2d921d9d7c
Explicitly propagate filter results from statuses to their boosts in API responses ()
Related to 
2024-07-23 20:51:07 +01:00
Vyr Cossont 8fdd358f4b
[feature] Conversations API ()
* Implement conversations API

* Sort and page conversations by last status ID

* Appease linter

* Fix deleting conversations and statuses

* Refactor to make migrations automatic

* Lint

* Update tests post-merge

* Fixes from live-fire testing

* Linter caught a format problem

* Refactor tests, fix cache

* Negative test for non-DMs

* Run conversations advanced migration on testrig startup as well as regular server startup

* Document (lack of) side effects of API method for deleting a conversation

* Make not-found check less nested for readability

* Rename PutConversation to UpsertConversation

* Use util.Ptr instead of IIFE

* Reduce cache used by conversations

* Remove unnecessary TableExpr/ColumnExpr

* Use struct tags for both unique constraints on Conversation

* Make it clear how paging with GetDirectStatusIDsBatch should be used

* Let conversation paging skip conversations it can't render

* Use Bun NewDropTable

* Convert delete raw query to Bun

* Convert update raw query to Bun

* Convert latestConversationStatusesTempTable raw query partially to Bun

* Convert conversationStatusesTempTable raw query partially to Bun

* Rename field used to store result of MaxDirectStatusID

* Move advanced migrations to their own tiny processor

* Catch up util function name with main

* Remove json.… wrappers

* Remove redundant check

* Combine error checks

* Replace map with slice of structs

* Address processor/type converter comments

- Add context info for errors
- Extract some common processor code into shared methods
- Move conversation eligibility check ahead of populating conversation

* Add error context when dropping temp tables
2024-07-23 20:44:31 +01:00
kim 31294f7c78
[bugfix] media.Processor{}.GetFile() returning 404s on first call, correctly loading on 2nd ()
* refactor file handling a tiny bit

* whoops

* make processing media / emoji defers a bit clear to see that it's the "on finished processing" path

* some wording

* add some debug logging

* add mutex locks for processing remote media

* try removing freshness check

* fix derefMedia not being allocated

* fix log format string

* handle case of empty file paths (i.e. not stored)

* remove media / emoji once finished processing from dereferencer maps

* whoops, fix the cached / force checks

* move url parsing outside of 'process___Safely()' funcs to prevalidate url

* use emoji.ShortcodeDomain()

* update RefreshEmoji() to also match RefreshMedia() changes

---------

Co-authored-by: tobi <tobi.smethurst@protonmail.com>
2024-07-22 18:45:48 +01:00
kim c635080a0b
[feature] add flac support ()
* add flac support to the ffprobe format/stream -> filetype parser

* also add audio/flac for flac (not just x-flac)

* update tests

---------

Co-authored-by: tobi <tobi.smethurst@protonmail.com>
2024-07-21 14:42:51 +02:00
tobi 027a93facc
[feature/frontend] Respect prefers-reduced-motion for avatars, headers, and emojis ()
* [feature/frontend] Respect `prefers-reduced-motion` for avatars, headers, and emojis

* go fmt

* fix tests

* use static version of instance thumbnail when appropriate

* use prefers-reduced-motion

* simplify account conversion a bit

* fix c&p error
2024-07-21 14:22:08 +02:00
kim b415337d40
[bugfix] update common get target account / status doing refresh async () 2024-07-21 14:04:19 +02:00
tobi 8a2fb48ef4
[chore] Remove duplicate <hr> ()
* [chore] Remove duplicate `<hr>`

* update test
2024-07-21 11:30:22 +02:00
kim 409b3987ac
update go-ffmpreg to v0.2.3 () 2024-07-20 14:14:58 +01:00
tobi ad700c5029
[chore/frontend] Use smaller webp images for logo and default avatars () 2024-07-20 14:02:22 +01:00
kim 50c9b5498b
[feature] use webp for thumbnails ()
* update to use webp for thumbnails

* bump webp quality up to 40% from 12% (it's a bit different to jpeg quality setting)

* update to use yuva colorspace, and use thumbnail=n=10 to select frame

* fix missing comma in ffmpeg args

* add links to appropriate ffmpeg docs

* update tests

* add file size tests for thumbnails

---------

Co-authored-by: tobi <tobi.smethurst@protonmail.com>
2024-07-19 17:28:43 +02:00
tobi 2b895f822a
[chore] Fix typo () 2024-07-17 22:04:52 +02:00
kim 72ba5666a6
[chore] media pipeline improvements ()
* don't set emoji / media image paths on failed download, migrate FileType from string to integer

* fix incorrect uses of util.PtrOr, fix returned frontend media

* fix migration not setting arguments correctly in where clause

* fix not providing default with not null column

* whoops

* ensure a default gets set for media attachment file type

* remove the exclusive flag from writing files in disk storage

* rename PtrOr -> PtrOrZero, and rename PtrValueOr -> PtrOrValue to match

* slight wording changes

* use singular / plural word forms (no parentheses), is better for screen readers

* update testmodels with unknown media type to have unset file details, update attachment focus handling converting to frontend, update tests

* store first instance in ffmpeg wasm pool, fill remaining with closed instances
2024-07-17 15:26:33 +00:00
tobi 0aadc2db2a
[feature] Allow users to set default interaction policies per status visibility ()
* [feature] Allow users to set default interaction policies

* use vars for default policies

* avoid some code repetition

* unfuck form binding

* avoid bonkers loop

* beep boop

* put policyValsToAPIPolicyVals in separate function

* don't bother with slices.Grow

* oops
2024-07-17 16:46:52 +02:00
kim 401098191b
give read-only access to /dev for ffmpeg to access /dev/urandom () 2024-07-16 13:32:48 +01:00
kim de45c0be60
[feature] more filetype support! ()
* add more supported file types to our media processor that ffmpeg supports, update supported mime type lists

* add code comments to the supported mime types slice

* don't check for zero value string, just parse

* remove some unneeded consts which make the code a bit harder to read

* fix test expected instance media mime types, use compact ffprobe json, simple media processing by type

* final tweaks to media processing code

* don't use safe divide where we don't need to
2024-07-15 15:24:53 +01:00
tobi 9efb11d848
[feature/frontend] Add player for audio files; use thumbnail for poster ()
* [feature/frontend] Audio player for audio media types

* use video preview images for previews instead of video itself

* don't preload

* update tests for new zork status

* collapse media gallery into single row when small
2024-07-15 10:47:57 +01:00
tobi 9adcce9494
[chore] add step-by-step logging to long migration () 2024-07-15 11:12:56 +02:00
tobi c83e96b8a7
[chore/frontend] Tweak threading a bit, inform about hidden replies ()
* [chore/frontend] Tweak threading a bit, inform about hidden replies

* whoops

* round off bottom of replies col-header if no replies visible
2024-07-13 12:26:16 +02:00
tobi bbbdf01213
[bugfix] Fix panic in isStatusHomeTimelineable () 2024-07-13 11:07:01 +02:00
tobi aeb65bceae
[feature/frontend] Better visual separation between "main" thread and "replies" ()
* [feature/frontend] Better web threading model

* fix test

* bwap

* tweaks

* more tweaks to wording

* typo

* indenting

* adjust wording

* aaa
2024-07-12 20:36:03 +02:00
kim cde2fb6244
[feature] support processing of (many) more media types ()
* initial work replacing our media decoding / encoding pipeline with ffprobe + ffmpeg

* specify the video codec to use when generating static image from emoji

* update go-storage library (fixes incompatibility after updating go-iotools)

* maintain image aspect ratio when generating a thumbnail for it

* update readme to show go-ffmpreg

* fix a bunch of media tests, move filesize checking to callers of media manager for more flexibility

* remove extra debug from error message

* fix up incorrect function signatures

* update PutFile to just use regular file copy, as changes are file is on separate partition

* fix remaining tests, remove some unneeded tests now we're working with ffmpeg/ffprobe

* update more tests, add more code comments

* add utilities to generate processed emoji / media outputs

* fix remaining tests

* add test for opus media file, add license header to utility cmds

* limit the number of concurrently available ffmpeg / ffprobe instances

* reduce number of instances

* further reduce number of instances

* fix envparsing test with configuration variables

* update docs and configuration with new media-{local,remote}-max-size variables
2024-07-12 09:39:47 +00:00
tobi 5bc567196b
[chore] Add interaction policy gtsmodels ()
* [chore] introduce interaction policy gts models

* update migration a smidge

* fix copy paste typo

* update migration

* use int for InteractionType
2024-07-11 15:44:29 +01:00
snow 8f8093aea4
[bugfix] Don't throw error when parent statuses are missing () ()
* [bugfix] Don't throw error when parent statuses are missing ()

* Split missing parent status case from error check
2024-07-10 14:38:25 +01:00
Daenney ad93e57d08
[choore] Update robots.txt ()
Recategorises a pair of scrapers according to their use.
2024-07-10 15:10:34 +02:00
tobi d70f4e166d
[feature/frontend] Allow setting alt-text for avatar + header () 2024-07-08 15:47:03 +02:00
Vyr Cossont be5e532cd2
[bugfix] Handle ErrHideStatus when preparing timeline statuses () 2024-07-04 19:29:28 -07:00
Daenney 02d6e2e3bc
[feature] Set some security related headers ()
* Set frame-ancestors in the CSP
   This ensures we can't be loaded/embedded in an iframe. It also sets the
   older X-Frame-Options for fallback.
* Disable MIME type sniffing
* Set Referrer-Policy
   This sets the policy such that browsers will never send the Referer
   header along with a request, unless it's a request to the same protocol,
   host/domain and port. Basically, only send it when navigating through
   our own UI, but not anything external.

   The default is strict-origin-when-cross-origin when unset, which sends
   the Referer header for requests unless it's going from HTTPS to HTTP
   (i.e a security downgrade, hence the 'strict').
2024-07-04 10:07:02 +02:00
Vyr Cossont fde0c6bc8c
[chore] Allow gtsmodel to depend on util ()
Convert one free function into a gtsmodel.Emoji method so that util does not depend on gtsmodel and can be used from it in the future
2024-07-03 15:53:54 -07:00
CDN 235bea04fd
[bugfix] update SignupURL routing in email template. () 2024-06-30 01:08:13 +02:00
Daenney 137ef5a9ff
[feature] Default to WASM-based SQLite driver ()
* [feature] Default to WASM-based SQLite driver

With 0.16 out this switches our default SQLite driver to the WASM-based
solution instead. So far the driver seems to perform just as well.
Switching our default should result in it getting a bit more testing
during the 0.17 development cycle.

* add the ol' john hancock

---------

Co-authored-by: tobi <tobi.smethurst@protonmail.com>
2024-06-29 09:35:57 +02:00
tobi 86786ae5b3
[bugfix] Fix typo in instance cache copyF () 2024-06-28 16:43:00 +02:00
kim 21bb324156
[chore] media and emoji refactoring ()
* start updating media manager interface ready for storing attachments / emoji right away

* store emoji and media as uncached immediately, then (re-)cache on Processing{}.Load()

* remove now unused media workers

* fix tests and issues

* fix another test!

* fix emoji activitypub uri setting behaviour, fix remainder of test compilation issues

* fix more tests

* fix (most of) remaining tests, add debouncing to repeatedly failing media / emojis

* whoops, rebase issue

* remove kim's whacky experiments

* do some reshuffling, ensure emoji uri gets set

* ensure marked as not cached on cleanup

* tweaks to media / emoji processing to handle context canceled better

* ensure newly fetched emojis actually get set in returned slice

* use different varnames to be a bit more obvious

* move emoji refresh rate limiting to dereferencer

* add exported dereferencer functions for remote media, use these for recaching in processor

* add check for nil attachment in updateAttachment()

* remove unused emoji and media fields + columns

* see previous commit

* fix old migrations expecting image_updated_at to exists (from copies of old models)

* remove freshness checking code (seems to be broken...)

* fix error arg causing nil ptr exception

* finish documentating functions with comments, slight tweaks to media / emoji deref error logic

* remove some extra unneeded boolean checking

* finish writing documentation (code comments) for exported media manager methods

* undo changes to migration snapshot gtsmodels, updated failing migration to have its own snapshot

* move doesColumnExist() to util.go in migrations package
2024-06-26 16:01:16 +01:00
dependabot[bot] 03c5d016a7
[chore]: Bump github.com/gorilla/feeds from 1.1.2 to 1.2.0 () 2024-06-24 09:04:27 +00:00
Daenney 4604224c4d
[chore] Update our robots.txt ()
This syncs our copy with the current state of the ai.robots.txt
repository. Upstream has tightened their scope to be AI-only, whereas
before it included a bunch of SEO and "web intelligence" marketing
stuff. I've kept those but moved them into their own section.
2024-06-23 15:34:21 +02:00
Julian c2738474d5
[bugfix] add Date and Message-ID headers for email ()
* [bugfix] add Date and Message-ID headers for email

This should make spam filters more happy, as most of them grant some
negative score for not having those headers. Also the Date is convenient
for the user receiving the mail.

* make golangci-lint happy
2024-06-22 23:36:30 +02:00
kim 7b1ccbd65a
[feature] add worker task serialization logic ()
* improved server shutdown with more precise shutdown of modules + deferring of ALL of it

* move delivery and workers into separate files

* add worker task model and Serialize() / Deserialize() methods for message types

* start adding message serialize / deserialize tests

* start adding test cases

* update body rewinding to rely on standard library mechanism of r.GetBody()

* remove request rewinding (http.Client{} should already handle this)

* standard library already handles rewinding

* improved code comment

* move the newPOST() function contents to prepare(), fits better with current API

* add Serialize() / Deserialize() implementations for Delivery{} type

* finish writing FromClientAPI sserialize / deserialize tests

* start adding FromFediAPI{} serialize / deserialize test cases

* fix FromFediAPI{} tests

* add tests for delivery message type

* fix repeat code

* missing license header

* use testrig status and accounts for marshal / unmarshaling tests

* add a specific test for checking account RSA keys are preserved
2024-06-20 17:06:20 +00:00
tobi 4ce5c37df5
[bugfix] rename include_types[] to types[] () 2024-06-18 18:18:35 +02:00
tobi d2b3d37724
[feature/frontend] Reports frontend v2 ()
* use apiutil + paging in admin processor+handlers

* we're making it happen

* fix little whoopsie

* styling for report list

* don't youuuu forget about meee don't don't don't don't

* last bits

* sanitize content before showing in report statuses

* update report docs
2024-06-18 18:18:00 +02:00
Vyr Cossont b08c1bd0cb
[feature] Implement types[] param for notifications ()
Counterpart of exclude_types[].

Also updates Swagger spec for types[] to use the correct param name and enumerate possible values.

Fixes 
2024-06-17 20:50:50 +01:00
tobi db803617db
[bugfix] avoid v. long notification clear query () 2024-06-14 10:14:55 +00:00
Vyr Cossont b789fe2bc7
[feature] filter API v2: Restore keywords_attributes and statuses_attributes ()
These filter API v2 features were cut late in development because the form encoding version is hard to implement correctly and because I thought no clients actually used `keywords_attributes`. Unfortunately, Phanpy does use `keywords_attributes`.
2024-06-14 10:11:41 +02:00
tobi 9d9013db4c
[chore] Update move test () 2024-06-13 15:08:43 +02:00
tobi fbdfce3190
[chore] Don't render sign-up form when registrations closed () 2024-06-12 15:00:21 +01:00
tobi 20877c1474
[bugfix] Ensure side effects handled for local unlocked follows () 2024-06-12 13:23:10 +01:00
Vyr Cossont dcf825bd87
[bugfix] Correct Swagger path for poll voting ()
Updates the Swagger spec to match the router.

Fixes 
2024-06-12 11:16:46 +02:00
tobi 611f9de39b
[bugfix] Deref stats async, serve stub collections if handshaking ()
* [bugfix] Deref stats async, allow peek if handshaking

* don't return totalItems when handshaking or hiding collections

* use GetLimit()

* use StubAccountStats
2024-06-11 11:54:59 +02:00
kim fd6637df4a
[bugfix] boost and account recursion ()
* fix possible infinite recursion if moved accounts are self-referential

* adds a defensive check for a boost being a boost of a boost wrapper

* add checks on input for a boost of a boost

* remove unnecessary check

* add protections on account move to prevent move recursion loops

* separate status conversion without boost logic into separate function to remove risk of recursion

* move boost check to boost function itself

* formatting

* use error 422 instead of 500

* use gtserror not standard errors package for error creation
2024-06-10 19:42:41 +01:00
tobi ebdcb00d0a
[chore] Roll back use of (created) pseudo-header pending () 2024-06-10 19:42:26 +01:00
tobi ce3b8aacf7
[chore] Warn about email/password change when using OIDC ()
* [chore] Warn about email/password change when using OIDC

* go fmt
2024-06-07 15:21:57 +01:00
Vyr Cossont afcfa48a7d
[feature] Implement filters_changed stream event () 2024-06-07 10:51:13 +02:00
Vyr Cossont e605788b4b
[bugfix] Filter fixes ()
* Add Filter.Expired convenience method, consistent with mutes

* Add missing Swagger for filter keyword/status delete and for 403s on moved accounts
2024-06-06 20:16:20 +02:00
Vyr Cossont 5e2d4fdb19
[feature] User muting ()
* User muting

* Address review feedback

* Rename uniqueness constraint on user_mutes to match convention

* Remove unused account_id from where clause

* Add UserMute to NewTestDB

* Update test/envparsing.sh with new and fixed cache stuff

* Address tobi's review comments

* Make compiledUserMuteListEntry.expired consistent with UserMute.Expired

* Make sure mute_expires_at is serialized as an explicit null for indefinite mutes

---------

Co-authored-by: tobi <tobi.smethurst@protonmail.com>
2024-06-06 16:38:02 +00:00
kim b371c2db47
[bugfix] update media if more than just url changes ()
* refactor status media handling into separate functions, handle case of changed metadata

* update fetchRemoteAccount{Avatar,Header} to use new refactored {load,update}Attachment() functions

* whoops, nearly marked avatars as headers :')

* reformatting to improve legibility
2024-06-06 15:35:50 +01:00
tobi bcda048eab
[feature] Self-serve email change for users ()
* [feature] Email change

* frontend stuff for changing email

* docs

* tests etc

* differentiate more clearly between local user+account and account

* populate user
2024-06-06 14:43:25 +01:00
tobi 131020faeb
drop date () 2024-06-06 12:50:56 +01:00
kim 5dcc954072
[feature] do not uncache status / emoji media if attached status is bookmarked ()
* do not uncache status / emoji media if attached status is bookmarked

* add status bookmark and bookmark IDs caches

* update status bookmark tests

* move IsStatusBookmarkedBy() to StatusBookmark{} interface, rely on cache

* fix envparsing.sh test
2024-06-06 11:44:43 +01:00
tobi 6f26b32ec3
[bugfix] Don't nil emojis + fields on blocked accounts ()
* [bugfix] Don't nil emojis + fields on blocked accounts

* comment

* swagger
2024-06-06 11:22:16 +01:00
kim 3b7faac604
[bugfix] concurrent map writes in dereferencer media processing maps ()
* removes the avatar / header deref maps as we now have per-uri status / account locks, adds retries on data-races, adds separate emoji map mutex

* work with a copy of account / status for each retry loop

* revert to old data race behaviour, it gets too complicated otherwise

---------

Co-authored-by: tobi <tobi.smethurst@protonmail.com>
2024-06-06 10:50:14 +02:00
tobi 9caf29bed2
[chore] Fiddle with CI tests; use wasmsqlite3 for CI tests ()
* [chore] Update CI test stuff

* don't bother with ldflags

* use wasmsqlite3
2024-06-05 21:40:08 +02:00
Vyr Cossont 45fe295caa
Compile filter keyword regexps when touched through PutFilter or UpdateFilter ()
Followup to 
2024-06-04 10:33:39 +01:00
kim f17dd62ff5
[feature] log worker startup counts ()
* log number of each worker kinds started, and log when stopped

* remove worker debug logging

* whoops, fix the count of media workers
2024-06-03 22:55:50 +01:00
tobi d79c2f2688
[feature/frontend] Add debug sections to settings panel ()
* [feature/frontend] Add debug sections to settings panel

* max-width

* swagger
2024-06-03 11:20:53 +02:00
Vyr Cossont 04bcde08a1
[feature] Add from: search operator and account_id query param ()
* Add from: search operator

* Fix whitespace in Swagger YAML comment

* Move query parsing into its own method

* Document search

* Clarify post search scope
2024-05-31 12:57:42 +02:00
Vyr Cossont 61a8d36255
[feature] Implement Filter API v2 ()
* Use correct entity name

* We support server-side filters now

* Document filter v1 methods that can throw a 409

* Validate v1 filter phrase as filter title

* Always check v1 filter API status codes in tests

* Document keyword minimum requirement on filter API v1

* Make it possible to specify filter keyword update columns per filter keyword

* Implement v2 filter API

* Fix lint and tests

* Update Swagger spec

* Fix filter update test

* Update Swagger spec *correctly*

* Update actual files Swagger spec was generated from

* Remove keywords_attributes and statuses_attributes

* Add test for serialization of empty filter

* More helpful messages when object is owned by wrong account
2024-05-31 12:55:56 +02:00
Daenney 2fd69ec58b
[chore] Make worker run messages debug output ()
On startup and shutdown of a worker, we log a message of the worker
being started together with a textual representation of a memory
address. Though this can be handy for developers to debug
startup/shutdown sequencing issues of the workers, it's typically not
very useful or informative for an admin. We can also output a lot of
these (on my system I get 265 lines of these during startup).

This changes the messages from Info to Debug, to not print them under
normal circumstances.
2024-05-30 11:00:47 +02:00
kim 32e570abfd
[chore] improved startup / shutdown ()
* improved server shutdown with more precise shutdown of modules + deferring of ALL of it

* make the same changes to the testrig server

* use testrig specific func

* update variable name to fix nilptr

* fix removal of setting db on state
2024-05-29 13:21:04 +02:00
Vyr Cossont 975e92b7f1
[feature] Implement profile API ()
* Implement profile API

This Mastodon 4.2 extension provides capabilities missing from the existing Mastodon account update API: deleting an account's avatar or header.

See: https://docs.joinmastodon.org/methods/profile/

* Move profile media methods to media processor

* Remove check for moved account
2024-05-29 12:57:44 +02:00
tobi f9a4a6120d
[feature] Debug admin endpoint to clear caches ()
* [feature] Debug admin endpoint to clear caches

* go fmt
2024-05-29 12:56:17 +02:00
tobi fa9a3075a5
[chore/bugfix] Don't cache MovedTo account () 2024-05-28 15:39:45 +02:00
tobi a276b1ca06
[feature/frontend] Let admins send test email to validate SMTP config ()
* [feature/frontend] Let admins send test email to validate SMTP config

* wee
2024-05-27 17:03:54 +00:00
kim 1e7b32490d
[experiment] add alternative wasm sqlite3 implementation available via build-tag ()
This allows for building GoToSocial with [SQLite transpiled to WASM](https://github.com/ncruces/go-sqlite3) and accessed through [Wazero](https://wazero.io/).
2024-05-27 17:46:15 +02:00
tobi 5bee30d60c
[chore] Fix report email link () 2024-05-27 12:27:49 +02:00
kim 3d3e99ae52
[performance] update storage backend and make use of seek syscall when available ()
* update to use go-storage/ instead of go-store/v2/storage/

* pull in latest version from codeberg

* remove test output 😇

* add code comments

* set the exclusive bit when creating new files in disk config

* bump to actual release version

* bump to v0.1.1 (tis a simple no-logic change)

* update readme

* only use a temporary read seeker when decoding video if required (should only be S3 now)

* use fastcopy library to use memory pooled buffers when calling TempFileSeeker()

* update to use seek call in serveFileRange()
2024-05-22 11:46:24 +02:00
kim b092da6d28
[performance] cache v2 filter keyword regular expressions ()
* add caching of filterkeyword regular expressions

* formatting

* fix WholeWord nil check
2024-05-21 14:20:19 +01:00
kim c06e6fb656
[performance] update go-structr and go-mutexes with memory usage improvements ()
* update go-structr and go-mutexes with memory usage improvements

* bump to go-structr v0.8.4
2024-05-13 08:05:46 +00:00
kim f456bd3401
update the total ratios calculation to include ALL caches (previously was missing a few!) () 2024-05-06 22:29:31 +01:00
Vyr Cossont 45f4afe60e
feature: filters v2 server-side warning/hiding ()
* Remove dead code

* Filter statuses when converting to frontend representation

* status.filtered is an array

* Make matching case-insensitive

* Remove TODOs that don't need to be done now

* Add missing filter check for notification

* lint: rename ErrHideStatus

* APIFilterActionToFilterAction not used yet

* swaggerino docseroni

* Address review comments

* Add apimodel.FilterActionNone

---------

Co-authored-by: tobi <31960611+tsmethurst@users.noreply.github.com>
Co-authored-by: tobi <tobi.smethurst@protonmail.com>
2024-05-06 12:49:08 +01:00
kim d3f6960ba0
close + drain body if response body is too large () 2024-05-05 16:43:38 +01:00
tobi 6171dcbe51
[feature] Add HTTP header permission section to frontend ()
* [feature] Add HTTP header filter section to frontend

* tweak naming a bit
2024-05-05 11:47:22 +00:00
kim a840f4d49d
add missing caches to the main cache sweep command () 2024-05-02 14:09:59 +01:00
tobi ebec95a522
[bugfix] Lock when checking/creating notifs to avoid race ()
* [bugfix] Lock when checking/creating notifs to avoid race

* test notif spam
2024-05-02 13:43:00 +01:00
tobi 725a21b027
[feature] Page through accounts as moderator ()
* [feature] Page through accounts as moderator

* aaaaa

* use COLLATE "C" for Postgres to ensure same ordering as SQLite

* fix typo, test paging up

* don't show moderation / info for our instance acct
2024-05-01 14:11:22 +01:00
kim 2300d5e73b
[bugfix] function queue memory pools limitlessly grow ()
* updates the simple queue memory pool to actually self-clean + limit growth

* update memory pool cleaning frequency
2024-05-01 13:30:43 +02:00
kim eb61c783ed
[bugfix] flaky paging test () 2024-05-01 13:29:42 +02:00
kim a8254a40e7
[bugfix] further paging mishaps ()
* FURTHER paging shenanigans 🥲

* remove cursor logic from ToLinkURL()

* fix up paging tests

---------

Co-authored-by: tobi <tobi.smethurst@protonmail.com>
2024-04-30 16:22:23 +02:00
kim ec7c983e46
[bugfix] retry on http 500 errors *inclusive* () 2024-04-30 16:18:32 +02:00
kim ec334ece20
[chore] include attemptno in httpclient logs ()
* include request attempt number in httpclient logs

* slightly nicer attempt number formatting
2024-04-30 16:15:50 +02:00
kim 4f87ef246c
[bugfix] paging rel links ()
* fix paging so it uses correct cursor query parameter name

* improved code comment

* whoops, flip the cursoring 🤦

* fix the broken test
2024-04-30 12:19:33 +02:00
tobi 40ece19055
[chore] Fix conflict in workers tests ()
* [chore] Fix conflict in workers tests

* commenty-wenty
2024-04-29 11:43:18 +02:00
kim 48b91ca239
[bugfix] Fix error string typo () 2024-04-26 15:20:56 +02:00
kim c9c0773f2c
[performance] update remaining worker pools to use queues ()
* start replacing client + federator + media workers with new worker + queue types

* refactor federatingDB.Delete(), drop queued messages when deleting account / status

* move all queue purging to the processor workers

* undo toolchain updates

* code comments, ensure dereferencer worker pool gets started

* update gruf libraries in readme

* start the job scheduler separately to the worker pools

* reshuffle ordering or server.go + remove duplicate worker start / stop

* update go-list version

* fix vendoring

* move queue invalidation to before wipeing / deletion, to ensure queued work not dropped

* add logging to worker processing functions in testrig, don't start workers in unexpected places

* update go-structr to add (+then rely on) QueueCtx{} type

* ensure more worker pools get started properly in tests

* fix remaining broken tests relying on worker queue logic

* fix account test suite queue popping logic, ensure noop workers do not pull from queue

* move back accidentally shuffled account deletion order

* ensure error (non nil!!) gets passed in refactored federatingDB{}.Delete()

* silently drop deletes from accounts not permitted to

* don't warn log on forwarded deletes

* make if else clauses easier to parse

* use getFederatorMsg()

* improved code comment

* improved code comment re: requesting account delete checks

* remove boolean result from worker start / stop since false = already running or already stopped

* remove optional passed-in http.client

* remove worker starting from the admin CLI commands (we don't need to handle side-effects)

* update prune cli to start scheduler but not all of the workers

* fix rebase issues

* remove redundant return statements

* i'm sorry sir linter
2024-04-26 13:50:46 +01:00
Daenney 3a369d834a
[chore] Upgrade our Go version to 1.22 ()
* [chore] Upgrade our Go version to 1.22

With Go 1.22 having been released at the start of February, it's now
been a few months. No major issues have shown up, and the two point
release since then have primarily been security fixes plus some general
bug fixing.

This sets the required Go version to 1.22, as there's nothing in 1.22.1
or 1.22.2 that we would explicitly require. It sets the toolchain to the
latest point release, to ensure we pick up any fixes from there when
building releases etc.

* [chore] Update CI to Go 1.22

* [chore] Update golangci-lint to 1.25.7

Newer version should know about Go 1.22 and run fine.

* [chore] Update Docker container to Go 1.22

* [chore] Update Dockerfile to newer Alpine version

* sign drone.yml

* add missing license header

---------

Co-authored-by: tobi <tobi.smethurst@protonmail.com>
2024-04-26 10:40:29 +02:00
Daenney dcab555a6b
[chore] Update robots.txt ()
This updates the robots.txt based on the list of the ai.robots.txt
repository. We can look at automating that at some point.

It's worth pointing out that some robots, namely the ones by Bytedance,
are known to ignore robots.txt entirely.
2024-04-22 11:01:37 +02:00
tobi b7c629a18a
[bugfix] Fix incorrect field name for status source, add helpful message ()
* [bugfix] Fix incorrect field name for status source, add helpful message

* swagger

* yyammm
2024-04-18 13:22:55 +02:00
tobi 431505b3e4
[feature] Stub conversations endpoint () 2024-04-18 12:36:02 +02:00
tobi 34d0159ad5
[feature] Stub account mutes endpoint ()
* [feature] Stub account mutes endpoint

* swagger? i barely know 'er!
2024-04-18 09:59:47 +01:00
kim c67bbe5ba0
update to set requesting account when deleting status () 2024-04-17 16:54:40 +01:00
tobi cef9924d9a
[feature] Status source endpoint ()
* [feature] statusSource endpoint

* finish up
2024-04-17 13:49:20 +01:00
tobi ef16919d4a
[feature] Stub status history endpoint () 2024-04-17 13:06:49 +01:00
tobi 8cf685fbe9
[bugfix] Fix minor API issue w/ boosted statuses () 2024-04-17 12:41:40 +02:00
tobi 6de5717d7f
[bugfix] fix get all tokens () 2024-04-16 13:25:47 +02:00
tobi 3cceed11b2
[feature/performance] Store account stats in separate table ()
* [feature/performance] Store account stats in separate table, get stats from remote

* test account stats

* add some missing increment / decrement calls

* change stats function signatures

* rejig logging a bit

* use lock when updating stats
2024-04-16 13:10:13 +02:00
kim f79d50b9b2
[performance] cached oauth database types ()
* update token + client code to use struct caches

* add code comments

* slight tweak to default mem ratios

* fix envparsing

* add appropriate invalidate hooks

* update the tokenstore sweeping function to rely on caches

* update to use PutClient()

* add ClientID to list of token struct indices
2024-04-15 14:22:21 +01:00