* improvements to caching for lists and relationship to accounts / follows
* fix nil panic in AddToList()
* ensure list related caches are correctly invalidated
* ensure returned ID lists are ordered correctly
* bump go-structr to v0.8.9 (returns early if zero uncached keys to be loaded)
* remove zero checks in uncached key load functions (go-structr now handles this)
* fix issues after rebase on upstream/main
* update the expected return order of CSV exports (since list entries are now down by entry creation date)
* rename some funcs, allow deleting list entries for multiple follow IDs at a time, fix up more tests
* use returning statements on delete to get cache invalidation info
* fixes to recent database delete changes
* fix broken list entries delete sql
* remove unused db function
* update remainder of delete functions to behave in similar way, some other small tweaks
* fix delete user sql, allow returning on err no entries
* uncomment + fix list database tests
* update remaining list tests
* update envparsing test
* add comments to each specific key being invalidated
* add more cache invalidation explanatory comments
* whoops; actually delete poll votes from database in the DeletePollByID() func
* remove added but-commented-out field
* improved comment regarding paging being disabled
* make cache invalidation comments match what's actually happening
* fix up delete query comments to match what is happening
* rename function to read a bit better
* don't use ErrNoEntries on delete when not needed (it's only needed for a RETURNING call)
* update function name in test
* move list exclusivity check to AFTER eligibility check. use log.Panic() instead of panic()
* use the poll_id column in poll_votes for selecting votes in poll ID
* fix function name
* [feature] Process `Reject` of interaction via fedi API, put rejected statuses in the "sin bin"
* update test
* move nil check back to `rejectStatusIRI`
* [feature/frontend] Add options to include Unlisted posts or hide all posts
* finish up
* swagger
* move invalidate call into bundb package, avoid invalidating if not necessary
* rename show_web_statuses => web_visibility
* don't use ptr for webvisibility
* last bits
* on account update, ensure that public key has not changed
* change expected error message
* also support the case of changing account keys when expired (not waiting for handshake)
* tweak account update hardening logic, add tests for updating account with pubkey expired
* add check for whether incoming data was via federator, accepting keys if so
* use freshest window for federated account updates + comment about it
* include pinned status when incrementing / decrementing status counts
* remove the pinned increment on status creation
* code comments
* microoptimize decr
* wrap thumbnailing code to handle generation natively where possible
* more code comments!
* add even more code comments!
* add code comments about blurhash generation
* maintain image rotation if contained in exif data
* move rotation before resizing
* ensure pix_fmt actually selected by ffprobe, check for alpha layer with gifs
* use linear instead of nearest-neighbour for resizing
* work with image "orientation" instead of "rotation". use default 75% quality for both webp and jpeg generation
* add header to new file
* use thumb extension when getting thumb mime type
* update test models and tests with new media processing
* add suggested code comments
* add note about thumbnail filter count reducing memory usage
* add test for emoji update image
* update emoji recache to set the instance account id
* don't refresh emoji if only not cached. in that case literally just recache
* code comment
* rename + move a few things
* add some more code comments, and rename some functions to make logic a bit clearer
* remove unnecessary nil check (the value can be nil)
* comment wording
* remove test data output
* handle the case of caching an emoji which has been refreshed then uncached
* allow overwriting on testrig storage as we do now on regular storage
* fix emoji category ID not getting updated
---------
Co-authored-by: tobi <tobi.smethurst@protonmail.com>
* persist queued worker tasks to database on shutdown, fill worker queues from database on startup
* ensure the tasks are sorted by creation time before pushing them
* add migration to insert WorkerTask{} into database, add test for worker task persistence
* add test for recovering worker queues from database
* quick tweak
* whoops we ended up with double cleaner job scheduling
* insert each task separately, because bun is throwing some reflection error??
* add specific checking of cancelled worker contexts
* add http request signing to deliveries recovered from database
* add test for outgoing public key ID being correctly set on delivery
* replace select with Queue.PopCtx()
* get rid of loop now we don't use it
* remove field now we don't use it
* ensure that signing func is set
* header values weren't being copied over 🤦
* use ptr for httpclient.Request in delivery
* move worker queue filling to later in server init process
* fix rebase issues
* make logging less shouty
* use slices.Delete() instead of copying / reslicing
* have database return tasks in ascending order instead of sorting them
* add a 1 minute timeout to persisting worker queues
* Implement followed tags API
* Insert statuses with followed tags into home timelines
* Test following and unfollowing tags
* Correct Swagger path params
* Trim conversation caches
* Migration for followed_tags table
* Followed tag caches and DB implementation
* Lint and tests
* Add missing tag info endpoint, reorganize tag API
* Unwrap boosts when timelining based on tags
* Apply visibility filters to tag followers
* Address review comments
* [feature] Federate interaction policies + Accepts; enforce policies
* use Acceptable type
* fix index
* remove appendIRIStrs
* add GetAccept federatingdb function
* lock on object IRI
* [chore] Add interaction filter to complement existing visibility filter
* pass in ptr to visibility and interaction filters to Processor{} to ensure shared
* use int constants for for match type, cache db calls in filterctx
* function name typo 😇
---------
Co-authored-by: kim <grufwub@gmail.com>
* Implement conversations API
* Sort and page conversations by last status ID
* Appease linter
* Fix deleting conversations and statuses
* Refactor to make migrations automatic
* Lint
* Update tests post-merge
* Fixes from live-fire testing
* Linter caught a format problem
* Refactor tests, fix cache
* Negative test for non-DMs
* Run conversations advanced migration on testrig startup as well as regular server startup
* Document (lack of) side effects of API method for deleting a conversation
* Make not-found check less nested for readability
* Rename PutConversation to UpsertConversation
* Use util.Ptr instead of IIFE
* Reduce cache used by conversations
* Remove unnecessary TableExpr/ColumnExpr
* Use struct tags for both unique constraints on Conversation
* Make it clear how paging with GetDirectStatusIDsBatch should be used
* Let conversation paging skip conversations it can't render
* Use Bun NewDropTable
* Convert delete raw query to Bun
* Convert update raw query to Bun
* Convert latestConversationStatusesTempTable raw query partially to Bun
* Convert conversationStatusesTempTable raw query partially to Bun
* Rename field used to store result of MaxDirectStatusID
* Move advanced migrations to their own tiny processor
* Catch up util function name with main
* Remove json.… wrappers
* Remove redundant check
* Combine error checks
* Replace map with slice of structs
* Address processor/type converter comments
- Add context info for errors
- Extract some common processor code into shared methods
- Move conversation eligibility check ahead of populating conversation
* Add error context when dropping temp tables
* refactor file handling a tiny bit
* whoops
* make processing media / emoji defers a bit clear to see that it's the "on finished processing" path
* some wording
* add some debug logging
* add mutex locks for processing remote media
* try removing freshness check
* fix derefMedia not being allocated
* fix log format string
* handle case of empty file paths (i.e. not stored)
* remove media / emoji once finished processing from dereferencer maps
* whoops, fix the cached / force checks
* move url parsing outside of 'process___Safely()' funcs to prevalidate url
* use emoji.ShortcodeDomain()
* update RefreshEmoji() to also match RefreshMedia() changes
---------
Co-authored-by: tobi <tobi.smethurst@protonmail.com>
* [feature/frontend] Respect `prefers-reduced-motion` for avatars, headers, and emojis
* go fmt
* fix tests
* use static version of instance thumbnail when appropriate
* use prefers-reduced-motion
* simplify account conversion a bit
* fix c&p error
* update to use webp for thumbnails
* bump webp quality up to 40% from 12% (it's a bit different to jpeg quality setting)
* update to use yuva colorspace, and use thumbnail=n=10 to select frame
* fix missing comma in ffmpeg args
* add links to appropriate ffmpeg docs
* update tests
* add file size tests for thumbnails
---------
Co-authored-by: tobi <tobi.smethurst@protonmail.com>
* don't set emoji / media image paths on failed download, migrate FileType from string to integer
* fix incorrect uses of util.PtrOr, fix returned frontend media
* fix migration not setting arguments correctly in where clause
* fix not providing default with not null column
* whoops
* ensure a default gets set for media attachment file type
* remove the exclusive flag from writing files in disk storage
* rename PtrOr -> PtrOrZero, and rename PtrValueOr -> PtrOrValue to match
* slight wording changes
* use singular / plural word forms (no parentheses), is better for screen readers
* update testmodels with unknown media type to have unset file details, update attachment focus handling converting to frontend, update tests
* store first instance in ffmpeg wasm pool, fill remaining with closed instances
* [feature] Allow users to set default interaction policies
* use vars for default policies
* avoid some code repetition
* unfuck form binding
* avoid bonkers loop
* beep boop
* put policyValsToAPIPolicyVals in separate function
* don't bother with slices.Grow
* oops
* [feature/frontend] Audio player for audio media types
* use video preview images for previews instead of video itself
* don't preload
* update tests for new zork status
* collapse media gallery into single row when small
* initial work replacing our media decoding / encoding pipeline with ffprobe + ffmpeg
* specify the video codec to use when generating static image from emoji
* update go-storage library (fixes incompatibility after updating go-iotools)
* maintain image aspect ratio when generating a thumbnail for it
* update readme to show go-ffmpreg
* fix a bunch of media tests, move filesize checking to callers of media manager for more flexibility
* remove extra debug from error message
* fix up incorrect function signatures
* update PutFile to just use regular file copy, as changes are file is on separate partition
* fix remaining tests, remove some unneeded tests now we're working with ffmpeg/ffprobe
* update more tests, add more code comments
* add utilities to generate processed emoji / media outputs
* fix remaining tests
* add test for opus media file, add license header to utility cmds
* limit the number of concurrently available ffmpeg / ffprobe instances
* reduce number of instances
* further reduce number of instances
* fix envparsing test with configuration variables
* update docs and configuration with new media-{local,remote}-max-size variables
* start updating media manager interface ready for storing attachments / emoji right away
* store emoji and media as uncached immediately, then (re-)cache on Processing{}.Load()
* remove now unused media workers
* fix tests and issues
* fix another test!
* fix emoji activitypub uri setting behaviour, fix remainder of test compilation issues
* fix more tests
* fix (most of) remaining tests, add debouncing to repeatedly failing media / emojis
* whoops, rebase issue
* remove kim's whacky experiments
* do some reshuffling, ensure emoji uri gets set
* ensure marked as not cached on cleanup
* tweaks to media / emoji processing to handle context canceled better
* ensure newly fetched emojis actually get set in returned slice
* use different varnames to be a bit more obvious
* move emoji refresh rate limiting to dereferencer
* add exported dereferencer functions for remote media, use these for recaching in processor
* add check for nil attachment in updateAttachment()
* remove unused emoji and media fields + columns
* see previous commit
* fix old migrations expecting image_updated_at to exists (from copies of old models)
* remove freshness checking code (seems to be broken...)
* fix error arg causing nil ptr exception
* finish documentating functions with comments, slight tweaks to media / emoji deref error logic
* remove some extra unneeded boolean checking
* finish writing documentation (code comments) for exported media manager methods
* undo changes to migration snapshot gtsmodels, updated failing migration to have its own snapshot
* move doesColumnExist() to util.go in migrations package
* use apiutil + paging in admin processor+handlers
* we're making it happen
* fix little whoopsie
* styling for report list
* don't youuuu forget about meee don't don't don't don't
* last bits
* sanitize content before showing in report statuses
* update report docs
These filter API v2 features were cut late in development because the form encoding version is hard to implement correctly and because I thought no clients actually used `keywords_attributes`. Unfortunately, Phanpy does use `keywords_attributes`.
