* initial work replacing our media decoding / encoding pipeline with ffprobe + ffmpeg
* specify the video codec to use when generating static image from emoji
* update go-storage library (fixes incompatibility after updating go-iotools)
* maintain image aspect ratio when generating a thumbnail for it
* update readme to show go-ffmpreg
* fix a bunch of media tests, move filesize checking to callers of media manager for more flexibility
* remove extra debug from error message
* fix up incorrect function signatures
* update PutFile to just use regular file copy, as changes are file is on separate partition
* fix remaining tests, remove some unneeded tests now we're working with ffmpeg/ffprobe
* update more tests, add more code comments
* add utilities to generate processed emoji / media outputs
* fix remaining tests
* add test for opus media file, add license header to utility cmds
* limit the number of concurrently available ffmpeg / ffprobe instances
* reduce number of instances
* further reduce number of instances
* fix envparsing test with configuration variables
* update docs and configuration with new media-{local,remote}-max-size variables
* Set frame-ancestors in the CSP
This ensures we can't be loaded/embedded in an iframe. It also sets the
older X-Frame-Options for fallback.
* Disable MIME type sniffing
* Set Referrer-Policy
This sets the policy such that browsers will never send the Referer
header along with a request, unless it's a request to the same protocol,
host/domain and port. Basically, only send it when navigating through
our own UI, but not anything external.
The default is strict-origin-when-cross-origin when unset, which sends
the Referer header for requests unless it's going from HTTPS to HTTP
(i.e a security downgrade, hence the 'strict').
* [feature] Default to WASM-based SQLite driver
With 0.16 out this switches our default SQLite driver to the WASM-based
solution instead. So far the driver seems to perform just as well.
Switching our default should result in it getting a bit more testing
during the 0.17 development cycle.
* add the ol' john hancock
---------
Co-authored-by: tobi <tobi.smethurst@protonmail.com>
* start updating media manager interface ready for storing attachments / emoji right away
* store emoji and media as uncached immediately, then (re-)cache on Processing{}.Load()
* remove now unused media workers
* fix tests and issues
* fix another test!
* fix emoji activitypub uri setting behaviour, fix remainder of test compilation issues
* fix more tests
* fix (most of) remaining tests, add debouncing to repeatedly failing media / emojis
* whoops, rebase issue
* remove kim's whacky experiments
* do some reshuffling, ensure emoji uri gets set
* ensure marked as not cached on cleanup
* tweaks to media / emoji processing to handle context canceled better
* ensure newly fetched emojis actually get set in returned slice
* use different varnames to be a bit more obvious
* move emoji refresh rate limiting to dereferencer
* add exported dereferencer functions for remote media, use these for recaching in processor
* add check for nil attachment in updateAttachment()
* remove unused emoji and media fields + columns
* see previous commit
* fix old migrations expecting image_updated_at to exists (from copies of old models)
* remove freshness checking code (seems to be broken...)
* fix error arg causing nil ptr exception
* finish documentating functions with comments, slight tweaks to media / emoji deref error logic
* remove some extra unneeded boolean checking
* finish writing documentation (code comments) for exported media manager methods
* undo changes to migration snapshot gtsmodels, updated failing migration to have its own snapshot
* move doesColumnExist() to util.go in migrations package
This syncs our copy with the current state of the ai.robots.txt
repository. Upstream has tightened their scope to be AI-only, whereas
before it included a bunch of SEO and "web intelligence" marketing
stuff. I've kept those but moved them into their own section.
* [bugfix] add Date and Message-ID headers for email
This should make spam filters more happy, as most of them grant some
negative score for not having those headers. Also the Date is convenient
for the user receiving the mail.
* make golangci-lint happy
* improved server shutdown with more precise shutdown of modules + deferring of ALL of it
* move delivery and workers into separate files
* add worker task model and Serialize() / Deserialize() methods for message types
* start adding message serialize / deserialize tests
* start adding test cases
* update body rewinding to rely on standard library mechanism of r.GetBody()
* remove request rewinding (http.Client{} should already handle this)
* standard library already handles rewinding
* improved code comment
* move the newPOST() function contents to prepare(), fits better with current API
* add Serialize() / Deserialize() implementations for Delivery{} type
* finish writing FromClientAPI sserialize / deserialize tests
* start adding FromFediAPI{} serialize / deserialize test cases
* fix FromFediAPI{} tests
* add tests for delivery message type
* fix repeat code
* missing license header
* use testrig status and accounts for marshal / unmarshaling tests
* add a specific test for checking account RSA keys are preserved
* use apiutil + paging in admin processor+handlers
* we're making it happen
* fix little whoopsie
* styling for report list
* don't youuuu forget about meee don't don't don't don't
* last bits
* sanitize content before showing in report statuses
* update report docs
These filter API v2 features were cut late in development because the form encoding version is hard to implement correctly and because I thought no clients actually used `keywords_attributes`. Unfortunately, Phanpy does use `keywords_attributes`.
* [bugfix] Deref stats async, allow peek if handshaking
* don't return totalItems when handshaking or hiding collections
* use GetLimit()
* use StubAccountStats
* fix possible infinite recursion if moved accounts are self-referential
* adds a defensive check for a boost being a boost of a boost wrapper
* add checks on input for a boost of a boost
* remove unnecessary check
* add protections on account move to prevent move recursion loops
* separate status conversion without boost logic into separate function to remove risk of recursion
* move boost check to boost function itself
* formatting
* use error 422 instead of 500
* use gtserror not standard errors package for error creation
* User muting
* Address review feedback
* Rename uniqueness constraint on user_mutes to match convention
* Remove unused account_id from where clause
* Add UserMute to NewTestDB
* Update test/envparsing.sh with new and fixed cache stuff
* Address tobi's review comments
* Make compiledUserMuteListEntry.expired consistent with UserMute.Expired
* Make sure mute_expires_at is serialized as an explicit null for indefinite mutes
---------
Co-authored-by: tobi <tobi.smethurst@protonmail.com>
* refactor status media handling into separate functions, handle case of changed metadata
* update fetchRemoteAccount{Avatar,Header} to use new refactored {load,update}Attachment() functions
* whoops, nearly marked avatars as headers :')
* reformatting to improve legibility
* [feature] Email change
* frontend stuff for changing email
* docs
* tests etc
* differentiate more clearly between local user+account and account
* populate user
* do not uncache status / emoji media if attached status is bookmarked
* add status bookmark and bookmark IDs caches
* update status bookmark tests
* move IsStatusBookmarkedBy() to StatusBookmark{} interface, rely on cache
* fix envparsing.sh test
* removes the avatar / header deref maps as we now have per-uri status / account locks, adds retries on data-races, adds separate emoji map mutex
* work with a copy of account / status for each retry loop
* revert to old data race behaviour, it gets too complicated otherwise
---------
Co-authored-by: tobi <tobi.smethurst@protonmail.com>
* Use correct entity name
* We support server-side filters now
* Document filter v1 methods that can throw a 409
* Validate v1 filter phrase as filter title
* Always check v1 filter API status codes in tests
* Document keyword minimum requirement on filter API v1
* Make it possible to specify filter keyword update columns per filter keyword
* Implement v2 filter API
* Fix lint and tests
* Update Swagger spec
* Fix filter update test
* Update Swagger spec *correctly*
* Update actual files Swagger spec was generated from
* Remove keywords_attributes and statuses_attributes
* Add test for serialization of empty filter
* More helpful messages when object is owned by wrong account
On startup and shutdown of a worker, we log a message of the worker
being started together with a textual representation of a memory
address. Though this can be handy for developers to debug
startup/shutdown sequencing issues of the workers, it's typically not
very useful or informative for an admin. We can also output a lot of
these (on my system I get 265 lines of these during startup).
This changes the messages from Info to Debug, to not print them under
normal circumstances.
* improved server shutdown with more precise shutdown of modules + deferring of ALL of it
* make the same changes to the testrig server
* use testrig specific func
* update variable name to fix nilptr
* fix removal of setting db on state
* Implement profile API
This Mastodon 4.2 extension provides capabilities missing from the existing Mastodon account update API: deleting an account's avatar or header.
See: https://docs.joinmastodon.org/methods/profile/
* Move profile media methods to media processor
* Remove check for moved account
