* start updating media manager interface ready for storing attachments / emoji right away
* store emoji and media as uncached immediately, then (re-)cache on Processing{}.Load()
* remove now unused media workers
* fix tests and issues
* fix another test!
* fix emoji activitypub uri setting behaviour, fix remainder of test compilation issues
* fix more tests
* fix (most of) remaining tests, add debouncing to repeatedly failing media / emojis
* whoops, rebase issue
* remove kim's whacky experiments
* do some reshuffling, ensure emoji uri gets set
* ensure marked as not cached on cleanup
* tweaks to media / emoji processing to handle context canceled better
* ensure newly fetched emojis actually get set in returned slice
* use different varnames to be a bit more obvious
* move emoji refresh rate limiting to dereferencer
* add exported dereferencer functions for remote media, use these for recaching in processor
* add check for nil attachment in updateAttachment()
* remove unused emoji and media fields + columns
* see previous commit
* fix old migrations expecting image_updated_at to exists (from copies of old models)
* remove freshness checking code (seems to be broken...)
* fix error arg causing nil ptr exception
* finish documentating functions with comments, slight tweaks to media / emoji deref error logic
* remove some extra unneeded boolean checking
* finish writing documentation (code comments) for exported media manager methods
* undo changes to migration snapshot gtsmodels, updated failing migration to have its own snapshot
* move doesColumnExist() to util.go in migrations package
* update typeconverter to use state structure
* deinterface the typeutils.TypeConverter -> typeutils.Converter
* finish copying over old type converter code comments
* fix cherry-pick merge issues, fix tests pointing to old typeutils interface type still
* media manager tidy-up: de-interface and remove unused PostDataFunc
Signed-off-by: kim <grufwub@gmail.com>
* remove last traces of media.Manager being an interface
Signed-off-by: kim <grufwub@gmail.com>
* update error to provide caller, allow tuneable via build tags
Signed-off-by: kim <grufwub@gmail.com>
* remove kim-specific build script changes
Signed-off-by: kim <grufwub@gmail.com>
* fix merge conflicts
Signed-off-by: kim <grufwub@gmail.com>
* update build-script to support externally setting build variables
Signed-off-by: kim <grufwub@gmail.com>
---------
Signed-off-by: kim <grufwub@gmail.com>
* [feature] Allow admins to send test emails
* implement unwrap on new error type
* add + use gtserror types
* GoToSocial Email Test -> GoToSocial Test Email
* add + use getInstance db call
* removed unused "unknown" error type
* [chore] Remove years from all license headers
Years or year ranges aren't required in license headers. Many projects
have removed them in recent years and it avoids a bit of yearly toil.
In many cases our copyright claim was also a bit dodgy since we added
the 2021-2023 header to files created after 2021 but you can't claim
copyright into the past that way.
* [chore] Add license header check
This ensures a license header is always added to any new file. This
avoids maintainers/reviewers needing to remember to check for and ask
for it in case a contribution doesn't include it.
* [chore] Add missing license headers
* [chore] Further updates to license header
* Use the more common // indentend comment format
* Remove the hack we had for the linter now that we use the // format
* Add SPDX license identifier
* [chore] Move ShortcodeDomain to its own little util func
* [feature] Add RefetchEmojis function to media manager
* [feature] Expose admin media refresh via admin API
* update following review feedback
- change/fix log levels
- make sure not to try to refetch local emojis
- small style refactoring + comments
* log on emoji refetch start
Signed-off-by: kim <grufwub@gmail.com>
Co-authored-by: kim <grufwub@gmail.com>
* start adding admin emoji PATCH stuff
* updating works OK, now how about copying
* allow emojis to be copied
* update swagger docs
* update admin processer to use non-interface storage driver
* remove shortcode updating for local emojis
* go fmt
Co-authored-by: f0x52 <f0x@cthu.lu>
* [feature] Add emoji categories GET
Serialize emojis in appropriate categories; make it possible to get categories via the admin API
* [feature] Create (or use existing) category for new emoji uploads
* fix lint issue
* update misleading line in swagger docs
* add admin emojis get path + model + docs
* stub admin emojis get processor function
* add id + disabled fields to admin emoji
* add emoji -> api admin emoji converter
* tidy up a bit
* add GetEmojis function
* finish up get emojis function
* order by shortcodedomain
* ASC
* tidy up + explain
* update to allow paging
* make admin emojis pageable
* fix mixed case paging
* normalize emoji queries a bit better
* test emoji get paging
* make limit optional
* fix incorrect path in media cleanup tests
* i have bad coder syndrome
* don't trimspace
* rename -> GetUseableEmojis
* wrap emoji query in subquery
avoid selecting more than we need
* fix a bit of sillyness teehee
* fix subquery postgres woes
* rename + tidy up remote pruning
* fix media attachment account join
see https://bun.uptrace.dev/guide/golang-orm.html#table-relationships
* update logging to new function name
* add get avatars and headers to bun
* add pruneallmeta function
* don't set uncached since we're deleting anyway
* fix totalPruned being in wrong place
* test pruning meta
* go fmt ./...
* rename mediaprune
* add meta pruning to routine mediaprune
* tidy up cleanup job scheduling
* rename adminmediaremoteprune
* update mediacleanup to use renamed prune func
* update swagger docs a little bit
* reuse cancel + context
* cache transports in controller by privkey-generated pubkey, add retry logic to transport requests
Signed-off-by: kim <grufwub@gmail.com>
* update code comments, defer mutex unlocks
Signed-off-by: kim <grufwub@gmail.com>
* add count to 'performing request' log message
Signed-off-by: kim <grufwub@gmail.com>
* reduce repeated conversions of same url.URL object
Signed-off-by: kim <grufwub@gmail.com>
* move worker.Worker to concurrency subpackage, add WorkQueue type, limit transport http client use by WorkQueue
Signed-off-by: kim <grufwub@gmail.com>
* fix security advisories regarding max outgoing conns, max rsp body size
- implemented by a new httpclient.Client{} that wraps an underlying
client with a queue to limit connections, and limit reader wrapping
a response body with a configured maximum size
- update pub.HttpClient args passed around to be this new httpclient.Client{}
Signed-off-by: kim <grufwub@gmail.com>
* add httpclient tests, move ip validation to separate package + change mechanism
Signed-off-by: kim <grufwub@gmail.com>
* fix merge conflicts
Signed-off-by: kim <grufwub@gmail.com>
* use singular mutex in transport rather than separate signer mus
Signed-off-by: kim <grufwub@gmail.com>
* improved useragent string
Signed-off-by: kim <grufwub@gmail.com>
* add note regarding missing test
Signed-off-by: kim <grufwub@gmail.com>
* remove useragent field from transport (instead store in controller)
Signed-off-by: kim <grufwub@gmail.com>
* shutup linter
Signed-off-by: kim <grufwub@gmail.com>
* reset other signing headers on each loop iteration
Signed-off-by: kim <grufwub@gmail.com>
* respect request ctx during retry-backoff sleep period
Signed-off-by: kim <grufwub@gmail.com>
* use external pkg with docs explaining performance "hack"
Signed-off-by: kim <grufwub@gmail.com>
* use http package constants instead of string method literals
Signed-off-by: kim <grufwub@gmail.com>
* add license file headers
Signed-off-by: kim <grufwub@gmail.com>
* update code comment to match new func names
Signed-off-by: kim <grufwub@gmail.com>
* updates to user-agent string
Signed-off-by: kim <grufwub@gmail.com>
* update signed testrig models to fit with new transport logic (instead uses separate signer now)
Signed-off-by: kim <grufwub@gmail.com>
* fuck you linter
Signed-off-by: kim <grufwub@gmail.com>
* add accountAction to the admin API
* model admin account action
* add admin account action to the processor
* add migration for new AdminAccountActions table
* fix accounts admin path
* Update swagger docs
* start pulling out + replacing urfave and config
* replace many many instances of config
* move more stuff => viper
* properly remove urfave
* move some flags to root command
* add testrig commands to root
* alias config file keys
* start adding cli parsing tests
* reorder viper init
* remove config path alias
* fmt
* change config file keys to non-nested
* we're more or less in business now
* tidy up the common func
* go fmt
* get tests passing again
* add note about the cliparsing tests
* reorganize
* update docs with changes
* structure cmd dir better
* rename + move some files around
* fix dangling comma
* reference logrus' global logger instead of passing and storing a logger reference everywhere
* always directly use global logrus logger instead of referencing an instance
* test suites should also directly use the global logrus logger
* rename gin logging function to clarify that it's middleware
* correct comments which erroneously referenced removed logger parameter
* setting log level for tests now uses logrus' exported type instead of the string value, to guarantee error isn't possible
* start work on admin domain blocking
* move stuff around + further work on domain blocks
* move + restructure processor
* prep work for deleting account
* tidy
* go fmt
* formatting
* domain blocking more work
* check domain blocks way earlier on
* progress on delete account
* delete more stuff when an account is gone
* and more...
* domain blocky block block
* get individual domain block, delete a block
