When we receive an HTTP 410 on webfinger it means the resource we asked
for (the account) is gone, but the endpoint itself responded. In such
cases we want to treat the request as successful from a cache (renewal)
point of view, while still returning an error from Finger.
Follow-up for #1588
* slight refactor and improved logging on failed webfinger in enrichAccount()
* use correct log format directive
---------
Signed-off-by: kim <grufwub@gmail.com>
* [feature] Discover webfinger through host-meta
This implements a fallback for discovering the webfinger endpoint in
case the /.well-known/webfinger endpoint wasn't properly redirected.
Some instances do this because the recommendation used to be to use
host-meta for the webfinger redirect in the before times.
Closes#1558.
* [bug] Ensure we only ever update cache on success
* [chore] Move finger tests to their own place
This adds a test suite for transport and moves the finger cache tests
into there instead of abusing the search test suite.
* [chore] cleanup the test a bit more
We don't really need a separate function for the oddly located webfinger
response as we check the full URL string anyway
* Address review comments
* [chore] update config example
* [chore] access DB only through state in controller
* add error value wrapping, include status code / not found flags from transport errors, update error usages
Signed-off-by: kim <grufwub@gmail.com>
* add code commenting for gtserror functions
Signed-off-by: kim <grufwub@gmail.com>
---------
Signed-off-by: kim <grufwub@gmail.com>
* Add go-playground/form pkg
* [feature] Add support for profile fields
* Add field attributes test
* Validate profile fields form
* Add profile field validation tests
* Add Field Attributes definition to swagger
---------
Co-authored-by: tobi <31960611+tsmethurst@users.noreply.github.com>
Currently, GtS only supports using the built-in LE client directly for
TLS. However, admins may still want to use GtS directly (so without a
reverse proxy) but with certificates provided through some other
mechanism. They may have some centralised way of provisioning these
things themselves, or simply prefer to use LE but with a different
challenge like DNS-01 which is not supported by autocert.
This adds support for loading a public/private keypair from disk instead
of using LE and reconfigures the server to use a TLS listener if we
succeed in doing so.
Additionally, being able to load TLS keypair from disk opens up the path
to using a custom CA for testing purposes avoinding the need for a
constellation of containers and something like Pebble or Step CA to
provide LE APIs.
* replace concurrency worker pools with base models in State.Workers, update code and tests accordingly
* add media attachment caching, slightly tweak default cache config
* further tweak default cache config values
* replace other media attachment db calls to go through cache
* update envparsing test
* fix delete media attachment sql
* fix media sql query
* invalidate cached media entries during status create / update
* fix envparsing test
* fix typo in panic log message...
* add 'updated_at' column during UpdateAttachment
* remove unused func
---------
Signed-off-by: kim <grufwub@gmail.com>
* Advertise rich text formats, support content_type field
* Update JSON in instance patch tests
* Replace format with content_type everywhere
* update migration to work with both pg and sqlite
* regenerate swagger docs
* update instance serialization + tests
* fix up
* learn to code tobi please, i'm begging you
---------
Co-authored-by: tsmethurst <tobi.smethurst@protonmail.com>
* start fiddling
* the ol' fiddle + update
* start working on fetching statuses
* poopy doopy doo where r u uwu
* further adventures in featuring statuses
* finishing up
* fmt
* simply status unpin loop
* move empty featured check back to caller function
* remove unnecessary log.WithContext calls
* remove unnecessary IsIRI() checks
* add explanatory comment about status URIs
* change log level to error
* better test names
* simply use storage.Storage, removing wrapping KVStore as we don't need KV store locking functionality
Signed-off-by: kim <grufwub@gmail.com>
* fix missing unwrapped function
Signed-off-by: kim <grufwub@gmail.com>
* add code comment
Signed-off-by: kim <grufwub@gmail.com>
* linter, please take my offering in peace
Signed-off-by: kim <grufwub@gmail.com>
---------
Signed-off-by: kim <grufwub@gmail.com>
This attempts to provide a slightly more comprehensive error message for
the end user when an incorrect grant type is used. This is not something
the user can typically resolve but should hopefully be informative for
the (client) developer.
This removes the current default of checking for membership of the admin
or admins group and makes it required to explicitly configure which
groups should grant admin access, if any.
Relying on the implicit default of admin or admins is potentially
dangerous as that group may contain a different subset of people that we
may wish to grant admin access to GtS. This is probably not an issue for
a single-person instance, but for a community instance different admin
groups may exist in an OIDC provider for different applications.
I'm explicitly opting for not defaulting the value of oidc-admin-groups
to admin,admins because I think it's better for those things to be
explicitly configured.
* implement status pin client api + web handler
* make test names + comments more descriptive
* don't use separate table for status pins
* remove unused add + remove checking
* tidy up + add some more tests
This changes parseDescription to properly encode things to be safe for
usage without removing things like backslashes that may be relevant.
* text.SanitizePlaintext already calls html.UnescapeString so we don't
have to do that
* Replace \n with space early
* Remove duplicate white-space by splitting on fields and joining
* HTML-escape the string we have
* For extra certainty, encode the backslash as \
Fixes#1549
* Tidy up emoji parsing on profile submission
Don't bother reparsing for emoji unless one of the fields that can have
emoji in it has changed.
Deduplicate emoji between the display name and profile note - I'm not
sure whether this was hurting anything, but better safe.
* Deduplicate emoji when parsing remote accounts
Some servers - Misskey at least - don't deduplicate emoji, so it's
possible to get an account which has the same emoji used in both the
display name and note and therefore includes that emoji twice in its
metadata. When we start trying to put those into our database, we run
into a uniqueness constraint and fall over.
This change just deduplicates at the point of construction of an
account.
This moves checking if the request is valid as early as possible in the
chain. This should ensure that for an invalid request we never bother
acquiring the wait queue and taking up a spot in it.
* improve error readability, mark "bad hosts" as fastFail
Signed-off-by: kim <grufwub@gmail.com>
* pull in latest go-byteutil version with byteutil.Reader{}
Signed-off-by: kim <grufwub@gmail.com>
* use rewindable body reader for post requests
Signed-off-by: kim <grufwub@gmail.com>
---------
Signed-off-by: kim <grufwub@gmail.com>
This updates the middleware log.WithField calls that create new loggers
to include the context the first time around. Without it the requestID
does not get logged.
Fixup from #1476
This adds a lightweight form of tracing to GTS. Each incoming request is
assigned a Request ID which we then pass on and log in all our log
lines. Any function that gets called downstream from an HTTP handler
should now emit a requestID=value pair whenever it logs something.
Co-authored-by: kim <grufwub@gmail.com>
* [bugfix] set cache-control max-age dynamically for s3
* woops
* double whoops
* time until, thank you linter, bless you, you're the best, no matter what kim says
* aa
* start fiddling with media + account queries a little
* initialize state when pruning
* allow for unsetting remote media
make sure to wait til media loaded
fix silly tiny bug
* move comment a bit for readability
* slight reformat of fetchRemoteAccount{Avatar,Header}
* fix issue after rebase
* slightly neaten up logic of avatar/header media handling
* remove log prefix (callername log field handles this)
---------
Signed-off-by: kim <grufwub@gmail.com>
Co-authored-by: kim <grufwub@gmail.com>
* replace media workers with just runners.WorkerPool, move to state structure, use go-sched for global task scheduling
* improved code comment
* fix worker tryUntil function, update go-runners/go-sched
* make preprocess functions package public, use these where possible to stop doubled up processing
* remove separate emoji worker pool
* limit calls to time.Now() during media preprocessing
* use Processor{} to manage singular runtime of processing media
* ensure workers get started when media manager is used
* improved error setting in processing media, fix media test
* port changes from processingmedia to processing emoji
* finish code commenting
* finish code commenting and comment-out client API + federator worker pools until concurrency worker pools replaced
* linterrrrrrrrrrrrrrrr
---------
Signed-off-by: kim <grufwub@gmail.com>
* remove throttling timers, support setting retry-after, use retry-after in transport
* remove unused variables
* add throttling-retry-after to cmd flags
* update envparsing to include new throttling-retry-after
* update example config to include retry-after documentation
* also support retry-after formatted as date-time, ensure max backoff time
---------
Signed-off-by: kim <grufwub@gmail.com>
* small formatting changes, rewrite fetchRemoteMedia to use separate funcs + use mutex lock correctly
* move url parsing before acquiring mutex locks
* use wrapped mutexes to allow safe unlocking. (previously i did a fucky and passed mutex by value...)
* remove unused code
* use consistent map keying for dereferencing headers/avatars
---------
Signed-off-by: kim <grufwub@gmail.com>
* serve publickey separately from AP, don't throttle it
* update nginx cache documentation, cache main-key too
* throttle public key, but separately from other endpoints
