Commit graph

617 commits

Author SHA1 Message Date
Daenney 68e6d08c76
[feature] Add a request ID and include it in logs ()
This adds a lightweight form of tracing to GTS. Each incoming request is
assigned a Request ID which we then pass on and log in all our log
lines. Any function that gets called downstream from an HTTP handler
should now emit a requestID=value pair whenever it logs something.

Co-authored-by: kim <grufwub@gmail.com>
2023-02-17 12:02:29 +01:00
tobi b5993095fa
[bugfix] Set 'discoverable' properly on API accounts () 2023-02-16 13:20:23 +00:00
tobi 6ee0dc8c7d
[bugfix] Set cache-control max-age dynamically for s3 ()
* [bugfix] set cache-control max-age dynamically for s3

* woops

* double whoops

* time until, thank you linter, bless you, you're the best, no matter what kim says

* aa
2023-02-16 13:18:53 +00:00
Sam Lade 40b584c219
Fix 410 Gone race on account deletes () 2023-02-15 19:41:16 +00:00
tobi 6c6f042290
[bugfix] Return empty result rather than 500 error when searching for blocked domains ()
* [bugfix] Return empty result when searching for blocked domains

* add tests
2023-02-14 11:55:02 +01:00
tobi 561ad71e58
[bugfix] Fix up error getting account avatar/header errors, other small fixes ()
* start fiddling with media + account queries a little

* initialize state when pruning

* allow for unsetting remote media
make sure to wait til media loaded
fix silly tiny bug

* move comment a bit for readability

* slight reformat of fetchRemoteAccount{Avatar,Header}

* fix issue after rebase

* slightly neaten up logic of avatar/header media handling

* remove log prefix (callername log field handles this)

---------

Signed-off-by: kim <grufwub@gmail.com>
Co-authored-by: kim <grufwub@gmail.com>
2023-02-13 20:19:51 +00:00
kim acc95923da
[performance] processing media and scheduled jobs improvements ()
* replace media workers with just runners.WorkerPool, move to state structure, use go-sched for global task scheduling

* improved code comment

* fix worker tryUntil function, update go-runners/go-sched

* make preprocess functions package public, use these where possible to stop doubled up processing

* remove separate emoji worker pool

* limit calls to time.Now() during media preprocessing

* use Processor{} to manage singular runtime of processing media

* ensure workers get started when media manager is used

* improved error setting in processing media, fix media test

* port changes from processingmedia to processing emoji

* finish code commenting

* finish code commenting and comment-out client API + federator worker pools until concurrency worker pools replaced

* linterrrrrrrrrrrrrrrr

---------

Signed-off-by: kim <grufwub@gmail.com>
2023-02-13 18:40:48 +00:00
tobi 041c8e695e
[chore] Do cache-control in a less silly way to avoid writing header twice ()
* do cache-control in a less silly way to avoid writing header twice

* add comment back in
2023-02-13 12:58:22 +01:00
tobi c223c7598d
[bugfix] Set appropriate cache-control when using presigned s3 links () 2023-02-12 14:42:28 +01:00
tobi 40bc03e717
[chore/performance] Update media prune logic, add extra CLI command ()
* start updating media prune stuff a wee bit

* continue prune / uncache work

* more tidying + consistency stuff

* add prune CLI command

* docs

* arg
2023-02-11 12:48:38 +01:00
kim 70739d32cc
[performance] remove throttling timers ()
* remove throttling timers, support setting retry-after, use retry-after in transport

* remove unused variables

* add throttling-retry-after to cmd flags

* update envparsing to include new throttling-retry-after

* update example config to include retry-after documentation

* also support retry-after formatted as date-time, ensure max backoff time

---------

Signed-off-by: kim <grufwub@gmail.com>
2023-02-10 20:16:01 +00:00
kim 6ac1dda96f
[chore] small changes missed in previous dereferencer.GetAccount() PRs ()
* small formatting changes, rewrite fetchRemoteMedia to use separate funcs + use mutex lock correctly

* move url parsing before acquiring mutex locks

* use wrapped mutexes to allow safe unlocking. (previously i did a fucky and passed mutex by value...)

* remove unused code

* use consistent map keying for dereferencing headers/avatars

---------

Signed-off-by: kim <grufwub@gmail.com>
2023-02-10 20:15:23 +00:00
tobi e5e257c259
[bugfix] Fix error on searching for account w/accountDomain by host () 2023-02-09 10:34:44 +01:00
tobi 95715f9251
[performance] Don't fetch avatar + header if uri hasn't changed () 2023-02-09 09:27:07 +01:00
tobi 27e95fd123
[chore/bugfix] Serve + throttle publickey separately from rest of ActivityPub API ()
* serve publickey separately from AP, don't throttle it

* update nginx cache documentation, cache main-key too

* throttle public key, but separately from other endpoints
2023-02-08 15:10:56 +01:00
tobi 4e4da19720
[bugfix] Use SignatureCheck middleware for web profile endpoints too () 2023-02-07 14:57:09 +01:00
Sam Lade ad6ab037e4
[bugfix] don't trash emoji in profile fields on edit () 2023-02-07 09:58:36 +01:00
kim ac2bdbbc62
[bugfix] fix file range length calculation being off by 1 ()
* small formatting change

* fix range handling new length calculation

---------

Signed-off-by: kim <grufwub@gmail.com>
2023-02-07 08:51:15 +00:00
Sam Lade 6a6647d68b
Ignore missing files when cleaning up media () 2023-02-06 14:58:54 +00:00
kim 02767bfc7d
[performance] remove local copying of file for satisfying range headers ()
* remove local copying of file for satisfying range headers

Signed-off-by: kim <grufwub@gmail.com>

* remove unused import

Signed-off-by: kim <grufwub@gmail.com>

* fix range header reader limit calculation

Signed-off-by: kim <grufwub@gmail.com>

---------

Signed-off-by: kim <grufwub@gmail.com>
2023-02-06 09:50:16 +01:00
tobi 75e1b9c896
[bugfix] fix old password hash staying in cache () 2023-02-06 09:44:13 +01:00
tobi 7f3245738d
[chore] stub /api/v1/featured_tags endpoint () 2023-02-05 10:50:09 +01:00
tobi 80c26d61f7
[bugfix] Allow instance thumbnail description to be set separately from image () 2023-02-04 15:53:11 +01:00
kim abe9447d28
fix cache startup ()
Signed-off-by: kim <grufwub@gmail.com>
2023-02-03 20:16:11 +00:00
kim 33aee1b1e9
[chore] reformat GetAccount() functionality, support updating accounts based on last_fetch ()
* reformat GetAccount() functionality, and add UpdateAccount() function.

* use fetched_at instead of last_webfingered_at

* catch local "not found" errors. small formatting / error string changes

* remove now unused error type

* return nil when wrapping nil error

* update expected error messages

* return correct url for foss satan webfinger

* add AP model for Some_User

* normalize local domain

* return notretrievable where appropriate

* expose NewErrNotRetrievable

* ensure webfinger for new accounts searched by uri

* update local account short circuit

* allow enrich to fail for already-known accounts

* remove unused LastWebfingeredAt

* expose test maps on mock http client

* update Update test

* reformat GetAccount() functionality, and add UpdateAccount() function.

* use fetched_at instead of last_webfingered_at

* catch local "not found" errors. small formatting / error string changes

* remove nil error checks (we shouldn't be passing nil errors to newError() initializers)

* remove mutex unlock on transport init fail (it hasn't yet been locked!)

* woops add back the error wrapping to use ErrNotRetrievable

* caches were never being started... 🙈

---------

Signed-off-by: kim <grufwub@gmail.com>
Co-authored-by: tsmethurst <tobi.smethurst@protonmail.com>
2023-02-03 20:03:05 +00:00
Autumn! 49beb17a8f
[chore] Text formatting overhaul ()
* Implement goldmark debug print for hashtags and mentions

* Minify HTML in FromPlain

* Convert plaintext status parser to goldmark

* Move mention/tag/emoji finding logic into formatter

* Combine mention and hashtag boundary characters

* Normalize unicode when rendering hashtags
2023-02-03 11:58:58 +01:00
tobi 271da016b9
[bugfix] Read Bookwyrm Articles more thoroughly () 2023-02-02 16:41:02 +01:00
tobi 382512a5a6
[feature] Implement /api/v2/instance endpoint ()
* interim: start adding /api/v2/instance

* finish up
2023-02-02 14:08:13 +01:00
tobi 4ee4cd2da1
[chore/performance] use only 1 sqlite db connection regardless of multiplier () 2023-02-01 11:55:34 +01:00
tobi b80be48fed
[chore] Use 'immediate' lock for sqlite transactions ()
* [chore] Use 'immediate' lock for sqlite transactions

* allow 1 connection regardless of cpu amount
2023-01-31 13:46:45 +01:00
tobi 3283900b0d
[feature] Federate reports to remote instance as Flag (if desired) ()
* reports federate out, we did it lxds

* fix optional line start (should be optional slash)
2023-01-27 14:48:11 +01:00
tobi 782169da76
[chore] set max open / idle conns + conn max lifetime for both postgres and sqlite ()
* [chore] set max open / idle conns + conn max lifetime for both postgres and sqlite

* reduce cache size default to 8MiB, reduce connections to 2 * cpu

* introduce max open conns multiplier, tune sqlite and pg separately

* go fmt
2023-01-26 15:12:48 +01:00
f0x52 17eecfb6d9
[feature] Public list of suspended domains ()
* basic rendered domain blocklist (unauthenticated!)

* style basic domain block list

* better formatting for domain blocklist

* add opt-in config option for showing suspended domains

* format/linter

* re-use InstancePeersGet for web-accessible domain blocklist

* reword explanation, border styling

* always attach blocklist handler, update error message

* domain blocklist error message grammar
2023-01-25 18:06:41 +01:00
tobi 993aae5e48
[feature] Accept incoming federated Flag activity ()
* start working on handling incoming Flag activity

* interim commit

* federate Flag in successfully
2023-01-25 11:12:27 +01:00
tobi faeb7ded3b
[feature] Implement reports admin API so admins can view + close reports ()
* add admin report api endpoints + tests

* [chore] remove funky duplicate attachment in testrig
2023-01-25 11:12:17 +01:00
tobi 36f62d6e60
[chore] remove funky duplicate attachment in testrig ()
* [chore] remove funky duplicate attachment in testrig

* whoops
2023-01-23 13:39:01 +01:00
tobi e9747247d5
[feature] Implement /api/v1/reports endpoints on client API ()
* start adding report client api

* route + test reports get

* start report create endpoint

* you can create reports now babyy

* stub account report processor

* add single reportGet endpoint

* fix test

* add more filtering params to /api/v1/reports GET

* update swagger

* use marshalIndent in tests

* add + test missing Link info
2023-01-23 13:14:21 +01:00
즈눅 13ec15d2bb
[chore] extending maximumPasswordLength to 256 ()
* [chore] extending maximumPasswordLength to 256

Resolves: 

* Make long test longer
2023-01-22 23:25:11 +01:00
tobi 974ec80a20
[chore] Change default sqlite busy timeout to 5m () 2023-01-18 14:37:52 +01:00
tobi 627b8eeae6
[feature] Tune sqlite pragmas ()
* sqlite pragma tuning

* use formatuint

* add sqlite busy timeout

* fix incorrect cache size format

* update envparsing test

* add sqlite tuning flags to cli

* set sqlite timeout to 30s default
2023-01-17 12:29:44 +00:00
tobi d4cddf460a
[bugfix] Parse video metadata more accurately; allow Range in fileserver ()
* don't serve unused fields for video attachments

* parse video bitrate + duration more accurately

* use ServeContent where appropriate to respect Range

* abstract temp file seeker into its own function
2023-01-16 16:19:17 +01:00
tobi 132c73883f
[bugfix] Mount bookmarks endpoint correctly () 2023-01-13 17:07:52 +01:00
tobi eafd73c292
[chore] Remove omitempty on account source; refactor tests to use prettyprint json ()
* remove omitEmpty tag on account source items

* update tests
2023-01-13 17:02:45 +01:00
kim 5318054808
[performance] media processing improvements ()
* media processor consolidation and reformatting, reduce amount of required syscalls

Signed-off-by: kim <grufwub@gmail.com>

* update go-store library, stream jpeg/png encoding + use buffer pools, improved media processing AlreadyExists error handling

Signed-off-by: kim <grufwub@gmail.com>

* fix duration not being set, fix mp4 test expecting error

Signed-off-by: kim <grufwub@gmail.com>

* fix test expecting media files with different extension

Signed-off-by: kim <grufwub@gmail.com>

* remove unused code

Signed-off-by: kim <grufwub@gmail.com>

* fix expected storage paths in tests, update expected test thumbnails

Signed-off-by: kim <grufwub@gmail.com>

* remove dead code

Signed-off-by: kim <grufwub@gmail.com>

* fix cached presigned s3 url fetching

Signed-off-by: kim <grufwub@gmail.com>

* fix tests

Signed-off-by: kim <grufwub@gmail.com>

* fix test models

Signed-off-by: kim <grufwub@gmail.com>

* update media processing to use sync.Once{} for concurrency protection

Signed-off-by: kim <grufwub@gmail.com>

* shutup linter

Signed-off-by: kim <grufwub@gmail.com>

* fix passing in KVStore GetStream() as stream to PutStream()

Signed-off-by: kim <grufwub@gmail.com>

* fix unlocks of storage keys

Signed-off-by: kim <grufwub@gmail.com>

* whoops, return the error...

Signed-off-by: kim <grufwub@gmail.com>

* pour one out for tobi's code <3

Signed-off-by: kim <grufwub@gmail.com>

* add back the byte slurping code

Signed-off-by: kim <grufwub@gmail.com>

* check for both ErrUnexpectedEOF and EOF

Signed-off-by: kim <grufwub@gmail.com>

* add back links to file format header information

Signed-off-by: kim <grufwub@gmail.com>

Signed-off-by: kim <grufwub@gmail.com>
2023-01-11 12:13:13 +01:00
Sleep 3512325e46
[feature] Add local user and post count to nodeinfo responses ()
* Add local user and post count to nodeinfo responses

This fixes  (at least partially). The nodeinfo endpoint should now
return the total users on an instance, along with their post count.

* Update NodeInfoUsers docstring and swagger yaml file
2023-01-11 11:36:36 +01:00
tobi d6487933c7
[feature] Implement Report database model and utility functions ()
* implement report database model

* implement report cache + config changes

* implement report database functions

* report uri / regex functions

* update envparsing test

* remove unnecessary uri index

* remove unused function + cache lookup

* process error when storing report
2023-01-10 14:19:05 +00:00
kim 1bda6a2002
[bugfix] return early in websocket upgrade handler ()
* launch websocket streaming in goroutine to allow upgrade handler to return

* don't send any message on ping, improved close check on failed read

* use context to signal wsconn close, ensure canceled in read goroutine

Signed-off-by: kim <grufwub@gmail.com>
2023-01-08 11:43:08 +00:00
kim 2bf9bfa24f
[bugfix] fix panic during status delete loop by breaking out early on len(statuses) == 0 ()
Signed-off-by: kim <grufwub@gmail.com>

Signed-off-by: kim <grufwub@gmail.com>
2023-01-08 11:33:03 +01:00
tobi 0dbe6c514f
[chore] Update/add license headers for 2023 () 2023-01-05 12:43:00 +01:00
tobi ff46dd485a
[chore] Fix emoji notnull constraint on initial gtsmodel () 2023-01-05 12:37:50 +01:00