Commit graph

1046 commits

Author SHA1 Message Date
kim 3d3e99ae52
[performance] update storage backend and make use of seek syscall when available (#2924)
* update to use go-storage/ instead of go-store/v2/storage/

* pull in latest version from codeberg

* remove test output 😇

* add code comments

* set the exclusive bit when creating new files in disk config

* bump to actual release version

* bump to v0.1.1 (tis a simple no-logic change)

* update readme

* only use a temporary read seeker when decoding video if required (should only be S3 now)

* use fastcopy library to use memory pooled buffers when calling TempFileSeeker()

* update to use seek call in serveFileRange()
2024-05-22 11:46:24 +02:00
kim b092da6d28
[performance] cache v2 filter keyword regular expressions (#2903)
* add caching of filterkeyword regular expressions

* formatting

* fix WholeWord nil check
2024-05-21 14:20:19 +01:00
kim c06e6fb656
[performance] update go-structr and go-mutexes with memory usage improvements (#2909)
* update go-structr and go-mutexes with memory usage improvements

* bump to go-structr v0.8.4
2024-05-13 08:05:46 +00:00
kim f456bd3401
update the total ratios calculation to include ALL caches (previously was missing a few!) (#2907) 2024-05-06 22:29:31 +01:00
Vyr Cossont 45f4afe60e
feature: filters v2 server-side warning/hiding (#2793)
* Remove dead code

* Filter statuses when converting to frontend representation

* status.filtered is an array

* Make matching case-insensitive

* Remove TODOs that don't need to be done now

* Add missing filter check for notification

* lint: rename ErrHideStatus

* APIFilterActionToFilterAction not used yet

* swaggerino docseroni

* Address review comments

* Add apimodel.FilterActionNone

---------

Co-authored-by: tobi <31960611+tsmethurst@users.noreply.github.com>
Co-authored-by: tobi <tobi.smethurst@protonmail.com>
2024-05-06 12:49:08 +01:00
kim d3f6960ba0
close + drain body if response body is too large (#2897) 2024-05-05 16:43:38 +01:00
tobi 6171dcbe51
[feature] Add HTTP header permission section to frontend (#2893)
* [feature] Add HTTP header filter section to frontend

* tweak naming a bit
2024-05-05 11:47:22 +00:00
kim a840f4d49d
add missing caches to the main cache sweep command (#2891) 2024-05-02 14:09:59 +01:00
tobi ebec95a522
[bugfix] Lock when checking/creating notifs to avoid race (#2890)
* [bugfix] Lock when checking/creating notifs to avoid race

* test notif spam
2024-05-02 13:43:00 +01:00
tobi 725a21b027
[feature] Page through accounts as moderator (#2881)
* [feature] Page through accounts as moderator

* aaaaa

* use COLLATE "C" for Postgres to ensure same ordering as SQLite

* fix typo, test paging up

* don't show moderation / info for our instance acct
2024-05-01 14:11:22 +01:00
kim 2300d5e73b
[bugfix] function queue memory pools limitlessly grow (#2882)
* updates the simple queue memory pool to actually self-clean + limit growth

* update memory pool cleaning frequency
2024-05-01 13:30:43 +02:00
kim eb61c783ed
[bugfix] flaky paging test (#2888) 2024-05-01 13:29:42 +02:00
kim a8254a40e7
[bugfix] further paging mishaps (#2884)
* FURTHER paging shenanigans 🥲

* remove cursor logic from ToLinkURL()

* fix up paging tests

---------

Co-authored-by: tobi <tobi.smethurst@protonmail.com>
2024-04-30 16:22:23 +02:00
kim ec7c983e46
[bugfix] retry on http 500 errors *inclusive* (#2886) 2024-04-30 16:18:32 +02:00
kim ec334ece20
[chore] include attemptno in httpclient logs (#2887)
* include request attempt number in httpclient logs

* slightly nicer attempt number formatting
2024-04-30 16:15:50 +02:00
kim 4f87ef246c
[bugfix] paging rel links (#2883)
* fix paging so it uses correct cursor query parameter name

* improved code comment

* whoops, flip the cursoring 🤦

* fix the broken test
2024-04-30 12:19:33 +02:00
tobi 40ece19055
[chore] Fix conflict in workers tests (#2880)
* [chore] Fix conflict in workers tests

* commenty-wenty
2024-04-29 11:43:18 +02:00
kim 48b91ca239
[bugfix] Fix error string typo (#2873) 2024-04-26 15:20:56 +02:00
kim c9c0773f2c
[performance] update remaining worker pools to use queues (#2865)
* start replacing client + federator + media workers with new worker + queue types

* refactor federatingDB.Delete(), drop queued messages when deleting account / status

* move all queue purging to the processor workers

* undo toolchain updates

* code comments, ensure dereferencer worker pool gets started

* update gruf libraries in readme

* start the job scheduler separately to the worker pools

* reshuffle ordering or server.go + remove duplicate worker start / stop

* update go-list version

* fix vendoring

* move queue invalidation to before wipeing / deletion, to ensure queued work not dropped

* add logging to worker processing functions in testrig, don't start workers in unexpected places

* update go-structr to add (+then rely on) QueueCtx{} type

* ensure more worker pools get started properly in tests

* fix remaining broken tests relying on worker queue logic

* fix account test suite queue popping logic, ensure noop workers do not pull from queue

* move back accidentally shuffled account deletion order

* ensure error (non nil!!) gets passed in refactored federatingDB{}.Delete()

* silently drop deletes from accounts not permitted to

* don't warn log on forwarded deletes

* make if else clauses easier to parse

* use getFederatorMsg()

* improved code comment

* improved code comment re: requesting account delete checks

* remove boolean result from worker start / stop since false = already running or already stopped

* remove optional passed-in http.client

* remove worker starting from the admin CLI commands (we don't need to handle side-effects)

* update prune cli to start scheduler but not all of the workers

* fix rebase issues

* remove redundant return statements

* i'm sorry sir linter
2024-04-26 13:50:46 +01:00
Daenney 3a369d834a
[chore] Upgrade our Go version to 1.22 (#2862)
* [chore] Upgrade our Go version to 1.22

With Go 1.22 having been released at the start of February, it's now
been a few months. No major issues have shown up, and the two point
release since then have primarily been security fixes plus some general
bug fixing.

This sets the required Go version to 1.22, as there's nothing in 1.22.1
or 1.22.2 that we would explicitly require. It sets the toolchain to the
latest point release, to ensure we pick up any fixes from there when
building releases etc.

* [chore] Update CI to Go 1.22

* [chore] Update golangci-lint to 1.25.7

Newer version should know about Go 1.22 and run fine.

* [chore] Update Docker container to Go 1.22

* [chore] Update Dockerfile to newer Alpine version

* sign drone.yml

* add missing license header

---------

Co-authored-by: tobi <tobi.smethurst@protonmail.com>
2024-04-26 10:40:29 +02:00
Daenney dcab555a6b
[chore] Update robots.txt (#2856)
This updates the robots.txt based on the list of the ai.robots.txt
repository. We can look at automating that at some point.

It's worth pointing out that some robots, namely the ones by Bytedance,
are known to ignore robots.txt entirely.
2024-04-22 11:01:37 +02:00
tobi b7c629a18a
[bugfix] Fix incorrect field name for status source, add helpful message (#2854)
* [bugfix] Fix incorrect field name for status source, add helpful message

* swagger

* yyammm
2024-04-18 13:22:55 +02:00
tobi 431505b3e4
[feature] Stub conversations endpoint (#2853) 2024-04-18 12:36:02 +02:00
tobi 34d0159ad5
[feature] Stub account mutes endpoint (#2852)
* [feature] Stub account mutes endpoint

* swagger? i barely know 'er!
2024-04-18 09:59:47 +01:00
kim c67bbe5ba0
update to set requesting account when deleting status (#2849) 2024-04-17 16:54:40 +01:00
tobi cef9924d9a
[feature] Status source endpoint (#2848)
* [feature] statusSource endpoint

* finish up
2024-04-17 13:49:20 +01:00
tobi ef16919d4a
[feature] Stub status history endpoint (#2847) 2024-04-17 13:06:49 +01:00
tobi 8cf685fbe9
[bugfix] Fix minor API issue w/ boosted statuses (#2846) 2024-04-17 12:41:40 +02:00
tobi 6de5717d7f
[bugfix] fix get all tokens (#2841) 2024-04-16 13:25:47 +02:00
tobi 3cceed11b2
[feature/performance] Store account stats in separate table (#2831)
* [feature/performance] Store account stats in separate table, get stats from remote

* test account stats

* add some missing increment / decrement calls

* change stats function signatures

* rejig logging a bit

* use lock when updating stats
2024-04-16 13:10:13 +02:00
kim f79d50b9b2
[performance] cached oauth database types (#2838)
* update token + client code to use struct caches

* add code comments

* slight tweak to default mem ratios

* fix envparsing

* add appropriate invalidate hooks

* update the tokenstore sweeping function to rely on caches

* update to use PutClient()

* add ClientID to list of token struct indices
2024-04-15 14:22:21 +01:00
tobi 8b30709791
[chore] Turn accounts-registration-open false by default (#2839) 2024-04-15 14:41:15 +02:00
tobi 89e0cfd874
[feature] Admin accounts endpoints; approve/reject sign-ups (#2826)
* update settings panels, add pending overview + approve/deny functions

* add admin accounts get, approve, reject

* send approved/rejected emails

* use signup URL

* docs!

* email

* swagger

* web linting

* fix email tests

* wee lil fixerinos

* use new paging logic for GetAccounts() series of admin endpoints, small changes to query building

* shuffle useAccountIDIn check *before* adding to query

* fix parse from toot react error

* use `netip.Addr`

* put valid slices in globals

* optimistic updates for account state

---------

Co-authored-by: kim <grufwub@gmail.com>
2024-04-13 13:25:10 +02:00
kim 1439042104
[performance] update GetAccountsByIDs() to use the new multi cache loader endpoint (#2828)
* update GetAccountsByIDs() to use the new multi cache loader endpoint

* fix broken import
2024-04-13 11:36:10 +02:00
tobi a5c6dc9716
[bugfix] Include MIME email headers to avoid mangling non-ascii text (#2827) 2024-04-12 15:18:14 +01:00
tobi 9fb8a78f91
[feature] New user sign-up via web page (#2796)
* [feature] User sign-up form and admin notifs

* add chosen + filtered languages to migration

* remove stray comment

* chosen languages schmosen schmanguages

* proper error on local account missing
2024-04-11 11:45:53 +02:00
kim a483bd9e38
[performance] massively improved ActivityPub delivery worker efficiency (#2812)
* add delivery worker type that pulls from queue to httpclient package

* finish up some code commenting, bodge a vendored activity library change, integrate the deliverypool changes into transportcontroller

* hook up queue deletion logic

* support deleting queued http requests by target ID

* don't index APRequest by hostname in the queue

* use gorun

* use the original context's values when wrapping msg type as delivery{}

* actually log in the AP delivery worker ...

* add uncommitted changes

* use errors.AsV2()

* use errorsv2.AsV2()

* finish adding some code comments, add bad host handling to delivery workers

* slightly tweak deliveryworkerpool API, use advanced sender multiplier

* remove PopCtx() method, let others instead rely on Wait()

* shuffle things around to move delivery stuff into transport/ subpkg

* remove dead code

* formatting

* validate request before queueing for delivery

* finish adding code comments, fix up backoff code

* finish adding more code comments

* clamp minimum no. senders to 1

* add start/stop logging to delivery worker, some slight changes

* remove double logging

* use worker ptrs

* expose the embedded log fields in httpclient.Request{}

* ensure request context values are preserved when updating ctx

* add delivery worker tests

* fix linter issues

* ensure delivery worker gets inited in testrig

* fix tests to delivering messages to check worker delivery queue

* update error type to use ptr instead of value receiver

* fix test calling Workers{}.Start() instead of testrig.StartWorkers()

* update docs for advanced-sender-multiplier

* update to the latest activity library version

* add comment about not using httptest.Server{}
2024-04-11 11:45:35 +02:00
tobi 6db7d014dc
[bugfix] Ensure side effects for local -> local follows get processed (#2820) 2024-04-08 11:51:51 +02:00
kim 83e7847cdf
fix possible nil panic (#2809) 2024-04-04 16:45:59 +01:00
Daenney 236604bd6b
[chore] Log less output on failed test (#2804)
* [chore] Log less output on failed test

This changes the testrig log level to be error by default instead of
info. This makes test failures a lot easier to read, as we don't have
the parade of info logs for each failure to scroll through.

It speeds up the test suite by a couple of seconds since we need
to buffer and flush a lot less messages. On a clean run, so no test
failures, it's about a 3s difference on my machine. Depending on the
amount of test failures, total time saved can vary.

This also introduces a GTS_TESTRIG_LOG_LEVEL environment variable that
we explicitly check for, making it easy to override the log level should
we have a need for it. This would be primarily for running locally, and
not so much as part of go test.

Lastly, it updates the syslog tests to use log.Error because if the log
level is set to error but we call log.Info no message is emitted and we
hang indefinitely on the channel read.

* [chore] Rename the testrig log level env var
2024-04-04 16:42:42 +01:00
Daenney 8ed1b8142c
[bugfix] Sort follows chronologically (#2801)
The id on the follows table is not a ULID, but a random ID. Sorting on
them results in a completely random order. Instead, sort on created_at,
which sould result in a stable and intended sort order.

Fixes: #2769

Co-authored-by: kim <89579420+NyaaaWhatsUpDoc@users.noreply.github.com>
2024-04-03 14:06:39 +01:00
kim 15ede4c1ea
[bugfix] improved authenticate post inbox error handling (#2803)
* improved PostInboxScheme() error handling / logging in case of failed auth

* dumbass kim. returning err instead of errWithCode...

* add checks for the slightly changed error handling in tests, add notes in codebase about the odd way of working
2024-04-03 13:57:07 +01:00
tobi 65b5366031
[bugfix] Set domain for empty-domain Friendica accounts (#2800) 2024-04-02 13:40:54 +01:00
kim e664d0918b
[bugfix] Set the Host header within the signing transport (#2799) 2024-04-02 14:28:36 +02:00
kim d61d5c8a6a
[bugfix] httpclient not signing subsequent redirect requests (#2798)
* move http request signing to transport

* actually hook up the http roundtripper ...

* add code comments for the new gtscontext functions
2024-04-02 13:12:26 +02:00
tobi 4bbdef02f1
[chore] Try to parse public key as both Actor + bare key (#2710)
* [chore] Try to parse public key as both Actor + bare key

* fix weird test fail
2024-04-02 11:30:10 +01:00
kim adf345f1ec
[chore] bump go structr cache version -> v0.6.0 (#2773)
* update go-structr library -> v0.6.0, add necessary wrapping types + code changes to support these changes

* update readme with go-structr package changes

* improved wrapping of the SliceCache type

* add code comments for the cache wrapper types

* remove test.out 😇

---------

Co-authored-by: tobi <31960611+tsmethurst@users.noreply.github.com>
2024-04-02 12:03:40 +02:00
tobi f05874be30
[feature] Option to hide followers/following (#2788) 2024-04-02 10:42:24 +01:00
tobi 4f66dba139
[bugfix] Serve correct URI for AP following collection (#2787) 2024-03-28 12:10:16 +01:00
tobi 8953f57d88
[feature] User-selectable preset CSS themes for accounts (#2777)
* [feature] User-selectable preset themes

* docs, more theme stuff

* lint, tests

* fix css name

* correct some little issues

* add another theme

* fix poll background

* okay last theme i swear

* make retrieval of apimodel themes more conventional

* preallocate stylesheet slices
2024-03-25 17:32:24 +00:00