Commit graph

24 commits

Author SHA1 Message Date
kim 23fc70f4e6
[feature] add support for receiving federated status edits (#3597)
* add support for extracting Updated field from Statusable implementers

* add support for status edits in the database, and update status dereferencer to handle them

* remove unused AdditionalInfo{}.CreatedAt

* remove unused AdditionalEmojiInfo{}.CreatedAt

* update new mention creation to use status.UpdatedAt

* remove mention.UpdatedAt, fixes related to NewULIDFromTime() change

* add migration to remove Mention{}.UpdatedAt field

* add migration to add the StatusEdit{} table

* start adding tests, add delete function for status edits

* add more of status edit migrations, fill in more of the necessary edit delete functionality

* remove unused function

* allow generating gotosocial compatible ulid via CLI with `go run ./cmd/gen-ulid`

* add StatusEdit{} test models

* fix new statusedits sql

* use model instead of table name

* actually remove the Mention.UpdatedAt field...

* fix tests now new models are added, add more status edit DB tests

* fix panic wording

* add test for deleting status edits

* don't automatically set `updated_at` field on updated statuses

* flesh out more of the dereferencer status edit tests, ensure updated at field set on outgoing AS statuses

* remove media_attachments.updated_at column

* fix up more tests, further complete the dereferencer status edit tests

* update more status serialization tests not expecting 'updated' AS property

* gah!! json serialization tests!!

* undo some gtscontext wrapping changes

* more serialization test fixing 🥲

* more test fixing, ensure the edit.status_id field is actually set 🤦

* fix status edit test

* grrr linter

* add edited_at field to apimodel status

* remove the choice of paging on the timeline public filtered test (otherwise it needs updating every time you add statuses ...)

* ensure that status.updated_at always fits chronologically

* fix more serialization tests ...

* add more code comments

* fix envparsing

* update swagger file

* properly handle media description changes during status edits

* slight formatting tweak

* code comment
2024-12-05 13:35:07 +00:00
tobi 301543616b
[feature] Add domain permission drafts and excludes (#3547)
* [feature] Add domain permission drafts and excludes

* fix typescript complaining

* lint

* make filenames more consistent

* test own domain excluded
2024-11-21 13:09:58 +00:00
kim 4592e29087
[chore] local instance count query caching, improved status context endpoint logging, don't log ErrHideStatus when timelining (#3330)
* ensure that errors checking status visibility / converting aren't dropped

* add some more context to error messages

* include calling function name in log entries

* don't error on timelining hidden status

* further code to ignore statusfilter.ErrHideStatus type errors

* remove unused error type

* add local instance status / domain / user counts

* add checks for localhost

* rename from InstanceCounts to LocalInstance

* improved code comment
2024-09-23 11:53:42 +00:00
kim 84279f6a6a
[performance] cache more database calls, reduce required database calls overall (#3290)
* improvements to caching for lists and relationship to accounts / follows

* fix nil panic in AddToList()

* ensure list related caches are correctly invalidated

* ensure returned ID lists are ordered correctly

* bump go-structr to v0.8.9 (returns early if zero uncached keys to be loaded)

* remove zero checks in uncached key load functions (go-structr now handles this)

* fix issues after rebase on upstream/main

* update the expected return order of CSV exports (since list entries are now down by entry creation date)

* rename some funcs, allow deleting list entries for multiple follow IDs at a time, fix up more tests

* use returning statements on delete to get cache invalidation info

* fixes to recent database delete changes

* fix broken list entries delete sql

* remove unused db function

* update remainder of delete functions to behave in similar way, some other small tweaks

* fix delete user sql, allow returning on err no entries

* uncomment + fix list database tests

* update remaining list tests

* update envparsing test

* add comments to each specific key being invalidated

* add more cache invalidation explanatory comments

* whoops; actually delete poll votes from database in the DeletePollByID() func

* remove added but-commented-out field

* improved comment regarding paging being disabled

* make cache invalidation comments match what's actually happening

* fix up delete query comments to match what is happening

* rename function to read a bit better

* don't use ErrNoEntries on delete when not needed (it's only needed for a RETURNING call)

* update function name in test

* move list exclusivity check to AFTER eligibility check. use log.Panic() instead of panic()

* use the poll_id column in poll_votes for selecting votes in poll ID

* fix function name
2024-09-16 16:46:09 +00:00
tobi 307d98e386
[feature] Process Reject of interaction via fedi API, put rejected statuses in the "sin bin" 😈 (#3271)
* [feature] Process `Reject` of interaction via fedi API, put rejected statuses in the "sin bin"

* update test

* move nil check back to `rejectStatusIRI`
2024-09-10 12:34:49 +00:00
tobi f23f04e0b1
[feature] Interaction requests client api + settings panel (#3215)
* [feature] Interaction requests client api + settings panel

* test accept / reject

* fmt

* don't pin rejected interaction

* use single db model for interaction accept, reject, and request

* swaggor

* env sharting

* append errors

* remove ErrNoEntries checks

* change intReqID to reqID

* rename "pend" to "request"

* markIntsPending -> mark interactionsPending

* use log instead of returning error when rejecting interaction

* empty migration

* jolly renaming

* make interactionURI unique again

* swag grr

* remove unnecessary locks

* invalidate as last step
2024-08-24 11:49:37 +02:00
Vyr Cossont a237e2b295
[feature] Implement following hashtags (#3141)
* Implement followed tags API

* Insert statuses with followed tags into home timelines

* Test following and unfollowing tags

* Correct Swagger path params

* Trim conversation caches

* Migration for followed_tags table

* Followed tag caches and DB implementation

* Lint and tests

* Add missing tag info endpoint, reorganize tag API

* Unwrap boosts when timelining based on tags

* Apply visibility filters to tag followers

* Address review comments
2024-07-29 19:26:31 +01:00
kim 63fc9b6c3e
[chore] renames the GTS caches to DB caches (#3127)
* renames the `GTS` caches to `DB` caches, as it better references what they are

* change remaining Caches.GTS uses to Caches.DB
2024-07-24 10:41:43 +02:00
Vyr Cossont 8fdd358f4b
[feature] Conversations API (#3013)
* Implement conversations API

* Sort and page conversations by last status ID

* Appease linter

* Fix deleting conversations and statuses

* Refactor to make migrations automatic

* Lint

* Update tests post-merge

* Fixes from live-fire testing

* Linter caught a format problem

* Refactor tests, fix cache

* Negative test for non-DMs

* Run conversations advanced migration on testrig startup as well as regular server startup

* Document (lack of) side effects of API method for deleting a conversation

* Make not-found check less nested for readability

* Rename PutConversation to UpsertConversation

* Use util.Ptr instead of IIFE

* Reduce cache used by conversations

* Remove unnecessary TableExpr/ColumnExpr

* Use struct tags for both unique constraints on Conversation

* Make it clear how paging with GetDirectStatusIDsBatch should be used

* Let conversation paging skip conversations it can't render

* Use Bun NewDropTable

* Convert delete raw query to Bun

* Convert update raw query to Bun

* Convert latestConversationStatusesTempTable raw query partially to Bun

* Convert conversationStatusesTempTable raw query partially to Bun

* Rename field used to store result of MaxDirectStatusID

* Move advanced migrations to their own tiny processor

* Catch up util function name with main

* Remove json.… wrappers

* Remove redundant check

* Combine error checks

* Replace map with slice of structs

* Address processor/type converter comments

- Add context info for errors
- Extract some common processor code into shared methods
- Move conversation eligibility check ahead of populating conversation

* Add error context when dropping temp tables
2024-07-23 20:44:31 +01:00
tobi 5bc567196b
[chore] Add interaction policy gtsmodels (#3075)
* [chore] introduce interaction policy gts models

* update migration a smidge

* fix copy paste typo

* update migration

* use int for InteractionType
2024-07-11 15:44:29 +01:00
tobi 86786ae5b3
[bugfix] Fix typo in instance cache copyF (#3052) 2024-06-28 16:43:00 +02:00
Vyr Cossont 5e2d4fdb19
[feature] User muting (#2960)
* User muting

* Address review feedback

* Rename uniqueness constraint on user_mutes to match convention

* Remove unused account_id from where clause

* Add UserMute to NewTestDB

* Update test/envparsing.sh with new and fixed cache stuff

* Address tobi's review comments

* Make compiledUserMuteListEntry.expired consistent with UserMute.Expired

* Make sure mute_expires_at is serialized as an explicit null for indefinite mutes

---------

Co-authored-by: tobi <tobi.smethurst@protonmail.com>
2024-06-06 16:38:02 +00:00
kim 5dcc954072
[feature] do not uncache status / emoji media if attached status is bookmarked (#2956)
* do not uncache status / emoji media if attached status is bookmarked

* add status bookmark and bookmark IDs caches

* update status bookmark tests

* move IsStatusBookmarkedBy() to StatusBookmark{} interface, rely on cache

* fix envparsing.sh test
2024-06-06 11:44:43 +01:00
tobi fa9a3075a5
[chore/bugfix] Don't cache MovedTo account (#2939) 2024-05-28 15:39:45 +02:00
kim b092da6d28
[performance] cache v2 filter keyword regular expressions (#2903)
* add caching of filterkeyword regular expressions

* formatting

* fix WholeWord nil check
2024-05-21 14:20:19 +01:00
kim f456bd3401
update the total ratios calculation to include ALL caches (previously was missing a few!) (#2907) 2024-05-06 22:29:31 +01:00
tobi 3cceed11b2
[feature/performance] Store account stats in separate table (#2831)
* [feature/performance] Store account stats in separate table, get stats from remote

* test account stats

* add some missing increment / decrement calls

* change stats function signatures

* rejig logging a bit

* use lock when updating stats
2024-04-16 13:10:13 +02:00
kim f79d50b9b2
[performance] cached oauth database types (#2838)
* update token + client code to use struct caches

* add code comments

* slight tweak to default mem ratios

* fix envparsing

* add appropriate invalidate hooks

* update the tokenstore sweeping function to rely on caches

* update to use PutClient()

* add ClientID to list of token struct indices
2024-04-15 14:22:21 +01:00
kim adf345f1ec
[chore] bump go structr cache version -> v0.6.0 (#2773)
* update go-structr library -> v0.6.0, add necessary wrapping types + code changes to support these changes

* update readme with go-structr package changes

* improved wrapping of the SliceCache type

* add code comments for the cache wrapper types

* remove test.out 😇

---------

Co-authored-by: tobi <31960611+tsmethurst@users.noreply.github.com>
2024-04-02 12:03:40 +02:00
tobi 7f4a0a1aeb
[chore] Move local account settings to separate db table (#2770)
* [chore] Move local account settings to separate database model

* don't use separate settings_id
2024-03-22 14:03:46 +01:00
tobi b22e213e15
[feature/chore] Add Move database functions + cache (#2647)
* [feature/chore] Add Move database functions + cache

* add move mem ratio to envparsing.sh

* update comment
2024-03-06 11:18:57 +01:00
Vyr Cossont 61a2b91f45
[feature] Filters v1 (#2594)
* Implement client-side v1 filters

* Exclude linter false positives

* Update test/envparsing.sh

* Fix minor Swagger, style, and Bun usage issues

* Regenerate Swagger

* De-generify filter keywords

* Remove updating filter statuses

This is an operation that the Mastodon v2 filter API doesn't actually have, because filter statuses, unlike keywords, don't have options: the only info they contain is the status ID to be filtered.

* Add a test for filter statuses specifically

* De-generify filter statuses

* Inline FilterEntry

* Use vertical style for Bun operations consistently

* Add comment on Filter DB interface

* Remove GoLand linter control comments

Our existing linters should catch these, or they don't matter very much

* Reduce memory ratio for filters
2024-03-06 11:15:58 +01:00
kim ede8f43635
[performance] temporarily cache account status counts to reduce no. account counts (#2620)
* temporarily cache account status counts to reduce no. account counts

* whoops, forgot to initAccountCounts()

* use already fetched cache capacity value

* make cache a ptr type

* whoops, use count instead of just select

* fix to correctly use the transaction

* properly wrap that tx 😇

* correctly wrap both tx types

* outline retryOnBusy() to allow the fast path to be inlined

* return err on context cancelled

* remove unnecessary storage of context in stmt, fix Exec and Query interface implementations

* shutup linter
2024-02-12 11:52:12 +00:00
kim 7ec1e1332e
[performance] overhaul struct (+ result) caching library for simplicity, performance and multiple-result lookups (#2535)
* rewrite cache library as codeberg.org/gruf/go-structr, implement in gotosocial

* use actual go-structr release version (not just commit hash)

* revert go toolchain changes (damn you go for auto changing this)

* fix go mod woes

* ensure %w is used in calls to errs.Appendf()

* fix error checking

* fix possible panic

* remove unnecessary start/stop functions, move to main Cache{} struct, add note regarding which caches require start/stop

* fix copy-paste artifact... 😇

* fix all comment copy-paste artifacts

* remove dropID() function, now we can just use slices.DeleteFunc()

* use util.Deduplicate() instead of collate(), move collate to util

* move orderByIDs() to util package and "generify"

* add a util.DeleteIf() function, use this to delete entries on failed population

* use slices.DeleteFunc() instead of util.DeleteIf() (i had the logic mixed up in my head somehow lol)

* add note about how collate differs from deduplicate
2024-01-19 12:57:29 +00:00