Commit graph

322 commits

Author SHA1 Message Date
John Winston 94c615d417
[feature] Add db-postgres-connection-string option (#3178)
* handle db-url

* lint and add doc

* add more doc

* fix config test

* return error

* change name from db-url to db-postgres-connection-string
2024-08-08 14:00:19 +02:00
kim 0f734a2410
[chore] move PopulateAccountStats() nil check often performed into function itself (#3158)
* move PopulateAccountStats() nil check often performed into function itself

* fix test to take in mind we don't repopulate account stats if not-nil
2024-08-02 13:15:11 +01:00
tobi 38f041cea1
[feature] Allow users to export data via the settings panel (#3140)
* [feature] Allow users to export data via the settings panel

* rename/move some stuff
2024-07-31 15:03:34 +01:00
kim 87cff71af9
[feature] persist worker queues to db (#3042)
* persist queued worker tasks to database on shutdown, fill worker queues from database on startup

* ensure the tasks are sorted by creation time before pushing them

* add migration to insert WorkerTask{} into database, add test for worker task persistence

* add test for recovering worker queues from database

* quick tweak

* whoops we ended up with double cleaner job scheduling

* insert each task separately, because bun is throwing some reflection error??

* add specific checking of cancelled worker contexts

* add http request signing to deliveries recovered from database

* add test for outgoing public key ID being correctly set on delivery

* replace select with Queue.PopCtx()

* get rid of loop now we don't use it

* remove field now we don't use it

* ensure that signing func is set

* header values weren't being copied over 🤦

* use ptr for httpclient.Request in delivery

* move worker queue filling to later in server init process

* fix rebase issues

* make logging less shouty

* use slices.Delete() instead of copying / reslicing

* have database return tasks in ascending order instead of sorting them

* add a 1 minute timeout to persisting worker queues
2024-07-30 13:58:31 +02:00
tobi 42932f9820
[chore] replace UniqueStrings with Deduplicate (#3154) 2024-07-30 12:20:06 +02:00
Vyr Cossont a237e2b295
[feature] Implement following hashtags (#3141)
* Implement followed tags API

* Insert statuses with followed tags into home timelines

* Test following and unfollowing tags

* Correct Swagger path params

* Trim conversation caches

* Migration for followed_tags table

* Followed tag caches and DB implementation

* Lint and tests

* Add missing tag info endpoint, reorganize tag API

* Unwrap boosts when timelining based on tags

* Apply visibility filters to tag followers

* Address review comments
2024-07-29 19:26:31 +01:00
tobi 8ab2b19a94
[feature] Federate interaction policies + Accepts; enforce policies (#3138)
* [feature] Federate interaction policies + Accepts; enforce policies

* use Acceptable type

* fix index

* remove appendIRIStrs

* add GetAccept federatingdb function

* lock on object IRI
2024-07-26 12:04:28 +02:00
kim 63fc9b6c3e
[chore] renames the GTS caches to DB caches (#3127)
* renames the `GTS` caches to `DB` caches, as it better references what they are

* change remaining Caches.GTS uses to Caches.DB
2024-07-24 10:41:43 +02:00
tobi 6533531cf1
[chore] Add some log lines to recent migrations warning not to interrupt (#3134)
* [chore] Add some log lines to recent migrations warning not to interrupt

* arse
2024-07-23 23:06:20 +02:00
Vyr Cossont 86a59db711
Remove content and related fields from boosts (#3131)
These duplicate the content of the target and aren't necessary for anything.

- Stops copying some fields from target when boosting or processing a remote boost
- Adds a migration to null out existing duplicate data
- Updates tests
2024-07-23 20:51:57 +01:00
Vyr Cossont 8fdd358f4b
[feature] Conversations API (#3013)
* Implement conversations API

* Sort and page conversations by last status ID

* Appease linter

* Fix deleting conversations and statuses

* Refactor to make migrations automatic

* Lint

* Update tests post-merge

* Fixes from live-fire testing

* Linter caught a format problem

* Refactor tests, fix cache

* Negative test for non-DMs

* Run conversations advanced migration on testrig startup as well as regular server startup

* Document (lack of) side effects of API method for deleting a conversation

* Make not-found check less nested for readability

* Rename PutConversation to UpsertConversation

* Use util.Ptr instead of IIFE

* Reduce cache used by conversations

* Remove unnecessary TableExpr/ColumnExpr

* Use struct tags for both unique constraints on Conversation

* Make it clear how paging with GetDirectStatusIDsBatch should be used

* Let conversation paging skip conversations it can't render

* Use Bun NewDropTable

* Convert delete raw query to Bun

* Convert update raw query to Bun

* Convert latestConversationStatusesTempTable raw query partially to Bun

* Convert conversationStatusesTempTable raw query partially to Bun

* Rename field used to store result of MaxDirectStatusID

* Move advanced migrations to their own tiny processor

* Catch up util function name with main

* Remove json.… wrappers

* Remove redundant check

* Combine error checks

* Replace map with slice of structs

* Address processor/type converter comments

- Add context info for errors
- Extract some common processor code into shared methods
- Move conversation eligibility check ahead of populating conversation

* Add error context when dropping temp tables
2024-07-23 20:44:31 +01:00
kim 72ba5666a6
[chore] media pipeline improvements (#3110)
* don't set emoji / media image paths on failed download, migrate FileType from string to integer

* fix incorrect uses of util.PtrOr, fix returned frontend media

* fix migration not setting arguments correctly in where clause

* fix not providing default with not null column

* whoops

* ensure a default gets set for media attachment file type

* remove the exclusive flag from writing files in disk storage

* rename PtrOr -> PtrOrZero, and rename PtrValueOr -> PtrOrValue to match

* slight wording changes

* use singular / plural word forms (no parentheses), is better for screen readers

* update testmodels with unknown media type to have unset file details, update attachment focus handling converting to frontend, update tests

* store first instance in ffmpeg wasm pool, fill remaining with closed instances
2024-07-17 15:26:33 +00:00
tobi 9efb11d848
[feature/frontend] Add player for audio files; use thumbnail for poster (#3099)
* [feature/frontend] Audio player for audio media types

* use video preview images for previews instead of video itself

* don't preload

* update tests for new zork status

* collapse media gallery into single row when small
2024-07-15 10:47:57 +01:00
tobi 9adcce9494
[chore] add step-by-step logging to long migration (#3102) 2024-07-15 11:12:56 +02:00
tobi 5bc567196b
[chore] Add interaction policy gtsmodels (#3075)
* [chore] introduce interaction policy gts models

* update migration a smidge

* fix copy paste typo

* update migration

* use int for InteractionType
2024-07-11 15:44:29 +01:00
snow 8f8093aea4
[bugfix] Don't throw error when parent statuses are missing (#2011) (#3088)
* [bugfix] Don't throw error when parent statuses are missing (#2011)

* Split missing parent status case from error check
2024-07-10 14:38:25 +01:00
kim 21bb324156
[chore] media and emoji refactoring (#3000)
* start updating media manager interface ready for storing attachments / emoji right away

* store emoji and media as uncached immediately, then (re-)cache on Processing{}.Load()

* remove now unused media workers

* fix tests and issues

* fix another test!

* fix emoji activitypub uri setting behaviour, fix remainder of test compilation issues

* fix more tests

* fix (most of) remaining tests, add debouncing to repeatedly failing media / emojis

* whoops, rebase issue

* remove kim's whacky experiments

* do some reshuffling, ensure emoji uri gets set

* ensure marked as not cached on cleanup

* tweaks to media / emoji processing to handle context canceled better

* ensure newly fetched emojis actually get set in returned slice

* use different varnames to be a bit more obvious

* move emoji refresh rate limiting to dereferencer

* add exported dereferencer functions for remote media, use these for recaching in processor

* add check for nil attachment in updateAttachment()

* remove unused emoji and media fields + columns

* see previous commit

* fix old migrations expecting image_updated_at to exists (from copies of old models)

* remove freshness checking code (seems to be broken...)

* fix error arg causing nil ptr exception

* finish documentating functions with comments, slight tweaks to media / emoji deref error logic

* remove some extra unneeded boolean checking

* finish writing documentation (code comments) for exported media manager methods

* undo changes to migration snapshot gtsmodels, updated failing migration to have its own snapshot

* move doesColumnExist() to util.go in migrations package
2024-06-26 16:01:16 +01:00
tobi 4ce5c37df5
[bugfix] rename include_types[] to types[] (#3023) 2024-06-18 18:18:35 +02:00
tobi d2b3d37724
[feature/frontend] Reports frontend v2 (#3022)
* use apiutil + paging in admin processor+handlers

* we're making it happen

* fix little whoopsie

* styling for report list

* don't youuuu forget about meee don't don't don't don't

* last bits

* sanitize content before showing in report statuses

* update report docs
2024-06-18 18:18:00 +02:00
Vyr Cossont b08c1bd0cb
[feature] Implement types[] param for notifications (#3009)
Counterpart of exclude_types[].

Also updates Swagger spec for types[] to use the correct param name and enumerate possible values.

Fixes #3003
2024-06-17 20:50:50 +01:00
tobi db803617db
[bugfix] avoid v. long notification clear query (#3007) 2024-06-14 10:14:55 +00:00
tobi 611f9de39b
[bugfix] Deref stats async, serve stub collections if handshaking (#2990)
* [bugfix] Deref stats async, allow peek if handshaking

* don't return totalItems when handshaking or hiding collections

* use GetLimit()

* use StubAccountStats
2024-06-11 11:54:59 +02:00
kim fd6637df4a
[bugfix] boost and account recursion (#2982)
* fix possible infinite recursion if moved accounts are self-referential

* adds a defensive check for a boost being a boost of a boost wrapper

* add checks on input for a boost of a boost

* remove unnecessary check

* add protections on account move to prevent move recursion loops

* separate status conversion without boost logic into separate function to remove risk of recursion

* move boost check to boost function itself

* formatting

* use error 422 instead of 500

* use gtserror not standard errors package for error creation
2024-06-10 19:42:41 +01:00
Vyr Cossont 5e2d4fdb19
[feature] User muting (#2960)
* User muting

* Address review feedback

* Rename uniqueness constraint on user_mutes to match convention

* Remove unused account_id from where clause

* Add UserMute to NewTestDB

* Update test/envparsing.sh with new and fixed cache stuff

* Address tobi's review comments

* Make compiledUserMuteListEntry.expired consistent with UserMute.Expired

* Make sure mute_expires_at is serialized as an explicit null for indefinite mutes

---------

Co-authored-by: tobi <tobi.smethurst@protonmail.com>
2024-06-06 16:38:02 +00:00
kim 5dcc954072
[feature] do not uncache status / emoji media if attached status is bookmarked (#2956)
* do not uncache status / emoji media if attached status is bookmarked

* add status bookmark and bookmark IDs caches

* update status bookmark tests

* move IsStatusBookmarkedBy() to StatusBookmark{} interface, rely on cache

* fix envparsing.sh test
2024-06-06 11:44:43 +01:00
tobi 9caf29bed2
[chore] Fiddle with CI tests; use wasmsqlite3 for CI tests (#2966)
* [chore] Update CI test stuff

* don't bother with ldflags

* use wasmsqlite3
2024-06-05 21:40:08 +02:00
Vyr Cossont 45fe295caa
Compile filter keyword regexps when touched through PutFilter or UpdateFilter (#2951)
Followup to #2903
2024-06-04 10:33:39 +01:00
Vyr Cossont 04bcde08a1
[feature] Add from: search operator and account_id query param (#2943)
* Add from: search operator

* Fix whitespace in Swagger YAML comment

* Move query parsing into its own method

* Document search

* Clarify post search scope
2024-05-31 12:57:42 +02:00
Vyr Cossont 61a8d36255
[feature] Implement Filter API v2 (#2936)
* Use correct entity name

* We support server-side filters now

* Document filter v1 methods that can throw a 409

* Validate v1 filter phrase as filter title

* Always check v1 filter API status codes in tests

* Document keyword minimum requirement on filter API v1

* Make it possible to specify filter keyword update columns per filter keyword

* Implement v2 filter API

* Fix lint and tests

* Update Swagger spec

* Fix filter update test

* Update Swagger spec *correctly*

* Update actual files Swagger spec was generated from

* Remove keywords_attributes and statuses_attributes

* Add test for serialization of empty filter

* More helpful messages when object is owned by wrong account
2024-05-31 12:55:56 +02:00
kim 1e7b32490d
[experiment] add alternative wasm sqlite3 implementation available via build-tag (#2863)
This allows for building GoToSocial with [SQLite transpiled to WASM](https://github.com/ncruces/go-sqlite3) and accessed through [Wazero](https://wazero.io/).
2024-05-27 17:46:15 +02:00
kim 3d3e99ae52
[performance] update storage backend and make use of seek syscall when available (#2924)
* update to use go-storage/ instead of go-store/v2/storage/

* pull in latest version from codeberg

* remove test output 😇

* add code comments

* set the exclusive bit when creating new files in disk config

* bump to actual release version

* bump to v0.1.1 (tis a simple no-logic change)

* update readme

* only use a temporary read seeker when decoding video if required (should only be S3 now)

* use fastcopy library to use memory pooled buffers when calling TempFileSeeker()

* update to use seek call in serveFileRange()
2024-05-22 11:46:24 +02:00
kim b092da6d28
[performance] cache v2 filter keyword regular expressions (#2903)
* add caching of filterkeyword regular expressions

* formatting

* fix WholeWord nil check
2024-05-21 14:20:19 +01:00
tobi 725a21b027
[feature] Page through accounts as moderator (#2881)
* [feature] Page through accounts as moderator

* aaaaa

* use COLLATE "C" for Postgres to ensure same ordering as SQLite

* fix typo, test paging up

* don't show moderation / info for our instance acct
2024-05-01 14:11:22 +01:00
tobi 6de5717d7f
[bugfix] fix get all tokens (#2841) 2024-04-16 13:25:47 +02:00
tobi 3cceed11b2
[feature/performance] Store account stats in separate table (#2831)
* [feature/performance] Store account stats in separate table, get stats from remote

* test account stats

* add some missing increment / decrement calls

* change stats function signatures

* rejig logging a bit

* use lock when updating stats
2024-04-16 13:10:13 +02:00
kim f79d50b9b2
[performance] cached oauth database types (#2838)
* update token + client code to use struct caches

* add code comments

* slight tweak to default mem ratios

* fix envparsing

* add appropriate invalidate hooks

* update the tokenstore sweeping function to rely on caches

* update to use PutClient()

* add ClientID to list of token struct indices
2024-04-15 14:22:21 +01:00
tobi 89e0cfd874
[feature] Admin accounts endpoints; approve/reject sign-ups (#2826)
* update settings panels, add pending overview + approve/deny functions

* add admin accounts get, approve, reject

* send approved/rejected emails

* use signup URL

* docs!

* email

* swagger

* web linting

* fix email tests

* wee lil fixerinos

* use new paging logic for GetAccounts() series of admin endpoints, small changes to query building

* shuffle useAccountIDIn check *before* adding to query

* fix parse from toot react error

* use `netip.Addr`

* put valid slices in globals

* optimistic updates for account state

---------

Co-authored-by: kim <grufwub@gmail.com>
2024-04-13 13:25:10 +02:00
kim 1439042104
[performance] update GetAccountsByIDs() to use the new multi cache loader endpoint (#2828)
* update GetAccountsByIDs() to use the new multi cache loader endpoint

* fix broken import
2024-04-13 11:36:10 +02:00
tobi 9fb8a78f91
[feature] New user sign-up via web page (#2796)
* [feature] User sign-up form and admin notifs

* add chosen + filtered languages to migration

* remove stray comment

* chosen languages schmosen schmanguages

* proper error on local account missing
2024-04-11 11:45:53 +02:00
Daenney 8ed1b8142c
[bugfix] Sort follows chronologically (#2801)
The id on the follows table is not a ULID, but a random ID. Sorting on
them results in a completely random order. Instead, sort on created_at,
which sould result in a stable and intended sort order.

Fixes: #2769

Co-authored-by: kim <89579420+NyaaaWhatsUpDoc@users.noreply.github.com>
2024-04-03 14:06:39 +01:00
tobi 65b5366031
[bugfix] Set domain for empty-domain Friendica accounts (#2800) 2024-04-02 13:40:54 +01:00
kim adf345f1ec
[chore] bump go structr cache version -> v0.6.0 (#2773)
* update go-structr library -> v0.6.0, add necessary wrapping types + code changes to support these changes

* update readme with go-structr package changes

* improved wrapping of the SliceCache type

* add code comments for the cache wrapper types

* remove test.out 😇

---------

Co-authored-by: tobi <31960611+tsmethurst@users.noreply.github.com>
2024-04-02 12:03:40 +02:00
tobi 8953f57d88
[feature] User-selectable preset CSS themes for accounts (#2777)
* [feature] User-selectable preset themes

* docs, more theme stuff

* lint, tests

* fix css name

* correct some little issues

* add another theme

* fix poll background

* okay last theme i swear

* make retrieval of apimodel themes more conventional

* preallocate stylesheet slices
2024-03-25 17:32:24 +00:00
tobi b7b42e832a
[feature] Add healthcheck endpoints /livez and /readyz (#2783)
* [feature] Add healthcheck endpoints `/livez` and `/readyz`

* use select that returns no data
2024-03-25 17:05:14 +00:00
tobi 7f4a0a1aeb
[chore] Move local account settings to separate db table (#2770)
* [chore] Move local account settings to separate database model

* don't use separate settings_id
2024-03-22 14:03:46 +01:00
kim 0767647056
[bugfix] add all possible busy result codes to the sqlite errBusy catching check (#2775) 2024-03-22 13:26:02 +01:00
tobi 9b94b1faa2
[docs] Add database maintenance section; update info message on ANALYZE run (sqlite) (#2756)
* [chore] Limit size of ANALYZE run after migration (sqlite)

* add basic db maintenance tips

* update docs, analyze

* amend info message a wee bit

* update docs/admin/database_maintenance.md wording

Co-authored-by: Daenney <daenney@users.noreply.github.com>

---------

Co-authored-by: kim <89579420+NyaaaWhatsUpDoc@users.noreply.github.com>
Co-authored-by: Daenney <daenney@users.noreply.github.com>
2024-03-14 17:40:36 +00:00
tobi ab2d063fcb
[feature] Process outgoing Move from clientAPI (#2750)
* prevent moved accounts from taking create-type actions

* update move logic

* federate move out

* indicate on web profile when an account has moved

* [docs] Add migration docs section

* lock while checking + setting move state

* use redirectFollowers func for clientAPI as well

* comment typo

* linter? i barely know 'er!

* Update internal/uris/uri.go

Co-authored-by: Daenney <daenney@users.noreply.github.com>

* add a couple tests for move

* fix little mistake exposed by tests (thanks tests)

* ensure Move marked as successful

* attach shared util funcs to struct

* lock whole account when doing move

* move moving check to after error check

* replace repeated text with error func

* linterrrrrr!!!!

* catch self follow case

---------

Co-authored-by: Daenney <daenney@users.noreply.github.com>
2024-03-13 13:53:29 +01:00
tobi 4c155aa847
[bugfix] Fix whitespace move_id issue (#2742) 2024-03-10 13:27:31 +01:00
tobi de3c15a425
[bugfix] Don't error when populating MovedTo if account not found (#2741)
* [bugfix] Don't error when populating MovedTo if account not found

* test the thing
2024-03-10 11:59:55 +01:00