Commit graph

138 commits

Author SHA1 Message Date
tobi 65cd1acbdf
[chore] tidy + update StatusToAPIStatus (#1754) 2023-05-09 13:25:48 +02:00
tobi 0e29f1f5bb
[feature] Enable federation in/out of profile PropertyValue fields (#1722)
Co-authored-by: kim <grufwub@gmail.com>
Co-authored-by: kim <89579420+NyaaaWhatsUpDoc@users.noreply.github.com>
2023-05-09 11:16:10 +01:00
tobi 37b4d9d179
[bugfix] Punycode fixes (#1743)
Co-authored-by: kim <grufwub@gmail.com>
Co-authored-by: kim <89579420+NyaaaWhatsUpDoc@users.noreply.github.com>
2023-05-07 18:53:21 +01:00
Daenney b7dd32da42
[bugfix] Return languages in api/v1/instance (#1741)
It turns out that in Masto v2.3.0 the languages key was added to the V1
Instance and that it's effectively mandatory. Though in GtS we don't
really have this concept yet, some apps will explode if the languages
key is missing altogether. So at least return the empty array on V1 too
in the hopes that it makes things work well enough.

For history's sake, you can see the attributes that will get serialised
in
f877aa9d70/app/serializers/rest/v1/instance_serializer.rb (L6-L9).
Because the attribute does not have a conditional defined for it,
there isn't a filter that optionally omits it, or a def languages to
modify the behaviour the attribute is effectively always included and
serialised.

Fixes: #1662
2023-05-06 22:38:19 +02:00
tobi 4f322f527f
[bugfix] Always serialize orderedItems as array (#1673) 2023-04-06 12:16:53 +01:00
tobi d9bbcc60a6
[bugfix] Fix report serialization errors caused by user delete (#1659)
* [bugfix] Fix report serialization errors caused by user delete

* fix tests
2023-03-31 15:01:29 +02:00
kim de6e3e5f2a
[performance] refactoring + add fave / follow / request / visibility caching (#1607)
* refactor visibility checking, add caching for visibility

* invalidate visibility cache items on account / status deletes

* fix requester ID passed to visibility cache nil ptr

* de-interface caches, fix home / public timeline caching + visibility

* finish adding code comments for visibility filter

* fix angry goconst linter warnings

* actually finish adding filter visibility code comments for timeline functions

* move home timeline status author check to after visibility

* remove now-unused code

* add more code comments

* add TODO code comment, update printed cache start names

* update printed cache names on stop

* start adding separate follow(request) delete db functions, add specific visibility cache tests

* add relationship type caching

* fix getting local account follows / followed-bys, other small codebase improvements

* simplify invalidation using cache hooks, add more GetAccountBy___() functions

* fix boosting to return 404 if not boostable but no error (to not leak status ID)

* remove dead code

* improved placement of cache invalidation

* update license headers

* add example follow, follow-request config entries

* add example visibility cache configuration to config file

* use specific PutFollowRequest() instead of just Put()

* add tests for all GetAccountBy()

* add GetBlockBy() tests

* update block to check primitive fields

* update and finish adding Get{Account,Block,Follow,FollowRequest}By() tests

* fix copy-pasted code

* update envparsing test

* whitespace

* fix bun struct tag

* add license header to gtscontext

* fix old license header

* improved error creation to not use fmt.Errorf() when not needed

* fix various rebase conflicts, fix account test

* remove commented-out code, fix-up mention caching

* fix mention select bun statement

* ensure mention target account populated, pass in context to customrenderer logging

* remove more uncommented code, fix typeutil test

* add statusfave database model caching

* add status fave cache configuration

* add status fave cache example config

* woops, catch missed error. nice catch linter!

* add back testrig panic on nil db

* update example configuration to match defaults, slight tweak to cache configuration defaults

* update envparsing test with new defaults

* fetch followingget to use the follow target account

* use accounnt.IsLocal() instead of empty domain check

* use constants for the cache visibility type check

* use bun.In() for notification type restriction in db query

* include replies when fetching PublicTimeline() (to account for single-author threads in Visibility{}.StatusPublicTimelineable())

* use bun query building for nested select statements to ensure working with postgres

* update public timeline future status checks to match visibility filter

* same as previous, for home timeline

* update public timeline tests to dynamically check for appropriate statuses

* migrate accounts to allow unique constraint on public_key

* provide minimal account with publicKey

---------

Signed-off-by: kim <grufwub@gmail.com>
Co-authored-by: tsmethurst <tobi.smethurst@protonmail.com>
2023-03-28 14:03:14 +01:00
tobi e8595f0c64
[chore] Refactor account deleting/block logic, tidy up some other processing things (#1599)
* start refactoring account deletion

* update to use state.DB

* further messing about

* some more tidying up

* more tidying, cleaning, nice-making

* further adventures in refactoring and the woes of technical debt

* update fr accept/reject

* poking + prodding

* fix up deleting

* create fave uri

* don't log using requestingAccount.ID because it might be nil

* move getBookmarks function

* use exists query to check for status bookmark

* use deletenotifications func

* fiddle

* delete follow request notif

* split up some db functions

* Fix possible nil pointer panic

* fix more possible nil pointers

* fix license headers

* warn when follow missing (target) account

* return wrapped err when bookmark/fave models can't be retrieved

* simplify self account delete

* warn log likely race condition

* de-sillify status delete loop

* move error check due north

* warn when unfollowSideEffects has no target account

* warn when no boost account is found

* warn + dump follow when no account

* more warnings

* warn on fave account not set

* move for loop inside anonymous function

* fix funky logic

* don't remove mutual account items on block;
do make sure unfollow occurs in both directions!
2023-03-20 18:10:08 +00:00
Daenney 5e2bf0bdca
[chore] Improve copyright header handling (#1608)
* [chore] Remove years from all license headers

Years or year ranges aren't required in license headers. Many projects
have removed them in recent years and it avoids a bit of yearly toil.

In many cases our copyright claim was also a bit dodgy since we added
the 2021-2023 header to files created after 2021 but you can't claim
copyright into the past that way.

* [chore] Add license header check

This ensures a license header is always added to any new file. This
avoids maintainers/reviewers needing to remember to check for and ask
for it in case a contribution doesn't include it.

* [chore] Add missing license headers

* [chore] Further updates to license header

* Use the more common // indentend comment format
* Remove the hack we had for the linter now that we use the // format
* Add SPDX license identifier
2023-03-12 16:00:57 +01:00
tobi 5be59f4a25
[bugfix] Federate status delete using just the URI (#1584) 2023-03-03 19:56:34 +00:00
Vyr Cossont e6cde25466
[feature] Advertise rich text formats, support content_type field (#1370)
* Advertise rich text formats, support content_type field

* Update JSON in instance patch tests

* Replace format with content_type everywhere

* update migration to work with both pg and sqlite

* regenerate swagger docs

* update instance serialization + tests

* fix up

* learn to code tobi please, i'm begging you

---------

Co-authored-by: tsmethurst <tobi.smethurst@protonmail.com>
2023-03-02 12:06:40 +01:00
kim baf933cb9f
[chore] move client/federator workerpools to Workers{} (#1575)
* replace concurrency worker pools with base models in State.Workers, update code and tests accordingly

* improve code comment

* change back testrig default log level

* un-comment-out TestAnnounceTwice() and fix

---------

Signed-off-by: kim <grufwub@gmail.com>
Reviewed-by: tobi
2023-03-01 18:26:53 +00:00
tobi 24cec4e7aa
[feature] Federate pinned posts (aka featuredCollection) in and out (#1560)
* start fiddling

* the ol' fiddle + update

* start working on fetching statuses

* poopy doopy doo where r u uwu

* further adventures in featuring statuses

* finishing up

* fmt

* simply status unpin loop

* move empty featured check back to caller function

* remove unnecessary log.WithContext calls

* remove unnecessary IsIRI() checks

* add explanatory comment about status URIs

* change log level to error

* better test names
2023-03-01 17:52:44 +00:00
tobi c27b4d7ed0
[feature] Client API endpoints + v. basic web view for pinned posts (#1547)
* implement status pin client api + web handler

* make test names + comments more descriptive

* don't use separate table for status pins

* remove unused add + remove checking

* tidy up + add some more tests
2023-02-25 12:16:30 +00:00
tobi b6fbdc66c1
[chore] Deinterface processor and subprocessors (#1501)
* [chore] Deinterface processor and subprocessors

* expose subprocessors via function calls

* missing license header
2023-02-22 16:05:26 +01:00
tobi e8a04b7ce1
[bugfix] Fix account roles (#1542)
* Change account role from string to object

* Update tests

* small fixes + swagger docs

---------

Co-authored-by: zowhoey <11893985+zowhoey@users.noreply.github.com>
2023-02-20 17:00:44 +01:00
Daenney b4d18887d3
[chore] Update more log calls to include context (#1517)
In #1476 we updated log.WithFields() but we forgot about
log.WithField(). Also updates a few explicit log.Entry{} creations.
2023-02-17 19:05:43 +00:00
Daenney 68e6d08c76
[feature] Add a request ID and include it in logs (#1476)
This adds a lightweight form of tracing to GTS. Each incoming request is
assigned a Request ID which we then pass on and log in all our log
lines. Any function that gets called downstream from an HTTP handler
should now emit a requestID=value pair whenever it logs something.

Co-authored-by: kim <grufwub@gmail.com>
2023-02-17 12:02:29 +01:00
tobi b5993095fa
[bugfix] Set 'discoverable' properly on API accounts (#1511) 2023-02-16 13:20:23 +00:00
kim 33aee1b1e9
[chore] reformat GetAccount() functionality, support updating accounts based on last_fetch (#1411)
* reformat GetAccount() functionality, and add UpdateAccount() function.

* use fetched_at instead of last_webfingered_at

* catch local "not found" errors. small formatting / error string changes

* remove now unused error type

* return nil when wrapping nil error

* update expected error messages

* return correct url for foss satan webfinger

* add AP model for Some_User

* normalize local domain

* return notretrievable where appropriate

* expose NewErrNotRetrievable

* ensure webfinger for new accounts searched by uri

* update local account short circuit

* allow enrich to fail for already-known accounts

* remove unused LastWebfingeredAt

* expose test maps on mock http client

* update Update test

* reformat GetAccount() functionality, and add UpdateAccount() function.

* use fetched_at instead of last_webfingered_at

* catch local "not found" errors. small formatting / error string changes

* remove nil error checks (we shouldn't be passing nil errors to newError() initializers)

* remove mutex unlock on transport init fail (it hasn't yet been locked!)

* woops add back the error wrapping to use ErrNotRetrievable

* caches were never being started... 🙈

---------

Signed-off-by: kim <grufwub@gmail.com>
Co-authored-by: tsmethurst <tobi.smethurst@protonmail.com>
2023-02-03 20:03:05 +00:00
tobi 271da016b9
[bugfix] Read Bookwyrm Articles more thoroughly (#1410) 2023-02-02 16:41:02 +01:00
tobi 382512a5a6
[feature] Implement /api/v2/instance endpoint (#1409)
* interim: start adding /api/v2/instance

* finish up
2023-02-02 14:08:13 +01:00
tobi 3283900b0d
[feature] Federate reports to remote instance as Flag (if desired) (#1386)
* reports federate out, we did it lxds

* fix optional line start (should be optional slash)
2023-01-27 14:48:11 +01:00
tobi 993aae5e48
[feature] Accept incoming federated Flag activity (#1382)
* start working on handling incoming Flag activity

* interim commit

* federate Flag in successfully
2023-01-25 11:12:27 +01:00
tobi faeb7ded3b
[feature] Implement reports admin API so admins can view + close reports (#1378)
* add admin report api endpoints + tests

* [chore] remove funky duplicate attachment in testrig
2023-01-25 11:12:17 +01:00
tobi e9747247d5
[feature] Implement /api/v1/reports endpoints on client API (#1330)
* start adding report client api

* route + test reports get

* start report create endpoint

* you can create reports now babyy

* stub account report processor

* add single reportGet endpoint

* fix test

* add more filtering params to /api/v1/reports GET

* update swagger

* use marshalIndent in tests

* add + test missing Link info
2023-01-23 13:14:21 +01:00
tobi d4cddf460a
[bugfix] Parse video metadata more accurately; allow Range in fileserver (#1342)
* don't serve unused fields for video attachments

* parse video bitrate + duration more accurately

* use ServeContent where appropriate to respect Range

* abstract temp file seeker into its own function
2023-01-16 16:19:17 +01:00
tobi eafd73c292
[chore] Remove omitempty on account source; refactor tests to use prettyprint json (#1337)
* remove omitEmpty tag on account source items

* update tests
2023-01-13 17:02:45 +01:00
kim 5318054808
[performance] media processing improvements (#1288)
* media processor consolidation and reformatting, reduce amount of required syscalls

Signed-off-by: kim <grufwub@gmail.com>

* update go-store library, stream jpeg/png encoding + use buffer pools, improved media processing AlreadyExists error handling

Signed-off-by: kim <grufwub@gmail.com>

* fix duration not being set, fix mp4 test expecting error

Signed-off-by: kim <grufwub@gmail.com>

* fix test expecting media files with different extension

Signed-off-by: kim <grufwub@gmail.com>

* remove unused code

Signed-off-by: kim <grufwub@gmail.com>

* fix expected storage paths in tests, update expected test thumbnails

Signed-off-by: kim <grufwub@gmail.com>

* remove dead code

Signed-off-by: kim <grufwub@gmail.com>

* fix cached presigned s3 url fetching

Signed-off-by: kim <grufwub@gmail.com>

* fix tests

Signed-off-by: kim <grufwub@gmail.com>

* fix test models

Signed-off-by: kim <grufwub@gmail.com>

* update media processing to use sync.Once{} for concurrency protection

Signed-off-by: kim <grufwub@gmail.com>

* shutup linter

Signed-off-by: kim <grufwub@gmail.com>

* fix passing in KVStore GetStream() as stream to PutStream()

Signed-off-by: kim <grufwub@gmail.com>

* fix unlocks of storage keys

Signed-off-by: kim <grufwub@gmail.com>

* whoops, return the error...

Signed-off-by: kim <grufwub@gmail.com>

* pour one out for tobi's code <3

Signed-off-by: kim <grufwub@gmail.com>

* add back the byte slurping code

Signed-off-by: kim <grufwub@gmail.com>

* check for both ErrUnexpectedEOF and EOF

Signed-off-by: kim <grufwub@gmail.com>

* add back links to file format header information

Signed-off-by: kim <grufwub@gmail.com>

Signed-off-by: kim <grufwub@gmail.com>
2023-01-11 12:13:13 +01:00
tobi 0dbe6c514f
[chore] Update/add license headers for 2023 (#1304) 2023-01-05 12:43:00 +01:00
tobi 941893a774
[chore] The Big Middleware and API Refactor (tm) (#1250)
* interim commit: start refactoring middlewares into package under router

* another interim commit, this is becoming a big job

* another fucking massive interim commit

* refactor bookmarks to new style

* ambassador, wiz zeze commits you are spoiling uz

* she compiles, we're getting there

* we're just normal men; we're just innocent men

* apiutil

* whoopsie

* i'm glad noone reads commit msgs haha :blob_sweat:

* use that weirdo go-bytesize library for maxMultipartMemory

* fix media module paths
2023-01-02 12:10:50 +00:00
tobi 1659f75ae6
[feature] For video attachments, store + return fps, bitrate, duration (#1282)
* start messing about with different mp4 metadata extraction

* heyyooo it works

* add test cow

* move useful multierror to gtserror package

* error out if video doesn't seem to be a real mp4

* test parsing mkv in disguise as mp4

* tidy up error handling

* remove extraneous line

* update framerate formatting

* use float32 for aspect

* fixy mctesterson
2022-12-22 11:48:28 +01:00
tobi 1d24c1c283
[bugfix] Use null for empty api status language (#1268)
* [bugfix] Use null for empty api status `language`

* update swagger docs
2022-12-15 09:41:49 +01:00
Matthew Phillips 477ae50933
[feature] Allow users to create + delete bookbarks, and view bookmarked statuses (#1168)
* Implement Bookmarks

* Update based on review comments

* Update swagger doc

* Fix argument passing to status.Bookmark

* Update changed test

* Updates based on latest PR review
2022-12-09 11:37:12 +01:00
kim e58d2d8122
[chore] move caches to a separate State{} structure (#1078)
* move caches to a separate State{} structure

Signed-off-by: kim <grufwub@gmail.com>

* fix call to log.Panic not using formatted call

Signed-off-by: kim <grufwub@gmail.com>

* move caches to use interfaces, to make switchouts easier in future

Signed-off-by: kim <grufwub@gmail.com>

* fix rebase issue

Signed-off-by: kim <grufwub@gmail.com>

* improve code comment

Signed-off-by: kim <grufwub@gmail.com>

* fix further issues after rebase

Signed-off-by: kim <grufwub@gmail.com>

* heh

Signed-off-by: kim <grufwub@gmail.com>

* add missing license text

Signed-off-by: kim <grufwub@gmail.com>

Signed-off-by: kim <grufwub@gmail.com>
2022-12-08 18:35:14 +01:00
tobi 3a11861ac6
[bugfix] Don't call strings.ToLower() on usernames when selecting account by domain+username (#1190)
* don't lowercase account username when doing a select

* test getting remote user with uppercase username
2022-11-30 17:44:02 +01:00
tobi 927117d8e3
[bugfix] Avoid accidentally marking changed emojis as orphaned + pruning them (#1188)
* add predictable instance account to tests, use it in emoji urls + paths

* use static image url to select emojis when pruning orphaned
2022-11-30 16:20:57 +01:00
kim cfa8d7900c
[bugfix] don't return error during account serialize on deleted emoji (#1177)
* only return error for emoji fetch if NOT errnoentries

Signed-off-by: kim <grufwub@gmail.com>

* reformat gts->api model slice conversion to standard error behaviours and reduce code reuse

Signed-off-by: kim <grufwub@gmail.com>

Signed-off-by: kim <grufwub@gmail.com>
2022-11-29 17:59:59 +00:00
tobi 50dc179d33
[feature] Prune timelines once per hour to plug memory leak (#1117)
* export highest/lowest ULIDs as proper const

* add stop + start to timeline manager, other small fixes

* unexport unused interface funcs + tidy up

* add LastGot func

* add timeline Prune function

* test prune

* update lastGot
2022-11-22 18:38:10 +00:00
tobi a39a664525
[feature] Serialize local account role via API, and show it via web view (#1045)
* [feature] Add 'role' field to api serialization of local accounts

* [chore] Add a bit of license text while I'm here

* [frogend] render account role on same line as username in web view of profile

* style tweaking on role badges, general profile header layout

* profile stats wrapping

* don't render standard 'user' role on web view

Co-authored-by: f0x <f0x@cthu.lu>
2022-11-15 10:19:32 +01:00
tobi 4cd00d546c
[feature] Allow newly uploaded emojis to be placed in categories (#939)
* [feature] Add emoji categories GET
Serialize emojis in appropriate categories; make it possible to get categories via the admin API

* [feature] Create (or use existing) category for new emoji uploads

* fix lint issue

* update misleading line in swagger docs
2022-11-14 22:47:27 +00:00
David Baker 96f11e757c
[bugfix] Fix login on Mastodon iOS app for users with no statuses (#1042)
* Fix login on Mastodon iOS app for users with no statuses

Mastodon for iOS can't cope with an empty string for a date and
expect a JSON `null` instead.

Fixes https://github.com/superseriousbusiness/gotosocial/issues/1010

* Fix expected values in tests to match
2022-11-13 21:38:01 +01:00
tobi b4f7316a4c
[feature] Make instance thumbnail configurable via admin panel (#973)
* [feature] Make instance thumbnail configurable via admin panel

* log db errors in InstanceToAPIInstance

* only update instance in db if necessary

* start adding tests

* finish test
2022-11-08 17:11:06 +00:00
tobi 70d65b683f
[feature] Refetch emojis when they change on remote instances (#905)
* select emoji using image_static_url

* use updated on AP emojis

* allow refetch of updated emojis

* cheeky workaround for test

* clean up old files for refreshed emoji

* check error for originalPostData

* shorten GetEmojiByStaticImageURL

* delete kirby (sorry nintendo)
2022-10-13 15:16:24 +02:00
tobi eb85ef7325
[feature] Add /api/v1/admin/custom_emojis endpoint (#902)
* add admin emojis get path + model + docs

* stub admin emojis get processor function

* add id + disabled fields to admin emoji

* add emoji -> api admin emoji converter

* tidy up a bit

* add GetEmojis function

* finish up get emojis function

* order by shortcodedomain

* ASC

* tidy up + explain

* update to allow paging

* make admin emojis pageable

* fix mixed case paging

* normalize emoji queries a bit better

* test emoji get paging

* make limit optional

* fix incorrect path in media cleanup tests

* i have bad coder syndrome

* don't trimspace

* rename -> GetUseableEmojis

* wrap emoji query in subquery
avoid selecting more than we need

* fix a bit of sillyness teehee

* fix subquery postgres woes
2022-10-12 15:01:42 +02:00
tobi 80663061d8
[feature] Add opt-in RSS feed for account's latest Public posts (#897)
* start adding rss functionality

* add gorilla/feeds dependency

* first bash at building rss feed
still needs work, this is an interim commit

* tidy up a bit

* add publicOnly option to GetAccountLastPosted

* implement rss endpoint

* fix test

* add initial user docs for rss

* update rss logo

* docs update

* add rssFeed to frontend

* feed -> feed.rss

* enableRSS

* increase rss logo size a lil bit

* add rss toggle

* move emojify to text package

* fiddle with rss feed formatting

* add Text field to test statuses

* move status to rss item to typeconverter

* update bun schema for enablerss

* simplify 304 checking

* assume account not rss

* update tests

* update swagger docs

* allow more characters in title, trim nicer

* update last posted to be more consistent
2022-10-08 14:00:39 +02:00
tobi f8528aa689
[feature] Add emoji image size to instance response (#892) 2022-10-06 12:00:53 +02:00
tobi 56f53a2a6f
[performance] add user cache and database (#879)
* go fmt

* add + use user cache and database

* fix import

* update tests

* remove unused relation
2022-10-03 10:46:11 +02:00
kim 1d999712e6
[feature] update config types to use bytesize.Size (#828)
* update config size types to use bytesize.Size

* submit unchecked-out file ... 🤦

* fix bytesize config var decoding

* bump bytesize version

* update kim's libraries in readme

* update envparse.sh to output more useful errors

* improve envparse.sh

* remove reliance on jq

* instead, use uint64 for bytesize flag types

* remove redundant type

* fix viper unmarshaling

* Update envparsing.sh

* fix envparsing test

Signed-off-by: kim <grufwub@gmail.com>
Co-authored-by: tobi <31960611+tsmethurst@users.noreply.github.com>
2022-09-29 21:50:43 +01:00
kim a156188b3e
[chore] update dependencies, bump to Go 1.19.1 (#826)
* update dependencies, bump Go version to 1.19

* bump test image Go version

* update golangci-lint

* update gotosocial-drone-build

* sign

* linting, go fmt

* update swagger docs

* update swagger docs

* whitespace

* update contributing.md

* fuckin whoopsie doopsie

* linterino, linteroni

* fix followrequest test not starting processor

* fix other api/client tests not starting processor

* fix remaining tests where processor not started

* bump go-runners version

* don't check last-webfingered-at, processor may have updated this

* update swagger command

* update bun to latest version

* fix embed to work the same as before with new bun

Signed-off-by: kim <grufwub@gmail.com>
Co-authored-by: tsmethurst <tobi.smethurst@protonmail.com>
2022-09-28 18:30:40 +01:00