Commit graph

615 commits

Author SHA1 Message Date
tobi 941893a774
[chore] The Big Middleware and API Refactor (tm) (#1250)
* interim commit: start refactoring middlewares into package under router

* another interim commit, this is becoming a big job

* another fucking massive interim commit

* refactor bookmarks to new style

* ambassador, wiz zeze commits you are spoiling uz

* she compiles, we're getting there

* we're just normal men; we're just innocent men

* apiutil

* whoopsie

* i'm glad noone reads commit msgs haha :blob_sweat:

* use that weirdo go-bytesize library for maxMultipartMemory

* fix media module paths
2023-01-02 12:10:50 +00:00
tobi 1659f75ae6
[feature] For video attachments, store + return fps, bitrate, duration (#1282)
* start messing about with different mp4 metadata extraction

* heyyooo it works

* add test cow

* move useful multierror to gtserror package

* error out if video doesn't seem to be a real mp4

* test parsing mkv in disguise as mp4

* tidy up error handling

* remove extraneous line

* update framerate formatting

* use float32 for aspect

* fixy mctesterson
2022-12-22 11:48:28 +01:00
tobi eabb906268
[bugfix] fix media create error not being checked (#1283) 2022-12-21 17:52:07 +01:00
tobi 6ebdc306ed
[bugfix] Close reader gracefully when streaming recache of remote media to fileserver api caller (#1281)
* close pipereader on failed data function

* gently slurp the bytes

* readability updates

* go fmt

* tidy up file server tests + add more cases

* start moving io wrappers to separate iotools package. Remove use of buffering while piping recache stream

Signed-off-by: kim <grufwub@gmail.com>

* add license text

Signed-off-by: kim <grufwub@gmail.com>
Co-authored-by: kim <grufwub@gmail.com>
2022-12-21 11:17:43 +01:00
tobi 2bbc64be43
[feature] Enable basic video support (mp4 only) (#1274)
* [feature] basic video support

* fix missing semicolon

* replace text shadow with stacked icons

Co-authored-by: f0x <f0x@cthu.lu>
2022-12-17 04:38:56 +00:00
tobi 0f38e7c9b0
[chore] fix some little config whoopsies (#1272) 2022-12-16 22:38:41 +00:00
kim da751c02fb
update go-cache to v3.2.0 with support for ignoring errors (#1273) 2022-12-16 22:36:52 +00:00
Autumn! eb08529f35
[chore/bugfix] Switch markdown from blackfriday to goldmark (#1267)
Co-authored-by: Autumn! <autumnull@posteo.net>
2022-12-16 12:20:22 +01:00
tobi 1d24c1c283
[bugfix] Use null for empty api status language (#1268)
* [bugfix] Use null for empty api status `language`

* update swagger docs
2022-12-15 09:41:49 +01:00
Matthew Phillips d10388cc28
[feature] support Sec-Websocket-Protocol in streaming API (#1254)
* [feature] support Sec-Websocket-Protocol in streaming API

* Fix lint problem

* Update based on reviews
2022-12-14 10:56:42 +01:00
kim 69dd5fed2c
[feature] domain block wildcarding (#1178)
* for domain block lookups, lookup along subdomain parts

Signed-off-by: kim <grufwub@gmail.com>

* only lookup up to a max of 5 domain parts to prevent DOS, limit inserted domains to max of 5 subdomains

Signed-off-by: kim <grufwub@gmail.com>

* add test for domain block wildcarding

Signed-off-by: kim <grufwub@gmail.com>

* check cached status first, increase cached domain time

Signed-off-by: kim <grufwub@gmail.com>

* fix domain wildcard part building logic

Signed-off-by: kim <grufwub@gmail.com>

* create separate domain.BlockCache{} type to hold all domain blocks in memory

Signed-off-by: kim <grufwub@gmail.com>

* remove unused variable

Signed-off-by: kim <grufwub@gmail.com>

* add docs and test to domain block cache, check for domain == host in domain block getter funcs

Signed-off-by: kim <grufwub@gmail.com>

* add license text

Signed-off-by: kim <grufwub@gmail.com>

* check order in which we check primary cache

Signed-off-by: kim <grufwub@gmail.com>

* add better documentation of how domain block checking is performed

Signed-off-by: kim <grufwub@gmail.com>

* change

Signed-off-by: kim <grufwub@gmail.com>

Signed-off-by: kim <grufwub@gmail.com>
2022-12-14 10:55:36 +01:00
Christoph Voigt 8703933df4
[bugfix] fix unordered favorites (#1245)
* [bugfix] fix unordered favorites

* add test for favouritesget

* add license to new test files
2022-12-13 12:33:49 +01:00
kim 58c87bdd7f
[feature] allow uncaching of other media types (#1234)
* simplify pruneRemote, remove unncecessary media trace logging, update RemoteOlderThan() to include headers/avis

Signed-off-by: kim <grufwub@gmail.com>

* cleanup pruneallmeta, add remote header to pruneremote tests

Signed-off-by: kim <grufwub@gmail.com>

* fix olderthan duration additions

Signed-off-by: kim <grufwub@gmail.com>

* fix broken test now that test model header changed

Signed-off-by: kim <grufwub@gmail.com>

* instead use new remote test account for new header model

Signed-off-by: kim <grufwub@gmail.com>

* use newer generated ULID for remote_account_3 to ensure it is sorted last

Signed-off-by: kim <grufwub@gmail.com>

* reorganize serialized keys to match expected test account model order

Signed-off-by: kim <grufwub@gmail.com>

Signed-off-by: kim <grufwub@gmail.com>
2022-12-12 12:22:19 +01:00
kim cb2b2fd805
[feature] support configuring database caches (#1246)
* update config generator to support nested structs, add cache configuration options

* update envparsing test

* add cache configuration to config parse tests

* set cache configuration in testrig

* move caches to sub-cache "gts" namespace, update envparsing, add cache config docs to example config

Signed-off-by: kim <grufwub@gmail.com>
2022-12-11 13:03:15 +00:00
tobi 5e060d0bcb
[feature] Start implementing refetch of lost media files via /api/v1/admin/media_refetch (#1221)
* [chore] Move ShortcodeDomain to its own little util func

* [feature] Add RefetchEmojis function to media manager

* [feature] Expose admin media refresh via admin API

* update following review feedback

- change/fix log levels
- make sure not to try to refetch local emojis
- small style refactoring + comments

* log on emoji refetch start

Signed-off-by: kim <grufwub@gmail.com>
Co-authored-by: kim <grufwub@gmail.com>
2022-12-10 21:43:11 +00:00
Christoph Voigt 199672e586
[bugfix] fix unordered favorites (#1236) 2022-12-09 11:46:52 +01:00
Matthew Phillips 477ae50933
[feature] Allow users to create + delete bookbarks, and view bookmarked statuses (#1168)
* Implement Bookmarks

* Update based on review comments

* Update swagger doc

* Fix argument passing to status.Bookmark

* Update changed test

* Updates based on latest PR review
2022-12-09 11:37:12 +01:00
kim e58d2d8122
[chore] move caches to a separate State{} structure (#1078)
* move caches to a separate State{} structure

Signed-off-by: kim <grufwub@gmail.com>

* fix call to log.Panic not using formatted call

Signed-off-by: kim <grufwub@gmail.com>

* move caches to use interfaces, to make switchouts easier in future

Signed-off-by: kim <grufwub@gmail.com>

* fix rebase issue

Signed-off-by: kim <grufwub@gmail.com>

* improve code comment

Signed-off-by: kim <grufwub@gmail.com>

* fix further issues after rebase

Signed-off-by: kim <grufwub@gmail.com>

* heh

Signed-off-by: kim <grufwub@gmail.com>

* add missing license text

Signed-off-by: kim <grufwub@gmail.com>

Signed-off-by: kim <grufwub@gmail.com>
2022-12-08 18:35:14 +01:00
Dominik Süß 199b685f43
[feature] overhaul the oidc system (#961)
* [feature] overhaul the oidc system

this allows for more flexible username handling and prevents account
takeover using old email addresses

* [feature] add migration path for old OIDC users

* [feature] nicer error reporting for users

* [docs] document the new OIDC flow

* [fix] return early on oidc error

* [docs]: add comments on the finalization logic
2022-12-06 14:15:56 +01:00
Sigrid Solveig Haflínudóttir 1a3f26fb5c
[feature] media: add webp support (#1155)
* media: add webp support

Signed-off-by: Sigrid Solveig Haflínudóttir <sigrid@ftrv.se>

* bump exif-terminator to v0.5.0

Signed-off-by: Sigrid Solveig Haflínudóttir <sigrid@ftrv.se>

Signed-off-by: Sigrid Solveig Haflínudóttir <sigrid@ftrv.se>
2022-12-06 14:15:25 +01:00
Dominik Süß 611df7b22d
[bugfix]: Prevent extension of S3 presigned url TTL (#1208)
Thanks :)
2022-12-05 11:09:22 +01:00
Daniele Sluijters 847e7c7c3a
[chore] Fix a few possible cases of int truncation (#1207)
This fixes a couple of cases where due to int being platform dependent a
value could get truncated if running on 32bits.
2022-12-04 14:20:41 +01:00
Dominik Süß bc917a4085
[performance]: make s3 urls cacheable (#1194)
Implements #864 and should speed up s3 based installations by a lot.

With more static urls, we can then also implement #1026 for even
better performance when used in conjunction with CDNs
2022-12-02 18:40:49 +00:00
tobi 281f712477
[bugfix] Fix admin account confirm on pre-confirmed account setting email address to empty string (#1203) 2022-12-02 17:41:10 +01:00
tobi 34716d7d7d
[bugfix] Add missing continues in emoji get funcs (#1200) 2022-12-02 16:41:05 +01:00
Daniele Sluijters 558b448ab2
[chore] Remove duplicate prefixes from sanitizer (#1195)
In the previous changes that expanded the IPv4 and IPv6 deny lists based
on the IANA registries we inadvertently added a number of duplicates.
This is unnecessary as they're already caught by larger prefixes and
means there's less entries to scan.

This change removes all prefixes that are subnets of other prefixes.
2022-12-01 20:34:55 +01:00
tobi cf20397f26
[bugfix] Use case-insensitive selects when getting remote accounts by username/domain (#1191)
* [bugfix] Case-insensitive account selection

* don't lowercase cache key
2022-12-01 16:06:09 +01:00
Sigrid Solveig Haflínudóttir 5a0e418281
[feature] Support PKCS1 "RSA PUBLIC KEY" pem block type (#1179)
* ap: add support for PKCS1 "RSA PUBLIC KEY" pem block type

Signed-off-by: Sigrid Solveig Haflínudóttir <sigrid@ftrv.se>

* ap: report no PEM data or unknown pem block type

Signed-off-by: Sigrid Solveig Haflínudóttir <sigrid@ftrv.se>

Signed-off-by: Sigrid Solveig Haflínudóttir <sigrid@ftrv.se>
2022-11-30 23:13:13 +01:00
tobi 3a11861ac6
[bugfix] Don't call strings.ToLower() on usernames when selecting account by domain+username (#1190)
* don't lowercase account username when doing a select

* test getting remote user with uppercase username
2022-11-30 17:44:02 +01:00
tobi 927117d8e3
[bugfix] Avoid accidentally marking changed emojis as orphaned + pruning them (#1188)
* add predictable instance account to tests, use it in emoji urls + paths

* use static image url to select emojis when pruning orphaned
2022-11-30 16:20:57 +01:00
kim cfa8d7900c
[bugfix] don't return error during account serialize on deleted emoji (#1177)
* only return error for emoji fetch if NOT errnoentries

Signed-off-by: kim <grufwub@gmail.com>

* reformat gts->api model slice conversion to standard error behaviours and reduce code reuse

Signed-off-by: kim <grufwub@gmail.com>

Signed-off-by: kim <grufwub@gmail.com>
2022-11-29 17:59:59 +00:00
kim d445c60a26
[bugfix] wrap the correct error on failed account update (#1176)
Signed-off-by: kim <grufwub@gmail.com>

Signed-off-by: kim <grufwub@gmail.com>
2022-11-29 18:40:30 +01:00
tobi 97f5453378
[chore] Tidy up some of the search logic (#1082)
* start refactoring some of the search + deref logic

* add tests for search api

* rename GetRemoteAccount + GetRemoteStatus

* make search function a bit simpler + clearer

* fix little fucky wucky uwu owo i'm just a little guy

* update faulty switch statements

* update test to use storage struct

* redo switches for clarity

* reduce repeated logic in search tests

* fastfail getstatus by uri

* debug log + trace log better

* add implementation note

* return early if no result for namestring search

* return + check on dereferencing error types

* errors hah what errors

* remove unneeded error type alias, add custom error text during stringification itself

* fix a woops recursion 🙈

Signed-off-by: kim <grufwub@gmail.com>
Co-authored-by: kim <grufwub@gmail.com>
2022-11-29 09:24:55 +00:00
kim 0c1b1b01f8
fix missing lookup cache key for invalid domain block (#1158) 2022-11-27 14:11:49 +00:00
Daniele Sluijters c534230600
[bugfix] Use Host domain for UA (#1156)
In d6f4d196c9 we swapped to use the
AccountDomain but that actually goes against the intent of the change.
This reverts that change and uses the host domain again.
2022-11-27 00:09:09 +00:00
Daniele Sluijters c5ae88c51b
[chore] Set User-Agent header in transport (#1154)
Currently requests set their own User-Agent. This moves it down to set
it in the transport's do() method, to guarantee it's always set on all
requests.
2022-11-26 20:19:42 +00:00
Daniele Sluijters d6f4d196c9
Tweak the User-Agent a bit (#1153)
* [bugfix] Use AccountDomain for user agent

By using the account domain we can pinpoint the source of the request
more accurately when looking at the User-Agent header.

* [chore] Align user-agent header with spec

Based on RFC 7231, our User-Agent header doesn't quite match. It seems
to always want Name [/ Version] pairs, with comments in parenthesis and
multiple comments separated by a semicolon.

Align our UA with that, using application name first by itself in case
someone has customised it with the source instance in a comment. Follow
that up with gotosocial/<version> and a comment pointing at the source
code.

This also drops the mention of gofed/activity since a fork is in use.
2022-11-26 20:15:19 +00:00
Daniele Sluijters 746f3fa4e6
Additional IP range validations (#1152)
* [bugfix] Ensure requests happen over TCP

It's possible for the network to be udp4 or udp6. This is rather
unlikely to occur, but since we're given the network anyway as part of
the Sanitize function getting called we might as well check for it.

* [chore] Align reserved v6 blocks to IANA registry

* [chore] Add test for ValidateIP

The net and netip packages diverge in that net.ParseIP will consider an
IPv4-mapped address to be an IPv4 address and as such it would get
caught by the IPv4Reserved list. However, netip considers it an IPv6
address, so we need to ensure the mapped range is in IPv6Reserved.

* [chore] Align reserved v4 blocks to IANA registry

This includes a number of tests for /32's explicitly called out in the
registry to ensure we always consider those invalid.
2022-11-26 11:09:55 +00:00
Daniele Sluijters e6cd81babc
[bugfix]: Fix IPv6 validation (#1150)
* [bugfix]: Fix IPv6 validation

The current code considers ff00::/8 valid, but contrary to the comment
that's not the global unicast range. ff-prefixes in IPv6 denote
multicast.

This adapts the code to take the same approach as IPv4, explicitly
blacklisting reserved internal/private ranges.

* [chore] Add missing 4 in IPv4Reserved doc comment
2022-11-25 23:28:03 +00:00
tobi 13e9abd02a
[feature] Add admin media prune orphaned CLI command (#1146)
* add FilePath regex

* add `admin media prune orphaned` command

* add prune orphaned function to media manager

* don't mark flag as required

* document admin media prune orphaned cmd

* oh envparsing.sh you coy minx
2022-11-25 17:23:42 +00:00
tobi b6dbe21026
[feature] PATCH /api/v1/admin/custom_emojis/{id} endpoint (#1061)
* start adding admin emoji PATCH stuff

* updating works OK, now how about copying

* allow emojis to be copied

* update swagger docs

* update admin processer to use non-interface storage driver

* remove shortcode updating for local emojis

* go fmt

Co-authored-by: f0x52 <f0x@cthu.lu>
2022-11-24 18:12:07 +00:00
tobi 3e82196d5e
[bugfix] Fix status boosts giving 404 (#1137)
We broke this at some point recently, and i'm not sure when.

In any case, i updated some of the logic in there + added a test for it.
2022-11-24 16:12:43 +01:00
tobi da8954858a
[bugfix] Prevent future statuses entering timelines (#1134)
* [bugfix] Prevent future statuses entering timeline
Statuses created more than 5 minutes into the future are now rejected in the visibility package.

* Come on buddy
2022-11-24 13:54:49 +01:00
kim fcb9c0bb8b
[chore] cleanup storage implementation, no need for multiple interface types (#1131)
Signed-off-by: kim <grufwub@gmail.com>

Signed-off-by: kim <grufwub@gmail.com>
2022-11-24 09:35:46 +01:00
tobi c9d893fec1
[feature/performance] Fail fast when doing remote transport calls inside incoming request contexts (#1119)
* [feature/performance] Fail fast when doing remote transport calls inside incoming request contexts

* [chore] Reduce outgoing request timeout to 15s

* log error messages when fastfailing

* use context.Value() instead of wrapped context, wrap error with fastfail instead of extra log entry

* add fast-fail context key test

Signed-off-by: kim <grufwub@gmail.com>
Co-authored-by: kim <grufwub@gmail.com>
2022-11-23 21:40:07 +00:00
tobi 1281a9fd4a
[chore/frontend] remove references to unused bundle.js (#1128) 2022-11-23 15:32:57 +01:00
tobi 50dc179d33
[feature] Prune timelines once per hour to plug memory leak (#1117)
* export highest/lowest ULIDs as proper const

* add stop + start to timeline manager, other small fixes

* unexport unused interface funcs + tidy up

* add LastGot func

* add timeline Prune function

* test prune

* update lastGot
2022-11-22 18:38:10 +00:00
kim a898160b0c
[chore] use kv.KVStore also for S3 storage (#1113)
* replace s3 storage implementation to also use kv.KVStore

Signed-off-by: kim <grufwub@gmail.com>

* pull in latest `go-store` fix

Signed-off-by: kim <grufwub@gmail.com>

* pull-in go-store v2.0.9 fixes, update s3 put chunk size to 5MiB

Signed-off-by: kim <grufwub@gmail.com>

Signed-off-by: kim <grufwub@gmail.com>
2022-11-22 14:28:55 +01:00
Norwin f88cb3aebc
[chore] markdown: disable Smartypants rendering (#1111)
* [enhancement] markdown: disable SmartyPantsFractions

fixes #1028

* some fractions are still converted to unicode

to fix that, we might need to disable smartypants indeed in its entirety

* disable smartypants completely

for lack of a better simple solution
2022-11-22 11:08:50 +01:00
kim 4a9538593c
[chore] simplify logging to remove "safetywriter" given os.File has concurrency protection (#1083)
Signed-off-by: kim <grufwub@gmail.com>

Signed-off-by: kim <grufwub@gmail.com>
2022-11-20 17:40:15 +01:00
kim 2eea3d57e0
[chore] improve errors returned fetching account/statues from DB (#1084)
Signed-off-by: kim <grufwub@gmail.com>

Signed-off-by: kim <grufwub@gmail.com>
2022-11-20 17:38:58 +01:00
kim 5d55e8d920
[performance] add account block DB cache and remove block query joins (#1085)
* add account block DB cache and remove reliance on relational joins

* actually include cache key arguments...

* add a PutBlock() method which also updates the block cache, update tests accordingly

* use `PutBlock` instead of `Put(ctx, block)`

* add + use functions for deleting + invalidating blocks

Signed-off-by: kim <grufwub@gmail.com>
Co-authored-by: tsmethurst <tobi.smethurst@protonmail.com>
2022-11-20 16:33:49 +00:00
kim 9be16852f2
[bugfix] fix possible infinite loop on federated AP profile delete (#1091)
* refactor federator account statuses delete to better catch errors, ensure next maxID is always set

Signed-off-by: kim <grufwub@gmail.com>

* fix error statement missing 2nd format operator

Signed-off-by: kim <grufwub@gmail.com>

Signed-off-by: kim <grufwub@gmail.com>
2022-11-20 15:57:19 +01:00
Michael Manfre 67106c9dc4
[feature] Support markdown format for Account bio/note (#1037)
* [feature] Status format also controls bio format

* test
2022-11-19 10:07:51 +01:00
kim 45ae719bd9
[performance] replace account emojis relational query with separate calls to emojiDB to rely on cache (#1074)
Signed-off-by: kim <grufwub@gmail.com>

Signed-off-by: kim <grufwub@gmail.com>
2022-11-18 18:29:25 +01:00
kim dccc2eee81
[performance] replace status query relationals with separate calls in order to rely on caches more (#1073)
Signed-off-by: kim <grufwub@gmail.com>

Signed-off-by: kim <grufwub@gmail.com>
2022-11-18 18:28:33 +01:00
tobi d98a48b446
[performance] don't use relations to select notification structs, use caches instead (#1072) 2022-11-18 17:23:14 +01:00
psa c1824b034b
[chore] Replace blocking Floc with Topics. (#1068)
Google abandoned Floc in favour of Topics and changed the
necessary permissions policy. Currently, the Floc policy will
block Topics. This change includes switching to the updated
policy to be ahead of Google abandoning recognising the Floc
policy.

Update the function documentation to include the current relevant
documentation.
2022-11-17 15:04:06 +01:00
tobi 940abc279c
[chore] reversion: use specific columns for updating user again (#1059) 2022-11-16 11:27:08 +01:00
kim 8598dea98b
[chore] update database caching library (#1040)
* convert most of the caches to use result.Cache{}

* add caching of emojis

* fix issues causing failing tests

* update go-cache/v2 instances with v3

* fix getnotification

* add a note about the left-in StatusCreate comment

* update EmojiCategory db access to use new result.Cache{}

* fix possible panic in getstatusparents

* further proof that kim is not stinky
2022-11-15 18:45:15 +00:00
Mara Sophie Grosch 1f256e288b
[chore] refactor test/cliparsing.sh into a go test below internal/config (#1036)
Also adds AddGlobalFlags and AddServerFlags as methods on ConfigState,
very useful for testing.
2022-11-15 16:53:19 +01:00
ugla 52109776f6
[bugfix] Fix unicode-unaware word boundary check in hashtags (#1049)
* [bugfix] Fix unicode-unaware word boundary check in hashtag regex

Go `\b` does not care for Unicode, and without lookahead, the workarounds got
very ugly. So I replaced the regex with a parser.

The parser runs in O(n) time and performance should not be affected.

* [bugfix] Add back hashtag max length and add tests for it
2022-11-15 16:05:34 +01:00
tobi a39a664525
[feature] Serialize local account role via API, and show it via web view (#1045)
* [feature] Add 'role' field to api serialization of local accounts

* [chore] Add a bit of license text while I'm here

* [frogend] render account role on same line as username in web view of profile

* style tweaking on role badges, general profile header layout

* profile stats wrapping

* don't render standard 'user' role on web view

Co-authored-by: f0x <f0x@cthu.lu>
2022-11-15 10:19:32 +01:00
tobi 4cd00d546c
[feature] Allow newly uploaded emojis to be placed in categories (#939)
* [feature] Add emoji categories GET
Serialize emojis in appropriate categories; make it possible to get categories via the admin API

* [feature] Create (or use existing) category for new emoji uploads

* fix lint issue

* update misleading line in swagger docs
2022-11-14 22:47:27 +00:00
Rob Sargant d120743e8b
[feature] add instance-expose-public-timeline flag (#1039)
* Add instance-expose-public-timeline flag

Adds a config flag that allows unauthenticated access to /api/v1/timelines/public. Defaults to false to replicate existing behaviour.

* Update structure following review

* Add comment

* Fix linting
2022-11-14 10:30:01 +01:00
David Baker 96f11e757c
[bugfix] Fix login on Mastodon iOS app for users with no statuses (#1042)
* Fix login on Mastodon iOS app for users with no statuses

Mastodon for iOS can't cope with an empty string for a date and
expect a JSON `null` instead.

Fixes https://github.com/superseriousbusiness/gotosocial/issues/1010

* Fix expected values in tests to match
2022-11-13 21:38:01 +01:00
kim e8c733da3f
[chore] bump go-cache to v3.1.7 to fix possible issues with zero value keys (#1038)
Signed-off-by: kim <grufwub@gmail.com>

Signed-off-by: kim <grufwub@gmail.com>
2022-11-13 14:02:07 +01:00
tobi 6fb47651c8
[chore] Fixes + updates in emoji dereferencing logic (#1022)
* fix incorrect static remote url use for emojis

* warn when emoji/attachment already exists

* defer emoji postdata execution

* rename ctx to innerCtx for clarity

* warn on emoji too large

* small efficiency fix in fetchRemoteAccountEmojis

* tidy up lock+load

* lock processing emojis

* fix little fucky wucky

* this wasn't go fmted for some reason
2022-11-11 19:27:37 +00:00
tobi edcee14d07
[feature] Read + Write tombstones for deleted Actors (#1005)
* [feature] Read + Write tombstones for deleted Actors

* copyTombstone

* update to use resultcache instead of old ttl cache

Signed-off-by: kim <grufwub@gmail.com>

* update go-cache library to fix result cache capacity / ordering bugs

Signed-off-by: kim <grufwub@gmail.com>

* bump go-cache/v3 to v3.1.6 to fix bugs

Signed-off-by: kim <grufwub@gmail.com>

* switch on status code

* better explain ErrGone reasoning

Signed-off-by: kim <grufwub@gmail.com>
Co-authored-by: kim <grufwub@gmail.com>
2022-11-11 12:18:38 +01:00
Mara Sophie Grosch 948e90b95a
[feature] S3: add config flag to proxy S3 media (#1014)
* S3: add config value "proxy" for not redirecting

Signed-off-by: Mara Sophie Grosch <littlefox@lf-net.org>

* S3: document new config value "proxy"

* S3: add new config value "proxy" to test scripts

Signed-off-by: Mara Sophie Grosch <littlefox@lf-net.org>
2022-11-11 12:03:18 +01:00
kim 3ce0e33f99
[chore] close in-storage media reader _before_ opening write, no need to leave it hanging around (#1016)
Signed-off-by: kim <grufwub@gmail.com>

Signed-off-by: kim <grufwub@gmail.com>
2022-11-11 12:01:53 +01:00
tobi b4f7316a4c
[feature] Make instance thumbnail configurable via admin panel (#973)
* [feature] Make instance thumbnail configurable via admin panel

* log db errors in InstanceToAPIInstance

* only update instance in db if necessary

* start adding tests

* finish test
2022-11-08 17:11:06 +00:00
f0x52 eb25739c34
[frontend] Custom Emoji Deletion (#994)
* re-add eslint

* fix oauth url getting too long

* actually attach single emoji get and delete routes

* basic emoji details + deletion using rtk query

* refactor emoji upload to rtk query

* clean up old redux api+reducers for custom emoji

* fix validation order

* refactor custom emoji form fields

* remove unused requires

* cleanup, fix most eslint errors

* more small eslint fixes

* fix max emoji size

* tiny bit of function documentation
2022-11-08 17:51:44 +01:00
kim 0e57246083
[feature] various worker / request queue improvements (#995)
* greatly simplify httpclient request queuing

Signed-off-by: kim <grufwub@gmail.com>

* improved request queue mutex logic

Signed-off-by: kim <grufwub@gmail.com>

* use improved hashmap library

Signed-off-by: kim <grufwub@gmail.com>

* add warn logging when request queues are full

Signed-off-by: kim <grufwub@gmail.com>

* improve worker pool prefix var naming

Signed-off-by: kim <grufwub@gmail.com>

* improved worker pool error logging

Signed-off-by: kim <grufwub@gmail.com>

* move error message into separate field

Signed-off-by: kim <grufwub@gmail.com>

* remove old log statement

Signed-off-by: kim <grufwub@gmail.com>

* don't export worker message, it gets very spammy :')

Signed-off-by: kim <grufwub@gmail.com>

Signed-off-by: kim <grufwub@gmail.com>
2022-11-08 10:35:24 +01:00
D 53a1e47791
[chore] Fix rel="nofollow" so that it only applies to external links (#991)
Resolves https://github.com/superseriousbusiness/gotosocial/issues/990
2022-11-07 14:25:36 +01:00
kim 05a8baa53a
[bugfix] KVStore doesn't like lost+found directory (#972)
* bump go-store version to v2.0.5, init kv.KVStore without initial clean (as we are using for storage, not as a key-value store)

Signed-off-by: kim <grufwub@gmail.com>

* remove newline

Signed-off-by: kim <grufwub@gmail.com>

Signed-off-by: kim <grufwub@gmail.com>
2022-11-06 13:30:08 +01:00
tobi 4d66fb9603
[feature] Make rate limit requests amount configurable (#966)
* update rate limit documentation

* regenerate landingpage config helpers

* make rate limit rate configurable
2022-11-06 09:47:48 +00:00
tobi 15be35673e
[bugfix] Check media-description-min-chars on submission of new status (#960) 2022-11-05 12:33:38 +00:00
tobi bcb80d3ff4
[chore] bump gruf/go-store to v2 (#953)
* [chore] bump gruf/go-store to v2

* no more boobs
2022-11-05 11:10:19 +00:00
tobi a9addb59b6
[bugfix] Increase field size limits when registering apps (#958) 2022-11-05 11:06:50 +01:00
Adam Harvey ee6a0a1e15
[bugfix] use the landing page user directly (#956)
If set, the landing page user configuration value is used as a Gin
context parameter, which seems incorrect, since a normal request isn't
going to have a parameter named after an arbitrarily configured user.
Instead, the user name should be used directly when building the
redirect URL.
2022-11-05 08:39:36 +01:00
tobi 1dfa7fe0d5
[bugfix] Wrap media in read closer (#941)
* use readcloser for content.Content

* call media postdata function no matter what

* return a readcloser from data func

* tidy of logic of readertostore

* fix whoopsie
2022-11-03 15:03:12 +01:00
tobi bd05040133
[bugfix] Use []rune to check length of user-submitted text (#948) 2022-11-03 14:38:06 +01:00
Mia Heidenstedt f3fc040c2e
[feature] Allow user to show instead of landing page on / (#922)
* configurable user instead of landing page

* rename DefaultUser to LandingPageUser

* code review changes and fix tests

* try to fix pipeline error

* code review changes

* code review changes

* code review changes

* remove unnecessary line
2022-11-03 13:15:31 +01:00
f0x52 c4c713988a
[frontend] change bundler to skulk (#942)
* replace web bundler with skulk

* upgrade skulk

* add license
2022-11-02 16:31:43 +01:00
Terin Stock 4a925e49b1
[bugfix] create admin_account_actions table in tx (#940)
The migration that adds the `admin_account_actions` table did so at the
same time as adding indexes onto the new table. This code was ran inside
a `RunInTx` function, but the table creation did not use the transaction
reference, while the creation of the indexes did. This could cause a
race between the table and index creations, depending on the scheduling
order. If the table creation did not win the race, then the migration
would fail.

This changeset corrects the table creation to also be done inside the
same transaction as the index creation.

Signed-off-by: Terin Stock <terinjokes@gmail.com>

Signed-off-by: Terin Stock <terinjokes@gmail.com>
2022-11-01 15:30:02 +01:00
Mia Heidenstedt 31f0f27bed
[bugfix] Fix imports for helpers.gen.go (#924) 2022-10-29 17:14:57 +02:00
tobi c7ba195907
[bugfix] Prevent future statuses showing in timelines (#932) 2022-10-29 17:10:28 +02:00
tobi ccd5b34c2c
[docs] Use correct form for status submit (#915) 2022-10-17 12:54:36 +02:00
tobi f7416d6e94
[feature] Add emoji DELETE handler at /api/v1/admin/custom_emojis (#913)
* add emoji DELETE handler

* no need to process error (thanks kim)

* don't double check if user is admin

* add missing security annotation
2022-10-14 17:30:04 +02:00
tobi 6a95f5fa67
[feature] Add /api/v1/admin/custom_emojis/{id} endpoint for single emoji GET (#910)
* fix error in prev swagger docs

* add GET for single admin emoji
2022-10-13 16:37:55 +02:00
tobi 70d65b683f
[feature] Refetch emojis when they change on remote instances (#905)
* select emoji using image_static_url

* use updated on AP emojis

* allow refetch of updated emojis

* cheeky workaround for test

* clean up old files for refreshed emoji

* check error for originalPostData

* shorten GetEmojiByStaticImageURL

* delete kirby (sorry nintendo)
2022-10-13 15:16:24 +02:00
tobi eb85ef7325
[feature] Add /api/v1/admin/custom_emojis endpoint (#902)
* add admin emojis get path + model + docs

* stub admin emojis get processor function

* add id + disabled fields to admin emoji

* add emoji -> api admin emoji converter

* tidy up a bit

* add GetEmojis function

* finish up get emojis function

* order by shortcodedomain

* ASC

* tidy up + explain

* update to allow paging

* make admin emojis pageable

* fix mixed case paging

* normalize emoji queries a bit better

* test emoji get paging

* make limit optional

* fix incorrect path in media cleanup tests

* i have bad coder syndrome

* don't trimspace

* rename -> GetUseableEmojis

* wrap emoji query in subquery
avoid selecting more than we need

* fix a bit of sillyness teehee

* fix subquery postgres woes
2022-10-12 15:01:42 +02:00
tobi 832befd727
[chore] Make paging logic more generic (#901)
* make paging logic more generic
not just for timelines!

* linty linterson
2022-10-10 15:52:49 +02:00
tobi 80663061d8
[feature] Add opt-in RSS feed for account's latest Public posts (#897)
* start adding rss functionality

* add gorilla/feeds dependency

* first bash at building rss feed
still needs work, this is an interim commit

* tidy up a bit

* add publicOnly option to GetAccountLastPosted

* implement rss endpoint

* fix test

* add initial user docs for rss

* update rss logo

* docs update

* add rssFeed to frontend

* feed -> feed.rss

* enableRSS

* increase rss logo size a lil bit

* add rss toggle

* move emojify to text package

* fiddle with rss feed formatting

* add Text field to test statuses

* move status to rss item to typeconverter

* update bun schema for enablerss

* simplify 304 checking

* assume account not rss

* update tests

* update swagger docs

* allow more characters in title, trim nicer

* update last posted to be more consistent
2022-10-08 14:00:39 +02:00
tobi aa07750bdb
[chore] Standardize database queries, use bun.Ident() properly (#886)
* use bun.Ident for user queries

* use bun.Ident for account queries

* use bun.Ident for media queries

* add DeleteAccount func

* remove CaseInsensitive in Where+use Ident ipv Safe

* update admin db

* update domain, use ident

* update emoji, use ident

* update instance queries, use bun.Ident

* fix media

* update mentions, use bun ident

* update relationship + tests

* use tableexpr

* add test follows to bun db test suite

* update notifications

* updatebyprimarykey => updatebyid

* fix session

* prefer explicit ID to pk

* fix little fucky wucky

* remove workaround

* use proper db func for attachment selection

* update status db

* add m2m entries in test rig

* fix up timeline

* go fmt

* fix status put issue

* update GetAccountStatuses
2022-10-08 13:50:48 +02:00
kim e58a6a2da3
[performance] cache domains after max retries in transport (#884) 2022-10-08 13:50:16 +02:00
tobi 3bb45b7179
[feature] oob oauth token support (#889)
* move helpful advice into oauth server

* rewrite HandleAuthorizeRequest to allow oob
2022-10-08 13:49:56 +02:00
tobi 5cf0f9950a
[bugfix] Fix new domain block date (#893) 2022-10-06 12:48:17 +02:00
tobi f8528aa689
[feature] Add emoji image size to instance response (#892) 2022-10-06 12:00:53 +02:00