Commit graph

711 commits

Author SHA1 Message Date
tobi 2bbc64be43
[feature] Enable basic video support (mp4 only) (#1274)
* [feature] basic video support

* fix missing semicolon

* replace text shadow with stacked icons

Co-authored-by: f0x <f0x@cthu.lu>
2022-12-17 04:38:56 +00:00
tobi 0f38e7c9b0
[chore] fix some little config whoopsies (#1272) 2022-12-16 22:38:41 +00:00
kim da751c02fb
update go-cache to v3.2.0 with support for ignoring errors (#1273) 2022-12-16 22:36:52 +00:00
Autumn! eb08529f35
[chore/bugfix] Switch markdown from blackfriday to goldmark (#1267)
Co-authored-by: Autumn! <autumnull@posteo.net>
2022-12-16 12:20:22 +01:00
tobi 1d24c1c283
[bugfix] Use null for empty api status language (#1268)
* [bugfix] Use null for empty api status `language`

* update swagger docs
2022-12-15 09:41:49 +01:00
Matthew Phillips d10388cc28
[feature] support Sec-Websocket-Protocol in streaming API (#1254)
* [feature] support Sec-Websocket-Protocol in streaming API

* Fix lint problem

* Update based on reviews
2022-12-14 10:56:42 +01:00
kim 69dd5fed2c
[feature] domain block wildcarding (#1178)
* for domain block lookups, lookup along subdomain parts

Signed-off-by: kim <grufwub@gmail.com>

* only lookup up to a max of 5 domain parts to prevent DOS, limit inserted domains to max of 5 subdomains

Signed-off-by: kim <grufwub@gmail.com>

* add test for domain block wildcarding

Signed-off-by: kim <grufwub@gmail.com>

* check cached status first, increase cached domain time

Signed-off-by: kim <grufwub@gmail.com>

* fix domain wildcard part building logic

Signed-off-by: kim <grufwub@gmail.com>

* create separate domain.BlockCache{} type to hold all domain blocks in memory

Signed-off-by: kim <grufwub@gmail.com>

* remove unused variable

Signed-off-by: kim <grufwub@gmail.com>

* add docs and test to domain block cache, check for domain == host in domain block getter funcs

Signed-off-by: kim <grufwub@gmail.com>

* add license text

Signed-off-by: kim <grufwub@gmail.com>

* check order in which we check primary cache

Signed-off-by: kim <grufwub@gmail.com>

* add better documentation of how domain block checking is performed

Signed-off-by: kim <grufwub@gmail.com>

* change

Signed-off-by: kim <grufwub@gmail.com>

Signed-off-by: kim <grufwub@gmail.com>
2022-12-14 10:55:36 +01:00
Christoph Voigt 8703933df4
[bugfix] fix unordered favorites (#1245)
* [bugfix] fix unordered favorites

* add test for favouritesget

* add license to new test files
2022-12-13 12:33:49 +01:00
kim 58c87bdd7f
[feature] allow uncaching of other media types (#1234)
* simplify pruneRemote, remove unncecessary media trace logging, update RemoteOlderThan() to include headers/avis

Signed-off-by: kim <grufwub@gmail.com>

* cleanup pruneallmeta, add remote header to pruneremote tests

Signed-off-by: kim <grufwub@gmail.com>

* fix olderthan duration additions

Signed-off-by: kim <grufwub@gmail.com>

* fix broken test now that test model header changed

Signed-off-by: kim <grufwub@gmail.com>

* instead use new remote test account for new header model

Signed-off-by: kim <grufwub@gmail.com>

* use newer generated ULID for remote_account_3 to ensure it is sorted last

Signed-off-by: kim <grufwub@gmail.com>

* reorganize serialized keys to match expected test account model order

Signed-off-by: kim <grufwub@gmail.com>

Signed-off-by: kim <grufwub@gmail.com>
2022-12-12 12:22:19 +01:00
kim cb2b2fd805
[feature] support configuring database caches (#1246)
* update config generator to support nested structs, add cache configuration options

* update envparsing test

* add cache configuration to config parse tests

* set cache configuration in testrig

* move caches to sub-cache "gts" namespace, update envparsing, add cache config docs to example config

Signed-off-by: kim <grufwub@gmail.com>
2022-12-11 13:03:15 +00:00
tobi 5e060d0bcb
[feature] Start implementing refetch of lost media files via /api/v1/admin/media_refetch (#1221)
* [chore] Move ShortcodeDomain to its own little util func

* [feature] Add RefetchEmojis function to media manager

* [feature] Expose admin media refresh via admin API

* update following review feedback

- change/fix log levels
- make sure not to try to refetch local emojis
- small style refactoring + comments

* log on emoji refetch start

Signed-off-by: kim <grufwub@gmail.com>
Co-authored-by: kim <grufwub@gmail.com>
2022-12-10 21:43:11 +00:00
Christoph Voigt 199672e586
[bugfix] fix unordered favorites (#1236) 2022-12-09 11:46:52 +01:00
Matthew Phillips 477ae50933
[feature] Allow users to create + delete bookbarks, and view bookmarked statuses (#1168)
* Implement Bookmarks

* Update based on review comments

* Update swagger doc

* Fix argument passing to status.Bookmark

* Update changed test

* Updates based on latest PR review
2022-12-09 11:37:12 +01:00
kim e58d2d8122
[chore] move caches to a separate State{} structure (#1078)
* move caches to a separate State{} structure

Signed-off-by: kim <grufwub@gmail.com>

* fix call to log.Panic not using formatted call

Signed-off-by: kim <grufwub@gmail.com>

* move caches to use interfaces, to make switchouts easier in future

Signed-off-by: kim <grufwub@gmail.com>

* fix rebase issue

Signed-off-by: kim <grufwub@gmail.com>

* improve code comment

Signed-off-by: kim <grufwub@gmail.com>

* fix further issues after rebase

Signed-off-by: kim <grufwub@gmail.com>

* heh

Signed-off-by: kim <grufwub@gmail.com>

* add missing license text

Signed-off-by: kim <grufwub@gmail.com>

Signed-off-by: kim <grufwub@gmail.com>
2022-12-08 18:35:14 +01:00
Dominik Süß 199b685f43
[feature] overhaul the oidc system (#961)
* [feature] overhaul the oidc system

this allows for more flexible username handling and prevents account
takeover using old email addresses

* [feature] add migration path for old OIDC users

* [feature] nicer error reporting for users

* [docs] document the new OIDC flow

* [fix] return early on oidc error

* [docs]: add comments on the finalization logic
2022-12-06 14:15:56 +01:00
Sigrid Solveig Haflínudóttir 1a3f26fb5c
[feature] media: add webp support (#1155)
* media: add webp support

Signed-off-by: Sigrid Solveig Haflínudóttir <sigrid@ftrv.se>

* bump exif-terminator to v0.5.0

Signed-off-by: Sigrid Solveig Haflínudóttir <sigrid@ftrv.se>

Signed-off-by: Sigrid Solveig Haflínudóttir <sigrid@ftrv.se>
2022-12-06 14:15:25 +01:00
Dominik Süß 611df7b22d
[bugfix]: Prevent extension of S3 presigned url TTL (#1208)
Thanks :)
2022-12-05 11:09:22 +01:00
Daniele Sluijters 847e7c7c3a
[chore] Fix a few possible cases of int truncation (#1207)
This fixes a couple of cases where due to int being platform dependent a
value could get truncated if running on 32bits.
2022-12-04 14:20:41 +01:00
Dominik Süß bc917a4085
[performance]: make s3 urls cacheable (#1194)
Implements #864 and should speed up s3 based installations by a lot.

With more static urls, we can then also implement #1026 for even
better performance when used in conjunction with CDNs
2022-12-02 18:40:49 +00:00
tobi 281f712477
[bugfix] Fix admin account confirm on pre-confirmed account setting email address to empty string (#1203) 2022-12-02 17:41:10 +01:00
tobi 34716d7d7d
[bugfix] Add missing continues in emoji get funcs (#1200) 2022-12-02 16:41:05 +01:00
Daniele Sluijters 558b448ab2
[chore] Remove duplicate prefixes from sanitizer (#1195)
In the previous changes that expanded the IPv4 and IPv6 deny lists based
on the IANA registries we inadvertently added a number of duplicates.
This is unnecessary as they're already caught by larger prefixes and
means there's less entries to scan.

This change removes all prefixes that are subnets of other prefixes.
2022-12-01 20:34:55 +01:00
tobi cf20397f26
[bugfix] Use case-insensitive selects when getting remote accounts by username/domain (#1191)
* [bugfix] Case-insensitive account selection

* don't lowercase cache key
2022-12-01 16:06:09 +01:00
Sigrid Solveig Haflínudóttir 5a0e418281
[feature] Support PKCS1 "RSA PUBLIC KEY" pem block type (#1179)
* ap: add support for PKCS1 "RSA PUBLIC KEY" pem block type

Signed-off-by: Sigrid Solveig Haflínudóttir <sigrid@ftrv.se>

* ap: report no PEM data or unknown pem block type

Signed-off-by: Sigrid Solveig Haflínudóttir <sigrid@ftrv.se>

Signed-off-by: Sigrid Solveig Haflínudóttir <sigrid@ftrv.se>
2022-11-30 23:13:13 +01:00
tobi 3a11861ac6
[bugfix] Don't call strings.ToLower() on usernames when selecting account by domain+username (#1190)
* don't lowercase account username when doing a select

* test getting remote user with uppercase username
2022-11-30 17:44:02 +01:00
tobi 927117d8e3
[bugfix] Avoid accidentally marking changed emojis as orphaned + pruning them (#1188)
* add predictable instance account to tests, use it in emoji urls + paths

* use static image url to select emojis when pruning orphaned
2022-11-30 16:20:57 +01:00
kim cfa8d7900c
[bugfix] don't return error during account serialize on deleted emoji (#1177)
* only return error for emoji fetch if NOT errnoentries

Signed-off-by: kim <grufwub@gmail.com>

* reformat gts->api model slice conversion to standard error behaviours and reduce code reuse

Signed-off-by: kim <grufwub@gmail.com>

Signed-off-by: kim <grufwub@gmail.com>
2022-11-29 17:59:59 +00:00
kim d445c60a26
[bugfix] wrap the correct error on failed account update (#1176)
Signed-off-by: kim <grufwub@gmail.com>

Signed-off-by: kim <grufwub@gmail.com>
2022-11-29 18:40:30 +01:00
tobi 97f5453378
[chore] Tidy up some of the search logic (#1082)
* start refactoring some of the search + deref logic

* add tests for search api

* rename GetRemoteAccount + GetRemoteStatus

* make search function a bit simpler + clearer

* fix little fucky wucky uwu owo i'm just a little guy

* update faulty switch statements

* update test to use storage struct

* redo switches for clarity

* reduce repeated logic in search tests

* fastfail getstatus by uri

* debug log + trace log better

* add implementation note

* return early if no result for namestring search

* return + check on dereferencing error types

* errors hah what errors

* remove unneeded error type alias, add custom error text during stringification itself

* fix a woops recursion 🙈

Signed-off-by: kim <grufwub@gmail.com>
Co-authored-by: kim <grufwub@gmail.com>
2022-11-29 09:24:55 +00:00
kim 0c1b1b01f8
fix missing lookup cache key for invalid domain block (#1158) 2022-11-27 14:11:49 +00:00
Daniele Sluijters c534230600
[bugfix] Use Host domain for UA (#1156)
In d6f4d196c9 we swapped to use the
AccountDomain but that actually goes against the intent of the change.
This reverts that change and uses the host domain again.
2022-11-27 00:09:09 +00:00
Daniele Sluijters c5ae88c51b
[chore] Set User-Agent header in transport (#1154)
Currently requests set their own User-Agent. This moves it down to set
it in the transport's do() method, to guarantee it's always set on all
requests.
2022-11-26 20:19:42 +00:00
Daniele Sluijters d6f4d196c9
Tweak the User-Agent a bit (#1153)
* [bugfix] Use AccountDomain for user agent

By using the account domain we can pinpoint the source of the request
more accurately when looking at the User-Agent header.

* [chore] Align user-agent header with spec

Based on RFC 7231, our User-Agent header doesn't quite match. It seems
to always want Name [/ Version] pairs, with comments in parenthesis and
multiple comments separated by a semicolon.

Align our UA with that, using application name first by itself in case
someone has customised it with the source instance in a comment. Follow
that up with gotosocial/<version> and a comment pointing at the source
code.

This also drops the mention of gofed/activity since a fork is in use.
2022-11-26 20:15:19 +00:00
Daniele Sluijters 746f3fa4e6
Additional IP range validations (#1152)
* [bugfix] Ensure requests happen over TCP

It's possible for the network to be udp4 or udp6. This is rather
unlikely to occur, but since we're given the network anyway as part of
the Sanitize function getting called we might as well check for it.

* [chore] Align reserved v6 blocks to IANA registry

* [chore] Add test for ValidateIP

The net and netip packages diverge in that net.ParseIP will consider an
IPv4-mapped address to be an IPv4 address and as such it would get
caught by the IPv4Reserved list. However, netip considers it an IPv6
address, so we need to ensure the mapped range is in IPv6Reserved.

* [chore] Align reserved v4 blocks to IANA registry

This includes a number of tests for /32's explicitly called out in the
registry to ensure we always consider those invalid.
2022-11-26 11:09:55 +00:00
Daniele Sluijters e6cd81babc
[bugfix]: Fix IPv6 validation (#1150)
* [bugfix]: Fix IPv6 validation

The current code considers ff00::/8 valid, but contrary to the comment
that's not the global unicast range. ff-prefixes in IPv6 denote
multicast.

This adapts the code to take the same approach as IPv4, explicitly
blacklisting reserved internal/private ranges.

* [chore] Add missing 4 in IPv4Reserved doc comment
2022-11-25 23:28:03 +00:00
tobi 13e9abd02a
[feature] Add admin media prune orphaned CLI command (#1146)
* add FilePath regex

* add `admin media prune orphaned` command

* add prune orphaned function to media manager

* don't mark flag as required

* document admin media prune orphaned cmd

* oh envparsing.sh you coy minx
2022-11-25 17:23:42 +00:00
tobi b6dbe21026
[feature] PATCH /api/v1/admin/custom_emojis/{id} endpoint (#1061)
* start adding admin emoji PATCH stuff

* updating works OK, now how about copying

* allow emojis to be copied

* update swagger docs

* update admin processer to use non-interface storage driver

* remove shortcode updating for local emojis

* go fmt

Co-authored-by: f0x52 <f0x@cthu.lu>
2022-11-24 18:12:07 +00:00
tobi 3e82196d5e
[bugfix] Fix status boosts giving 404 (#1137)
We broke this at some point recently, and i'm not sure when.

In any case, i updated some of the logic in there + added a test for it.
2022-11-24 16:12:43 +01:00
tobi da8954858a
[bugfix] Prevent future statuses entering timelines (#1134)
* [bugfix] Prevent future statuses entering timeline
Statuses created more than 5 minutes into the future are now rejected in the visibility package.

* Come on buddy
2022-11-24 13:54:49 +01:00
kim fcb9c0bb8b
[chore] cleanup storage implementation, no need for multiple interface types (#1131)
Signed-off-by: kim <grufwub@gmail.com>

Signed-off-by: kim <grufwub@gmail.com>
2022-11-24 09:35:46 +01:00
tobi c9d893fec1
[feature/performance] Fail fast when doing remote transport calls inside incoming request contexts (#1119)
* [feature/performance] Fail fast when doing remote transport calls inside incoming request contexts

* [chore] Reduce outgoing request timeout to 15s

* log error messages when fastfailing

* use context.Value() instead of wrapped context, wrap error with fastfail instead of extra log entry

* add fast-fail context key test

Signed-off-by: kim <grufwub@gmail.com>
Co-authored-by: kim <grufwub@gmail.com>
2022-11-23 21:40:07 +00:00
tobi 1281a9fd4a
[chore/frontend] remove references to unused bundle.js (#1128) 2022-11-23 15:32:57 +01:00
tobi 50dc179d33
[feature] Prune timelines once per hour to plug memory leak (#1117)
* export highest/lowest ULIDs as proper const

* add stop + start to timeline manager, other small fixes

* unexport unused interface funcs + tidy up

* add LastGot func

* add timeline Prune function

* test prune

* update lastGot
2022-11-22 18:38:10 +00:00
kim a898160b0c
[chore] use kv.KVStore also for S3 storage (#1113)
* replace s3 storage implementation to also use kv.KVStore

Signed-off-by: kim <grufwub@gmail.com>

* pull in latest `go-store` fix

Signed-off-by: kim <grufwub@gmail.com>

* pull-in go-store v2.0.9 fixes, update s3 put chunk size to 5MiB

Signed-off-by: kim <grufwub@gmail.com>

Signed-off-by: kim <grufwub@gmail.com>
2022-11-22 14:28:55 +01:00
Norwin f88cb3aebc
[chore] markdown: disable Smartypants rendering (#1111)
* [enhancement] markdown: disable SmartyPantsFractions

fixes #1028

* some fractions are still converted to unicode

to fix that, we might need to disable smartypants indeed in its entirety

* disable smartypants completely

for lack of a better simple solution
2022-11-22 11:08:50 +01:00
kim 4a9538593c
[chore] simplify logging to remove "safetywriter" given os.File has concurrency protection (#1083)
Signed-off-by: kim <grufwub@gmail.com>

Signed-off-by: kim <grufwub@gmail.com>
2022-11-20 17:40:15 +01:00
kim 2eea3d57e0
[chore] improve errors returned fetching account/statues from DB (#1084)
Signed-off-by: kim <grufwub@gmail.com>

Signed-off-by: kim <grufwub@gmail.com>
2022-11-20 17:38:58 +01:00
kim 5d55e8d920
[performance] add account block DB cache and remove block query joins (#1085)
* add account block DB cache and remove reliance on relational joins

* actually include cache key arguments...

* add a PutBlock() method which also updates the block cache, update tests accordingly

* use `PutBlock` instead of `Put(ctx, block)`

* add + use functions for deleting + invalidating blocks

Signed-off-by: kim <grufwub@gmail.com>
Co-authored-by: tsmethurst <tobi.smethurst@protonmail.com>
2022-11-20 16:33:49 +00:00
kim 9be16852f2
[bugfix] fix possible infinite loop on federated AP profile delete (#1091)
* refactor federator account statuses delete to better catch errors, ensure next maxID is always set

Signed-off-by: kim <grufwub@gmail.com>

* fix error statement missing 2nd format operator

Signed-off-by: kim <grufwub@gmail.com>

Signed-off-by: kim <grufwub@gmail.com>
2022-11-20 15:57:19 +01:00
Michael Manfre 67106c9dc4
[feature] Support markdown format for Account bio/note (#1037)
* [feature] Status format also controls bio format

* test
2022-11-19 10:07:51 +01:00
kim 45ae719bd9
[performance] replace account emojis relational query with separate calls to emojiDB to rely on cache (#1074)
Signed-off-by: kim <grufwub@gmail.com>

Signed-off-by: kim <grufwub@gmail.com>
2022-11-18 18:29:25 +01:00
kim dccc2eee81
[performance] replace status query relationals with separate calls in order to rely on caches more (#1073)
Signed-off-by: kim <grufwub@gmail.com>

Signed-off-by: kim <grufwub@gmail.com>
2022-11-18 18:28:33 +01:00
tobi d98a48b446
[performance] don't use relations to select notification structs, use caches instead (#1072) 2022-11-18 17:23:14 +01:00
psa c1824b034b
[chore] Replace blocking Floc with Topics. (#1068)
Google abandoned Floc in favour of Topics and changed the
necessary permissions policy. Currently, the Floc policy will
block Topics. This change includes switching to the updated
policy to be ahead of Google abandoning recognising the Floc
policy.

Update the function documentation to include the current relevant
documentation.
2022-11-17 15:04:06 +01:00
tobi 940abc279c
[chore] reversion: use specific columns for updating user again (#1059) 2022-11-16 11:27:08 +01:00
kim 8598dea98b
[chore] update database caching library (#1040)
* convert most of the caches to use result.Cache{}

* add caching of emojis

* fix issues causing failing tests

* update go-cache/v2 instances with v3

* fix getnotification

* add a note about the left-in StatusCreate comment

* update EmojiCategory db access to use new result.Cache{}

* fix possible panic in getstatusparents

* further proof that kim is not stinky
2022-11-15 18:45:15 +00:00
Mara Sophie Grosch 1f256e288b
[chore] refactor test/cliparsing.sh into a go test below internal/config (#1036)
Also adds AddGlobalFlags and AddServerFlags as methods on ConfigState,
very useful for testing.
2022-11-15 16:53:19 +01:00
ugla 52109776f6
[bugfix] Fix unicode-unaware word boundary check in hashtags (#1049)
* [bugfix] Fix unicode-unaware word boundary check in hashtag regex

Go `\b` does not care for Unicode, and without lookahead, the workarounds got
very ugly. So I replaced the regex with a parser.

The parser runs in O(n) time and performance should not be affected.

* [bugfix] Add back hashtag max length and add tests for it
2022-11-15 16:05:34 +01:00
tobi a39a664525
[feature] Serialize local account role via API, and show it via web view (#1045)
* [feature] Add 'role' field to api serialization of local accounts

* [chore] Add a bit of license text while I'm here

* [frogend] render account role on same line as username in web view of profile

* style tweaking on role badges, general profile header layout

* profile stats wrapping

* don't render standard 'user' role on web view

Co-authored-by: f0x <f0x@cthu.lu>
2022-11-15 10:19:32 +01:00
tobi 4cd00d546c
[feature] Allow newly uploaded emojis to be placed in categories (#939)
* [feature] Add emoji categories GET
Serialize emojis in appropriate categories; make it possible to get categories via the admin API

* [feature] Create (or use existing) category for new emoji uploads

* fix lint issue

* update misleading line in swagger docs
2022-11-14 22:47:27 +00:00
Rob Sargant d120743e8b
[feature] add instance-expose-public-timeline flag (#1039)
* Add instance-expose-public-timeline flag

Adds a config flag that allows unauthenticated access to /api/v1/timelines/public. Defaults to false to replicate existing behaviour.

* Update structure following review

* Add comment

* Fix linting
2022-11-14 10:30:01 +01:00
David Baker 96f11e757c
[bugfix] Fix login on Mastodon iOS app for users with no statuses (#1042)
* Fix login on Mastodon iOS app for users with no statuses

Mastodon for iOS can't cope with an empty string for a date and
expect a JSON `null` instead.

Fixes https://github.com/superseriousbusiness/gotosocial/issues/1010

* Fix expected values in tests to match
2022-11-13 21:38:01 +01:00
kim e8c733da3f
[chore] bump go-cache to v3.1.7 to fix possible issues with zero value keys (#1038)
Signed-off-by: kim <grufwub@gmail.com>

Signed-off-by: kim <grufwub@gmail.com>
2022-11-13 14:02:07 +01:00
tobi 6fb47651c8
[chore] Fixes + updates in emoji dereferencing logic (#1022)
* fix incorrect static remote url use for emojis

* warn when emoji/attachment already exists

* defer emoji postdata execution

* rename ctx to innerCtx for clarity

* warn on emoji too large

* small efficiency fix in fetchRemoteAccountEmojis

* tidy up lock+load

* lock processing emojis

* fix little fucky wucky

* this wasn't go fmted for some reason
2022-11-11 19:27:37 +00:00
tobi edcee14d07
[feature] Read + Write tombstones for deleted Actors (#1005)
* [feature] Read + Write tombstones for deleted Actors

* copyTombstone

* update to use resultcache instead of old ttl cache

Signed-off-by: kim <grufwub@gmail.com>

* update go-cache library to fix result cache capacity / ordering bugs

Signed-off-by: kim <grufwub@gmail.com>

* bump go-cache/v3 to v3.1.6 to fix bugs

Signed-off-by: kim <grufwub@gmail.com>

* switch on status code

* better explain ErrGone reasoning

Signed-off-by: kim <grufwub@gmail.com>
Co-authored-by: kim <grufwub@gmail.com>
2022-11-11 12:18:38 +01:00
Mara Sophie Grosch 948e90b95a
[feature] S3: add config flag to proxy S3 media (#1014)
* S3: add config value "proxy" for not redirecting

Signed-off-by: Mara Sophie Grosch <littlefox@lf-net.org>

* S3: document new config value "proxy"

* S3: add new config value "proxy" to test scripts

Signed-off-by: Mara Sophie Grosch <littlefox@lf-net.org>
2022-11-11 12:03:18 +01:00
kim 3ce0e33f99
[chore] close in-storage media reader _before_ opening write, no need to leave it hanging around (#1016)
Signed-off-by: kim <grufwub@gmail.com>

Signed-off-by: kim <grufwub@gmail.com>
2022-11-11 12:01:53 +01:00
tobi b4f7316a4c
[feature] Make instance thumbnail configurable via admin panel (#973)
* [feature] Make instance thumbnail configurable via admin panel

* log db errors in InstanceToAPIInstance

* only update instance in db if necessary

* start adding tests

* finish test
2022-11-08 17:11:06 +00:00
f0x52 eb25739c34
[frontend] Custom Emoji Deletion (#994)
* re-add eslint

* fix oauth url getting too long

* actually attach single emoji get and delete routes

* basic emoji details + deletion using rtk query

* refactor emoji upload to rtk query

* clean up old redux api+reducers for custom emoji

* fix validation order

* refactor custom emoji form fields

* remove unused requires

* cleanup, fix most eslint errors

* more small eslint fixes

* fix max emoji size

* tiny bit of function documentation
2022-11-08 17:51:44 +01:00
kim 0e57246083
[feature] various worker / request queue improvements (#995)
* greatly simplify httpclient request queuing

Signed-off-by: kim <grufwub@gmail.com>

* improved request queue mutex logic

Signed-off-by: kim <grufwub@gmail.com>

* use improved hashmap library

Signed-off-by: kim <grufwub@gmail.com>

* add warn logging when request queues are full

Signed-off-by: kim <grufwub@gmail.com>

* improve worker pool prefix var naming

Signed-off-by: kim <grufwub@gmail.com>

* improved worker pool error logging

Signed-off-by: kim <grufwub@gmail.com>

* move error message into separate field

Signed-off-by: kim <grufwub@gmail.com>

* remove old log statement

Signed-off-by: kim <grufwub@gmail.com>

* don't export worker message, it gets very spammy :')

Signed-off-by: kim <grufwub@gmail.com>

Signed-off-by: kim <grufwub@gmail.com>
2022-11-08 10:35:24 +01:00
D 53a1e47791
[chore] Fix rel="nofollow" so that it only applies to external links (#991)
Resolves https://github.com/superseriousbusiness/gotosocial/issues/990
2022-11-07 14:25:36 +01:00
kim 05a8baa53a
[bugfix] KVStore doesn't like lost+found directory (#972)
* bump go-store version to v2.0.5, init kv.KVStore without initial clean (as we are using for storage, not as a key-value store)

Signed-off-by: kim <grufwub@gmail.com>

* remove newline

Signed-off-by: kim <grufwub@gmail.com>

Signed-off-by: kim <grufwub@gmail.com>
2022-11-06 13:30:08 +01:00
tobi 4d66fb9603
[feature] Make rate limit requests amount configurable (#966)
* update rate limit documentation

* regenerate landingpage config helpers

* make rate limit rate configurable
2022-11-06 09:47:48 +00:00
tobi 15be35673e
[bugfix] Check media-description-min-chars on submission of new status (#960) 2022-11-05 12:33:38 +00:00
tobi bcb80d3ff4
[chore] bump gruf/go-store to v2 (#953)
* [chore] bump gruf/go-store to v2

* no more boobs
2022-11-05 11:10:19 +00:00
tobi a9addb59b6
[bugfix] Increase field size limits when registering apps (#958) 2022-11-05 11:06:50 +01:00
Adam Harvey ee6a0a1e15
[bugfix] use the landing page user directly (#956)
If set, the landing page user configuration value is used as a Gin
context parameter, which seems incorrect, since a normal request isn't
going to have a parameter named after an arbitrarily configured user.
Instead, the user name should be used directly when building the
redirect URL.
2022-11-05 08:39:36 +01:00
tobi 1dfa7fe0d5
[bugfix] Wrap media in read closer (#941)
* use readcloser for content.Content

* call media postdata function no matter what

* return a readcloser from data func

* tidy of logic of readertostore

* fix whoopsie
2022-11-03 15:03:12 +01:00
tobi bd05040133
[bugfix] Use []rune to check length of user-submitted text (#948) 2022-11-03 14:38:06 +01:00
Mia Heidenstedt f3fc040c2e
[feature] Allow user to show instead of landing page on / (#922)
* configurable user instead of landing page

* rename DefaultUser to LandingPageUser

* code review changes and fix tests

* try to fix pipeline error

* code review changes

* code review changes

* code review changes

* remove unnecessary line
2022-11-03 13:15:31 +01:00
f0x52 c4c713988a
[frontend] change bundler to skulk (#942)
* replace web bundler with skulk

* upgrade skulk

* add license
2022-11-02 16:31:43 +01:00
Terin Stock 4a925e49b1
[bugfix] create admin_account_actions table in tx (#940)
The migration that adds the `admin_account_actions` table did so at the
same time as adding indexes onto the new table. This code was ran inside
a `RunInTx` function, but the table creation did not use the transaction
reference, while the creation of the indexes did. This could cause a
race between the table and index creations, depending on the scheduling
order. If the table creation did not win the race, then the migration
would fail.

This changeset corrects the table creation to also be done inside the
same transaction as the index creation.

Signed-off-by: Terin Stock <terinjokes@gmail.com>

Signed-off-by: Terin Stock <terinjokes@gmail.com>
2022-11-01 15:30:02 +01:00
Mia Heidenstedt 31f0f27bed
[bugfix] Fix imports for helpers.gen.go (#924) 2022-10-29 17:14:57 +02:00
tobi c7ba195907
[bugfix] Prevent future statuses showing in timelines (#932) 2022-10-29 17:10:28 +02:00
tobi ccd5b34c2c
[docs] Use correct form for status submit (#915) 2022-10-17 12:54:36 +02:00
tobi f7416d6e94
[feature] Add emoji DELETE handler at /api/v1/admin/custom_emojis (#913)
* add emoji DELETE handler

* no need to process error (thanks kim)

* don't double check if user is admin

* add missing security annotation
2022-10-14 17:30:04 +02:00
tobi 6a95f5fa67
[feature] Add /api/v1/admin/custom_emojis/{id} endpoint for single emoji GET (#910)
* fix error in prev swagger docs

* add GET for single admin emoji
2022-10-13 16:37:55 +02:00
tobi 70d65b683f
[feature] Refetch emojis when they change on remote instances (#905)
* select emoji using image_static_url

* use updated on AP emojis

* allow refetch of updated emojis

* cheeky workaround for test

* clean up old files for refreshed emoji

* check error for originalPostData

* shorten GetEmojiByStaticImageURL

* delete kirby (sorry nintendo)
2022-10-13 15:16:24 +02:00
tobi eb85ef7325
[feature] Add /api/v1/admin/custom_emojis endpoint (#902)
* add admin emojis get path + model + docs

* stub admin emojis get processor function

* add id + disabled fields to admin emoji

* add emoji -> api admin emoji converter

* tidy up a bit

* add GetEmojis function

* finish up get emojis function

* order by shortcodedomain

* ASC

* tidy up + explain

* update to allow paging

* make admin emojis pageable

* fix mixed case paging

* normalize emoji queries a bit better

* test emoji get paging

* make limit optional

* fix incorrect path in media cleanup tests

* i have bad coder syndrome

* don't trimspace

* rename -> GetUseableEmojis

* wrap emoji query in subquery
avoid selecting more than we need

* fix a bit of sillyness teehee

* fix subquery postgres woes
2022-10-12 15:01:42 +02:00
tobi 832befd727
[chore] Make paging logic more generic (#901)
* make paging logic more generic
not just for timelines!

* linty linterson
2022-10-10 15:52:49 +02:00
tobi 80663061d8
[feature] Add opt-in RSS feed for account's latest Public posts (#897)
* start adding rss functionality

* add gorilla/feeds dependency

* first bash at building rss feed
still needs work, this is an interim commit

* tidy up a bit

* add publicOnly option to GetAccountLastPosted

* implement rss endpoint

* fix test

* add initial user docs for rss

* update rss logo

* docs update

* add rssFeed to frontend

* feed -> feed.rss

* enableRSS

* increase rss logo size a lil bit

* add rss toggle

* move emojify to text package

* fiddle with rss feed formatting

* add Text field to test statuses

* move status to rss item to typeconverter

* update bun schema for enablerss

* simplify 304 checking

* assume account not rss

* update tests

* update swagger docs

* allow more characters in title, trim nicer

* update last posted to be more consistent
2022-10-08 14:00:39 +02:00
tobi aa07750bdb
[chore] Standardize database queries, use bun.Ident() properly (#886)
* use bun.Ident for user queries

* use bun.Ident for account queries

* use bun.Ident for media queries

* add DeleteAccount func

* remove CaseInsensitive in Where+use Ident ipv Safe

* update admin db

* update domain, use ident

* update emoji, use ident

* update instance queries, use bun.Ident

* fix media

* update mentions, use bun ident

* update relationship + tests

* use tableexpr

* add test follows to bun db test suite

* update notifications

* updatebyprimarykey => updatebyid

* fix session

* prefer explicit ID to pk

* fix little fucky wucky

* remove workaround

* use proper db func for attachment selection

* update status db

* add m2m entries in test rig

* fix up timeline

* go fmt

* fix status put issue

* update GetAccountStatuses
2022-10-08 13:50:48 +02:00
kim e58a6a2da3
[performance] cache domains after max retries in transport (#884) 2022-10-08 13:50:16 +02:00
tobi 3bb45b7179
[feature] oob oauth token support (#889)
* move helpful advice into oauth server

* rewrite HandleAuthorizeRequest to allow oob
2022-10-08 13:49:56 +02:00
tobi 5cf0f9950a
[bugfix] Fix new domain block date (#893) 2022-10-06 12:48:17 +02:00
tobi f8528aa689
[feature] Add emoji image size to instance response (#892) 2022-10-06 12:00:53 +02:00
tobi 359ed1bcb5
[bugfix] Use background context instead of request context for async processing (#888)
Fixes an issue where async processing was not completing correctly.

In particular this applies to side effects of domain blocks: while the domain block was being entered and enforced correctly, side effects like deleting accounts and updating the instance entry for the blocked instance were not. This fixes that :)
2022-10-04 17:50:29 +02:00
f0x52 5249294a16
[chore] Bundler restructure (#880)
* re-structure bundler, settings panel files

* add more info logging

* tidy up CSS syntax errors

* split into lib/ files

* livereloading server

* fix factor function for production builds

* remove testing console.log

* default to production env, saves 300kb bundle size
2022-10-03 16:46:38 +02:00
tobi 56f53a2a6f
[performance] add user cache and database (#879)
* go fmt

* add + use user cache and database

* fix import

* update tests

* remove unused relation
2022-10-03 10:46:11 +02:00
tobi deba75cad1
[chore] Use shorter timestamps in frontend for replies (#875)
* rename timestampShort -> timestampVague

* add ParseISO8601

* start fiddling with timestamp

* pad/margin a bit more consistently

* remove visibilty icon, change timestamp use

* update timestamp logic

* check + log errors

* properly cut-off long display- and usernames

Co-authored-by: f0x <f0x@cthu.lu>
2022-10-02 15:54:42 +02:00
tobi 196e474e43
fix emojis not showing in spoiler text (#878) 2022-10-02 13:01:47 +01:00
tobi 651303d970
[chore] Normalize domain blocks to punycode (#873)
* normalize domain blocks to punycode

* normalize -> normalizeDomain
2022-10-01 17:48:38 +02:00
kim 9fcfe61410
[chore] small logging cleanup (#869)
* handle more syslogging levels, use singular time format variable, add entry .New() function

* pass in calldepth to lower log functions to ensure correctly set

* update truncate length in syslog test
2022-10-01 16:36:08 +01:00
kim 1d999712e6
[feature] update config types to use bytesize.Size (#828)
* update config size types to use bytesize.Size

* submit unchecked-out file ... 🤦

* fix bytesize config var decoding

* bump bytesize version

* update kim's libraries in readme

* update envparse.sh to output more useful errors

* improve envparse.sh

* remove reliance on jq

* instead, use uint64 for bytesize flag types

* remove redundant type

* fix viper unmarshaling

* Update envparsing.sh

* fix envparsing test

Signed-off-by: kim <grufwub@gmail.com>
Co-authored-by: tobi <31960611+tsmethurst@users.noreply.github.com>
2022-09-29 21:50:43 +01:00
tobi f0bf69d4d0
[chore] Add ipv6 localhost to trusted proxies by default (#868)
* add ipv6 localhost to trusted proxies by default

* test different trusted proxies
2022-09-29 17:08:56 +02:00
tobi 97b747b452
[bugfix] Fix incorrect use of bun.Ident (#866)
* remove misused bun.Ident

* test status delete via API

* fix test
2022-09-29 12:33:33 +02:00
tobi dd83ad053c
[feature] Add meta robots tag; allow robots to index profile card if user is Discoverable (#842)
* rework robots.txt response

* don't let robots snippet from statuses/threads

* allow robots to index if user is Discoverable

* add license text
2022-09-29 12:03:17 +02:00
f0x52 938328cd07
[frontend] Unified panels (#812)
* settings panel restructuring

* clean up old Gin handlers

* colorscheme redesign, some other small css tweaks

* basic router layout, error boundary

* colorscheme redesign, some other small css tweaks

* kebab-case consistency

* superfluous padding on applist

* remove unused consts

* redux, whitespace changes..

* use .jsx extensions for components

* login flow up till app registration

* full redux oauth implementation, with basic error handling

* split oauth api functions

* oauth api revocation handling

* basic profile change submission

* move old dir

* profile overview

* fix keeping track of the wrong instance url (for different instance/api domains)

* use redux state for profile form

* delete old/index.js, old/basic.js, fully implemented

* implement old/user/profile.js

* implement password change

* remove debug logging

* support future api for removing files

* customize profile css

* remove unneeded wrapper components

* restructure form fields

* start on admin pages

* admin panel settings

* admin settings panel

* remove old/admin files

* add top-level redirect

* refactor/cleanup forms

* only do API checks on logged-in state

* admin-status based routing

* federation block routing

* federation blocks

* upgrade dependencies

* react 18 changes

* media cleanup

* fix useEffect hooks

* remove unused require

* custom emoji base

* emoji uploader

* delete last old panel files

* sidebar styling, remove unused page

* refactor submit functions

* fix sidebar boxshadow-border

* fix old css variables

* fix fake-toot avatar

* fix non-square emoji

* fix user settings redux keys

* properly get admin account contact from instance response

* Account.source default values

* source.status_format key

* mobile responsiveness

* mobile element tweaks

* proper redirect after removing block

* add redirects for old setting panel urls

* deletes

* fix mobile overflow

* clean up debug logging calls
2022-09-29 12:02:41 +02:00
kim 2f22780800
[chore] simplify generating log entry caller information (#863)
* vastly simplify logging caller information

Signed-off-by: kim <grufwub@gmail.com>

* fix failing test due to multiple calls to processor.Start()

Signed-off-by: kim <grufwub@gmail.com>

Signed-off-by: kim <grufwub@gmail.com>
2022-09-29 12:02:12 +02:00
kim a156188b3e
[chore] update dependencies, bump to Go 1.19.1 (#826)
* update dependencies, bump Go version to 1.19

* bump test image Go version

* update golangci-lint

* update gotosocial-drone-build

* sign

* linting, go fmt

* update swagger docs

* update swagger docs

* whitespace

* update contributing.md

* fuckin whoopsie doopsie

* linterino, linteroni

* fix followrequest test not starting processor

* fix other api/client tests not starting processor

* fix remaining tests where processor not started

* bump go-runners version

* don't check last-webfingered-at, processor may have updated this

* update swagger command

* update bun to latest version

* fix embed to work the same as before with new bun

Signed-off-by: kim <grufwub@gmail.com>
Co-authored-by: tsmethurst <tobi.smethurst@protonmail.com>
2022-09-28 18:30:40 +01:00
tobi 00d38855d4
[bugfix] Fix emphasis being added to emoji shortcodes with markdown parsing (#856)
* fix underscored emoji shortcodes being emphasized

* remove footnote parsing from md
2022-09-27 14:27:53 +02:00
tobi 429bb770e2
[performance] Update indexes that were causing slow db queries (#855)
* add status.url index, use id desc in some indexes

* test account last posted
2022-09-26 14:43:19 +02:00
tobi c4a08292ee
[feature] Show + federate emojis in accounts (#837)
* Start adding account emoji

* get emojis serialized + deserialized nicely

* update tests

* set / retrieve emojis on accounts

* show account emojis in web view

* fetch emojis from db based on ids

* fix typo in test

* lint

* fix pg migration

* update tests

* update emoji checking logic

* update comment

* clarify comments + add some spacing

* tidy up loops a lil (thanks kim)
2022-09-26 11:56:01 +02:00
kim 15a67b7bef
[bugfix] more nil checks baybeeeeeeeeeeeeeeeeeeee (#854)
* more nil checks baybeeeeeeeeeeeeeeeeeeee

Signed-off-by: kim <grufwub@gmail.com>

* move item nil checks

Signed-off-by: kim <grufwub@gmail.com>

Signed-off-by: kim <grufwub@gmail.com>
2022-09-26 10:50:14 +02:00
kim 7bea107608
[bugfix] add so, many, nil, checks (#853)
* add so, many, nil, checks.

* remove comment
2022-09-26 10:14:36 +02:00
kim 82061b1202
[bugfix] panic during child thread iteration (#852)
* *actually* start at top of stack loop on find remote child, fix iter indexing

Signed-off-by: kim <grufwub@gmail.com>

* add improved code comment

Signed-off-by: kim <grufwub@gmail.com>

Signed-off-by: kim <grufwub@gmail.com>
2022-09-26 09:39:59 +02:00
kim 7a1aa04bbb
[bugfix] update thread iterators to not use recursion (#851)
* update thread iterators to not use recursion, rewrote both

Signed-off-by: kim <grufwub@gmail.com>

* fix endless descendant deref, don't error if fetching existing status

Signed-off-by: kim <grufwub@gmail.com>

* don't refetch remote ancestor statuses, improve descendant iter commenting

Signed-off-by: kim <grufwub@gmail.com>

* move collection page next logic so we capture first page of entities

Signed-off-by: kim <grufwub@gmail.com>

* improve log format argument quoting

Signed-off-by: kim <grufwub@gmail.com>

* improve code commenting of collection paging

Signed-off-by: kim <grufwub@gmail.com>

* only dereference announce's originating status if _not_ local. update DereferenceThread() signature. cleanup searchStatusByURI()

Signed-off-by: kim <grufwub@gmail.com>

Signed-off-by: kim <grufwub@gmail.com>
2022-09-25 13:09:41 +02:00
tobi 78409f1985
[bugfix] Wrap media reader in length reader to determine length if no content-length given (#848)
* use lengthReader 2 determine fileSize if not given

* update tests

* small fixes

* go fmt
2022-09-24 11:11:47 +02:00
tobi 69a193dae5
[feature] Allow delivery to sharedInboxes where possible (#847)
* update Activity

* add instance-deliver-to-shared-inboxes setting

* update activity version again

* add SharedInboxURI field to accounts

* serdes for endpoints/sharedInbox

* deliver to sharedInbox if one is available

* update tests

* only assign shared inbox if shared domain

* look for shared inbox if currently nil

* go fmt

* finger to get params.RemoteAccountID if necessary

* make comments clearer

* compare dns more consistently
2022-09-23 20:27:35 +01:00
tobi 4cf76a2bfc
[chore] Tidy up status deletion, remove from cache too (#845)
* add func for deleting status from db + cache

* move deletes entirely back to processor
and also only do a delete if the requesting account owns the item being deleted

* tidy up unboost processing

* delete status more efficiently

* fix wrong account id on remote test attachments

* fix federator test
2022-09-21 19:55:52 +02:00
tobi de26924a4a
don't error out if storage key already exists (#840) 2022-09-19 12:59:11 +01:00
tobi 3777f5c684
[bugfix] Server and closer bugfixes (#839)
* defer streaming from storage more forcefully

* shut down Server more gracefully

* use command context as server BaseContext
2022-09-19 12:43:22 +01:00
tobi c1585d5f8a
[bugfix] Fix domains not being unblockable, log internal server errors from API (#833)
* log internal server errors from 500 api calls

* don't exec into nil dest

* don't exec into nil dest

* log error in router logger not api errorhandling

* update logging a tad

* linter
2022-09-17 19:12:12 +02:00
tobi 1149310673
[bugfix] Fix emojis, attachments, and mentions not being serialized correctly sometimes via AP (#829) 2022-09-15 11:29:05 +02:00
tobi 6a10123790
[docs] Swagger document /api/v1/custom_emojis (#824) 2022-09-13 13:30:07 +02:00
tobi 2b34a9b51c
[bug] fix unique constraint conflict (#820) 2022-09-12 13:58:31 +02:00
tobi b42469e4e0
[feature] Allow users to set custom css for their profiles + threads (#808)
* add custom css account property + db func to fetch

* allow account to get/set custom css

* serve custom css for an account

* go fmt

* use monospace for customcss, add link

* add custom css to account cache

* fix broken field

* add custom css docs to user guide

* add `accounts-allow-custom-css` config flag

* add allow custom css to /api/v1/instance response

* only show/set custom css if allowed to do so

* only set/serve custom account css if enabled

* update swagger docs

* chain promise

* make bool a bit clearer

* use cache for GetAccountCustomCSSByUsername
2022-09-12 13:14:29 +02:00
tobi 268f252e0d
[feature] Fetch + display custom emoji in statuses from remote instances (#807)
* start implementing remote emoji fetcher

* update status where pk

* aaa

* tidy up a little

* check size limits for emojis

* thank you linter, i love you <3

* update swagger docs

* add emoji dereference test

* make emoji max sizes configurable

* normalize db.ErrAlreadyExists
2022-09-12 13:03:23 +02:00
tobi a26280b31a
[bugfix] Fix preposterous characters reserved per URL amount (#809)
* clarify docs

* tidy up consts, set reserved chars much lower

* update tests
2022-09-08 12:36:42 +02:00
f0x52 d09ddb4769
[feature] opengraph meta tags (#806)
* f0x gitignore additions

* better meta title and descriptions

* user avatar icon for thread and profile meta tags

* use proper tag for image

* whitespace

* add noescapeAttr template function

* use ogMeta struct for opengraph

* maxOGDescriptionLength = 300

Co-authored-by: tsmethurst <tobi.smethurst@protonmail.com>
2022-09-07 16:53:12 +02:00
tobi a872ddebe6
[feature] Custom emoji updates (serve emoji via s2s api, tune db models) (#805)
* migrate emojis

* add get emoji to s2s (federation) API

* add new emoji db + cache functions

* add shortcodeDomain lookup for emojis

* check existing emojis w/cache, not w/constraints

* go fmt

* add putEmoji func

* use new db emoji funcs instead of where

* remove emojistringstotags func

* add unique constraint back in

* fix up broken migration

* update index
2022-09-06 12:42:55 +02:00
tobi 2db0c64738
[bugfix] Use 'Image' instead of unrecognized 'Gif' type for media attachments (#801)
* Store gifs as Image type

* remove Gif attachment type, add Gifv type

* update test
2022-09-04 14:58:58 +02:00
tobi 006c8b604b
[feature] Set default header and avatar for API accounts to GtS ones (#799)
* validate web-asset-base-dir

* move default icons into converter

* always ensure avatar + header on api accounts

* update tests

* add default header

* don't return error from web module creation anymore

* tidy a bit

* use pngs for default avatars rather than svgs
2022-09-04 14:41:42 +02:00
tobi 4e13408fd4
[bugfix] Fix status fields in_reply_to_id and in_reply_to_account_id not being nullable (#798)
* make reply status fields nullable pointers

* update tests
2022-09-02 17:00:11 +02:00
kim d68c04a6c0
[performance] cache recently allowed/denied domains to cut down on db calls (#794)
* fetch creation and fetching domain blocks from db

Signed-off-by: kim <grufwub@gmail.com>

* add separate domainblock cache type, handle removing block from cache on delete

Signed-off-by: kim <grufwub@gmail.com>

* fix sentinel nil values being passed into cache

Signed-off-by: kim <grufwub@gmail.com>

Signed-off-by: kim <grufwub@gmail.com>
2022-09-02 12:17:46 +02:00
Blackle Morisanchetto bf9d146987
[feature] Federate custom emoji (outbound) (#791)
* Federate local custom emoji

* Add test for converting a status with tags to AP
2022-09-02 12:11:43 +02:00
kim 077e66381f
[performance] cache account db lookups by public key URI (#795)
Signed-off-by: kim <grufwub@gmail.com>

Signed-off-by: kim <grufwub@gmail.com>
2022-09-02 11:58:42 +02:00
kim 614ab12733
[performance] use GetAccountByUsernameDomain() for local account lookups to rely on cache (#793)
Signed-off-by: kim <grufwub@gmail.com>

Signed-off-by: kim <grufwub@gmail.com>
2022-09-02 11:56:33 +02:00
Blackle Morisanchetto 1e1cdee06a
[feature] Emojify spoiler and content in web templates (#785)
* Emojify spoiler and content in web templates

* Use more performance emojify code (thanks NyaaaWhatsUpDoc!)
2022-09-02 11:54:32 +02:00
Blackle Morisanchetto ecb97f4e0b
[feature] Add support for the exclude_types[] parameter on the notifications endpoint (#784)
* Add support for the exclude_types[] parameter on the notifications endpoint

* Add swagger docs to notifications
2022-08-31 19:20:52 +02:00
Blackle Morisanchetto f01492ae48
[bugfix] Use custom blackfriday renderer to only add mention/hashtag links in normal text (#787)
* Use custom blackfriday renderer to only add mention/hashtag links in normal text

* Add additional markdown tests
2022-08-31 17:40:11 +02:00
tobi 0245c606d7
[chore] Test fixes (#788)
* use 'test' value for testrig storage backend

* update test dependency

* add WaitFor func in testrig

* use WaitFor function instead of time.Sleep

* tidy up tests

* make SentMessages a sync.map

* go fmt
2022-08-31 17:31:21 +02:00
nya1 bee8458a2d
[feature] add rate limit middleware (#741)
* feat: add rate limit middleware

* chore: update vendor dir

* chore: update readme with new dependency

* chore: add rate limit infos to swagger.md file

* refactor: add ipv6 mask limiter option

Add IPv6 CIDR /64 mask

* refactor: increase rate limit to 1000

Address https://github.com/superseriousbusiness/gotosocial/pull/741#discussion_r945584800

Co-authored-by: tobi <31960611+tsmethurst@users.noreply.github.com>
2022-08-31 12:06:14 +02:00
Blackle Morisanchetto daec9ab10e
[feature] Sort follow requests, followers, and following by updated_at (#774)
* Sort follow requests, followers, and following by updated_at

* Add migration to regenerate indexes for follows and follow requests
2022-08-31 11:27:39 +02:00
Blackle Morisanchetto c5c425b4e7
[bugfix] Fix tusky search issue by returning empty if offset is greater than zero (#786) 2022-08-31 09:57:50 +02:00
Blackle Morisanchetto 5d9c6b0e5a
[bugfix] Do not generate a notification when favouriting your own post (#780) 2022-08-30 11:43:29 +02:00
Blackle Morisanchetto 1f8ecf1c15
[bugfix] Fix issue where the 'favourited' field of a notification's status is always false (#779) 2022-08-30 11:42:52 +02:00
Blackle Morisanchetto 3ce26a60f8
[bugfix] Use reblogged status in notification, instead of wrapper status (#775) 2022-08-29 11:06:37 +02:00
tobi 969c194fcd
[bugfix] Relax outgoing http request queue (#760)
* add request queue trace logging

* fix  misleading wording

* implement request slots per host per method

* undo formatting change (?)

* remove gratuitous trace logging

* rename done -> release
avoids confusion with ctx.Done
2022-08-27 12:00:19 +02:00
Blackle Morisanchetto 54f6caed65
[bugfix] Status visibility + excludeReplies fixes (#769)
* Fix some bugs when viewing a user's posts: include their self-replies (threads) even when excludeReplies is set, and use in_reply_to_uri instead of in_reply_to_id to filter out replies

* Assign values to InReplyToURI when creating statuses. Add index and update old statuses with a migration
2022-08-27 11:35:31 +02:00
Blackle Morisanchetto e9b5ba0502
[bugfix] Check the length of form.MediaIDs instead of just checking for null (#766) 2022-08-26 17:37:51 +02:00
tobi 79fb8bad04
[feature] Allow footnotes in markdown, use <br> instead of \n (#767)
* allow markdown footnotes + hard line breaks

* don't keep whitespace w/minify (unnecessary now)

* test markdown a bit more
2022-08-26 13:28:06 +02:00
tobi 2fe3a2b5b8
[bugfix] Fix boost of boost issue (#764) 2022-08-23 10:46:10 +02:00
tobi b96533ca8f
[bugfix] Fix loss of account info on export/import, add tests (#759)
* start adding additional tests

* use random database address for in-memory sqlite

* add more fields to account export
2022-08-22 11:21:36 +02:00
tobi 570fa7c359
[bugfix] Fix potential dereference of accounts on own instance (#757)
* add GetAccountByUsernameDomain

* simplify search

* add escape to not deref accounts on own domain

* check if local + we have account by ap uri
2022-08-20 21:47:19 +01:00
tobi 586ebb5059
return after redirects to web representation (#755) 2022-08-16 19:50:26 +01:00
tobi ac6ed3d939
[chore] Update bun / sqlite versions; update gtsmodels (#754)
* upstep bun and sqlite versions

* allow specific columns to be updated in the db

* only update necessary columns for user

* bit tidier

* only update necessary fields of media_attachment

* only update relevant instance fields

* update tests

* update only specific account columns

* use bool pointers on gtsmodels
includes attachment, status, account, user

* update columns more selectively

* test all default fields on new account insert

* updating remaining bools on gtsmodels

* initialize pointer fields when extracting AP emoji

* copy bools properly

* add copyBoolPtr convenience function + test it

* initialize false bool ptrs a bit more neatly
2022-08-15 11:35:05 +01:00
tobi 9a9702c964
[bugfix] Check orientation when reading width/height of images (#749) 2022-08-10 15:54:15 +02:00
tobi 91c8d5d20d
[bugfix] Fix thumbnails not taking exif rotation into account (#746)
* use disintegration/imaging instead of nfnt/resize

* update tests

* use disintegration lib for thumbing (if necessary)
2022-08-10 14:05:14 +02:00
tobi 117888cf59
[feature] Add first iteration of a user panel at /user (#736)
* start work on user panel

* parse source first before checking if empty form

* newline

* set avi + header nicely

* add posts settings

* render signin a bit nicer on mobile

* return OK json on successful change

* return unauthorized on bad password

* clarify message on insecure password

* make login a bit prettier

* add alt text + border round image previews

* add logout button

* add password change

* styling updates

* redirect /auth/edit to /user

* update tests

* fix validation tests

* better labels, link to more info

* make submit button generic component

* move submit button inside forms

* add autocomplete labels to password fields

* fix indentation (thx eslint)

* update eslintrc

* eslint: no-unescaped-entities

* initial deduplication between user and admin panel

* add default status/post format setting

* user panel styling for inputs

* update user panel styling, include normalize css

* add placeholder text

* input padding

Co-authored-by: f0x <f0x@cthu.lu>
2022-08-08 10:40:51 +02:00
tobi 879b4abde7
[bugfix] Markdown formatting updates (#743)
* add minify dependency specifically for markdown

* rearrange markdown formatting

* update markdown tests
2022-08-07 18:19:16 +02:00
tobi f5689a9e5f
[feature] Let accounts set default status format, and use this when processing new statuses (#739)
* add post_format to acct & use it when making post

* update swagger docs

* add status_format updating to frontend

* fix up tests

* post_format => status_format

* add status_format to account validation
2022-08-06 12:09:21 +02:00
tobi 3ab3f58342
[bugfix] Parse source first before checking if empty form (#738) 2022-08-05 12:30:47 +02:00
Artémis 4fdbef04b4
[feature] Implemented notification clear (#720)
* Implemented notification clear

* Added the cache clear mechanism

* added multi user check test
2022-08-01 11:13:49 +02:00
tobi 8fdc9ed552
serve HEAD requests via the fileserver (#735) 2022-07-30 13:42:47 +01:00
tobi fc81e6443a
[bugfix] remove <= 0 expires_in from oauth token response (#731)
* remove <= 0 expired_in from oauth token response

* go fmt
2022-07-28 16:43:42 +02:00
tobi 8106b69856
[feature] add 'state' oauth2 param to /oauth/authorize (#730) 2022-07-28 16:43:27 +02:00
tobi 7ca5bac7c6
[bugfix] Fix Toot CLI media attachments not working properly (#726) 2022-07-22 13:43:51 +02:00
tobi 73b8839c5d
[bugfix] Make /api/v2/media more compatible with masto API (#724)
* update docs

* make api version into a path param

* update tests

* workaround to unset URL if using v2 of api

* make some fields into pointers
2022-07-22 12:48:19 +02:00
tobi 40f91d052c
[chore] Update image/video size defaults to mastodon's (#723) 2022-07-20 12:25:06 +02:00
tobi c84384e660
[bugfix] html escape special characters in text instead of totally removing them (#719)
* remove minify dependency

* tidy up some tests

* remove pre + postformat funcs

* rework sanitization + formatting

* update tests

* add some more markdown tests
2022-07-19 15:21:17 +02:00
kim 098dbe6ff4
[chore] use our own logging implementation (#716)
* first commit

Signed-off-by: kim <grufwub@gmail.com>

* replace logging with our own log library

Signed-off-by: kim <grufwub@gmail.com>

* fix imports

Signed-off-by: kim <grufwub@gmail.com>

* fix log imports

Signed-off-by: kim <grufwub@gmail.com>

* add license text

Signed-off-by: kim <grufwub@gmail.com>

* fix package import cycle between config and log package

Signed-off-by: kim <grufwub@gmail.com>

* fix empty kv.Fields{} being passed to WithFields()

Signed-off-by: kim <grufwub@gmail.com>

* fix uses of log.WithFields() with whitespace issues and empty slices

Signed-off-by: kim <grufwub@gmail.com>

* *linter related grumbling*

Signed-off-by: kim <grufwub@gmail.com>

* gofmt the codebase! also fix more log.WithFields() formatting issues

Signed-off-by: kim <grufwub@gmail.com>

* update testrig code to match new changes

Signed-off-by: kim <grufwub@gmail.com>

* fix error wrapping in non fmt.Errorf function

Signed-off-by: kim <grufwub@gmail.com>

* add benchmarking of log.Caller() vs non-cached

Signed-off-by: kim <grufwub@gmail.com>

* fix syslog tests, add standard build tags to test runner to ensure consistency

Signed-off-by: kim <grufwub@gmail.com>

* make syslog tests more robust

Signed-off-by: kim <grufwub@gmail.com>

* fix caller depth arithmatic (is that how you spell it?)

Signed-off-by: kim <grufwub@gmail.com>

* update to use unkeyed fields in kv.Field{} instances

Signed-off-by: kim <grufwub@gmail.com>

* update go-kv library

Signed-off-by: kim <grufwub@gmail.com>

* update libraries list

Signed-off-by: kim <grufwub@gmail.com>

* fuck you linter get nerfed

Signed-off-by: kim <grufwub@gmail.com>

Co-authored-by: tobi <31960611+tsmethurst@users.noreply.github.com>
2022-07-19 10:47:55 +02:00
tobi 59be7466f3
[bugfix] Markdown format fixes (#718)
* just sanitize markdown, don't minify or escape

* tidy tests, add one for inline code

* add another test, it works!
2022-07-19 10:41:16 +02:00
tobi c3b6a5b0f9
[feature] Implement cache-control and etags for static assets (#711)
* start working on etag stuff

* add + use cache middleware

* generate etags on the fly

* remove unused field

* clean up filepath

* add license headers to cache files

* add attachgroup function to router interface

* move cache into web module

* rename a couple things

* remove attachStaticFS function from router

* rename + tidy up a few things

* mount assets filesystem

* create assetsFileInfoCache

* update comment

* simplify hash

* fix string fmt

* skip last mod chk, prefer strong etags w/long cache

* move base handler to its own file
this matches the modules in the api folder

* generate new etag if file was modified

* wrap strong etag in quotation marks as per spec

* clarify logic in avatar search

* make hashing a little niftier
2022-07-18 12:55:06 +02:00
tobi ab03318b7a
[chore] move dialer inside new (#715)
* move dialer inside new, use default resolver

* instantiate resolver
2022-07-18 11:25:26 +02:00
tobi 839c4346ad
[performance] Add long cache-control max age to fileserver (#710) 2022-07-15 13:23:04 +02:00
tobi 6418307c64
[feature] Add back/next buttons to profiles for paging through statuses (#708)
* add GetAccountWebStatuses to db

* add WebStatusesGet func to processor

* don't add limit to next/prev links if 0

* take query params for next/prev statuses

* add separate next + prev links for convenience

* show 'nothing here' message if no statuses exist

* add back / next links to profiles

* allow paging down only

* go fmt ./...

* 'recent public toots' -> 'latest public toots'
2022-07-13 09:57:47 +02:00
kim 6934ae378a
[chore] improved router logging, recovery and error handling (#705)
* move panic recovery to logging middleware, improve logging + panic recovery logic

Signed-off-by: kim <grufwub@gmail.com>

* remove dead code

Signed-off-by: kim <grufwub@gmail.com>

* remove skip paths code

Signed-off-by: kim <grufwub@gmail.com>

* re-enable log quoting

Signed-off-by: kim <grufwub@gmail.com>

* use human-friendly bytesize in logging body size

Signed-off-by: kim <grufwub@gmail.com>

* only disable quoting in debug builds

Signed-off-by: kim <grufwub@gmail.com>

* use logrus level instead of debug.DEBUG() to enable/disable quoting

Signed-off-by: kim <grufwub@gmail.com>

* shutup linter

Signed-off-by: kim <grufwub@gmail.com>

* fix instance tests

Signed-off-by: kim <grufwub@gmail.com>

* fix gin test contexts created with missing engine HTML renderer

Signed-off-by: kim <grufwub@gmail.com>

* add note regarding not logging query parameters

Signed-off-by: kim <grufwub@gmail.com>

* better explain 'DisableQuoting' logic

Signed-off-by: kim <grufwub@gmail.com>

* add license text

Signed-off-by: kim <grufwub@gmail.com>
2022-07-12 08:32:20 +01:00
tobi a465cefb8c
[performance] Add new index to fix slow web profile queries (#706)
* start adding new index migration

* fix up index
2022-07-11 12:52:12 +01:00
kim 7cc40302a5
[chore] consolidate caching libraries (#704)
* add miekg/dns dependency

* set/validate accountDomain

* move finger to dereferencer

* totally break GetRemoteAccount

* start reworking finger func a bit

* start reworking getRemoteAccount a bit

* move mention parts to namestring

* rework webfingerget

* use util function to extract webfinger parts

* use accountDomain

* rework finger again, final form

* just a real nasty commit, the worst

* remove refresh from account

* use new ASRepToAccount signature

* fix incorrect debug call

* fix for new getRemoteAccount

* rework GetRemoteAccount

* start updating tests to remove repetition

* break a lot of tests
Move shared test logic into the testrig,
rather than having it scattered all over
the place. This allows us to just mock
the transport controller once, and have
all tests use it (unless they need not to
for some other reason).

* fix up tests to use main mock httpclient

* webfinger only if necessary

* cheeky linting with the lads

* update mentionName regex
recognize instance accounts

* don't finger instance accounts

* test webfinger part extraction

* increase default worker count to 4 per cpu

* don't repeat regex parsing

* final search for discovered accountDomain

* be more permissive in namestring lookup

* add more extraction tests

* simplify GetParseMentionFunc

* skip long search if local account

* fix broken test

* consolidate to all use same caching libraries

Signed-off-by: kim <grufwub@gmail.com>

* perform more caching in the database layer

Signed-off-by: kim <grufwub@gmail.com>

* remove ASNote cache

Signed-off-by: kim <grufwub@gmail.com>

* update cache library, improve db tracing hooks

Signed-off-by: kim <grufwub@gmail.com>

* return ErrNoEntries if no account status IDs found, small formatting changes

Signed-off-by: kim <grufwub@gmail.com>

* fix tests, thanks tobi!

Signed-off-by: kim <grufwub@gmail.com>

Co-authored-by: tsmethurst <tobi.smethurst@protonmail.com>
2022-07-10 17:18:21 +02:00
kim e75d742afe
[chore] Improve panic handling (#700)
* add miekg/dns dependency

* set/validate accountDomain

* move finger to dereferencer

* totally break GetRemoteAccount

* start reworking finger func a bit

* start reworking getRemoteAccount a bit

* move mention parts to namestring

* rework webfingerget

* use util function to extract webfinger parts

* use accountDomain

* rework finger again, final form

* just a real nasty commit, the worst

* remove refresh from account

* use new ASRepToAccount signature

* fix incorrect debug call

* fix for new getRemoteAccount

* rework GetRemoteAccount

* start updating tests to remove repetition

* break a lot of tests
Move shared test logic into the testrig,
rather than having it scattered all over
the place. This allows us to just mock
the transport controller once, and have
all tests use it (unless they need not to
for some other reason).

* fix up tests to use main mock httpclient

* webfinger only if necessary

* cheeky linting with the lads

* update mentionName regex
recognize instance accounts

* don't finger instance accounts

* test webfinger part extraction

* increase default worker count to 4 per cpu

* don't repeat regex parsing

* final search for discovered accountDomain

* be more permissive in namestring lookup

* add more extraction tests

* simplify GetParseMentionFunc

* skip long search if local account

* fix broken test

* panics get logged at error level, now include stacktrace

Signed-off-by: kim <grufwub@gmail.com>

Co-authored-by: tsmethurst <tobi.smethurst@protonmail.com>
2022-07-08 12:01:44 +02:00
tobi 81dd4f3660
[bugfix] Fix footer info fields not wrapping (#694)
* flex-wrap footer items

* add testrig instance entry with more info set
2022-07-05 16:44:58 +02:00
tobi b61b000e0a
[bugfix] Fix incorrect domain showing in profiles (#693)
* use instance account_domain in profile

* add instance account_domain field
2022-07-05 14:03:44 +02:00
tobi 4a69651a7c
[bugfix] Visibility fixes (#687)
* test self boost

* only CC to public when necessary
2022-07-04 15:41:20 +02:00
f0x52 c6d1b82f48
[chore] configure instance.Version for testrig (#659)
* configure instance.Version for testrig

* update instancepatch tests

Co-authored-by: tobi <31960611+tsmethurst@users.noreply.github.com>
Co-authored-by: tsmethurst <tobi.smethurst@protonmail.com>
2022-07-03 12:27:32 +02:00
tobi de4d75246c
[chore] Update pruneOneLocal to use ctx (#684)
* update pruneOneLocal to use ctx

* update test
2022-07-03 12:18:59 +02:00
Dominik Süß 9d0df426da
[feature] S3 support (#674)
* feat: vendor minio client

* feat: introduce storage package with s3 support

* feat: serve s3 files directly

this saves a lot of bandwith as the files are fetched from the object
store directly

* fix: use explicit local storage in tests

* feat: integrate s3 storage with the main server

* fix: add s3 config to cli tests

* docs: explicitly set values in example config

also adds license header to the storage package

* fix: use better http status code on s3 redirect

HTTP 302 Found is the best fit, as it signifies that the resource
requested was found but not under its presumed URL

307/TemporaryRedirect would mean that this resource is usually located
here, not in this case

303/SeeOther indicates that the redirection does not link to the
requested resource but to another page

* refactor: use context in storage driver interface
2022-07-03 12:08:30 +02:00
tobi 664713ddd4
[bugfix] Make hashtag regex work with non-ascii characters (#682) 2022-07-03 11:03:03 +02:00
tobi 9e7d022a06
[feature] Cleanup unattached local media (#680)
* add localUnattached db function

* add parseOlderThan util function

* add pruneunusedlocalattachments to media manager

* add unusedlocal pruning to schedule + admin call

* set number of days to keep as a const

* fix test
2022-06-30 12:22:10 +02:00
tobi 07620acc0e
[feature] Use default instance thumbnail if instance account header not set (#672) 2022-06-26 12:33:11 +02:00
tobi 68736efd20
[feature] add configuration to /api/v1/instance response (#670)
* add configuration object to api instance model

* regenerate swagger docs

* add func to return all supported mimes for media

* add instance configuration to api serialization

* fix json tags

* update instance endpoint tests

* fix typeutils tests

* final regen of swagger docs

* omitempty instance configuration
2022-06-26 10:58:45 +02:00
ugla 3e4e57d554
[bugfix] disallow following or blocking yoursel (#667)
Closes #664
2022-06-25 11:14:05 +02:00
tobi 0846b76e93
[bugfix] Fix 404 on status delete redraft (#668)
* add unattach function to media processor

* call delete or unattach appropriately
unattach from client api, delete from federated api

* typo fix
2022-06-24 17:17:40 +02:00
tobi 7eacbd064b
[bugfix] allow setting empty email via instance patch (#665) 2022-06-24 10:43:21 +02:00
tobi 5f00d4980b
[feature] Implement /api/v1/instance/peers endpoint (#660)
* add missing license headers

* start adding instance peers get

* rename domainblock.go

* embed domain in domainblock so it can be reused

* update swagger docs

* add test instances to db

* update tests

* add/update instancepeersget

* update domain model

* add getinstancepeers to db

* instance-expose-peers, instance-expose-suspended

* add auth checks for both current filters

* attach endpoint to router

* include public comment

* obfuscate domain if required

* go mod tidy

* update swagger docs

* remove unnecessary comment

* return 'flat' peerlist if no query params provided
2022-06-23 16:54:54 +02:00
f0x52 7c6c0cd547
[frontend] Profile pages upgrade (#640)
* fix css indentation

* profile styling update

* update status styling to match profile

* empty header fix

* generate random avatars for thread views

* appease the linter gods

* upgrade deps

* turn profile accent into border + $bg background

* upgrade deps

* small accessibility tweaks

Co-authored-by: tobi <31960611+tsmethurst@users.noreply.github.com>
2022-06-21 10:48:42 +02:00
tobi 8c7945fb78
[bugfix] Account self finger fix (#658)
* check only 1 of status/account when search by uri

* don't try to resolve local uris when searching
2022-06-20 14:11:36 +02:00
tobi 610395d5a5
[chore] make tests more cacheable by avoiding time.Now() (#656) 2022-06-19 17:10:24 +02:00
Mara Sophie Grosch c48266c459
[bugfix] for the second accounts.note_raw migration (#653) 2022-06-16 19:39:57 +02:00
Mara Sophie Grosch 0e12ee0aa1
postgres locale: fix accounts.note_raw migration (#651)
Database migration 20220506110822_add_account_raw_note.go has some error
handling code to detect some error messages as "ok", but only done for
english error messages. This commit adds a check for the specific error
code, which should be locale agnostic.
2022-06-16 11:22:51 +02:00