Commit graph

1014 commits

Author SHA1 Message Date
Tsuribori 1ba3e14b36
[feature] Initial Prometheus metrics implementation (#2334)
* feat: Initial OTEL metrics

* docs: add metrics documentation

* fix: metrics endpoint conditional check

* feat: metrics endpoint basic auth

* fix: make metrics-auth-enabled default false

* fix: go fmt helpers.gen.go

* fix: add metric-related env vars to envparsing.sh

* fix: metrics docs

* fix: metrics related stuff in envparsing.sh

* fix: metrics docs

* chore: metrics docs wording

* fix: metrics stuff in envparsing?

* bump otel versions

---------

Co-authored-by: Tsuribori <user@acertaindebian>
Co-authored-by: Tsuribori <none@example.org>
Co-authored-by: tsmethurst <tobi.smethurst@protonmail.com>
2023-11-20 16:43:55 +01:00
kim 16275853eb
[bugfix] self-referencing collection pages for status replies (#2364) 2023-11-20 12:22:28 +00:00
tobi 9b6d972c74
[chore/whoops] re-add missing languages flag (#2365) 2023-11-17 12:29:03 +01:00
tobi fc02d3c6f7
[feature] Set/show instance language(s); show post language on frontend (#2362)
* update go text, include text/display

* [feature] Set instance langs, show post lang on frontend

* go fmt

* WebGet

* set language for whole article, don't use FA icon

* mention instance languages + other optional config vars

* little tweak

* put languages in config properly

* warn log language parse

* change some naming around

* tidy up validate a bit

* lint

* rename LanguageTmpl in template
2023-11-17 11:35:28 +01:00
tobi 4ee436e98a
[bugfix] process account delete side effects in serial, not in parallel (#2360)
* [bugfix] process account delete side effects in serial, not in parallel

* StartWorkers / StartNoopWorkers for tests

* undo testrig trace logging

* log errors instead of immediately returning
2023-11-14 15:57:25 +01:00
tobi 0b99f14d64
[bugfix] Update poll delete/update db queries (#2361) 2023-11-14 12:43:27 +00:00
tobi 8d0c017cf2
[feature/performance] Wrap incoming HTTP requests in timeout handler (#2353)
* deinterface router, start messing about with deadlines

* weeeee

* thanks linter (thinter)

* write Connection: close when timing out requests

* update wording

* don't replace req

* don't bother with fancy Cause functions (I'll use them one day...)
2023-11-13 19:48:51 +01:00
tobi 7753f42132
[chore] update otel -> v1.20.0 (#2358) 2023-11-13 11:08:02 +01:00
kim deaea100c3
[bugfix] support endless polls, and misskey's' method of inferring expiry in closed polls (#2349) 2023-11-11 10:15:04 +00:00
tobi ba9d6b467a
[feature] Media attachment placeholders (#2331)
* [feature] Use placeholders for unknown media types

* fix read of underreported small files

* switch to reduce nesting

* simplify cleanup
2023-11-10 19:29:26 +01:00
tobi c7ecab9e6f
[chore/bugfix/horror] Allow expires_in and poll choices to be parsed from strings (#2346) 2023-11-10 16:42:48 +00:00
tobi 7ce3a1e6f3
[bugfix] Don't try to update suspended accounts (#2348)
* [bugfix] Don't try to update suspended accounts

* bail early if requesting account suspended
2023-11-10 17:16:58 +01:00
tobi 42a19cf390
[bugfix/docs] Poll api fixups + swagger docs (#2345) 2023-11-09 12:06:37 +00:00
kim b1c65ed9ac
[bugfix] actually decrement votes during poll vote delete ... (#2344) 2023-11-09 09:08:46 +01:00
kim 34d0879c16
[bugfix] fix poll vote count responses on client and fedi API vote creation (#2343)
* increment poll votes *before* enqueuing vote to client API worker

* increment vote counts before federating status update after vote in local poll

* improved vote count calculation during backend -> frontend model conversion
2023-11-08 23:37:35 +01:00
kim e9e5dc5a40
[feature] add support for polls + receiving federated status edits (#2330) 2023-11-08 14:32:17 +00:00
kim 41435a6c4e
[feature] support canceling scheduled tasks, some federation API performance improvements (#2329) 2023-11-04 20:21:20 +00:00
Sam Lade 145dc515a2
Remove account_suspended_at_idx to resolve slow query issues (#2310) 2023-10-31 13:43:33 +00:00
kim ce71a5a790
[feature] add per-uri dereferencer locks (#2291) 2023-10-31 11:12:22 +00:00
tobi 51d0a0bba5
[bugfix] Relax Mention parsing, allowing either href or name (#2320) 2023-10-31 11:05:17 +00:00
tobi dd4b0241ea
[bugfix] Allow blocked accounts to show in precise search (#2321) 2023-10-30 19:01:00 +01:00
tobi 4dc0547dc0
[feature] Customizable media cleaner schedule (#2304) 2023-10-30 17:35:11 +00:00
tobi 0b978f2c56
[bugfix] Extract description as summary first, fall back to name (#2303) 2023-10-26 11:59:10 +02:00
kim ece2e795e0
[feature] attach any request errors if found, only set level=ERROR if code >= 500 (#2300) 2023-10-25 16:11:40 +01:00
tobi c7b6cd7770
[feature] Status thread mute/unmute functionality (#2278)
* add db models + functions for keeping track of threads

* give em the old linty testy

* create, remove, check mutes

* swagger

* testerino

* test mute/unmute via api

* add info log about new index creation

* thread + allow muting of any remote statuses that mention a local account

* IsStatusThreadMutedBy -> IsThreadMutedByAccount

* use common processing functions in status processor

* set = NULL

* favee!

* get rekt darlings, darlings get rekt

* testrig please, have mercy muy liege
2023-10-25 15:04:53 +01:00
KEINOS 27f4659139
[bugfix] allow store smaller PNG image than 261 bytes (#2263) (#2298)
* chore: add test of golden cases before fix of #2263

* chore: add test case to reproduce error of #2263

* [bugfix] allow store smaller PNG image than 261 bytes (#2263)
2023-10-25 14:58:45 +02:00
tobi cc313f58cb
[bugfix] serialize instance terms via API (#2293) 2023-10-24 14:23:22 +02:00
tobi 013a17f5e8
[chore] bump go version -> 1.21.x (#2287)
* [chore] bump go version -> 1.21.x

* versions

* lint
2023-10-23 13:07:31 +01:00
kim 69ba9a79a1
[chore] de-interface{} the federator and dereferencer structs (#2285)
* de-interface{} the federator and dereferencer structs

* fix broken type signatures
2023-10-23 10:58:13 +01:00
tobi 9114c5ca1b
[bugfix/frontend] fix typo and other oddness in patchRemoteEmojis (#2281)
* fix emoji test model

* found the bug!

* remove unused 'current' import

* comment useChecklistReducer

* wah

* lint

* fix cleaner tests
2023-10-21 17:23:05 +02:00
tobi 805c67b759
[bugfix] add missing "local" param in public timeline link headers (#2277) 2023-10-18 12:32:40 +02:00
tobi 77c5d74113
[bugfix] Postgres error wrapping fix (#2276) 2023-10-18 12:17:26 +02:00
tobi 637f188ebe
[feature] Allow import/export/creation of domain allows via admin panel (#2264)
* it's happening!

* aaa

* fix silly whoopsie

* it's working pa! it's working ma!

* model report parameters

* shuffle some more stuff around

* getting there

* oo hoo

* finish tidying up for now

* aaa

* fix use form submit errors

* peepee poo poo

* aaaaa

* ffff

* they see me typin', they hatin'

* boop

* aaa

* oooo

* typing typing tappa tappa

* almost done typing

* weee

* alright

* push it push it real good doo doo doo doo doo doo

* thingy no worky

* almost done

* mutation modifers not quite right

* hmm

* it works

* view blocks + allows nicely

* it works!

* typia install

* the old linterino

* linter plz
2023-10-17 12:46:06 +02:00
kim 6e508830e1
updates markdown parsing to reduce allocations in the same way as the plain text formatter (#2252) 2023-10-05 13:22:40 +01:00
恐咖兵糖 e0f0d320f6
[docs] statuses-query-pinned #2250 (#2251) 2023-10-04 14:19:24 +02:00
kim c6e00afc7c
[feature] tentatively start adding polls support (#2249) 2023-10-04 13:09:42 +01:00
kim 297b6eeaaa
[chore] internal/ap: add pollable AS types, code reformatting, general niceties (#2248) 2023-10-03 14:59:30 +01:00
Daenney 0cce2c0838
[feature] Block a bunch of "AI" crawlers (#2239)
* [feature] Block Google Bard/AI crawlers

* [feature] Block the other OpenAI crawler

* [feature] Block Common Crawl crawler

This is used in research, but also gleefully advertises itself as the
training source used in all LLMs and GPT-3.

Fixes: #2240

* [feature] Block Omgilikebot

Used by some shady big web data engine company.

* [feature] Block Meta's language model crawler

* [feature] Block well-known.dev crawler
2023-09-30 20:44:57 +01:00
tobi 2b6b9cdf83
[bugfix] Fix paging for empty items (#2236)
* use minID properly for public timeline

* return paged response properly even when 0 items

* use gtserror

* page more consistently (for now)

* test

* aaa
2023-09-29 15:31:10 +02:00
tobi 536d9e482d
[chore/bugfix] Deinterface text.Formatter, allow underscores in hashtags (#2233) 2023-09-29 10:39:56 +02:00
tobi b6b8f82c87
[bugfix] Move follow.show_reblogs check further up to avoid showing unwanted reblogs in home timeline (#2234) 2023-09-29 10:39:35 +02:00
Daenney 2a9927dfdc
[chore] Enable S256 code challenge (#2224)
For some reason we only had plain enabled. This adds S256.
2023-09-28 11:21:19 +02:00
kim 9f9fcf743d
[bugfix] support both CollectionPage AND OrderedCollectionPage in status replies (#2220) 2023-09-23 18:28:12 +01:00
kim 8f67dd583d
[chore] deinterface the typeutils.Converter and update to use state structure (#2217)
* update typeconverter to use state structure

* deinterface the typeutils.TypeConverter -> typeutils.Converter

* finish copying over old type converter code comments

* fix cherry-pick merge issues, fix tests pointing to old typeutils interface type still
2023-09-23 18:44:11 +02:00
K 1b848aa1e5
[chore] fix typo in slice.go (#2219) 2023-09-23 18:42:34 +02:00
tobi 183eaa5b29
[feature] Implement explicit domain allows + allowlist federation mode (#2200)
* love like winter! wohoah, wohoah

* domain allow side effects

* tests! logging! unallow!

* document federation modes

* linty linterson

* test

* further adventures in documentation

* finish up domain block documentation (i think)

* change wording a wee little bit

* docs, example

* consolidate shared domainPermission code

* call mode once

* fetch federation mode within domain blocked func

* read domain perm import in streaming manner

* don't use pointer to slice for domain perms

* don't bother copying blocks + allows before deleting

* admonish!

* change wording just a scooch

* update docs
2023-09-21 12:12:04 +02:00
kim fc11deeb83
[feature] add paging to AP following / followers endpoints (#2198) 2023-09-20 16:49:46 +01:00
kim ba0a31d224
[bugfix] fix flakey paging test (#2210) 2023-09-18 13:45:10 +01:00
kim 7293d6029b
[feature] add paging to account follows, followers and follow requests endpoints (#2186) 2023-09-12 14:00:35 +01:00
tobi 4b594516ec
[feature] Allow admins to expire remote public keys; refetch expired keys on demand (#2183) 2023-09-12 10:43:12 +01:00
Daenney 2cac5a4613
[feature] Support Actor URIs for webfinger queries (#2187)
* [feature] Support Actor URIs for webfinger queries

It's now possible to pass an Actor URI as the resource to query for when
doing a webfinger query. The code now extracts the username and domain
from the URI. The URI needs to be fully qualified, including having a
scheme of http or https to be recognised as such.

The acct scheme is handled as we used to, including dealing with an
erroneous leading @ on the username. We retain the ability to handle
resources without a scheme by parsing them again with the acct scheme if
the original parse failed. This can happen due to parsing ambiguities
when dealing with a string like user@domain.tld:port.

* [bugfix] Remove debugging changes

* [chore] Make TestExtractNamestring table-driven

* [chore] Unnest Trim and Split for readability
2023-09-11 18:38:31 +02:00
kim b093947d84
[chore] much improved paging package (#2182) 2023-09-07 15:58:37 +01:00
Daenney 14ef098099
[feature] Support OTLP HTTP, drop Jaeger (#2184)
* [feature] Add http trace exporter, drop Jaeger

Jaeger supports ingesting traces using the OpenTelemetry gRPC or HTTP
methods. The Jaeger project has deprecated the old jaeger transport.

* Add support for submitting traces over HTTP
* Drop support for the old Jaeger protocol
* Upgrade the trace libraries to v1.17

Fixes: #2176
Fixes: #2179
2023-09-07 13:20:37 +02:00
kim 916c6d07ba
[bugfix] fix checks for deref the same status descendants / ascendants (#2181) 2023-09-05 11:22:02 +01:00
Daenney 5f10885dde
[feature] Don't trace non-existing routes (#2172)
c.FullPath() is the empty string if a request doesn't match any route on
our mux. In those cases, there's no value in emitting a trace. The trace
will be empty, containing no other information beyond the fact that we
didn't match a route. Since Gin breaks off the processing early we don't
need to trace this request as it won't do anything and consumes no
further resources.

The 404 will still be emitted by our logs and will be visible from a
reverse proxy too.
2023-09-04 17:15:14 +02:00
tobi 3ed1ca68e5
[feature] Store admin actions in the db, prevent conflicting actions (#2167) 2023-09-04 14:55:17 +01:00
kim 4eb77ff5d7
[bugfix] move SQLite pragmas into connection string (#2171)
* move SQLite pragmas into connection string

Signed-off-by: kim <grufwub@gmail.com>

* use url.Values type for SQLite connection preferences

Signed-off-by: kim <grufwub@gmail.com>

* set SQLite URI prefs properly using _pragma query key

Signed-off-by: kim <grufwub@gmail.com>

* add notes on SQLite connection preferences

Signed-off-by: kim <grufwub@gmail.com>

* fix typo

Signed-off-by: kim <grufwub@gmail.com>

* add one extra line regarding connection pooling

Signed-off-by: kim <grufwub@gmail.com>

---------

Signed-off-by: kim <grufwub@gmail.com>
2023-09-01 15:13:33 +02:00
kim 1ee99fc165
[bugfix] wrap bun.Tx to add our own error processing (#2169)
* wrap bun.Tx to add our own error processing

Signed-off-by: kim <grufwub@gmail.com>

* add compile-time check for updateRowError() compatibility with sql.Row, fix wrapTx() not being used properly

Signed-off-by: kim <grufwub@gmail.com>

---------

Signed-off-by: kim <grufwub@gmail.com>
2023-08-31 11:46:15 +02:00
rdelaage 7b48437f17
[feature] list commands for both attachment and emojis (#2121)
* [feature] list commands for both attachment and emojis

* use fewer commands, provide `local-only` and `remote-only` as filters

* envparsing

---------

Co-authored-by: Romain de Laage <romain.delaage@rdelaage.ovh>
Co-authored-by: tsmethurst <tobi.smethurst@protonmail.com>
2023-08-23 18:01:16 +02:00
tobi 8f38dc2e7f
[feature] Add rate limit exceptions option, use ISO8601 for rate limit reset (#2151)
* start updating rate limiting, add exceptions

* tests, comments, tidying up

* add rate limiting exceptions to example config

* envparsing

* nolint

* apply kimbediff

* add examples
2023-08-23 14:32:27 +02:00
tobi 94d16631bc
[performance] Rework home timeline query to use cache more (#2148) 2023-08-22 15:41:51 +02:00
Daenney 4ae16bce8c
[feature] Make log format configurable (#2130)
* [feature] Don't emit timestamp in log lines

When running gotosocial with a service manager like systemd, or a
container runtime, the associated log driver usually emits timestamps
itself. In those cases, having the extra timestamp from our own log
lines ends up being a bit noisy and when centrally ingesting logs is
duplicate information.

This introduces a configuration flag that allows disabling emitting the
timestamp. It's only wired up for "daemonised" processes, meaning server
and testrig.

* [chore] Add docs for log-timestamp

* [feature] Simplify timestamp handling

Co-Authored-By: kim <89579420+NyaaaWhatsUpDoc@users.noreply.github.com>

* [chore] Less escaped double-quotes

* [chore] Fix help string

---------

Co-authored-by: kim <89579420+NyaaaWhatsUpDoc@users.noreply.github.com>
2023-08-21 19:07:55 +01:00
tobi 638f023a1c
[performance] Tweak media attachment cleanup; replace stale index (#2143) 2023-08-21 14:03:04 +01:00
tobi 1e2db7a32f
[feature/bugfix] Probe S3 storage for CSP uri, add config flag for extra URIs (#2134)
* [feature/bugfix] Probe S3 storage for CSP uri, add config flag for extra URIs

* env parsing tests, my coy mistress
2023-08-20 13:35:55 +02:00
f0x52 92de8fb396
[feature] Instance rules (#2125)
* init instance rules database model, admin api

* expose instance rules in public instance api

* public /api/v1/instance/rules route

* GET ruleById

* createRule route

* createRule auth check

* updateRule

* deleteRule

* list rules on about page

* ruleGet auth

* add about page ids for anchors

* process and store adding violated rules to reports

* admin api models for instance rules

* instance rule edit frontend

* change rule inputs to textareas

* database fixes after rebase (#2124)

* remove unused imports

* fix db migration column name

* fix tests

* fix more tests

* fix postgres error with wrongly used Ident

* add some tests, fiddle with rule model a bit, fix postgres migration

* swagger docs

---------

Co-authored-by: tsmethurst <tobi.smethurst@protonmail.com>
2023-08-19 14:33:15 +02:00
kim d5d6ad406f
[bugfix] fix double firing bun.DB query hooks (#2124)
* improve bun.DB wrapping readability + comments, fix double-firing query hooks

* fix incorrect code comment placement

* fix linter issues

* Update internal/db/basic.go

* do as the linter commmands ...

---------

Signed-off-by: kim <grufwub@gmail.com>
Co-authored-by: Daenney <daenney@users.noreply.github.com>
2023-08-17 17:26:21 +01:00
kim e9c3663cce
[chore] ensure worker contexts have request ID (#2120) 2023-08-15 17:01:01 +01:00
kim 815b5291e0
[bugfix] fix inconsistent calculated cache sizes (#2115)
* use calculated exampleTime instead of `time.Now()` to ensure no locale data, retweak cache ratios

* update envparsing test

* update default cache memory to 100MiB

* fix envparsing with latest cache target default

---------

Signed-off-by: kim <grufwub@gmail.com>
2023-08-14 17:08:19 +01:00
f0x52 912a104aed
[fix] Update CSP header for blob images (upload preview) and dev livereload (#2109)
* update CSP header for blob images (upload preview) and dev livereload websocket

* update csp for s3, update csp tests
2023-08-14 12:30:09 +02:00
Daenney 5e368d3089
[bugfix] CSP policy fixes for S3/object storage (#2104)
* [bugfix] CSP policy fixes for S3 in non-proxied mode

* It should be img-src
* In both img-src and media-src we still need to include 'self'
2023-08-12 12:21:48 +02:00
tobi b7274545e0
[bugfix] Add s3 endpoint as image-src and media-src for CSP (#2103)
* [bugfix] Add s3 endpoint as image-src and media-src for CSP

* use https if secure

* reorder comment
2023-08-11 17:49:17 +02:00
tobi a1768a83e0
[bugfix] Suppress 'errNoEntries' warnings from InboxForwarding function call (#2102) 2023-08-11 14:17:36 +01:00
tobi dc99e9e10b
[bugfix] Fix using wrong key for clientID during oauth callback (#2101) 2023-08-11 14:58:47 +02:00
tobi dc96562b40
[bugfix] Use custom bluemonday policy to disallow inline img tags (#2100) 2023-08-11 14:40:11 +02:00
Daenney 3aedd937c3
[feature] Set Content-Security-Policy header (#2095)
This adds the CSP header with a policy of only loading from the same
domain. We don't make use of external media, CSS, JS, fonts, so we don't
ever need external data loaded in our context.

When building a DEBUG build, the policy gets extended to include
localhost:*, i.e localhost on any port. This keeps the live-reloading
flow for JS development working. localhost and 127.0.0.1 are considered
to be the same so mixing and matching those doesn't result in a CSP
violation.
2023-08-11 13:20:56 +02:00
tobi a26af1310f
[bugfix] Populate followReq before accessing targetaccount pointer (#2099) 2023-08-11 11:05:49 +02:00
tobi 5588d4e88e
[bugfix] Use length in runes when trimming for RSS (#2094) 2023-08-10 18:26:56 +02:00
tobi 992c7ce4c2
[chore] Add test to ensure show_reblogs on follow works as expected (#2093) 2023-08-10 17:10:27 +02:00
kim 91cbcd589e
[performance] remove last of relational queries to instead rely on caches (#2091) 2023-08-10 15:08:41 +01:00
tobi 9770d54237
[feature] List replies policy, refactor async workers (#2087)
* Add/update some DB functions.

* move async workers into subprocessor

* rename FromFederator -> FromFediAPI

* update home timeline check to include check for current status first before moving to parent status

* change streamMap to pointer to mollify linter

* update followtoas func signature

* fix merge

* remove errant debug log

* don't use separate errs.Combine() check to wrap errs

* wrap parts of workers functionality in sub-structs

* populate report using new db funcs

* embed federator (tiny bit tidier)

* flesh out error msg, add continue(!)

* fix other error messages to be more specific

* better, nicer

* give parseURI util function a bit more util

* missing headers

* use pointers for subprocessors
2023-08-09 19:14:33 +02:00
Daenney dbf487effb
[bugfix] Fix incorrect per-loop variable capture (#2092)
These should be per iteration, not per loop. This was caught by running
a build with the loopvar experiment: go build -gcflags=-d=loopvar=2.
2023-08-09 18:40:32 +02:00
kim 0ddc2edf19
[bugfix] only set content-length AFTER rewinding body bytes (#2086) 2023-08-08 12:45:29 +01:00
kim 3920bc87d1
[bugfix] don't accept unrelated statuses (#2078)
Co-authored-by: Daenney <daenney@users.noreply.github.com>
Co-authored-by: tsmethurst <tobi.smethurst@protonmail.com>
2023-08-08 12:26:34 +01:00
tobi 4b05dcde43
[chore] Update robots.txt, give chatgpt the middle finger (#2085) 2023-08-08 13:16:34 +02:00
Daenney be3718f6e4
[chore] Use generic pointer function (#2080)
This replaces the different $TypePtr functions with a generic
implementation.
2023-08-07 18:38:11 +01:00
Vyr Cossont 0f812746b7
[feature] Allow full BCP 47 in language inputs (#2067)
* Allow full BCP 47 in language inputs

Fixes #2066

* Fuse validation and normalization for languages

* Remove outdated comment line

* Move post language canonicalization test
2023-08-07 10:25:54 +02:00
Daenney c1375ca5c1
[chore] Remove go-playground/validator (#2069)
* [chore] Remove go-playground/validator

It turns out we're not actually using the validator code. This is a
remnant from when we intended to use it, but the presence of it and its
struct tags creates the illusion we're validating a lot of things we're
not. It resulted in some confusion when we were trying to figure out
language valdiation.

Remove all this code, so that only the validation functions from the
validate package we actually use remain. I'm not touching the struct
tags in the migrations in order to avoid things potentially thinking
migrations need to be re-run.

* [chore] Bring back a struct tag on api

The validate on internal/api is Gin doing form validation, not the
validator from go-playground/validator.
2023-08-06 12:22:40 +02:00
kim 6f4ae8f58d
[bugfix] update cache library with nil ptr fix (#2070)
Signed-off-by: kim <grufwub@gmail.com>
2023-08-06 11:55:04 +02:00
kim 9a291dea84
[performance] add caching of status fave, boost of, in reply to ID lists (#2060) 2023-08-04 12:28:33 +01:00
kim 00adf18c24
[feature] simpler cache size configuration (#2051)
* add automatic cache max size generation based on ratios of a singular fixed memory target

Signed-off-by: kim <grufwub@gmail.com>

* remove now-unused cache max-size config variables

Signed-off-by: kim <grufwub@gmail.com>

* slight ratio tweak

Signed-off-by: kim <grufwub@gmail.com>

* remove unused visibility config var

Signed-off-by: kim <grufwub@gmail.com>

* add secret little ratio config trick

Signed-off-by: kim <grufwub@gmail.com>

* fixed a word

Signed-off-by: kim <grufwub@gmail.com>

* update cache library to remove use of TTL in result caches + slice cache

Signed-off-by: kim <grufwub@gmail.com>

* update other cache usages to use correct interface

Signed-off-by: kim <grufwub@gmail.com>

* update example config to explain the cache memory target

Signed-off-by: kim <grufwub@gmail.com>

* update env parsing test with new config values

Signed-off-by: kim <grufwub@gmail.com>

* do some ratio twiddling

Signed-off-by: kim <grufwub@gmail.com>

* add missing header

* update envparsing with latest defaults

Signed-off-by: kim <grufwub@gmail.com>

* update size calculations to take into account result cache, simple cache and extra map overheads

Signed-off-by: kim <grufwub@gmail.com>

* tweak the ratios some more

Signed-off-by: kim <grufwub@gmail.com>

* more nan rampaging

Signed-off-by: kim <grufwub@gmail.com>

* fix envparsing script

Signed-off-by: kim <grufwub@gmail.com>

* update cache library, add sweep function to keep caches trim

Signed-off-by: kim <grufwub@gmail.com>

* sweep caches once a minute

Signed-off-by: kim <grufwub@gmail.com>

* add a regular job to sweep caches and keep under 80% utilisation

Signed-off-by: kim <grufwub@gmail.com>

* remove dead code

Signed-off-by: kim <grufwub@gmail.com>

* add new size library used to libraries section of readme

Signed-off-by: kim <grufwub@gmail.com>

* add better explanations for the mem-ratio numbers

Signed-off-by: kim <grufwub@gmail.com>

* update go-cache

Signed-off-by: kim <grufwub@gmail.com>

* library version bump

Signed-off-by: kim <grufwub@gmail.com>

* update cache.result{} size model estimation

Signed-off-by: kim <grufwub@gmail.com>

---------

Signed-off-by: kim <grufwub@gmail.com>
2023-08-03 11:34:35 +02:00
tobi e8a20f587c
[bugfix] Rework MultiError to wrap + unwrap errors properly (#2057)
* rework multierror a bit

* test multierror
2023-08-02 17:21:46 +02:00
kim 2cee8f2dd8
[bugfix] fix slow accounts / statuses using emojis lookups (#2056)
* update DeleteEmoji to use faster relational tables for status / account finding

Signed-off-by: kim <grufwub@gmail.com>

* update Get{Accounts,Statuses}UsingEmoji() to also use relational tables

Signed-off-by: kim <grufwub@gmail.com>

* remove the now unneeded tags relation from newStatusQ()

Signed-off-by: kim <grufwub@gmail.com>

* fix table names

Signed-off-by: kim <grufwub@gmail.com>

* fix account and status selects using emojis

Signed-off-by: kim <grufwub@gmail.com>

---------

Signed-off-by: kim <grufwub@gmail.com>
2023-08-02 16:11:23 +02:00
kim 24516b84c2
[bugfix] handle HEAD requests more elegantly (#2055)
Signed-off-by: kim <grufwub@gmail.com>
2023-08-02 10:28:20 +02:00
tobi cec29e2a8d
[bugfix] Allow instance accounts to be shown in search results in certain circumstances (#2053) 2023-08-02 08:31:09 +01:00
tobi 2be83fdca5
[feature] Allow users to skip http client tls verification for testing purposes (with appropriately loud warnings) (#2052) 2023-08-01 18:50:17 +01:00
tobi 2796a2e82f
[feature] Hashtag federation (in/out), hashtag client API endpoints (#2032)
* update go-fed

* do the things

* remove unused columns from tags

* update to latest lingo from main

* further tag shenanigans

* serve stub page at tag endpoint

* we did it lads

* tests, oh tests, ohhh tests, oh tests (doo doo doo doo)

* swagger docs

* document hashtag usage + federation

* instanceGet

* don't bother parsing tag href

* rename whereStartsWith -> whereStartsLike

* remove GetOrCreateTag

* dont cache status tag timelineability
2023-07-31 15:47:35 +02:00
kim ed2477ebea
[performance] cache follow, follow request and block ID lists (#2027) 2023-07-31 11:25:29 +01:00
Vyr Cossont b874e9251e
[feature] Implement markers API (#1989)
* Implement markers API

Fixes #1856

* Correct import grouping in markers files

* Regenerate Swagger for markers API

* Shorten names for readability

* Cache markers for 6 hours

* Update DB ref

* Update envparsing.sh
2023-07-29 12:49:14 +02:00
tobi e747c14685
[chore] fix merge issue (#2029) 2023-07-27 12:54:27 +02:00
Vyr Cossont 22ac4607a1
[feature] Support setting private notes on accounts (#1982)
* Support setting private notes on accounts

* Reformat comment whitespace

* Add missing license headers

* Use apiutil.ParseID

* Rename Note model and cache to AccountNote

* Update golden cache config in test/envparsing.sh

* Rename gtsmodel/note.go to gtsmodel/accountnote.go

* Update AccountNote uniqueness constraint name

Now has same prefix as other indexes on this table.

---------

Co-authored-by: tobi <31960611+tsmethurst@users.noreply.github.com>
2023-07-27 10:30:39 +02:00
kim 5f3e095717
[performance] retry db queries on busy errors (#2025)
* catch SQLITE_BUSY errors, wrap bun.DB to use our own busy retrier, remove unnecessary db.Error type

Signed-off-by: kim <grufwub@gmail.com>

* remove dead code

Signed-off-by: kim <grufwub@gmail.com>

* remove more dead code, add missing error arguments

Signed-off-by: kim <grufwub@gmail.com>

* update sqlite to use maxOpenConns()

Signed-off-by: kim <grufwub@gmail.com>

* add uncommitted changes

Signed-off-by: kim <grufwub@gmail.com>

* use direct calls-through for the ConnIface to make sure we don't double query hook

Signed-off-by: kim <grufwub@gmail.com>

* expose underlying bun.DB better

Signed-off-by: kim <grufwub@gmail.com>

* retry on the correct busy error

Signed-off-by: kim <grufwub@gmail.com>

* use longer possible maxRetries for db retry-backoff

Signed-off-by: kim <grufwub@gmail.com>

* remove the note regarding max-open-conns only applying to postgres

Signed-off-by: kim <grufwub@gmail.com>

* improved code commenting

Signed-off-by: kim <grufwub@gmail.com>

* remove unnecessary infof call (just use info)

Signed-off-by: kim <grufwub@gmail.com>

* rename DBConn to WrappedDB to better follow sql package name conventions

Signed-off-by: kim <grufwub@gmail.com>

* update test error string checks

Signed-off-by: kim <grufwub@gmail.com>

* shush linter

Signed-off-by: kim <grufwub@gmail.com>

* update backoff logic to be more transparent

Signed-off-by: kim <grufwub@gmail.com>

---------

Signed-off-by: kim <grufwub@gmail.com>
2023-07-25 10:34:05 +02:00
kim 9eff0d46e4
[feature/performance] support uncaching remote emoji + scheduled cleanup functions (#1987) 2023-07-24 13:14:13 +01:00
Daenney 994d5e80d9
[chore] Disable the right webfinger test (#2018)
We don't talk about 8868889 / #1983.
2023-07-23 16:33:16 +02:00
tobi 5a29a031ad
[chore] Admin CLI + new account creation refactoring (#2008)
* set maxPasswordLength to 72 bytes, rename validate function

* refactor NewSignup

* refactor admin account CLI commands

* refactor oidc create user

* refactor processor create

* tweak password change, check old != new password
2023-07-23 12:33:17 +02:00
Daenney f8f0312042
[feature] Report Masto version in /api/v1/instance (#1977) 2023-07-21 18:49:13 +01:00
Daenney da8282081c
[feature] Lower remote media cache config duration (#2007)
The old default of 30d can lead to a lot of media getting cached and
significant disk usage, even on small or single person instances. A lot
of deployments decrease this value, to 15 or even less. This is less of
an issue when using object storage, but for local storage which is the
more popular deployment option running out of disk space is unpleasant.

With GoToSocial's aim to fit in small places, this changes the default
to a much more conservative 7 days. In all likelihood people aren't
scrolling that far back in their timeline so this change shouldn't
result in any issue. Existing deployments will only be affected by
this change if the admin hasn't already configured this value, or didn't
bootstrap from the example configuration.
2023-07-21 16:22:59 +02:00
tobi 89ee9d5004
[bugfix] Return all accounts when list accounts limit <= 0 (#2014) 2023-07-21 13:56:38 +01:00
tobi d6fa74e5dc
[bugfix] Change maximumPasswordLength to 72 bytes (#2012) 2023-07-21 10:29:18 +01:00
kim f4319740ab
[bugfix] more robust list timeline invalidation (#1995) 2023-07-18 09:43:17 +01:00
tobi 346ecabd07
[bugfix] Add missing continue statement in prepareXBetweenIDs (#1996) 2023-07-18 09:39:16 +01:00
tobi 12b6cdcd8c
[bugfix] Set Vary header correctly on cache-control (#1988)
* [bugfix] Set Vary header correctly on cache-control

* Prefer activitypub types on AP endpoints

* use immutable on file server, vary by range

* vary auth on Accept
2023-07-13 21:27:25 +02:00
Daenney 88688899aa
[chore] Skip webfinger test on CI (#1983) 2023-07-12 16:09:49 +01:00
tobi 1951e6c840
[bugfix] Update account Update logic (#1984) 2023-07-12 12:20:15 +01:00
firescry 8d92b2479f
[bugfix] Align default values in the configuration file with the code (#1971)
* Set default value of SMTPFrom to empty string

This parameter should contain proper e-mail address (to be provided by user during configuration).

* Update default values in example/config.yaml

Default values and related comments in example/config.yaml are aligned
with values defined in internal/config/defaults.go.
Small improvements to foramting of config.yaml file.

* Add default value for AdvancedThrottlingRetryAfter to internal/config/defaults.go

AdvancedThrottlingRetryAfter was introduced in 70739d3 (superseriousbusiness/gotosocial#1466).

* Update config.yaml snippets in documentation
2023-07-12 10:28:41 +02:00
MaeIsBad 0d267fd598
[bugfix] Properly handle range > content-length (#1979)
This makes the serveFileRange function return the entire file
if suffix-range is larger than content-length in compliance with RFC9110

Co-authored-by: mae <git@badat.dev>
2023-07-12 09:51:51 +02:00
tobi ca5492b65f
[bugfix] Tidy up rss feed serving; don't error on empty feed (#1970)
* [bugfix] Tidy up rss feed serving; don't error on empty feed

* fall back to account creation time as rss feed update time

* return feed early when account has no eligible statuses
2023-07-10 17:05:59 +02:00
Daenney f0dad439f6
[feature] Use Read/Write lock for config (#1969) 2023-07-10 12:56:14 +01:00
Daenney 65c89709bc
[chore] Fix password typos (#1966)
Fixes #1963
2023-07-09 18:25:37 +02:00
tobi 6dbb8ba7aa
[bugfix] Fix delete follow req instead of follow (#1962) 2023-07-08 23:12:06 +02:00
tobi f40bb02f31
[bugfix] Delete mutual follow (requests) when receiving block from remote (#1960)
* [bugfix] Delete mutual follow (requests) on block

* fix test
2023-07-08 16:43:12 +02:00
tobi 2a99df0588
[feature] enable + document explicit IP dialer allowing/denying (#1950)
* [feature] enable + document explicit IP dialer allowing/denying

* lord have mercy

* allee jonge

* shortcut check ipv6 prefixes

* comment

* separate httpclient_test, export Sanitizer
2023-07-07 16:17:39 +02:00
tobi ac564c1862
[bugfix] Reorder web view logic, other small fixes (#1954) 2023-07-07 14:58:53 +02:00
tobi e70bf8a6c8
[chore/bugfix] Domain block tidying up, Implement first pass of 207 Multi-Status (#1886)
* [chore/refactor] update domain block processing

* expose domain block import errors a lil better

* move/remove unused query keys
2023-07-07 11:34:12 +02:00
tobi d9c69f6ce0
[chore/performance] Remove remaining 'whereEmptyOrNull' funcs (#1946) 2023-07-05 12:34:37 +02:00
tobi 3d16962173
[chore/bugfix] Break Websockets logic into smaller read/write functions, don't log expected errors (#1932)
* [chore/bugfix] Break Websockets logic into smaller read/write functions, don't log expected errors

* tweak

* tidy up, use control message
2023-07-04 12:55:10 +02:00
Daenney 2a40c81f10
[bugfix] Try to fix the webfinger test, again (#1931)
For some reason we hit the case in CI where the
TestFingerWithHostMetaCacheStrategy seems to experience some time
dilation. It's possible this is a genuine bug, but I can't for the life
of me reproduce it locally, even after having run this test thousands of
times (-count=1000 when invoking go test etc.)

This changes the test to explicitly stop the webfinger cache, set TTL
and Sweep frequency to something well beyond the lifetime of the cache
during the test and then starts the cache again. Hopefully that does it,
because the other option that remains is that for some reason
timekeeping in CI/Docker is not as precise as when running the test on a
host.
2023-06-27 18:32:03 +02:00
tobi d98b6318ac
[bugfix] Use gtserror package for WrongType errs (#1930)
* [bugfix] Use gtserror package for WrongType errs

* test
2023-06-27 11:37:42 +02:00
tobi 52ebff5e7e
[bugfix] Only mark status orphaned on 410 Gone (#1923) 2023-06-24 13:59:28 +02:00
tobi 3e19f480e6
[bugfix] Ensure InReplyToID set properly, update dereference ancestors func (#1921) 2023-06-24 08:32:10 +01:00
kim 9a22102fa8
[bugfix/chore] oauth entropy fix + media cleanup tasks rewrite (#1853) 2023-06-22 20:46:36 +01:00
kim 8e0043104d
[performance] update go-cache library (#1917)
* update go-cache library

Signed-off-by: kim <grufwub@gmail.com>

* fix broken test after cache library upgrade

Signed-off-by: kim <grufwub@gmail.com>

* fix the webfinger test

Signed-off-by: kim <grufwub@gmail.com>

---------

Signed-off-by: kim <grufwub@gmail.com>
2023-06-21 21:08:48 +02:00
tobi 831ae09f8b
[feature] Add partial text search for accounts + statuses (#1836) 2023-06-21 17:26:40 +01:00
Umar Getagazov 84eb7eaf26
[chore] Fix Swagger paths for lists API endpoints (#1904) 2023-06-19 10:00:19 +02:00
Umar Getagazov 73bfb5fbff
[bugfix] Parse POST-style forms in the list member removal endpoint (#1903) 2023-06-19 09:58:58 +02:00
tobi d8e16a226a
[chore/bugfix] Refactor ap/extract.go functions, return URIs more reliably (#1897) 2023-06-17 16:49:11 +01:00
Umar Getagazov 0fa06c0cde
[bugfix] Accept non-multipart forms for account updates (#1896)
* [bugfix] Update Swagger schema per max_profile_fields addition

* [bugfix] Accept non-multipart forms for account updates
2023-06-16 11:16:04 +02:00
tobi 827cc4df56
[chore/bugfix] Demote failed inbox forwarding to warn log rather than error return (#1892) 2023-06-14 15:08:31 +02:00
tobi 24fbdf2b0a
[chore] Refactor AP authentication, other small bits of tidying up (#1874) 2023-06-13 15:47:56 +01:00
f0x52 8fb5a7e7f8
[Frontend] Settings for profile fields (#1885)
* get max emoji size from instance settings

* expose (hardcoded) max amount of profile fields in instance api

* basic profile field setting

* fix profile field hook structure for updates

* *twirls mustache* fix ze tests

---------

Co-authored-by: tsmethurst <tobi.smethurst@protonmail.com>
2023-06-13 12:21:26 +02:00
tobi 5e2897e35c
[bugfix] Invalidate timeline entries for status when stats change (#1879) 2023-06-11 10:18:44 +01:00
tobi b0015a3604
[bugfix] Delete from list timeline on status deletion (#1878) 2023-06-10 11:11:54 +02:00
tobi d7d660374a
[bugfix] Fix slow joined home timeline query (#1867) 2023-06-04 20:17:28 +01:00
tobi 97bc2e713a
[chore] tidy + test timelines a bit better (#1865)
* [chore] tidy + test timelines a bit better

* thanks linter
2023-06-04 18:55:30 +02:00
tobi 21c1552daa
[chore] Update versions, fix lint errors (#1860) 2023-06-03 13:58:57 +02:00
tobi 1d4137fb88
[bugfix] Fix first item of thread dereferencing always being skipped (#1858)
* [bugfix] Fix first item of thread dereferencing always being skipped

* tweak to status descendant item iteration

Signed-off-by: kim <grufwub@gmail.com>

---------

Signed-off-by: kim <grufwub@gmail.com>
Co-authored-by: kim <grufwub@gmail.com>
2023-06-03 11:35:15 +02:00
Julian-Samuel Gebühr 1f39275c0f
[bugfix] Allow lowercase emoji shortcode in frontend (#1851)
* Replace pinafore with semaphore

* Typo

* Allow lowercase emoji shortcode in frontend

* Fix failing test (corrected expected outcome)
2023-06-02 17:42:14 +02:00
tobi 2358cf4e43
[bugfix] Overwrite API client closed errors with 499 - Client Closed Request (#1857)
* [bugfix] Overwrite client closed errors with 499

* bleep bloop

* review changes
2023-06-02 15:19:43 +02:00
kim 20978b1278
[performance] wrap httpclient response body to ensure drained before close (#1854)
Signed-off-by: kim <grufwub@gmail.com>
2023-06-02 10:34:52 +02:00
kim 9da20eeecb
[bugfix] only attempt to populate account/statuses from DB if already exist (#1839)
* only attempt to populate account/statuses from DB if already up-to-date

Signed-off-by: kim <grufwub@gmail.com>

* add missing status is-up-to-date check :grimace: + ensure populated if so

Signed-off-by: kim <grufwub@gmail.com>

---------

Signed-off-by: kim <grufwub@gmail.com>
2023-05-31 10:39:54 +02:00
tobi 46d4ec0f05
[bugfix/chore] Inbox post updates (#1821)
Co-authored-by: kim <grufwub@gmail.com>
2023-05-28 20:05:15 +01:00
kim 5faeb4de20
[chore] tidy up media manager, add calling func to errors, build-script improvements (#1835)
* media manager tidy-up: de-interface and remove unused PostDataFunc

Signed-off-by: kim <grufwub@gmail.com>

* remove last traces of media.Manager being an interface

Signed-off-by: kim <grufwub@gmail.com>

* update error to provide caller, allow tuneable via build tags

Signed-off-by: kim <grufwub@gmail.com>

* remove kim-specific build script changes

Signed-off-by: kim <grufwub@gmail.com>

* fix merge conflicts

Signed-off-by: kim <grufwub@gmail.com>

* update build-script to support externally setting build variables

Signed-off-by: kim <grufwub@gmail.com>

---------

Signed-off-by: kim <grufwub@gmail.com>
2023-05-28 14:08:35 +02:00
Daenney e0962987aa
[bugfix/docs] Fix clear notifications API docs (#1831)
The handler is already hooked to the right path, it's just the docs that
were missing the path.
2023-05-25 16:56:33 +02:00
tobi fc524f8cf1
[feature] Configurable custom css length (#1827)
* [feature] Make accounts custom css length configurable

* test custom css validation
2023-05-25 15:18:15 +02:00
decentral1se 45f961a9fe
[bugfix] allow usernames of length 1 (#1823)
Closes https://github.com/superseriousbusiness/gotosocial/issues/1691.

Co-authored-by: decentral1se <cellarspoon@riseup.net>
2023-05-25 11:15:10 +02:00
tobi f5c004d67d
[feature] Add List functionality (#1802)
* start working on lists

* further list work

* test list db functions nicely

* more work on lists

* peepoopeepoo

* poke

* start list timeline func

* we're getting there lads

* couldn't be me working on stuff... could it?

* hook up handlers

* fiddling

* weeee

* woah

* screaming, pissing

* fix streaming being a whiny baby

* lint, small test fix, swagger

* tidying up, testing

* fucked! by the linter

* move timelines to state like a boss

* add timeline start to tests using state

* invalidate lists
2023-05-25 10:37:38 +02:00
tobi c48abd8bc0
[chore] update account statuses paging logic (#1814) 2023-05-22 16:32:36 +02:00
Julian-Samuel Gebühr 9c24dee01f
[chore] Replace pinafore with semaphore (#1801)
* Replace pinafore with semaphore

* Typo
2023-05-21 22:40:43 +02:00
kim 2063d01cdb
[bugfix] Add back removed ValidateRequest() before backoff-retry loop (#1805)
* add back removed ValidateRequest() before backoff-retry loop

Signed-off-by: kim <grufwub@gmail.com>

* include response body in error response log

Signed-off-by: kim <grufwub@gmail.com>

* improved error response body draining

Signed-off-by: kim <grufwub@gmail.com>

* add more code commenting

Signed-off-by: kim <grufwub@gmail.com>

* move new error response logic to gtserror, handle instead in transport.Transport{} impl

Signed-off-by: kim <grufwub@gmail.com>

* appease ye oh mighty linter

Signed-off-by: kim <grufwub@gmail.com>

* fix mockhttpclient not setting request in http response

Signed-off-by: kim <grufwub@gmail.com>

---------

Signed-off-by: kim <grufwub@gmail.com>
2023-05-21 18:59:14 +02:00
Daenney 107237c8e8
[feature] Make client IP logging configurable (#1799) 2023-05-21 16:12:47 +01:00
tobi 99ddaba57e
[bugfix] Fix incorrect default for empty emoji domain (#1803) 2023-05-21 13:45:24 +02:00
Daenney c7702c47bb
[chore] Change time comparison in webfinger test (#1798)
Every now and then the TestFingerWithHostMetaCacheStrategy would fail on
a time related error. I suspect suite.Equal doesn't quite work as
expected when given two time.Time's, so instead explicitly check with
the time.Equal.
2023-05-16 15:09:29 +02:00
Daenney e91cabb704
[bugfix] Fix NegotiateFormat with multiple accept headers (#1797)
* [bugfix] Fix NegotiateAccept with multi accept

There's a bug in Gin's NegotiateFormat that doesn't handle the presence
of multilpe accept headers. This lifts the code from the PR @tsmethurst
sent a year ago to Gin into our codebase to fix the issue.

* [bugfix] Concat accept header in webfinger

Some implementations bug out when there's multiple accept headers,
including Gin (see 7050112af1). But things
seem to work reliably with a single accept header with multiple parts.

Fixes: #1793
2023-05-16 15:08:45 +02:00
tobi 17b9a937b1
[bugfix] Fix duplicating fields on profile edit (#1788)
* [bugfix] Fix duplicating fields on profile edit

* test non-duplicate fields
2023-05-15 12:52:40 +02:00
f0x52 89dcbd5a20
[frontend] Basic user moderation actions (#1728)
* remove info banner

* update swagger definition for AccountAction

* basic user view, suspend action

* clean up suspended user display

* basic user searching

* rename User -> Account for clarity

* refactor error boundary component to give better info

* appease the linter
2023-05-13 12:17:22 +02:00
tobi ec325fee14
[chore] Update a bunch of database dependencies (#1772)
* [chore] Update a bunch of database dependencies

* fix lil thing
2023-05-12 14:33:40 +02:00
tobi 8eda0051ec
[bugfix] Ensure account fields can be set by JSON (#1762) 2023-05-12 11:17:31 +02:00
kim 6c9d8e78eb
[feature] status refetch support (#1690)
* revamp http client to not limit requests, instead use sender worker

Signed-off-by: kim <grufwub@gmail.com>

* remove separate sender worker pool, spawn 2*GOMAXPROCS batch senders each time, no need for transport cache sweeping

Signed-off-by: kim <grufwub@gmail.com>

* improve batch senders to keep popping recipients until remote URL found

Signed-off-by: kim <grufwub@gmail.com>

* fix recipient looping issue

Signed-off-by: kim <grufwub@gmail.com>

* move request id ctx key to gtscontext, finish filling out more code comments, add basic support for not logging client IP

Signed-off-by: kim <grufwub@gmail.com>

* first draft of status refetching logic

Signed-off-by: kim <grufwub@gmail.com>

* fix testrig to use new federation alloc func signature

Signed-off-by: kim <grufwub@gmail.com>

* fix log format directive

Signed-off-by: kim <grufwub@gmail.com>

* add status fetched_at migration

Signed-off-by: kim <grufwub@gmail.com>

* remove unused / unchecked for error types

Signed-off-by: kim <grufwub@gmail.com>

* add back the used type...

Signed-off-by: kim <grufwub@gmail.com>

* add separate internal getStatus() function for derefThread() that doesn't recurse

Signed-off-by: kim <grufwub@gmail.com>

* improved mention and media attachment error handling

Signed-off-by: kim <grufwub@gmail.com>

* fix log and error format directives

Signed-off-by: kim <grufwub@gmail.com>

* update account deref to match status deref changes

Signed-off-by: kim <grufwub@gmail.com>

* very small code formatting change to make things clearer

Signed-off-by: kim <grufwub@gmail.com>

* add more code comments

Signed-off-by: kim <grufwub@gmail.com>

* improved code commenting

Signed-off-by: kim <grufwub@gmail.com>

* only check for required further derefs if needed

Signed-off-by: kim <grufwub@gmail.com>

* improved cache invalidation

Signed-off-by: kim <grufwub@gmail.com>

* tweak cache restarting to use a (very small) backoff

Signed-off-by: kim <grufwub@gmail.com>

* small readability changes and fixes

Signed-off-by: kim <grufwub@gmail.com>

* fix account sync issues

Signed-off-by: kim <grufwub@gmail.com>

* fix merge conflicts + update account enrichment to accept already-passed accountable

Signed-off-by: kim <grufwub@gmail.com>

* remove secondary function declaration

Signed-off-by: kim <grufwub@gmail.com>

* normalise dereferencer get status / account behaviour, fix remaining tests

Signed-off-by: kim <grufwub@gmail.com>

* fix remaining rebase conflicts, finish commenting code

Signed-off-by: kim <grufwub@gmail.com>

* appease the linter

Signed-off-by: kim <grufwub@gmail.com>

* add source file header

Signed-off-by: kim <grufwub@gmail.com>

* update to use TIMESTAMPTZ column type instead of just TIMESTAMP

Signed-off-by: kim <grufwub@gmail.com>

* don't pass in 'updated_at' to UpdateEmoji()

Signed-off-by: kim <grufwub@gmail.com>

* use new ap.Resolve{Account,Status}able() functions

Signed-off-by: kim <grufwub@gmail.com>

* remove the somewhat confusing rescoping of the same variable names

Signed-off-by: kim <grufwub@gmail.com>

* update migration file name, improved database delete error returns

Signed-off-by: kim <grufwub@gmail.com>

* formatting

Signed-off-by: kim <grufwub@gmail.com>

* improved multi-delete database functions to minimise DB calls

Signed-off-by: kim <grufwub@gmail.com>

* remove unused type

Signed-off-by: kim <grufwub@gmail.com>

* fix delete statements

Signed-off-by: kim <grufwub@gmail.com>

---------

Signed-off-by: kim <grufwub@gmail.com>
2023-05-12 11:15:54 +02:00
Natsu Kagami ba5a464ca5
[chore] Prefer JSON errors in API endpoints (#1766)
* Default to JSON over HTML for error handling

* Change the default error display for web endpoints to html
2023-05-12 10:16:41 +02:00
f0x52 9cc9ffc5a7
[frontend] Profiles with fields & more (#1764)
* redesign status template

* separate index page styling

* redesign profile template

* fix header styling/wrapping

* remove old spoiler js

* fix status cw button wrapping

* fix status info variables

* profile responsiveness, accessibility tweaks

* fix variable use, mobile

* remove duplicate id's

* rss icon, fix indent

* fix toot border-radius

* fix toot spacing

* emojify and html profile fields

* refactor (sensitive) media rendering

* plaintext profile fields

* bundle plyr icon svg

* only pause video when switching photoswipe slides

* yarn upgrade

* profile fields formatting

* replace uglifyify with @browserify updated fork

* fix profile field templating (yet again)

* fix React classes

* testrig: add testing profile field for admin user

* fix sensitive media interactions

* Revert "testrig: add testing profile field for admin user"

This reverts commit 80490c183e.

* settings interface wrapping

* fix reported toot styling

* add role to profile sr-only text

* comment fallback rule

* remove currently unused image description lacking indicator
2023-05-11 17:46:32 +02:00
Dominik Süß 6392e00653
feat: initial tracing support (#1623) 2023-05-09 18:19:48 +01:00
tobi 878ed48de3
[bugfix] Don't try to get user when serializing local instance account (#1757) 2023-05-09 16:05:35 +01:00
kim 11e843a273
[bugfix] fix possible domain blockcache nil ptr + add debug String() func (#1755) 2023-05-09 15:18:51 +01:00
kim 8275d70e38
[bugfix] update go-cache version to support multi-keying (#1756)
* update go-cache version to support multi-keying

Signed-off-by: kim <grufwub@gmail.com>

* improved cache invalidation

Signed-off-by: kim <grufwub@gmail.com>

---------

Signed-off-by: kim <grufwub@gmail.com>
2023-05-09 16:17:43 +02:00
tobi 65cd1acbdf
[chore] tidy + update StatusToAPIStatus (#1754) 2023-05-09 13:25:48 +02:00
tobi 0e29f1f5bb
[feature] Enable federation in/out of profile PropertyValue fields (#1722)
Co-authored-by: kim <grufwub@gmail.com>
Co-authored-by: kim <89579420+NyaaaWhatsUpDoc@users.noreply.github.com>
2023-05-09 11:16:10 +01:00
tobi cbb9e2d3f0
[chore/performance] Make sender multiplier configurable (#1750) 2023-05-08 18:03:38 +01:00
tobi 37b4d9d179
[bugfix] Punycode fixes (#1743)
Co-authored-by: kim <grufwub@gmail.com>
Co-authored-by: kim <89579420+NyaaaWhatsUpDoc@users.noreply.github.com>
2023-05-07 18:53:21 +01:00
Daenney b7dd32da42
[bugfix] Return languages in api/v1/instance (#1741)
It turns out that in Masto v2.3.0 the languages key was added to the V1
Instance and that it's effectively mandatory. Though in GtS we don't
really have this concept yet, some apps will explode if the languages
key is missing altogether. So at least return the empty array on V1 too
in the hopes that it makes things work well enough.

For history's sake, you can see the attributes that will get serialised
in
f877aa9d70/app/serializers/rest/v1/instance_serializer.rb (L6-L9).
Because the attribute does not have a conditional defined for it,
there isn't a filter that optionally omits it, or a def languages to
modify the behaviour the attribute is effectively always included and
serialised.

Fixes: #1662
2023-05-06 22:38:19 +02:00
Daenney 6d138588d8
[feature] Implement the preferences client API (#1740)
This adds the preferences endpoint to our Mastodon Client API
implementation. It's a read-only endpoint that returns a number of
user preferences. Applications can query these settings when logging in
a user (for the first time) to configure themselves.
2023-05-06 17:42:58 +02:00
tobi 5027d0ced2
[bugfix] Serve correct 'application/jrd+json' content type for webfinger requests (#1738)
* [bugfix] Return `application/jrd+json` from webfinger queries

* update finger req content-type
2023-05-04 12:28:50 +02:00
tobi 4a012acd52
[bugfix] Rework notifs to use min_id for paging up (#1734) 2023-05-04 12:27:24 +02:00
tobi a6ec2a5bc2
[bugfix] Fix invalid og:description on account w/ empty note (#1733) 2023-05-03 16:18:34 +02:00
kim 3ff1391a9d
[performance] replace domain block cache with an in-memory radix trie (#1714)
* replace domain block cache with an in-memory radix tree

Signed-off-by: kim <grufwub@gmail.com>

* fix domain block cache init

Signed-off-by: kim <grufwub@gmail.com>

---------

Signed-off-by: kim <grufwub@gmail.com>
2023-05-01 12:36:46 +02:00
kim ead286a67b
[bugfix] tweak httpclient error handling again ... (#1721)
* check for tls, x509 errors using string.Contains() since crypto/tls sucks

Signed-off-by: kim <grufwub@gmail.com>

* use 2* maxprocs

Signed-off-by: kim <grufwub@gmail.com>

---------

Signed-off-by: kim <grufwub@gmail.com>
2023-04-30 10:11:18 +02:00
kim 68b91d2128
[performance] tweak http client error handling (#1718)
* update errors library, check for more TLS type error in http client

Signed-off-by: kim <grufwub@gmail.com>

* bump cache library version to match errors library

Signed-off-by: kim <grufwub@gmail.com>

---------

Signed-off-by: kim <grufwub@gmail.com>
2023-04-29 18:44:20 +02:00
tobi 8b1e2288d8
[feature] Add GET endpoint for single notification (#1719) 2023-04-29 17:29:51 +01:00
Mal Hancock fdd2487cfb
[bugfix] add From to email header (#1717)
* add From to email header

* update tests
2023-04-29 11:47:14 +02:00
kim 6a29c5ffd4
[performance] improved request batching (removes need for queueing) (#1687)
* revamp http client to not limit requests, instead use sender worker

Signed-off-by: kim <grufwub@gmail.com>

* remove separate sender worker pool, spawn 2*GOMAXPROCS batch senders each time, no need for transport cache sweeping

Signed-off-by: kim <grufwub@gmail.com>

* improve batch senders to keep popping recipients until remote URL found

Signed-off-by: kim <grufwub@gmail.com>

* fix recipient looping issue

Signed-off-by: kim <grufwub@gmail.com>

* fix missing mutex unlock

Signed-off-by: kim <grufwub@gmail.com>

* move request id ctx key to gtscontext, finish filling out more code comments, add basic support for not logging client IP

Signed-off-by: kim <grufwub@gmail.com>

* slight code reformatting

Signed-off-by: kim <grufwub@gmail.com>

* a whitespace

Signed-off-by: kim <grufwub@gmail.com>

* remove unused code

Signed-off-by: kim <grufwub@gmail.com>

* add missing license headers

Signed-off-by: kim <grufwub@gmail.com>

* fix request backoff calculation

Signed-off-by: kim <grufwub@gmail.com>

---------

Signed-off-by: kim <grufwub@gmail.com>
2023-04-28 17:45:21 +02:00
tobi 6b4f6dc755
[bugfix] Fix remaining mangled URI escaping issues in statuses + accounts (#1712)
* start fiddling with normalize + extract functions

* normalize attachment name (image description)

* NormalizeAccountableSummary

* normalize summary + name
2023-04-26 16:17:22 +01:00
Umar Getagazov cb1f935013
[bugfix] Fix the bookmarks list API endpoint returning an empty array (#1700) 2023-04-19 13:42:00 +02:00
tobi 093cf2ab12
[feature] Receive notification when followed account posts (if desired) (#1680)
* start working on notifs for new posts

* tidy up a bit

* update swagger

* carry over show reblogs + notify from follow req

* test notify on status post

* update column slice

* dedupe update logic + add tests

* fix own boosts not being timelined

* avoid type check, passing unnecessary accounts

* remove unnecessary 'inReplyToID' check

* add a couple todo's for future db functions
2023-04-10 20:56:02 +01:00
tobi f28ed21343
[bugfix] Fix single author threads not appearing in home timeline (#1679) 2023-04-08 12:56:55 +02:00
kim e46323c207
[bugfix] visibility caching and hometimeline (#1675)
* fix visibility caching to use correct type key

Signed-off-by: kim <grufwub@gmail.com>

* check for ID check > max possible ID

Signed-off-by: kim <grufwub@gmail.com>

* update home timeline code to include relevant threads to owner (e.g. between mutuals/follows)

Signed-off-by: kim <grufwub@gmail.com>

---------

Signed-off-by: kim <grufwub@gmail.com>
2023-04-06 17:11:25 +02:00
tobi 3510454768
[bugfix/chore] Refactor timeline code (#1656)
* start poking timelines

* OK yes we're refactoring, but it's nothing like the last time so don't worry

* more fiddling

* update tests, simplify Get

* thanks linter, you're the best, mwah mwah kisses

* do a bit more tidying up

* start buggering about with the prepare function

* fix little oopsie

* start merging lists into 1

* ik heb een heel zwaar leven
nee nee echt waar

* hey it works we did it reddit

* regenerate swagger docs

* tidy up a wee bit

* adjust paging

* fix little error, remove unused functions
2023-04-06 12:43:13 +01:00
tobi c54510bc74
[bugfix] Normalize status content (don't parse status content as IRI) (#1665)
* start fannying about

* finish up Normalize

* tidy up

* pin to tag

* move errors about just a little bit
2023-04-06 12:19:55 +01:00
tobi 4f322f527f
[bugfix] Always serialize orderedItems as array (#1673) 2023-04-06 12:16:53 +01:00
tobi 8d2a76c58c
[bugfix] Add proper constraints on status faves, dedupe (#1674)
* [bugfix] Start working on multiple like issue

* finish up
2023-04-05 20:10:05 +02:00
tobi b847af1dbd
[bugfix] Add idempotency-key to allowed CORS headers (#1670) 2023-04-03 12:01:24 +02:00
tobi 60639a6a0e
[bugfix] Fix multiple "updated_at" columns for media updates (#1660)
* [bugfix] Fix multiple "updated_at" columns for media updates

* silly unrelated race condition
2023-03-31 15:19:50 +02:00
tobi d9bbcc60a6
[bugfix] Fix report serialization errors caused by user delete (#1659)
* [bugfix] Fix report serialization errors caused by user delete

* fix tests
2023-03-31 15:01:29 +02:00