Commit graph

93 commits

Author SHA1 Message Date
firescry 8d92b2479f
[bugfix] Align default values in the configuration file with the code (#1971)
* Set default value of SMTPFrom to empty string

This parameter should contain proper e-mail address (to be provided by user during configuration).

* Update default values in example/config.yaml

Default values and related comments in example/config.yaml are aligned
with values defined in internal/config/defaults.go.
Small improvements to foramting of config.yaml file.

* Add default value for AdvancedThrottlingRetryAfter to internal/config/defaults.go

AdvancedThrottlingRetryAfter was introduced in 70739d3 (superseriousbusiness/gotosocial#1466).

* Update config.yaml snippets in documentation
2023-07-12 10:28:41 +02:00
Daenney f0dad439f6
[feature] Use Read/Write lock for config (#1969) 2023-07-10 12:56:14 +01:00
tobi 2a99df0588
[feature] enable + document explicit IP dialer allowing/denying (#1950)
* [feature] enable + document explicit IP dialer allowing/denying

* lord have mercy

* allee jonge

* shortcut check ipv6 prefixes

* comment

* separate httpclient_test, export Sanitizer
2023-07-07 16:17:39 +02:00
tobi e70bf8a6c8
[chore/bugfix] Domain block tidying up, Implement first pass of 207 Multi-Status (#1886)
* [chore/refactor] update domain block processing

* expose domain block import errors a lil better

* move/remove unused query keys
2023-07-07 11:34:12 +02:00
tobi fc524f8cf1
[feature] Configurable custom css length (#1827)
* [feature] Make accounts custom css length configurable

* test custom css validation
2023-05-25 15:18:15 +02:00
tobi f5c004d67d
[feature] Add List functionality (#1802)
* start working on lists

* further list work

* test list db functions nicely

* more work on lists

* peepoopeepoo

* poke

* start list timeline func

* we're getting there lads

* couldn't be me working on stuff... could it?

* hook up handlers

* fiddling

* weeee

* woah

* screaming, pissing

* fix streaming being a whiny baby

* lint, small test fix, swagger

* tidying up, testing

* fucked! by the linter

* move timelines to state like a boss

* add timeline start to tests using state

* invalidate lists
2023-05-25 10:37:38 +02:00
Daenney 107237c8e8
[feature] Make client IP logging configurable (#1799) 2023-05-21 16:12:47 +01:00
Dominik Süß 6392e00653
feat: initial tracing support (#1623) 2023-05-09 18:19:48 +01:00
tobi cbb9e2d3f0
[chore/performance] Make sender multiplier configurable (#1750) 2023-05-08 18:03:38 +01:00
kim de6e3e5f2a
[performance] refactoring + add fave / follow / request / visibility caching (#1607)
* refactor visibility checking, add caching for visibility

* invalidate visibility cache items on account / status deletes

* fix requester ID passed to visibility cache nil ptr

* de-interface caches, fix home / public timeline caching + visibility

* finish adding code comments for visibility filter

* fix angry goconst linter warnings

* actually finish adding filter visibility code comments for timeline functions

* move home timeline status author check to after visibility

* remove now-unused code

* add more code comments

* add TODO code comment, update printed cache start names

* update printed cache names on stop

* start adding separate follow(request) delete db functions, add specific visibility cache tests

* add relationship type caching

* fix getting local account follows / followed-bys, other small codebase improvements

* simplify invalidation using cache hooks, add more GetAccountBy___() functions

* fix boosting to return 404 if not boostable but no error (to not leak status ID)

* remove dead code

* improved placement of cache invalidation

* update license headers

* add example follow, follow-request config entries

* add example visibility cache configuration to config file

* use specific PutFollowRequest() instead of just Put()

* add tests for all GetAccountBy()

* add GetBlockBy() tests

* update block to check primitive fields

* update and finish adding Get{Account,Block,Follow,FollowRequest}By() tests

* fix copy-pasted code

* update envparsing test

* whitespace

* fix bun struct tag

* add license header to gtscontext

* fix old license header

* improved error creation to not use fmt.Errorf() when not needed

* fix various rebase conflicts, fix account test

* remove commented-out code, fix-up mention caching

* fix mention select bun statement

* ensure mention target account populated, pass in context to customrenderer logging

* remove more uncommented code, fix typeutil test

* add statusfave database model caching

* add status fave cache configuration

* add status fave cache example config

* woops, catch missed error. nice catch linter!

* add back testrig panic on nil db

* update example configuration to match defaults, slight tweak to cache configuration defaults

* update envparsing test with new defaults

* fetch followingget to use the follow target account

* use accounnt.IsLocal() instead of empty domain check

* use constants for the cache visibility type check

* use bun.In() for notification type restriction in db query

* include replies when fetching PublicTimeline() (to account for single-author threads in Visibility{}.StatusPublicTimelineable())

* use bun query building for nested select statements to ensure working with postgres

* update public timeline future status checks to match visibility filter

* same as previous, for home timeline

* update public timeline tests to dynamically check for appropriate statuses

* migrate accounts to allow unique constraint on public_key

* provide minimal account with publicKey

---------

Signed-off-by: kim <grufwub@gmail.com>
Co-authored-by: tsmethurst <tobi.smethurst@protonmail.com>
2023-03-28 14:03:14 +01:00
tobi 7db81cde44
[feature] Email notifications for new / closed moderation reports (#1628)
* start fiddling about with email sending to allow multiple recipients

* do some fiddling

* notifs working

* notify on closed report

* finishing up

* envparsing

* use strings.ContainsAny
2023-03-19 13:11:46 +01:00
Daenney 5e2bf0bdca
[chore] Improve copyright header handling (#1608)
* [chore] Remove years from all license headers

Years or year ranges aren't required in license headers. Many projects
have removed them in recent years and it avoids a bit of yearly toil.

In many cases our copyright claim was also a bit dodgy since we added
the 2021-2023 header to files created after 2021 but you can't claim
copyright into the past that way.

* [chore] Add license header check

This ensures a license header is always added to any new file. This
avoids maintainers/reviewers needing to remember to check for and ask
for it in case a contribution doesn't include it.

* [chore] Add missing license headers

* [chore] Further updates to license header

* Use the more common // indentend comment format
* Remove the hack we had for the linter now that we use the // format
* Add SPDX license identifier
2023-03-12 16:00:57 +01:00
Daenney e397272fe8
[feature] Discover webfinger through host-meta (#1588)
* [feature] Discover webfinger through host-meta

This implements a fallback for discovering the webfinger endpoint in
case the /.well-known/webfinger endpoint wasn't properly redirected.
Some instances do this because the recommendation used to be to use
host-meta for the webfinger redirect in the before times.

Closes #1558.

* [bug] Ensure we only ever update cache on success

* [chore] Move finger tests to their own place

This adds a test suite for transport and moves the finger cache tests
into there instead of abusing the search test suite.

* [chore] cleanup the test a bit more

We don't really need a separate function for the oddly located webfinger
response as we check the full URL string anyway

* Address review comments

* [chore] update config example

* [chore] access DB only through state in controller
2023-03-08 12:57:41 +00:00
Daenney d2f6de0185
[feature] Allow loading TLS certs from disk (#1586)
Currently, GtS only supports using the built-in LE client directly for
TLS. However, admins may still want to use GtS directly (so without a
reverse proxy) but with certificates provided through some other
mechanism. They may have some centralised way of provisioning these
things themselves, or simply prefer to use LE but with a different
challenge like DNS-01 which is not supported by autocert.

This adds support for loading a public/private keypair from disk instead
of using LE and reconfigures the server to use a TLS listener if we
succeed in doing so.

Additionally, being able to load TLS keypair from disk opens up the path
to using a custom CA for testing purposes avoinding the need for a
constellation of containers and something like Pebble or Step CA to
provide LE APIs.
2023-03-04 17:24:02 +00:00
kim a8e6bdfa33
[performance] cache media attachments (#1525)
* replace concurrency worker pools with base models in State.Workers, update code and tests accordingly

* add media attachment caching, slightly tweak default cache config

* further tweak default cache config values

* replace other media attachment db calls to go through cache

* update envparsing test

* fix delete media attachment sql

* fix media sql query

* invalidate cached media entries during status create / update

* fix envparsing test

* fix typo in panic log message...

* add 'updated_at' column during UpdateAttachment

* remove unused func

---------

Signed-off-by: kim <grufwub@gmail.com>
2023-03-03 23:02:23 +00:00
Daenney 9cfb69f75d
[feature] Make OIDC admin groups configurable (#1555)
This removes the current default of checking for membership of the admin
or admins group and makes it required to explicitly configure which
groups should grant admin access, if any.

Relying on the implicit default of admin or admins is potentially
dangerous as that group may contain a different subset of people that we
may wish to grant admin access to GtS. This is probably not an issue for
a single-person instance, but for a community instance different admin
groups may exist in an OIDC provider for different applications.

I'm explicitly opting for not defaulting the value of oidc-admin-groups
to admin,admins because I think it's better for those things to be
explicitly configured.
2023-02-25 16:37:39 +00:00
Daenney 68e6d08c76
[feature] Add a request ID and include it in logs (#1476)
This adds a lightweight form of tracing to GTS. Each incoming request is
assigned a Request ID which we then pass on and log in all our log
lines. Any function that gets called downstream from an HTTP handler
should now emit a requestID=value pair whenever it logs something.

Co-authored-by: kim <grufwub@gmail.com>
2023-02-17 12:02:29 +01:00
tobi 40bc03e717
[chore/performance] Update media prune logic, add extra CLI command (#1474)
* start updating media prune stuff a wee bit

* continue prune / uncache work

* more tidying + consistency stuff

* add prune CLI command

* docs

* arg
2023-02-11 12:48:38 +01:00
kim 70739d32cc
[performance] remove throttling timers (#1466)
* remove throttling timers, support setting retry-after, use retry-after in transport

* remove unused variables

* add throttling-retry-after to cmd flags

* update envparsing to include new throttling-retry-after

* update example config to include retry-after documentation

* also support retry-after formatted as date-time, ensure max backoff time

---------

Signed-off-by: kim <grufwub@gmail.com>
2023-02-10 20:16:01 +00:00
tobi 782169da76
[chore] set max open / idle conns + conn max lifetime for both postgres and sqlite (#1369)
* [chore] set max open / idle conns + conn max lifetime for both postgres and sqlite

* reduce cache size default to 8MiB, reduce connections to 2 * cpu

* introduce max open conns multiplier, tune sqlite and pg separately

* go fmt
2023-01-26 15:12:48 +01:00
f0x52 17eecfb6d9
[feature] Public list of suspended domains (#1362)
* basic rendered domain blocklist (unauthenticated!)

* style basic domain block list

* better formatting for domain blocklist

* add opt-in config option for showing suspended domains

* format/linter

* re-use InstancePeersGet for web-accessible domain blocklist

* reword explanation, border styling

* always attach blocklist handler, update error message

* domain blocklist error message grammar
2023-01-25 18:06:41 +01:00
tobi 974ec80a20
[chore] Change default sqlite busy timeout to 5m (#1352) 2023-01-18 14:37:52 +01:00
tobi 627b8eeae6
[feature] Tune sqlite pragmas (#1349)
* sqlite pragma tuning

* use formatuint

* add sqlite busy timeout

* fix incorrect cache size format

* update envparsing test

* add sqlite tuning flags to cli

* set sqlite timeout to 30s default
2023-01-17 12:29:44 +00:00
tobi d6487933c7
[feature] Implement Report database model and utility functions (#1310)
* implement report database model

* implement report cache + config changes

* implement report database functions

* report uri / regex functions

* update envparsing test

* remove unnecessary uri index

* remove unused function + cache lookup

* process error when storing report
2023-01-10 14:19:05 +00:00
tobi 0dbe6c514f
[chore] Update/add license headers for 2023 (#1304) 2023-01-05 12:43:00 +01:00
tobi 90a14abb0c
[feature] HTTP request throttling middleware (#1297)
* [feature] Add throttling middleware to AP endpoints

* refactor a lil bit

* use config setting, start updating docs

* doc updates

* use relative links in faq doc

* small docs fixes

* return code 503 instead of 429 when throttled

* throttle other endpoints too

* simplify token channel prefills
2023-01-04 11:57:59 +01:00
tobi 941893a774
[chore] The Big Middleware and API Refactor (tm) (#1250)
* interim commit: start refactoring middlewares into package under router

* another interim commit, this is becoming a big job

* another fucking massive interim commit

* refactor bookmarks to new style

* ambassador, wiz zeze commits you are spoiling uz

* she compiles, we're getting there

* we're just normal men; we're just innocent men

* apiutil

* whoopsie

* i'm glad noone reads commit msgs haha :blob_sweat:

* use that weirdo go-bytesize library for maxMultipartMemory

* fix media module paths
2023-01-02 12:10:50 +00:00
tobi 0f38e7c9b0
[chore] fix some little config whoopsies (#1272) 2022-12-16 22:38:41 +00:00
kim cb2b2fd805
[feature] support configuring database caches (#1246)
* update config generator to support nested structs, add cache configuration options

* update envparsing test

* add cache configuration to config parse tests

* set cache configuration in testrig

* move caches to sub-cache "gts" namespace, update envparsing, add cache config docs to example config

Signed-off-by: kim <grufwub@gmail.com>
2022-12-11 13:03:15 +00:00
Dominik Süß 199b685f43
[feature] overhaul the oidc system (#961)
* [feature] overhaul the oidc system

this allows for more flexible username handling and prevents account
takeover using old email addresses

* [feature] add migration path for old OIDC users

* [feature] nicer error reporting for users

* [docs] document the new OIDC flow

* [fix] return early on oidc error

* [docs]: add comments on the finalization logic
2022-12-06 14:15:56 +01:00
tobi 13e9abd02a
[feature] Add admin media prune orphaned CLI command (#1146)
* add FilePath regex

* add `admin media prune orphaned` command

* add prune orphaned function to media manager

* don't mark flag as required

* document admin media prune orphaned cmd

* oh envparsing.sh you coy minx
2022-11-25 17:23:42 +00:00
Mara Sophie Grosch 1f256e288b
[chore] refactor test/cliparsing.sh into a go test below internal/config (#1036)
Also adds AddGlobalFlags and AddServerFlags as methods on ConfigState,
very useful for testing.
2022-11-15 16:53:19 +01:00
Rob Sargant d120743e8b
[feature] add instance-expose-public-timeline flag (#1039)
* Add instance-expose-public-timeline flag

Adds a config flag that allows unauthenticated access to /api/v1/timelines/public. Defaults to false to replicate existing behaviour.

* Update structure following review

* Add comment

* Fix linting
2022-11-14 10:30:01 +01:00
tobi 6fb47651c8
[chore] Fixes + updates in emoji dereferencing logic (#1022)
* fix incorrect static remote url use for emojis

* warn when emoji/attachment already exists

* defer emoji postdata execution

* rename ctx to innerCtx for clarity

* warn on emoji too large

* small efficiency fix in fetchRemoteAccountEmojis

* tidy up lock+load

* lock processing emojis

* fix little fucky wucky

* this wasn't go fmted for some reason
2022-11-11 19:27:37 +00:00
Mara Sophie Grosch 948e90b95a
[feature] S3: add config flag to proxy S3 media (#1014)
* S3: add config value "proxy" for not redirecting

Signed-off-by: Mara Sophie Grosch <littlefox@lf-net.org>

* S3: document new config value "proxy"

* S3: add new config value "proxy" to test scripts

Signed-off-by: Mara Sophie Grosch <littlefox@lf-net.org>
2022-11-11 12:03:18 +01:00
tobi 4d66fb9603
[feature] Make rate limit requests amount configurable (#966)
* update rate limit documentation

* regenerate landingpage config helpers

* make rate limit rate configurable
2022-11-06 09:47:48 +00:00
Mia Heidenstedt f3fc040c2e
[feature] Allow user to show instead of landing page on / (#922)
* configurable user instead of landing page

* rename DefaultUser to LandingPageUser

* code review changes and fix tests

* try to fix pipeline error

* code review changes

* code review changes

* code review changes

* remove unnecessary line
2022-11-03 13:15:31 +01:00
Mia Heidenstedt 31f0f27bed
[bugfix] Fix imports for helpers.gen.go (#924) 2022-10-29 17:14:57 +02:00
kim 1d999712e6
[feature] update config types to use bytesize.Size (#828)
* update config size types to use bytesize.Size

* submit unchecked-out file ... 🤦

* fix bytesize config var decoding

* bump bytesize version

* update kim's libraries in readme

* update envparse.sh to output more useful errors

* improve envparse.sh

* remove reliance on jq

* instead, use uint64 for bytesize flag types

* remove redundant type

* fix viper unmarshaling

* Update envparsing.sh

* fix envparsing test

Signed-off-by: kim <grufwub@gmail.com>
Co-authored-by: tobi <31960611+tsmethurst@users.noreply.github.com>
2022-09-29 21:50:43 +01:00
tobi f0bf69d4d0
[chore] Add ipv6 localhost to trusted proxies by default (#868)
* add ipv6 localhost to trusted proxies by default

* test different trusted proxies
2022-09-29 17:08:56 +02:00
tobi 69a193dae5
[feature] Allow delivery to sharedInboxes where possible (#847)
* update Activity

* add instance-deliver-to-shared-inboxes setting

* update activity version again

* add SharedInboxURI field to accounts

* serdes for endpoints/sharedInbox

* deliver to sharedInbox if one is available

* update tests

* only assign shared inbox if shared domain

* look for shared inbox if currently nil

* go fmt

* finger to get params.RemoteAccountID if necessary

* make comments clearer

* compare dns more consistently
2022-09-23 20:27:35 +01:00
tobi b42469e4e0
[feature] Allow users to set custom css for their profiles + threads (#808)
* add custom css account property + db func to fetch

* allow account to get/set custom css

* serve custom css for an account

* go fmt

* use monospace for customcss, add link

* add custom css to account cache

* fix broken field

* add custom css docs to user guide

* add `accounts-allow-custom-css` config flag

* add allow custom css to /api/v1/instance response

* only show/set custom css if allowed to do so

* only set/serve custom account css if enabled

* update swagger docs

* chain promise

* make bool a bit clearer

* use cache for GetAccountCustomCSSByUsername
2022-09-12 13:14:29 +02:00
tobi 268f252e0d
[feature] Fetch + display custom emoji in statuses from remote instances (#807)
* start implementing remote emoji fetcher

* update status where pk

* aaa

* tidy up a little

* check size limits for emojis

* thank you linter, i love you <3

* update swagger docs

* add emoji dereference test

* make emoji max sizes configurable

* normalize db.ErrAlreadyExists
2022-09-12 13:03:23 +02:00
tobi 006c8b604b
[feature] Set default header and avatar for API accounts to GtS ones (#799)
* validate web-asset-base-dir

* move default icons into converter

* always ensure avatar + header on api accounts

* update tests

* add default header

* don't return error from web module creation anymore

* tidy a bit

* use pngs for default avatars rather than svgs
2022-09-04 14:41:42 +02:00
tobi 40f91d052c
[chore] Update image/video size defaults to mastodon's (#723) 2022-07-20 12:25:06 +02:00
kim 098dbe6ff4
[chore] use our own logging implementation (#716)
* first commit

Signed-off-by: kim <grufwub@gmail.com>

* replace logging with our own log library

Signed-off-by: kim <grufwub@gmail.com>

* fix imports

Signed-off-by: kim <grufwub@gmail.com>

* fix log imports

Signed-off-by: kim <grufwub@gmail.com>

* add license text

Signed-off-by: kim <grufwub@gmail.com>

* fix package import cycle between config and log package

Signed-off-by: kim <grufwub@gmail.com>

* fix empty kv.Fields{} being passed to WithFields()

Signed-off-by: kim <grufwub@gmail.com>

* fix uses of log.WithFields() with whitespace issues and empty slices

Signed-off-by: kim <grufwub@gmail.com>

* *linter related grumbling*

Signed-off-by: kim <grufwub@gmail.com>

* gofmt the codebase! also fix more log.WithFields() formatting issues

Signed-off-by: kim <grufwub@gmail.com>

* update testrig code to match new changes

Signed-off-by: kim <grufwub@gmail.com>

* fix error wrapping in non fmt.Errorf function

Signed-off-by: kim <grufwub@gmail.com>

* add benchmarking of log.Caller() vs non-cached

Signed-off-by: kim <grufwub@gmail.com>

* fix syslog tests, add standard build tags to test runner to ensure consistency

Signed-off-by: kim <grufwub@gmail.com>

* make syslog tests more robust

Signed-off-by: kim <grufwub@gmail.com>

* fix caller depth arithmatic (is that how you spell it?)

Signed-off-by: kim <grufwub@gmail.com>

* update to use unkeyed fields in kv.Field{} instances

Signed-off-by: kim <grufwub@gmail.com>

* update go-kv library

Signed-off-by: kim <grufwub@gmail.com>

* update libraries list

Signed-off-by: kim <grufwub@gmail.com>

* fuck you linter get nerfed

Signed-off-by: kim <grufwub@gmail.com>

Co-authored-by: tobi <31960611+tsmethurst@users.noreply.github.com>
2022-07-19 10:47:55 +02:00
Dominik Süß 9d0df426da
[feature] S3 support (#674)
* feat: vendor minio client

* feat: introduce storage package with s3 support

* feat: serve s3 files directly

this saves a lot of bandwith as the files are fetched from the object
store directly

* fix: use explicit local storage in tests

* feat: integrate s3 storage with the main server

* fix: add s3 config to cli tests

* docs: explicitly set values in example config

also adds license header to the storage package

* fix: use better http status code on s3 redirect

HTTP 302 Found is the best fit, as it signifies that the resource
requested was found but not under its presumed URL

307/TemporaryRedirect would mean that this resource is usually located
here, not in this case

303/SeeOther indicates that the redirection does not link to the
requested resource but to another page

* refactor: use context in storage driver interface
2022-07-03 12:08:30 +02:00
tobi 5f00d4980b
[feature] Implement /api/v1/instance/peers endpoint (#660)
* add missing license headers

* start adding instance peers get

* rename domainblock.go

* embed domain in domainblock so it can be reused

* update swagger docs

* add test instances to db

* update tests

* add/update instancepeersget

* update domain model

* add getinstancepeers to db

* instance-expose-peers, instance-expose-suspended

* add auth checks for both current filters

* attach endpoint to router

* include public comment

* obfuscate domain if required

* go mod tidy

* update swagger docs

* remove unnecessary comment

* return 'flat' peerlist if no query params provided
2022-06-23 16:54:54 +02:00
tobi cf5c6d724d
[chore] Validate/set account domain (#619)
* add miekg/dns dependency

* set/validate accountDomain
2022-06-11 11:09:31 +02:00
kim 91c0ed863a
[bugfix] #621: add weak type handing to mapstructure decode (#625)
* Drone sig (#623)

* accept weakly typed input on mapstructure decode i.e. .UnmarshalMap()

Signed-off-by: kim <grufwub@gmail.com>

* add envparsing script to test for panics during environment variable parsing

Signed-off-by: kim <grufwub@gmail.com>

* add envparsing.sh script to drone commands

Signed-off-by: kim <grufwub@gmail.com>

* update drone signature

Co-authored-by: kim <grufwub@gmail.com>

* compare expected with output

* update expected output of envparsing

* update expected output to correct value

* use viper's unmarshal function instead
There were problems with marshalling
string slices from viper into the st.config
struct with the other function. Now, we
can use viper's unmarshal function and pass
in the custom decoder config that we need
as a hook. This ensures that we marshal
string slices from viper into our config
struct correctly.

Co-authored-by: tobi <31960611+tsmethurst@users.noreply.github.com>
Co-authored-by: tsmethurst <tobi.smethurst@protonmail.com>
2022-06-08 20:28:28 +02:00