Commit graph

1010 commits

Author SHA1 Message Date
kim a483bd9e38
[performance] massively improved ActivityPub delivery worker efficiency (#2812)
* add delivery worker type that pulls from queue to httpclient package

* finish up some code commenting, bodge a vendored activity library change, integrate the deliverypool changes into transportcontroller

* hook up queue deletion logic

* support deleting queued http requests by target ID

* don't index APRequest by hostname in the queue

* use gorun

* use the original context's values when wrapping msg type as delivery{}

* actually log in the AP delivery worker ...

* add uncommitted changes

* use errors.AsV2()

* use errorsv2.AsV2()

* finish adding some code comments, add bad host handling to delivery workers

* slightly tweak deliveryworkerpool API, use advanced sender multiplier

* remove PopCtx() method, let others instead rely on Wait()

* shuffle things around to move delivery stuff into transport/ subpkg

* remove dead code

* formatting

* validate request before queueing for delivery

* finish adding code comments, fix up backoff code

* finish adding more code comments

* clamp minimum no. senders to 1

* add start/stop logging to delivery worker, some slight changes

* remove double logging

* use worker ptrs

* expose the embedded log fields in httpclient.Request{}

* ensure request context values are preserved when updating ctx

* add delivery worker tests

* fix linter issues

* ensure delivery worker gets inited in testrig

* fix tests to delivering messages to check worker delivery queue

* update error type to use ptr instead of value receiver

* fix test calling Workers{}.Start() instead of testrig.StartWorkers()

* update docs for advanced-sender-multiplier

* update to the latest activity library version

* add comment about not using httptest.Server{}
2024-04-11 11:45:35 +02:00
tobi 6db7d014dc
[bugfix] Ensure side effects for local -> local follows get processed (#2820) 2024-04-08 11:51:51 +02:00
kim 83e7847cdf
fix possible nil panic (#2809) 2024-04-04 16:45:59 +01:00
Daenney 236604bd6b
[chore] Log less output on failed test (#2804)
* [chore] Log less output on failed test

This changes the testrig log level to be error by default instead of
info. This makes test failures a lot easier to read, as we don't have
the parade of info logs for each failure to scroll through.

It speeds up the test suite by a couple of seconds since we need
to buffer and flush a lot less messages. On a clean run, so no test
failures, it's about a 3s difference on my machine. Depending on the
amount of test failures, total time saved can vary.

This also introduces a GTS_TESTRIG_LOG_LEVEL environment variable that
we explicitly check for, making it easy to override the log level should
we have a need for it. This would be primarily for running locally, and
not so much as part of go test.

Lastly, it updates the syslog tests to use log.Error because if the log
level is set to error but we call log.Info no message is emitted and we
hang indefinitely on the channel read.

* [chore] Rename the testrig log level env var
2024-04-04 16:42:42 +01:00
Daenney 8ed1b8142c
[bugfix] Sort follows chronologically (#2801)
The id on the follows table is not a ULID, but a random ID. Sorting on
them results in a completely random order. Instead, sort on created_at,
which sould result in a stable and intended sort order.

Fixes: #2769

Co-authored-by: kim <89579420+NyaaaWhatsUpDoc@users.noreply.github.com>
2024-04-03 14:06:39 +01:00
kim 15ede4c1ea
[bugfix] improved authenticate post inbox error handling (#2803)
* improved PostInboxScheme() error handling / logging in case of failed auth

* dumbass kim. returning err instead of errWithCode...

* add checks for the slightly changed error handling in tests, add notes in codebase about the odd way of working
2024-04-03 13:57:07 +01:00
tobi 65b5366031
[bugfix] Set domain for empty-domain Friendica accounts (#2800) 2024-04-02 13:40:54 +01:00
kim e664d0918b
[bugfix] Set the Host header within the signing transport (#2799) 2024-04-02 14:28:36 +02:00
kim d61d5c8a6a
[bugfix] httpclient not signing subsequent redirect requests (#2798)
* move http request signing to transport

* actually hook up the http roundtripper ...

* add code comments for the new gtscontext functions
2024-04-02 13:12:26 +02:00
tobi 4bbdef02f1
[chore] Try to parse public key as both Actor + bare key (#2710)
* [chore] Try to parse public key as both Actor + bare key

* fix weird test fail
2024-04-02 11:30:10 +01:00
kim adf345f1ec
[chore] bump go structr cache version -> v0.6.0 (#2773)
* update go-structr library -> v0.6.0, add necessary wrapping types + code changes to support these changes

* update readme with go-structr package changes

* improved wrapping of the SliceCache type

* add code comments for the cache wrapper types

* remove test.out 😇

---------

Co-authored-by: tobi <31960611+tsmethurst@users.noreply.github.com>
2024-04-02 12:03:40 +02:00
tobi f05874be30
[feature] Option to hide followers/following (#2788) 2024-04-02 10:42:24 +01:00
tobi 4f66dba139
[bugfix] Serve correct URI for AP following collection (#2787) 2024-03-28 12:10:16 +01:00
tobi 8953f57d88
[feature] User-selectable preset CSS themes for accounts (#2777)
* [feature] User-selectable preset themes

* docs, more theme stuff

* lint, tests

* fix css name

* correct some little issues

* add another theme

* fix poll background

* okay last theme i swear

* make retrieval of apimodel themes more conventional

* preallocate stylesheet slices
2024-03-25 17:32:24 +00:00
tobi b7b42e832a
[feature] Add healthcheck endpoints /livez and /readyz (#2783)
* [feature] Add healthcheck endpoints `/livez` and `/readyz`

* use select that returns no data
2024-03-25 17:05:14 +00:00
tobi 36f79e650c
[bugfix] Avoid empty public/local timeline queries (#2784) 2024-03-25 16:20:45 +00:00
tobi 7f4a0a1aeb
[chore] Move local account settings to separate db table (#2770)
* [chore] Move local account settings to separate database model

* don't use separate settings_id
2024-03-22 14:03:46 +01:00
kim 0767647056
[bugfix] add all possible busy result codes to the sqlite errBusy catching check (#2775) 2024-03-22 13:26:02 +01:00
tobi 0362d49da0
[bugfix] Parse links that contain non-ascii characters (#2762) 2024-03-15 17:26:53 +00:00
Vyr Cossont 4a56f31047
[chore] Swagger: add missing paging params to bookmarks list (#2759)
No code changes.
2024-03-15 11:43:24 +01:00
Blake Smith 15578835a8
[chore] Different error message for net/mail parsing on go 1.21.8 and above (#2760)
go 1.21.8 fixed some minor issues in net/mail that causes the test suite to fail
for some mail validation cases. Although we're not on go 1.21.8 yet, make the test
forward and backwards compatible.

See: 263c059b09
2024-03-15 11:36:43 +01:00
Vyr Cossont efbc8151db
[chore] Swagger: correct names and formats for several array params (#2758)
No code changes.
2024-03-15 11:31:58 +01:00
tobi 9b94b1faa2
[docs] Add database maintenance section; update info message on ANALYZE run (sqlite) (#2756)
* [chore] Limit size of ANALYZE run after migration (sqlite)

* add basic db maintenance tips

* update docs, analyze

* amend info message a wee bit

* update docs/admin/database_maintenance.md wording

Co-authored-by: Daenney <daenney@users.noreply.github.com>

---------

Co-authored-by: kim <89579420+NyaaaWhatsUpDoc@users.noreply.github.com>
Co-authored-by: Daenney <daenney@users.noreply.github.com>
2024-03-14 17:40:36 +00:00
tobi 1fe4cdaf46
[bugfix] Don't return 500 when searching for unpermitted status (#2753) 2024-03-14 17:57:54 +01:00
kim 459e75a9db
[bugfix] only check replyTo visibility during permission checks IF status is local (#2757) 2024-03-14 17:55:35 +01:00
tobi 92bf1f779b
[chore] Expose move endpoint again, small settings panel fixes (#2752) 2024-03-13 14:45:50 +01:00
tobi ab2d063fcb
[feature] Process outgoing Move from clientAPI (#2750)
* prevent moved accounts from taking create-type actions

* update move logic

* federate move out

* indicate on web profile when an account has moved

* [docs] Add migration docs section

* lock while checking + setting move state

* use redirectFollowers func for clientAPI as well

* comment typo

* linter? i barely know 'er!

* Update internal/uris/uri.go

Co-authored-by: Daenney <daenney@users.noreply.github.com>

* add a couple tests for move

* fix little mistake exposed by tests (thanks tests)

* ensure Move marked as successful

* attach shared util funcs to struct

* lock whole account when doing move

* move moving check to after error check

* replace repeated text with error func

* linterrrrrr!!!!

* catch self follow case

---------

Co-authored-by: Daenney <daenney@users.noreply.github.com>
2024-03-13 13:53:29 +01:00
Vyr Cossont 13b9fd5f92
[bugfix]: Add missing Link headers in Swagger spec (#2751)
* Adds Link headers to all endpoints that use it for paging
* Fixes the return type for listing what accounts someone follows
2024-03-13 12:23:26 +01:00
tobi 1bcdf1da3b
[feature] Process incoming Move activity (#2724)
* [feature] Process incoming account Move activity

* fix targetAcct typo

* put move origin account on fMsg

* shift more move functionality back to the worker fn

* simplify error logic
2024-03-12 14:34:08 +00:00
Daenney 5e871e81a8
[chore] Update usage of OTEL libraries (#2725)
* otel to 1.24
* prometheus exporter to 0.46
* bunotel to 1.1.17

Also:
* Use schemaless URL for metrics
* Add software version to tracing schema
2024-03-11 15:34:34 +01:00
tobi 4c155aa847
[bugfix] Fix whitespace move_id issue (#2742) 2024-03-10 13:27:31 +01:00
tobi de3c15a425
[bugfix] Don't error when populating MovedTo if account not found (#2741)
* [bugfix] Don't error when populating MovedTo if account not found

* test the thing
2024-03-10 11:59:55 +01:00
kim 016923b4dc
[bugfix] add workaround for Xsqlite_interrupt() permanently breaking connection (#2731) 2024-03-07 14:56:33 +01:00
Vyr Cossont 74e84cf8fa
[docs/chore] Swagger fixes for filters (#2730)
* Swagger: filtersV1Get should return an array

* Swagger: context must use multi format
2024-03-07 11:28:57 +01:00
Vyr Cossont 5159664a51
[chore] Fix a Swagger warning that only manifests during Go client code generation (#2729) 2024-03-06 18:44:58 +01:00
Vyr Cossont fc3741365c
[bugfix] Fix Swagger spec and add test script (#2698)
* Add Swagger spec test script

* Fix Swagger spec errors not related to statuses with polls

* Add API tests that post a status with a poll

* Fix creating a status with a poll from form params

* Fix Swagger spec errors related to statuses with polls (this is the last error)

* Fix Swagger spec warnings not related to unused definitions

* Suppress a duplicate list update params definition that was somehow causing wrong param names

* Add Swagger test to CI

- updates Drone config
- vendorizes go-swagger
- fixes a file extension issue that caused the test script to generate JSON instead of YAML with the vendorized version

* Put `Sample: ` on its own line everywhere

* Remove unused id param from emojiCategoriesGet

* Add 5 more pairs of profile fields to account update API Swagger

* Remove Swagger prefix from dummy fields

It makes the generated code look weird

* Manually annotate params for statusCreate operation

* Fix all remaining Swagger spec warnings

- Change some models into operation parameters
- Ignore models that already correspond to manually documented operation parameters but can't be trivially changed (those with file fields)

* Documented that creating a status with scheduled_at isn't implemented yet

* sign drone.yml

* Fix filter API Swagger errors

* fixup! Fix filter API Swagger errors

---------

Co-authored-by: tobi <tobi.smethurst@protonmail.com>
2024-03-06 18:05:45 +01:00
tobi 68c8fe67cc
[chore/bugfix] Little DB fixes (#2726) 2024-03-06 15:40:37 +01:00
tobi b22e213e15
[feature/chore] Add Move database functions + cache (#2647)
* [feature/chore] Add Move database functions + cache

* add move mem ratio to envparsing.sh

* update comment
2024-03-06 11:18:57 +01:00
Vyr Cossont 61a2b91f45
[feature] Filters v1 (#2594)
* Implement client-side v1 filters

* Exclude linter false positives

* Update test/envparsing.sh

* Fix minor Swagger, style, and Bun usage issues

* Regenerate Swagger

* De-generify filter keywords

* Remove updating filter statuses

This is an operation that the Mastodon v2 filter API doesn't actually have, because filter statuses, unlike keywords, don't have options: the only info they contain is the status ID to be filtered.

* Add a test for filter statuses specifically

* De-generify filter statuses

* Inline FilterEntry

* Use vertical style for Bun operations consistently

* Add comment on Filter DB interface

* Remove GoLand linter control comments

Our existing linters should catch these, or they don't matter very much

* Reduce memory ratio for filters
2024-03-06 11:15:58 +01:00
dependabot[bot] 7bc536d1f7
[chore]: Bump github.com/stretchr/testify from 1.8.4 to 1.9.0 (#2714)
* [chore]: Bump github.com/stretchr/testify from 1.8.4 to 1.9.0

Bumps [github.com/stretchr/testify](https://github.com/stretchr/testify) from 1.8.4 to 1.9.0.
- [Release notes](https://github.com/stretchr/testify/releases)
- [Commits](https://github.com/stretchr/testify/compare/v1.8.4...v1.9.0)

---
updated-dependencies:
- dependency-name: github.com/stretchr/testify
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* cast the exxpected test values as float32 to bypass (possible) testify suite regression

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: kim <grufwub@gmail.com>
2024-03-04 15:05:24 +00:00
kim d85727e184
[bugfix] check remote status permissibility (#2703)
* add more stringent checks for remote status permissibility

* add check for inreplyto of a remote status being a boost

* do not permit inReplyTo boost wrapper statuses

* change comment wording

* fix calls to NewFederator()

* add code comments for NotPermitted() and SetNotPermitted()

* improve comment

* check that existing != nil before attempting delete

* ensure replying account isn't suspended

* use a debug log instead of info. check for boost using ID

* shorten log string length. make info level

* add note that replying to boost wrapper status shouldn't be able to happen anyways

* update to use onFail() function
2024-03-04 12:30:12 +00:00
tobi f487fc5d4b
[bugfix] Sanitize incoming PropertyValue fields (#2722) 2024-03-04 11:46:59 +01:00
Milas Bowman 66d9297e64
[bugfix/tracing] fix broken tracing due to conflicting schema url (#2712)
The OpenTelemetry SDK is very strict about the schema version when
the `Resource` is initialized.

Specifically, different schema versions _CANNOT_ be mixed, and since
the default SDK resource (which is merged with the user-defined one)
defines a schema URL, the `semconv` imports are really prone to being
out-of-sync.

The best way to avoid this is to merge a _schemaless_ resource. This
is fine...there's plenty of other ways to get `semconv` out of sync,
and the core service attributes (e.g. `service.name`) should not ever
change.

Additionally, any errors here are now propagated so that they'll be
visible instead of silently swallowed.
2024-03-04 11:10:15 +01:00
kim ea0efb8094
[bugfix] update postgresqlstmt to correctly use postgres err hook (#2711) 2024-03-03 11:21:25 +01:00
kim fcecd0c952
[bugfix] unwrap boosts when checking in-reply-to status (#2702)
* add stronger checks on status being replied to

* update error code test is expecting
2024-02-29 15:20:57 +01:00
Vyr Cossont c2a691fd83
[chore] Disable the syslog long message over Unix datagram socket test on macOS (#2700) 2024-02-29 13:25:15 +01:00
tobi 504c4f227f
[chore] re-add statuses_account_id_id_idx if removed (#2699)
* [chore] re-add `statuses_account_id_id_idx` if removed

* if not exists
2024-02-28 12:53:46 +01:00
tobi f29d429699
[chore] add log line about lengthy reindex migration (#2695) 2024-02-27 18:46:48 +01:00
Vyr Cossont ad28b9f166
[bugfix] Account timeline: exclude self-replies that mention other accounts (#2670)
* Account timeline: exclude self-replies that mention other accounts

* Add index for querying unmentioned statuses

* remove now unused statuses_account_id_id_idx

---------

Co-authored-by: tobi <tobi.smethurst@protonmail.com>
2024-02-27 18:18:40 +01:00
9p4 9bf448be7a
[feature/oidc] Add support for very basic RBAC (#2642)
* Add support for very basic RBAC

* Add some small tests for allowedGroup and adminGroup

* Switch to table-driven tests
2024-02-27 16:07:29 +01:00