Commit graph

1187 commits

Author SHA1 Message Date
tobi efd1a4f717
[bugfix] Use better plaintext representation of status for filtering (#3301)
* [bugfix] Use better plaintext representation of status for filtering

* add new deps to readme

* lint

* update tests

* update regexes

* address review comments

* remove now unused xxhash

* whoops, wrong logger

* Merge branch 'main' into status_filtering_bugfix

* put cache in caches struct

* pain
2024-09-16 14:00:23 +02:00
kim 6dd936fbe1
[bugfix/chore] Always set the status sensitive if media + content-warning present (#3308)
* always set the status sensitive flag to true if it has a content-warning with media

* whoops use attachment ids instead of mention ids ... 🤦
2024-09-16 13:38:12 +02:00
Vyr Cossont f3311d6273
[bugfix] Hoist filterable text field extraction out of loop (#3297)
This doesn't change within the loop, so there's no point in repeating it.
2024-09-15 10:42:04 +02:00
Daenney 4fa0d41202
[chore] Update go-sqlite3 to 0.18.3 (#3295)
* [chore] Update go-sqlite3 to 0.18.3

* [chore] Fix getting the sqlite3.Conn
2024-09-14 16:36:25 +02:00
tobi 20fe430ef9
[bugfix] Update home timeline query to ignore exclusive list entries (#3289)
* [bugfix] Update home timeline query to ignore exclusive list entries

* a
2024-09-11 10:55:25 +00:00
tobi 307d98e386
[feature] Process Reject of interaction via fedi API, put rejected statuses in the "sin bin" 😈 (#3271)
* [feature] Process `Reject` of interaction via fedi API, put rejected statuses in the "sin bin"

* update test

* move nil check back to `rejectStatusIRI`
2024-09-10 12:34:49 +00:00
kim 3254ef1923
[chore] status dereferencing improvements (#3255)
* search for mentions also by username,domain in status deref, handle deleted statuses in enrichStatusSafely()

* return d.enrichStatusSafely() directly
2024-09-10 12:33:32 +00:00
Vyr Cossont 540edef0c2
[feature] Implement exclusive lists (#3280)
Fixes #2616
2024-09-09 15:56:58 -07:00
tobi 5543fd5340
[feature/frontend] Add options to include Unlisted posts or hide all posts (#3272)
* [feature/frontend] Add options to include Unlisted posts or hide all posts

* finish up

* swagger

* move invalidate call into bundb package, avoid invalidating if not necessary

* rename show_web_statuses => web_visibility

* don't use ptr for webvisibility

* last bits
2024-09-09 18:07:25 +02:00
Vyr Cossont 100d660797
[bugfix] Swagger: add missing filter_action param for v2 filter PUT (#3281) 2024-09-09 09:52:49 +02:00
tobi b17010cf17
[bugfix] Fix temp table deletion causing runaway allocations (#3278)
* [bugfix] Fix temp table deletion causing runaway allocations

* move some vars around

* small fixes

* rely on conn max age to recycle temp tables

* fackin' ell m8
2024-09-08 16:14:56 +02:00
Daenney 0e2ee857ac
[chore] More AI blocking (#3273) 2024-09-07 17:21:36 +02:00
kim 6ee540a503
fix new processing media / emoji not being added to dereferencer maps (#3269) 2024-09-03 13:30:41 +00:00
tobi 7b7659f1fa
[chore/performance] Further reduce nil uncached queries (#3267)
* [chore/performance] Further reduce nil uncached queries

* more checks
2024-09-02 18:15:12 +02:00
tobi 0560c5ce89
[chore] Don't try to select zero uncached filters (#3266) 2024-09-02 17:13:54 +02:00
tobi 25a815a8a4
[chore/performance] Avoid unnecessary "uncached" queries (#3265)
* [chore/performance] Avoid unnecessary "uncached" queries

* go fmt
2024-09-02 14:00:17 +02:00
kim 0a1555521d
[performance] use single-threaded image transforms (#3252)
* use single-threaded image resizing in native code so we have more control over goroutines

* implement parallel-free versions of image transform functions also

* remove debug code
2024-08-31 10:41:38 +02:00
tobi 88a81fbcaf
[chore] Close copied request body in SignDelivery (#3254) 2024-08-30 17:00:55 +00:00
tobi cd93a5baf3
[security] Implement allowFiles fs for better isolation of ffmpeg / ffprobe (#3251)
* [chore] Implement readOneFile fs

* further isolation

* remove fmt call

* tweaks
2024-08-30 14:03:59 +02:00
tobi e10aa76612
[chore] Only call imaging.Resize when necessary, use even tinier blurhashes (#3247)
* [chore] Use `imaging.Fit`, use even tinier blurhashes

* avoid calling resize if not necessary

* update blurhashes + thumb
2024-08-29 17:43:14 +02:00
Daenney 277b043633
[chore] Update robots.txt with more AI scrapers (#3248) 2024-08-29 17:42:48 +02:00
tobi 8a34e4c28f
[bugfix] Fix incorrect json-ld @context serialization (#3243) 2024-08-26 19:17:45 +02:00
tobi db59ca36b1
[chore] Remove unused "env" module (#3235) 2024-08-25 15:37:20 +00:00
tobi 2bd31ab710
[chore] Tiny tweak to ApprovedByURI (#3234) 2024-08-25 15:44:08 +02:00
tobi d3887bf6cc
[bugfix] Let prometheus client do its own compression handling (#3232) 2024-08-25 13:25:32 +02:00
tobi bef0dfc66c
[bugfix] Carry-over ApprovedByURI to avoid marking already-approved remote statuses as pending approval (#3231)
* [bugfix] Carry-over pendingApproval + approvedByURI to avoid pending already-approved remote statuses

* don't carry over pending_approval, not necessary

* comment
2024-08-25 12:18:39 +02:00
tobi da3fa2d4a2
[chore/bugfix] Fix missing insertion of preapproved interaction requests (#3228) 2024-08-24 12:17:55 +00:00
tobi f23f04e0b1
[feature] Interaction requests client api + settings panel (#3215)
* [feature] Interaction requests client api + settings panel

* test accept / reject

* fmt

* don't pin rejected interaction

* use single db model for interaction accept, reject, and request

* swaggor

* env sharting

* append errors

* remove ErrNoEntries checks

* change intReqID to reqID

* rename "pend" to "request"

* markIntsPending -> mark interactionsPending

* use log instead of returning error when rejecting interaction

* empty migration

* jolly renaming

* make interactionURI unique again

* swag grr

* remove unnecessary locks

* invalidate as last step
2024-08-24 11:49:37 +02:00
kim 8e5a72ac5c
[performance] ffmpeg ffprobe wrapper improvements (#3225)
* use a single instance of wazero runtime and compiled modules

* remove test output 🤦

* undo process-{media,emoji} changes

* update test runner to include wazero compilation cache

* sign drone.yml

---------

Co-authored-by: tobi <tobi.smethurst@protonmail.com>
2024-08-23 17:15:35 +02:00
tobi 53fccb8af8
[feature] Use local_only field, deprecate federated field (#3222)
* [feature] Use `local_only` field, deprecate `federated` field

* use `deprecated` comment for form.Federated

* nolint
2024-08-22 19:47:10 +02:00
kim 889d4756ea
[performance] use native Go code to probe JPEGs (#3206)
* use native Go code to probe JPEGs

* add note about copying from github.com/disintegration/imaging

* add more code comments
2024-08-19 13:38:10 +00:00
kim e1154453bb
[chore] update default http client timeout to 30s (#3214) 2024-08-19 14:56:43 +02:00
kim ae14aa004d
[bugfix] permit unspecified orientation data (#3205) 2024-08-16 14:10:03 +02:00
kim 9cd27b412d
[security] harden account update logic (#3198)
* on account update, ensure that public key has not changed

* change expected error message

* also support the case of changing account keys when expired (not waiting for handshake)

* tweak account update hardening logic, add tests for updating account with pubkey expired

* add check for whether incoming data was via federator, accepting keys if so

* use freshest window for federated account updates + comment about it
2024-08-13 15:37:09 +00:00
kim 5212a1057e
[bugfix] relax missing preferred_username, instead using webfingered username (#3189)
* support no preferred_username, instead using webfingered username

* add tests for the new preferred_username behaviour
2024-08-13 09:01:50 +00:00
kim 4cb3e4d3e6
[bugfix] incorrect AP serialize function used serializing worker data (#3196) 2024-08-12 18:23:24 +02:00
kim 865b3aeaac
[bugfix] updated pinned counts on status delete (#3188)
* include pinned status when incrementing / decrementing status counts

* remove the pinned increment on status creation

* code comments

* microoptimize decr
2024-08-11 11:23:36 +02:00
kim f77005128a
[performance] move thumbnail generation to go code where possible (#3183)
* wrap thumbnailing code to handle generation natively where possible

* more code comments!

* add even more code comments!

* add code comments about blurhash generation

* maintain image rotation if contained in exif data

* move rotation before resizing

* ensure pix_fmt actually selected by ffprobe, check for alpha layer with gifs

* use linear instead of nearest-neighbour for resizing

* work with image "orientation" instead of "rotation". use default 75% quality for both webp and jpeg generation

* add header to new file

* use thumb extension when getting thumb mime type

* update test models and tests with new media processing

* add suggested code comments

* add note about thumbnail filter count reducing memory usage
2024-08-08 17:12:13 +00:00
John Winston 94c615d417
[feature] Add db-postgres-connection-string option (#3178)
* handle db-url

* lint and add doc

* add more doc

* fix config test

* return error

* change name from db-url to db-postgres-connection-string
2024-08-08 14:00:19 +02:00
tobi b19cfee7ae
[feature] Use gifv type for short soundless mp4 videos (#3182) 2024-08-08 08:12:16 +00:00
Leah Neukirchen 4697271cef
[bugfix] send back Sec-Websocket-Protocol header for streaming WebSocket (#3169)
* [bugfix] send back Sec-Websocket-Protocol header for streaming WebSocket

Chrome expects the selected Sec-Websocket-Protocol to be sent back
on the WebSocket upgrade request (RFC6455 1.9).

* fiddle a bit to avoid getting headers multiple times

* add some explanatory notes

---------

Co-authored-by: tobi <tobi.smethurst@protonmail.com>
2024-08-06 11:52:27 +02:00
kim b85a9983d0
[bugfix] fix emoji recaching operations (#3167)
* add test for emoji update image

* update emoji recache to set the instance account id

* don't refresh emoji if only not cached. in that case literally just recache

* code comment

* rename + move a few things

* add some more code comments, and rename some functions to make logic a bit clearer

* remove unnecessary nil check (the value can be nil)

* comment wording

* remove test data output

* handle the case of caching an emoji which has been refreshed then uncached

* allow overwriting on testrig storage as we do now on regular storage

* fix emoji category ID not getting updated

---------

Co-authored-by: tobi <tobi.smethurst@protonmail.com>
2024-08-03 19:05:38 +02:00
tobi fa59c3713c
[chore] Add media-ffmpeg-pool-size config var (#3164) 2024-08-03 16:40:26 +02:00
Daenney 9b50151f17
[feature] Beef up our AI opt-outs (#3165)
* [chore] Synchronise our robots.txt with upstream

* [feature] Add headers to escape AI crawlers

This adds 2 headers that a number of AI crawlers respect to signal that
content should not be included in their datasets.
2024-08-02 18:22:39 +02:00
kim e5e996b28a
[bugfix] close files before error return (#3163)
* close files before error return

* use defer statements

* shuffle around some defers
2024-08-02 15:11:24 +01:00
kim 0f734a2410
[chore] move PopulateAccountStats() nil check often performed into function itself (#3158)
* move PopulateAccountStats() nil check often performed into function itself

* fix test to take in mind we don't repopulate account stats if not-nil
2024-08-02 13:15:11 +01:00
kim 94e87610c4
[chore] add back exif-terminator and use only for jpeg,png,webp (#3161)
* add back exif-terminator and use only for jpeg,png,webp

* fix arguments passed to terminateExif()

* pull in latest exif-terminator

* fix test

* update processed img

---------

Co-authored-by: tobi <tobi.smethurst@protonmail.com>
2024-08-02 12:46:41 +01:00
tobi 7b5917d6ae
[feature] Allow import of following and blocks via CSV (#3150)
* [feature] Import follows + blocks via settings panel

* test import follows
2024-08-02 12:41:46 +01:00
tobi 697261da53
[chore] Take account of rotation data when calculating full size image dimensions (#3159)
* [chore] Take account of rotation data when calculating full size image dimensions

* boobies
2024-07-31 18:43:39 +00:00
Vyr Cossont fd837776e2
[feature] Implement Mastodon-compatible roles (#3136)
* Implement Mastodon-compatible roles

- `Account.role` should only be available through verify_credentials for checking current user's permissions
- `Account.role` now carries a Mastodon-compatible permissions bitmap and a marker for whether it should be shown to the public
- `Account.roles` added for *public* display roles (undocumented but stable since Mastodon 4.1)
- Web template now uses only public display roles (no user-visible change here, we already special-cased the `user` role)

* Handle verify_credentials case for default role

* Update JSON exact-match tests

* Address review comments

* Add blocks bit to admin permissions bitmap
2024-07-31 09:26:09 -07:00