Commit graph

191 commits

Author SHA1 Message Date
kim d68c04a6c0
[performance] cache recently allowed/denied domains to cut down on db calls (#794)
* fetch creation and fetching domain blocks from db

Signed-off-by: kim <grufwub@gmail.com>

* add separate domainblock cache type, handle removing block from cache on delete

Signed-off-by: kim <grufwub@gmail.com>

* fix sentinel nil values being passed into cache

Signed-off-by: kim <grufwub@gmail.com>

Signed-off-by: kim <grufwub@gmail.com>
2022-09-02 12:17:46 +02:00
kim 614ab12733
[performance] use GetAccountByUsernameDomain() for local account lookups to rely on cache (#793)
Signed-off-by: kim <grufwub@gmail.com>

Signed-off-by: kim <grufwub@gmail.com>
2022-09-02 11:56:33 +02:00
Blackle Morisanchetto ecb97f4e0b
[feature] Add support for the exclude_types[] parameter on the notifications endpoint (#784)
* Add support for the exclude_types[] parameter on the notifications endpoint

* Add swagger docs to notifications
2022-08-31 19:20:52 +02:00
tobi 0245c606d7
[chore] Test fixes (#788)
* use 'test' value for testrig storage backend

* update test dependency

* add WaitFor func in testrig

* use WaitFor function instead of time.Sleep

* tidy up tests

* make SentMessages a sync.map

* go fmt
2022-08-31 17:31:21 +02:00
Blackle Morisanchetto c5c425b4e7
[bugfix] Fix tusky search issue by returning empty if offset is greater than zero (#786) 2022-08-31 09:57:50 +02:00
Blackle Morisanchetto 5d9c6b0e5a
[bugfix] Do not generate a notification when favouriting your own post (#780) 2022-08-30 11:43:29 +02:00
Blackle Morisanchetto 54f6caed65
[bugfix] Status visibility + excludeReplies fixes (#769)
* Fix some bugs when viewing a user's posts: include their self-replies (threads) even when excludeReplies is set, and use in_reply_to_uri instead of in_reply_to_id to filter out replies

* Assign values to InReplyToURI when creating statuses. Add index and update old statuses with a migration
2022-08-27 11:35:31 +02:00
tobi 2fe3a2b5b8
[bugfix] Fix boost of boost issue (#764) 2022-08-23 10:46:10 +02:00
tobi 570fa7c359
[bugfix] Fix potential dereference of accounts on own instance (#757)
* add GetAccountByUsernameDomain

* simplify search

* add escape to not deref accounts on own domain

* check if local + we have account by ap uri
2022-08-20 21:47:19 +01:00
tobi ac6ed3d939
[chore] Update bun / sqlite versions; update gtsmodels (#754)
* upstep bun and sqlite versions

* allow specific columns to be updated in the db

* only update necessary columns for user

* bit tidier

* only update necessary fields of media_attachment

* only update relevant instance fields

* update tests

* update only specific account columns

* use bool pointers on gtsmodels
includes attachment, status, account, user

* update columns more selectively

* test all default fields on new account insert

* updating remaining bools on gtsmodels

* initialize pointer fields when extracting AP emoji

* copy bools properly

* add copyBoolPtr convenience function + test it

* initialize false bool ptrs a bit more neatly
2022-08-15 11:35:05 +01:00
tobi 117888cf59
[feature] Add first iteration of a user panel at /user (#736)
* start work on user panel

* parse source first before checking if empty form

* newline

* set avi + header nicely

* add posts settings

* render signin a bit nicer on mobile

* return OK json on successful change

* return unauthorized on bad password

* clarify message on insecure password

* make login a bit prettier

* add alt text + border round image previews

* add logout button

* add password change

* styling updates

* redirect /auth/edit to /user

* update tests

* fix validation tests

* better labels, link to more info

* make submit button generic component

* move submit button inside forms

* add autocomplete labels to password fields

* fix indentation (thx eslint)

* update eslintrc

* eslint: no-unescaped-entities

* initial deduplication between user and admin panel

* add default status/post format setting

* user panel styling for inputs

* update user panel styling, include normalize css

* add placeholder text

* input padding

Co-authored-by: f0x <f0x@cthu.lu>
2022-08-08 10:40:51 +02:00
tobi f5689a9e5f
[feature] Let accounts set default status format, and use this when processing new statuses (#739)
* add post_format to acct & use it when making post

* update swagger docs

* add status_format updating to frontend

* fix up tests

* post_format => status_format

* add status_format to account validation
2022-08-06 12:09:21 +02:00
Artémis 4fdbef04b4
[feature] Implemented notification clear (#720)
* Implemented notification clear

* Added the cache clear mechanism

* added multi user check test
2022-08-01 11:13:49 +02:00
tobi c84384e660
[bugfix] html escape special characters in text instead of totally removing them (#719)
* remove minify dependency

* tidy up some tests

* remove pre + postformat funcs

* rework sanitization + formatting

* update tests

* add some more markdown tests
2022-07-19 15:21:17 +02:00
kim 098dbe6ff4
[chore] use our own logging implementation (#716)
* first commit

Signed-off-by: kim <grufwub@gmail.com>

* replace logging with our own log library

Signed-off-by: kim <grufwub@gmail.com>

* fix imports

Signed-off-by: kim <grufwub@gmail.com>

* fix log imports

Signed-off-by: kim <grufwub@gmail.com>

* add license text

Signed-off-by: kim <grufwub@gmail.com>

* fix package import cycle between config and log package

Signed-off-by: kim <grufwub@gmail.com>

* fix empty kv.Fields{} being passed to WithFields()

Signed-off-by: kim <grufwub@gmail.com>

* fix uses of log.WithFields() with whitespace issues and empty slices

Signed-off-by: kim <grufwub@gmail.com>

* *linter related grumbling*

Signed-off-by: kim <grufwub@gmail.com>

* gofmt the codebase! also fix more log.WithFields() formatting issues

Signed-off-by: kim <grufwub@gmail.com>

* update testrig code to match new changes

Signed-off-by: kim <grufwub@gmail.com>

* fix error wrapping in non fmt.Errorf function

Signed-off-by: kim <grufwub@gmail.com>

* add benchmarking of log.Caller() vs non-cached

Signed-off-by: kim <grufwub@gmail.com>

* fix syslog tests, add standard build tags to test runner to ensure consistency

Signed-off-by: kim <grufwub@gmail.com>

* make syslog tests more robust

Signed-off-by: kim <grufwub@gmail.com>

* fix caller depth arithmatic (is that how you spell it?)

Signed-off-by: kim <grufwub@gmail.com>

* update to use unkeyed fields in kv.Field{} instances

Signed-off-by: kim <grufwub@gmail.com>

* update go-kv library

Signed-off-by: kim <grufwub@gmail.com>

* update libraries list

Signed-off-by: kim <grufwub@gmail.com>

* fuck you linter get nerfed

Signed-off-by: kim <grufwub@gmail.com>

Co-authored-by: tobi <31960611+tsmethurst@users.noreply.github.com>
2022-07-19 10:47:55 +02:00
tobi 6418307c64
[feature] Add back/next buttons to profiles for paging through statuses (#708)
* add GetAccountWebStatuses to db

* add WebStatusesGet func to processor

* don't add limit to next/prev links if 0

* take query params for next/prev statuses

* add separate next + prev links for convenience

* show 'nothing here' message if no statuses exist

* add back / next links to profiles

* allow paging down only

* go fmt ./...

* 'recent public toots' -> 'latest public toots'
2022-07-13 09:57:47 +02:00
kim 7cc40302a5
[chore] consolidate caching libraries (#704)
* add miekg/dns dependency

* set/validate accountDomain

* move finger to dereferencer

* totally break GetRemoteAccount

* start reworking finger func a bit

* start reworking getRemoteAccount a bit

* move mention parts to namestring

* rework webfingerget

* use util function to extract webfinger parts

* use accountDomain

* rework finger again, final form

* just a real nasty commit, the worst

* remove refresh from account

* use new ASRepToAccount signature

* fix incorrect debug call

* fix for new getRemoteAccount

* rework GetRemoteAccount

* start updating tests to remove repetition

* break a lot of tests
Move shared test logic into the testrig,
rather than having it scattered all over
the place. This allows us to just mock
the transport controller once, and have
all tests use it (unless they need not to
for some other reason).

* fix up tests to use main mock httpclient

* webfinger only if necessary

* cheeky linting with the lads

* update mentionName regex
recognize instance accounts

* don't finger instance accounts

* test webfinger part extraction

* increase default worker count to 4 per cpu

* don't repeat regex parsing

* final search for discovered accountDomain

* be more permissive in namestring lookup

* add more extraction tests

* simplify GetParseMentionFunc

* skip long search if local account

* fix broken test

* consolidate to all use same caching libraries

Signed-off-by: kim <grufwub@gmail.com>

* perform more caching in the database layer

Signed-off-by: kim <grufwub@gmail.com>

* remove ASNote cache

Signed-off-by: kim <grufwub@gmail.com>

* update cache library, improve db tracing hooks

Signed-off-by: kim <grufwub@gmail.com>

* return ErrNoEntries if no account status IDs found, small formatting changes

Signed-off-by: kim <grufwub@gmail.com>

* fix tests, thanks tobi!

Signed-off-by: kim <grufwub@gmail.com>

Co-authored-by: tsmethurst <tobi.smethurst@protonmail.com>
2022-07-10 17:18:21 +02:00
Dominik Süß 9d0df426da
[feature] S3 support (#674)
* feat: vendor minio client

* feat: introduce storage package with s3 support

* feat: serve s3 files directly

this saves a lot of bandwith as the files are fetched from the object
store directly

* fix: use explicit local storage in tests

* feat: integrate s3 storage with the main server

* fix: add s3 config to cli tests

* docs: explicitly set values in example config

also adds license header to the storage package

* fix: use better http status code on s3 redirect

HTTP 302 Found is the best fit, as it signifies that the resource
requested was found but not under its presumed URL

307/TemporaryRedirect would mean that this resource is usually located
here, not in this case

303/SeeOther indicates that the redirection does not link to the
requested resource but to another page

* refactor: use context in storage driver interface
2022-07-03 12:08:30 +02:00
tobi 9e7d022a06
[feature] Cleanup unattached local media (#680)
* add localUnattached db function

* add parseOlderThan util function

* add pruneunusedlocalattachments to media manager

* add unusedlocal pruning to schedule + admin call

* set number of days to keep as a const

* fix test
2022-06-30 12:22:10 +02:00
ugla 3e4e57d554
[bugfix] disallow following or blocking yoursel (#667)
Closes #664
2022-06-25 11:14:05 +02:00
tobi 0846b76e93
[bugfix] Fix 404 on status delete redraft (#668)
* add unattach function to media processor

* call delete or unattach appropriately
unattach from client api, delete from federated api

* typo fix
2022-06-24 17:17:40 +02:00
tobi 7eacbd064b
[bugfix] allow setting empty email via instance patch (#665) 2022-06-24 10:43:21 +02:00
tobi 5f00d4980b
[feature] Implement /api/v1/instance/peers endpoint (#660)
* add missing license headers

* start adding instance peers get

* rename domainblock.go

* embed domain in domainblock so it can be reused

* update swagger docs

* add test instances to db

* update tests

* add/update instancepeersget

* update domain model

* add getinstancepeers to db

* instance-expose-peers, instance-expose-suspended

* add auth checks for both current filters

* attach endpoint to router

* include public comment

* obfuscate domain if required

* go mod tidy

* update swagger docs

* remove unnecessary comment

* return 'flat' peerlist if no query params provided
2022-06-23 16:54:54 +02:00
tobi 8c7945fb78
[bugfix] Account self finger fix (#658)
* check only 1 of status/account when search by uri

* don't try to resolve local uris when searching
2022-06-20 14:11:36 +02:00
tobi dfdc473cef
[chore] Webfinger rework (#627)
* move finger to dereferencer

* totally break GetRemoteAccount

* start reworking finger func a bit

* start reworking getRemoteAccount a bit

* move mention parts to namestring

* rework webfingerget

* use util function to extract webfinger parts

* use accountDomain

* rework finger again, final form

* just a real nasty commit, the worst

* remove refresh from account

* use new ASRepToAccount signature

* fix incorrect debug call

* fix for new getRemoteAccount

* rework GetRemoteAccount

* start updating tests to remove repetition

* break a lot of tests
Move shared test logic into the testrig,
rather than having it scattered all over
the place. This allows us to just mock
the transport controller once, and have
all tests use it (unless they need not to
for some other reason).

* fix up tests to use main mock httpclient

* webfinger only if necessary

* cheeky linting with the lads

* update mentionName regex
recognize instance accounts

* don't finger instance accounts

* test webfinger part extraction

* increase default worker count to 4 per cpu

* don't repeat regex parsing

* final search for discovered accountDomain

* be more permissive in namestring lookup

* add more extraction tests

* simplify GetParseMentionFunc

* skip long search if local account

* fix broken test
2022-06-11 11:01:34 +02:00
tobi 694a490589
[feature] Add created_at and error_description to /oauth/token endpoint (#645)
* start fiddling about with oauth server

* start returning more helpful errors from oauth

* test helpful(ish) token errors

* add missing license header
2022-06-11 10:39:39 +02:00
tobi 1ede54ddf6
[feature] More consistent API error handling (#637)
* update templates

* start reworking api error handling

* update template

* return AP status at web endpoint if negotiated

* start making api error handling much more consistent

* update account endpoints to new error handling

* use new api error handling in admin endpoints

* go fmt ./...

* use api error logic in app

* use generic error handling in auth

* don't export generic error handler

* don't defer clearing session

* user nicer error handling on oidc callback handler

* tidy up the sign in handler

* tidy up the token handler

* use nicer error handling in blocksget

* auth emojis endpoint

* fix up remaining api endpoints

* fix whoopsie during login flow

* regenerate swagger docs

* change http error logging to debug
2022-06-08 20:38:03 +02:00
tobi 6f6e89e271
[feature] Add paging via Link header for notifications and account statuses (#629)
* test link headers

* page get account statuses properly

* page get notifications

* add util func for packaging timeline responses

* return timelined stuff from accountstatusesget

* rename timeline response

* use new convenience function

* go fmt
2022-06-08 20:22:49 +02:00
tobi f3b44426f4
[bugfix] Update time marshalling format to provide 3 digits of ms (#630) 2022-06-04 20:03:01 +02:00
kim 43ac0cdb9c
[chore] Global server configuration overhaul (#575)
* move config flag names and usage to config package, rewrite config package to use global Configuration{} struct

Signed-off-by: kim <grufwub@gmail.com>

* improved code comment

Signed-off-by: kim <grufwub@gmail.com>

* linter

Signed-off-by: kim <grufwub@gmail.com>

* fix unmarshaling

Signed-off-by: kim <grufwub@gmail.com>

* remove kim's custom go compiler changes

Signed-off-by: kim <grufwub@gmail.com>

* generate setter and flag-name functions, implement these in codebase

Signed-off-by: kim <grufwub@gmail.com>

* update deps

Signed-off-by: kim <grufwub@gmail.com>

* small change

Signed-off-by: kim <grufwub@gmail.com>

* appease the linter...

Signed-off-by: kim <grufwub@gmail.com>

* move configuration into ConfigState structure, ensure reloading to/from viper settings to keep in sync

Signed-off-by: kim <grufwub@gmail.com>

* lint

Signed-off-by: kim <grufwub@gmail.com>

* update code comments

Signed-off-by: kim <grufwub@gmail.com>

* fix merge issue

Signed-off-by: kim <grufwub@gmail.com>

* fix merge issue

Signed-off-by: kim <grufwub@gmail.com>

* improved version string (removes time + go version)

Signed-off-by: kim <grufwub@gmail.com>

* fix version string build to pass test script + consolidate logic in func

Signed-off-by: kim <grufwub@gmail.com>

* add license text, update config.Defaults comment

Signed-off-by: kim <grufwub@gmail.com>

* add license text to generated config helpers file

Signed-off-by: kim <grufwub@gmail.com>

* defer unlock on config.Set___(), to ensure unlocked on panic

Signed-off-by: kim <grufwub@gmail.com>

* make it more obvious which cmd flags are being attached

Signed-off-by: kim <grufwub@gmail.com>
2022-05-30 14:41:24 +02:00
tobi ae5402ada6
[chore] Mastodon api fixups (#617)
* don't omitempty on description

* don't omitempty on any fields

* add ms to timestamp format

* don't omitempty on text_url

* rearrange attachment fields a bit

* just give URL again as attachment text url

* update tests

* fix accidental replace
2022-05-28 18:59:55 +01:00
tobi 73d5766572
[chore] Debug failed account get (#612) 2022-05-27 16:44:33 +02:00
tobi 5668ce1ec7
[bugfix] Fix HTML escaping in instance title (#607)
* move caption sanitization -> sanitize.go

* use sanitizeplaintext rather than removehtml

* rename sanitizecaption to sanitizeplaintext

* avoid removing html twice from statuses

* unexport remoteHTML
it's no longer used outside the text package so this
makes it less confusing

* test instance PATCH
2022-05-26 11:37:13 +02:00
tobi a54efa09f9
[chore] Serialize times as UTC ISO8601 instead of RFC3339 (#602)
* add time util to mimic utc ISO8601

* use ISO8601 when serializing to frontend

* update test notification
2022-05-24 17:21:27 +01:00
tobi 2d748a68ae
[bugfix] Only search remote if protocol is http(s) (#601) 2022-05-24 13:38:11 +02:00
tobi f0c9f4169b
[bugfix] Fix multiple dereferences of boosted status causing media duplication (#589)
* add some announces to test models

* start on announce test logic

* test federatingDB.Announce

* change signature of GetRemoteStatus

* remove 'refresh' logic and replace it with refetch

* go fmt

* remove timeline manager from processor test

* make zork created at determinate

* test get account statuses

* test get + serialize zork

* make account keys determinate

* make admin accountCreate time determinate

* test account to as

* init test config before test log

* test status to frontend

* remove daft Within check

* hack around a bit

* use index of slice
2022-05-23 16:40:03 +01:00
Adelie Paull caa0cde0e0
[feature] implement custom_emojis endpoint (#563)
* implement custom_emojis api endpoint

* add tests for getting custom emoji out of the database and converting to api emoji

* change sort direction of emoji query

* change logging level and initialize array with known length as per kim's suggestions

* add continue to lessen risk of making a malformed struct during conversion from db to api emojis
2022-05-20 10:34:36 +02:00
tobi b2810fedf2
[bugfix] Clean up boosts of status when the status itself is deleted (#579)
* move status wiping logic to fromcommon.go

* delete reblogs of status when a status is deleted

* add admin boost of zork to test model

* update tests to make them more determinate

* Merge branch 'main' into status_reblog_cleanup

* move status wiping logic to fromcommon.go

* delete reblogs of status when a status is deleted

* add admin boost of zork to test model

* update tests to make them more determinate

* Merge branch 'main' into status_reblog_cleanup

* test status delete via client api

* go fmt
2022-05-18 22:13:03 +01:00
tobi b143877995
[feature] Unused avatar and header cleanup (#574)
* rename + tidy up remote pruning

* fix media attachment account join
see https://bun.uptrace.dev/guide/golang-orm.html#table-relationships

* update logging to new function name

* add get avatars and headers to bun

* add pruneallmeta function

* don't set uncached since we're deleting anyway

* fix totalPruned being in wrong place

* test pruning meta

* go fmt ./...

* rename mediaprune

* add meta pruning to routine mediaprune

* tidy up cleanup job scheduling

* rename adminmediaremoteprune

* update mediacleanup to use renamed prune func

* update swagger docs a little bit

* reuse cancel + context
2022-05-15 15:45:04 +01:00
kim 223025fc27
[security] transport.Controller{} and transport.Transport{} security and performance improvements (#564)
* cache transports in controller by privkey-generated pubkey, add retry logic to transport requests

Signed-off-by: kim <grufwub@gmail.com>

* update code comments, defer mutex unlocks

Signed-off-by: kim <grufwub@gmail.com>

* add count to 'performing request' log message

Signed-off-by: kim <grufwub@gmail.com>

* reduce repeated conversions of same url.URL object

Signed-off-by: kim <grufwub@gmail.com>

* move worker.Worker to concurrency subpackage, add WorkQueue type, limit transport http client use by WorkQueue

Signed-off-by: kim <grufwub@gmail.com>

* fix security advisories regarding max outgoing conns, max rsp body size

- implemented by a new httpclient.Client{} that wraps an underlying
  client with a queue to limit connections, and limit reader wrapping
  a response body with a configured maximum size
- update pub.HttpClient args passed around to be this new httpclient.Client{}

Signed-off-by: kim <grufwub@gmail.com>

* add httpclient tests, move ip validation to separate package + change mechanism

Signed-off-by: kim <grufwub@gmail.com>

* fix merge conflicts

Signed-off-by: kim <grufwub@gmail.com>

* use singular mutex in transport rather than separate signer mus

Signed-off-by: kim <grufwub@gmail.com>

* improved useragent string

Signed-off-by: kim <grufwub@gmail.com>

* add note regarding missing test

Signed-off-by: kim <grufwub@gmail.com>

* remove useragent field from transport (instead store in controller)

Signed-off-by: kim <grufwub@gmail.com>

* shutup linter

Signed-off-by: kim <grufwub@gmail.com>

* reset other signing headers on each loop iteration

Signed-off-by: kim <grufwub@gmail.com>

* respect request ctx during retry-backoff sleep period

Signed-off-by: kim <grufwub@gmail.com>

* use external pkg with docs explaining performance "hack"

Signed-off-by: kim <grufwub@gmail.com>

* use http package constants instead of string method literals

Signed-off-by: kim <grufwub@gmail.com>

* add license file headers

Signed-off-by: kim <grufwub@gmail.com>

* update code comment to match new func names

Signed-off-by: kim <grufwub@gmail.com>

* updates to user-agent string

Signed-off-by: kim <grufwub@gmail.com>

* update signed testrig models to fit with new transport logic (instead uses separate signer now)

Signed-off-by: kim <grufwub@gmail.com>

* fuck you linter

Signed-off-by: kim <grufwub@gmail.com>
2022-05-15 11:16:43 +02:00
Eamonn O'Brien-Strain b24b71c0a4
[feature] Include password strength in error message when password strength is too low (#550)
* When password validation fails, return how close to enough entropy it has.

* Shorter version of low-strength password error message
2022-05-09 10:31:46 +02:00
kim 26b74aefaf
[bugfix] Fix existing bio text showing as HTML (#531)
* fix existing bio text showing as HTML

- updated replaced mentions to include instance
- strips HTML from account source note in Verify handler
- update text formatter to use buffers for string writes

Signed-off-by: kim <grufwub@gmail.com>

* go away linter

Signed-off-by: kim <grufwub@gmail.com>

* change buf reset location, change html mention tags

Signed-off-by: kim <grufwub@gmail.com>

* reduce FindLinks code complexity

Signed-off-by: kim <grufwub@gmail.com>

* fix HTML to text conversion

Signed-off-by: kim <grufwub@gmail.com>

* Update internal/regexes/regexes.go

Co-authored-by: Mina Galić <mina.galic@puppet.com>

* use improved html2text lib with more options

Signed-off-by: kim <grufwub@gmail.com>

* fix to produce actual plaintext from html

Signed-off-by: kim <grufwub@gmail.com>

* fix span tags instead written as space

Signed-off-by: kim <grufwub@gmail.com>

* performance improvements to regex replacements, fix link replace logic for un-html-ing in the future

Signed-off-by: kim <grufwub@gmail.com>

* fix tag/mention replacements to use input string, fix link replace to not include scheme

Signed-off-by: kim <grufwub@gmail.com>

* use matched input string for link replace href text

Signed-off-by: kim <grufwub@gmail.com>

* remove unused code (to appease linter :sobs:)

Signed-off-by: kim <grufwub@gmail.com>

* improve hashtagFinger regex to be more compliant

Signed-off-by: kim <grufwub@gmail.com>

* update breakReplacer to include both unix and windows line endings

Signed-off-by: kim <grufwub@gmail.com>

* add NoteRaw field to Account to store plaintext account bio, add migration for this, set for sensitive accounts

Signed-off-by: kim <grufwub@gmail.com>

* drop unnecessary code

Signed-off-by: kim <grufwub@gmail.com>

* update text package tests to fix logic changes

Signed-off-by: kim <grufwub@gmail.com>

* add raw note content testing to account update and account verify

Signed-off-by: kim <grufwub@gmail.com>

* remove unused modules

Signed-off-by: kim <grufwub@gmail.com>

* fix emoji regex

Signed-off-by: kim <grufwub@gmail.com>

* fix replacement of hashtags

Signed-off-by: kim <grufwub@gmail.com>

* update code comment

Signed-off-by: kim <grufwub@gmail.com>

Co-authored-by: Mina Galić <mina.galic@puppet.com>
2022-05-07 17:55:27 +02:00
tobi d7b46a4b63
[chore] Move initial MediaRemotePrune to cmd.server function (#536)
* move initial prune up to the server start function

* don't do initial prune when starting media manager

* add admin/mediaremoteprune in processor
2022-05-03 10:56:18 +01:00
Adelie Paull 9265a09a65
[bugfix] Allow self-boosting for any visibility but direct (#510)
* create visibility filter for boostability and allow self-boosting for any visbility but direct messages

* add a followers-only status to local_account_2

* fix typo in comment

* add license header, unwrap errors, be explicit about non-boostable visibility settings to avoid rogue boosting from miscoded clients, use ID compare for checking if self-boosting

* add tests for statusboostable filter

* fix tests that were affected by adding a new status to the test data

* fix the rest of tests affected by adding a status to the textrig data
2022-05-02 15:23:37 +02:00
tobi a5852fd7e4
[performance] Speed up some of the slower db queries (#523)
* remove unnecessary LOWER() db calls

* warn during slow db queries

* use bundb built-in exists function

* add db block test

* update account block query

* add domain block db test

* optimize domain block query

* fix implementing wrong test

* exclude most columns when checking block

* go fmt

* remote more unnecessary use of LOWER()
2022-05-02 11:53:46 +01:00
kim 420e2fb22b
replace async client API / federator msg processing with worker pools (#497)
* replace async client API / federator msg processing with worker pools
* appease our lord-and-saviour, the linter
2022-04-28 13:23:11 +01:00
tobi 9cf66bf298
[chore] Return more useful errors from auth failure (#494)
* try rsa_sha256 sig algo first

* return more informative errors from auth

* adapt to reworked auth function
2022-04-26 18:10:11 +02:00
tobi 0d05bf473b
[bug] Fix status API / status web API being case sensitive (#481)
* make getStatus by id case-insensitive

* test get status case insensitive

* init config before log
2022-04-24 14:24:43 +02:00
tobi af97d6bb7e
[bugfix] Use background ctx for domain block side effects (#457) 2022-04-16 12:56:41 +02:00
tobi 26683b3d49
[feature] Web profile pages for accounts (#449)
* add default avatars

* allow webModule to error

* return errWithCode from account get

* add AccountGetLocalByUsername

* check nil requesting account

* add timestampShort function for just month/year

* move loading logic to New + add default avatars

* add profile page view

* update swagger docs

* add excludeReblogs to GetAccountStatuses

* ignore casing when selecting local account by username

* appropriate redirects

* css fiddling

* add 'about' heading

* adjust thread page to work with routing

* return AP representation if requested + authorized

* simplify auth check

* go fmt

* golangci-lint ignore math/rand
2022-04-15 14:33:01 +02:00
tobi 03d7c75ebf
[chore] Update Go version to 1.18 (#444)
* linting with new golangci-lint version

* update go to 1.18

* bump versions in drone.yml

* use new runtime/debug package for version info

* remove Commit build flag from goreleaser

* remove mock commit + version from build script

* go fmt

* add dummy version env flag to test container

* install git in golang container for testing

* only set versionString if Version is defined
2022-04-02 15:40:09 +02:00
tobi 37d310f981
[feature] Dereference remote mentions when the account is not already known (#442)
* remove mention util function from db

* add ParseMentionFunc to gtsmodel

* add parseMentionFunc to processor

* refactor search to simplify it a bit

* add parseMentionFunc to account

* add parseMentionFunc to status

* some renaming for clarity

* test dereference of unknown mentioned account
2022-03-29 11:54:56 +02:00
tobi 73e9cca701
[bugfix] Close ReadClosers properly in the media package (#434)
* defer lock reader

* close readers when finished with them

* close the reader in the teereader when finished
2022-03-21 13:41:44 +01:00
tobi 55ad6dee71
[feature] Admin account actions (#432)
* add accountAction to the admin API

* model admin account action

* add admin account action to the processor

* add migration for new AdminAccountActions table

* fix accounts admin path

* Update swagger docs
2022-03-19 12:01:40 +01:00
tobi 532c4cc697
[feature] Federate local account deletion (#431)
* add account delete to API

* model account delete request

* add AccountDeleteLocal

* federate local account deletes

* add DeleteLocal

* update transport (controller) to allow shortcuts

* delete logic + testing

* update swagger docs

* more tests + fixes
2022-03-15 16:12:35 +01:00
tobi e306233166
[bugfix] Fix html-escaped characters in content warnings (#426)
* test status create with odd CWs

* use SanitizeCaption for content warning escaping
2022-03-13 16:37:45 +01:00
tobi 07727753b9
[feature] Clean up/uncache remote media (#407)
* Add whereNotEmptyAndNotNull

* Add GetRemoteOlderThanDays

* Add GetRemoteOlderThanDays

* Add PruneRemote to Manager interface

* Start implementing PruneRemote

* add new attachment + status to tests

* fix up and test GetRemoteOlderThan

* fix bad import

* PruneRemote: return number pruned

* add Cached column to mediaattachment

* update + test pruneRemote

* update mediaTest

* use Cached column

* upstep bun to latest version

* embed structs in mediaAttachment

* migrate mediaAttachment to new format

* don't default cached to true

* select only remote media

* update db dependencies

* step bun back to last working version

* update pruneRemote to use Cached field

* fix storage path of test attachments

* add recache logic to manager

* fix trimmed aspect ratio

* test prune and recache

* return errwithcode

* tidy up different paths for emoji vs attachment

* fix incorrect thumbnail type being stored

* expose TransportController to media processor

* implement tee-ing recached content

* add thoughts of dog to test fedi attachments

* test get remote files

* add comment on PruneRemote

* add postData cleanup to recache

* test thumbnail fetching

* add incredible diagram

* go mod tidy

* buffer pipes for recache streaming

* test for client stops reading after 1kb

* add media-remote-cache-days to config

* add cron package

* wrap logrus so it's available to cron

* start and stop cron jobs gracefully
2022-03-07 11:08:26 +01:00
tobi 55b83bea7c
[feature] Add postDataCallbackFunc to allow cleanup (#408) 2022-02-22 13:50:33 +01:00
tobi 23034ec145
[feature] Stream files via reader (#404)
* serve files via reader rather than byte slice

* close readcloser when we're done with it

* cast reader to readcloser
2022-02-19 11:44:56 +01:00
tsmethurst 3301148bb7 merge fixup 2022-02-08 12:17:53 +01:00
tsmethurst 4c294a596a Merge branch 'main' into media_refactor 2022-02-08 12:17:18 +01:00
Forest Johnson 6ed368cbeb
[feature] add authorization to the already-existing authentication (#365)
* add ensureUserIsAuthorizedOrRedirect to /oauth/authorize

* adding authorization (email confirm, account approve, etc) to TokenCheck

* revert un-needed changes to signin.go

* oops what happened here

* error css

* add account.SuspendedAt check

* remove redundant checks from oauth util Authed function

* wip tests

* tests passing

* stop stripping useful information from ErrAlreadyExists

* that feeling of scraping the dryer LINT off the screen

* oops I didn't mean to get rid of this NewTestRouter function

* make tests work with recorder

* re-add ConfigureTemplatesWithGin to handle template path err

Co-authored-by: tsmethurst <tobi.smethurst@protonmail.com>
2022-02-07 12:04:31 +01:00
tobi 85b4f96263
if accountDomain isn't set, use Host value instead (#380) 2022-02-06 14:07:38 +01:00
tobi 1b36e85840
[feature] Rework timeline code to make it useful for more than just statuses (#373)
* add preparable and timelineable interfaces

* initialize timeline manager within the processor

* generic renaming

* move status-specific timeline logic into the processor

* refactor timeline to make it useful for more than statuses
2022-02-05 12:47:38 +01:00
Forest Johnson 5be8a7a7ea
[bug] Send plaintext emails to fix "message refused: Message is not RFC 2822 compliant" (#366)
* trying to fix "message refused: Message is not RFC 2822 compliant"

* fix "message refused: Message is not RFC 2822 compliant"

550 5.7.1 Delivery not authorized, message refused: Message is not RFC
2822 compliant

* remove silly regex

* lint

* fix tests

* we should use text/template instead of html/template now
2022-01-31 11:46:20 +01:00
tsmethurst c156602c66 ensure blocking calls to getRemoteAccount before showing stuff to client 2022-01-25 13:48:13 +01:00
tsmethurst a6fb93ae2a ctx => innerctx 2022-01-24 18:12:28 +01:00
tsmethurst 667e7f112c update remote account get/deref logic 2022-01-24 13:12:17 +01:00
tsmethurst c157b1b20b rework data function to provide filesize 2022-01-23 14:41:58 +01:00
tsmethurst 589bb9df02 pass reader around instead of []byte 2022-01-16 18:52:55 +01:00
tsmethurst 6bf39d0fc1 emoji code passing muster 2022-01-15 17:36:15 +01:00
tsmethurst c4a533db72 start fixing up emoji processing code 2022-01-15 14:33:58 +01:00
tsmethurst 113f9d9ab4 pass a function into the manager, start work on emoji 2022-01-11 17:49:14 +01:00
tsmethurst e0f9323b9a test the media manager a bit, add shutdown logic 2022-01-10 18:36:09 +01:00
tsmethurst dccf21dd87 tests are passing, but there's still much to be done 2022-01-09 18:41:22 +01:00
tsmethurst f61c3ddcf7 compiling now 2022-01-08 17:17:01 +01:00
tsmethurst c2ff8f392b further refinements 2022-01-08 13:45:42 +01:00
tsmethurst c4d63d125b more refactoring, media handler => manager 2021-12-28 16:36:00 +01:00
tobi ef5a9256a8
Extend license notices to 2022 (#354) 2021-12-20 18:42:19 +01:00
tobi cb8688f429
Remove unnecessary storage config variables (#344)
* rewire config to not use extraneous serve vars

* rename 'file' to 'local' for consistency

* use Type and Size again
2021-12-20 15:19:53 +01:00
tobi 0884f89431
Implement Cobra CLI tooling, Viper config tooling (#336)
* start pulling out + replacing urfave and config

* replace many many instances of config

* move more stuff => viper

* properly remove urfave

* move some flags to root command

* add testrig commands to root

* alias config file keys

* start adding cli parsing tests

* reorder viper init

* remove config path alias

* fmt

* change config file keys to non-nested

* we're more or less in business now

* tidy up the common func

* go fmt

* get tests passing again

* add note about the cliparsing tests

* reorganize

* update docs with changes

* structure cmd dir better

* rename + move some files around

* fix dangling comma
2021-12-07 13:31:39 +01:00
tobi 3caae376e7
Fix streamed messages ending up in wrong timeline(s) (#325)
* define timeline consts

* remove double stream of status

* change test stream creation up a bit

* stream messages more selectively

* add test for streaming new status creation via clientAPI

* tidy code + comments a bit

* tidy up tests

* make sure new status isn't streamed to public
2021-11-22 19:03:21 +01:00
tobi 24f9e11221
Fix image description unnecessarily html-escaping innocent characters (#321)
* implement SanitizeCaption function

* tidy up text test setup
2021-11-22 11:49:11 +01:00
tobi f8630348b4
Enable stricter linting with golangci-lint (#316)
* update golangci-lint

* add golangci config file w/ more linters

* correct issues flagged by stricter linters

* add more generous timeout for golangci-lint

* add some style + formatting guidelines

* move timeout to config file

* go fmt
2021-11-22 08:46:19 +01:00
tobi 09ef9e639e
move to ssb gofed fork (#298) 2021-11-13 17:29:43 +01:00
tobi 829a934d23
update dependencies (#296) 2021-11-13 12:29:08 +01:00
tobi 2aaec82732
smtp + email confirmation (#285)
* add smtp configuration

* add email confirm + reset templates

* add email sender to testrig

* flesh out the email sender interface

* go fmt

* golint

* update from field with more clarity

* tidy up the email formatting

* fix tests

* add email sender to processor

* tidy client api processing a bit

* further tidying in fromClientAPI

* pin new account to user

* send msg to processor on new account creation

* generate confirm email uri

* remove emailer from account processor again

* add processCreateAccountFromClientAPI

* move emailer accountprocessor => userprocessor

* add email sender to user processor

* SendConfirmEmail function

* add noop email sender

* use noop email sender in tests

* only assemble message if callback is not nil

* use noop email sender if no smtp host is defined

* minify email html before sending

* fix wrong email address

* email confirm test

* fmt

* serve web hndler

* add email confirm handler

* init test log properly on testrig

* log emails that *would* have been sent

* go fmt ./...

* unexport confirm email handler

* updatedAt

* test confirm email function

* don't allow tokens older than 7 days

* change error message a bit

* add basic smtp docs

* add a few more snippets

* typo

* add email sender to outbox tests

* don't use dutch wikipedia link

* don't minify email html
2021-10-31 15:46:23 +01:00
tobi 4b1d9d3780
Serve outbox for Actor (#289)
* add statusesvisible convenience function

* add minID + onlyPublic to account statuses get

* move swagger collection stuff to common

* start working on Outbox GETting

* move functions into federationProcessor

* outboxToASCollection

* add statusesvisible convenience function

* add minID + onlyPublic to account statuses get

* move swagger collection stuff to common

* start working on Outbox GETting

* move functions into federationProcessor

* outboxToASCollection

* bit more work on outbox paging

* wrapNoteInCreate function

* test + hook up the processor functions

* don't do prev + next links on empty reply

* test get outbox through api

* don't fail on no status entries

* add outbox implementation doc

* typo
2021-10-24 11:57:39 +02:00
kim 7e4c3fa5c7
fix mention extracting when no domain exists (usually intra-instance mentions) (#272)
* fix mention extracting when no domain exists (usually when intra-instance mentions)

Signed-off-by: kim <grufwub@gmail.com>

* fix search logic to match new mention matching logic

Signed-off-by: kim <grufwub@gmail.com>

* appease the linter :p

Signed-off-by: kim <grufwub@gmail.com>
2021-10-17 14:19:49 +02:00
tobi 15621f5324
Follow request improvements (#282)
* tiny doc update

* add rejectfollowrequest to db

* add follow request reject to processor

* add reject handler

* tidy up follow request api

* tidy up federation call

* regenerate swagger docs

* api endpoint tests

* processor test

* add reject federatingdb handler

* start writing reject tests

* test reject follow request

* go fmt

* increase sleep for slow test setups

* more relaxed time.sleep
2021-10-16 13:27:43 +02:00
tobi 107685e22e
User password change (#280)
* start passwordChangeHandler

* add user scope

* add user module / api path

* add password change request

* make comment clearer

* add user to processor

* required true

* add processor call to handler

* don't pass tc or channel

* change password func + tests

* add some first docs about password management

* update swagger docs

* add api tests

* go fmt

* test fixes
2021-10-14 14:26:04 +02:00
R. Aidan Campbell 083099a957
reference global logrus (#274)
* reference logrus' global logger instead of passing and storing a logger reference everywhere

* always directly use global logrus logger instead of referencing an instance

* test suites should also directly use the global logrus logger

* rename gin logging function to clarify that it's middleware

* correct comments which erroneously referenced removed logger parameter

* setting log level for tests now uses logrus' exported type instead of the string value, to guarantee error isn't possible
2021-10-11 14:37:33 +02:00
tobi 367bdca250
Handle forwarded messages (#273)
* correct path of foss_satan

* add APIri and notes

* test create forward note

* rename target => receiving account

* split up create into separate funcs

* update extractFromCtx

* tidy up from federator processing

* foss satan => http not https

* check if status in db

* mock dereference of status from IRI

* add forward message deref test

* update test with activities

* add remote_account_2 to test rig
2021-10-10 12:39:25 +02:00
tobi e04b187702
Refactor/tidy (#261)
* tidy up streaming

* cut down code duplication

* test get followers/following

* test streaming processor

* fix some test models

* add TimeMustParse

* fix uri / url typo

* make trace logging less verbose

* make logging more consistent

* disable quote on logging

* remove context.Background

* remove many extraneous mastodon references

* regenerate swagger

* don't log query on no rows result

* log latency first for easier reading
2021-10-04 15:24:19 +02:00
tobi 9ce4234b9f
Follow request auto approval (#259)
* start messing about

* fiddle more

* Tests & fiddling
2021-10-01 19:08:50 +02:00
tobi 36a09dd0df
handle remote account deletion more systematically (#254) 2021-09-30 10:56:02 +02:00
tobi b5a7e1ba32
Account update issue (#250)
* start poking around

* tests

* notes and fiddling
2021-09-28 15:21:59 +02:00
tobi b3fd9c39a3
Weird notif issue (#248)
* start working on weird issue

* go fmt ./...

* more tests
2021-09-27 17:42:20 +02:00
tobi b315eee43b
Update webfingering a little, add tests (#236)
* Update webfingering a little, add tests

* fix broken tests oops
2021-09-20 16:46:45 +02:00
tobi 2e5dcc2929
Fix mentions not notifying (#230)
* set default privacy for new accounts

* teshts

* found it

* tiny change

* aaaa
2021-09-14 12:23:56 +02:00