* move caption sanitization -> sanitize.go
* use sanitizeplaintext rather than removehtml
* rename sanitizecaption to sanitizeplaintext
* avoid removing html twice from statuses
* unexport remoteHTML
it's no longer used outside the text package so this
makes it less confusing
* test instance PATCH
* add some announces to test models
* start on announce test logic
* test federatingDB.Announce
* change signature of GetRemoteStatus
* remove 'refresh' logic and replace it with refetch
* go fmt
* remove timeline manager from processor test
* make zork created at determinate
* test get account statuses
* test get + serialize zork
* make account keys determinate
* make admin accountCreate time determinate
* test account to as
* init test config before test log
* test status to frontend
* remove daft Within check
* hack around a bit
* use index of slice
* implement custom_emojis api endpoint
* add tests for getting custom emoji out of the database and converting to api emoji
* change sort direction of emoji query
* change logging level and initialize array with known length as per kim's suggestions
* add continue to lessen risk of making a malformed struct during conversion from db to api emojis
* move status wiping logic to fromcommon.go
* delete reblogs of status when a status is deleted
* add admin boost of zork to test model
* update tests to make them more determinate
* Merge branch 'main' into status_reblog_cleanup
* move status wiping logic to fromcommon.go
* delete reblogs of status when a status is deleted
* add admin boost of zork to test model
* update tests to make them more determinate
* Merge branch 'main' into status_reblog_cleanup
* test status delete via client api
* go fmt
* rename + tidy up remote pruning
* fix media attachment account join
see https://bun.uptrace.dev/guide/golang-orm.html#table-relationships
* update logging to new function name
* add get avatars and headers to bun
* add pruneallmeta function
* don't set uncached since we're deleting anyway
* fix totalPruned being in wrong place
* test pruning meta
* go fmt ./...
* rename mediaprune
* add meta pruning to routine mediaprune
* tidy up cleanup job scheduling
* rename adminmediaremoteprune
* update mediacleanup to use renamed prune func
* update swagger docs a little bit
* reuse cancel + context
* cache transports in controller by privkey-generated pubkey, add retry logic to transport requests
Signed-off-by: kim <grufwub@gmail.com>
* update code comments, defer mutex unlocks
Signed-off-by: kim <grufwub@gmail.com>
* add count to 'performing request' log message
Signed-off-by: kim <grufwub@gmail.com>
* reduce repeated conversions of same url.URL object
Signed-off-by: kim <grufwub@gmail.com>
* move worker.Worker to concurrency subpackage, add WorkQueue type, limit transport http client use by WorkQueue
Signed-off-by: kim <grufwub@gmail.com>
* fix security advisories regarding max outgoing conns, max rsp body size
- implemented by a new httpclient.Client{} that wraps an underlying
client with a queue to limit connections, and limit reader wrapping
a response body with a configured maximum size
- update pub.HttpClient args passed around to be this new httpclient.Client{}
Signed-off-by: kim <grufwub@gmail.com>
* add httpclient tests, move ip validation to separate package + change mechanism
Signed-off-by: kim <grufwub@gmail.com>
* fix merge conflicts
Signed-off-by: kim <grufwub@gmail.com>
* use singular mutex in transport rather than separate signer mus
Signed-off-by: kim <grufwub@gmail.com>
* improved useragent string
Signed-off-by: kim <grufwub@gmail.com>
* add note regarding missing test
Signed-off-by: kim <grufwub@gmail.com>
* remove useragent field from transport (instead store in controller)
Signed-off-by: kim <grufwub@gmail.com>
* shutup linter
Signed-off-by: kim <grufwub@gmail.com>
* reset other signing headers on each loop iteration
Signed-off-by: kim <grufwub@gmail.com>
* respect request ctx during retry-backoff sleep period
Signed-off-by: kim <grufwub@gmail.com>
* use external pkg with docs explaining performance "hack"
Signed-off-by: kim <grufwub@gmail.com>
* use http package constants instead of string method literals
Signed-off-by: kim <grufwub@gmail.com>
* add license file headers
Signed-off-by: kim <grufwub@gmail.com>
* update code comment to match new func names
Signed-off-by: kim <grufwub@gmail.com>
* updates to user-agent string
Signed-off-by: kim <grufwub@gmail.com>
* update signed testrig models to fit with new transport logic (instead uses separate signer now)
Signed-off-by: kim <grufwub@gmail.com>
* fuck you linter
Signed-off-by: kim <grufwub@gmail.com>
* fix existing bio text showing as HTML
- updated replaced mentions to include instance
- strips HTML from account source note in Verify handler
- update text formatter to use buffers for string writes
Signed-off-by: kim <grufwub@gmail.com>
* go away linter
Signed-off-by: kim <grufwub@gmail.com>
* change buf reset location, change html mention tags
Signed-off-by: kim <grufwub@gmail.com>
* reduce FindLinks code complexity
Signed-off-by: kim <grufwub@gmail.com>
* fix HTML to text conversion
Signed-off-by: kim <grufwub@gmail.com>
* Update internal/regexes/regexes.go
Co-authored-by: Mina Galić <mina.galic@puppet.com>
* use improved html2text lib with more options
Signed-off-by: kim <grufwub@gmail.com>
* fix to produce actual plaintext from html
Signed-off-by: kim <grufwub@gmail.com>
* fix span tags instead written as space
Signed-off-by: kim <grufwub@gmail.com>
* performance improvements to regex replacements, fix link replace logic for un-html-ing in the future
Signed-off-by: kim <grufwub@gmail.com>
* fix tag/mention replacements to use input string, fix link replace to not include scheme
Signed-off-by: kim <grufwub@gmail.com>
* use matched input string for link replace href text
Signed-off-by: kim <grufwub@gmail.com>
* remove unused code (to appease linter :sobs:)
Signed-off-by: kim <grufwub@gmail.com>
* improve hashtagFinger regex to be more compliant
Signed-off-by: kim <grufwub@gmail.com>
* update breakReplacer to include both unix and windows line endings
Signed-off-by: kim <grufwub@gmail.com>
* add NoteRaw field to Account to store plaintext account bio, add migration for this, set for sensitive accounts
Signed-off-by: kim <grufwub@gmail.com>
* drop unnecessary code
Signed-off-by: kim <grufwub@gmail.com>
* update text package tests to fix logic changes
Signed-off-by: kim <grufwub@gmail.com>
* add raw note content testing to account update and account verify
Signed-off-by: kim <grufwub@gmail.com>
* remove unused modules
Signed-off-by: kim <grufwub@gmail.com>
* fix emoji regex
Signed-off-by: kim <grufwub@gmail.com>
* fix replacement of hashtags
Signed-off-by: kim <grufwub@gmail.com>
* update code comment
Signed-off-by: kim <grufwub@gmail.com>
Co-authored-by: Mina Galić <mina.galic@puppet.com>
* create visibility filter for boostability and allow self-boosting for any visbility but direct messages
* add a followers-only status to local_account_2
* fix typo in comment
* add license header, unwrap errors, be explicit about non-boostable visibility settings to avoid rogue boosting from miscoded clients, use ID compare for checking if self-boosting
* add tests for statusboostable filter
* fix tests that were affected by adding a new status to the test data
* fix the rest of tests affected by adding a status to the textrig data
* remove unnecessary LOWER() db calls
* warn during slow db queries
* use bundb built-in exists function
* add db block test
* update account block query
* add domain block db test
* optimize domain block query
* fix implementing wrong test
* exclude most columns when checking block
* go fmt
* remote more unnecessary use of LOWER()
* linting with new golangci-lint version
* update go to 1.18
* bump versions in drone.yml
* use new runtime/debug package for version info
* remove Commit build flag from goreleaser
* remove mock commit + version from build script
* go fmt
* add dummy version env flag to test container
* install git in golang container for testing
* only set versionString if Version is defined
* remove mention util function from db
* add ParseMentionFunc to gtsmodel
* add parseMentionFunc to processor
* refactor search to simplify it a bit
* add parseMentionFunc to account
* add parseMentionFunc to status
* some renaming for clarity
* test dereference of unknown mentioned account
* add accountAction to the admin API
* model admin account action
* add admin account action to the processor
* add migration for new AdminAccountActions table
* fix accounts admin path
* Update swagger docs
* Add whereNotEmptyAndNotNull
* Add GetRemoteOlderThanDays
* Add GetRemoteOlderThanDays
* Add PruneRemote to Manager interface
* Start implementing PruneRemote
* add new attachment + status to tests
* fix up and test GetRemoteOlderThan
* fix bad import
* PruneRemote: return number pruned
* add Cached column to mediaattachment
* update + test pruneRemote
* update mediaTest
* use Cached column
* upstep bun to latest version
* embed structs in mediaAttachment
* migrate mediaAttachment to new format
* don't default cached to true
* select only remote media
* update db dependencies
* step bun back to last working version
* update pruneRemote to use Cached field
* fix storage path of test attachments
* add recache logic to manager
* fix trimmed aspect ratio
* test prune and recache
* return errwithcode
* tidy up different paths for emoji vs attachment
* fix incorrect thumbnail type being stored
* expose TransportController to media processor
* implement tee-ing recached content
* add thoughts of dog to test fedi attachments
* test get remote files
* add comment on PruneRemote
* add postData cleanup to recache
* test thumbnail fetching
* add incredible diagram
* go mod tidy
* buffer pipes for recache streaming
* test for client stops reading after 1kb
* add media-remote-cache-days to config
* add cron package
* wrap logrus so it's available to cron
* start and stop cron jobs gracefully
* add ensureUserIsAuthorizedOrRedirect to /oauth/authorize
* adding authorization (email confirm, account approve, etc) to TokenCheck
* revert un-needed changes to signin.go
* oops what happened here
* error css
* add account.SuspendedAt check
* remove redundant checks from oauth util Authed function
* wip tests
* tests passing
* stop stripping useful information from ErrAlreadyExists
* that feeling of scraping the dryer LINT off the screen
* oops I didn't mean to get rid of this NewTestRouter function
* make tests work with recorder
* re-add ConfigureTemplatesWithGin to handle template path err
Co-authored-by: tsmethurst <tobi.smethurst@protonmail.com>
* add preparable and timelineable interfaces
* initialize timeline manager within the processor
* generic renaming
* move status-specific timeline logic into the processor
* refactor timeline to make it useful for more than statuses
* trying to fix "message refused: Message is not RFC 2822 compliant"
* fix "message refused: Message is not RFC 2822 compliant"
550 5.7.1 Delivery not authorized, message refused: Message is not RFC
2822 compliant
* remove silly regex
* lint
* fix tests
* we should use text/template instead of html/template now
* start pulling out + replacing urfave and config
* replace many many instances of config
* move more stuff => viper
* properly remove urfave
* move some flags to root command
* add testrig commands to root
* alias config file keys
* start adding cli parsing tests
* reorder viper init
* remove config path alias
* fmt
* change config file keys to non-nested
* we're more or less in business now
* tidy up the common func
* go fmt
* get tests passing again
* add note about the cliparsing tests
* reorganize
* update docs with changes
* structure cmd dir better
* rename + move some files around
* fix dangling comma
* define timeline consts
* remove double stream of status
* change test stream creation up a bit
* stream messages more selectively
* add test for streaming new status creation via clientAPI
* tidy code + comments a bit
* tidy up tests
* make sure new status isn't streamed to public
