* use minID properly for public timeline
* return paged response properly even when 0 items
* use gtserror
* page more consistently (for now)
* test
* aaa
* update typeconverter to use state structure
* deinterface the typeutils.TypeConverter -> typeutils.Converter
* finish copying over old type converter code comments
* fix cherry-pick merge issues, fix tests pointing to old typeutils interface type still
* Allow full BCP 47 in language inputs
Fixes#2066
* Fuse validation and normalization for languages
* Remove outdated comment line
* Move post language canonicalization test
* Support setting private notes on accounts
* Reformat comment whitespace
* Add missing license headers
* Use apiutil.ParseID
* Rename Note model and cache to AccountNote
* Update golden cache config in test/envparsing.sh
* Rename gtsmodel/note.go to gtsmodel/accountnote.go
* Update AccountNote uniqueness constraint name
Now has same prefix as other indexes on this table.
---------
Co-authored-by: tobi <31960611+tsmethurst@users.noreply.github.com>
* [bugfix] Tidy up rss feed serving; don't error on empty feed
* fall back to account creation time as rss feed update time
* return feed early when account has no eligible statuses
* media manager tidy-up: de-interface and remove unused PostDataFunc
Signed-off-by: kim <grufwub@gmail.com>
* remove last traces of media.Manager being an interface
Signed-off-by: kim <grufwub@gmail.com>
* update error to provide caller, allow tuneable via build tags
Signed-off-by: kim <grufwub@gmail.com>
* remove kim-specific build script changes
Signed-off-by: kim <grufwub@gmail.com>
* fix merge conflicts
Signed-off-by: kim <grufwub@gmail.com>
* update build-script to support externally setting build variables
Signed-off-by: kim <grufwub@gmail.com>
---------
Signed-off-by: kim <grufwub@gmail.com>
* start working on lists
* further list work
* test list db functions nicely
* more work on lists
* peepoopeepoo
* poke
* start list timeline func
* we're getting there lads
* couldn't be me working on stuff... could it?
* hook up handlers
* fiddling
* weeee
* woah
* screaming, pissing
* fix streaming being a whiny baby
* lint, small test fix, swagger
* tidying up, testing
* fucked! by the linter
* move timelines to state like a boss
* add timeline start to tests using state
* invalidate lists
* revamp http client to not limit requests, instead use sender worker
Signed-off-by: kim <grufwub@gmail.com>
* remove separate sender worker pool, spawn 2*GOMAXPROCS batch senders each time, no need for transport cache sweeping
Signed-off-by: kim <grufwub@gmail.com>
* improve batch senders to keep popping recipients until remote URL found
Signed-off-by: kim <grufwub@gmail.com>
* fix recipient looping issue
Signed-off-by: kim <grufwub@gmail.com>
* move request id ctx key to gtscontext, finish filling out more code comments, add basic support for not logging client IP
Signed-off-by: kim <grufwub@gmail.com>
* first draft of status refetching logic
Signed-off-by: kim <grufwub@gmail.com>
* fix testrig to use new federation alloc func signature
Signed-off-by: kim <grufwub@gmail.com>
* fix log format directive
Signed-off-by: kim <grufwub@gmail.com>
* add status fetched_at migration
Signed-off-by: kim <grufwub@gmail.com>
* remove unused / unchecked for error types
Signed-off-by: kim <grufwub@gmail.com>
* add back the used type...
Signed-off-by: kim <grufwub@gmail.com>
* add separate internal getStatus() function for derefThread() that doesn't recurse
Signed-off-by: kim <grufwub@gmail.com>
* improved mention and media attachment error handling
Signed-off-by: kim <grufwub@gmail.com>
* fix log and error format directives
Signed-off-by: kim <grufwub@gmail.com>
* update account deref to match status deref changes
Signed-off-by: kim <grufwub@gmail.com>
* very small code formatting change to make things clearer
Signed-off-by: kim <grufwub@gmail.com>
* add more code comments
Signed-off-by: kim <grufwub@gmail.com>
* improved code commenting
Signed-off-by: kim <grufwub@gmail.com>
* only check for required further derefs if needed
Signed-off-by: kim <grufwub@gmail.com>
* improved cache invalidation
Signed-off-by: kim <grufwub@gmail.com>
* tweak cache restarting to use a (very small) backoff
Signed-off-by: kim <grufwub@gmail.com>
* small readability changes and fixes
Signed-off-by: kim <grufwub@gmail.com>
* fix account sync issues
Signed-off-by: kim <grufwub@gmail.com>
* fix merge conflicts + update account enrichment to accept already-passed accountable
Signed-off-by: kim <grufwub@gmail.com>
* remove secondary function declaration
Signed-off-by: kim <grufwub@gmail.com>
* normalise dereferencer get status / account behaviour, fix remaining tests
Signed-off-by: kim <grufwub@gmail.com>
* fix remaining rebase conflicts, finish commenting code
Signed-off-by: kim <grufwub@gmail.com>
* appease the linter
Signed-off-by: kim <grufwub@gmail.com>
* add source file header
Signed-off-by: kim <grufwub@gmail.com>
* update to use TIMESTAMPTZ column type instead of just TIMESTAMP
Signed-off-by: kim <grufwub@gmail.com>
* don't pass in 'updated_at' to UpdateEmoji()
Signed-off-by: kim <grufwub@gmail.com>
* use new ap.Resolve{Account,Status}able() functions
Signed-off-by: kim <grufwub@gmail.com>
* remove the somewhat confusing rescoping of the same variable names
Signed-off-by: kim <grufwub@gmail.com>
* update migration file name, improved database delete error returns
Signed-off-by: kim <grufwub@gmail.com>
* formatting
Signed-off-by: kim <grufwub@gmail.com>
* improved multi-delete database functions to minimise DB calls
Signed-off-by: kim <grufwub@gmail.com>
* remove unused type
Signed-off-by: kim <grufwub@gmail.com>
* fix delete statements
Signed-off-by: kim <grufwub@gmail.com>
---------
Signed-off-by: kim <grufwub@gmail.com>
* revamp http client to not limit requests, instead use sender worker
Signed-off-by: kim <grufwub@gmail.com>
* remove separate sender worker pool, spawn 2*GOMAXPROCS batch senders each time, no need for transport cache sweeping
Signed-off-by: kim <grufwub@gmail.com>
* improve batch senders to keep popping recipients until remote URL found
Signed-off-by: kim <grufwub@gmail.com>
* fix recipient looping issue
Signed-off-by: kim <grufwub@gmail.com>
* fix missing mutex unlock
Signed-off-by: kim <grufwub@gmail.com>
* move request id ctx key to gtscontext, finish filling out more code comments, add basic support for not logging client IP
Signed-off-by: kim <grufwub@gmail.com>
* slight code reformatting
Signed-off-by: kim <grufwub@gmail.com>
* a whitespace
Signed-off-by: kim <grufwub@gmail.com>
* remove unused code
Signed-off-by: kim <grufwub@gmail.com>
* add missing license headers
Signed-off-by: kim <grufwub@gmail.com>
* fix request backoff calculation
Signed-off-by: kim <grufwub@gmail.com>
---------
Signed-off-by: kim <grufwub@gmail.com>
* start working on notifs for new posts
* tidy up a bit
* update swagger
* carry over show reblogs + notify from follow req
* test notify on status post
* update column slice
* dedupe update logic + add tests
* fix own boosts not being timelined
* avoid type check, passing unnecessary accounts
* remove unnecessary 'inReplyToID' check
* add a couple todo's for future db functions
* refactor visibility checking, add caching for visibility
* invalidate visibility cache items on account / status deletes
* fix requester ID passed to visibility cache nil ptr
* de-interface caches, fix home / public timeline caching + visibility
* finish adding code comments for visibility filter
* fix angry goconst linter warnings
* actually finish adding filter visibility code comments for timeline functions
* move home timeline status author check to after visibility
* remove now-unused code
* add more code comments
* add TODO code comment, update printed cache start names
* update printed cache names on stop
* start adding separate follow(request) delete db functions, add specific visibility cache tests
* add relationship type caching
* fix getting local account follows / followed-bys, other small codebase improvements
* simplify invalidation using cache hooks, add more GetAccountBy___() functions
* fix boosting to return 404 if not boostable but no error (to not leak status ID)
* remove dead code
* improved placement of cache invalidation
* update license headers
* add example follow, follow-request config entries
* add example visibility cache configuration to config file
* use specific PutFollowRequest() instead of just Put()
* add tests for all GetAccountBy()
* add GetBlockBy() tests
* update block to check primitive fields
* update and finish adding Get{Account,Block,Follow,FollowRequest}By() tests
* fix copy-pasted code
* update envparsing test
* whitespace
* fix bun struct tag
* add license header to gtscontext
* fix old license header
* improved error creation to not use fmt.Errorf() when not needed
* fix various rebase conflicts, fix account test
* remove commented-out code, fix-up mention caching
* fix mention select bun statement
* ensure mention target account populated, pass in context to customrenderer logging
* remove more uncommented code, fix typeutil test
* add statusfave database model caching
* add status fave cache configuration
* add status fave cache example config
* woops, catch missed error. nice catch linter!
* add back testrig panic on nil db
* update example configuration to match defaults, slight tweak to cache configuration defaults
* update envparsing test with new defaults
* fetch followingget to use the follow target account
* use accounnt.IsLocal() instead of empty domain check
* use constants for the cache visibility type check
* use bun.In() for notification type restriction in db query
* include replies when fetching PublicTimeline() (to account for single-author threads in Visibility{}.StatusPublicTimelineable())
* use bun query building for nested select statements to ensure working with postgres
* update public timeline future status checks to match visibility filter
* same as previous, for home timeline
* update public timeline tests to dynamically check for appropriate statuses
* migrate accounts to allow unique constraint on public_key
* provide minimal account with publicKey
---------
Signed-off-by: kim <grufwub@gmail.com>
Co-authored-by: tsmethurst <tobi.smethurst@protonmail.com>
* start refactoring account deletion
* update to use state.DB
* further messing about
* some more tidying up
* more tidying, cleaning, nice-making
* further adventures in refactoring and the woes of technical debt
* update fr accept/reject
* poking + prodding
* fix up deleting
* create fave uri
* don't log using requestingAccount.ID because it might be nil
* move getBookmarks function
* use exists query to check for status bookmark
* use deletenotifications func
* fiddle
* delete follow request notif
* split up some db functions
* Fix possible nil pointer panic
* fix more possible nil pointers
* fix license headers
* warn when follow missing (target) account
* return wrapped err when bookmark/fave models can't be retrieved
* simplify self account delete
* warn log likely race condition
* de-sillify status delete loop
* move error check due north
* warn when unfollowSideEffects has no target account
* warn when no boost account is found
* warn + dump follow when no account
* more warnings
* warn on fave account not set
* move for loop inside anonymous function
* fix funky logic
* don't remove mutual account items on block;
do make sure unfollow occurs in both directions!
* [chore] Remove years from all license headers
Years or year ranges aren't required in license headers. Many projects
have removed them in recent years and it avoids a bit of yearly toil.
In many cases our copyright claim was also a bit dodgy since we added
the 2021-2023 header to files created after 2021 but you can't claim
copyright into the past that way.
* [chore] Add license header check
This ensures a license header is always added to any new file. This
avoids maintainers/reviewers needing to remember to check for and ask
for it in case a contribution doesn't include it.
* [chore] Add missing license headers
* [chore] Further updates to license header
* Use the more common // indentend comment format
* Remove the hack we had for the linter now that we use the // format
* Add SPDX license identifier
* Add go-playground/form pkg
* [feature] Add support for profile fields
* Add field attributes test
* Validate profile fields form
* Add profile field validation tests
* Add Field Attributes definition to swagger
---------
Co-authored-by: tobi <31960611+tsmethurst@users.noreply.github.com>
* Advertise rich text formats, support content_type field
* Update JSON in instance patch tests
* Replace format with content_type everywhere
* update migration to work with both pg and sqlite
* regenerate swagger docs
* update instance serialization + tests
* fix up
* learn to code tobi please, i'm begging you
---------
Co-authored-by: tsmethurst <tobi.smethurst@protonmail.com>
* implement status pin client api + web handler
* make test names + comments more descriptive
* don't use separate table for status pins
* remove unused add + remove checking
* tidy up + add some more tests
* Tidy up emoji parsing on profile submission
Don't bother reparsing for emoji unless one of the fields that can have
emoji in it has changed.
Deduplicate emoji between the display name and profile note - I'm not
sure whether this was hurting anything, but better safe.
* Deduplicate emoji when parsing remote accounts
Some servers - Misskey at least - don't deduplicate emoji, so it's
possible to get an account which has the same emoji used in both the
display name and note and therefore includes that emoji twice in its
metadata. When we start trying to put those into our database, we run
into a uniqueness constraint and fall over.
This change just deduplicates at the point of construction of an
account.
This adds a lightweight form of tracing to GTS. Each incoming request is
assigned a Request ID which we then pass on and log in all our log
lines. Any function that gets called downstream from an HTTP handler
should now emit a requestID=value pair whenever it logs something.
Co-authored-by: kim <grufwub@gmail.com>
* replace media workers with just runners.WorkerPool, move to state structure, use go-sched for global task scheduling
* improved code comment
* fix worker tryUntil function, update go-runners/go-sched
* make preprocess functions package public, use these where possible to stop doubled up processing
* remove separate emoji worker pool
* limit calls to time.Now() during media preprocessing
* use Processor{} to manage singular runtime of processing media
* ensure workers get started when media manager is used
* improved error setting in processing media, fix media test
* port changes from processingmedia to processing emoji
* finish code commenting
* finish code commenting and comment-out client API + federator worker pools until concurrency worker pools replaced
* linterrrrrrrrrrrrrrrr
---------
Signed-off-by: kim <grufwub@gmail.com>
* reformat GetAccount() functionality, and add UpdateAccount() function.
* use fetched_at instead of last_webfingered_at
* catch local "not found" errors. small formatting / error string changes
* remove now unused error type
* return nil when wrapping nil error
* update expected error messages
* return correct url for foss satan webfinger
* add AP model for Some_User
* normalize local domain
* return notretrievable where appropriate
* expose NewErrNotRetrievable
* ensure webfinger for new accounts searched by uri
* update local account short circuit
* allow enrich to fail for already-known accounts
* remove unused LastWebfingeredAt
* expose test maps on mock http client
* update Update test
* reformat GetAccount() functionality, and add UpdateAccount() function.
* use fetched_at instead of last_webfingered_at
* catch local "not found" errors. small formatting / error string changes
* remove nil error checks (we shouldn't be passing nil errors to newError() initializers)
* remove mutex unlock on transport init fail (it hasn't yet been locked!)
* woops add back the error wrapping to use ErrNotRetrievable
* caches were never being started... 🙈
---------
Signed-off-by: kim <grufwub@gmail.com>
Co-authored-by: tsmethurst <tobi.smethurst@protonmail.com>
* Implement goldmark debug print for hashtags and mentions
* Minify HTML in FromPlain
* Convert plaintext status parser to goldmark
* Move mention/tag/emoji finding logic into formatter
* Combine mention and hashtag boundary characters
* Normalize unicode when rendering hashtags
* media processor consolidation and reformatting, reduce amount of required syscalls
Signed-off-by: kim <grufwub@gmail.com>
* update go-store library, stream jpeg/png encoding + use buffer pools, improved media processing AlreadyExists error handling
Signed-off-by: kim <grufwub@gmail.com>
* fix duration not being set, fix mp4 test expecting error
Signed-off-by: kim <grufwub@gmail.com>
* fix test expecting media files with different extension
Signed-off-by: kim <grufwub@gmail.com>
* remove unused code
Signed-off-by: kim <grufwub@gmail.com>
* fix expected storage paths in tests, update expected test thumbnails
Signed-off-by: kim <grufwub@gmail.com>
* remove dead code
Signed-off-by: kim <grufwub@gmail.com>
* fix cached presigned s3 url fetching
Signed-off-by: kim <grufwub@gmail.com>
* fix tests
Signed-off-by: kim <grufwub@gmail.com>
* fix test models
Signed-off-by: kim <grufwub@gmail.com>
* update media processing to use sync.Once{} for concurrency protection
Signed-off-by: kim <grufwub@gmail.com>
* shutup linter
Signed-off-by: kim <grufwub@gmail.com>
* fix passing in KVStore GetStream() as stream to PutStream()
Signed-off-by: kim <grufwub@gmail.com>
* fix unlocks of storage keys
Signed-off-by: kim <grufwub@gmail.com>
* whoops, return the error...
Signed-off-by: kim <grufwub@gmail.com>
* pour one out for tobi's code <3
Signed-off-by: kim <grufwub@gmail.com>
* add back the byte slurping code
Signed-off-by: kim <grufwub@gmail.com>
* check for both ErrUnexpectedEOF and EOF
Signed-off-by: kim <grufwub@gmail.com>
* add back links to file format header information
Signed-off-by: kim <grufwub@gmail.com>
Signed-off-by: kim <grufwub@gmail.com>
* Implement Bookmarks
* Update based on review comments
* Update swagger doc
* Fix argument passing to status.Bookmark
* Update changed test
* Updates based on latest PR review
* [feature] overhaul the oidc system
this allows for more flexible username handling and prevents account
takeover using old email addresses
* [feature] add migration path for old OIDC users
* [feature] nicer error reporting for users
* [docs] document the new OIDC flow
* [fix] return early on oidc error
* [docs]: add comments on the finalization logic
* start refactoring some of the search + deref logic
* add tests for search api
* rename GetRemoteAccount + GetRemoteStatus
* make search function a bit simpler + clearer
* fix little fucky wucky uwu owo i'm just a little guy
* update faulty switch statements
* update test to use storage struct
* redo switches for clarity
* reduce repeated logic in search tests
* fastfail getstatus by uri
* debug log + trace log better
* add implementation note
* return early if no result for namestring search
* return + check on dereferencing error types
* errors hah what errors
* remove unneeded error type alias, add custom error text during stringification itself
* fix a woops recursion 🙈
Signed-off-by: kim <grufwub@gmail.com>
Co-authored-by: kim <grufwub@gmail.com>
* [feature/performance] Fail fast when doing remote transport calls inside incoming request contexts
* [chore] Reduce outgoing request timeout to 15s
* log error messages when fastfailing
* use context.Value() instead of wrapped context, wrap error with fastfail instead of extra log entry
* add fast-fail context key test
Signed-off-by: kim <grufwub@gmail.com>
Co-authored-by: kim <grufwub@gmail.com>
* add account block DB cache and remove reliance on relational joins
* actually include cache key arguments...
* add a PutBlock() method which also updates the block cache, update tests accordingly
* use `PutBlock` instead of `Put(ctx, block)`
* add + use functions for deleting + invalidating blocks
Signed-off-by: kim <grufwub@gmail.com>
Co-authored-by: tsmethurst <tobi.smethurst@protonmail.com>
* refactor federator account statuses delete to better catch errors, ensure next maxID is always set
Signed-off-by: kim <grufwub@gmail.com>
* fix error statement missing 2nd format operator
Signed-off-by: kim <grufwub@gmail.com>
Signed-off-by: kim <grufwub@gmail.com>
* convert most of the caches to use result.Cache{}
* add caching of emojis
* fix issues causing failing tests
* update go-cache/v2 instances with v3
* fix getnotification
* add a note about the left-in StatusCreate comment
* update EmojiCategory db access to use new result.Cache{}
* fix possible panic in getstatusparents
* further proof that kim is not stinky
* [feature] Make instance thumbnail configurable via admin panel
* log db errors in InstanceToAPIInstance
* only update instance in db if necessary
* start adding tests
* finish test
* use readcloser for content.Content
* call media postdata function no matter what
* return a readcloser from data func
* tidy of logic of readertostore
* fix whoopsie
* start adding rss functionality
* add gorilla/feeds dependency
* first bash at building rss feed
still needs work, this is an interim commit
* tidy up a bit
* add publicOnly option to GetAccountLastPosted
* implement rss endpoint
* fix test
* add initial user docs for rss
* update rss logo
* docs update
* add rssFeed to frontend
* feed -> feed.rss
* enableRSS
* increase rss logo size a lil bit
* add rss toggle
* move emojify to text package
* fiddle with rss feed formatting
* add Text field to test statuses
* move status to rss item to typeconverter
* update bun schema for enablerss
* simplify 304 checking
* assume account not rss
* update tests
* update swagger docs
* allow more characters in title, trim nicer
* update last posted to be more consistent
* Start adding account emoji
* get emojis serialized + deserialized nicely
* update tests
* set / retrieve emojis on accounts
* show account emojis in web view
* fetch emojis from db based on ids
* fix typo in test
* lint
* fix pg migration
* update tests
* update emoji checking logic
* update comment
* clarify comments + add some spacing
* tidy up loops a lil (thanks kim)
* add func for deleting status from db + cache
* move deletes entirely back to processor
and also only do a delete if the requesting account owns the item being deleted
* tidy up unboost processing
* delete status more efficiently
* fix wrong account id on remote test attachments
* fix federator test
* add custom css account property + db func to fetch
* allow account to get/set custom css
* serve custom css for an account
* go fmt
* use monospace for customcss, add link
* add custom css to account cache
* fix broken field
* add custom css docs to user guide
* add `accounts-allow-custom-css` config flag
* add allow custom css to /api/v1/instance response
* only show/set custom css if allowed to do so
* only set/serve custom account css if enabled
* update swagger docs
* chain promise
* make bool a bit clearer
* use cache for GetAccountCustomCSSByUsername
* upstep bun and sqlite versions
* allow specific columns to be updated in the db
* only update necessary columns for user
* bit tidier
* only update necessary fields of media_attachment
* only update relevant instance fields
* update tests
* update only specific account columns
* use bool pointers on gtsmodels
includes attachment, status, account, user
* update columns more selectively
* test all default fields on new account insert
* updating remaining bools on gtsmodels
* initialize pointer fields when extracting AP emoji
* copy bools properly
* add copyBoolPtr convenience function + test it
* initialize false bool ptrs a bit more neatly
* add post_format to acct & use it when making post
* update swagger docs
* add status_format updating to frontend
* fix up tests
* post_format => status_format
* add status_format to account validation
* first commit
Signed-off-by: kim <grufwub@gmail.com>
* replace logging with our own log library
Signed-off-by: kim <grufwub@gmail.com>
* fix imports
Signed-off-by: kim <grufwub@gmail.com>
* fix log imports
Signed-off-by: kim <grufwub@gmail.com>
* add license text
Signed-off-by: kim <grufwub@gmail.com>
* fix package import cycle between config and log package
Signed-off-by: kim <grufwub@gmail.com>
* fix empty kv.Fields{} being passed to WithFields()
Signed-off-by: kim <grufwub@gmail.com>
* fix uses of log.WithFields() with whitespace issues and empty slices
Signed-off-by: kim <grufwub@gmail.com>
* *linter related grumbling*
Signed-off-by: kim <grufwub@gmail.com>
* gofmt the codebase! also fix more log.WithFields() formatting issues
Signed-off-by: kim <grufwub@gmail.com>
* update testrig code to match new changes
Signed-off-by: kim <grufwub@gmail.com>
* fix error wrapping in non fmt.Errorf function
Signed-off-by: kim <grufwub@gmail.com>
* add benchmarking of log.Caller() vs non-cached
Signed-off-by: kim <grufwub@gmail.com>
* fix syslog tests, add standard build tags to test runner to ensure consistency
Signed-off-by: kim <grufwub@gmail.com>
* make syslog tests more robust
Signed-off-by: kim <grufwub@gmail.com>
* fix caller depth arithmatic (is that how you spell it?)
Signed-off-by: kim <grufwub@gmail.com>
* update to use unkeyed fields in kv.Field{} instances
Signed-off-by: kim <grufwub@gmail.com>
* update go-kv library
Signed-off-by: kim <grufwub@gmail.com>
* update libraries list
Signed-off-by: kim <grufwub@gmail.com>
* fuck you linter get nerfed
Signed-off-by: kim <grufwub@gmail.com>
Co-authored-by: tobi <31960611+tsmethurst@users.noreply.github.com>
* add GetAccountWebStatuses to db
* add WebStatusesGet func to processor
* don't add limit to next/prev links if 0
* take query params for next/prev statuses
* add separate next + prev links for convenience
* show 'nothing here' message if no statuses exist
* add back / next links to profiles
* allow paging down only
* go fmt ./...
* 'recent public toots' -> 'latest public toots'
* add miekg/dns dependency
* set/validate accountDomain
* move finger to dereferencer
* totally break GetRemoteAccount
* start reworking finger func a bit
* start reworking getRemoteAccount a bit
* move mention parts to namestring
* rework webfingerget
* use util function to extract webfinger parts
* use accountDomain
* rework finger again, final form
* just a real nasty commit, the worst
* remove refresh from account
* use new ASRepToAccount signature
* fix incorrect debug call
* fix for new getRemoteAccount
* rework GetRemoteAccount
* start updating tests to remove repetition
* break a lot of tests
Move shared test logic into the testrig,
rather than having it scattered all over
the place. This allows us to just mock
the transport controller once, and have
all tests use it (unless they need not to
for some other reason).
* fix up tests to use main mock httpclient
* webfinger only if necessary
* cheeky linting with the lads
* update mentionName regex
recognize instance accounts
* don't finger instance accounts
* test webfinger part extraction
* increase default worker count to 4 per cpu
* don't repeat regex parsing
* final search for discovered accountDomain
* be more permissive in namestring lookup
* add more extraction tests
* simplify GetParseMentionFunc
* skip long search if local account
* fix broken test
* consolidate to all use same caching libraries
Signed-off-by: kim <grufwub@gmail.com>
* perform more caching in the database layer
Signed-off-by: kim <grufwub@gmail.com>
* remove ASNote cache
Signed-off-by: kim <grufwub@gmail.com>
* update cache library, improve db tracing hooks
Signed-off-by: kim <grufwub@gmail.com>
* return ErrNoEntries if no account status IDs found, small formatting changes
Signed-off-by: kim <grufwub@gmail.com>
* fix tests, thanks tobi!
Signed-off-by: kim <grufwub@gmail.com>
Co-authored-by: tsmethurst <tobi.smethurst@protonmail.com>
* feat: vendor minio client
* feat: introduce storage package with s3 support
* feat: serve s3 files directly
this saves a lot of bandwith as the files are fetched from the object
store directly
* fix: use explicit local storage in tests
* feat: integrate s3 storage with the main server
* fix: add s3 config to cli tests
* docs: explicitly set values in example config
also adds license header to the storage package
* fix: use better http status code on s3 redirect
HTTP 302 Found is the best fit, as it signifies that the resource
requested was found but not under its presumed URL
307/TemporaryRedirect would mean that this resource is usually located
here, not in this case
303/SeeOther indicates that the redirection does not link to the
requested resource but to another page
* refactor: use context in storage driver interface
* move finger to dereferencer
* totally break GetRemoteAccount
* start reworking finger func a bit
* start reworking getRemoteAccount a bit
* move mention parts to namestring
* rework webfingerget
* use util function to extract webfinger parts
* use accountDomain
* rework finger again, final form
* just a real nasty commit, the worst
* remove refresh from account
* use new ASRepToAccount signature
* fix incorrect debug call
* fix for new getRemoteAccount
* rework GetRemoteAccount
* start updating tests to remove repetition
* break a lot of tests
Move shared test logic into the testrig,
rather than having it scattered all over
the place. This allows us to just mock
the transport controller once, and have
all tests use it (unless they need not to
for some other reason).
* fix up tests to use main mock httpclient
* webfinger only if necessary
* cheeky linting with the lads
* update mentionName regex
recognize instance accounts
* don't finger instance accounts
* test webfinger part extraction
* increase default worker count to 4 per cpu
* don't repeat regex parsing
* final search for discovered accountDomain
* be more permissive in namestring lookup
* add more extraction tests
* simplify GetParseMentionFunc
* skip long search if local account
* fix broken test
* update templates
* start reworking api error handling
* update template
* return AP status at web endpoint if negotiated
* start making api error handling much more consistent
* update account endpoints to new error handling
* use new api error handling in admin endpoints
* go fmt ./...
* use api error logic in app
* use generic error handling in auth
* don't export generic error handler
* don't defer clearing session
* user nicer error handling on oidc callback handler
* tidy up the sign in handler
* tidy up the token handler
* use nicer error handling in blocksget
* auth emojis endpoint
* fix up remaining api endpoints
* fix whoopsie during login flow
* regenerate swagger docs
* change http error logging to debug
* test link headers
* page get account statuses properly
* page get notifications
* add util func for packaging timeline responses
* return timelined stuff from accountstatusesget
* rename timeline response
* use new convenience function
* go fmt
* move config flag names and usage to config package, rewrite config package to use global Configuration{} struct
Signed-off-by: kim <grufwub@gmail.com>
* improved code comment
Signed-off-by: kim <grufwub@gmail.com>
* linter
Signed-off-by: kim <grufwub@gmail.com>
* fix unmarshaling
Signed-off-by: kim <grufwub@gmail.com>
* remove kim's custom go compiler changes
Signed-off-by: kim <grufwub@gmail.com>
* generate setter and flag-name functions, implement these in codebase
Signed-off-by: kim <grufwub@gmail.com>
* update deps
Signed-off-by: kim <grufwub@gmail.com>
* small change
Signed-off-by: kim <grufwub@gmail.com>
* appease the linter...
Signed-off-by: kim <grufwub@gmail.com>
* move configuration into ConfigState structure, ensure reloading to/from viper settings to keep in sync
Signed-off-by: kim <grufwub@gmail.com>
* lint
Signed-off-by: kim <grufwub@gmail.com>
* update code comments
Signed-off-by: kim <grufwub@gmail.com>
* fix merge issue
Signed-off-by: kim <grufwub@gmail.com>
* fix merge issue
Signed-off-by: kim <grufwub@gmail.com>
* improved version string (removes time + go version)
Signed-off-by: kim <grufwub@gmail.com>
* fix version string build to pass test script + consolidate logic in func
Signed-off-by: kim <grufwub@gmail.com>
* add license text, update config.Defaults comment
Signed-off-by: kim <grufwub@gmail.com>
* add license text to generated config helpers file
Signed-off-by: kim <grufwub@gmail.com>
* defer unlock on config.Set___(), to ensure unlocked on panic
Signed-off-by: kim <grufwub@gmail.com>
* make it more obvious which cmd flags are being attached
Signed-off-by: kim <grufwub@gmail.com>
* move caption sanitization -> sanitize.go
* use sanitizeplaintext rather than removehtml
* rename sanitizecaption to sanitizeplaintext
* avoid removing html twice from statuses
* unexport remoteHTML
it's no longer used outside the text package so this
makes it less confusing
* test instance PATCH
* cache transports in controller by privkey-generated pubkey, add retry logic to transport requests
Signed-off-by: kim <grufwub@gmail.com>
* update code comments, defer mutex unlocks
Signed-off-by: kim <grufwub@gmail.com>
* add count to 'performing request' log message
Signed-off-by: kim <grufwub@gmail.com>
* reduce repeated conversions of same url.URL object
Signed-off-by: kim <grufwub@gmail.com>
* move worker.Worker to concurrency subpackage, add WorkQueue type, limit transport http client use by WorkQueue
Signed-off-by: kim <grufwub@gmail.com>
* fix security advisories regarding max outgoing conns, max rsp body size
- implemented by a new httpclient.Client{} that wraps an underlying
client with a queue to limit connections, and limit reader wrapping
a response body with a configured maximum size
- update pub.HttpClient args passed around to be this new httpclient.Client{}
Signed-off-by: kim <grufwub@gmail.com>
* add httpclient tests, move ip validation to separate package + change mechanism
Signed-off-by: kim <grufwub@gmail.com>
* fix merge conflicts
Signed-off-by: kim <grufwub@gmail.com>
* use singular mutex in transport rather than separate signer mus
Signed-off-by: kim <grufwub@gmail.com>
* improved useragent string
Signed-off-by: kim <grufwub@gmail.com>
* add note regarding missing test
Signed-off-by: kim <grufwub@gmail.com>
* remove useragent field from transport (instead store in controller)
Signed-off-by: kim <grufwub@gmail.com>
* shutup linter
Signed-off-by: kim <grufwub@gmail.com>
* reset other signing headers on each loop iteration
Signed-off-by: kim <grufwub@gmail.com>
* respect request ctx during retry-backoff sleep period
Signed-off-by: kim <grufwub@gmail.com>
* use external pkg with docs explaining performance "hack"
Signed-off-by: kim <grufwub@gmail.com>
* use http package constants instead of string method literals
Signed-off-by: kim <grufwub@gmail.com>
* add license file headers
Signed-off-by: kim <grufwub@gmail.com>
* update code comment to match new func names
Signed-off-by: kim <grufwub@gmail.com>
* updates to user-agent string
Signed-off-by: kim <grufwub@gmail.com>
* update signed testrig models to fit with new transport logic (instead uses separate signer now)
Signed-off-by: kim <grufwub@gmail.com>
* fuck you linter
Signed-off-by: kim <grufwub@gmail.com>
* fix existing bio text showing as HTML
- updated replaced mentions to include instance
- strips HTML from account source note in Verify handler
- update text formatter to use buffers for string writes
Signed-off-by: kim <grufwub@gmail.com>
* go away linter
Signed-off-by: kim <grufwub@gmail.com>
* change buf reset location, change html mention tags
Signed-off-by: kim <grufwub@gmail.com>
* reduce FindLinks code complexity
Signed-off-by: kim <grufwub@gmail.com>
* fix HTML to text conversion
Signed-off-by: kim <grufwub@gmail.com>
* Update internal/regexes/regexes.go
Co-authored-by: Mina Galić <mina.galic@puppet.com>
* use improved html2text lib with more options
Signed-off-by: kim <grufwub@gmail.com>
* fix to produce actual plaintext from html
Signed-off-by: kim <grufwub@gmail.com>
* fix span tags instead written as space
Signed-off-by: kim <grufwub@gmail.com>
* performance improvements to regex replacements, fix link replace logic for un-html-ing in the future
Signed-off-by: kim <grufwub@gmail.com>
* fix tag/mention replacements to use input string, fix link replace to not include scheme
Signed-off-by: kim <grufwub@gmail.com>
* use matched input string for link replace href text
Signed-off-by: kim <grufwub@gmail.com>
* remove unused code (to appease linter :sobs:)
Signed-off-by: kim <grufwub@gmail.com>
* improve hashtagFinger regex to be more compliant
Signed-off-by: kim <grufwub@gmail.com>
* update breakReplacer to include both unix and windows line endings
Signed-off-by: kim <grufwub@gmail.com>
* add NoteRaw field to Account to store plaintext account bio, add migration for this, set for sensitive accounts
Signed-off-by: kim <grufwub@gmail.com>
* drop unnecessary code
Signed-off-by: kim <grufwub@gmail.com>
* update text package tests to fix logic changes
Signed-off-by: kim <grufwub@gmail.com>
* add raw note content testing to account update and account verify
Signed-off-by: kim <grufwub@gmail.com>
* remove unused modules
Signed-off-by: kim <grufwub@gmail.com>
* fix emoji regex
Signed-off-by: kim <grufwub@gmail.com>
* fix replacement of hashtags
Signed-off-by: kim <grufwub@gmail.com>
* update code comment
Signed-off-by: kim <grufwub@gmail.com>
Co-authored-by: Mina Galić <mina.galic@puppet.com>
* remove mention util function from db
* add ParseMentionFunc to gtsmodel
* add parseMentionFunc to processor
* refactor search to simplify it a bit
* add parseMentionFunc to account
* add parseMentionFunc to status
* some renaming for clarity
* test dereference of unknown mentioned account
* start pulling out + replacing urfave and config
* replace many many instances of config
* move more stuff => viper
* properly remove urfave
* move some flags to root command
* add testrig commands to root
* alias config file keys
* start adding cli parsing tests
* reorder viper init
* remove config path alias
* fmt
* change config file keys to non-nested
* we're more or less in business now
* tidy up the common func
* go fmt
* get tests passing again
* add note about the cliparsing tests
* reorganize
* update docs with changes
* structure cmd dir better
* rename + move some files around
* fix dangling comma
* add smtp configuration
* add email confirm + reset templates
* add email sender to testrig
* flesh out the email sender interface
* go fmt
* golint
* update from field with more clarity
* tidy up the email formatting
* fix tests
* add email sender to processor
* tidy client api processing a bit
* further tidying in fromClientAPI
* pin new account to user
* send msg to processor on new account creation
* generate confirm email uri
* remove emailer from account processor again
* add processCreateAccountFromClientAPI
* move emailer accountprocessor => userprocessor
* add email sender to user processor
* SendConfirmEmail function
* add noop email sender
* use noop email sender in tests
* only assemble message if callback is not nil
* use noop email sender if no smtp host is defined
* minify email html before sending
* fix wrong email address
* email confirm test
* fmt
* serve web hndler
* add email confirm handler
* init test log properly on testrig
* log emails that *would* have been sent
* go fmt ./...
* unexport confirm email handler
* updatedAt
* test confirm email function
* don't allow tokens older than 7 days
* change error message a bit
* add basic smtp docs
* add a few more snippets
* typo
* add email sender to outbox tests
* don't use dutch wikipedia link
* don't minify email html
* add statusesvisible convenience function
* add minID + onlyPublic to account statuses get
* move swagger collection stuff to common
* start working on Outbox GETting
* move functions into federationProcessor
* outboxToASCollection
* add statusesvisible convenience function
* add minID + onlyPublic to account statuses get
* move swagger collection stuff to common
* start working on Outbox GETting
* move functions into federationProcessor
* outboxToASCollection
* bit more work on outbox paging
* wrapNoteInCreate function
* test + hook up the processor functions
* don't do prev + next links on empty reply
* test get outbox through api
* don't fail on no status entries
* add outbox implementation doc
* typo
* reference logrus' global logger instead of passing and storing a logger reference everywhere
* always directly use global logrus logger instead of referencing an instance
* test suites should also directly use the global logrus logger
* rename gin logging function to clarify that it's middleware
* correct comments which erroneously referenced removed logger parameter
* setting log level for tests now uses logrus' exported type instead of the string value, to guarantee error isn't possible
