Implements #864 and should speed up s3 based installations by a lot.
With more static urls, we can then also implement #1026 for even
better performance when used in conjunction with CDNs
In the previous changes that expanded the IPv4 and IPv6 deny lists based
on the IANA registries we inadvertently added a number of duplicates.
This is unnecessary as they're already caught by larger prefixes and
means there's less entries to scan.
This change removes all prefixes that are subnets of other prefixes.
* ap: add support for PKCS1 "RSA PUBLIC KEY" pem block type
Signed-off-by: Sigrid Solveig Haflínudóttir <sigrid@ftrv.se>
* ap: report no PEM data or unknown pem block type
Signed-off-by: Sigrid Solveig Haflínudóttir <sigrid@ftrv.se>
Signed-off-by: Sigrid Solveig Haflínudóttir <sigrid@ftrv.se>
* only return error for emoji fetch if NOT errnoentries
Signed-off-by: kim <grufwub@gmail.com>
* reformat gts->api model slice conversion to standard error behaviours and reduce code reuse
Signed-off-by: kim <grufwub@gmail.com>
Signed-off-by: kim <grufwub@gmail.com>
* start refactoring some of the search + deref logic
* add tests for search api
* rename GetRemoteAccount + GetRemoteStatus
* make search function a bit simpler + clearer
* fix little fucky wucky uwu owo i'm just a little guy
* update faulty switch statements
* update test to use storage struct
* redo switches for clarity
* reduce repeated logic in search tests
* fastfail getstatus by uri
* debug log + trace log better
* add implementation note
* return early if no result for namestring search
* return + check on dereferencing error types
* errors hah what errors
* remove unneeded error type alias, add custom error text during stringification itself
* fix a woops recursion 🙈
Signed-off-by: kim <grufwub@gmail.com>
Co-authored-by: kim <grufwub@gmail.com>
In d6f4d196c9 we swapped to use the
AccountDomain but that actually goes against the intent of the change.
This reverts that change and uses the host domain again.
Currently requests set their own User-Agent. This moves it down to set
it in the transport's do() method, to guarantee it's always set on all
requests.
* [bugfix] Use AccountDomain for user agent
By using the account domain we can pinpoint the source of the request
more accurately when looking at the User-Agent header.
* [chore] Align user-agent header with spec
Based on RFC 7231, our User-Agent header doesn't quite match. It seems
to always want Name [/ Version] pairs, with comments in parenthesis and
multiple comments separated by a semicolon.
Align our UA with that, using application name first by itself in case
someone has customised it with the source instance in a comment. Follow
that up with gotosocial/<version> and a comment pointing at the source
code.
This also drops the mention of gofed/activity since a fork is in use.
* [bugfix] Ensure requests happen over TCP
It's possible for the network to be udp4 or udp6. This is rather
unlikely to occur, but since we're given the network anyway as part of
the Sanitize function getting called we might as well check for it.
* [chore] Align reserved v6 blocks to IANA registry
* [chore] Add test for ValidateIP
The net and netip packages diverge in that net.ParseIP will consider an
IPv4-mapped address to be an IPv4 address and as such it would get
caught by the IPv4Reserved list. However, netip considers it an IPv6
address, so we need to ensure the mapped range is in IPv6Reserved.
* [chore] Align reserved v4 blocks to IANA registry
This includes a number of tests for /32's explicitly called out in the
registry to ensure we always consider those invalid.
* [bugfix]: Fix IPv6 validation
The current code considers ff00::/8 valid, but contrary to the comment
that's not the global unicast range. ff-prefixes in IPv6 denote
multicast.
This adapts the code to take the same approach as IPv4, explicitly
blacklisting reserved internal/private ranges.
* [chore] Add missing 4 in IPv4Reserved doc comment
* add FilePath regex
* add `admin media prune orphaned` command
* add prune orphaned function to media manager
* don't mark flag as required
* document admin media prune orphaned cmd
* oh envparsing.sh you coy minx
* start adding admin emoji PATCH stuff
* updating works OK, now how about copying
* allow emojis to be copied
* update swagger docs
* update admin processer to use non-interface storage driver
* remove shortcode updating for local emojis
* go fmt
Co-authored-by: f0x52 <f0x@cthu.lu>
* [bugfix] Prevent future statuses entering timeline
Statuses created more than 5 minutes into the future are now rejected in the visibility package.
* Come on buddy
* [feature/performance] Fail fast when doing remote transport calls inside incoming request contexts
* [chore] Reduce outgoing request timeout to 15s
* log error messages when fastfailing
* use context.Value() instead of wrapped context, wrap error with fastfail instead of extra log entry
* add fast-fail context key test
Signed-off-by: kim <grufwub@gmail.com>
Co-authored-by: kim <grufwub@gmail.com>
* replace s3 storage implementation to also use kv.KVStore
Signed-off-by: kim <grufwub@gmail.com>
* pull in latest `go-store` fix
Signed-off-by: kim <grufwub@gmail.com>
* pull-in go-store v2.0.9 fixes, update s3 put chunk size to 5MiB
Signed-off-by: kim <grufwub@gmail.com>
Signed-off-by: kim <grufwub@gmail.com>
* [enhancement] markdown: disable SmartyPantsFractions
fixes#1028
* some fractions are still converted to unicode
to fix that, we might need to disable smartypants indeed in its entirety
* disable smartypants completely
for lack of a better simple solution
* add account block DB cache and remove reliance on relational joins
* actually include cache key arguments...
* add a PutBlock() method which also updates the block cache, update tests accordingly
* use `PutBlock` instead of `Put(ctx, block)`
* add + use functions for deleting + invalidating blocks
Signed-off-by: kim <grufwub@gmail.com>
Co-authored-by: tsmethurst <tobi.smethurst@protonmail.com>
* refactor federator account statuses delete to better catch errors, ensure next maxID is always set
Signed-off-by: kim <grufwub@gmail.com>
* fix error statement missing 2nd format operator
Signed-off-by: kim <grufwub@gmail.com>
Signed-off-by: kim <grufwub@gmail.com>
Google abandoned Floc in favour of Topics and changed the
necessary permissions policy. Currently, the Floc policy will
block Topics. This change includes switching to the updated
policy to be ahead of Google abandoning recognising the Floc
policy.
Update the function documentation to include the current relevant
documentation.
* convert most of the caches to use result.Cache{}
* add caching of emojis
* fix issues causing failing tests
* update go-cache/v2 instances with v3
* fix getnotification
* add a note about the left-in StatusCreate comment
* update EmojiCategory db access to use new result.Cache{}
* fix possible panic in getstatusparents
* further proof that kim is not stinky
* [bugfix] Fix unicode-unaware word boundary check in hashtag regex
Go `\b` does not care for Unicode, and without lookahead, the workarounds got
very ugly. So I replaced the regex with a parser.
The parser runs in O(n) time and performance should not be affected.
* [bugfix] Add back hashtag max length and add tests for it
* [feature] Add 'role' field to api serialization of local accounts
* [chore] Add a bit of license text while I'm here
* [frogend] render account role on same line as username in web view of profile
* style tweaking on role badges, general profile header layout
* profile stats wrapping
* don't render standard 'user' role on web view
Co-authored-by: f0x <f0x@cthu.lu>
* [feature] Add emoji categories GET
Serialize emojis in appropriate categories; make it possible to get categories via the admin API
* [feature] Create (or use existing) category for new emoji uploads
* fix lint issue
* update misleading line in swagger docs
* Add instance-expose-public-timeline flag
Adds a config flag that allows unauthenticated access to /api/v1/timelines/public. Defaults to false to replicate existing behaviour.
* Update structure following review
* Add comment
* Fix linting
* Fix login on Mastodon iOS app for users with no statuses
Mastodon for iOS can't cope with an empty string for a date and
expect a JSON `null` instead.
Fixes https://github.com/superseriousbusiness/gotosocial/issues/1010
* Fix expected values in tests to match
* fix incorrect static remote url use for emojis
* warn when emoji/attachment already exists
* defer emoji postdata execution
* rename ctx to innerCtx for clarity
* warn on emoji too large
* small efficiency fix in fetchRemoteAccountEmojis
* tidy up lock+load
* lock processing emojis
* fix little fucky wucky
* this wasn't go fmted for some reason
* [feature] Read + Write tombstones for deleted Actors
* copyTombstone
* update to use resultcache instead of old ttl cache
Signed-off-by: kim <grufwub@gmail.com>
* update go-cache library to fix result cache capacity / ordering bugs
Signed-off-by: kim <grufwub@gmail.com>
* bump go-cache/v3 to v3.1.6 to fix bugs
Signed-off-by: kim <grufwub@gmail.com>
* switch on status code
* better explain ErrGone reasoning
Signed-off-by: kim <grufwub@gmail.com>
Co-authored-by: kim <grufwub@gmail.com>
* S3: add config value "proxy" for not redirecting
Signed-off-by: Mara Sophie Grosch <littlefox@lf-net.org>
* S3: document new config value "proxy"
* S3: add new config value "proxy" to test scripts
Signed-off-by: Mara Sophie Grosch <littlefox@lf-net.org>
* [feature] Make instance thumbnail configurable via admin panel
* log db errors in InstanceToAPIInstance
* only update instance in db if necessary
* start adding tests
* finish test
* re-add eslint
* fix oauth url getting too long
* actually attach single emoji get and delete routes
* basic emoji details + deletion using rtk query
* refactor emoji upload to rtk query
* clean up old redux api+reducers for custom emoji
* fix validation order
* refactor custom emoji form fields
* remove unused requires
* cleanup, fix most eslint errors
* more small eslint fixes
* fix max emoji size
* tiny bit of function documentation
* greatly simplify httpclient request queuing
Signed-off-by: kim <grufwub@gmail.com>
* improved request queue mutex logic
Signed-off-by: kim <grufwub@gmail.com>
* use improved hashmap library
Signed-off-by: kim <grufwub@gmail.com>
* add warn logging when request queues are full
Signed-off-by: kim <grufwub@gmail.com>
* improve worker pool prefix var naming
Signed-off-by: kim <grufwub@gmail.com>
* improved worker pool error logging
Signed-off-by: kim <grufwub@gmail.com>
* move error message into separate field
Signed-off-by: kim <grufwub@gmail.com>
* remove old log statement
Signed-off-by: kim <grufwub@gmail.com>
* don't export worker message, it gets very spammy :')
Signed-off-by: kim <grufwub@gmail.com>
Signed-off-by: kim <grufwub@gmail.com>
* bump go-store version to v2.0.5, init kv.KVStore without initial clean (as we are using for storage, not as a key-value store)
Signed-off-by: kim <grufwub@gmail.com>
* remove newline
Signed-off-by: kim <grufwub@gmail.com>
Signed-off-by: kim <grufwub@gmail.com>
If set, the landing page user configuration value is used as a Gin
context parameter, which seems incorrect, since a normal request isn't
going to have a parameter named after an arbitrarily configured user.
Instead, the user name should be used directly when building the
redirect URL.
* use readcloser for content.Content
* call media postdata function no matter what
* return a readcloser from data func
* tidy of logic of readertostore
* fix whoopsie
The migration that adds the `admin_account_actions` table did so at the
same time as adding indexes onto the new table. This code was ran inside
a `RunInTx` function, but the table creation did not use the transaction
reference, while the creation of the indexes did. This could cause a
race between the table and index creations, depending on the scheduling
order. If the table creation did not win the race, then the migration
would fail.
This changeset corrects the table creation to also be done inside the
same transaction as the index creation.
Signed-off-by: Terin Stock <terinjokes@gmail.com>
Signed-off-by: Terin Stock <terinjokes@gmail.com>
* select emoji using image_static_url
* use updated on AP emojis
* allow refetch of updated emojis
* cheeky workaround for test
* clean up old files for refreshed emoji
* check error for originalPostData
* shorten GetEmojiByStaticImageURL
* delete kirby (sorry nintendo)
* add admin emojis get path + model + docs
* stub admin emojis get processor function
* add id + disabled fields to admin emoji
* add emoji -> api admin emoji converter
* tidy up a bit
* add GetEmojis function
* finish up get emojis function
* order by shortcodedomain
* ASC
* tidy up + explain
* update to allow paging
* make admin emojis pageable
* fix mixed case paging
* normalize emoji queries a bit better
* test emoji get paging
* make limit optional
* fix incorrect path in media cleanup tests
* i have bad coder syndrome
* don't trimspace
* rename -> GetUseableEmojis
* wrap emoji query in subquery
avoid selecting more than we need
* fix a bit of sillyness teehee
* fix subquery postgres woes
* start adding rss functionality
* add gorilla/feeds dependency
* first bash at building rss feed
still needs work, this is an interim commit
* tidy up a bit
* add publicOnly option to GetAccountLastPosted
* implement rss endpoint
* fix test
* add initial user docs for rss
* update rss logo
* docs update
* add rssFeed to frontend
* feed -> feed.rss
* enableRSS
* increase rss logo size a lil bit
* add rss toggle
* move emojify to text package
* fiddle with rss feed formatting
* add Text field to test statuses
* move status to rss item to typeconverter
* update bun schema for enablerss
* simplify 304 checking
* assume account not rss
* update tests
* update swagger docs
* allow more characters in title, trim nicer
* update last posted to be more consistent
* use bun.Ident for user queries
* use bun.Ident for account queries
* use bun.Ident for media queries
* add DeleteAccount func
* remove CaseInsensitive in Where+use Ident ipv Safe
* update admin db
* update domain, use ident
* update emoji, use ident
* update instance queries, use bun.Ident
* fix media
* update mentions, use bun ident
* update relationship + tests
* use tableexpr
* add test follows to bun db test suite
* update notifications
* updatebyprimarykey => updatebyid
* fix session
* prefer explicit ID to pk
* fix little fucky wucky
* remove workaround
* use proper db func for attachment selection
* update status db
* add m2m entries in test rig
* fix up timeline
* go fmt
* fix status put issue
* update GetAccountStatuses
Fixes an issue where async processing was not completing correctly.
In particular this applies to side effects of domain blocks: while the domain block was being entered and enforced correctly, side effects like deleting accounts and updating the instance entry for the blocked instance were not. This fixes that :)
* re-structure bundler, settings panel files
* add more info logging
* tidy up CSS syntax errors
* split into lib/ files
* livereloading server
* fix factor function for production builds
* remove testing console.log
* default to production env, saves 300kb bundle size
* handle more syslogging levels, use singular time format variable, add entry .New() function
* pass in calldepth to lower log functions to ensure correctly set
* update truncate length in syslog test
* vastly simplify logging caller information
Signed-off-by: kim <grufwub@gmail.com>
* fix failing test due to multiple calls to processor.Start()
Signed-off-by: kim <grufwub@gmail.com>
Signed-off-by: kim <grufwub@gmail.com>
* update dependencies, bump Go version to 1.19
* bump test image Go version
* update golangci-lint
* update gotosocial-drone-build
* sign
* linting, go fmt
* update swagger docs
* update swagger docs
* whitespace
* update contributing.md
* fuckin whoopsie doopsie
* linterino, linteroni
* fix followrequest test not starting processor
* fix other api/client tests not starting processor
* fix remaining tests where processor not started
* bump go-runners version
* don't check last-webfingered-at, processor may have updated this
* update swagger command
* update bun to latest version
* fix embed to work the same as before with new bun
Signed-off-by: kim <grufwub@gmail.com>
Co-authored-by: tsmethurst <tobi.smethurst@protonmail.com>
* Start adding account emoji
* get emojis serialized + deserialized nicely
* update tests
* set / retrieve emojis on accounts
* show account emojis in web view
* fetch emojis from db based on ids
* fix typo in test
* lint
* fix pg migration
* update tests
* update emoji checking logic
* update comment
* clarify comments + add some spacing
* tidy up loops a lil (thanks kim)
* more nil checks baybeeeeeeeeeeeeeeeeeeee
Signed-off-by: kim <grufwub@gmail.com>
* move item nil checks
Signed-off-by: kim <grufwub@gmail.com>
Signed-off-by: kim <grufwub@gmail.com>
* *actually* start at top of stack loop on find remote child, fix iter indexing
Signed-off-by: kim <grufwub@gmail.com>
* add improved code comment
Signed-off-by: kim <grufwub@gmail.com>
Signed-off-by: kim <grufwub@gmail.com>
* update thread iterators to not use recursion, rewrote both
Signed-off-by: kim <grufwub@gmail.com>
* fix endless descendant deref, don't error if fetching existing status
Signed-off-by: kim <grufwub@gmail.com>
* don't refetch remote ancestor statuses, improve descendant iter commenting
Signed-off-by: kim <grufwub@gmail.com>
* move collection page next logic so we capture first page of entities
Signed-off-by: kim <grufwub@gmail.com>
* improve log format argument quoting
Signed-off-by: kim <grufwub@gmail.com>
* improve code commenting of collection paging
Signed-off-by: kim <grufwub@gmail.com>
* only dereference announce's originating status if _not_ local. update DereferenceThread() signature. cleanup searchStatusByURI()
Signed-off-by: kim <grufwub@gmail.com>
Signed-off-by: kim <grufwub@gmail.com>
* update Activity
* add instance-deliver-to-shared-inboxes setting
* update activity version again
* add SharedInboxURI field to accounts
* serdes for endpoints/sharedInbox
* deliver to sharedInbox if one is available
* update tests
* only assign shared inbox if shared domain
* look for shared inbox if currently nil
* go fmt
* finger to get params.RemoteAccountID if necessary
* make comments clearer
* compare dns more consistently
* add func for deleting status from db + cache
* move deletes entirely back to processor
and also only do a delete if the requesting account owns the item being deleted
* tidy up unboost processing
* delete status more efficiently
* fix wrong account id on remote test attachments
* fix federator test
* log internal server errors from 500 api calls
* don't exec into nil dest
* don't exec into nil dest
* log error in router logger not api errorhandling
* update logging a tad
* linter
* add custom css account property + db func to fetch
* allow account to get/set custom css
* serve custom css for an account
* go fmt
* use monospace for customcss, add link
* add custom css to account cache
* fix broken field
* add custom css docs to user guide
* add `accounts-allow-custom-css` config flag
* add allow custom css to /api/v1/instance response
* only show/set custom css if allowed to do so
* only set/serve custom account css if enabled
* update swagger docs
* chain promise
* make bool a bit clearer
* use cache for GetAccountCustomCSSByUsername
* start implementing remote emoji fetcher
* update status where pk
* aaa
* tidy up a little
* check size limits for emojis
* thank you linter, i love you <3
* update swagger docs
* add emoji dereference test
* make emoji max sizes configurable
* normalize db.ErrAlreadyExists
* f0x gitignore additions
* better meta title and descriptions
* user avatar icon for thread and profile meta tags
* use proper tag for image
* whitespace
* add noescapeAttr template function
* use ogMeta struct for opengraph
* maxOGDescriptionLength = 300
Co-authored-by: tsmethurst <tobi.smethurst@protonmail.com>
* migrate emojis
* add get emoji to s2s (federation) API
* add new emoji db + cache functions
* add shortcodeDomain lookup for emojis
* check existing emojis w/cache, not w/constraints
* go fmt
* add putEmoji func
* use new db emoji funcs instead of where
* remove emojistringstotags func
* add unique constraint back in
* fix up broken migration
* update index
* validate web-asset-base-dir
* move default icons into converter
* always ensure avatar + header on api accounts
* update tests
* add default header
* don't return error from web module creation anymore
* tidy a bit
* use pngs for default avatars rather than svgs
* fetch creation and fetching domain blocks from db
Signed-off-by: kim <grufwub@gmail.com>
* add separate domainblock cache type, handle removing block from cache on delete
Signed-off-by: kim <grufwub@gmail.com>
* fix sentinel nil values being passed into cache
Signed-off-by: kim <grufwub@gmail.com>
Signed-off-by: kim <grufwub@gmail.com>
* use 'test' value for testrig storage backend
* update test dependency
* add WaitFor func in testrig
* use WaitFor function instead of time.Sleep
* tidy up tests
* make SentMessages a sync.map
* go fmt
* Fix some bugs when viewing a user's posts: include their self-replies (threads) even when excludeReplies is set, and use in_reply_to_uri instead of in_reply_to_id to filter out replies
* Assign values to InReplyToURI when creating statuses. Add index and update old statuses with a migration
* upstep bun and sqlite versions
* allow specific columns to be updated in the db
* only update necessary columns for user
* bit tidier
* only update necessary fields of media_attachment
* only update relevant instance fields
* update tests
* update only specific account columns
* use bool pointers on gtsmodels
includes attachment, status, account, user
* update columns more selectively
* test all default fields on new account insert
* updating remaining bools on gtsmodels
* initialize pointer fields when extracting AP emoji
* copy bools properly
* add copyBoolPtr convenience function + test it
* initialize false bool ptrs a bit more neatly
* start work on user panel
* parse source first before checking if empty form
* newline
* set avi + header nicely
* add posts settings
* render signin a bit nicer on mobile
* return OK json on successful change
* return unauthorized on bad password
* clarify message on insecure password
* make login a bit prettier
* add alt text + border round image previews
* add logout button
* add password change
* styling updates
* redirect /auth/edit to /user
* update tests
* fix validation tests
* better labels, link to more info
* make submit button generic component
* move submit button inside forms
* add autocomplete labels to password fields
* fix indentation (thx eslint)
* update eslintrc
* eslint: no-unescaped-entities
* initial deduplication between user and admin panel
* add default status/post format setting
* user panel styling for inputs
* update user panel styling, include normalize css
* add placeholder text
* input padding
Co-authored-by: f0x <f0x@cthu.lu>
* add post_format to acct & use it when making post
* update swagger docs
* add status_format updating to frontend
* fix up tests
* post_format => status_format
* add status_format to account validation
