* add delivery worker type that pulls from queue to httpclient package
* finish up some code commenting, bodge a vendored activity library change, integrate the deliverypool changes into transportcontroller
* hook up queue deletion logic
* support deleting queued http requests by target ID
* don't index APRequest by hostname in the queue
* use gorun
* use the original context's values when wrapping msg type as delivery{}
* actually log in the AP delivery worker ...
* add uncommitted changes
* use errors.AsV2()
* use errorsv2.AsV2()
* finish adding some code comments, add bad host handling to delivery workers
* slightly tweak deliveryworkerpool API, use advanced sender multiplier
* remove PopCtx() method, let others instead rely on Wait()
* shuffle things around to move delivery stuff into transport/ subpkg
* remove dead code
* formatting
* validate request before queueing for delivery
* finish adding code comments, fix up backoff code
* finish adding more code comments
* clamp minimum no. senders to 1
* add start/stop logging to delivery worker, some slight changes
* remove double logging
* use worker ptrs
* expose the embedded log fields in httpclient.Request{}
* ensure request context values are preserved when updating ctx
* add delivery worker tests
* fix linter issues
* ensure delivery worker gets inited in testrig
* fix tests to delivering messages to check worker delivery queue
* update error type to use ptr instead of value receiver
* fix test calling Workers{}.Start() instead of testrig.StartWorkers()
* update docs for advanced-sender-multiplier
* update to the latest activity library version
* add comment about not using httptest.Server{}
* [feature] User-selectable preset themes
* docs, more theme stuff
* lint, tests
* fix css name
* correct some little issues
* add another theme
* fix poll background
* okay last theme i swear
* make retrieval of apimodel themes more conventional
* preallocate stylesheet slices
* prevent moved accounts from taking create-type actions
* update move logic
* federate move out
* indicate on web profile when an account has moved
* [docs] Add migration docs section
* lock while checking + setting move state
* use redirectFollowers func for clientAPI as well
* comment typo
* linter? i barely know 'er!
* Update internal/uris/uri.go
Co-authored-by: Daenney <daenney@users.noreply.github.com>
* add a couple tests for move
* fix little mistake exposed by tests (thanks tests)
* ensure Move marked as successful
* attach shared util funcs to struct
* lock whole account when doing move
* move moving check to after error check
* replace repeated text with error func
* linterrrrrr!!!!
* catch self follow case
---------
Co-authored-by: Daenney <daenney@users.noreply.github.com>
* [feature] Process incoming account Move activity
* fix targetAcct typo
* put move origin account on fMsg
* shift more move functionality back to the worker fn
* simplify error logic
* Add Swagger spec test script
* Fix Swagger spec errors not related to statuses with polls
* Add API tests that post a status with a poll
* Fix creating a status with a poll from form params
* Fix Swagger spec errors related to statuses with polls (this is the last error)
* Fix Swagger spec warnings not related to unused definitions
* Suppress a duplicate list update params definition that was somehow causing wrong param names
* Add Swagger test to CI
- updates Drone config
- vendorizes go-swagger
- fixes a file extension issue that caused the test script to generate JSON instead of YAML with the vendorized version
* Put `Sample: ` on its own line everywhere
* Remove unused id param from emojiCategoriesGet
* Add 5 more pairs of profile fields to account update API Swagger
* Remove Swagger prefix from dummy fields
It makes the generated code look weird
* Manually annotate params for statusCreate operation
* Fix all remaining Swagger spec warnings
- Change some models into operation parameters
- Ignore models that already correspond to manually documented operation parameters but can't be trivially changed (those with file fields)
* Documented that creating a status with scheduled_at isn't implemented yet
* sign drone.yml
* Fix filter API Swagger errors
* fixup! Fix filter API Swagger errors
---------
Co-authored-by: tobi <tobi.smethurst@protonmail.com>
* Implement client-side v1 filters
* Exclude linter false positives
* Update test/envparsing.sh
* Fix minor Swagger, style, and Bun usage issues
* Regenerate Swagger
* De-generify filter keywords
* Remove updating filter statuses
This is an operation that the Mastodon v2 filter API doesn't actually have, because filter statuses, unlike keywords, don't have options: the only info they contain is the status ID to be filtered.
* Add a test for filter statuses specifically
* De-generify filter statuses
* Inline FilterEntry
* Use vertical style for Bun operations consistently
* Add comment on Filter DB interface
* Remove GoLand linter control comments
Our existing linters should catch these, or they don't matter very much
* Reduce memory ratio for filters
This switches the documentation to use human sizes, so 50MiB instead of
an integer number of bytes. This makes it much easier to understand what
values are set, and less likely to accidentally set the wrong value.
* add authentication workflow to docs
* fixed the markdown callouts and added the new page to the mkdocs.yml
* fixed the markdown callouts properly this time
* applied the suggested edits
* inline styles don't render in code blocks; added bash to the curl code blocks
* scopes list was not rendering as a list; fixed a typo in getting your token warning
* scopes can be a space separated list and not just one
* apply suggested change to authentication.md
Co-authored-by: Daenney <daenney@users.noreply.github.com>
* applied latest suggested edits
* added information on Gts scoped tokens
* fixed admonition formatting
---------
Co-authored-by: Daenney <daenney@users.noreply.github.com>
* [bugfix] Pass `latest` to dereferenceThread instead of barebones status
* only mark status orphaned if visibility suggests parent is really deleted
* tone down "not deref'd" warnings, since they represent a legit visibility situation
* remove FAQ entry for "status not deref'd yet"
* [docs] Mounting config file in container
This adds a small section clarifying how to mount a config.yaml in a
container.
* [docs] Work in GTS_CONFIG_PATH
* [feature] serdes for moved/also_known_as
* document `alsoKnownAs` and `movedTo` properties
* only implicitly populate AKA uris from DB for local accounts
* don't let remotes store more than 20 AKA uris to avoid shenanigans
* Restructure the sections
* Mention how to clear out remote avatars and headers, related to #2328
* Add more links to authoritative documentation on the AWS side
* Show how to use the official AWS CLI
* Make a separate section for migration to S3 compatible object backends
* [feature] Account alias / move API + db models
* go fmt
* fix little cherry-pick issues
* update error checking, formatting
* add and use new util functions to simplify alias logic
* [docs] Describe how to block IPs using a firewall
This adds some documentation on how to block one, multiple or whole IP
ranges using the firewall. This can be helpful to protect from DDoS
attacks or block certain parties from being able to communicate with
your instance at all.
* [docs] Fix indenting of nft rule
* [docs] Split iptables and nftables into sections
* [docs] Change configuration creation instructions
This changes the wording to push people towards creating their own
configuration, without copying the whole example configuration. This
makes it much easier to reconcile necessary configruation changes on
upgrades.
* [docs] Reword container version section
This changes the wording in the Version section to make it more clear
what the risks of a moving tag are. It pushes people to use an explicit
release tag instead.
* [feature] Federate status language in + out
* go fmt
* tests, little fix
* improve comments
* unnest a bit
* avoid unnecessary nil check
* use more descriptive variable for contentMap
* prefer instance languages when selecting from contentMap
* update docs to reflect lang selection
* rename rdfLangString -> rdfLangs
* update comments to mention Pollable
* iter through slice instead of map
* update go text, include text/display
* [feature] Set instance langs, show post lang on frontend
* go fmt
* WebGet
* set language for whole article, don't use FA icon
* mention instance languages + other optional config vars
* little tweak
* put languages in config properly
* warn log language parse
* change some naming around
* tidy up validate a bit
* lint
* rename LanguageTmpl in template
* add db models + functions for keeping track of threads
* give em the old linty testy
* create, remove, check mutes
* swagger
* testerino
* test mute/unmute via api
* add info log about new index creation
* thread + allow muting of any remote statuses that mention a local account
* IsStatusThreadMutedBy -> IsThreadMutedByAccount
* use common processing functions in status processor
* set = NULL
* favee!
* get rekt darlings, darlings get rekt
* testrig please, have mercy muy liege
* love like winter! wohoah, wohoah
* domain allow side effects
* tests! logging! unallow!
* document federation modes
* linty linterson
* test
* further adventures in documentation
* finish up domain block documentation (i think)
* change wording a wee little bit
* docs, example
* consolidate shared domainPermission code
* call mode once
* fetch federation mode within domain blocked func
* read domain perm import in streaming manner
* don't use pointer to slice for domain perms
* don't bother copying blocks + allows before deleting
* admonish!
* change wording just a scooch
* update docs
* [docs] Callout no cluster/multi-node support
Though we have a GitHub issue for this, we don't mention it anywhere in
the docs. This adds it to the deployment considerations so hopefully
folks will see it when standing up their own instance.
* [docs] Add daenney as a developer
* [feature] Add http trace exporter, drop Jaeger
Jaeger supports ingesting traces using the OpenTelemetry gRPC or HTTP
methods. The Jaeger project has deprecated the old jaeger transport.
* Add support for submitting traces over HTTP
* Drop support for the old Jaeger protocol
* Upgrade the trace libraries to v1.17
Fixes: #2176Fixes: #2179
* [feature] list commands for both attachment and emojis
* use fewer commands, provide `local-only` and `remote-only` as filters
* envparsing
---------
Co-authored-by: Romain de Laage <romain.delaage@rdelaage.ovh>
Co-authored-by: tsmethurst <tobi.smethurst@protonmail.com>
* [feature] Don't emit timestamp in log lines
When running gotosocial with a service manager like systemd, or a
container runtime, the associated log driver usually emits timestamps
itself. In those cases, having the extra timestamp from our own log
lines ends up being a bit noisy and when centrally ingesting logs is
duplicate information.
This introduces a configuration flag that allows disabling emitting the
timestamp. It's only wired up for "daemonised" processes, meaning server
and testrig.
* [chore] Add docs for log-timestamp
* [feature] Simplify timestamp handling
Co-Authored-By: kim <89579420+NyaaaWhatsUpDoc@users.noreply.github.com>
* [chore] Less escaped double-quotes
* [chore] Fix help string
---------
Co-authored-by: kim <89579420+NyaaaWhatsUpDoc@users.noreply.github.com>
* init instance rules database model, admin api
* expose instance rules in public instance api
* public /api/v1/instance/rules route
* GET ruleById
* createRule route
* createRule auth check
* updateRule
* deleteRule
* list rules on about page
* ruleGet auth
* add about page ids for anchors
* process and store adding violated rules to reports
* admin api models for instance rules
* instance rule edit frontend
* change rule inputs to textareas
* database fixes after rebase (#2124)
* remove unused imports
* fix db migration column name
* fix tests
* fix more tests
* fix postgres error with wrongly used Ident
* add some tests, fiddle with rule model a bit, fix postgres migration
* swagger docs
---------
Co-authored-by: tsmethurst <tobi.smethurst@protonmail.com>
This adds the CSP header with a policy of only loading from the same
domain. We don't make use of external media, CSS, JS, fonts, so we don't
ever need external data loaded in our context.
When building a DEBUG build, the policy gets extended to include
localhost:*, i.e localhost on any port. This keeps the live-reloading
flow for JS development working. localhost and 127.0.0.1 are considered
to be the same so mixing and matching those doesn't result in a CSP
violation.
With Feditext now accepting beta users, this adds it as the third
client to recommend so we have web and the dominant mobile platforms
covered.
This also removes the screenshots from the README, because it became a
mess trying to add a third one. Either the cells become very narrow, or
the table doubles in height. As the UI may also change over time, it
might be better to point folks at the apps instead who'll hopefully have
up to date screenshots in their storefronts.
The current command `create database gotosocial with locale C.UTF-8 template template0;` fails because the locale has to be quoted:
```
postgres=# create database gotosocial with locale C.UTF-8 template template0;
ERROR: syntax error at or near "."
LINE 1: create database gotosocial with locale C.UTF-8 template temp...
```
Tested manually that it works with the quotes and the official postgres examples also use quotes around the locales: https://www.postgresql.org/docs/current/sql-createdatabase.html
* update go-fed
* do the things
* remove unused columns from tags
* update to latest lingo from main
* further tag shenanigans
* serve stub page at tag endpoint
* we did it lads
* tests, oh tests, ohhh tests, oh tests (doo doo doo doo)
* swagger docs
* document hashtag usage + federation
* instanceGet
* don't bother parsing tag href
* rename whereStartsWith -> whereStartsLike
* remove GetOrCreateTag
* dont cache status tag timelineability
* Support setting private notes on accounts
* Reformat comment whitespace
* Add missing license headers
* Use apiutil.ParseID
* Rename Note model and cache to AccountNote
* Update golden cache config in test/envparsing.sh
* Rename gtsmodel/note.go to gtsmodel/accountnote.go
* Update AccountNote uniqueness constraint name
Now has same prefix as other indexes on this table.
---------
Co-authored-by: tobi <31960611+tsmethurst@users.noreply.github.com>
The old default of 30d can lead to a lot of media getting cached and
significant disk usage, even on small or single person instances. A lot
of deployments decrease this value, to 15 or even less. This is less of
an issue when using object storage, but for local storage which is the
more popular deployment option running out of disk space is unpleasant.
With GoToSocial's aim to fit in small places, this changes the default
to a much more conservative 7 days. In all likelihood people aren't
scrolling that far back in their timeline so this change shouldn't
result in any issue. Existing deployments will only be affected by
this change if the admin hasn't already configured this value, or didn't
bootstrap from the example configuration.
* Set default value of SMTPFrom to empty string
This parameter should contain proper e-mail address (to be provided by user during configuration).
* Update default values in example/config.yaml
Default values and related comments in example/config.yaml are aligned
with values defined in internal/config/defaults.go.
Small improvements to foramting of config.yaml file.
* Add default value for AdvancedThrottlingRetryAfter to internal/config/defaults.go
AdvancedThrottlingRetryAfter was introduced in 70739d3 (superseriousbusiness/gotosocial#1466).
* Update config.yaml snippets in documentation
* [docs] Rework backups a bit
This changes the existing backup documentation to:
* Push a bit harder on people to perform backups, it's not really just a
nice to have
* Removes the language about migrating to/from GoToSocial and a
different ActivityPub server since that's really not supported
* Adds a section about using backup software and provides an example on
how to do this using Borgmatic
* [docs] Remove too much info in db section
* [docs] Add docs on how to backup local media
This adds documentation pointing people at the media list-local command
in order to determine what media files they need to include as part of
their backups.
Provides a Python script that people can use to transform the media
listing from the CLI into Borg patterns. It also includes a Borgmatic
config.yaml in the repository so people can easily fetch it instead of
copy-pasting from the docs.
* [bugfix] Ensure we emit an absolute path prefix
It works either way, as a pattern like data/files/<ID> would match a
file on /data/files/<ID>. But it would potentially also match any path
that happens to include data/files/<ID> but not rooted at the
storage-local-base-path.
* [docs] Add more links to media list CLI reference
* [feature] Add media list command
This is an attempt to help alleviate #1776. Using admin media list
--local the full path to each local media file will be printed, with a
newline. The output of this should be feadable into backup tools in
order to allow to backup local media too. Together with the database
this should allow to fully recover from the loss of an instance.
The list command also gets a --remote flag for symmetry. In the case
of --remote we print the RemoteURL instead, the location the asset can
be retrieved from.
To get all media, you can run with --local and --remote.
* [bugfix] Fix the test failures
* [feature] Reimplement list media as top commands
This changes the implementation of admin media list --<variant> to two
separate top-level commands, list-local and list-remote.
The implementation now iterates over over the database in batches of 200
in order to avoid loading all media metadata into memory.
* [feature] Implement ListMedia with filter callback
This does away with the somewhat odd iterator-like structure we had
before and does away with most of the loop duplication in list-local and
list-remote. Instead they call GetAllMediaPaths with a filter func to
select the media they want. That's accumulated into a slice and
eventually returned.
* [bugfix] Simplify remote filter
Since we don't append the empty string anywhere, the remote filter can
be limited to returning RemoteURL, as that'll be an empty string for
local media.
* [docs] Add media list commands to CLI reference
---------
Co-authored-by: tobi <31960611+tsmethurst@users.noreply.github.com>
* [docs] Add a certificates guide in Advanced
This adds some documentation about the process of getting certificates
through ACME in general. It also provides a number of links to
alternative clients and certbot deployment guides that are up to date.
Slightly restructure the NGINX and Apache reverse proxy documentation
and insert mentions to the Provisioning TLS certificates advanced
documentation in them.
* [docs] Add firewall section in Advanced
* [docs] Add new guides to section indexes
* [docs] Fix spelling issue
* [docs] Fix a few typos
* [docs] Made Advanced its own section
This splits the Advanced page off from the Getting Started guide and
makes it its own thing. It now has some additional sub-sections for
bigger topics like caching and enhanced security. This also moves
tracing from Getting Started to Advanced as that feels like a more
appropriate location for it.
The enhanced security looks a little silly with a single section, but I
have guides pending for firewall configurations and I'd also like to
consolidate our how to provision TLS certificates in there as we repeat
this information multiple times.
* [docs] Fix all my spelling errors
* [docs] Inline the links in sandboxing
This tries to revamp/restructure the installation guide. It's renamed to
"Getting Started" as it now contains a few more things than just the
installation, especially the deployment considerations which we didn't
use to spell out as much ahead of time.
Installation is now a section with the guides on their own. I've removed
a bit of redundancy like the user creation. I also removed the rogue
reverse proxy section in the Docker guide and lifted that into the
reverse proxy section.
This adds an example on how to get Grafana Tempo up to receive spans as
well as Grafana itself to view them.
I've added this as a separate Tracing doc in the installation guide as
the Advanced one was starting to get rather full.
Fixes: #1791
Our default configuration places the SQLite DB in /gotosocial/, but the
AppArmor profile doesn't allow us to write there. Instead of making the
whole directory writable, add a writable area in /gotosocial/db/ instead
and advise in the docs to move the DB there.
Instead of a manually curated list of distribution packages, this adds
the Repology[1] widget instead.
This also separates out the distribution packages from the
self-hosting/deployment options into their own sections.
[1]: https://repology.org/project/gotosocial/versions
* start working on lists
* further list work
* test list db functions nicely
* more work on lists
* peepoopeepoo
* poke
* start list timeline func
* we're getting there lads
* couldn't be me working on stuff... could it?
* hook up handlers
* fiddling
* weeee
* woah
* screaming, pissing
* fix streaming being a whiny baby
* lint, small test fix, swagger
* tidying up, testing
* fucked! by the linter
* move timelines to state like a boss
* add timeline start to tests using state
* invalidate lists
This adds the preferences endpoint to our Mastodon Client API
implementation. It's a read-only endpoint that returns a number of
user preferences. Applications can query these settings when logging in
a user (for the first time) to configure themselves.
* Explain that markdown is enabled on the user settings page.
The current wording seems to imply that the input formatting can be
set on a per-post basis, which no client currently supports. It's not
clear that you have to go to the user settings page to change it.
* Update FAQ; some of the missing features are not missing any more!
* start working on notifs for new posts
* tidy up a bit
* update swagger
* carry over show reblogs + notify from follow req
* test notify on status post
* update column slice
* dedupe update logic + add tests
* fix own boosts not being timelined
* avoid type check, passing unnecessary accounts
* remove unnecessary 'inReplyToID' check
* add a couple todo's for future db functions
* start poking timelines
* OK yes we're refactoring, but it's nothing like the last time so don't worry
* more fiddling
* update tests, simplify Get
* thanks linter, you're the best, mwah mwah kisses
* do a bit more tidying up
* start buggering about with the prepare function
* fix little oopsie
* start merging lists into 1
* ik heb een heel zwaar leven
nee nee echt waar
* hey it works we did it reddit
* regenerate swagger docs
* tidy up a wee bit
* adjust paging
* fix little error, remove unused functions
* start fiddling about with email sending to allow multiple recipients
* do some fiddling
* notifs working
* notify on closed report
* finishing up
* envparsing
* use strings.ContainsAny
This adds a section to the docs instructing how to ensure apps will be
able to successfully login to an instance when host- and account-domain
differ.
Resolves#1609
* [feature] Allow admins to send test emails
* implement unwrap on new error type
* add + use gtserror types
* GoToSocial Email Test -> GoToSocial Test Email
* add + use getInstance db call
* removed unused "unknown" error type
* [chore] Remove years from all license headers
Years or year ranges aren't required in license headers. Many projects
have removed them in recent years and it avoids a bit of yearly toil.
In many cases our copyright claim was also a bit dodgy since we added
the 2021-2023 header to files created after 2021 but you can't claim
copyright into the past that way.
* [chore] Add license header check
This ensures a license header is always added to any new file. This
avoids maintainers/reviewers needing to remember to check for and ask
for it in case a contribution doesn't include it.
* [chore] Add missing license headers
* [chore] Further updates to license header
* Use the more common // indentend comment format
* Remove the hack we had for the linter now that we use the // format
* Add SPDX license identifier
* [feature] Provide .well-known/host-meta endpoint
This adds the host-meta endpoint as Mastodon clients use this to
discover the API domain to use when the host and account domains aren't
the same.
* Address review comments
Currently, GtS only supports using the built-in LE client directly for
TLS. However, admins may still want to use GtS directly (so without a
reverse proxy) but with certificates provided through some other
mechanism. They may have some centralised way of provisioning these
things themselves, or simply prefer to use LE but with a different
challenge like DNS-01 which is not supported by autocert.
This adds support for loading a public/private keypair from disk instead
of using LE and reconfigures the server to use a TLS listener if we
succeed in doing so.
Additionally, being able to load TLS keypair from disk opens up the path
to using a custom CA for testing purposes avoinding the need for a
constellation of containers and something like Pebble or Step CA to
provide LE APIs.
* Advertise rich text formats, support content_type field
* Update JSON in instance patch tests
* Replace format with content_type everywhere
* update migration to work with both pg and sqlite
* regenerate swagger docs
* update instance serialization + tests
* fix up
* learn to code tobi please, i'm begging you
---------
Co-authored-by: tsmethurst <tobi.smethurst@protonmail.com>
* start fiddling
* the ol' fiddle + update
* start working on fetching statuses
* poopy doopy doo where r u uwu
* further adventures in featuring statuses
* finishing up
* fmt
* simply status unpin loop
* move empty featured check back to caller function
* remove unnecessary log.WithContext calls
* remove unnecessary IsIRI() checks
* add explanatory comment about status URIs
* change log level to error
* better test names
This removes the current default of checking for membership of the admin
or admins group and makes it required to explicitly configure which
groups should grant admin access, if any.
Relying on the implicit default of admin or admins is potentially
dangerous as that group may contain a different subset of people that we
may wish to grant admin access to GtS. This is probably not an issue for
a single-person instance, but for a community instance different admin
groups may exist in an OIDC provider for different applications.
I'm explicitly opting for not defaulting the value of oidc-admin-groups
to admin,admins because I think it's better for those things to be
explicitly configured.
* implement status pin client api + web handler
* make test names + comments more descriptive
* don't use separate table for status pins
* remove unused add + remove checking
* tidy up + add some more tests
* [feature] Use Material theme for docs
This changes the theme to use Material with the slate (dark) theme and
the accent colour set to orange. It also replaces the swagger plugin
with one that works correctly with the Material theming.
* Add theme by @f0x52
Co-authored-by: f0x52 <f0x@cthu.lu>
* Enable social cards
The dependencies for cairosvg and pillow can be satisfied on RTD, so
this adds them to our requirements.txt and enables the social plugin.
This generates opengraph and twitter card tags as well as fancy preview
images in GTS colour style for platforms that show that as part of link
previews.
* Update Conda environment.yml
* update theme for swagger, basic light theme option
* dark mode mobile header bg color
---------
Co-authored-by: f0x52 <f0x@cthu.lu>
This adds a lightweight form of tracing to GTS. Each incoming request is
assigned a Request ID which we then pass on and log in all our log
lines. Any function that gets called downstream from an HTTP handler
should now emit a requestID=value pair whenever it logs something.
Co-authored-by: kim <grufwub@gmail.com>
