Database migration 20220506110822_add_account_raw_note.go has some error
handling code to detect some error messages as "ok", but only done for
english error messages. This commit adds a check for the specific error
code, which should be locale agnostic.
* add migration to clean up duplicated media
* use /tmp/gotosocial for testrig storage path
* defer remove storage tempdir
* skip if not attached to status or status not found
* log errors at error level
* only log delete as else clause if successful
* just return nil on down
* reword delete logic a little bit
* check if storage base path is defined
* check for status id more thoroughly
* don't log error if just no rows
* go fmt
* break statusIDLoop when found
* break currentlyUsedLoop when found
* move config flag names and usage to config package, rewrite config package to use global Configuration{} struct
Signed-off-by: kim <grufwub@gmail.com>
* improved code comment
Signed-off-by: kim <grufwub@gmail.com>
* linter
Signed-off-by: kim <grufwub@gmail.com>
* fix unmarshaling
Signed-off-by: kim <grufwub@gmail.com>
* remove kim's custom go compiler changes
Signed-off-by: kim <grufwub@gmail.com>
* generate setter and flag-name functions, implement these in codebase
Signed-off-by: kim <grufwub@gmail.com>
* update deps
Signed-off-by: kim <grufwub@gmail.com>
* small change
Signed-off-by: kim <grufwub@gmail.com>
* appease the linter...
Signed-off-by: kim <grufwub@gmail.com>
* move configuration into ConfigState structure, ensure reloading to/from viper settings to keep in sync
Signed-off-by: kim <grufwub@gmail.com>
* lint
Signed-off-by: kim <grufwub@gmail.com>
* update code comments
Signed-off-by: kim <grufwub@gmail.com>
* fix merge issue
Signed-off-by: kim <grufwub@gmail.com>
* fix merge issue
Signed-off-by: kim <grufwub@gmail.com>
* improved version string (removes time + go version)
Signed-off-by: kim <grufwub@gmail.com>
* fix version string build to pass test script + consolidate logic in func
Signed-off-by: kim <grufwub@gmail.com>
* add license text, update config.Defaults comment
Signed-off-by: kim <grufwub@gmail.com>
* add license text to generated config helpers file
Signed-off-by: kim <grufwub@gmail.com>
* defer unlock on config.Set___(), to ensure unlocked on panic
Signed-off-by: kim <grufwub@gmail.com>
* make it more obvious which cmd flags are being attached
Signed-off-by: kim <grufwub@gmail.com>
* add some announces to test models
* start on announce test logic
* test federatingDB.Announce
* change signature of GetRemoteStatus
* remove 'refresh' logic and replace it with refetch
* go fmt
* remove timeline manager from processor test
* make zork created at determinate
* test get account statuses
* test get + serialize zork
* make account keys determinate
* make admin accountCreate time determinate
* test account to as
* init test config before test log
* test status to frontend
* remove daft Within check
* hack around a bit
* use index of slice
* tidy up context keys, add otherInvolvedIRIs
* add ReplyToable interface
* skip block check if we own the requesting domain
* add block check for other involved IRIs
* use cacheable status fetch
* remove unused ContextActivity
* remove unused ContextActivity
* add helper for unique URIs
* check through CCs and clean slice
* add GetAccountIDForStatusURI
* add GetAccountIDForAccountURI
* check blocks on involved account
* add statuses to tests
* add some blocked tests
* go fmt
* extract Tos as well as CCs
* test PostInboxRequestBodyHook
* add some more testActivities
* deduplicate involvedAccountIDs
* go fmt
* use cacheable db functions, remove new functions
* implement custom_emojis api endpoint
* add tests for getting custom emoji out of the database and converting to api emoji
* change sort direction of emoji query
* change logging level and initialize array with known length as per kim's suggestions
* add continue to lessen risk of making a malformed struct during conversion from db to api emojis
* rename + tidy up remote pruning
* fix media attachment account join
see https://bun.uptrace.dev/guide/golang-orm.html#table-relationships
* update logging to new function name
* add get avatars and headers to bun
* add pruneallmeta function
* don't set uncached since we're deleting anyway
* fix totalPruned being in wrong place
* test pruning meta
* go fmt ./...
* rename mediaprune
* add meta pruning to routine mediaprune
* tidy up cleanup job scheduling
* rename adminmediaremoteprune
* update mediacleanup to use renamed prune func
* update swagger docs a little bit
* reuse cancel + context
* fix existing bio text showing as HTML
- updated replaced mentions to include instance
- strips HTML from account source note in Verify handler
- update text formatter to use buffers for string writes
Signed-off-by: kim <grufwub@gmail.com>
* go away linter
Signed-off-by: kim <grufwub@gmail.com>
* change buf reset location, change html mention tags
Signed-off-by: kim <grufwub@gmail.com>
* reduce FindLinks code complexity
Signed-off-by: kim <grufwub@gmail.com>
* fix HTML to text conversion
Signed-off-by: kim <grufwub@gmail.com>
* Update internal/regexes/regexes.go
Co-authored-by: Mina Galić <mina.galic@puppet.com>
* use improved html2text lib with more options
Signed-off-by: kim <grufwub@gmail.com>
* fix to produce actual plaintext from html
Signed-off-by: kim <grufwub@gmail.com>
* fix span tags instead written as space
Signed-off-by: kim <grufwub@gmail.com>
* performance improvements to regex replacements, fix link replace logic for un-html-ing in the future
Signed-off-by: kim <grufwub@gmail.com>
* fix tag/mention replacements to use input string, fix link replace to not include scheme
Signed-off-by: kim <grufwub@gmail.com>
* use matched input string for link replace href text
Signed-off-by: kim <grufwub@gmail.com>
* remove unused code (to appease linter :sobs:)
Signed-off-by: kim <grufwub@gmail.com>
* improve hashtagFinger regex to be more compliant
Signed-off-by: kim <grufwub@gmail.com>
* update breakReplacer to include both unix and windows line endings
Signed-off-by: kim <grufwub@gmail.com>
* add NoteRaw field to Account to store plaintext account bio, add migration for this, set for sensitive accounts
Signed-off-by: kim <grufwub@gmail.com>
* drop unnecessary code
Signed-off-by: kim <grufwub@gmail.com>
* update text package tests to fix logic changes
Signed-off-by: kim <grufwub@gmail.com>
* add raw note content testing to account update and account verify
Signed-off-by: kim <grufwub@gmail.com>
* remove unused modules
Signed-off-by: kim <grufwub@gmail.com>
* fix emoji regex
Signed-off-by: kim <grufwub@gmail.com>
* fix replacement of hashtags
Signed-off-by: kim <grufwub@gmail.com>
* update code comment
Signed-off-by: kim <grufwub@gmail.com>
Co-authored-by: Mina Galić <mina.galic@puppet.com>
* create visibility filter for boostability and allow self-boosting for any visbility but direct messages
* add a followers-only status to local_account_2
* fix typo in comment
* add license header, unwrap errors, be explicit about non-boostable visibility settings to avoid rogue boosting from miscoded clients, use ID compare for checking if self-boosting
* add tests for statusboostable filter
* fix tests that were affected by adding a new status to the test data
* fix the rest of tests affected by adding a status to the textrig data
* remove unnecessary LOWER() db calls
* warn during slow db queries
* use bundb built-in exists function
* add db block test
* update account block query
* add domain block db test
* optimize domain block query
* fix implementing wrong test
* exclude most columns when checking block
* go fmt
* remote more unnecessary use of LOWER()
The `database/sql` package in the Go stdlib has two methods that can run
queries against the database driver: `Query` and `Exec`. When the query
returns rows, such as "SELECT", the package expects the use of `Query`,
and returns `*sql.Rows`. When the query does not return rows, the
package expects the use of `Exec`, which returns `sql.Result`.
This changeset corrects the "media_attachments" migration to using
`ExecContext`, as "ALTER TABLE" does not return rows.
Signed-off-by: Terin Stock <terinjokes@gmail.com>
* bump superseriousbusiness/activity version
* fetch list of follower inbox iris
* add test for getting account statuses
* fix incorrect publicOnly parsing
* update media not null query
* linting with new golangci-lint version
* update go to 1.18
* bump versions in drone.yml
* use new runtime/debug package for version info
* remove Commit build flag from goreleaser
* remove mock commit + version from build script
* go fmt
* add dummy version env flag to test container
* install git in golang container for testing
* only set versionString if Version is defined
* remove mention util function from db
* add ParseMentionFunc to gtsmodel
* add parseMentionFunc to processor
* refactor search to simplify it a bit
* add parseMentionFunc to account
* add parseMentionFunc to status
* some renaming for clarity
* test dereference of unknown mentioned account
* add accountAction to the admin API
* model admin account action
* add admin account action to the processor
* add migration for new AdminAccountActions table
* fix accounts admin path
* Update swagger docs
* create first index on notifications
* tidy up + add tests
* log queries for trace, ops for debug
* index commonly used fields
* rearrange query
* add a few more indexes
* remove schema-breaking index
(add this back in later)
* re-add cleanup query index
* Add whereNotEmptyAndNotNull
* Add GetRemoteOlderThanDays
* Add GetRemoteOlderThanDays
* Add PruneRemote to Manager interface
* Start implementing PruneRemote
* add new attachment + status to tests
* fix up and test GetRemoteOlderThan
* fix bad import
* PruneRemote: return number pruned
* add Cached column to mediaattachment
* update + test pruneRemote
* update mediaTest
* use Cached column
* upstep bun to latest version
* embed structs in mediaAttachment
* migrate mediaAttachment to new format
* don't default cached to true
* select only remote media
* update db dependencies
* step bun back to last working version
* update pruneRemote to use Cached field
* fix storage path of test attachments
* add recache logic to manager
* fix trimmed aspect ratio
* test prune and recache
* return errwithcode
* tidy up different paths for emoji vs attachment
* fix incorrect thumbnail type being stored
* expose TransportController to media processor
* implement tee-ing recached content
* add thoughts of dog to test fedi attachments
* test get remote files
* add comment on PruneRemote
* add postData cleanup to recache
* test thumbnail fetching
* add incredible diagram
* go mod tidy
* buffer pipes for recache streaming
* test for client stops reading after 1kb
* add media-remote-cache-days to config
* add cron package
* wrap logrus so it's available to cron
* start and stop cron jobs gracefully
* add ensureUserIsAuthorizedOrRedirect to /oauth/authorize
* adding authorization (email confirm, account approve, etc) to TokenCheck
* revert un-needed changes to signin.go
* oops what happened here
* error css
* add account.SuspendedAt check
* remove redundant checks from oauth util Authed function
* wip tests
* tests passing
* stop stripping useful information from ErrAlreadyExists
* that feeling of scraping the dryer LINT off the screen
* oops I didn't mean to get rid of this NewTestRouter function
* make tests work with recorder
* re-add ConfigureTemplatesWithGin to handle template path err
Co-authored-by: tsmethurst <tobi.smethurst@protonmail.com>
* Don't use the system 'postgres' database by default
* Use postgres adapter defaults
The pgx code actually goes to great lengths to make postgres connections
Just Work(tm) out of the box, including supporting `~/.pg_service.conf`,
SSL certificates, UNIX sockets if it can find a socket at a common path,
and falling back to TCP to localhost if not.
(On Windows, it won't try to use UNIX sockets, but will read credentials
from %appdata% as is standard over there.)
By applying our flags as overrides only when they're specified, database
connections should Just Work(tm) anywhere `psql gotosocial` does.
* start pulling out + replacing urfave and config
* replace many many instances of config
* move more stuff => viper
* properly remove urfave
* move some flags to root command
* add testrig commands to root
* alias config file keys
* start adding cli parsing tests
* reorder viper init
* remove config path alias
* fmt
* change config file keys to non-nested
* we're more or less in business now
* tidy up the common func
* go fmt
* get tests passing again
* add note about the cliparsing tests
* reorganize
* update docs with changes
* structure cmd dir better
* rename + move some files around
* fix dangling comma
* update other tests
* set test status to followers_only
* add test dm
* fix mentioned accounts not being added to relevantAccounts
* add some visibility tests for statuses
* add smtp configuration
* add email confirm + reset templates
* add email sender to testrig
* flesh out the email sender interface
* go fmt
* golint
* update from field with more clarity
* tidy up the email formatting
* fix tests
* add email sender to processor
* tidy client api processing a bit
* further tidying in fromClientAPI
* pin new account to user
* send msg to processor on new account creation
* generate confirm email uri
* remove emailer from account processor again
* add processCreateAccountFromClientAPI
* move emailer accountprocessor => userprocessor
* add email sender to user processor
* SendConfirmEmail function
* add noop email sender
* use noop email sender in tests
* only assemble message if callback is not nil
* use noop email sender if no smtp host is defined
* minify email html before sending
* fix wrong email address
* email confirm test
* fmt
* serve web hndler
* add email confirm handler
* init test log properly on testrig
* log emails that *would* have been sent
* go fmt ./...
* unexport confirm email handler
* updatedAt
* test confirm email function
* don't allow tokens older than 7 days
* change error message a bit
* add basic smtp docs
* add a few more snippets
* typo
* add email sender to outbox tests
* don't use dutch wikipedia link
* don't minify email html
* add statusesvisible convenience function
* add minID + onlyPublic to account statuses get
* move swagger collection stuff to common
* start working on Outbox GETting
* move functions into federationProcessor
* outboxToASCollection
* add statusesvisible convenience function
* add minID + onlyPublic to account statuses get
* move swagger collection stuff to common
* start working on Outbox GETting
* move functions into federationProcessor
* outboxToASCollection
* bit more work on outbox paging
* wrapNoteInCreate function
* test + hook up the processor functions
* don't do prev + next links on empty reply
* test get outbox through api
* don't fail on no status entries
* add outbox implementation doc
* typo
* reference logrus' global logger instead of passing and storing a logger reference everywhere
* always directly use global logrus logger instead of referencing an instance
* test suites should also directly use the global logrus logger
* rename gin logging function to clarify that it's middleware
* correct comments which erroneously referenced removed logger parameter
* setting log level for tests now uses logrus' exported type instead of the string value, to guarantee error isn't possible
* tidy up streaming
* cut down code duplication
* test get followers/following
* test streaming processor
* fix some test models
* add TimeMustParse
* fix uri / url typo
* make trace logging less verbose
* make logging more consistent
* disable quote on logging
* remove context.Background
* remove many extraneous mastodon references
* regenerate swagger
* don't log query on no rows result
* log latency first for easier reading
* only fetch status parents / children if explicity requested when dereferencing
Signed-off-by: kim (grufwub) <grufwub@gmail.com>
* Remove recursive DB GetStatus logic, don't fetch parent unless requested
Signed-off-by: kim (grufwub) <grufwub@gmail.com>
* StatusCache copies status so there are no thread-safety issues with modified status objects
Signed-off-by: kim (grufwub) <grufwub@gmail.com>
* remove sqlite test files
Signed-off-by: kim (grufwub) <grufwub@gmail.com>
* fix bugs introduced by previous commit
Signed-off-by: kim (grufwub) <grufwub@gmail.com>
* fix not continue on error in loop
Signed-off-by: kim (grufwub) <grufwub@gmail.com>
* use our own RunInTx implementation (possible fix for nested tx error)
Signed-off-by: kim (grufwub) <grufwub@gmail.com>
* fix cast statement to work with SQLite
Signed-off-by: kim (grufwub) <grufwub@gmail.com>
* be less strict about valid status in cache
Signed-off-by: kim (grufwub) <grufwub@gmail.com>
* add cache=shared ALWAYS for SQLite db instances
Signed-off-by: kim (grufwub) <grufwub@gmail.com>
* Fix EnrichRemoteAccount when updating account fails
Signed-off-by: kim (grufwub) <grufwub@gmail.com>
* add nolint tag
Signed-off-by: kim (grufwub) <grufwub@gmail.com>
* ensure file: prefixes the filename in sqlite addr
Signed-off-by: kim (grufwub) <grufwub@gmail.com>
* add an account cache, add status author account from db
Signed-off-by: kim (grufwub) <grufwub@gmail.com>
* Fix incompatible SQLite query
Signed-off-by: kim (grufwub) <grufwub@gmail.com>
* *actually* use the new getAccount() function in accountsDB
Signed-off-by: kim (grufwub) <grufwub@gmail.com>
* update cache tests to use test suite
Signed-off-by: kim (grufwub) <grufwub@gmail.com>
* add RelationshipTestSuite, add tests for methods with changed SQL
Signed-off-by: kim (grufwub) <grufwub@gmail.com>
* Add SQLite support, fix un-thread-safe DB caches, small performance fixes
Signed-off-by: kim (grufwub) <grufwub@gmail.com>
* add SQLite licenses to README
Signed-off-by: kim (grufwub) <grufwub@gmail.com>
* appease the linter, and fix my dumbass-ery
Signed-off-by: kim (grufwub) <grufwub@gmail.com>
* make requested changes
Signed-off-by: kim (grufwub) <grufwub@gmail.com>
* add back comment
Signed-off-by: kim (grufwub) <grufwub@gmail.com>
* rework mention creation a bit
* rework mention creation a bit
* tidy up status dereferencing
* start adding tests for dereferencing
* fixups
* fix
* review changes
