Commit graph

2145 commits

Author SHA1 Message Date
kim 9a291dea84
[performance] add caching of status fave, boost of, in reply to ID lists (#2060) 2023-08-04 12:28:33 +01:00
kim 00adf18c24
[feature] simpler cache size configuration (#2051)
* add automatic cache max size generation based on ratios of a singular fixed memory target

Signed-off-by: kim <grufwub@gmail.com>

* remove now-unused cache max-size config variables

Signed-off-by: kim <grufwub@gmail.com>

* slight ratio tweak

Signed-off-by: kim <grufwub@gmail.com>

* remove unused visibility config var

Signed-off-by: kim <grufwub@gmail.com>

* add secret little ratio config trick

Signed-off-by: kim <grufwub@gmail.com>

* fixed a word

Signed-off-by: kim <grufwub@gmail.com>

* update cache library to remove use of TTL in result caches + slice cache

Signed-off-by: kim <grufwub@gmail.com>

* update other cache usages to use correct interface

Signed-off-by: kim <grufwub@gmail.com>

* update example config to explain the cache memory target

Signed-off-by: kim <grufwub@gmail.com>

* update env parsing test with new config values

Signed-off-by: kim <grufwub@gmail.com>

* do some ratio twiddling

Signed-off-by: kim <grufwub@gmail.com>

* add missing header

* update envparsing with latest defaults

Signed-off-by: kim <grufwub@gmail.com>

* update size calculations to take into account result cache, simple cache and extra map overheads

Signed-off-by: kim <grufwub@gmail.com>

* tweak the ratios some more

Signed-off-by: kim <grufwub@gmail.com>

* more nan rampaging

Signed-off-by: kim <grufwub@gmail.com>

* fix envparsing script

Signed-off-by: kim <grufwub@gmail.com>

* update cache library, add sweep function to keep caches trim

Signed-off-by: kim <grufwub@gmail.com>

* sweep caches once a minute

Signed-off-by: kim <grufwub@gmail.com>

* add a regular job to sweep caches and keep under 80% utilisation

Signed-off-by: kim <grufwub@gmail.com>

* remove dead code

Signed-off-by: kim <grufwub@gmail.com>

* add new size library used to libraries section of readme

Signed-off-by: kim <grufwub@gmail.com>

* add better explanations for the mem-ratio numbers

Signed-off-by: kim <grufwub@gmail.com>

* update go-cache

Signed-off-by: kim <grufwub@gmail.com>

* library version bump

Signed-off-by: kim <grufwub@gmail.com>

* update cache.result{} size model estimation

Signed-off-by: kim <grufwub@gmail.com>

---------

Signed-off-by: kim <grufwub@gmail.com>
2023-08-03 11:34:35 +02:00
tobi e8a20f587c
[bugfix] Rework MultiError to wrap + unwrap errors properly (#2057)
* rework multierror a bit

* test multierror
2023-08-02 17:21:46 +02:00
kim 2cee8f2dd8
[bugfix] fix slow accounts / statuses using emojis lookups (#2056)
* update DeleteEmoji to use faster relational tables for status / account finding

Signed-off-by: kim <grufwub@gmail.com>

* update Get{Accounts,Statuses}UsingEmoji() to also use relational tables

Signed-off-by: kim <grufwub@gmail.com>

* remove the now unneeded tags relation from newStatusQ()

Signed-off-by: kim <grufwub@gmail.com>

* fix table names

Signed-off-by: kim <grufwub@gmail.com>

* fix account and status selects using emojis

Signed-off-by: kim <grufwub@gmail.com>

---------

Signed-off-by: kim <grufwub@gmail.com>
2023-08-02 16:11:23 +02:00
kim 24516b84c2
[bugfix] handle HEAD requests more elegantly (#2055)
Signed-off-by: kim <grufwub@gmail.com>
2023-08-02 10:28:20 +02:00
tobi cec29e2a8d
[bugfix] Allow instance accounts to be shown in search results in certain circumstances (#2053) 2023-08-02 08:31:09 +01:00
tobi 2be83fdca5
[feature] Allow users to skip http client tls verification for testing purposes (with appropriately loud warnings) (#2052) 2023-08-01 18:50:17 +01:00
tobi 9bd03e122e
[feature] Set timezone in Docker using TZ env variable (#2050) 2023-08-01 14:47:53 +02:00
tobi 70929c1691
Revert "[feature] Configurable timezone in containers (#2046)" (#2048)
This reverts commit 96dd0e75f2.
2023-08-01 12:31:50 +02:00
Charles Chin 96dd0e75f2
[feature] Configurable timezone in containers (#2046) 2023-08-01 12:09:46 +02:00
tobi 2796a2e82f
[feature] Hashtag federation (in/out), hashtag client API endpoints (#2032)
* update go-fed

* do the things

* remove unused columns from tags

* update to latest lingo from main

* further tag shenanigans

* serve stub page at tag endpoint

* we did it lads

* tests, oh tests, ohhh tests, oh tests (doo doo doo doo)

* swagger docs

* document hashtag usage + federation

* instanceGet

* don't bother parsing tag href

* rename whereStartsWith -> whereStartsLike

* remove GetOrCreateTag

* dont cache status tag timelineability
2023-07-31 15:47:35 +02:00
kim ed2477ebea
[performance] cache follow, follow request and block ID lists (#2027) 2023-07-31 11:25:29 +01:00
tobi de148e9f9f
[bugfix/frontend] Decode URI component domain before showing on frontend (#2043) 2023-07-31 12:18:27 +02:00
dependabot[bot] a5a80b4bbe
[chore]: Bump github.com/abema/go-mp4 from 0.10.2-0.20230727031202-a1a707db6ecd to 0.11.0 (#2042)
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-07-31 09:47:56 +01:00
dependabot[bot] 9ed9d96597
[chore]: Bump github.com/minio/minio-go/v7 from 7.0.60 to 7.0.61 (#2041)
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-07-31 09:47:25 +01:00
Vyr Cossont b874e9251e
[feature] Implement markers API (#1989)
* Implement markers API

Fixes #1856

* Correct import grouping in markers files

* Regenerate Swagger for markers API

* Shorten names for readability

* Cache markers for 6 hours

* Update DB ref

* Update envparsing.sh
2023-07-29 12:49:14 +02:00
tobi cf4bd700fb
[chore] Update activity dependency (#2031) 2023-07-27 20:54:30 +02:00
tobi e747c14685
[chore] fix merge issue (#2029) 2023-07-27 12:54:27 +02:00
Dominik Süß 98c2b8ff7e
[chore] update go-mp4 to latest commit (#2028)
This adds support for probing mp4 files with a co64 box instead of an stco box,
which is the case for videos recorded on newer android devices.
2023-07-27 10:31:36 +02:00
Vyr Cossont 22ac4607a1
[feature] Support setting private notes on accounts (#1982)
* Support setting private notes on accounts

* Reformat comment whitespace

* Add missing license headers

* Use apiutil.ParseID

* Rename Note model and cache to AccountNote

* Update golden cache config in test/envparsing.sh

* Rename gtsmodel/note.go to gtsmodel/accountnote.go

* Update AccountNote uniqueness constraint name

Now has same prefix as other indexes on this table.

---------

Co-authored-by: tobi <31960611+tsmethurst@users.noreply.github.com>
2023-07-27 10:30:39 +02:00
kim 5f3e095717
[performance] retry db queries on busy errors (#2025)
* catch SQLITE_BUSY errors, wrap bun.DB to use our own busy retrier, remove unnecessary db.Error type

Signed-off-by: kim <grufwub@gmail.com>

* remove dead code

Signed-off-by: kim <grufwub@gmail.com>

* remove more dead code, add missing error arguments

Signed-off-by: kim <grufwub@gmail.com>

* update sqlite to use maxOpenConns()

Signed-off-by: kim <grufwub@gmail.com>

* add uncommitted changes

Signed-off-by: kim <grufwub@gmail.com>

* use direct calls-through for the ConnIface to make sure we don't double query hook

Signed-off-by: kim <grufwub@gmail.com>

* expose underlying bun.DB better

Signed-off-by: kim <grufwub@gmail.com>

* retry on the correct busy error

Signed-off-by: kim <grufwub@gmail.com>

* use longer possible maxRetries for db retry-backoff

Signed-off-by: kim <grufwub@gmail.com>

* remove the note regarding max-open-conns only applying to postgres

Signed-off-by: kim <grufwub@gmail.com>

* improved code commenting

Signed-off-by: kim <grufwub@gmail.com>

* remove unnecessary infof call (just use info)

Signed-off-by: kim <grufwub@gmail.com>

* rename DBConn to WrappedDB to better follow sql package name conventions

Signed-off-by: kim <grufwub@gmail.com>

* update test error string checks

Signed-off-by: kim <grufwub@gmail.com>

* shush linter

Signed-off-by: kim <grufwub@gmail.com>

* update backoff logic to be more transparent

Signed-off-by: kim <grufwub@gmail.com>

---------

Signed-off-by: kim <grufwub@gmail.com>
2023-07-25 10:34:05 +02:00
kim 9eff0d46e4
[feature/performance] support uncaching remote emoji + scheduled cleanup functions (#1987) 2023-07-24 13:14:13 +01:00
dependabot[bot] 81fe59dadc
[chore]: Bump github.com/microcosm-cc/bluemonday from 1.0.24 to 1.0.25 (#2021) 2023-07-24 10:24:03 +00:00
dependabot[bot] b05f6c8f56
[chore]: Bump github.com/yuin/goldmark from 1.5.4 to 1.5.5 (#2023) 2023-07-24 10:21:14 +00:00
dependabot[bot] c9f3539b5a
[chore]: Bump codeberg.org/gruf/go-cache/v3 from 3.4.1 to 3.4.3 (#2022) 2023-07-24 10:20:11 +00:00
dependabot[bot] 8b71567f9e
[chore]: Bump github.com/jackc/pgconn from 1.14.0 to 1.14.1 (#2024) 2023-07-24 10:14:16 +00:00
dependabot[bot] b7891bb462
[chore]: Bump go.uber.org/automaxprocs from 1.5.2 to 1.5.3 (#2020) 2023-07-24 10:13:31 +00:00
Daenney 994d5e80d9
[chore] Disable the right webfinger test (#2018)
We don't talk about 8868889 / #1983.
2023-07-23 16:33:16 +02:00
tobi 5a29a031ad
[chore] Admin CLI + new account creation refactoring (#2008)
* set maxPasswordLength to 72 bytes, rename validate function

* refactor NewSignup

* refactor admin account CLI commands

* refactor oidc create user

* refactor processor create

* tweak password change, check old != new password
2023-07-23 12:33:17 +02:00
Daenney f8f0312042
[feature] Report Masto version in /api/v1/instance (#1977) 2023-07-21 18:49:13 +01:00
dependabot[bot] 43a2753ef0
[chore]: Bump modernc.org/sqlite from 1.23.1 to 1.24.0 (#1993)
Bumps [modernc.org/sqlite](https://gitlab.com/cznic/sqlite) from 1.23.1 to 1.24.0.
- [Commits](https://gitlab.com/cznic/sqlite/compare/v1.23.1...v1.24.0)

---
updated-dependencies:
- dependency-name: modernc.org/sqlite
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-07-21 16:43:56 +02:00
Daenney da8282081c
[feature] Lower remote media cache config duration (#2007)
The old default of 30d can lead to a lot of media getting cached and
significant disk usage, even on small or single person instances. A lot
of deployments decrease this value, to 15 or even less. This is less of
an issue when using object storage, but for local storage which is the
more popular deployment option running out of disk space is unpleasant.

With GoToSocial's aim to fit in small places, this changes the default
to a much more conservative 7 days. In all likelihood people aren't
scrolling that far back in their timeline so this change shouldn't
result in any issue. Existing deployments will only be affected by
this change if the admin hasn't already configured this value, or didn't
bootstrap from the example configuration.
2023-07-21 16:22:59 +02:00
dependabot[bot] fa57c699fe
[chore]: Bump github.com/jackc/pgx/v5 from 5.4.1 to 5.4.2 (#1991) 2023-07-21 14:22:20 +00:00
dependabot[bot] 83139989b5
[chore]: Bump github.com/minio/minio-go/v7 from 7.0.59 to 7.0.60 (#1992) 2023-07-21 14:22:17 +00:00
dependabot[bot] d53449ccb4
[chore]: Bump github.com/go-playground/form/v4 from 4.2.0 to 4.2.1 (#1994) 2023-07-21 14:22:13 +00:00
dependabot[bot] 8ac3a3f688
Bump word-wrap from 1.2.3 to 1.2.4 in /web/source (#2001) 2023-07-21 14:21:50 +00:00
tobi 89ee9d5004
[bugfix] Return all accounts when list accounts limit <= 0 (#2014) 2023-07-21 13:56:38 +01:00
tobi d6fa74e5dc
[bugfix] Change maximumPasswordLength to 72 bytes (#2012) 2023-07-21 10:29:18 +01:00
mirabilos 95e2024c2a
[docs] Apache setup for Caching assets and media (#2005) (#2005)
Also change the nginx fileserver expiry, after discussion, to
one week, to match.
2023-07-20 18:48:52 +02:00
kim f4319740ab
[bugfix] more robust list timeline invalidation (#1995) 2023-07-18 09:43:17 +01:00
tobi 346ecabd07
[bugfix] Add missing continue statement in prepareXBetweenIDs (#1996) 2023-07-18 09:39:16 +01:00
tobi 12b6cdcd8c
[bugfix] Set Vary header correctly on cache-control (#1988)
* [bugfix] Set Vary header correctly on cache-control

* Prefer activitypub types on AP endpoints

* use immutable on file server, vary by range

* vary auth on Accept
2023-07-13 21:27:25 +02:00
Daenney 88688899aa
[chore] Skip webfinger test on CI (#1983) 2023-07-12 16:09:49 +01:00
tobi 1951e6c840
[bugfix] Update account Update logic (#1984) 2023-07-12 12:20:15 +01:00
firescry 8d92b2479f
[bugfix] Align default values in the configuration file with the code (#1971)
* Set default value of SMTPFrom to empty string

This parameter should contain proper e-mail address (to be provided by user during configuration).

* Update default values in example/config.yaml

Default values and related comments in example/config.yaml are aligned
with values defined in internal/config/defaults.go.
Small improvements to foramting of config.yaml file.

* Add default value for AdvancedThrottlingRetryAfter to internal/config/defaults.go

AdvancedThrottlingRetryAfter was introduced in 70739d3 (superseriousbusiness/gotosocial#1466).

* Update config.yaml snippets in documentation
2023-07-12 10:28:41 +02:00
dependabot[bot] f5d6503aad
Bump semver from 6.3.0 to 6.3.1 in /web/source (#1981)
Bumps [semver](https://github.com/npm/node-semver) from 6.3.0 to 6.3.1.
- [Release notes](https://github.com/npm/node-semver/releases)
- [Changelog](https://github.com/npm/node-semver/blob/v6.3.1/CHANGELOG.md)
- [Commits](https://github.com/npm/node-semver/compare/v6.3.0...v6.3.1)

---
updated-dependencies:
- dependency-name: semver
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-07-12 10:27:31 +02:00
MaeIsBad 0d267fd598
[bugfix] Properly handle range > content-length (#1979)
This makes the serveFileRange function return the entire file
if suffix-range is larger than content-length in compliance with RFC9110

Co-authored-by: mae <git@badat.dev>
2023-07-12 09:51:51 +02:00
tobi ca5492b65f
[bugfix] Tidy up rss feed serving; don't error on empty feed (#1970)
* [bugfix] Tidy up rss feed serving; don't error on empty feed

* fall back to account creation time as rss feed update time

* return feed early when account has no eligible statuses
2023-07-10 17:05:59 +02:00
dependabot[bot] a29b5affc8
[chore]: Bump golang.org/x/oauth2 from 0.9.0 to 0.10.0 (#1975)
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-07-10 12:58:06 +01:00
Daenney f0dad439f6
[feature] Use Read/Write lock for config (#1969) 2023-07-10 12:56:14 +01:00