Merge 9acf9ddc75 into 095663f5cc

* replicate issue

* update go-structr to v0.8.10 with internal linked-list fix, small tweaks to caching of interaction requests

* remove debug function

---------

Co-authored-by: tobi <tobi.smethurst@protonmail.com>

docs/advanced/index.md

@@ -16,3 +16,4 @@ We consider these topics advanced because applying them incorrectly does have th
 * [Tracing](tracing.md)
 * [Metrics](metrics.md)
 * [Replicating SQLite](replicating-sqlite.md)
+* [SQLite on networked storage](sqlite-networked-storage.md)

docs/advanced/sqlite-networked-storage.md

@@ -0,0 +1,35 @@
+# SQLite on networked storage
+
+SQLite's operating model assumes the database and the processes or applications using it are colocated on the same host. When running the database in WAL-mode, which is GoToSocial's default, it relies on shared memory between processes to ensure the integrity of your database.
+
+!!! quote
+    All processes using a database must be on the same host computer; WAL does not work over a network filesystem. This is because WAL requires all processes to share a small amount of memory and processes on separate host machines obviously cannot share memory with each other. 
+
+    — SQLite.org [Write-Ahead Logging](https://www.sqlite.org/wal.html)
+
+This also means that any other processes accessing the database need to run in the same namespace or container context.
+
+It is in theory possible to run SQLite over Samba, NFS, iSCSI or other forms of filesystems accessed over the network. But it is neither recommended nor supported by the SQLite maintainers, irrespective of whether you're running with write-ahead logging or not. Doing so puts you at risk of database corruption. There is a long history of networked storage having synchronisation issues in their locking primitives, or implementing them with weaker guarantees than what a local filesystem can provide.
+
+Your cloud provider's external volumes, like Hetzner Cloud Volumes, AWS EBS, GCP Persistent Disk etc. may also cause problems, and add variable latency. This has a tendency to severely degrade SQLite's performance.
+
+If you're going to access your database over the network, it's better to use a database with a client-server architecture. GoToSocial supports Postgres for such use-cases.
+
+For the purpose of having a copy of the SQLite database on durable long-term storage, refer to [SQLite streaming replication](replicating-sqlite.md) instead. Remember that neither replication nor using a networked filesystem are a substitute [for having backups](../admin/backup_and_restore.md).
+
+## Settings
+
+!!! danger "Corrupted database"
+    We do not support running GoToSocial with SQLite on a networked filesystem and we will not be able to help you if you damage your database this way.
+
+Should you really want to take this risk, you'll need to adjust the SQLite [synchronous][sqlite-sync] mode and [journal][sqlite-journal] mode to match the limitations of the filesystem.
+
+[sqlite-sync]: https://www.sqlite.org/pragma.html#pragma_synchronous
+[sqlite-journal]: https://www.sqlite.org/pragma.html#pragma_journal_mode
+
+You'll need to update the following settings:
+
+* `db-sqlite-journal-mode`
+* `db-sqlite-synchronous`
+
+We don't provide any recommendations as this will vary based on the solution you're using. See [this issue](https://github.com/superseriousbusiness/gotosocial/issues/3360#issuecomment-2380332027) for what you could potentially set those values to.

docs/getting_started/index.md

@@ -65,14 +65,16 @@ If you decide to use a VPS instead, you can spin yourself up something cheap wit
 
 [Greenhost](https://greenhost.net) is also great: it has zero CO2 emissions, but is a bit more costly. Their 1GB, 1-cpu VPS works great for a single-user or small instance.
 
+!!! warning "Cloud storage volumes"
+    Not all cloud VPS storage offerings are equal, and just because something claims to be backed by an SSD doesn't mean that it will necessarily be suitable to run a GoToSocial instance.
+    
+    The [performance of Hetzner Cloud Volumes](https://github.com/superseriousbusiness/gotosocial/issues/2471#issuecomment-1891098323) is not guaranteed and seems to have very volatile latency. This will result in your GoToSocial instance performing poorly.
+
 !!! danger "Oracle Free Tier"
     [Oracle Cloud Free Tier](https://www.oracle.com/cloud/free/) servers are not suitable for a GoToSocial deployment if you intend to federate with more than a handful of other instances and users.
     
     GoToSocial admins running on Oracle Cloud Free Tier have reported that their instances become extremely slow or unresponsive during periods of moderate load. This is most likely due to memory or storage latency, which causes even simple database queries to take a long time to run.
 
-!!! danger "Hetzner Cloud Volume"
-    The [performance of Hetzner Cloud Volumes](https://github.com/superseriousbusiness/gotosocial/issues/2471#issuecomment-1891098323) is not guaranteed and seems to have very volatile latency. You're going to have a bad time running your database on those, with extremely poor query performance for even the most basic operations. Before filing performance issues against GoToSocial, make sure the problems reproduce with local storage.
-
 ### Distribution system requirements
 
 Please make sure to check on your distribution system requirments, especially memory. Many distributions have baseline requirements and running them on a system that doesn't meet them will cause problems without further tuning and tweaking on your part.
@@ -99,13 +101,15 @@ GoToSocial supports both SQLite and Postgres as database drivers. Though it is p
 
 SQLite is the default driver and it's been shown to work brilliantly for instances in the range of 1-30 users (or maybe more).
 
+!!! danger "SQLite on networked storage"
+    Don't put your SQLite database on remote storage, whether that's NFS/Samba, iSCSI volumes, things like Ceph/Gluster or your cloud provider's network volume storage solution.
+
+    See [SQLite on networked storage](../advanced/sqlite-networked-storage.md) for further information.
+
 If you're planning on hosting more people than this on an instance, you may wish to use Postgres instead, as it offers the possibility of database clustering and redundancy, at the cost of some complexity.
 
 Regardless of which database driver you choose, for proper performance they should be run on fast storage that operates with low and stable latency. It is possible to run databases on network attached storage, but this adds variable latency and network congestion to the mix, as well as potential I/O contention on the origin storage.
 
-!!! danger "Cloud Storage Volumes"
-    Not all cloud VPS storage offerings are equal, and just because something claims to be backed by an SSD doesn't mean that it will necessarily be suitable to run a GoToSocial instance on. Please see the [Server/VPS section](#vps) section below.
-
 !!! tip
     Please [backup your database](../admin/backup_and_restore.md). The database contains encryption keys for the instance and any user accounts. You won't be able to federate again from the same domain if you lose these keys!
 

go.mod

@@ -22,7 +22,7 @@ require (
 	codeberg.org/gruf/go-runners v1.6.3
 	codeberg.org/gruf/go-sched v1.2.4
 	codeberg.org/gruf/go-storage v0.2.0
-	codeberg.org/gruf/go-structr v0.8.9
+	codeberg.org/gruf/go-structr v0.8.10
 	codeberg.org/superseriousbusiness/exif-terminator v0.9.0
 	github.com/DmitriyVTitov/size v1.5.0
 	github.com/KimMachineGun/automemlimit v0.6.1

go.sum

@@ -72,8 +72,8 @@ codeberg.org/gruf/go-sched v1.2.4 h1:ddBB9o0D/2oU8NbQ0ldN5aWxogpXPRBATWi58+p++Hw
 codeberg.org/gruf/go-sched v1.2.4/go.mod h1:wad6l+OcYGWMA2TzNLMmLObsrbBDxdJfEy5WvTgBjNk=
 codeberg.org/gruf/go-storage v0.2.0 h1:mKj3Lx6AavEkuXXtxqPhdq+akW9YwrnP16yQBF7K5ZI=
 codeberg.org/gruf/go-storage v0.2.0/go.mod h1:o3GzMDE5QNUaRnm/daUzFqvuAaC4utlgXDXYO79sWKU=
-codeberg.org/gruf/go-structr v0.8.9 h1:OyiSspWYCeJOm356fFPd+bDRumPrard2VAUXAPqZiJ0=
+codeberg.org/gruf/go-structr v0.8.10 h1:uSapW97/StRnYEhCtycaM0isCsEMYC+tx/knYr6SiVo=
-codeberg.org/gruf/go-structr v0.8.9/go.mod h1:zkoXVrAnKosh8VFAsbP/Hhs8FmLBjbVVy5w/Ngm8ApM=
+codeberg.org/gruf/go-structr v0.8.10/go.mod h1:zkoXVrAnKosh8VFAsbP/Hhs8FmLBjbVVy5w/Ngm8ApM=
 codeberg.org/superseriousbusiness/exif-terminator v0.9.0 h1:/EfyGI6HIrbkhFwgXGSjZ9o1kr/+k8v4mKdfXTH02Go=
 codeberg.org/superseriousbusiness/exif-terminator v0.9.0/go.mod h1:gCWKduudUWFzsnixoMzu0FYVdxHWG+AbXnZ50DqxsUE=
 dmitri.shuralyov.com/gpu/mtl v0.0.0-20190408044501-666a987793e9/go.mod h1:H6x//7gZCb22OMCxBHrMx7a5I7Hp++hsVxbQ4BYO7hU=

internal/api/client/accounts/statuses_test.go

@@ -73,7 +73,7 @@ func (suite *AccountStatusesTestSuite) TestGetStatusesPublicOnly() {
 		suite.Equal(apimodel.VisibilityPublic, s.Visibility)
 	}
 
-	suite.Equal(`<http://localhost:8080/api/v1/accounts/01F8MH17FWEB39HZJ76B6VXSKF/statuses?limit=20&max_id=01F8MH75CBF9JFX4ZAD54N0W0R&exclude_replies=false&exclude_reblogs=false&pinned=false&only_media=false&only_public=true>; rel="next", <http://localhost:8080/api/v1/accounts/01F8MH17FWEB39HZJ76B6VXSKF/statuses?limit=20&min_id=01G36SF3V6Y6V5BF9P4R7PQG7G&exclude_replies=false&exclude_reblogs=false&pinned=false&only_media=false&only_public=true>; rel="prev"`, result.Header.Get("link"))
+	suite.Equal(`<http://localhost:8080/api/v1/accounts/01F8MH17FWEB39HZJ76B6VXSKF/statuses?limit=20&max_id=01F8MH75CBF9JFX4ZAD54N0W0R&exclude_replies=false&exclude_reblogs=false&pinned=false&only_media=false&only_public=true>; rel="next", <http://localhost:8080/api/v1/accounts/01F8MH17FWEB39HZJ76B6VXSKF/statuses?limit=20&min_id=01J5QVB9VC76NPPRQ207GG4DRZ&exclude_replies=false&exclude_reblogs=false&pinned=false&only_media=false&only_public=true>; rel="prev"`, result.Header.Get("link"))
 }
 
 func (suite *AccountStatusesTestSuite) TestGetStatusesPublicOnlyMediaOnly() {

internal/api/client/statuses/statusboost_test.go

@@ -591,7 +591,7 @@ func (suite *StatusBoostTestSuite) TestPostBoostImplicitAccept() {
     "text": "Hi @1happyturtle, can I reply?",
     "uri": "http://localhost:8080/some/determinate/url",
     "url": "http://localhost:8080/some/determinate/url",
-    "visibility": "unlisted"
+    "visibility": "public"
   },
   "reblogged": true,
   "reblogs_count": 0,
@@ -601,7 +601,7 @@ func (suite *StatusBoostTestSuite) TestPostBoostImplicitAccept() {
   "tags": [],
   "uri": "http://localhost:8080/some/determinate/url",
   "url": "http://localhost:8080/some/determinate/url",
-  "visibility": "unlisted"
+  "visibility": "public"
 }`, out)
 
 	// Target status should no

internal/api/client/statuses/statusfave_test.go

@@ -27,6 +27,7 @@
 	"github.com/gin-gonic/gin"
 	"github.com/stretchr/testify/suite"
 	"github.com/superseriousbusiness/gotosocial/internal/api/client/statuses"
+	"github.com/superseriousbusiness/gotosocial/internal/filter/visibility"
 	"github.com/superseriousbusiness/gotosocial/internal/gtsmodel"
 
 	apiutil "github.com/superseriousbusiness/gotosocial/internal/api/util"
@@ -185,13 +186,24 @@ func (suite *StatusFaveTestSuite) TestPostUnfaveable() {
 // Fave a status that's pending approval by us.
 func (suite *StatusFaveTestSuite) TestPostFaveImplicitAccept() {
 	var (
+		ctx          = context.Background()
 		targetStatus = suite.testStatuses["admin_account_status_5"]
 		app          = suite.testApplications["application_1"]
 		token        = suite.testTokens["local_account_2"]
 		user         = suite.testUsers["local_account_2"]
 		account      = suite.testAccounts["local_account_2"]
+		visFilter    = visibility.NewFilter(&suite.state)
 	)
 
+	// Check visibility of status to public before posting fave.
+	visible, err := visFilter.StatusVisible(ctx, nil, targetStatus)
+	if err != nil {
+		suite.FailNow(err.Error())
+	}
+	if visible {
+		suite.FailNow("status should not be visible yet")
+	}
+
 	out, recorder := suite.postStatusFave(
 		targetStatus.ID,
 		app,
@@ -268,30 +280,40 @@ func (suite *StatusFaveTestSuite) TestPostFaveImplicitAccept() {
   "text": "Hi @1happyturtle, can I reply?",
   "uri": "http://localhost:8080/some/determinate/url",
   "url": "http://localhost:8080/some/determinate/url",
-  "visibility": "unlisted"
+  "visibility": "public"
 }`, out)
 
 	// Target status should no
 	// longer be pending approval.
 	dbStatus, err := suite.state.DB.GetStatusByID(
-		context.Background(),
+		ctx,
 		targetStatus.ID,
 	)
 	if err != nil {
 		suite.FailNow(err.Error())
 	}
 	suite.False(*dbStatus.PendingApproval)
+	suite.NotEmpty(dbStatus.ApprovedByURI)
 
 	// There should be an Accept
 	// stored for the target status.
 	intReq, err := suite.state.DB.GetInteractionRequestByInteractionURI(
-		context.Background(), targetStatus.URI,
+		ctx, targetStatus.URI,
 	)
 	if err != nil {
 		suite.FailNow(err.Error())
 	}
 	suite.NotZero(intReq.AcceptedAt)
 	suite.NotEmpty(intReq.URI)
+
+	// Check visibility of status to public after posting fave.
+	visible, err = visFilter.StatusVisible(ctx, nil, dbStatus)
+	if err != nil {
+		suite.FailNow(err.Error())
+	}
+	if !visible {
+		suite.FailNow("status should be visible")
+	}
 }
 
 func TestStatusFaveTestSuite(t *testing.T) {

internal/cache/util.go

@@ -18,7 +18,6 @@
 package cache
 
 import (
-	"database/sql"
 	"errors"
 	"time"
 
@@ -42,7 +41,6 @@ func ignoreErrors(err error) bool {
 		// (until invalidation).
 		db.ErrNoEntries,
 		db.ErrAlreadyExists,
-		sql.ErrNoRows,
 	)
 }
 

internal/cache/visibility.go

@@ -49,7 +49,13 @@ func (c *Caches) initVisibility() {
 			{Fields: "Type,RequesterID,ItemID"},
 		},
 		MaxSize: cap,
-		IgnoreErr: ignoreErrors,
+		IgnoreErr: func(err error) bool {
+			// don't cache any errors,
+			// it gets a little too tricky
+			// otherwise with ensuring
+			// errors are cleared out
+			return true
+		},
 		Copy: copyF,
 	})
 }

internal/db/bundb/interaction.go

@@ -26,8 +26,10 @@
 	"github.com/superseriousbusiness/gotosocial/internal/gtscontext"
 	"github.com/superseriousbusiness/gotosocial/internal/gtserror"
 	"github.com/superseriousbusiness/gotosocial/internal/gtsmodel"
+	"github.com/superseriousbusiness/gotosocial/internal/log"
 	"github.com/superseriousbusiness/gotosocial/internal/paging"
 	"github.com/superseriousbusiness/gotosocial/internal/state"
+	"github.com/superseriousbusiness/gotosocial/internal/util"
 	"github.com/uptrace/bun"
 )
 
@@ -84,6 +86,53 @@ func(request *gtsmodel.InteractionRequest) error {
 	)
 }
 
+func (i *interactionDB) GetInteractionRequestsByIDs(ctx context.Context, ids []string) ([]*gtsmodel.InteractionRequest, error) {
+	// Load all interaction request IDs via cache loader callbacks.
+	requests, err := i.state.Caches.DB.InteractionRequest.LoadIDs("ID",
+		ids,
+		func(uncached []string) ([]*gtsmodel.InteractionRequest, error) {
+			// Preallocate expected length of uncached interaction requests.
+			requests := make([]*gtsmodel.InteractionRequest, 0, len(uncached))
+
+			// Perform database query scanning
+			// the remaining (uncached) IDs.
+			if err := i.db.NewSelect().
+				Model(&requests).
+				Where("? IN (?)", bun.Ident("id"), bun.In(uncached)).
+				Scan(ctx); err != nil {
+				return nil, err
+			}
+
+			return requests, nil
+		},
+	)
+	if err != nil {
+		return nil, err
+	}
+
+	// Reorder the requests by their
+	// IDs to ensure in correct order.
+	getID := func(r *gtsmodel.InteractionRequest) string { return r.ID }
+	util.OrderBy(requests, ids, getID)
+
+	if gtscontext.Barebones(ctx) {
+		// no need to fully populate.
+		return requests, nil
+	}
+
+	// Populate all loaded interaction requests, removing those we
+	// fail to populate (removes needing so many nil checks everywhere).
+	requests = slices.DeleteFunc(requests, func(request *gtsmodel.InteractionRequest) bool {
+		if err := i.PopulateInteractionRequest(ctx, request); err != nil {
+			log.Errorf(ctx, "error populating %s: %v", request.ID, err)
+			return true
+		}
+		return false
+	})
+
+	return requests, nil
+}
+
 func (i *interactionDB) getInteractionRequest(
 	ctx context.Context,
 	lookup string,
@@ -205,15 +254,20 @@ func (i *interactionDB) UpdateInteractionRequest(ctx context.Context, request *g
 }
 
 func (i *interactionDB) DeleteInteractionRequestByID(ctx context.Context, id string) error {
-	defer i.state.Caches.DB.InteractionRequest.Invalidate("ID", id)
+	// Delete interaction request by ID.
-
+	if _, err := i.db.NewDelete().
-	_, err := i.db.NewDelete().
+		Table("interaction_requests").
-		TableExpr("? AS ?", bun.Ident("interaction_requests"), bun.Ident("interaction_request")).
+		Where("? = ?", bun.Ident("id"), id).
-		Where("? = ?", bun.Ident("interaction_request.id"), id).
+		Exec(ctx); err != nil {
-		Exec(ctx)
 		return err
 	}
 
+	// Invalidate cached interaction request with ID.
+	i.state.Caches.DB.InteractionRequest.Invalidate("ID", id)
+
+	return nil
+}
+
 func (i *interactionDB) GetInteractionsRequestsForAcct(
 	ctx context.Context,
 	acctID string,
@@ -317,19 +371,8 @@ func (i *interactionDB) GetInteractionsRequestsForAcct(
 		slices.Reverse(reqIDs)
 	}
 
-	// For each interaction request ID,
+	// Load all interaction requests by their IDs.
-	// select the interaction request.
+	return i.GetInteractionRequestsByIDs(ctx, reqIDs)
-	reqs := make([]*gtsmodel.InteractionRequest, 0, len(reqIDs))
-	for _, id := range reqIDs {
-		req, err := i.GetInteractionRequestByID(ctx, id)
-		if err != nil {
-			return nil, err
-		}
-
-		reqs = append(reqs, req)
-	}
-
-	return reqs, nil
 }
 
 func (i *interactionDB) IsInteractionRejected(ctx context.Context, interactionURI string) (bool, error) {

internal/db/bundb/timeline_test.go

@@ -74,7 +74,8 @@ func (suite *TimelineTestSuite) publicCount() int {
 	var publicCount int
 	for _, status := range suite.testStatuses {
 		if status.Visibility == gtsmodel.VisibilityPublic &&
-			status.BoostOfID == "" {
+			status.BoostOfID == "" &&
+			!util.PtrOrZero(status.PendingApproval) {
 			publicCount++
 		}
 	}

internal/filter/visibility/status_test.go

@@ -168,6 +168,9 @@ func (suite *StatusVisibleTestSuite) TestVisiblePending() {
 	testStatus := new(gtsmodel.Status)
 	*testStatus = *suite.testStatuses["admin_account_status_3"]
 	testStatus.PendingApproval = util.Ptr(true)
+	if err := suite.state.DB.UpdateStatus(ctx, testStatus); err != nil {
+		suite.FailNow(err.Error())
+	}
 
 	for _, testCase := range []struct {
 		acct    *gtsmodel.Account
@@ -198,6 +201,43 @@ func (suite *StatusVisibleTestSuite) TestVisiblePending() {
 		suite.NoError(err)
 		suite.Equal(testCase.visible, visible)
 	}
+
+	// Update the status to mark it as approved.
+	testStatus.PendingApproval = util.Ptr(false)
+	testStatus.ApprovedByURI = "http://localhost:8080/some/accept/uri"
+	if err := suite.state.DB.UpdateStatus(ctx, testStatus); err != nil {
+		suite.FailNow(err.Error())
+	}
+
+	for _, testCase := range []struct {
+		acct    *gtsmodel.Account
+		visible bool
+	}{
+		{
+			acct:    suite.testAccounts["admin_account"],
+			visible: true, // Own status, always visible.
+		},
+		{
+			acct:    suite.testAccounts["local_account_1"],
+			visible: true, // Reply to zork, always visible.
+		},
+		{
+			acct:    suite.testAccounts["local_account_2"],
+			visible: true, // Should be visible now.
+		},
+		{
+			acct:    suite.testAccounts["remote_account_1"],
+			visible: true, // Should be visible now.
+		},
+		{
+			acct:    nil,  // Unauthed request.
+			visible: true, // Should be visible now (public status).
+		},
+	} {
+		visible, err := suite.filter.StatusVisible(ctx, testCase.acct, testStatus)
+		suite.NoError(err)
+		suite.Equal(testCase.visible, visible)
+	}
 }
 
 func (suite *StatusVisibleTestSuite) TestVisibleLocalOnly() {

internal/httpclient/request.go

@@ -19,6 +19,7 @@
 
 import (
 	"net/http"
+	"strconv"
 	"time"
 
 	"github.com/superseriousbusiness/gotosocial/internal/log"
@@ -32,6 +33,7 @@
 // Request wraps an HTTP request
 // to add our own retry / backoff.
 type Request struct {
+
 	// Current backoff dur.
 	backoff time.Duration
 
@@ -57,8 +59,7 @@ func WrapRequest(r *http.Request) *Request {
 		// Only add content-type header if a request body exists.
 		entry = entry.WithField("contentType", r.Header.Get("Content-Type"))
 	}
-	// note our formatting library follows ptr values
+	entry = entry.WithField("attempt", uintPtr{&rr.attempts})
-	entry = entry.WithField("attempt", &rr.attempts)
 	rr.Entry = entry
 	return rr
 }
@@ -73,3 +74,12 @@ func (r *Request) BackOff() time.Duration {
 	}
 	return r.backoff
 }
+
+type uintPtr struct{ u *uint }
+
+func (f uintPtr) String() string {
+	if f.u == nil {
+		return "<nil>"
+	}
+	return strconv.FormatUint(uint64(*f.u), 10)
+}

internal/typeutils/internaltofrontend_test.go

@@ -1744,7 +1744,7 @@ func (suite *InternalToFrontendTestSuite) TestStatusToAPIStatusPendingApproval()
   "in_reply_to_account_id": "01F8MH5NBDF2MV7CTC4Q5128HF",
   "sensitive": false,
   "spoiler_text": "",
-  "visibility": "unlisted",
+  "visibility": "public",
   "language": null,
   "uri": "http://localhost:8080/users/admin/statuses/01J5QVB9VC76NPPRQ207GG4DRZ",
   "url": "http://localhost:8080/@admin/statuses/01J5QVB9VC76NPPRQ207GG4DRZ",
@@ -3177,7 +3177,7 @@ func (suite *InternalToFrontendTestSuite) TestIntReqToAPI() {
     "in_reply_to_account_id": null,
     "sensitive": true,
     "spoiler_text": "you won't be able to reply to this without my approval",
-    "visibility": "unlisted",
+    "visibility": "public",
     "language": "en",
     "uri": "http://localhost:8080/users/1happyturtle/statuses/01F8MHC8VWDRBQR0N1BATDDEM5",
     "url": "http://localhost:8080/@1happyturtle/statuses/01F8MHC8VWDRBQR0N1BATDDEM5",
@@ -3269,7 +3269,7 @@ func (suite *InternalToFrontendTestSuite) TestIntReqToAPI() {
     "in_reply_to_account_id": "01F8MH5NBDF2MV7CTC4Q5128HF",
     "sensitive": false,
     "spoiler_text": "",
-    "visibility": "unlisted",
+    "visibility": "public",
     "language": null,
     "uri": "http://localhost:8080/users/admin/statuses/01J5QVB9VC76NPPRQ207GG4DRZ",
     "url": "http://localhost:8080/@admin/statuses/01J5QVB9VC76NPPRQ207GG4DRZ",

mkdocs.yml

@@ -116,6 +116,7 @@ nav:
     - "advanced/tracing.md"
     - "advanced/metrics.md"
     - "advanced/replicating-sqlite.md"
+    - "advanced/sqlite-networked-storage.md"
 
   - "Admin":
       - "admin/settings.md"

testrig/testmodels.go

@@ -1531,7 +1531,7 @@ func NewTestStatuses() map[string]*gtsmodel.Status {
 			BoostOfID:                "",
 			BoostOfAccountID:         "",
 			ThreadID:                 "01HCWE4P0EW9HBA5WHW97D5YV0",
-			Visibility:               gtsmodel.VisibilityUnlocked,
+			Visibility:               gtsmodel.VisibilityPublic,
 			Sensitive:                util.Ptr(false),
 			CreatedWithApplicationID: "01F8MGXQRHYF5QPMTMXP78QC2F",
 			Federated:                util.Ptr(true),
@@ -1811,7 +1811,7 @@ func NewTestStatuses() map[string]*gtsmodel.Status {
 			BoostOfID:                "",
 			ThreadID:                 "01HCWE4P0EW9HBA5WHW97D5YV0",
 			ContentWarning:           "you won't be able to reply to this without my approval",
-			Visibility:               gtsmodel.VisibilityUnlocked,
+			Visibility:               gtsmodel.VisibilityPublic,
 			Sensitive:                util.Ptr(true),
 			Language:                 "en",
 			CreatedWithApplicationID: "01F8MGYG9E893WRHW0TAEXR8GJ",

vendor/codeberg.org/gruf/go-structr/cache.go

@@ -575,8 +575,9 @@ func (c *Cache[T]) store_value(index *Index, key string, value T) {
 	item.data = value
 
 	if index != nil {
-		// Append item to index.
+		// Append item to index a key
-		index.append(key, item)
+		// was already generated for.
+		index.append(&c.lru, key, item)
 	}
 
 	// Get ptr to value data.
@@ -607,8 +608,8 @@ func (c *Cache[T]) store_value(index *Index, key string, value T) {
 			continue
 		}
 
-		// Append item to index.
+		// Append item to this index.
-		idx.append(key, item)
+		idx.append(&c.lru, key, item)
 	}
 
 	// Add item to main lru list.
@@ -645,8 +646,9 @@ func (c *Cache[T]) store_error(index *Index, key string, err error) {
 	// Set error val.
 	item.data = err
 
-	// Append item to index.
+	// Append item to index a key
-	index.append(key, item)
+	// was already generated for.
+	index.append(&c.lru, key, item)
 
 	// Add item to main lru list.
 	c.lru.push_front(&item.elem)

vendor/codeberg.org/gruf/go-structr/index.go

@@ -174,7 +174,7 @@ func (i *Index) init(t reflect.Type, cfg IndexConfig, cap int) {
 // get_one will fetch one indexed item under key.
 func (i *Index) get_one(key Key) *indexed_item {
 	// Get list at hash.
-	l, _ := i.data.Get(key.key)
+	l := i.data.Get(key.key)
 	if l == nil {
 		return nil
 	}
@@ -192,7 +192,7 @@ func (i *Index) get(key string, hook func(*indexed_item)) {
 	}
 
 	// Get list at hash.
-	l, _ := i.data.Get(key)
+	l := i.data.Get(key)
 	if l == nil {
 		return
 	}
@@ -237,11 +237,12 @@ func (i *Index) key(buf *byteutil.Buffer, parts []unsafe.Pointer) string {
 }
 
 // append will append the given index entry to appropriate
-// doubly-linked-list in index hashmap. this handles case
+// doubly-linked-list in index hashmap. this handles case of
-// of key collisions and overwriting 'unique' entries.
+// overwriting "unique" index entries, and removes from given
-func (i *Index) append(key string, item *indexed_item) {
+// outer linked-list in the case that it is no longer indexed.
+func (i *Index) append(ll *list, key string, item *indexed_item) {
 	// Look for existing.
-	l, _ := i.data.Get(key)
+	l := i.data.Get(key)
 
 	if l == nil {
 
@@ -255,12 +256,21 @@ func (i *Index) append(key string, item *indexed_item) {
 		elem := l.head
 		l.remove(elem)
 
-		// Drop index from inner item.
+		// Drop index from inner item,
+		// catching the evicted item.
 		e := (*index_entry)(elem.data)
-		e.item.drop_index(e)
+		evicted := e.item
+		evicted.drop_index(e)
 
 		// Free unused entry.
 		free_index_entry(e)
+
+		if len(evicted.indexed) == 0 {
+			// Evicted item is not indexed,
+			// remove from outer linked list.
+			ll.remove(&evicted.elem)
+			free_indexed_item(evicted)
+		}
 	}
 
 	// Prepare new index entry.
@@ -283,7 +293,7 @@ func (i *Index) delete(key string, hook func(*indexed_item)) {
 	}
 
 	// Get list at hash.
-	l, _ := i.data.Get(key)
+	l := i.data.Get(key)
 	if l == nil {
 		return
 	}
@@ -292,10 +302,9 @@ func (i *Index) delete(key string, hook func(*indexed_item)) {
 	i.data.Delete(key)
 
 	// Iterate entries in list.
-	for x := 0; x < l.len; x++ {
+	l.rangefn(func(elem *list_elem) {
 
-		// Pop list head.
+		// Remove elem.
-		elem := l.head
 		l.remove(elem)
 
 		// Extract element entry + item.
@@ -310,7 +319,7 @@ func (i *Index) delete(key string, hook func(*indexed_item)) {
 
 		// Pass to hook.
 		hook(item)
-	}
+	})
 
 	// Release list.
 	free_list(l)
@@ -319,7 +328,7 @@ func (i *Index) delete(key string, hook func(*indexed_item)) {
 // delete_entry deletes the given index entry.
 func (i *Index) delete_entry(entry *index_entry) {
 	// Get list at hash sum.
-	l, _ := i.data.Get(entry.key)
+	l := i.data.Get(entry.key)
 	if l == nil {
 		return
 	}

vendor/codeberg.org/gruf/go-structr/item.go

@@ -50,12 +50,9 @@ func (i *indexed_item) drop_index(entry *index_entry) {
 			continue
 		}
 
-		// Unset tptr value to
-		// ensure GC can take it.
-		i.indexed[x] = nil
-
 		// Move all index entries down + reslice.
 		_ = copy(i.indexed[x:], i.indexed[x+1:])
+		i.indexed[len(i.indexed)-1] = nil
 		i.indexed = i.indexed[:len(i.indexed)-1]
 		break
 	}

vendor/codeberg.org/gruf/go-structr/list.go

@@ -48,27 +48,17 @@ func free_list(list *list) {
 
 // push_front will push the given elem to front (head) of list.
 func (l *list) push_front(elem *list_elem) {
-	if l.len == 0 {
+	// Set new head.
-		// Set new tail + head
-		l.head = elem
-		l.tail = elem
-
-		// Link elem to itself
-		elem.next = elem
-		elem.prev = elem
-	} else {
 	oldHead := l.head
+	l.head = elem
 
+	if oldHead != nil {
 		// Link to old head
 		elem.next = oldHead
 		oldHead.prev = elem
-
+	} else {
-		// Link up to tail
+		// First in list.
-		elem.prev = l.tail
+		l.tail = elem
-		l.tail.next = elem
-
-		// Set new head
-		l.head = elem
 	}
 
 	// Incr count
@@ -77,27 +67,17 @@ func (l *list) push_front(elem *list_elem) {
 
 // push_back will push the given elem to back (tail) of list.
 func (l *list) push_back(elem *list_elem) {
-	if l.len == 0 {
+	// Set new tail.
-		// Set new tail + head
+	oldTail := l.tail
-		l.head = elem
 	l.tail = elem
 
-		// Link elem to itself
+	if oldTail != nil {
-		elem.next = elem
-		elem.prev = elem
-	} else {
-		oldTail := l.tail
-
 		// Link to old tail
 		elem.prev = oldTail
 		oldTail.next = elem
-
+	} else {
-		// Link up to head
+		// First in list.
-		elem.next = l.head
+		l.head = elem
-		l.head.prev = elem
-
-		// Set new tail
-		l.tail = elem
 	}
 
 	// Incr count
@@ -105,53 +85,57 @@ func (l *list) push_back(elem *list_elem) {
 }
 
 // move_front will move given elem to front (head) of list.
+// if it is already at front this call is a no-op.
 func (l *list) move_front(elem *list_elem) {
+	if elem == l.head {
+		return
+	}
 	l.remove(elem)
 	l.push_front(elem)
 }
 
-// move_back will move given elem to back (tail) of list.
+// move_back will move given elem to back (tail) of list,
+// if it is already at back this call is a no-op.
 func (l *list) move_back(elem *list_elem) {
+	if elem == l.tail {
+		return
+	}
 	l.remove(elem)
 	l.push_back(elem)
 }
 
 // remove will remove given elem from list.
 func (l *list) remove(elem *list_elem) {
-	if l.len <= 1 {
+	// Get linked elems.
-		// Drop elem's links
-		elem.next = nil
-		elem.prev = nil
-
-		// Only elem in list
-		l.head = nil
-		l.tail = nil
-		l.len = 0
-		return
-	}
-
-	// Get surrounding elems
 	next := elem.next
 	prev := elem.prev
 
-	// Relink chain
+	// Unset elem.
-	next.prev = prev
-	prev.next = next
-
-	switch elem {
-	// Set new head
-	case l.head:
-		l.head = next
-
-	// Set new tail
-	case l.tail:
-		l.tail = prev
-	}
-
-	// Drop elem's links
 	elem.next = nil
 	elem.prev = nil
 
+	switch {
+	// elem is ONLY one in list.
+	case next == nil && prev == nil:
+		l.head = nil
+		l.tail = nil
+
+	// elem is front in list.
+	case next != nil && prev == nil:
+		l.head = next
+		next.prev = nil
+
+	// elem is last in list.
+	case prev != nil && next == nil:
+		l.tail = prev
+		prev.next = nil
+
+	// elem in middle of list.
+	default:
+		next.prev = prev
+		prev.next = next
+	}
+
 	// Decr count
 	l.len--
 }
@@ -161,9 +145,11 @@ func (l *list) rangefn(fn func(*list_elem)) {
 	if fn == nil {
 		panic("nil fn")
 	}
-	elem := l.head
+	for e := l.head; //
-	for i := 0; i < l.len; i++ {
+	e != nil;        //
-		fn(elem)
+	{
-		elem = elem.next
+		n := e.next
+		fn(e)
+		e = n
 	}
 }

vendor/codeberg.org/gruf/go-structr/map.go

@@ -10,9 +10,8 @@ func (m *hashmap) init(cap int) {
 	m.n = cap
 }
 
-func (m *hashmap) Get(key string) (*list, bool) {
+func (m *hashmap) Get(key string) *list {
-	list, ok := m.m[key]
+	return m.m[key]
-	return list, ok
 }
 
 func (m *hashmap) Put(key string, list *list) {

vendor/codeberg.org/gruf/go-structr/queue.go

@@ -308,8 +308,8 @@ func (q *Queue[T]) index(value T) *indexed_item {
 			continue
 		}
 
-		// Append item to index.
+		// Append item to this index.
-		idx.append(key, item)
+		idx.append(&q.queue, key, item)
 	}
 
 	// Done with buf.

vendor/modules.txt

@@ -66,7 +66,7 @@ codeberg.org/gruf/go-storage/disk
 codeberg.org/gruf/go-storage/internal
 codeberg.org/gruf/go-storage/memory
 codeberg.org/gruf/go-storage/s3
-# codeberg.org/gruf/go-structr v0.8.9
+# codeberg.org/gruf/go-structr v0.8.10
 ## explicit; go 1.21
 codeberg.org/gruf/go-structr
 # codeberg.org/superseriousbusiness/exif-terminator v0.9.0