gotosocial/internal/processing/processor.go

260 lines
9 KiB
Go
Raw Normal View History

// GoToSocial
// Copyright (C) GoToSocial Authors admin@gotosocial.org
// SPDX-License-Identifier: AGPL-3.0-or-later
//
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU Affero General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
//
// This program is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU Affero General Public License for more details.
//
// You should have received a copy of the GNU Affero General Public License
// along with this program. If not, see <http://www.gnu.org/licenses/>.
package processing
import (
"github.com/superseriousbusiness/gotosocial/internal/cleaner"
"github.com/superseriousbusiness/gotosocial/internal/email"
"github.com/superseriousbusiness/gotosocial/internal/federation"
"github.com/superseriousbusiness/gotosocial/internal/filter/interaction"
"github.com/superseriousbusiness/gotosocial/internal/filter/visibility"
"github.com/superseriousbusiness/gotosocial/internal/gtsmodel"
mm "github.com/superseriousbusiness/gotosocial/internal/media"
"github.com/superseriousbusiness/gotosocial/internal/oauth"
"github.com/superseriousbusiness/gotosocial/internal/processing/account"
"github.com/superseriousbusiness/gotosocial/internal/processing/admin"
[feature] Conversations API (#3013) * Implement conversations API * Sort and page conversations by last status ID * Appease linter * Fix deleting conversations and statuses * Refactor to make migrations automatic * Lint * Update tests post-merge * Fixes from live-fire testing * Linter caught a format problem * Refactor tests, fix cache * Negative test for non-DMs * Run conversations advanced migration on testrig startup as well as regular server startup * Document (lack of) side effects of API method for deleting a conversation * Make not-found check less nested for readability * Rename PutConversation to UpsertConversation * Use util.Ptr instead of IIFE * Reduce cache used by conversations * Remove unnecessary TableExpr/ColumnExpr * Use struct tags for both unique constraints on Conversation * Make it clear how paging with GetDirectStatusIDsBatch should be used * Let conversation paging skip conversations it can't render * Use Bun NewDropTable * Convert delete raw query to Bun * Convert update raw query to Bun * Convert latestConversationStatusesTempTable raw query partially to Bun * Convert conversationStatusesTempTable raw query partially to Bun * Rename field used to store result of MaxDirectStatusID * Move advanced migrations to their own tiny processor * Catch up util function name with main * Remove json.… wrappers * Remove redundant check * Combine error checks * Replace map with slice of structs * Address processor/type converter comments - Add context info for errors - Extract some common processor code into shared methods - Move conversation eligibility check ahead of populating conversation * Add error context when dropping temp tables
2024-07-23 19:44:31 +00:00
"github.com/superseriousbusiness/gotosocial/internal/processing/advancedmigrations"
"github.com/superseriousbusiness/gotosocial/internal/processing/common"
[feature] Conversations API (#3013) * Implement conversations API * Sort and page conversations by last status ID * Appease linter * Fix deleting conversations and statuses * Refactor to make migrations automatic * Lint * Update tests post-merge * Fixes from live-fire testing * Linter caught a format problem * Refactor tests, fix cache * Negative test for non-DMs * Run conversations advanced migration on testrig startup as well as regular server startup * Document (lack of) side effects of API method for deleting a conversation * Make not-found check less nested for readability * Rename PutConversation to UpsertConversation * Use util.Ptr instead of IIFE * Reduce cache used by conversations * Remove unnecessary TableExpr/ColumnExpr * Use struct tags for both unique constraints on Conversation * Make it clear how paging with GetDirectStatusIDsBatch should be used * Let conversation paging skip conversations it can't render * Use Bun NewDropTable * Convert delete raw query to Bun * Convert update raw query to Bun * Convert latestConversationStatusesTempTable raw query partially to Bun * Convert conversationStatusesTempTable raw query partially to Bun * Rename field used to store result of MaxDirectStatusID * Move advanced migrations to their own tiny processor * Catch up util function name with main * Remove json.… wrappers * Remove redundant check * Combine error checks * Replace map with slice of structs * Address processor/type converter comments - Add context info for errors - Extract some common processor code into shared methods - Move conversation eligibility check ahead of populating conversation * Add error context when dropping temp tables
2024-07-23 19:44:31 +00:00
"github.com/superseriousbusiness/gotosocial/internal/processing/conversations"
"github.com/superseriousbusiness/gotosocial/internal/processing/fedi"
filtersv1 "github.com/superseriousbusiness/gotosocial/internal/processing/filters/v1"
filtersv2 "github.com/superseriousbusiness/gotosocial/internal/processing/filters/v2"
"github.com/superseriousbusiness/gotosocial/internal/processing/interactionrequests"
"github.com/superseriousbusiness/gotosocial/internal/processing/list"
"github.com/superseriousbusiness/gotosocial/internal/processing/markers"
"github.com/superseriousbusiness/gotosocial/internal/processing/media"
"github.com/superseriousbusiness/gotosocial/internal/processing/polls"
2024-12-01 04:13:06 +00:00
"github.com/superseriousbusiness/gotosocial/internal/processing/push"
"github.com/superseriousbusiness/gotosocial/internal/processing/report"
"github.com/superseriousbusiness/gotosocial/internal/processing/search"
"github.com/superseriousbusiness/gotosocial/internal/processing/status"
"github.com/superseriousbusiness/gotosocial/internal/processing/stream"
"github.com/superseriousbusiness/gotosocial/internal/processing/tags"
"github.com/superseriousbusiness/gotosocial/internal/processing/timeline"
"github.com/superseriousbusiness/gotosocial/internal/processing/user"
"github.com/superseriousbusiness/gotosocial/internal/processing/workers"
"github.com/superseriousbusiness/gotosocial/internal/state"
"github.com/superseriousbusiness/gotosocial/internal/text"
"github.com/superseriousbusiness/gotosocial/internal/typeutils"
"github.com/superseriousbusiness/gotosocial/internal/webpush"
)
// Processor groups together processing functions and
// sub processors for handling actions + events coming
// from either the client or federating APIs.
//
// Many of the functions available through this struct
// or sub processors will trigger asynchronous processing
// via the workers contained in state.
type Processor struct {
converter *typeutils.Converter
oauthServer oauth.Server
state *state.State
/*
Required for instance description / terms updating.
*/
formatter *text.Formatter
parseMentionFunc gtsmodel.ParseMentionFunc
/*
SUB-PROCESSORS
*/
account account.Processor
admin admin.Processor
advancedmigrations advancedmigrations.Processor
conversations conversations.Processor
fedi fedi.Processor
filtersv1 filtersv1.Processor
filtersv2 filtersv2.Processor
interactionRequests interactionrequests.Processor
list list.Processor
markers markers.Processor
media media.Processor
polls polls.Processor
2024-12-01 04:13:06 +00:00
push push.Processor
report report.Processor
search search.Processor
status status.Processor
stream stream.Processor
tags tags.Processor
timeline timeline.Processor
user user.Processor
workers workers.Processor
}
func (p *Processor) Account() *account.Processor {
return &p.account
}
func (p *Processor) Admin() *admin.Processor {
return &p.admin
}
[feature] Conversations API (#3013) * Implement conversations API * Sort and page conversations by last status ID * Appease linter * Fix deleting conversations and statuses * Refactor to make migrations automatic * Lint * Update tests post-merge * Fixes from live-fire testing * Linter caught a format problem * Refactor tests, fix cache * Negative test for non-DMs * Run conversations advanced migration on testrig startup as well as regular server startup * Document (lack of) side effects of API method for deleting a conversation * Make not-found check less nested for readability * Rename PutConversation to UpsertConversation * Use util.Ptr instead of IIFE * Reduce cache used by conversations * Remove unnecessary TableExpr/ColumnExpr * Use struct tags for both unique constraints on Conversation * Make it clear how paging with GetDirectStatusIDsBatch should be used * Let conversation paging skip conversations it can't render * Use Bun NewDropTable * Convert delete raw query to Bun * Convert update raw query to Bun * Convert latestConversationStatusesTempTable raw query partially to Bun * Convert conversationStatusesTempTable raw query partially to Bun * Rename field used to store result of MaxDirectStatusID * Move advanced migrations to their own tiny processor * Catch up util function name with main * Remove json.… wrappers * Remove redundant check * Combine error checks * Replace map with slice of structs * Address processor/type converter comments - Add context info for errors - Extract some common processor code into shared methods - Move conversation eligibility check ahead of populating conversation * Add error context when dropping temp tables
2024-07-23 19:44:31 +00:00
func (p *Processor) AdvancedMigrations() *advancedmigrations.Processor {
return &p.advancedmigrations
}
func (p *Processor) Conversations() *conversations.Processor {
return &p.conversations
}
func (p *Processor) Fedi() *fedi.Processor {
return &p.fedi
}
func (p *Processor) FiltersV1() *filtersv1.Processor {
return &p.filtersv1
}
func (p *Processor) FiltersV2() *filtersv2.Processor {
return &p.filtersv2
}
func (p *Processor) InteractionRequests() *interactionrequests.Processor {
return &p.interactionRequests
}
func (p *Processor) List() *list.Processor {
return &p.list
}
func (p *Processor) Markers() *markers.Processor {
return &p.markers
}
func (p *Processor) Media() *media.Processor {
return &p.media
}
func (p *Processor) Polls() *polls.Processor {
return &p.polls
}
2024-12-01 04:13:06 +00:00
func (p *Processor) Push() *push.Processor {
return &p.push
}
func (p *Processor) Report() *report.Processor {
return &p.report
}
func (p *Processor) Search() *search.Processor {
return &p.search
}
func (p *Processor) Status() *status.Processor {
return &p.status
}
func (p *Processor) Stream() *stream.Processor {
return &p.stream
}
func (p *Processor) Tags() *tags.Processor {
return &p.tags
}
func (p *Processor) Timeline() *timeline.Processor {
return &p.timeline
}
func (p *Processor) User() *user.Processor {
return &p.user
}
func (p *Processor) Workers() *workers.Processor {
return &p.workers
}
// NewProcessor returns a new Processor.
func NewProcessor(
cleaner *cleaner.Cleaner,
converter *typeutils.Converter,
federator *federation.Federator,
oauthServer oauth.Server,
mediaManager *mm.Manager,
state *state.State,
emailSender email.Sender,
webPushSender webpush.Sender,
visFilter *visibility.Filter,
intFilter *interaction.Filter,
) *Processor {
parseMentionFunc := GetParseMentionFunc(state, federator)
processor := &Processor{
converter: converter,
oauthServer: oauthServer,
state: state,
formatter: text.NewFormatter(state.DB),
parseMentionFunc: parseMentionFunc,
}
// Instantiate sub processors.
//
// Start with sub processors that will
// be required by the workers processor.
common := common.New(state, mediaManager, converter, federator, visFilter)
processor.account = account.New(&common, state, converter, mediaManager, federator, visFilter, parseMentionFunc)
[chore] media and emoji refactoring (#3000) * start updating media manager interface ready for storing attachments / emoji right away * store emoji and media as uncached immediately, then (re-)cache on Processing{}.Load() * remove now unused media workers * fix tests and issues * fix another test! * fix emoji activitypub uri setting behaviour, fix remainder of test compilation issues * fix more tests * fix (most of) remaining tests, add debouncing to repeatedly failing media / emojis * whoops, rebase issue * remove kim's whacky experiments * do some reshuffling, ensure emoji uri gets set * ensure marked as not cached on cleanup * tweaks to media / emoji processing to handle context canceled better * ensure newly fetched emojis actually get set in returned slice * use different varnames to be a bit more obvious * move emoji refresh rate limiting to dereferencer * add exported dereferencer functions for remote media, use these for recaching in processor * add check for nil attachment in updateAttachment() * remove unused emoji and media fields + columns * see previous commit * fix old migrations expecting image_updated_at to exists (from copies of old models) * remove freshness checking code (seems to be broken...) * fix error arg causing nil ptr exception * finish documentating functions with comments, slight tweaks to media / emoji deref error logic * remove some extra unneeded boolean checking * finish writing documentation (code comments) for exported media manager methods * undo changes to migration snapshot gtsmodels, updated failing migration to have its own snapshot * move doesColumnExist() to util.go in migrations package
2024-06-26 15:01:16 +00:00
processor.media = media.New(&common, state, converter, federator, mediaManager, federator.TransportController())
processor.stream = stream.New(state, oauthServer)
// Instantiate the rest of the sub
// processors + pin them to this struct.
processor.account = account.New(&common, state, converter, mediaManager, federator, visFilter, parseMentionFunc)
[chore] media and emoji refactoring (#3000) * start updating media manager interface ready for storing attachments / emoji right away * store emoji and media as uncached immediately, then (re-)cache on Processing{}.Load() * remove now unused media workers * fix tests and issues * fix another test! * fix emoji activitypub uri setting behaviour, fix remainder of test compilation issues * fix more tests * fix (most of) remaining tests, add debouncing to repeatedly failing media / emojis * whoops, rebase issue * remove kim's whacky experiments * do some reshuffling, ensure emoji uri gets set * ensure marked as not cached on cleanup * tweaks to media / emoji processing to handle context canceled better * ensure newly fetched emojis actually get set in returned slice * use different varnames to be a bit more obvious * move emoji refresh rate limiting to dereferencer * add exported dereferencer functions for remote media, use these for recaching in processor * add check for nil attachment in updateAttachment() * remove unused emoji and media fields + columns * see previous commit * fix old migrations expecting image_updated_at to exists (from copies of old models) * remove freshness checking code (seems to be broken...) * fix error arg causing nil ptr exception * finish documentating functions with comments, slight tweaks to media / emoji deref error logic * remove some extra unneeded boolean checking * finish writing documentation (code comments) for exported media manager methods * undo changes to migration snapshot gtsmodels, updated failing migration to have its own snapshot * move doesColumnExist() to util.go in migrations package
2024-06-26 15:01:16 +00:00
processor.admin = admin.New(&common, state, cleaner, federator, converter, mediaManager, federator.TransportController(), emailSender)
processor.conversations = conversations.New(state, converter, visFilter)
processor.fedi = fedi.New(state, &common, converter, federator, visFilter)
processor.filtersv1 = filtersv1.New(state, converter, &processor.stream)
processor.filtersv2 = filtersv2.New(state, converter, &processor.stream)
processor.interactionRequests = interactionrequests.New(&common, state, converter)
processor.list = list.New(state, converter)
processor.markers = markers.New(state, converter)
processor.polls = polls.New(&common, state, converter)
2024-12-01 04:13:06 +00:00
processor.push = push.New(state, converter)
processor.report = report.New(state, converter)
processor.tags = tags.New(state, converter)
processor.timeline = timeline.New(state, converter, visFilter)
processor.search = search.New(state, federator, converter, visFilter)
processor.status = status.New(state, &common, &processor.polls, &processor.interactionRequests, federator, converter, visFilter, intFilter, parseMentionFunc)
processor.user = user.New(state, converter, oauthServer, emailSender)
[feature] Conversations API (#3013) * Implement conversations API * Sort and page conversations by last status ID * Appease linter * Fix deleting conversations and statuses * Refactor to make migrations automatic * Lint * Update tests post-merge * Fixes from live-fire testing * Linter caught a format problem * Refactor tests, fix cache * Negative test for non-DMs * Run conversations advanced migration on testrig startup as well as regular server startup * Document (lack of) side effects of API method for deleting a conversation * Make not-found check less nested for readability * Rename PutConversation to UpsertConversation * Use util.Ptr instead of IIFE * Reduce cache used by conversations * Remove unnecessary TableExpr/ColumnExpr * Use struct tags for both unique constraints on Conversation * Make it clear how paging with GetDirectStatusIDsBatch should be used * Let conversation paging skip conversations it can't render * Use Bun NewDropTable * Convert delete raw query to Bun * Convert update raw query to Bun * Convert latestConversationStatusesTempTable raw query partially to Bun * Convert conversationStatusesTempTable raw query partially to Bun * Rename field used to store result of MaxDirectStatusID * Move advanced migrations to their own tiny processor * Catch up util function name with main * Remove json.… wrappers * Remove redundant check * Combine error checks * Replace map with slice of structs * Address processor/type converter comments - Add context info for errors - Extract some common processor code into shared methods - Move conversation eligibility check ahead of populating conversation * Add error context when dropping temp tables
2024-07-23 19:44:31 +00:00
// The advanced migrations processor sequences advanced migrations from all other processors.
processor.advancedmigrations = advancedmigrations.New(&processor.conversations)
// Workers processor handles asynchronous
// worker jobs; instantiate it separately
// and pass subset of sub processors it needs.
processor.workers = workers.New(
state,
&common,
federator,
converter,
visFilter,
emailSender,
webPushSender,
&processor.account,
&processor.media,
&processor.stream,
[feature] Conversations API (#3013) * Implement conversations API * Sort and page conversations by last status ID * Appease linter * Fix deleting conversations and statuses * Refactor to make migrations automatic * Lint * Update tests post-merge * Fixes from live-fire testing * Linter caught a format problem * Refactor tests, fix cache * Negative test for non-DMs * Run conversations advanced migration on testrig startup as well as regular server startup * Document (lack of) side effects of API method for deleting a conversation * Make not-found check less nested for readability * Rename PutConversation to UpsertConversation * Use util.Ptr instead of IIFE * Reduce cache used by conversations * Remove unnecessary TableExpr/ColumnExpr * Use struct tags for both unique constraints on Conversation * Make it clear how paging with GetDirectStatusIDsBatch should be used * Let conversation paging skip conversations it can't render * Use Bun NewDropTable * Convert delete raw query to Bun * Convert update raw query to Bun * Convert latestConversationStatusesTempTable raw query partially to Bun * Convert conversationStatusesTempTable raw query partially to Bun * Rename field used to store result of MaxDirectStatusID * Move advanced migrations to their own tiny processor * Catch up util function name with main * Remove json.… wrappers * Remove redundant check * Combine error checks * Replace map with slice of structs * Address processor/type converter comments - Add context info for errors - Extract some common processor code into shared methods - Move conversation eligibility check ahead of populating conversation * Add error context when dropping temp tables
2024-07-23 19:44:31 +00:00
&processor.conversations,
)
return processor
}