Old versions of the Mattermost server did not qualify queries scanning both `Posts` and `Threads`, and choke on the ambiguity in deciding between the new `DeleteAt` on `Threads` and the `DeleteAt` on `Posts` in existing queries.
While this problem is transient only while running multiple server versions, it effectively makes our backwards compatibility guarantee void, not to mention complicating cloud deployments.
Work around this by renaming `Threads.DeleteAt` to `Threads.ThreadDeleteAt`. The old migration is nulled out, but remains, since some test servers have already upgraded and manually fixing each affected instance would be problematic. Thew new migration takes care of removing the old column -- if it ever existed.
Fixes: https://mattermost.atlassian.net/browse/MM-43770
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* [MM-42739] Initial setup for top channels for team
* [MM-42739] Add initial tests
* [MM-42739] Update tests
* [MM-42739] Add top channels for user
* [MM-42739] Fix query
* [MM-42739] Update query
* [MM-42739] Improve query performance
* [MM-42739] Remove rank
* [MM-42739] Fix tests to use new time range today
* [MM-42739] Add tests for top channels for user
* [MM-42739] Add test for pagination
* Remove top channels by time struct
* [MM-42739] Update test names
* [MM-42739] Remove rank from top reactions
* [MM-42739] Return empty array instead of nil when result is empty
* [MM-42739] Add additional tests and update permissions check for teams
* [MM-42739] Add excluded channel tests for top reactions
* [MM-42739] Move insights to api4/insights and keep time range as string until required
* [MM-42739] Update queries only check DeleteAt after union
* [MM-42739] Improve query performance by using publicchannels table
* [MM-42739] Fix broken query after merge
* removed appending the root post to the posts list
also changed `UpdateAt` to `CreateAt` in thread_store.go in accordance with kyriakos
* fixed failing test after latest change
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* Disambiguates some units.
* Updates DB attribute.
* Updates some more error-prone units.
* Updates some tests with legible constants.
* Updates query for MySQL case sensitivity.
* Fixes more casing issues.
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
On Postgres, `GetTeamsUnreadForUser` triggers a sequential scan on `Posts`. We can avoid this by querying the `Threads` table directly and only joining to `Posts` to eliminate deleted threads. (We could avoid the latter if we later denormalize `DeleteAt` onto `Threads`.)
Fixes: https://mattermost.atlassian.net/browse/MM-42919
We check for the presence of binary_parameters
in the DSN and add the 0x01 byte accordingly.
This helps us avoid casting to string
and efficiently use the database.
```release-note
NONE
```
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
The older method used to reply completely on timestamps
to take batches of items in a timestamp range and then
just incrementing the timestamp. This led to handling
edge-cases such as more items than the batch count, all
having the same timestamp.
Additionally, relying on timestamp as the page cursor
meant that indexing was not very efficient if you had
several items spread out across large spans of time.
To get away from all of that we use a proper cursor-based
approach consisting of createAt+Id. With this, we move
completely to a constant page size where we can fetch
a given number of objects irrespective of when they
were created. This makes indexing much more faster and
efficient.
https://mattermost.atlassian.net/browse/MM-41260
```release-note
Elasticsearch and Bleve indexing have been revamped to be much
more efficient and faster. The config parameter BulkIndexingTimeWindowSeconds
for both elasticsearch and bleve have been removed.
A new config parameter called BatchSize has been introduced instead.
This parameter controls the number of objects that
can be indexed in a single batch. This makes things
more efficient and maintains a constant workload.
```
We implement a cursor based pagination model
to page through the posts in a given thread.
The cursor is a combination of the post.CreateAt+
post.Id to differentiate multiple posts in a given
timestamp.
Some additional parameters like direction, fromPost,
fromCreateAt and perPage were introduced to implement
this.
```release-note
NONE
```
Due to the way our community deployment is done. The job server
is restarted every day. This means that whenever there is a job
that takes more than 24 hours, it will always get cancelled
when the server restarts and therefore will never finish.
This PR adds ability to resume any stopped jobs, by storing
intermediate progress in the job metadata and setting
the job to pending instead of cancelled when everything is
shut down.
The user can still cancel a job explicitly by clicking on
the cross button in the system console. That functionality
hasn't changed. Only server stop or stopping/starting
job server via config will pause/resume jobs.
```release-note
The elasticsearch indexing job is resumable now. Stopping a
server while the job is running will put the job in pending status
and will resume the job when the server starts.
The job can still be explicitly cancelled via the system console UI.
```
* revamp db version and add applied migrations endpoint
* replace old schema version with new
* add db version subcommand
* add to local api
* reflect review comments
* log errors
* remove setting the version from model.CurrentVersion
* fix a test
* use different field for schema version
* add build hash and current version to the support packet
* add tests
* update test to use new assets
* MM-42282: handle teamId parameter correctly
As per https://community-daily.mattermost.com/core/pl/ugs7ue6e4j8a7cgegk1bxje8to, `ThreadStore.GetThreadsForUser` accepts a `teamId` parameter, but incorrectly handles an empty value of `""` as looking only for channels with an empty `teamId` (aka DMs and GMs) instead of finding all channels and effectively ignoring the team property.
Fixes: https://mattermost.atlassian.net/browse/MM-42282
* break up getThreadsForUser, leverage errgroup
This change breaks up `GetThreadsForUser` in the `ThreadStore` into its constituent `GetTotalUnreadThreads`, `GetTotalThreads`, `GetTotalUnreadMentions`, and the original `GetThreadsForUser` but now solely returning the thread structures. Instead of a monolithic method at the store level, the application layer now handles calling bulk requests, leveraging `errgroup` for simpler parallelization.
This change brings with it a few benefits:
* Simpler code, including more idiomatic usage of squirrel
* Simpler SQL, joining tables only when configured conditions require same. (No performance benefit here, since an unused LEFT JOIN generally has no overhead.)
* Discrete Grafana metrics for each store method, giving us better insight into the performance characteristics in play.
* **Performance boost**: reduced overhead when clearing push notifications.
This last point is what prompted the re-re-reactoring in this PR. As I broke things up, I realized that `clearPushNotificationSync` only used the `TotalUnreadMentions`, but asked for the count of total threads and total unread threads. By exposing the discrete methods, this code path avoids two aggregate queries. We clear notifications when marking a thread as read, and when marking a channel with unread mentions as viewed, so I expect we'll see at least a modest boost to performance from simply not wasting these cycles anymore.
No performance improvements are expected from this PR for the general case of using `GetThreadsForUser` to populate the threads view.
* never discard errors from building queries
* no MustSql
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* about to merge with master.
* merge from latest master.
* refactor squirrel function names to match original. clean up extra debugging.
* fix lint.
* reverted websocket_norace_test
* Fix issue of incorrect error being returned.
* made changes from annotated code.
* fix small lint issue.
* remove comment.
* cleaned up code some more.
* fix error with archived tests.
* Cleanup
```release-note
NONE
```
* more cleanup
```release-note
NONE
```
* address review comments
```release-note
NONE
```
* sq.Eq optimization
```release-note
NONE
```
Co-authored-by: Agniva De Sarker <agnivade@yahoo.co.in>
* Ported GetMembersByIds and GetMembersByChannelIds to Squirrel
* Added checks for empty ID lists
* Updated GetAllChannelMembersById to use Squirrel
* Added a method to get the query placeholder directly
* Migrated UpdateMultipleMembers to use a Squirrel query
* Migrated UpdateMultipleMembers to Squirrel
* Initialize a prepared query builder for channelMembersForTeamWithSchemeSelectQuery
* Migrated GetMembersForUser to Squirrel
* Slight improvement for constructMySQLJSONArgs
* Migrated UpdateMemberNotifyProps to Squirrel
* Migrated GetMembers to Squirrel
* Migrated GetMember to Squirrel
* Avoid shadowing err
* Don't set query builder on copy of SqlChannelStore
* Fixed typo in error message
* Fixed missing elipsis
* Shorter SQL generation for GetAllChannelMembersById
* Don't unnecessarily copy the reference
* Use a function to generate the ChannelMember map
* Avoid shadowing err
* Don't use IN for matching multiple possible values
* Initialize the members lists
* Remove check for empty channelIDs list
* Fixed test and removed check for empty list
* Use err2 rather than eerr
Co-authored-by: Tim Scheuermann <tim@plusmid.dev>
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* Revert "[MM-41576] Revamp database schema version (#19586)"
This reverts commit 645fee3fe3.
* Revert "MM-42049 - license endpoint not working (#19686)"
This reverts commit 4fe89e5847.
* revamp db version and add applied migrations endpoint
* replace old schema version with new
* add db version subcommand
* add to local api
* reflect review comments
* log errors
* remove setting the version from model.CurrentVersion
* fix a test
* deadcode: remove UpdateChannelLastViewedAt
* deadcode: remove ThreadStore.(Save(Multiple)|Update|Delete)
* deadcode: followThead in App.MarkChannelAsUnreadFromPost
* document ThreadMembership, Thread structs
* maintain LastUpdated consistently
Whenever we touch a `ThreadMembership` record, we should be setting `LastUpdated` to the current timestamp. The mobile client relies on this to detect changes to these records.
* simplify: never updateThreads from `App.MarkChannelAsUnreadFromPost`
Change all invocations of `ChannelStore.UpdateLastViewedAtPost` from `App.MarkChannelAsUnreadFromPost` to pass `updateThreads` as `false`. When `ChannelStore.UpdateLastViewedAtPost` was invoked with `updateThreads` as `true`, it would in turn call `ThreadStore.UpdateUnreadsByChannel` but pass `updateViewedTimestamp` as `false`. This effectively updated the `LastUpdated` field of the corresponding thread memberships but never touched any of the actual data (such as `LastViewed`).
The overall CRT feature continued to work, because `App.MarkChannelAsUnreadFromPost` directly updates the relevant thread memberships via `ThreadStore.MaintainMembership`.
* deadcode: updateThreads in ChannelStore.UpdateLastViewedAtPost
* simplify: never updateThreads from App.SendNotifications
Change all invocations of `ChannelStore.IncrementMentionCount` from
`App.SendNotifications` to pass `updateThreads` as `false`. When `ChannelStore.IncrementMentionCount` was invoked with `updateThreads` as `true`, it would in turn call `ThreadStore.UpdateUnreadsByChannel` but pass `updateViewedTimestamp` as `false`. This effectively updated the `LastUpdated` field of the corresponding thread memberships but never touched any of the actual data (such as `UnreadMentions`).
The overall CRT feature continued to work, because `App.SendNotifications` directly updates the relevant thread memberships mention counts via `ThreadStore.MaintainMembership`.
* deadcode: updateThreads in ChannelStore.IncrementMentionCount
* fix & rename ThreadStore.UpdateUnreadsByChannel
Rename `ThreadStore.UpdateUnreadsByChannel` to `ThreadStore.UpdateLastViewedByThreadIds`, making it unconditionally set the `LastViewed` for the given threads (as well as `LastUpdated`).
All previous invocations of this method that passed `updateViewedTimestamp` have been previously removed.
* unrelated gofmt -w -s changes to satisfy linter
* always set LastUpdated to model.GetMillis()
* deadcode: ThreadStore.SaveMembership
* fix TestMarkUnreadWithThreads
* MM-40302: CRT, use updateThreads param vs. MarkAllAsReadInChannels
`MarkAllAsReadInChannels` was the subject of a significant performance regression in v5.37 and is known to be very inefficient, by virtue of always writing to an ever increasing number of rows, and doing so on common events like simply viewing a channel.
Fortunately, `ChannelStore.UpdateLastViewedAt` already supported an `updateThreads` parameter that implemented the start of an improved algorithm: query the set of threads with newer posts, and then update only /those/. Missing was the need to reset the `UnreadMentions`, but thanks to the previous simplifications in #19523, we can make this change largely without impacting other semantics.
Fixes: https://mattermost.atlassian.net/browse/MM-40302
* fix MySQL
* remove another JOIN
* remove outdated comment
* unit tests
Previously, we were incrementing mentions one-by-one
all concurrently in an unbounded fashion.
This would cause a big spike in memory usage if there
were an `@all` mention in a large channel.
We fix this by changing the SQL query to take all userIDs
at once.
https://mattermost.atlassian.net/browse/MM-41752
```release-note
NONE
```
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* deadcode: remove UpdateChannelLastViewedAt
* deadcode: remove ThreadStore.(Save(Multiple)|Update|Delete)
* deadcode: followThead in App.MarkChannelAsUnreadFromPost
* document ThreadMembership, Thread structs
* maintain LastUpdated consistently
Whenever we touch a `ThreadMembership` record, we should be setting `LastUpdated` to the current timestamp. The mobile client relies on this to detect changes to these records.
* simplify: never updateThreads from `App.MarkChannelAsUnreadFromPost`
Change all invocations of `ChannelStore.UpdateLastViewedAtPost` from `App.MarkChannelAsUnreadFromPost` to pass `updateThreads` as `false`. When `ChannelStore.UpdateLastViewedAtPost` was invoked with `updateThreads` as `true`, it would in turn call `ThreadStore.UpdateUnreadsByChannel` but pass `updateViewedTimestamp` as `false`. This effectively updated the `LastUpdated` field of the corresponding thread memberships but never touched any of the actual data (such as `LastViewed`).
The overall CRT feature continued to work, because `App.MarkChannelAsUnreadFromPost` directly updates the relevant thread memberships via `ThreadStore.MaintainMembership`.
* deadcode: updateThreads in ChannelStore.UpdateLastViewedAtPost
* simplify: never updateThreads from App.SendNotifications
Change all invocations of `ChannelStore.IncrementMentionCount` from
`App.SendNotifications` to pass `updateThreads` as `false`. When `ChannelStore.IncrementMentionCount` was invoked with `updateThreads` as `true`, it would in turn call `ThreadStore.UpdateUnreadsByChannel` but pass `updateViewedTimestamp` as `false`. This effectively updated the `LastUpdated` field of the corresponding thread memberships but never touched any of the actual data (such as `UnreadMentions`).
The overall CRT feature continued to work, because `App.SendNotifications` directly updates the relevant thread memberships mention counts via `ThreadStore.MaintainMembership`.
* deadcode: updateThreads in ChannelStore.IncrementMentionCount
* fix & rename ThreadStore.UpdateUnreadsByChannel
Rename `ThreadStore.UpdateUnreadsByChannel` to `ThreadStore.UpdateLastViewedByThreadIds`, making it unconditionally set the `LastViewed` for the given threads (as well as `LastUpdated`).
All previous invocations of this method that passed `updateViewedTimestamp` have been previously removed.
* unrelated gofmt -w -s changes to satisfy linter
* always set LastUpdated to model.GetMillis()
* deadcode: ThreadStore.SaveMembership
* fix TestMarkUnreadWithThreads
* GetMasterX
