Previously, we were incrementing mentions one-by-one
all concurrently in an unbounded fashion.
This would cause a big spike in memory usage if there
were an `@all` mention in a large channel.
We fix this by changing the SQL query to take all userIDs
at once.
https://mattermost.atlassian.net/browse/MM-41752
```release-note
NONE
```
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* deadcode: remove UpdateChannelLastViewedAt
* deadcode: remove ThreadStore.(Save(Multiple)|Update|Delete)
* deadcode: followThead in App.MarkChannelAsUnreadFromPost
* document ThreadMembership, Thread structs
* maintain LastUpdated consistently
Whenever we touch a `ThreadMembership` record, we should be setting `LastUpdated` to the current timestamp. The mobile client relies on this to detect changes to these records.
* simplify: never updateThreads from `App.MarkChannelAsUnreadFromPost`
Change all invocations of `ChannelStore.UpdateLastViewedAtPost` from `App.MarkChannelAsUnreadFromPost` to pass `updateThreads` as `false`. When `ChannelStore.UpdateLastViewedAtPost` was invoked with `updateThreads` as `true`, it would in turn call `ThreadStore.UpdateUnreadsByChannel` but pass `updateViewedTimestamp` as `false`. This effectively updated the `LastUpdated` field of the corresponding thread memberships but never touched any of the actual data (such as `LastViewed`).
The overall CRT feature continued to work, because `App.MarkChannelAsUnreadFromPost` directly updates the relevant thread memberships via `ThreadStore.MaintainMembership`.
* deadcode: updateThreads in ChannelStore.UpdateLastViewedAtPost
* simplify: never updateThreads from App.SendNotifications
Change all invocations of `ChannelStore.IncrementMentionCount` from
`App.SendNotifications` to pass `updateThreads` as `false`. When `ChannelStore.IncrementMentionCount` was invoked with `updateThreads` as `true`, it would in turn call `ThreadStore.UpdateUnreadsByChannel` but pass `updateViewedTimestamp` as `false`. This effectively updated the `LastUpdated` field of the corresponding thread memberships but never touched any of the actual data (such as `UnreadMentions`).
The overall CRT feature continued to work, because `App.SendNotifications` directly updates the relevant thread memberships mention counts via `ThreadStore.MaintainMembership`.
* deadcode: updateThreads in ChannelStore.IncrementMentionCount
* fix & rename ThreadStore.UpdateUnreadsByChannel
Rename `ThreadStore.UpdateUnreadsByChannel` to `ThreadStore.UpdateLastViewedByThreadIds`, making it unconditionally set the `LastViewed` for the given threads (as well as `LastUpdated`).
All previous invocations of this method that passed `updateViewedTimestamp` have been previously removed.
* unrelated gofmt -w -s changes to satisfy linter
* always set LastUpdated to model.GetMillis()
* deadcode: ThreadStore.SaveMembership
* fix TestMarkUnreadWithThreads
* GetMasterX
* WIP
* adding initial creategroup endpoint
* fetching by group source
* fixing startup error
* updating create endpoint to take an array of user_ids, this will allow us to create the group with one request
* adding delete group endpoint and appropriate test
* adding source param for getGroups
* adding add members and delete members endpoints
* locking down crud endpoints to only be allowed for custom groups
* user search stuff
* allowing remoteid be null by changing field to pointer
* code cleanup and store level tests
* adding new tests and removing unused endpoint
* resolving conflicts
* Adds authz check for group.
* Adds authz checks to groups APIs.
* Updated create group authz tests.
* Updates delete group tests.
* Tests create group.
* Adds some tests and validations.
* adding new parameter so I can get users not in a group
* Fixed all lint warnings.
* Fix type.
* fixing search users not in group
* Fixes some lint errors.
* Moves entry in JSON array.
* Fixed SQL query.
* Fixes permission migration test.
* Fixes migration test.
* Fixes some group store tests.
* Fix test.
* Fix test.
* Revert lint change.
* Migrated CreateWithUserIds to sqlx.
* Adds tests for GetMember; migrates implementation to sqlx.
* Tests GetNonMemberUsersPage and hanles wrong group id.
* Fixes test.
* Switches GetMaster to GetMasterX.
* Switches GetReplica to GetReplicaX.
* Fixes logic.
* Fixes shadow declaration.
* Adds include_member_count to get group API endpoint.
* Adds filter_has_member param to getGroups.
* Fixes.
* Removes array of group sources.
* fixing error
* Testing reverting CreateWithUserIds back to gorp.
* Added websocket event for CreateGroupWithUserIds.
* Changed a few response status codes. Switched to correct permission.
* Added member count to ws payload for group when updating or creating.
* Adds feature flag checks for custom groups.
* Added middleware function to require license. Added config to disable custom groups.
* Change for function signature change of executePossiblyEmptyQuery.
* Lint fixes.
* Adds telemetry none comment.
* Adds translations.
* Migrated to sqlx.
* Temp. removal of translation.
* Fixed typo.
* Added an intermediary model to query with a field that is now ignored by sqlx on read queries.
* Re-used existing store struct.
* Inludes member count.
* Fix for merge error.'
* Require license for group endpoints.
* Updates translations.
* Fix shadow declaration.
* Renames permissions. Switches to new method to retrieve remoteid.
* Added WS events for upsert and delete member(s).
* Added new store error type ErrUniqueConstraint.
* Added EnableCustonGroups to the client config.
* Sanitized some user records.
* Added parameter to include_total_count for listing groups.
* Added translations.
* adding deleteAt field to getByUsers query
* Revert sanitize.
* Added uniqueness constraint error to UpdateGroup.
* Removed the FutureFeatures flag so that the feature is not enabled on old Enterprise licenses.
* Renamed function.
* Updates authz check for user search related to groups.
* Removed debug statement.
* Removed unused app method.
* Added telemetry for enable_custom_groups.
* Returns early from nil license.
* Updates test.
* Returned early to avoid nesting in (*SqlGroupStore).checkUserExist. Switched to reading from replica in (*SqlGroupStore).GetMember. Handled JSON marshal error in (*Client4).UpsertGroupMembers
* Switched to SanitizeProfile.
* Switched to model.NewInt.
* Switched from status NotImplemented to Forbidden for missing license.
* Removed deactivated users from 'exists' set.
* Revert gotool update.
* Ignored lint error that I think is invalid.
* Added the approprate access tag for disabling custom groups.
* Revert change to response status.
* Fixed refactor mistake.
* Limited the group member WS events to individual users.
* Removed WS event of deleted groups.
* Updated license check for searchUsers endpoint.
* Switched from license feature to license sku.
* Update app/group.go
Co-authored-by: Claudio Costa <cstcld91@gmail.com>
* Update app/group.go
Co-authored-by: Claudio Costa <cstcld91@gmail.com>
* Remove linter ignore comment.
* Added function to create sku-specific license.
* Fixed typo. Removed comment.
* Fixed for wrong type.
* Added missing param to client. Removed unnecessary props setting. Added test for retrieving groups by source.
* Updated some tests now that we're validating group membership not created for deactivated user.
* Fix for groups endpoint returning all group types by default.
* Changes constant names. Adds migration for all users to manage custom group members.
* Removes requirement for manage_system permission to filter user search by group.
* Added migration mock.
* Removes default permissions from custom_group_user role.
* Fixes migration.
* Fixes emoji migration test.
* fixing issue with member counts
* fixing search issue for deleted members
Co-authored-by: Benjamin Cooke <benjamincooke@Benjamins-MacBook-Pro.local>
Co-authored-by: Benjamin Cooke <benjamincooke@Benjamins-MBP.ht.home>
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
Co-authored-by: Benjamin Cooke <benjamincooke@Benjamins-MacBook-Pro.fritz.box>
Co-authored-by: Claudio Costa <cstcld91@gmail.com>
The GetTeamsUnreadForUser call would be called for every team switch.
In CRT mode, it would make a separate store call for every team, which
would run the 3 aggregate SQL queries in GetThreadsForUser.
This is suboptimal because the complexity is linearly proportional
to the number of teams.
We make the following optimizations:
1. Change the query to a single one which aggregates all teams.
2. The query originally used just 2 out of the 3 queries, so one
query was fully redundant. We remove that query in the new one.
3. Further analysis was done whether it makes sense to run the 2
queries synchronously or not. The load-tests didn't show any degradation
in running them concurrently, so we keep the same behavior.
```release-note
NONE
```
The UPSERT statement was taking unexplained gap
locks, due to which the import was somehow failing.
I still don't know exactly how this is happening.
But splitting the UPSERT into 2 different statements
in the same transaction changes the locking semantics,
and avoids the deadlock.
Naturally, this proves that even our retrylayer
is sometimes ineffective against some problems.
```release-note
NONE
```
We create a custom io.Writer to pass to the logger
instance. For now, we keep everything to debug
as all errors are surfaced back to the server via
the API.
```release-note
NONE
```
* tools updates
* Revert "tools updates"
This reverts commit 6293297b55803c5a263e200ebd80192899666ae9.
* new endpoint to get users that should potentially be guests
* checking authservice to ensure they were an email signup
* adding tests for new endpoint
* fixing translation issue
* permissions for new endpoint
* fixing tests
* fixing when domain array is empty
* fixing when domain array is empty
* removing bots from request
Co-authored-by: Benjamin Cooke <benjamincooke@Benjamins-MacBook-Pro.local>
Co-authored-by: Benjamin Cooke <benjamincooke@Benjamins-MBP.ht.home>
Co-authored-by: mkraft <martinkraft@gmail.com>
* MM-36589: provide previous values for unreads
To successfully figure out the new counts of mentions or unread replies
for CRT we need to provide previous values alongside with the new.
This is needed so we'll know how many to subtract from the total.
This commit provides those numbers upon publishing websocket ThreadUpdated
and ThreadReadChanged events.
* Fixes errors
* Removes unneeded lines
* Adds GetThreadUnreadReplyCount store method
Uses the new store method to get unread replies instead of
GetThreadForUser.
* Tests, and some changes
- Adds api4 tests to test ws events
- Uses sqlx instead of gorp for new store method
- Fixes case where previous_unread_replies could be a negative value
* Refactors tests and adds more cases
* Fixes previous and current unread counts for commenter
When a user posts a reply to a thread the unread counts had a couple of
issues.
UnreadMentions where not zeroed out, and previous unread counts where
not set correctly.
This commit tries to fix that by marking the thread as read for the
current poster after we set previous unread counts to the websocket
event data.
Also MaintainMembership should zero out UnreadMentions when we are
setting the thread as read.
* Oops
* Fixes tests by updating when maintaining membership
OK, so some tests broke because we zero UnreadMentions in the membership
when we UpdateViewedTimestamp, since the new timestamp is always now.
Some tests broke because MaintainMembership for the commenter so that
the thread is read each time commenter posts moved further down in the
SendNotification method.
BOTH those test cases are fixed with this commit. To be sincere though I
don't understand why the second one is fixed by this.
* SystemAdminUser was not part of the channel
Some tests are failing because SystemAdminUser is not part of the
team and channels.
This commit adds user to team and channels, in an effort to fix
api4/user_tests
* Fixes tests
* Fixes tests
* Addresses review comments
* Fix if clause
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
Before gorp, to use sqlx we had to resort to using
internal structs without the `db:"-"` tag so that
it doesn't interfere with the gorp table generation.
Now that it's gone, we can use the original model structs.
```release-note
NONE
```
* MM-35298: Follow thread when added to channel
* return better error if thread doesn't exist
* update test for possible race
* use correct comparision operator
* Update morph dependency to use a newer version
* remove timeout check for migrations statements
* store/sqlstore: reset timeout for mysql while creating db for migrations
* update morph to v0.2.1
Due to the store method having named returns, the variable
was getting set to nil instead of being an empty slice.
We missed this during code review.
https://mattermost.atlassian.net/browse/MM-40594
```release-note
NONE
```
* MM-40799: Improve ReplyCount query
This PR optimizes the reply count query to use only
the indexed column to count the rows rather than using
the ID column. This results in an index-only scan
rather than an index scan. As a result, the query
can be satisfied directly by scanning the index
and there is no need to touch the heap.
Count(*) also gives the same result, but is not recommended
as it also verifies non-null columns and can take slightly
longer.
Before:
```
explain analyze SELECT p.*, (SELECT COUNT(Posts.Id) FROM Posts WHERE Posts.RootId = (CASE WHEN p.RootId = '' THEN p.Id ELSE p.RootId END) AND Posts.DeleteAt = 0) AS ReplyCount FROM Posts p WHERE (CreateAt < (SELECT CreateAt FROM Posts WHERE Id = 'bguxt6zzcjytpffgza3dc7xody') AND p.ChannelId = '5xmynf36cinrzksuzhq7my1nbc' AND DeleteAt = 0) ORDER BY p.ChannelId, DeleteAt, CreateAt DESC LIMIT 30 OFFSET 0;
QUERY PLAN
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Limit (cost=9.14..130616.04 rows=30 width=922) (actual time=19.208..19.454 rows=30 loops=1)
InitPlan 2 (returns $2)
-> Index Scan using posts_pkey on posts posts_1 (cost=0.56..8.58 rows=1 width=8) (actual time=0.028..0.030 rows=1 loops=1)
Index Cond: ((id)::text = 'bguxt6zzcjytpffgza3dc7xody'::text)
-> Index Scan Backward using idx_posts_channel_id_delete_at_create_at on posts p (cost=0.56..2720977.61 rows=625 width=922) (actual time=0.075..0.317 rows=30 loops=1)
Index Cond: (((channelid)::text = '5xmynf36cinrzksuzhq7my1nbc'::text) AND (deleteat = 0) AND (createat < $2))
SubPlan 1
-> Aggregate (cost=4349.49..4349.50 rows=1 width=8) (actual time=0.007..0.007 rows=1 loops=30)
-> Index Scan using idx_posts_root_id_delete_at on posts (cost=0.56..4346.36 rows=1253 width=27) (actual time=0.007..0.007 rows=0 loops=30)
Index Cond: (((rootid)::text = (CASE WHEN ((p.rootid)::text = ''::text) THEN p.id ELSE p.rootid END)::text) AND (deleteat = 0))
```
After:
```
explain analyze SELECT p.*, (SELECT COUNT(Posts.RootId) FROM Posts WHERE Posts.RootId = (CASE WHEN p.RootId = '' THEN p.Id ELSE p.RootId END) AND Posts.DeleteAt = 0) AS ReplyCount FROM Posts p WHERE (CreateAt < (SELECT CreateAt FROM Posts WHERE Id = 'bguxt6zzcjytpffgza3dc7xody') AND p.ChannelId = '5xmynf36cinrzksuzhq7my1nbc' AND DeleteAt = 0) ORDER BY p.ChannelId, DeleteAt, CreateAt DESC LIMIT 30 OFFSET 0;
QUERY PLAN
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Limit (cost=9.14..130616.04 rows=30 width=922) (actual time=12.936..13.188 rows=30 loops=1)
InitPlan 2 (returns $2)
-> Index Scan using posts_pkey on posts posts_1 (cost=0.56..8.58 rows=1 width=8) (actual time=0.029..0.030 rows=1 loops=1)
Index Cond: ((id)::text = 'bguxt6zzcjytpffgza3dc7xody'::text)
-> Index Scan Backward using idx_posts_channel_id_delete_at_create_at on posts p (cost=0.56..2720977.61 rows=625 width=922) (actual time=0.076..0.324 rows=30 loops=1)
Index Cond: (((channelid)::text = '5xmynf36cinrzksuzhq7my1nbc'::text) AND (deleteat = 0) AND (createat < $2))
SubPlan 1
-> Aggregate (cost=4349.49..4349.50 rows=1 width=8) (actual time=0.008..0.008 rows=1 loops=30)
-> Index Only Scan using idx_posts_root_id_delete_at on posts (cost=0.56..4346.36 rows=1253 width=8) (actual time=0.007..0.007 rows=0 loops=30)
Index Cond: ((rootid = (CASE WHEN ((p.rootid)::text = ''::text) THEN p.id ELSE p.rootid END)::text) AND (deleteat = 0))
Heap Fetches: 14
```
https://mattermost.atlassian.net/browse/MM-40799
```release-note
NONE
```
* Use COUNT(*) for faster
```release-note
NONE
```
This query deals with posts already created,
and it was originally already querying replica.
In recent performance investigations, this came up
a lot of times. I believe the change to query master
was unintentional.
```release-note
NONE
```
During v6 indexing changes, we replaced
idx_posts_root_id with idx_posts_root_id_delete_at.
This causes MySQL to trigger the index_merge path
again with PRIMARY and idx_posts_root_id_delete_at
as shown below:
```
mysql> UPDATE Posts SET DeleteAt = 1637998911685, UpdateAt = 1637998911685, Props = JSON_SET(Props, '$.deleteBy', 'buqskqrwmjnhfuqskqrwmjn4ca') Where Id = 'q38uaydtpink5f4wkmcsn8h47o' OR RootId = 'q38uaydtpink5f4wkmcsn8h47o';
Query OK, 9 rows affected (17.29 sec)
Rows matched: 10 Changed: 9 Warnings: 0
mysql> EXPLAIN UPDATE Posts SET DeleteAt = 1637998911685, UpdateAt = 1637998911685, Props = JSON_SET(Props, '$.deleteBy', 'buqskqrwmjnhfuqskqrwmjn4ca') Where Id = 'q38uaydtpink5f4wkmcsn8h47o' OR RootId = 'q38uaydtpink5f4wkmcsn8h47o'\G
*************************** 1. row ***************************
id: 1
select_type: UPDATE
table: Posts
partitions: NULL
type: index_merge
possible_keys: PRIMARY,idx_posts_root_id_delete_at
key: idx_posts_root_id_delete_at,PRIMARY
key_len: 107,106
ref: NULL
rows: 9
filtered: 100.00
Extra: Using sort_union(idx_posts_root_id_delete_at,PRIMARY); Using where; Using temporary
1 row in set, 1 warning (0.00 sec)
```
To fix the temporary sort, we order by Id
```
mysql> UPDATE Posts SET DeleteAt = 1637998911686, UpdateAt = 1637998911686, Props = JSON_SET(Props, '$.deleteBy', 'buqskqrwmjnhfuqskqrwmjn4ca') Where Id = 'q38uaydtpink5f4wkmcsn8h47o' OR RootId = 'q38uaydtpink5f4wkmcsn8h47o' ORDER BY Id;
Query OK, 9 rows affected (0.01 sec)
Rows matched: 9 Changed: 9 Warnings: 0
mysql> EXPLAIN UPDATE Posts SET DeleteAt = 1637998911686, UpdateAt = 1637998911686, Props = JSON_SET(Props, '$.deleteBy', 'buqskqrwmjnhfuqskqrwmjn4ca') Where Id = 'q38uaydtpink5f4wkmcsn8h47o' OR RootId = 'q38uaydtpink5f4wkmcsn8h47o' ORDER BY Id\G
*************************** 1. row ***************************
id: 1
select_type: UPDATE
table: Posts
partitions: NULL
type: index_merge
possible_keys: PRIMARY,idx_posts_root_id_delete_at
key: idx_posts_root_id_delete_at,PRIMARY
key_len: 107,106
ref: NULL
rows: 9
filtered: 100.00
Extra: Using sort_union(idx_posts_root_id_delete_at,PRIMARY); Using where; Using filesort
1 row in set, 1 warning (0.00 sec)
```
Postgres uses Bitmap Heap scan which does
a Bitmap OR of the index tuples and _then_
fetches the rows from the heap. This is a
much better and sophisticated way. Sadly, MySQL
will fetch the rows from the indexes first,
and then do an OR, which is why index_merge_intersection
is so bad.
See: https://developers.mattermost.com/blog/mysql-index-merge/
for more info.
```release-note
NONE
```
