Before gorp, to use sqlx we had to resort to using
internal structs without the `db:"-"` tag so that
it doesn't interfere with the gorp table generation.
Now that it's gone, we can use the original model structs.
```release-note
NONE
```
* MM-35298: Follow thread when added to channel
* return better error if thread doesn't exist
* update test for possible race
* use correct comparision operator
* Update morph dependency to use a newer version
* remove timeout check for migrations statements
* store/sqlstore: reset timeout for mysql while creating db for migrations
* update morph to v0.2.1
Due to the store method having named returns, the variable
was getting set to nil instead of being an empty slice.
We missed this during code review.
https://mattermost.atlassian.net/browse/MM-40594
```release-note
NONE
```
* MM-40799: Improve ReplyCount query
This PR optimizes the reply count query to use only
the indexed column to count the rows rather than using
the ID column. This results in an index-only scan
rather than an index scan. As a result, the query
can be satisfied directly by scanning the index
and there is no need to touch the heap.
Count(*) also gives the same result, but is not recommended
as it also verifies non-null columns and can take slightly
longer.
Before:
```
explain analyze SELECT p.*, (SELECT COUNT(Posts.Id) FROM Posts WHERE Posts.RootId = (CASE WHEN p.RootId = '' THEN p.Id ELSE p.RootId END) AND Posts.DeleteAt = 0) AS ReplyCount FROM Posts p WHERE (CreateAt < (SELECT CreateAt FROM Posts WHERE Id = 'bguxt6zzcjytpffgza3dc7xody') AND p.ChannelId = '5xmynf36cinrzksuzhq7my1nbc' AND DeleteAt = 0) ORDER BY p.ChannelId, DeleteAt, CreateAt DESC LIMIT 30 OFFSET 0;
QUERY PLAN
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Limit (cost=9.14..130616.04 rows=30 width=922) (actual time=19.208..19.454 rows=30 loops=1)
InitPlan 2 (returns $2)
-> Index Scan using posts_pkey on posts posts_1 (cost=0.56..8.58 rows=1 width=8) (actual time=0.028..0.030 rows=1 loops=1)
Index Cond: ((id)::text = 'bguxt6zzcjytpffgza3dc7xody'::text)
-> Index Scan Backward using idx_posts_channel_id_delete_at_create_at on posts p (cost=0.56..2720977.61 rows=625 width=922) (actual time=0.075..0.317 rows=30 loops=1)
Index Cond: (((channelid)::text = '5xmynf36cinrzksuzhq7my1nbc'::text) AND (deleteat = 0) AND (createat < $2))
SubPlan 1
-> Aggregate (cost=4349.49..4349.50 rows=1 width=8) (actual time=0.007..0.007 rows=1 loops=30)
-> Index Scan using idx_posts_root_id_delete_at on posts (cost=0.56..4346.36 rows=1253 width=27) (actual time=0.007..0.007 rows=0 loops=30)
Index Cond: (((rootid)::text = (CASE WHEN ((p.rootid)::text = ''::text) THEN p.id ELSE p.rootid END)::text) AND (deleteat = 0))
```
After:
```
explain analyze SELECT p.*, (SELECT COUNT(Posts.RootId) FROM Posts WHERE Posts.RootId = (CASE WHEN p.RootId = '' THEN p.Id ELSE p.RootId END) AND Posts.DeleteAt = 0) AS ReplyCount FROM Posts p WHERE (CreateAt < (SELECT CreateAt FROM Posts WHERE Id = 'bguxt6zzcjytpffgza3dc7xody') AND p.ChannelId = '5xmynf36cinrzksuzhq7my1nbc' AND DeleteAt = 0) ORDER BY p.ChannelId, DeleteAt, CreateAt DESC LIMIT 30 OFFSET 0;
QUERY PLAN
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Limit (cost=9.14..130616.04 rows=30 width=922) (actual time=12.936..13.188 rows=30 loops=1)
InitPlan 2 (returns $2)
-> Index Scan using posts_pkey on posts posts_1 (cost=0.56..8.58 rows=1 width=8) (actual time=0.029..0.030 rows=1 loops=1)
Index Cond: ((id)::text = 'bguxt6zzcjytpffgza3dc7xody'::text)
-> Index Scan Backward using idx_posts_channel_id_delete_at_create_at on posts p (cost=0.56..2720977.61 rows=625 width=922) (actual time=0.076..0.324 rows=30 loops=1)
Index Cond: (((channelid)::text = '5xmynf36cinrzksuzhq7my1nbc'::text) AND (deleteat = 0) AND (createat < $2))
SubPlan 1
-> Aggregate (cost=4349.49..4349.50 rows=1 width=8) (actual time=0.008..0.008 rows=1 loops=30)
-> Index Only Scan using idx_posts_root_id_delete_at on posts (cost=0.56..4346.36 rows=1253 width=8) (actual time=0.007..0.007 rows=0 loops=30)
Index Cond: ((rootid = (CASE WHEN ((p.rootid)::text = ''::text) THEN p.id ELSE p.rootid END)::text) AND (deleteat = 0))
Heap Fetches: 14
```
https://mattermost.atlassian.net/browse/MM-40799
```release-note
NONE
```
* Use COUNT(*) for faster
```release-note
NONE
```
This query deals with posts already created,
and it was originally already querying replica.
In recent performance investigations, this came up
a lot of times. I believe the change to query master
was unintentional.
```release-note
NONE
```
During v6 indexing changes, we replaced
idx_posts_root_id with idx_posts_root_id_delete_at.
This causes MySQL to trigger the index_merge path
again with PRIMARY and idx_posts_root_id_delete_at
as shown below:
```
mysql> UPDATE Posts SET DeleteAt = 1637998911685, UpdateAt = 1637998911685, Props = JSON_SET(Props, '$.deleteBy', 'buqskqrwmjnhfuqskqrwmjn4ca') Where Id = 'q38uaydtpink5f4wkmcsn8h47o' OR RootId = 'q38uaydtpink5f4wkmcsn8h47o';
Query OK, 9 rows affected (17.29 sec)
Rows matched: 10 Changed: 9 Warnings: 0
mysql> EXPLAIN UPDATE Posts SET DeleteAt = 1637998911685, UpdateAt = 1637998911685, Props = JSON_SET(Props, '$.deleteBy', 'buqskqrwmjnhfuqskqrwmjn4ca') Where Id = 'q38uaydtpink5f4wkmcsn8h47o' OR RootId = 'q38uaydtpink5f4wkmcsn8h47o'\G
*************************** 1. row ***************************
id: 1
select_type: UPDATE
table: Posts
partitions: NULL
type: index_merge
possible_keys: PRIMARY,idx_posts_root_id_delete_at
key: idx_posts_root_id_delete_at,PRIMARY
key_len: 107,106
ref: NULL
rows: 9
filtered: 100.00
Extra: Using sort_union(idx_posts_root_id_delete_at,PRIMARY); Using where; Using temporary
1 row in set, 1 warning (0.00 sec)
```
To fix the temporary sort, we order by Id
```
mysql> UPDATE Posts SET DeleteAt = 1637998911686, UpdateAt = 1637998911686, Props = JSON_SET(Props, '$.deleteBy', 'buqskqrwmjnhfuqskqrwmjn4ca') Where Id = 'q38uaydtpink5f4wkmcsn8h47o' OR RootId = 'q38uaydtpink5f4wkmcsn8h47o' ORDER BY Id;
Query OK, 9 rows affected (0.01 sec)
Rows matched: 9 Changed: 9 Warnings: 0
mysql> EXPLAIN UPDATE Posts SET DeleteAt = 1637998911686, UpdateAt = 1637998911686, Props = JSON_SET(Props, '$.deleteBy', 'buqskqrwmjnhfuqskqrwmjn4ca') Where Id = 'q38uaydtpink5f4wkmcsn8h47o' OR RootId = 'q38uaydtpink5f4wkmcsn8h47o' ORDER BY Id\G
*************************** 1. row ***************************
id: 1
select_type: UPDATE
table: Posts
partitions: NULL
type: index_merge
possible_keys: PRIMARY,idx_posts_root_id_delete_at
key: idx_posts_root_id_delete_at,PRIMARY
key_len: 107,106
ref: NULL
rows: 9
filtered: 100.00
Extra: Using sort_union(idx_posts_root_id_delete_at,PRIMARY); Using where; Using filesort
1 row in set, 1 warning (0.00 sec)
```
Postgres uses Bitmap Heap scan which does
a Bitmap OR of the index tuples and _then_
fetches the rows from the heap. This is a
much better and sophisticated way. Sadly, MySQL
will fetch the rows from the indexes first,
and then do an OR, which is why index_merge_intersection
is so bad.
See: https://developers.mattermost.com/blog/mysql-index-merge/
for more info.
```release-note
NONE
```
* Migrate Save from gorp to sqlx
* Migrate checkTeamsExist from gorp to sqlx
* Migrate checkChannelsExist from gorp to sqlx
* Migrate Patch from gorp to sqlx
* Migrate buildGetPoliciesQuery from gorp to sqlx
* Migrate Get from gorp to sqlx
* Change buildGetPolicyQuery return values
* Migrate GetAll from gorp to sqlx
* Migrate GetCount from gorp to sqlx
* Migrate Delete from gorp to sqlx
* Migrate GetChannels from gorp to sqlx
* Migrate GetChannelsCount from gorp to sqlx
* Migrate AddChannels from gorp to sqlx
* Migrate RemoveChannels from gorp to sqlx
* Migrate GetTeams from gorp to sqlx
* Migrate GetTeamsCount from gorp to sqlx
* Migrate AddTeams from gorp to sqlx
* Migrate RemoveTeams from gorp to sqlx
* Migrate DeleteOrphanedRows from gorp to sqlx
* Migrate GetTeamPoliciesForUser from gorp to sqlx
* Migrate GetTeamPoliciesCountForUser from gorp to sqlx
* Migrate GetChannelPoliciesForUser from gorp to sqlx
* Migrate GetChannelPoliciesCountForUser from gorp to sqlx
* Migrate Delete from gorp to sqlx
* Add mapper tag function before saving a record
Summary:
Since there is a db tag in this model for the ID column,
I set a mapper in this transaction to lowercase
all mapping columns before saving the record.
* Use quoted identifiers when selecting Id
* Address PR comments, replace variable declarations for their short hand syntax
* Enhance subquery implementation
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* Add playbooks related permissions.
* Add RolesGrantPermission to pluginapi
* Fixing scopes.
* New defaults.
* Fix defaults
* Fix tests.
* Fix migration.
* More test and migration fixes.
* Need to add everything to system admin too.
* Move to 63
* Feedback fixes.
* Fix system manager editing playbook permissions.
* MM-36862: removes participant from thread
Removing a participant upon last reply deleted from thread didn't work
reliably, a suspect on this is the replica lag, since we are first
deleting the post and then counting non-deleted posts of the participant
to decide on whether to delete or not.
The findings that led to this conclusion is that the reply count gets
updated but the participant is not removed (participant removal depends
on the number of replies this participant has in the thread.)
This commit fixes that by removing first the participant and then
deleting the post.
So we delete the participant if they have 1 post in that thread, and
then we delete the post, so now they have no posts in the thread.
* Makes deleting posts transactional
This commit makes deleting a post transactional and also tries to fix
permanent deletion of posts.
Currently when we permanently delete all posts by a user we don't update
the threads reply count nor the participant's array. This commit tries
to fix that.
* Adds comments on deleting posts
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* MM-39634: Migrated gorp to sqlx
* MM-39634: Removed redundant limit on get since only one value is returned
* MM-39485: Fixed returned errors + goimports
Co-authored-by: Alex <alex@rumandcode.io>
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
Summary
The addition of the TotalMsgCountRoot and MsgCountRoot columns to support CRT caused several issues with previously read threads and channels being marked as unread. Previously we attempted to fix this purely in a SQL migration [MM-35345][MM-35494] fixes for incorrect mentions and unreads for threads and channels #17803 but that turned out to be too heavy and it was decided to break up some of the fixes into async jobs.
This PR implements an async job to mark channels as read if there are no user posts since the last time the user viewed the channel.
Ticket Link
https://mattermost.atlassian.net/browse/MM-37013
