* MM-40799: Improve ReplyCount query
This PR optimizes the reply count query to use only
the indexed column to count the rows rather than using
the ID column. This results in an index-only scan
rather than an index scan. As a result, the query
can be satisfied directly by scanning the index
and there is no need to touch the heap.
Count(*) also gives the same result, but is not recommended
as it also verifies non-null columns and can take slightly
longer.
Before:
```
explain analyze SELECT p.*, (SELECT COUNT(Posts.Id) FROM Posts WHERE Posts.RootId = (CASE WHEN p.RootId = '' THEN p.Id ELSE p.RootId END) AND Posts.DeleteAt = 0) AS ReplyCount FROM Posts p WHERE (CreateAt < (SELECT CreateAt FROM Posts WHERE Id = 'bguxt6zzcjytpffgza3dc7xody') AND p.ChannelId = '5xmynf36cinrzksuzhq7my1nbc' AND DeleteAt = 0) ORDER BY p.ChannelId, DeleteAt, CreateAt DESC LIMIT 30 OFFSET 0;
QUERY PLAN
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Limit (cost=9.14..130616.04 rows=30 width=922) (actual time=19.208..19.454 rows=30 loops=1)
InitPlan 2 (returns $2)
-> Index Scan using posts_pkey on posts posts_1 (cost=0.56..8.58 rows=1 width=8) (actual time=0.028..0.030 rows=1 loops=1)
Index Cond: ((id)::text = 'bguxt6zzcjytpffgza3dc7xody'::text)
-> Index Scan Backward using idx_posts_channel_id_delete_at_create_at on posts p (cost=0.56..2720977.61 rows=625 width=922) (actual time=0.075..0.317 rows=30 loops=1)
Index Cond: (((channelid)::text = '5xmynf36cinrzksuzhq7my1nbc'::text) AND (deleteat = 0) AND (createat < $2))
SubPlan 1
-> Aggregate (cost=4349.49..4349.50 rows=1 width=8) (actual time=0.007..0.007 rows=1 loops=30)
-> Index Scan using idx_posts_root_id_delete_at on posts (cost=0.56..4346.36 rows=1253 width=27) (actual time=0.007..0.007 rows=0 loops=30)
Index Cond: (((rootid)::text = (CASE WHEN ((p.rootid)::text = ''::text) THEN p.id ELSE p.rootid END)::text) AND (deleteat = 0))
```
After:
```
explain analyze SELECT p.*, (SELECT COUNT(Posts.RootId) FROM Posts WHERE Posts.RootId = (CASE WHEN p.RootId = '' THEN p.Id ELSE p.RootId END) AND Posts.DeleteAt = 0) AS ReplyCount FROM Posts p WHERE (CreateAt < (SELECT CreateAt FROM Posts WHERE Id = 'bguxt6zzcjytpffgza3dc7xody') AND p.ChannelId = '5xmynf36cinrzksuzhq7my1nbc' AND DeleteAt = 0) ORDER BY p.ChannelId, DeleteAt, CreateAt DESC LIMIT 30 OFFSET 0;
QUERY PLAN
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Limit (cost=9.14..130616.04 rows=30 width=922) (actual time=12.936..13.188 rows=30 loops=1)
InitPlan 2 (returns $2)
-> Index Scan using posts_pkey on posts posts_1 (cost=0.56..8.58 rows=1 width=8) (actual time=0.029..0.030 rows=1 loops=1)
Index Cond: ((id)::text = 'bguxt6zzcjytpffgza3dc7xody'::text)
-> Index Scan Backward using idx_posts_channel_id_delete_at_create_at on posts p (cost=0.56..2720977.61 rows=625 width=922) (actual time=0.076..0.324 rows=30 loops=1)
Index Cond: (((channelid)::text = '5xmynf36cinrzksuzhq7my1nbc'::text) AND (deleteat = 0) AND (createat < $2))
SubPlan 1
-> Aggregate (cost=4349.49..4349.50 rows=1 width=8) (actual time=0.008..0.008 rows=1 loops=30)
-> Index Only Scan using idx_posts_root_id_delete_at on posts (cost=0.56..4346.36 rows=1253 width=8) (actual time=0.007..0.007 rows=0 loops=30)
Index Cond: ((rootid = (CASE WHEN ((p.rootid)::text = ''::text) THEN p.id ELSE p.rootid END)::text) AND (deleteat = 0))
Heap Fetches: 14
```
https://mattermost.atlassian.net/browse/MM-40799
```release-note
NONE
```
* Use COUNT(*) for faster
```release-note
NONE
```
This query deals with posts already created,
and it was originally already querying replica.
In recent performance investigations, this came up
a lot of times. I believe the change to query master
was unintentional.
```release-note
NONE
```
* [MM-40407] - Do not show the Renew Now if the license id does not exist in the portal
* improvements
* impl mock
* fix translations
* feedback impl
* fix typo
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* MM-40537: Bump bleve dependency to fix Sentry crash
We observed a sentry crash, which is now fixed upstream.
So we bump the dependency accordingly.
```release-note
NONE
```
* Fix vendor/modules.txt
```release-note
NONE
```
* New fix
```release-note
NONE
```
* Trigger CI
```release-note
NONE
```
During v6 indexing changes, we replaced
idx_posts_root_id with idx_posts_root_id_delete_at.
This causes MySQL to trigger the index_merge path
again with PRIMARY and idx_posts_root_id_delete_at
as shown below:
```
mysql> UPDATE Posts SET DeleteAt = 1637998911685, UpdateAt = 1637998911685, Props = JSON_SET(Props, '$.deleteBy', 'buqskqrwmjnhfuqskqrwmjn4ca') Where Id = 'q38uaydtpink5f4wkmcsn8h47o' OR RootId = 'q38uaydtpink5f4wkmcsn8h47o';
Query OK, 9 rows affected (17.29 sec)
Rows matched: 10 Changed: 9 Warnings: 0
mysql> EXPLAIN UPDATE Posts SET DeleteAt = 1637998911685, UpdateAt = 1637998911685, Props = JSON_SET(Props, '$.deleteBy', 'buqskqrwmjnhfuqskqrwmjn4ca') Where Id = 'q38uaydtpink5f4wkmcsn8h47o' OR RootId = 'q38uaydtpink5f4wkmcsn8h47o'\G
*************************** 1. row ***************************
id: 1
select_type: UPDATE
table: Posts
partitions: NULL
type: index_merge
possible_keys: PRIMARY,idx_posts_root_id_delete_at
key: idx_posts_root_id_delete_at,PRIMARY
key_len: 107,106
ref: NULL
rows: 9
filtered: 100.00
Extra: Using sort_union(idx_posts_root_id_delete_at,PRIMARY); Using where; Using temporary
1 row in set, 1 warning (0.00 sec)
```
To fix the temporary sort, we order by Id
```
mysql> UPDATE Posts SET DeleteAt = 1637998911686, UpdateAt = 1637998911686, Props = JSON_SET(Props, '$.deleteBy', 'buqskqrwmjnhfuqskqrwmjn4ca') Where Id = 'q38uaydtpink5f4wkmcsn8h47o' OR RootId = 'q38uaydtpink5f4wkmcsn8h47o' ORDER BY Id;
Query OK, 9 rows affected (0.01 sec)
Rows matched: 9 Changed: 9 Warnings: 0
mysql> EXPLAIN UPDATE Posts SET DeleteAt = 1637998911686, UpdateAt = 1637998911686, Props = JSON_SET(Props, '$.deleteBy', 'buqskqrwmjnhfuqskqrwmjn4ca') Where Id = 'q38uaydtpink5f4wkmcsn8h47o' OR RootId = 'q38uaydtpink5f4wkmcsn8h47o' ORDER BY Id\G
*************************** 1. row ***************************
id: 1
select_type: UPDATE
table: Posts
partitions: NULL
type: index_merge
possible_keys: PRIMARY,idx_posts_root_id_delete_at
key: idx_posts_root_id_delete_at,PRIMARY
key_len: 107,106
ref: NULL
rows: 9
filtered: 100.00
Extra: Using sort_union(idx_posts_root_id_delete_at,PRIMARY); Using where; Using filesort
1 row in set, 1 warning (0.00 sec)
```
Postgres uses Bitmap Heap scan which does
a Bitmap OR of the index tuples and _then_
fetches the rows from the heap. This is a
much better and sophisticated way. Sadly, MySQL
will fetch the rows from the indexes first,
and then do an OR, which is why index_merge_intersection
is so bad.
See: https://developers.mattermost.com/blog/mysql-index-merge/
for more info.
```release-note
NONE
```
The TestValidateToken was actually testing
the underlying library rather than testing our code.
So, in a way, it was ineffective.
The actual flaky test was that in a rare care,
the random string generated might actually lead
to 000000 being an actual code rather than invalid.
So using random strings is fundamentally incorrect.
Even the tests in the library use hardcoded strings
and not random strings.
To fix this properly would be to use hardcoded strings,
but then we would just be testing the library
and not our code. To keep things simple,
we just keep the test to verify the error message
and remove the others.
https://community-daily.mattermost.com/boards/workspace/zyoahc9uapdn3xdptac6jb69ic/285b80a3-257d-41f6-8cf4-ed80ca9d92e5/495cdb4d-c13a-4992-8eb9-80cfee2819a4/cidjhgy1ag3yktn5eszae1489yr
```release-note
NONE
```
* Migrate Save from gorp to sqlx
* Migrate checkTeamsExist from gorp to sqlx
* Migrate checkChannelsExist from gorp to sqlx
* Migrate Patch from gorp to sqlx
* Migrate buildGetPoliciesQuery from gorp to sqlx
* Migrate Get from gorp to sqlx
* Change buildGetPolicyQuery return values
* Migrate GetAll from gorp to sqlx
* Migrate GetCount from gorp to sqlx
* Migrate Delete from gorp to sqlx
* Migrate GetChannels from gorp to sqlx
* Migrate GetChannelsCount from gorp to sqlx
* Migrate AddChannels from gorp to sqlx
* Migrate RemoveChannels from gorp to sqlx
* Migrate GetTeams from gorp to sqlx
* Migrate GetTeamsCount from gorp to sqlx
* Migrate AddTeams from gorp to sqlx
* Migrate RemoveTeams from gorp to sqlx
* Migrate DeleteOrphanedRows from gorp to sqlx
* Migrate GetTeamPoliciesForUser from gorp to sqlx
* Migrate GetTeamPoliciesCountForUser from gorp to sqlx
* Migrate GetChannelPoliciesForUser from gorp to sqlx
* Migrate GetChannelPoliciesCountForUser from gorp to sqlx
* Migrate Delete from gorp to sqlx
* Add mapper tag function before saving a record
Summary:
Since there is a db tag in this model for the ID column,
I set a mapper in this transaction to lowercase
all mapping columns before saving the record.
* Use quoted identifiers when selecting Id
* Address PR comments, replace variable declarations for their short hand syntax
* Enhance subquery implementation
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* MM-40469: Handle invalid response codes from plugins
This was an interesting crash detected via Sentry.
Typically, any HTTP status code outside 100-999 range
will cause a crash in the HTTP server. And `ExecuteCommand`
is the only plugin hook which returns a model.AppError
instead of error.
So an incorrect plugin implementation could return
a status code of 0, and crash the server. We handle that
by rewriting any illegal response code to 500.
https://mattermost.atlassian.net/browse/MM-40469
```release-note
NONE
```
* added warning
```release-note
NONE
```
* Add playbooks related permissions.
* Add RolesGrantPermission to pluginapi
* Fixing scopes.
* New defaults.
* Fix defaults
* Fix tests.
* Fix migration.
* More test and migration fixes.
* Need to add everything to system admin too.
* Move to 63
* Feedback fixes.
* Fix system manager editing playbook permissions.
* Enable receiving binary websocket messages
* Improve error message
* Prefer anonymous declaration
* Simplify
* Improve test
* Use MessagePack to clone WebSocketRequest struct
* Use short form
* Fix test
