mirrors/mostlymatter
1
0
Форкнуть 0
Код Задачи Запросы на слияние Действия Пакеты Проекты Релизы Вики Активность
Files
cddda07de21a67c56eece1a20cd282b7eb66575a
mostlymatter/store/sqlstore
История
Agniva De Sarker ce9d5d77bb MM-50435: Improve data retention queries on MySQL (#22304) 
For some queries MySQL performed poorly until we slightly
tune the query. Here are the results:

Old query
```
// 5.7
mysql> explain SELECT Posts.Id FROM Posts LEFT JOIN Channels ON Posts.ChannelId = Channels.Id WHERE Channels.Id IS NULL LIMIT 3000;
+----+-------------+----------+------------+--------+---------------+--------------------------------+---------+----------------------------+---------+----------+--------------------------------------+
| id | select_type | table    | partitions | type   | possible_keys | key                            | key_len | ref                        | rows    | filtered | Extra                                |
+----+-------------+----------+------------+--------+---------------+--------------------------------+---------+----------------------------+---------+----------+--------------------------------------+
|  1 | SIMPLE      | Posts    | NULL       | index  | NULL          | idx_posts_channel_id_update_at | 116     | NULL                       | 6439997 |   100.00 | Using index                          |
|  1 | SIMPLE      | Channels | NULL       | eq_ref | PRIMARY       | PRIMARY                        | 106     | mattermost.Posts.ChannelId |       1 |   100.00 | Using where; Not exists; Using index |
+----+-------------+----------+------------+--------+---------------+--------------------------------+---------+----------------------------+---------+----------+--------------------------------------+
2 rows in set, 1 warning (0.16 sec)

// 8.0
| -> Limit: 3000 row(s)  (cost=13321292.71 rows=3000) (actual time=14291.753..14291.753 rows=0 loops=1)
    -> Filter: (Channels.Id is null)  (cost=13321292.71 rows=10895918) (actual time=14291.751..14291.751 rows=0 loops=1)
        -> Nested loop antijoin  (cost=13321292.71 rows=10895918) (actual time=14291.749..14291.749 rows=0 loops=1)
            -> Covering index scan on Posts using idx_posts_channel_id_update_at  (cost=1337847.05 rows=10895918) (actual time=6.258..10928.120 rows=11824842 loops=1)
            -> Single-row covering index lookup on Channels using PRIMARY (Id=Posts.ChannelId)  (cost=1.00 rows=1) (actual time=0.000..0.000 rows=1 loops=11824842)
```

New query
```
// 5.7
mysql> EXPLAIN SELECT Posts.Id FROM Posts WHERE Posts.ChannelId NOT IN (SELECT Id FROM Channels) LIMIT 3000;
+----+-------------+----------+------------+-------+---------------+--------------------------------+---------+------+---------+----------+--------------------------+
| id | select_type | table    | partitions | type  | possible_keys | key                            | key_len | ref  | rows    | filtered | Extra                    |
+----+-------------+----------+------------+-------+---------------+--------------------------------+---------+------+---------+----------+--------------------------+
|  1 | PRIMARY     | Posts    | NULL       | index | NULL          | idx_posts_channel_id_update_at | 116     | NULL | 6440001 |   100.00 | Using where; Using index |
|  2 | SUBQUERY    | Channels | NULL       | index | PRIMARY       | idx_channels_update_at         | 9       | NULL |   60964 |   100.00 | Using index              |
+----+-------------+----------+------------+-------+---------------+--------------------------------+---------+------+---------+----------+--------------------------+
2 rows in set, 1 warning (0.17 sec)

// 8.0
| -> Limit: 3000 row(s)  (cost=1337847.05 rows=3000) (actual time=8309.610..8309.610 rows=0 loops=1)
    -> Filter: <in_optimizer>(Posts.ChannelId,Posts.ChannelId in (select #2) is false)  (cost=1337847.05 rows=10895918) (actual time=8309.609..8309.609 rows=0 loops=1)
        -> Covering index scan on Posts using idx_posts_channel_id_update_at  (cost=1337847.05 rows=10895918) (actual time=0.127..2938.252 rows=11824842 loops=1)
        -> Select #2 (subquery in condition; run only once)
            -> Filter: ((Posts.ChannelId = `<materialized_subquery>`.Id))  (cost=57224.17..57224.17 rows=1) (actual time=0.014..0.014 rows=1 loops=136308)
                -> Limit: 1 row(s)  (cost=57224.07..57224.07 rows=1) (actual time=0.013..0.013 rows=1 loops=136308)
                    -> Index lookup on <materialized_subquery> using <auto_distinct_key> (Id=Posts.ChannelId)  (actual time=0.013..0.013 rows=1 loops=136308)
                        -> Materialize with deduplication  (cost=57224.07..57224.07 rows=266638) (actual time=1361.488..1361.488 rows=270959 loops=1)
                            -> Covering index scan on Channels using idx_channels_update_at  (cost=30560.27 rows=266638) (actual time=1.807..264.116 rows=270959 loops=1)
```

We can see that in the base case, MySQL does an anitjoin which is actually fine in most queries
but in this query we are selecting posts.Id, but joining on posts.channelid. In such a case, if we use a nested query,
suddenly MySQL can materialize the sub-query and do only a covering index scan on Posts. Also compare the costs.
Old one has 13321292.71 whereas new one has 1337847.05. So an order of magnitude improvement.

Old query
```
 Limit  (cost=6561.27..655145.99 rows=1 width=27) (actual time=3897.766..3902.683 rows=0 loops=1)
   ->  Gather  (cost=6561.27..655145.99 rows=1 width=27) (actual time=3702.143..3707.058 rows=0 loops=1)
         Workers Planned: 2
         Workers Launched: 2
         ->  Parallel Hash Anti Join  (cost=5561.27..654145.89 rows=1 width=27) (actual time=3631.030..3631.032 rows=0 loops=3)
               Hash Cond: ((posts.channelid)::text = (channels.id)::text)
               ->  Parallel Seq Scan on posts  (cost=0.00..627918.55 rows=5510955 width=54) (actual time=0.719..2701.470 rows=4410848 loops=3)
               ->  Parallel Hash  (cost=4874.45..4874.45 rows=54945 width=27) (actual time=193.211..193.212 rows=43956 loops=3)
                     Buckets: 262144  Batches: 1  Memory Usage: 10368kB
                     ->  Parallel Seq Scan on channels  (cost=0.00..4874.45 rows=54945 width=27) (actual time=105.575..146.071 rows=43956 loops=3)
 Planning Time: 41.365 ms
 JIT:
   Functions: 31
   Options: Inlining true, Optimization true, Expressions true, Deforming true
   Timing: Generation 11.106 ms, Inlining 237.381 ms, Optimization 169.412 ms, Emission 103.889 ms, Total 521.787 ms
 Execution Time: 4297.552 ms
```

New query
```
 Limit  (cost=1000.00..9419975.49 rows=3000 width=27)
   ->  Gather  (cost=1000.00..20761674119.53 rows=6612717 width=27)
         Workers Planned: 2
         ->  Parallel Seq Scan on posts  (cost=0.00..20761011847.83 rows=2755299 width=27)
               Filter: (NOT (SubPlan 1))
               SubPlan 1
                 ->  Materialize  (cost=0.00..7205.04 rows=131869 width=27)
                       ->  Seq Scan on channels  (cost=0.00..5643.69 rows=131869 width=27)
 JIT:
   Functions: 9
   Options: Inlining true, Optimization true, Expressions true, Deforming true
(11 rows)
```

Note: this was so bad, I couldn't even wait to finish EXPLAIN ANALYZE as it was taking more than
10 minutes to finish.

Old query
```
EXPLAIN ANALYZE SELECT Threads.PostId FROM Threads LEFT JOIN Channels ON Threads.ChannelId = Channels.Id WHERE Channels.Id IS NULL LIMIT 3000;
// 8.0
------------------+
| -> Limit: 3000 row(s)  (cost=727387.95 rows=3000) (actual time=725.905..725.905 rows=0 loops=1)
    -> Filter: (Channels.Id is null)  (cost=727387.95 rows=807045) (actual time=725.903..725.903 rows=0 loops=1)
        -> Nested loop antijoin  (cost=727387.95 rows=807045) (actual time=725.901..725.901 rows=0 loops=1)
            -> Covering index scan on Threads using idx_threads_channel_id_last_reply_at  (cost=92668.15 rows=807045) (actual time=0.201..214.332 rows=846313 loops=1)
            -> Single-row covering index lookup on Channels using PRIMARY (Id=Threads.ChannelId)  (cost=0.69 rows=1) (actual time=0.001..0.001 rows=1 loops=846313)
 |
```

New query
```
// 8.0
mysql> EXPLAIN ANALYZE  SELECT Threads.PostId FROM Threads WHERE Threads.ChannelId NOT IN (SELECT Id FROM Channels) LIMIT 3000;
 -> Limit: 3000 row(s)  (cost=92668.15 rows=3000) (actual time=1770.798..1770.798 rows=0 loops=1)
    -> Filter: <in_optimizer>(Threads.ChannelId,Threads.ChannelId in (select #2) is false)  (cost=92668.15 rows=807045) (actual time=1770.796..1770.796 rows=0 loops=1)
        -> Covering index scan on Threads using idx_threads_channel_id_last_reply_at  (cost=92668.15 rows=807045) (actual time=0.191..197.768 rows=846313 loops=1)
        -> Select #2 (subquery in condition; run only once)
            -> Filter: ((Threads.ChannelId = `<materialized_subquery>`.Id))  (cost=56081.83..56081.83 rows=1) (actual time=0.011..0.011 rows=1 loops=114202)
                -> Limit: 1 row(s)  (cost=56081.73..56081.73 rows=1) (actual time=0.011..0.011 rows=1 loops=114202)
                    -> Index lookup on <materialized_subquery> using <auto_distinct_key> (Id=Threads.ChannelId)  (actual time=0.011..0.011 rows=1 loops=114202)
                        -> Materialize with deduplication  (cost=56081.73..56081.73 rows=266638) (actual time=901.592..901.592 rows=270959 loops=1)
                            -> Covering index scan on Channels using idx_channels_update_at  (cost=29417.93 rows=266638) (actual time=0.129..57.806 rows=270959 loops=1)
 |
```

I did not test this query in Community but I expect the same logic to happen. Again, notice
the improvement in cost. 727387.95 vs 727387.95

Old query
```
EXPLAIN ANALYZE SELECT Threads.PostId FROM Threads LEFT JOIN Channels ON Threads.ChannelId = Channels.Id WHERE Channels.Id IS NULL LIMIT 3000;
 Limit  (cost=6561.27..74898.36 rows=1 width=27) (actual time=628.141..630.437 rows=0 loops=1)
   ->  Gather  (cost=6561.27..74898.36 rows=1 width=27) (actual time=628.138..630.433 rows=0 loops=1)
         Workers Planned: 2
         Workers Launched: 2
         ->  Parallel Hash Anti Join  (cost=5561.27..73898.26 rows=1 width=27) (actual time=612.672..612.674 rows=0 loops=3)
               Hash Cond: ((threads.channelid)::text = (channels.id)::text)
               ->  Parallel Seq Scan on threads  (cost=0.00..66852.00 rows=396000 width=54) (actual time=0.308..505.021 rows=315151 loops=3)
               ->  Parallel Hash  (cost=4874.45..4874.45 rows=54945 width=27) (actual time=21.031..21.031 rows=43956 loops=3)
                     Buckets: 262144  Batches: 1  Memory Usage: 10336kB
                     ->  Parallel Seq Scan on channels  (cost=0.00..4874.45 rows=54945 width=27) (actual time=0.018..7.959 rows=43956 loops=3)
```

New query
```
[bigdb] # EXPLAIN   SELECT Threads.PostId FROM Threads WHERE Threads.ChannelId NOT IN (SELECT Id FROM Channels) LIMIT 3000;
                                        QUERY PLAN
-------------------------------------------------------------------------------------------
 Limit  (cost=1000.00..9420102.76 rows=3000 width=27)
   ->  Gather  (cost=1000.00..1491986877.26 rows=475200 width=27)
         Workers Planned: 2
         ->  Parallel Seq Scan on threads  (cost=0.00..1491938357.26 rows=198000 width=27)
               Filter: (NOT (SubPlan 1))
               SubPlan 1
                 ->  Materialize  (cost=0.00..7205.04 rows=131869 width=27)
                       ->  Seq Scan on channels  (cost=0.00..5643.69 rows=131869 width=27)
 JIT:
   Functions: 9
   Options: Inlining true, Optimization true, Expressions true, Deforming true
```

As expected, Postgres behaves correctly in the original query.

https://mattermost.atlassian.net/browse/MM-50435

```release-note
NONE
```
2023-02-13 20:33:45 +05:30
..
adapters_test.go
Using native JSON operations to add thread participants (#18093)
2021-08-12 16:45:03 +02:00
adapters.go
MM-45994 ensure database operations return their errors (#20857)
2022-08-26 11:12:59 +02:00
audit_store_test.go
MM-36271: Bump major version to 6.0 (#17973)
2021-07-22 12:21:47 +05:30
audit_store.go
s/masterminds/mattermost for squirrel (#20162)
2022-05-10 11:48:59 -03:00
bot_store_test.go
MM-36271: Bump major version to 6.0 (#17973)
2021-07-22 12:21:47 +05:30
bot_store.go
MM-45994 ensure database operations return their errors (#20857)
2022-08-26 11:12:59 +02:00
channel_member_history_store_test.go
MM-36271: Bump major version to 6.0 (#17973)
2021-07-22 12:21:47 +05:30
channel_member_history_store.go
MM-45994 ensure database operations return their errors (#20857)
2022-08-26 11:12:59 +02:00
channel_store_categories_test.go
MM-36271: Bump major version to 6.0 (#17973)
2021-07-22 12:21:47 +05:30
channel_store_categories.go
Do not send categories for teams deleted or not member of (#21038)
2022-10-11 15:33:29 +02:00
channel_store_test.go
MM-19336 Migrated GetMembersByIds and GetMembersByChannelIds to Squirrel (#19728)
2022-03-11 01:27:09 +05:30
channel_store.go
MM-47537 Export and import read channels (#22145)
2023-01-31 09:18:20 +01:00
cluster_discovery_store_test.go
MM-36271: Bump major version to 6.0 (#17973)
2021-07-22 12:21:47 +05:30
cluster_discovery_store.go
MM-45994 ensure database operations return their errors (#20857)
2022-08-26 11:12:59 +02:00
command_store_test.go
MM-36271: Bump major version to 6.0 (#17973)
2021-07-22 12:21:47 +05:30
command_store.go
s/masterminds/mattermost for squirrel (#20162)
2022-05-10 11:48:59 -03:00
command_webhook_store_test.go
MM-36271: Bump major version to 6.0 (#17973)
2021-07-22 12:21:47 +05:30
command_webhook_store.go
MM-45994 ensure database operations return their errors (#20857)
2022-08-26 11:12:59 +02:00
compliance_store_test.go
MM-36271: Bump major version to 6.0 (#17973)
2021-07-22 12:21:47 +05:30
compliance_store.go
MM-49354: Make compliance export query cancellable (#21943)
2022-12-23 19:56:49 +05:30
context_test.go
MM-30882: Fix read-after-write issue for demoting user (#16911)
2021-02-12 19:04:05 +05:30
context.go
Kill gorp (#19786)
2022-03-15 19:49:19 +05:30
draft_store_test.go
MM-48614: saves priority for drafts (#21752)
2022-11-29 23:36:47 +02:00
draft_store.go
MM-48614: saves priority for drafts (#21752)
2022-11-29 23:36:47 +02:00
emoji_store_test.go
MM-36271: Bump major version to 6.0 (#17973)
2021-07-22 12:21:47 +05:30
emoji_store.go
MM-45994 ensure database operations return their errors (#20857)
2022-08-26 11:12:59 +02:00
file_info_store_test.go
MM-36271: Bump major version to 6.0 (#17973)
2021-07-22 12:21:47 +05:30
file_info_store.go
[MM-5046] Fix emails search for Postgres DB (#21590)
2022-11-22 12:02:33 -05:00
group_store_test.go
MM-36271: Bump major version to 6.0 (#17973)
2021-07-22 12:21:47 +05:30
group_store.go
Add parameter to return group members in order of display name (#21775)
2022-12-15 14:20:36 -05:00
init_test.go
Consistent license message for all the go files (#13235)
2019-11-29 12:59:40 +01:00
integrity_test.go
[MM-45868] Add teamId to Threads table (#20915)
2022-10-24 16:10:27 +04:00
integrity.go
[MM-45868] Replace TeamId column with ThreadTeamId (#21505)
2022-10-26 19:50:37 +04:00
job_store_test.go
MM-36271: Bump major version to 6.0 (#17973)
2021-07-22 12:21:47 +05:30
job_store.go
s/masterminds/mattermost for squirrel (#20162)
2022-05-10 11:48:59 -03:00
license_store_test.go
MM-36271: Bump major version to 6.0 (#17973)
2021-07-22 12:21:47 +05:30
license_store.go
s/masterminds/mattermost for squirrel (#20162)
2022-05-10 11:48:59 -03:00
link_metadata_store_test.go
MM-36271: Bump major version to 6.0 (#17973)
2021-07-22 12:21:47 +05:30
link_metadata_store.go
s/masterminds/mattermost for squirrel (#20162)
2022-05-10 11:48:59 -03:00
main_test.go
MM-36764 mlog refactor (#18118)
2021-08-17 16:08:04 -04:00
notify_admin_store_test.go
[MM-45564] - Notify Admin v2 (#20777)
2022-09-02 13:52:48 +03:00
notify_admin_store.go
[MM-48029] notify admin for plugin install from work templates (#22007)
2023-02-08 14:08:33 -07:00
oauth_store_test.go
MM-36271: Bump major version to 6.0 (#17973)
2021-07-22 12:21:47 +05:30
oauth_store.go
MM-45994 ensure database operations return their errors (#20857)
2022-08-26 11:12:59 +02:00
plugin_store_test.go
MM-36271: Bump major version to 6.0 (#17973)
2021-07-22 12:21:47 +05:30
plugin_store.go
s/masterminds/mattermost for squirrel (#20162)
2022-05-10 11:48:59 -03:00
post_acknowledgements_store_test.go
MM-47750: Adds PostAcknowledgements table and apis (#21689)
2022-11-23 19:41:23 -05:00
post_acknowledgements_store.go
MM-49845: fixes acknowledgements in getpostsince (#22124)
2023-01-25 14:45:45 +02:00
post_priority_store_test.go
MM-46410: adds urgency on mention counts (#20999)
2022-11-23 21:08:21 +02:00
post_priority_store.go
MM-46410: adds urgency on mention counts (#20999)
2022-11-23 21:08:21 +02:00
post_store_test.go
MM-36271: Bump major version to 6.0 (#17973)
2021-07-22 12:21:47 +05:30
post_store.go
MM-50435: Improve data retention queries on MySQL (#22304)
2023-02-13 20:33:45 +05:30
preference_store_test.go
Kill gorp (#19786)
2022-03-15 19:49:19 +05:30
preference_store.go
Revert "[MM-46579] Add a limit for preferences" (#22086)
2023-01-16 15:30:14 +02:00
product_notices_store_test.go
MM-36271: Bump major version to 6.0 (#17973)
2021-07-22 12:21:47 +05:30
product_notices_store.go
MM-45994 ensure database operations return their errors (#20857)
2022-08-26 11:12:59 +02:00
reaction_store_test.go
MM-36271: Bump major version to 6.0 (#17973)
2021-07-22 12:21:47 +05:30
reaction_store.go
MM-45994 ensure database operations return their errors (#20857)
2022-08-26 11:12:59 +02:00
remote_cluster_store_test.go
MM-36271: Bump major version to 6.0 (#17973)
2021-07-22 12:21:47 +05:30
remote_cluster_store.go
s/masterminds/mattermost for squirrel (#20162)
2022-05-10 11:48:59 -03:00
retention_policy_store_test.go
MM-36271: Bump major version to 6.0 (#17973)
2021-07-22 12:21:47 +05:30
retention_policy_store.go
MM-45994 ensure database operations return their errors (#20857)
2022-08-26 11:12:59 +02:00
role_store_test.go
MM-36271: Bump major version to 6.0 (#17973)
2021-07-22 12:21:47 +05:30
role_store.go
MM-45994 ensure database operations return their errors (#20857)
2022-08-26 11:12:59 +02:00
scheme_store_test.go
MM-36271: Bump major version to 6.0 (#17973)
2021-07-22 12:21:47 +05:30
scheme_store.go
New isrole replacement (#21688)
2022-11-24 07:41:09 -05:00
session_store_test.go
MM-36271: Bump major version to 6.0 (#17973)
2021-07-22 12:21:47 +05:30
session_store.go
MM-44101: GraphQL: Add sidebar categories at the top level (#20178)
2022-05-12 20:03:47 +05:30
shared_channel_store_test.go
MM-36271: Bump major version to 6.0 (#17973)
2021-07-22 12:21:47 +05:30
shared_channel_store.go
MM-45994 ensure database operations return their errors (#20857)
2022-08-26 11:12:59 +02:00
sqlx_wrapper_test.go
Added a sqlXPassThrough layer (#18238)
2021-08-30 10:20:28 +05:30
sqlx_wrapper.go
MM-49354: Make compliance export query cancellable (#21943)
2022-12-23 19:56:49 +05:30
status_store_test.go
MM-36271: Bump major version to 6.0 (#17973)
2021-07-22 12:21:47 +05:30
status_store.go
MM-45994 ensure database operations return their errors (#20857)
2022-08-26 11:12:59 +02:00
store_test.go
MM-49604: Fix race condition in test (#22128)
2023-01-25 14:43:13 +05:30
store.go
Merge branch 'master' into MM-47853-true-up-review-telemetry-off-non-air-gapped
2023-01-17 08:45:44 +02:00
system_store_test.go
MM-36271: Bump major version to 6.0 (#17973)
2021-07-22 12:21:47 +05:30
system_store.go
MM-45994 ensure database operations return their errors (#20857)
2022-08-26 11:12:59 +02:00
team_store_test.go
MM-36271: Bump major version to 6.0 (#17973)
2021-07-22 12:21:47 +05:30
team_store.go
[MM-46463] Return last_picture_update for new team members (#21758)
2022-12-07 11:16:56 +02:00
terms_of_service_store_test.go
MM-36271: Bump major version to 6.0 (#17973)
2021-07-22 12:21:47 +05:30
terms_of_service_store.go
Remove ColMap (#19544)
2022-02-15 11:44:04 +05:30
thread_store_test.go
MM-36271: Bump major version to 6.0 (#17973)
2021-07-22 12:21:47 +05:30
thread_store.go
MM-50435: Improve data retention queries on MySQL (#22304)
2023-02-13 20:33:45 +05:30
tokens_store_test.go
[GH-18965] - Pass expire time from outside Cleanup method (#19008)
2021-12-22 14:57:30 +05:30
tokens_store.go
s/masterminds/mattermost for squirrel (#20162)
2022-05-10 11:48:59 -03:00
true_up_review_store_test.go
Fix mysql migrations and db tests.
2022-12-23 11:08:41 -05:00
true_up_review_store.go
Fix typo, ensure we don't throw away errors that are not store.NotFound.
2022-12-28 10:01:20 -05:00
upload_session_store_test.go
MM-36271: Bump major version to 6.0 (#17973)
2021-07-22 12:21:47 +05:30
upload_session_store.go
[MM-48946] Fix read after write issue when uploading data (#21868)
2022-12-13 11:47:05 -06:00
user_access_token_store_test.go
MM-36271: Bump major version to 6.0 (#17973)
2021-07-22 12:21:47 +05:30
user_access_token_store.go
MM-45994 ensure database operations return their errors (#20857)
2022-08-26 11:12:59 +02:00
user_store_test.go
MM-36271: Bump major version to 6.0 (#17973)
2021-07-22 12:21:47 +05:30
user_store.go
Fix (#21816)
2023-02-02 18:52:48 +05:30
user_terms_of_service_store_test.go
MM-36271: Bump major version to 6.0 (#17973)
2021-07-22 12:21:47 +05:30
user_terms_of_service.go
MM-45994 ensure database operations return their errors (#20857)
2022-08-26 11:12:59 +02:00
utils_test.go
Use any instead of interface{} (#20577)
2022-07-05 09:46:50 +03:00
utils.go
MM-45994 ensure database operations return their errors (#20857)
2022-08-26 11:12:59 +02:00
webhook_store_test.go
MM-36271: Bump major version to 6.0 (#17973)
2021-07-22 12:21:47 +05:30
webhook_store.go
remove webhook queries in favor of using existing ones that ignore deleted webhooks, enforce not implemented for cloud users, get seats from license.features.users.
2022-12-21 12:09:33 -05:00