mirrors/mostlymatter
1
0
Форкнуть 0
Код Задачи Запросы на слияние Действия Пакеты Проекты Релизы Вики Активность
Files
971af6935c509191ab3a32d5d35435d60dab64f4
mostlymatter/store/sqlstore
История
Agniva De Sarker 971af6935c Optimize TotalUnreadThreads (#18788) 
We remove the left joins with inner joins and also
remove the coalesce on posts.DeleteAt which improves
the join plan from a nested loop to a hash join,
and also allows the DeleteAt to be used as part of the
index filter.

Old plan

```
 Aggregate  (cost=666288.49..666288.50 rows=1 width=8) (actual time=5114.328..5119.991 rows=1 loops=1)
   ->  Gather  (cost=2880.69..666288.27 rows=87 width=8) (actual time=614.214..5119.575 rows=533 loops=1)
         Workers Planned: 2
         Workers Launched: 2
         ->  Nested Loop  (cost=1880.69..665279.57 rows=36 width=8) (actual time=607.821..5099.078 rows=178 loops=3)
               ->  Hash Join  (cost=1880.28..664932.93 rows=229 width=16) (actual time=224.374..1494.698 rows=1866781 loops=3)
                     Hash Cond: ((posts.channelid)::text = (channels.id)::text)
                     ->  Parallel Seq Scan on posts  (cost=0.00..662980.29 rows=27568 width=43) (actual time=201.053..1013.362 rows=4410819 loops=3)
                           Filter: (COALESCE(deleteat, '0'::bigint) = 0)
                     ->  Hash  (cost=1866.60..1866.60 rows=1094 width=27) (actual time=1.486..1.486 rows=1002 loops=3)
                           Buckets: 2048  Batches: 1  Memory Usage: 74kB
                           ->  Index Scan using idx_channels_team_id_type on channels  (cost=0.42..1866.60 rows=1094 width=27) (actual time=0.028..1.331 rows=1002 loops=3)
                                 Index Cond: ((teamid)::text = '8ywxyw9ocp8smxrmjzrkqhrdwe'::text)
               ->  Index Scan using threadmemberships_pkey on threadmemberships  (cost=0.41..1.50 rows=1 width=35) (actual time=0.002..0.002 rows=0 loops=5600344)
                     Index Cond: (((postid)::text = (posts.rootid)::text) AND ((userid)::text = 'tc3p1yqw67d8idcp3g98awexqe'::text))
                     Filter: (following AND (posts.createat > lastviewed))
                     Rows Removed by Filter: 0
 Planning Time: 2.005 ms
 JIT:
   Functions: 59
   Options: Inlining true, Optimization true, Expressions true, Deforming true
   Timing: Generation 9.905 ms, Inlining 122.509 ms, Optimization 293.764 ms, Emission 186.064 ms, Total 612.241 ms
 Execution Time: 5125.739 ms
```

New plan

```
 Aggregate  (cost=683129.17..683129.18 rows=1 width=8) (actual time=1650.746..1656.455 rows=1 loops=1)
   ->  Gather  (cost=3766.65..683085.55 rows=17450 width=27) (actual time=641.073..1656.086 rows=533 loops=1)
         Workers Planned: 2
         Workers Launched: 2
         ->  Hash Join  (cost=2766.65..680340.55 rows=7271 width=27) (actual time=617.096..1635.203 rows=178 loops=3)
               Hash Cond: ((posts.rootid)::text = (threadmemberships.postid)::text)
               Join Filter: (posts.createat > threadmemberships.lastviewed)
               Rows Removed by Join Filter: 6
               ->  Hash Join  (cost=1880.28..679334.08 rows=45742 width=16) (actual time=15.196..1242.832 rows=1866781 loops=3)
                     Hash Cond: ((posts.channelid)::text = (channels.id)::text)
                     ->  Parallel Seq Scan on posts  (cost=0.00..662980.29 rows=5513543 width=43) (actual time=0.028..775.118 rows=4410819 loops=3)
                           Filter: (deleteat = 0)
                     ->  Hash  (cost=1866.60..1866.60 rows=1094 width=27) (actual time=1.572..1.573 rows=1002 loops=3)
                           Buckets: 2048  Batches: 1  Memory Usage: 74kB
                           ->  Index Scan using idx_channels_team_id_type on channels  (cost=0.42..1866.60 rows=1094 width=27) (actual time=0.025..1.372 rows=1002 loops=3)
                                 Index Cond: ((teamid)::text = '8ywxyw9ocp8smxrmjzrkqhrdwe'::text)
               ->  Hash  (cost=823.41..823.41 rows=5037 width=35) (actual time=256.240..256.241 rows=5007 loops=3)
                     Buckets: 8192  Batches: 1  Memory Usage: 417kB
                     ->  Bitmap Heap Scan on threadmemberships  (cost=159.45..823.41 rows=5037 width=35) (actual time=253.011..255.255 rows=5007 loops=3)
                           Recheck Cond: ((userid)::text = 'tc3p1yqw67d8idcp3g98awexqe'::text)
                           Filter: following
                           Heap Blocks: exact=601
                           ->  Bitmap Index Scan on idx_thread_memberships_user_id  (cost=0.00..158.19 rows=5037 width=0) (actual time=0.627..0.627 rows=5037 loops=3)
                                 Index Cond: ((userid)::text = 'tc3p1yqw67d8idcp3g98awexqe'::text)
 Planning Time: 1.952 ms
 JIT:
   Functions: 83
   Options: Inlining true, Optimization true, Expressions true, Deforming true
   Timing: Generation 10.148 ms, Inlining 115.819 ms, Optimization 395.923 ms, Emission 244.130 ms, Total 766.021 ms
 Execution Time: 1661.791 ms
```

No special test has been done on MySQL since these are pretty
basic optimizations and should apply to both equally.

https://community-daily.mattermost.com/boards/workspace/zyoahc9uapdn3xdptac6jb69ic/285b80a3-257d-41f6-8cf4-ed80ca9d92e5/495cdb4d-c13a-4992-8eb9-80cfee2819a4?c=c9aa17f3-b918-4a78-86aa-01562e418634

```release-note
NONE
```
2021-10-26 08:56:38 +05:30
..
adapters_test.go
Using native JSON operations to add thread participants (#18093)
2021-08-12 16:45:03 +02:00
adapters.go
[MM-22051] Remove To/From JSON (#18070)
2021-09-01 14:43:12 +02:00
audit_store_test.go
MM-36271: Bump major version to 6.0 (#17973)
2021-07-22 12:21:47 +05:30
audit_store.go
Migrate from gorp to sqlx in store/sqlstore/audit_store.go (#18409)
2021-09-20 18:55:36 +05:30
bot_store_test.go
MM-36271: Bump major version to 6.0 (#17973)
2021-07-22 12:21:47 +05:30
bot_store.go
Migrating bot store to use sqlx (#18300)
2021-09-02 17:08:11 +02:00
channel_member_history_store_test.go
MM-36271: Bump major version to 6.0 (#17973)
2021-07-22 12:21:47 +05:30
channel_member_history_store.go
MM-38624 Migrate ChannelMemberHistoryStore to sqlx (#18496)
2021-10-06 19:50:02 +02:00
channel_store_categories_test.go
MM-36271: Bump major version to 6.0 (#17973)
2021-07-22 12:21:47 +05:30
channel_store_categories.go
[MM-28985] Remove pointers to slice (part 1) (#18034)
2021-08-17 11:18:33 +02:00
channel_store_test.go
MM-36271: Bump major version to 6.0 (#17973)
2021-07-22 12:21:47 +05:30
channel_store.go
Fix migration scripts (#18756)
2021-10-19 19:56:09 +05:30
cluster_discovery_store_test.go
MM-36271: Bump major version to 6.0 (#17973)
2021-07-22 12:21:47 +05:30
cluster_discovery_store.go
Cluster Discovery store sqlx migration (#18605)
2021-10-07 16:50:03 +02:00
command_store_test.go
MM-36271: Bump major version to 6.0 (#17973)
2021-07-22 12:21:47 +05:30
command_store.go
Migrate from gorp to sqlx in store/sqlstore/command_store.go (#18728)
2021-10-21 09:57:33 +05:30
command_webhook_store_test.go
MM-36271: Bump major version to 6.0 (#17973)
2021-07-22 12:21:47 +05:30
command_webhook_store.go
Migrate from gorp to sqlx in store/sqlstore/command_webhook_store.go (#18517)
2021-10-05 21:22:50 +05:30
compliance_store_test.go
MM-36271: Bump major version to 6.0 (#17973)
2021-07-22 12:21:47 +05:30
compliance_store.go
[MM-38996] - Migrate from gorp to sqlx in store/sqlstore/compliance_store.go (#18633)
2021-10-15 20:06:57 +05:30
context_test.go
MM-30882: Fix read-after-write issue for demoting user (#16911)
2021-02-12 19:04:05 +05:30
context.go
Migrating Emoji store to use sqlx (#18301)
2021-09-14 13:46:42 +05:30
emoji_store_test.go
MM-36271: Bump major version to 6.0 (#17973)
2021-07-22 12:21:47 +05:30
emoji_store.go
Migrating Emoji store to use sqlx (#18301)
2021-09-14 13:46:42 +05:30
file_info_store_test.go
MM-36271: Bump major version to 6.0 (#17973)
2021-07-22 12:21:47 +05:30
file_info_store.go
[GH-18504] Migrate from gorp to sqlx in store/sqlstore/file_info_store.go (#18637)
2021-10-19 19:29:10 +05:30
group_store_test.go
MM-36271: Bump major version to 6.0 (#17973)
2021-07-22 12:21:47 +05:30
group_store.go
MM-36271: Bump major version to 6.0 (#17973)
2021-07-22 12:21:47 +05:30
init_test.go
Consistent license message for all the go files (#13235)
2019-11-29 12:59:40 +01:00
integrity_test.go
[MM-28985] Remove pointers to slice (part 1) (#18034)
2021-08-17 11:18:33 +02:00
integrity.go
[MM-29217] Remove Posts.ParentId (#17923)
2021-08-17 11:25:22 +03:00
job_store_test.go
MM-36271: Bump major version to 6.0 (#17973)
2021-07-22 12:21:47 +05:30
job_store.go
Migrate from gorp to sqlx in store/sqlstore/job_store.go (#18518)
2021-10-19 19:29:24 +05:30
license_store_test.go
MM-36271: Bump major version to 6.0 (#17973)
2021-07-22 12:21:47 +05:30
license_store.go
MM-36271: Bump major version to 6.0 (#17973)
2021-07-22 12:21:47 +05:30
link_metadata_store_test.go
MM-36271: Bump major version to 6.0 (#17973)
2021-07-22 12:21:47 +05:30
link_metadata_store.go
[MM-39229] - Migrate from gorp to sqlx in store/sqlstore/link_metadata_store.go (#18702)
2021-10-19 19:49:45 +05:30
main_test.go
MM-36764 mlog refactor (#18118)
2021-08-17 16:08:04 -04:00
oauth_store_test.go
MM-36271: Bump major version to 6.0 (#17973)
2021-07-22 12:21:47 +05:30
oauth_store.go
Migrating OAuthStore to sqlx (#18302)
2021-09-13 19:23:58 +05:30
plugin_store_test.go
MM-36271: Bump major version to 6.0 (#17973)
2021-07-22 12:21:47 +05:30
plugin_store.go
Migrated from gorp to sqlx in store/sqlstore/plugin_store.go (#18729)
2021-10-21 09:57:49 +05:30
post_store_test.go
MM-36271: Bump major version to 6.0 (#17973)
2021-07-22 12:21:47 +05:30
post_store.go
[MM-39157] Thread participant removal with just one query using DB's JSON functions (#18725)
2021-10-19 14:07:17 -04:00
preference_store_test.go
[MM-28985] Remove pointers to slice (part 1) (#18034)
2021-08-17 11:18:33 +02:00
preference_store.go
[MM-28985] Remove pointers to slice (part 1) (#18034)
2021-08-17 11:18:33 +02:00
product_notices_store_test.go
MM-36271: Bump major version to 6.0 (#17973)
2021-07-22 12:21:47 +05:30
product_notices_store.go
[MM-28985] Remove pointers to slice (part 1) (#18034)
2021-08-17 11:18:33 +02:00
reaction_store_test.go
MM-36271: Bump major version to 6.0 (#17973)
2021-07-22 12:21:47 +05:30
reaction_store.go
[MM-39232] Migrate from gorp to sqlx in store/sqlstore/reaction_store.go (#18657)
2021-10-21 09:55:49 +05:30
remote_cluster_store_test.go
MM-36271: Bump major version to 6.0 (#17973)
2021-07-22 12:21:47 +05:30
remote_cluster_store.go
MM-36271: Bump major version to 6.0 (#17973)
2021-07-22 12:21:47 +05:30
retention_policy_store_test.go
MM-36271: Bump major version to 6.0 (#17973)
2021-07-22 12:21:47 +05:30
retention_policy_store.go
MM-36271: Bump major version to 6.0 (#17973)
2021-07-22 12:21:47 +05:30
role_store_test.go
MM-36271: Bump major version to 6.0 (#17973)
2021-07-22 12:21:47 +05:30
role_store.go
MM-36271: Bump major version to 6.0 (#17973)
2021-07-22 12:21:47 +05:30
scheme_store_test.go
MM-36271: Bump major version to 6.0 (#17973)
2021-07-22 12:21:47 +05:30
scheme_store.go
MM-36271: Bump major version to 6.0 (#17973)
2021-07-22 12:21:47 +05:30
session_store_test.go
MM-36271: Bump major version to 6.0 (#17973)
2021-07-22 12:21:47 +05:30
session_store.go
[MM-36472] Fix inconsistencies in Roles columns (#18390)
2021-10-01 15:31:29 +02:00
shared_channel_store_test.go
MM-36271: Bump major version to 6.0 (#17973)
2021-07-22 12:21:47 +05:30
shared_channel_store.go
MM-38611 getSharedChannels: only return channels user is member of (#18417)
2021-09-20 15:00:47 -04:00
sqlx_wrapper_test.go
Added a sqlXPassThrough layer (#18238)
2021-08-30 10:20:28 +05:30
sqlx_wrapper.go
Migrating bot store to use sqlx (#18300)
2021-09-02 17:08:11 +02:00
status_store_test.go
MM-36271: Bump major version to 6.0 (#17973)
2021-07-22 12:21:47 +05:30
status_store.go
[MM-36387] Address db performance audit items (#18078)
2021-08-21 08:05:52 +03:00
store_test.go
MM-36271: Bump major version to 6.0 (#17973)
2021-07-22 12:21:47 +05:30
store.go
[GH-18504] Migrate from gorp to sqlx in store/sqlstore/file_info_store.go (#18637)
2021-10-19 19:29:10 +05:30
system_store_test.go
MM-36271: Bump major version to 6.0 (#17973)
2021-07-22 12:21:47 +05:30
system_store.go
MM-36990: Use upsert for system key update (#18037)
2021-08-03 14:23:47 +05:30
team_store_test.go
MM-36271: Bump major version to 6.0 (#17973)
2021-07-22 12:21:47 +05:30
team_store.go
[MM-36472] Fix inconsistencies in Roles columns (#18390)
2021-10-01 15:31:29 +02:00
terms_of_service_store_test.go
MM-36271: Bump major version to 6.0 (#17973)
2021-07-22 12:21:47 +05:30
terms_of_service_store.go
MM-36271: Bump major version to 6.0 (#17973)
2021-07-22 12:21:47 +05:30
thread_store_test.go
MM-36271: Bump major version to 6.0 (#17973)
2021-07-22 12:21:47 +05:30
thread_store.go
Optimize TotalUnreadThreads (#18788)
2021-10-26 08:56:38 +05:30
tokens_store.go
MM-36271: Bump major version to 6.0 (#17973)
2021-07-22 12:21:47 +05:30
upgrade_test.go
[MM-28985] Remove pointers to slice (part 1) (#18034)
2021-08-17 11:18:33 +02:00
upgrade.go
Drop existing default on Users.Timezone prior to column type change (#18669)
2021-10-14 17:18:18 +02:00
upload_session_store_test.go
MM-36271: Bump major version to 6.0 (#17973)
2021-07-22 12:21:47 +05:30
upload_session_store.go
MM-36271: Bump major version to 6.0 (#17973)
2021-07-22 12:21:47 +05:30
user_access_token_store_test.go
MM-36271: Bump major version to 6.0 (#17973)
2021-07-22 12:21:47 +05:30
user_access_token_store.go
MM-36271: Bump major version to 6.0 (#17973)
2021-07-22 12:21:47 +05:30
user_store_test.go
MM-36271: Bump major version to 6.0 (#17973)
2021-07-22 12:21:47 +05:30
user_store.go
[MM-36472] Fix inconsistencies in Roles columns (#18390)
2021-10-01 15:31:29 +02:00
user_terms_of_service_store_test.go
MM-36271: Bump major version to 6.0 (#17973)
2021-07-22 12:21:47 +05:30
user_terms_of_service.go
MM-36271: Bump major version to 6.0 (#17973)
2021-07-22 12:21:47 +05:30
utils_test.go
Update channel member notify props to use native JSON (#18114)
2021-08-23 10:04:30 +05:30
utils.go
[MM-39232] Migrate from gorp to sqlx in store/sqlstore/reaction_store.go (#18657)
2021-10-21 09:55:49 +05:30
webhook_store_test.go
MM-36271: Bump major version to 6.0 (#17973)
2021-07-22 12:21:47 +05:30
webhook_store.go
MM-36271: Bump major version to 6.0 (#17973)
2021-07-22 12:21:47 +05:30