mirrors/mostlymatter
1
0
Форкнуть 0
Код Задачи Запросы на слияние Действия Пакеты Проекты Релизы Вики Активность
Files
635cea19c77fba736f7368de5e22d49660405c49
mostlymatter/store/sqlstore
История
Alejandro García Montoro 635cea19c7 MM-46105: Simplify posts batch retrieval query (#21149) 
In MySQL, tested in Community, times for the old and new queries are
mostly the same:
- Original : 6 s 250 ms (execution: 258 ms, fetching: 5 s 992 ms)
- New      : 6 s 148 ms (execution: 148 ms, fetching: 6 s)

The output from EXPLAIN in Community is also similar for both queries:

- Original:

| id | select_type | table      | partitions | type   | possible_keys          | key                    | key_len | ref                  | rows    | filtered | Extra                 |
|----|-------------|------------|------------|--------|------------------------|------------------------|---------|----------------------|---------|----------|-----------------------|
|  1 | PRIMARY     | <derived2> |            | ALL    |                        |                        |         |                      |   10000 |   100.00 | Using filesort        |
|  1 | PRIMARY     | Channels   |            | eq_ref | PRIMARY                | PRIMARY                | 106     | PostsQuery.ChannelId |       1 |   100.00 |                       |
|  2 | DERIVED     | Posts      |            | range  | idx_posts_create_at_id | idx_posts_create_at_id | 115     |                      | 3108048 |   100.00 | Using index condition |

- New:

| id | select_type | table    | partitions | type   | possible_keys          | key                    | key_len | ref                        | rows    | filtered | Extra                 |
|----|-------------|----------|------------|--------|------------------------|------------------------|---------|----------------------------|---------|----------|-----------------------|
|  1 | SIMPLE      | Posts    |            | range  | idx_posts_create_at_id | idx_posts_create_at_id | 115     |                            | 3108050 |   100.00 | Using index condition |
|  1 | SIMPLE      | Channels |            | eq_ref | PRIMARY                | PRIMARY                | 106     | mattermost.Posts.ChannelId |       1 |   100.00 |                       |

In Postgres 12, this was tested locally with a 12M posts database, in a
machine with the following specs:
- CPU: i7-11800H (8C / 16T, 2.3 / 4.6GHz, 24MB)
- Memory: 32GB
- Storage: 1TB SSD M.2 2280 PCIe 4.0x4 Performance NVMe Opal2

The times of the queries in this setup are:
- Original query : 118.080 ms
- New query      : 94.454 ms

The complete output from EXPLAIN ANALYZE in Postgres 12:

- Original query:

mattermost=> EXPLAIN ANALYZE
SELECT PostsQuery.*, Channels.TeamId
FROM (
    SELECT *
    FROM
    Posts
    WHERE Posts.CreateAt > 1629244800000
        OR (Posts.CreateAt = 1629244800000 AND Posts.Id > '')
    ORDER BY CreateAt ASC, Id ASC
    LIMIT 10000
) AS PostsQuery
LEFT JOIN Channels ON PostsQuery.ChannelId = Channels.Id
ORDER BY CreateAt ASC, Id ASC;
                                                                                QUERY PLAN
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 Sort  (cost=156636.28..156661.28 rows=10000 width=375) (actual time=114.093..114.833 rows=10000 loops=1)
   Sort Key: posts.createat, posts.id
   Sort Method: external merge  Disk: 3128kB
   ->  Hash Right Join  (cost=136335.48..155971.89 rows=10000 width=375) (actual time=67.919..107.330 rows=10000 loops=1)
         Hash Cond: ((channels.id)::text = (posts.channelid)::text)
         ->  Seq Scan on channels  (cost=0.00..10902.68 rows=272168 width=28) (actual time=0.018..28.826 rows=272168 loops=1)
         ->  Hash  (cost=135721.48..135721.48 rows=10000 width=374) (actual time=40.141..40.144 rows=10000 loops=1)
               Buckets: 16384  Batches: 2  Memory Usage: 2244kB
               ->  Limit  (cost=135596.48..135621.48 rows=10000 width=374) (actual time=36.799..38.169 rows=10000 loops=1)
                     ->  Sort  (cost=135596.48..135704.93 rows=43380 width=374) (actual time=30.641..31.699 rows=10000 loops=1)
                           Sort Key: posts.createat, posts.id
                           Sort Method: external merge  Disk: 18840kB
                           ->  Bitmap Heap Scan on posts  (cost=827.91..132497.48 rows=43380 width=374) (actual time=1.856..8.630 rows=54398 loops=1)
                                 Recheck Cond: ((createat > '1629244800000'::bigint) OR (createat = '1629244800000'::bigint))
                                 Filter: ((createat > '1629244800000'::bigint) OR ((createat = '1629244800000'::bigint) AND ((id)::text > ''::text)))
                                 Heap Blocks: exact=2674
                                 ->  BitmapOr  (cost=827.91..827.91 rows=43380 width=0) (actual time=1.660..1.661 rows=0 loops=1)
                                       ->  Bitmap Index Scan on idx_posts_create_at  (cost=0.00..801.78 rows=43379 width=0) (actual time=1.657..1.657 rows=54398 loops=1)
                                             Index Cond: (createat > '1629244800000'::bigint)
                                       ->  Bitmap Index Scan on idx_posts_create_at  (cost=0.00..4.44 rows=1 width=0) (actual time=0.002..0.002 rows=0 loops=1)
                                             Index Cond: (createat = '1629244800000'::bigint)
 Planning Time: 0.230 ms
 JIT:
   Functions: 14
   Options: Inlining false, Optimization false, Expressions true, Deforming true
   Timing: Generation 0.562 ms, Inlining 0.000 ms, Optimization 0.320 ms, Emission 5.863 ms, Total 6.745 ms
 Execution Time: 118.080 ms
(27 rows)

- New query:

mattermost=> EXPLAIN ANALYZE
SELECT Posts.*, Channels.TeamId
FROM
Posts
LEFT JOIN Channels ON Posts.ChannelId = Channels.Id
WHERE Posts.CreateAt > 1629244800000
    OR (Posts.CreateAt = 1629244800000 AND Posts.Id > '')
ORDER BY Posts.CreateAt ASC, Posts.Id ASC
LIMIT 10000;
                                                                             QUERY PLAN
--------------------------------------------------------------------------------------------------------------------------------------------------------------------
 Limit  (cost=148443.93..149610.68 rows=10000 width=375) (actual time=83.610..92.492 rows=10000 loops=1)
   ->  Gather Merge  (cost=148443.93..152661.73 rows=36150 width=375) (actual time=74.409..82.975 rows=10000 loops=1)
         Workers Planned: 2
         Workers Launched: 2
         ->  Sort  (cost=147443.91..147489.10 rows=18075 width=375) (actual time=64.325..64.629 rows=3436 loops=3)
               Sort Key: posts.createat, posts.id
               Sort Method: external merge  Disk: 5992kB
               Worker 0:  Sort Method: external merge  Disk: 7144kB
               Worker 1:  Sort Method: external merge  Disk: 7152kB
               ->  Parallel Hash Left Join  (cost=12336.48..146152.66 rows=18075 width=375) (actual time=42.908..51.116 rows=18133 loops=3)
                     Hash Cond: ((posts.channelid)::text = (channels.id)::text)
                     ->  Parallel Bitmap Heap Scan on posts  (cost=827.91..132054.64 rows=18075 width=374) (actual time=2.448..5.417 rows=18133 loops=3)
                           Recheck Cond: ((createat > '1629244800000'::bigint) OR (createat = '1629244800000'::bigint))
                           Filter: ((createat > '1629244800000'::bigint) OR ((createat = '1629244800000'::bigint) AND ((id)::text > ''::text)))
                           Heap Blocks: exact=902
                           ->  BitmapOr  (cost=827.91..827.91 rows=43380 width=0) (actual time=2.111..2.112 rows=0 loops=1)
                                 ->  Bitmap Index Scan on idx_posts_create_at  (cost=0.00..801.78 rows=43379 width=0) (actual time=2.105..2.105 rows=54398 loops=1)
                                       Index Cond: (createat > '1629244800000'::bigint)
                                 ->  Bitmap Index Scan on idx_posts_create_at  (cost=0.00..4.44 rows=1 width=0) (actual time=0.004..0.005 rows=0 loops=1)
                                       Index Cond: (createat = '1629244800000'::bigint)
                     ->  Parallel Hash  (cost=9315.03..9315.03 rows=113403 width=28) (actual time=29.928..29.929 rows=90723 loops=3)
                           Buckets: 65536  Batches: 8  Memory Usage: 2688kB
                           ->  Parallel Seq Scan on channels  (cost=0.00..9315.03 rows=113403 width=28) (actual time=5.378..16.495 rows=90723 loops=3)
 Planning Time: 0.460 ms
 JIT:
   Functions: 46
   Options: Inlining false, Optimization false, Expressions true, Deforming true
   Timing: Generation 2.291 ms, Inlining 0.000 ms, Optimization 1.518 ms, Emission 23.827 ms, Total 27.636 ms
 Execution Time: 94.454 ms
(29 rows)
2022-09-30 18:18:26 +02:00
..
adapters_test.go
Using native JSON operations to add thread participants (#18093)
2021-08-12 16:45:03 +02:00
adapters.go
MM-45994 ensure database operations return their errors (#20857)
2022-08-26 11:12:59 +02:00
audit_store_test.go
MM-36271: Bump major version to 6.0 (#17973)
2021-07-22 12:21:47 +05:30
audit_store.go
s/masterminds/mattermost for squirrel (#20162)
2022-05-10 11:48:59 -03:00
bot_store_test.go
MM-36271: Bump major version to 6.0 (#17973)
2021-07-22 12:21:47 +05:30
bot_store.go
MM-45994 ensure database operations return their errors (#20857)
2022-08-26 11:12:59 +02:00
channel_member_history_store_test.go
MM-36271: Bump major version to 6.0 (#17973)
2021-07-22 12:21:47 +05:30
channel_member_history_store.go
MM-45994 ensure database operations return their errors (#20857)
2022-08-26 11:12:59 +02:00
channel_store_categories_test.go
MM-36271: Bump major version to 6.0 (#17973)
2021-07-22 12:21:47 +05:30
channel_store_categories.go
MM-45994 ensure database operations return their errors (#20857)
2022-08-26 11:12:59 +02:00
channel_store_test.go
MM-19336 Migrated GetMembersByIds and GetMembersByChannelIds to Squirrel (#19728)
2022-03-11 01:27:09 +05:30
channel_store.go
Avoid counting top channel posts for posts made by plugins and OAuth apps (#20943)
2022-09-30 04:12:15 -04:00
cluster_discovery_store_test.go
MM-36271: Bump major version to 6.0 (#17973)
2021-07-22 12:21:47 +05:30
cluster_discovery_store.go
MM-45994 ensure database operations return their errors (#20857)
2022-08-26 11:12:59 +02:00
command_store_test.go
MM-36271: Bump major version to 6.0 (#17973)
2021-07-22 12:21:47 +05:30
command_store.go
s/masterminds/mattermost for squirrel (#20162)
2022-05-10 11:48:59 -03:00
command_webhook_store_test.go
MM-36271: Bump major version to 6.0 (#17973)
2021-07-22 12:21:47 +05:30
command_webhook_store.go
MM-45994 ensure database operations return their errors (#20857)
2022-08-26 11:12:59 +02:00
compliance_store_test.go
MM-36271: Bump major version to 6.0 (#17973)
2021-07-22 12:21:47 +05:30
compliance_store.go
Use any instead of interface{} (#20577)
2022-07-05 09:46:50 +03:00
context_test.go
MM-30882: Fix read-after-write issue for demoting user (#16911)
2021-02-12 19:04:05 +05:30
context.go
Kill gorp (#19786)
2022-03-15 19:49:19 +05:30
emoji_store_test.go
MM-36271: Bump major version to 6.0 (#17973)
2021-07-22 12:21:47 +05:30
emoji_store.go
MM-45994 ensure database operations return their errors (#20857)
2022-08-26 11:12:59 +02:00
file_info_store_test.go
MM-36271: Bump major version to 6.0 (#17973)
2021-07-22 12:21:47 +05:30
file_info_store.go
[MM-44489] Cloud limits: enforcing files (#20703)
2022-09-28 12:52:53 -04:00
group_store_test.go
MM-36271: Bump major version to 6.0 (#17973)
2021-07-22 12:21:47 +05:30
group_store.go
MM-45994 ensure database operations return their errors (#20857)
2022-08-26 11:12:59 +02:00
init_test.go
Consistent license message for all the go files (#13235)
2019-11-29 12:59:40 +01:00
integrity_test.go
[MM-45444] Denormalize Reactions to add ChannelId for top reactions insights query (#20572)
2022-07-11 09:25:03 -04:00
integrity.go
MM-45994 ensure database operations return their errors (#20857)
2022-08-26 11:12:59 +02:00
job_store_test.go
MM-36271: Bump major version to 6.0 (#17973)
2021-07-22 12:21:47 +05:30
job_store.go
s/masterminds/mattermost for squirrel (#20162)
2022-05-10 11:48:59 -03:00
license_store_test.go
MM-36271: Bump major version to 6.0 (#17973)
2021-07-22 12:21:47 +05:30
license_store.go
s/masterminds/mattermost for squirrel (#20162)
2022-05-10 11:48:59 -03:00
link_metadata_store_test.go
MM-36271: Bump major version to 6.0 (#17973)
2021-07-22 12:21:47 +05:30
link_metadata_store.go
s/masterminds/mattermost for squirrel (#20162)
2022-05-10 11:48:59 -03:00
main_test.go
MM-36764 mlog refactor (#18118)
2021-08-17 16:08:04 -04:00
notify_admin_store_test.go
[MM-45564] - Notify Admin v2 (#20777)
2022-09-02 13:52:48 +03:00
notify_admin_store.go
[MM-45564] - Notify Admin v2 (#20777)
2022-09-02 13:52:48 +03:00
oauth_store_test.go
MM-36271: Bump major version to 6.0 (#17973)
2021-07-22 12:21:47 +05:30
oauth_store.go
MM-45994 ensure database operations return their errors (#20857)
2022-08-26 11:12:59 +02:00
plugin_store_test.go
MM-36271: Bump major version to 6.0 (#17973)
2021-07-22 12:21:47 +05:30
plugin_store.go
s/masterminds/mattermost for squirrel (#20162)
2022-05-10 11:48:59 -03:00
post_store_test.go
MM-36271: Bump major version to 6.0 (#17973)
2021-07-22 12:21:47 +05:30
post_store.go
MM-46105: Simplify posts batch retrieval query (#21149)
2022-09-30 18:18:26 +02:00
preference_store_test.go
Kill gorp (#19786)
2022-03-15 19:49:19 +05:30
preference_store.go
MM-45994 ensure database operations return their errors (#20857)
2022-08-26 11:12:59 +02:00
product_notices_store_test.go
MM-36271: Bump major version to 6.0 (#17973)
2021-07-22 12:21:47 +05:30
product_notices_store.go
MM-45994 ensure database operations return their errors (#20857)
2022-08-26 11:12:59 +02:00
reaction_store_test.go
MM-36271: Bump major version to 6.0 (#17973)
2021-07-22 12:21:47 +05:30
reaction_store.go
MM-45994 ensure database operations return their errors (#20857)
2022-08-26 11:12:59 +02:00
remote_cluster_store_test.go
MM-36271: Bump major version to 6.0 (#17973)
2021-07-22 12:21:47 +05:30
remote_cluster_store.go
s/masterminds/mattermost for squirrel (#20162)
2022-05-10 11:48:59 -03:00
retention_policy_store_test.go
MM-36271: Bump major version to 6.0 (#17973)
2021-07-22 12:21:47 +05:30
retention_policy_store.go
MM-45994 ensure database operations return their errors (#20857)
2022-08-26 11:12:59 +02:00
role_store_test.go
MM-36271: Bump major version to 6.0 (#17973)
2021-07-22 12:21:47 +05:30
role_store.go
MM-45994 ensure database operations return their errors (#20857)
2022-08-26 11:12:59 +02:00
scheme_store_test.go
MM-36271: Bump major version to 6.0 (#17973)
2021-07-22 12:21:47 +05:30
scheme_store.go
MM-45994 ensure database operations return their errors (#20857)
2022-08-26 11:12:59 +02:00
session_store_test.go
MM-36271: Bump major version to 6.0 (#17973)
2021-07-22 12:21:47 +05:30
session_store.go
MM-44101: GraphQL: Add sidebar categories at the top level (#20178)
2022-05-12 20:03:47 +05:30
shared_channel_store_test.go
MM-36271: Bump major version to 6.0 (#17973)
2021-07-22 12:21:47 +05:30
shared_channel_store.go
MM-45994 ensure database operations return their errors (#20857)
2022-08-26 11:12:59 +02:00
sqlx_wrapper_test.go
Added a sqlXPassThrough layer (#18238)
2021-08-30 10:20:28 +05:30
sqlx_wrapper.go
Use any instead of interface{} (#20577)
2022-07-05 09:46:50 +03:00
status_store_test.go
MM-36271: Bump major version to 6.0 (#17973)
2021-07-22 12:21:47 +05:30
status_store.go
MM-45994 ensure database operations return their errors (#20857)
2022-08-26 11:12:59 +02:00
store_test.go
Remove code duplication and move utility methods to the store utils (#20091)
2022-04-29 12:15:38 +02:00
store.go
[MM-45564] - Notify Admin v2 (#20777)
2022-09-02 13:52:48 +03:00
system_store_test.go
MM-36271: Bump major version to 6.0 (#17973)
2021-07-22 12:21:47 +05:30
system_store.go
MM-45994 ensure database operations return their errors (#20857)
2022-08-26 11:12:59 +02:00
team_store_test.go
MM-36271: Bump major version to 6.0 (#17973)
2021-07-22 12:21:47 +05:30
team_store.go
MM-45994 ensure database operations return their errors (#20857)
2022-08-26 11:12:59 +02:00
terms_of_service_store_test.go
MM-36271: Bump major version to 6.0 (#17973)
2021-07-22 12:21:47 +05:30
terms_of_service_store.go
Remove ColMap (#19544)
2022-02-15 11:44:04 +05:30
thread_store_test.go
MM-36271: Bump major version to 6.0 (#17973)
2021-07-22 12:21:47 +05:30
thread_store.go
MM-45994 ensure database operations return their errors (#20857)
2022-08-26 11:12:59 +02:00
tokens_store_test.go
[GH-18965] - Pass expire time from outside Cleanup method (#19008)
2021-12-22 14:57:30 +05:30
tokens_store.go
s/masterminds/mattermost for squirrel (#20162)
2022-05-10 11:48:59 -03:00
upload_session_store_test.go
MM-36271: Bump major version to 6.0 (#17973)
2021-07-22 12:21:47 +05:30
upload_session_store.go
s/masterminds/mattermost for squirrel (#20162)
2022-05-10 11:48:59 -03:00
user_access_token_store_test.go
MM-36271: Bump major version to 6.0 (#17973)
2021-07-22 12:21:47 +05:30
user_access_token_store.go
MM-45994 ensure database operations return their errors (#20857)
2022-08-26 11:12:59 +02:00
user_store_test.go
MM-36271: Bump major version to 6.0 (#17973)
2021-07-22 12:21:47 +05:30
user_store.go
Badge count fix for push notifications when CRT is enabled (#20898)
2022-09-12 14:51:33 +05:30
user_terms_of_service_store_test.go
MM-36271: Bump major version to 6.0 (#17973)
2021-07-22 12:21:47 +05:30
user_terms_of_service.go
MM-45994 ensure database operations return their errors (#20857)
2022-08-26 11:12:59 +02:00
utils_test.go
Use any instead of interface{} (#20577)
2022-07-05 09:46:50 +03:00
utils.go
MM-45994 ensure database operations return their errors (#20857)
2022-08-26 11:12:59 +02:00
webhook_store_test.go
MM-36271: Bump major version to 6.0 (#17973)
2021-07-22 12:21:47 +05:30
webhook_store.go
s/masterminds/mattermost for squirrel (#20162)
2022-05-10 11:48:59 -03:00