From 635cea19c77fba736f7368de5e22d49660405c49 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Alejandro=20Garc=C3=ADa=20Montoro?= Date: Fri, 30 Sep 2022 18:18:26 +0200 Subject: [PATCH] MM-46105: Simplify posts batch retrieval query (#21149) In MySQL, tested in Community, times for the old and new queries are mostly the same: - Original : 6 s 250 ms (execution: 258 ms, fetching: 5 s 992 ms) - New : 6 s 148 ms (execution: 148 ms, fetching: 6 s) The output from EXPLAIN in Community is also similar for both queries: - Original: | id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra | |----|-------------|------------|------------|--------|------------------------|------------------------|---------|----------------------|---------|----------|-----------------------| | 1 | PRIMARY | | | ALL | | | | | 10000 | 100.00 | Using filesort | | 1 | PRIMARY | Channels | | eq_ref | PRIMARY | PRIMARY | 106 | PostsQuery.ChannelId | 1 | 100.00 | | | 2 | DERIVED | Posts | | range | idx_posts_create_at_id | idx_posts_create_at_id | 115 | | 3108048 | 100.00 | Using index condition | - New: | id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra | |----|-------------|----------|------------|--------|------------------------|------------------------|---------|----------------------------|---------|----------|-----------------------| | 1 | SIMPLE | Posts | | range | idx_posts_create_at_id | idx_posts_create_at_id | 115 | | 3108050 | 100.00 | Using index condition | | 1 | SIMPLE | Channels | | eq_ref | PRIMARY | PRIMARY | 106 | mattermost.Posts.ChannelId | 1 | 100.00 | | In Postgres 12, this was tested locally with a 12M posts database, in a machine with the following specs: - CPU: i7-11800H (8C / 16T, 2.3 / 4.6GHz, 24MB) - Memory: 32GB - Storage: 1TB SSD M.2 2280 PCIe 4.0x4 Performance NVMe Opal2 The times of the queries in this setup are: - Original query : 118.080 ms - New query : 94.454 ms The complete output from EXPLAIN ANALYZE in Postgres 12: - Original query: mattermost=> EXPLAIN ANALYZE SELECT PostsQuery.*, Channels.TeamId FROM ( SELECT * FROM Posts WHERE Posts.CreateAt > 1629244800000 OR (Posts.CreateAt = 1629244800000 AND Posts.Id > '') ORDER BY CreateAt ASC, Id ASC LIMIT 10000 ) AS PostsQuery LEFT JOIN Channels ON PostsQuery.ChannelId = Channels.Id ORDER BY CreateAt ASC, Id ASC; QUERY PLAN -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Sort (cost=156636.28..156661.28 rows=10000 width=375) (actual time=114.093..114.833 rows=10000 loops=1) Sort Key: posts.createat, posts.id Sort Method: external merge Disk: 3128kB -> Hash Right Join (cost=136335.48..155971.89 rows=10000 width=375) (actual time=67.919..107.330 rows=10000 loops=1) Hash Cond: ((channels.id)::text = (posts.channelid)::text) -> Seq Scan on channels (cost=0.00..10902.68 rows=272168 width=28) (actual time=0.018..28.826 rows=272168 loops=1) -> Hash (cost=135721.48..135721.48 rows=10000 width=374) (actual time=40.141..40.144 rows=10000 loops=1) Buckets: 16384 Batches: 2 Memory Usage: 2244kB -> Limit (cost=135596.48..135621.48 rows=10000 width=374) (actual time=36.799..38.169 rows=10000 loops=1) -> Sort (cost=135596.48..135704.93 rows=43380 width=374) (actual time=30.641..31.699 rows=10000 loops=1) Sort Key: posts.createat, posts.id Sort Method: external merge Disk: 18840kB -> Bitmap Heap Scan on posts (cost=827.91..132497.48 rows=43380 width=374) (actual time=1.856..8.630 rows=54398 loops=1) Recheck Cond: ((createat > '1629244800000'::bigint) OR (createat = '1629244800000'::bigint)) Filter: ((createat > '1629244800000'::bigint) OR ((createat = '1629244800000'::bigint) AND ((id)::text > ''::text))) Heap Blocks: exact=2674 -> BitmapOr (cost=827.91..827.91 rows=43380 width=0) (actual time=1.660..1.661 rows=0 loops=1) -> Bitmap Index Scan on idx_posts_create_at (cost=0.00..801.78 rows=43379 width=0) (actual time=1.657..1.657 rows=54398 loops=1) Index Cond: (createat > '1629244800000'::bigint) -> Bitmap Index Scan on idx_posts_create_at (cost=0.00..4.44 rows=1 width=0) (actual time=0.002..0.002 rows=0 loops=1) Index Cond: (createat = '1629244800000'::bigint) Planning Time: 0.230 ms JIT: Functions: 14 Options: Inlining false, Optimization false, Expressions true, Deforming true Timing: Generation 0.562 ms, Inlining 0.000 ms, Optimization 0.320 ms, Emission 5.863 ms, Total 6.745 ms Execution Time: 118.080 ms (27 rows) - New query: mattermost=> EXPLAIN ANALYZE SELECT Posts.*, Channels.TeamId FROM Posts LEFT JOIN Channels ON Posts.ChannelId = Channels.Id WHERE Posts.CreateAt > 1629244800000 OR (Posts.CreateAt = 1629244800000 AND Posts.Id > '') ORDER BY Posts.CreateAt ASC, Posts.Id ASC LIMIT 10000; QUERY PLAN -------------------------------------------------------------------------------------------------------------------------------------------------------------------- Limit (cost=148443.93..149610.68 rows=10000 width=375) (actual time=83.610..92.492 rows=10000 loops=1) -> Gather Merge (cost=148443.93..152661.73 rows=36150 width=375) (actual time=74.409..82.975 rows=10000 loops=1) Workers Planned: 2 Workers Launched: 2 -> Sort (cost=147443.91..147489.10 rows=18075 width=375) (actual time=64.325..64.629 rows=3436 loops=3) Sort Key: posts.createat, posts.id Sort Method: external merge Disk: 5992kB Worker 0: Sort Method: external merge Disk: 7144kB Worker 1: Sort Method: external merge Disk: 7152kB -> Parallel Hash Left Join (cost=12336.48..146152.66 rows=18075 width=375) (actual time=42.908..51.116 rows=18133 loops=3) Hash Cond: ((posts.channelid)::text = (channels.id)::text) -> Parallel Bitmap Heap Scan on posts (cost=827.91..132054.64 rows=18075 width=374) (actual time=2.448..5.417 rows=18133 loops=3) Recheck Cond: ((createat > '1629244800000'::bigint) OR (createat = '1629244800000'::bigint)) Filter: ((createat > '1629244800000'::bigint) OR ((createat = '1629244800000'::bigint) AND ((id)::text > ''::text))) Heap Blocks: exact=902 -> BitmapOr (cost=827.91..827.91 rows=43380 width=0) (actual time=2.111..2.112 rows=0 loops=1) -> Bitmap Index Scan on idx_posts_create_at (cost=0.00..801.78 rows=43379 width=0) (actual time=2.105..2.105 rows=54398 loops=1) Index Cond: (createat > '1629244800000'::bigint) -> Bitmap Index Scan on idx_posts_create_at (cost=0.00..4.44 rows=1 width=0) (actual time=0.004..0.005 rows=0 loops=1) Index Cond: (createat = '1629244800000'::bigint) -> Parallel Hash (cost=9315.03..9315.03 rows=113403 width=28) (actual time=29.928..29.929 rows=90723 loops=3) Buckets: 65536 Batches: 8 Memory Usage: 2688kB -> Parallel Seq Scan on channels (cost=0.00..9315.03 rows=113403 width=28) (actual time=5.378..16.495 rows=90723 loops=3) Planning Time: 0.460 ms JIT: Functions: 46 Options: Inlining false, Optimization false, Expressions true, Deforming true Timing: Generation 2.291 ms, Inlining 0.000 ms, Optimization 1.518 ms, Emission 23.827 ms, Total 27.636 ms Execution Time: 94.454 ms (29 rows) --- store/sqlstore/post_store.go | 30 +++++++++++------------------- 1 file changed, 11 insertions(+), 19 deletions(-) diff --git a/store/sqlstore/post_store.go b/store/sqlstore/post_store.go index 747b9d0f81..33dee42c36 100644 --- a/store/sqlstore/post_store.go +++ b/store/sqlstore/post_store.go @@ -2331,28 +2331,20 @@ func (s *SqlPostStore) GetPostsBatchForIndexing(startTime int64, startPostID str table += " USE INDEX(idx_posts_create_at_id)" } query := `SELECT - PostsQuery.*, Channels.TeamId - FROM ( - SELECT - * - FROM - ` + table + ` - WHERE - Posts.CreateAt > ? - OR - (Posts.CreateAt = ? AND Posts.Id > ?) - ORDER BY - CreateAt ASC, Id ASC - LIMIT - ? - ) - AS - PostsQuery + Posts.*, Channels.TeamId + FROM ` + table + ` LEFT JOIN Channels ON - PostsQuery.ChannelId = Channels.Id - ORDER BY CreateAt ASC, Id ASC` + Posts.ChannelId = Channels.Id + WHERE + Posts.CreateAt > ? + OR + (Posts.CreateAt = ? AND Posts.Id > ?) + ORDER BY + Posts.CreateAt ASC, Posts.Id ASC + LIMIT + ?` err := s.GetSearchReplicaX().Select(&posts, query, startTime, startTime, startPostID, limit) if err != nil {