We use * instead of a column name to use
index-only scan always even when other column filters
are applied.
Right now, index-only scan will only get applied
in the basic query of "select count(p.id) as value
from posts p". But it won't get applied if the query
is "select count(p.id) as value from posts p where
p.deleteat=0".
So this is a minor optimization which improves
some corner cases.
This was found from the slow query monitoring.
```release-note
NONE
```
* MM-43939: fixes lastreplyat when deleting the last reply
Currently we are not updating the Threads.LastReplyAt when the last
reply gets deleted. This can lead to threads appearing unread when
actually there is no unread thread.
This commit updates the value of Threads.LastReplyAt when a reply gets
deleted, to the most recent post's timestamp in the thread.
* Updates ReplyCount to current value on post delete
* Addresses review comments
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
This PR adds the post reminder backend work.
We add a new API endpoint via which a user can set a reminder for a post. An ephemeral message will be sent down the line to let the user know about the action. And then after the time is over, the system admin bot will send a DM message to the user about the reminder post.
* Add new Job to keep updating the last_accessible_post time
* Filter out posts for funcs returning PostList model
* Separate methods to get and compute cache
* filter pinned posts
* For posts with sorted CreateAt order, support a faster form of filtering.
* Add inaccessible header for getPost and getPostsByIDs APIs
* replace manual binary search with the std. library
* in-place filter posts
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
Co-authored-by: Nathaniel Allred <neallred@protonmail.com>
We use the default_text_search_config param from the DB
and pass that as the search config instead of hardcoding
it to english.
The docs do say that if no param is passed, by default
it will automatically fall back to default_text_search_config.
But I have seen different query plans being created
when the parameter is not passed versus when it is passed.
I am not sure why this is happening, but to be safe,
I have taken the search config during startup and stored
it in the SqlStore struct.
For reference, this is what happens without a search config
passed:
```
[bigdb] # explain analyze SELECT *, (SELECT COUNT(*) FROM Posts WHERE Posts.RootId = (CASE WHEN q2.RootId = '' THEN q2.Id ELSE q2.RootId END) AND Posts.DeleteAt = 0) as ReplyCount FROM Posts q2 WHERE q2.DeleteAt = 0 AND q2.Type NOT LIKE 'system_%' AND to_tsvector(Message) @@ to_tsquery('sapiente') AND ChannelId IN (SELECT Id FROM Channels, ChannelMembers WHERE Id = ChannelId AND Channels.DeleteAt = 0 AND ChannelMembe
rs.UserId = 'tc3p1yqw67d8idcp3g98awexqe' AND (TeamId = '8ywxyw9ocp8smxrmjzrkqhrdwe' OR TeamId = '') AND Id IN ('h1x9asxr7idjpqfmg8q67us49h')) ORDER BY q2.CreateAt DESC LIMIT 100;
QUERY PLAN
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Limit (cost=8200.93..8720.23 rows=9 width=382) (actual time=177.277..177.948 rows=100 loops=1)
-> Result (cost=8200.93..8720.23 rows=9 width=382) (actual time=177.274..177.939 rows=100 loops=1)
-> Sort (cost=8200.93..8200.95 rows=9 width=374) (actual time=177.245..177.271 rows=100 loops=1)
Sort Key: q2.createat DESC
Sort Method: top-N heapsort Memory: 161kB
-> Nested Loop Semi Join (cost=71.53..8200.79 rows=9 width=374) (actual time=5.910..176.815 rows=532 loops=1)
-> Bitmap Heap Scan on posts q2 (cost=70.56..8183.64 rows=9 width=374) (actual time=5.815..176.457 rows=532 loops=1)
Recheck Cond: ((channelid)::text = 'h1x9asxr7idjpqfmg8q67us49h'::text)
Filter: (((type)::text !~~ 'system_%'::text) AND (deleteat = 0) AND (to_tsvector((message)::text) @@ to_tsquery('sapiente'::text)))
Rows Removed by Filter: 5244
Heap Blocks: exact=944
-> Bitmap Index Scan on idx_posts_channel_id_update_at (cost=0.00..70.56 rows=1866 width=0) (actual time=1.883..1.883 rows=5776 loops=1)
Index Cond: ((channelid)::text = 'h1x9asxr7idjpqfmg8q67us49h'::text)
-> Materialize (cost=0.98..17.04 rows=1 width=27) (actual time=0.000..0.000 rows=1 loops=532)
-> Nested Loop (cost=0.98..17.03 rows=1 width=27) (actual time=0.080..0.080 rows=1 loops=1)
-> Index Scan using channels_pkey on channels (cost=0.42..8.45 rows=1 width=27) (actual time=0.041..0.042 rows=1 loops=1)
Index Cond: ((id)::text = 'h1x9asxr7idjpqfmg8q67us49h'::text)
Filter: ((deleteat = 0) AND (((teamid)::text = '8ywxyw9ocp8smxrmjzrkqhrdwe'::text) OR ((teamid)::text = ''::text)))
-> Index Only Scan using idx_channelmembers_user_id_channel_id_last_viewed_at on channelmembers (cost=0.56..8.58 rows=1 width=27) (actual time=0.034..0.034 rows=1 loops=1)
Index Cond: ((userid = 'tc3p1yqw67d8idcp3g98awexqe'::text) AND (channelid = 'h1x9asxr7idjpqfmg8q67us49h'::text))
Heap Fetches: 1
SubPlan 1
-> Aggregate (cost=57.68..57.69 rows=1 width=8) (actual time=0.006..0.006 rows=1 loops=100)
-> Index Only Scan using idx_posts_root_id_delete_at on posts (cost=0.56..54.44 rows=1294 width=0) (actual time=0.005..0.005 rows=3 loops=100)
Index Cond: ((rootid = (CASE WHEN ((q2.rootid)::text = ''::text) THEN q2.id ELSE q2.rootid END)::text) AND (deleteat = 0))
Heap Fetches: 0
Planning Time: 2.178 ms
Execution Time: 178.155 ms
```
We can see it using a top-N heapsort.
And this is what happens when an explicit search config
is passed:
```
[bigdb] # explain analyze SELECT *, (SELECT COUNT(*) FROM Posts WHERE Posts.RootId = (CASE WHEN q2.RootId = '' THEN q2.Id ELSE q2.RootId END) AND Posts.DeleteAt = 0) as ReplyCount FROM Posts q2 WHERE q2.DeleteAt = 0 AND q2.Type NOT LIKE 'system_%' AND to_tsvector('pg_catalog.english', Message) @@ to_tsquery('pg_catalog.english', 'sapiente') AND ChannelId IN (SELECT Id FROM Channels, ChannelMembers WHERE Id = ChannelId AND Channels.DeleteAt = 0 AND ChannelMembers.UserId = 'tc3p1yqw67d8idcp3g98awexqe' AND (TeamId = '8ywxyw9ocp8smxrmjzrkqhrdwe' OR TeamId = '') AND Id IN ('h1x9asxr7idjpqfmg8q67us49h')) ORDER BY q2.CreateAt DESC LIMIT 100;
QUERY PLAN
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Limit (cost=1.54..10679.04 rows=100 width=382) (actual time=0.237..31.405 rows=100 loops=1)
-> Nested Loop Semi Join (cost=1.54..17512.64 rows=164 width=382) (actual time=0.236..31.387 rows=100 loops=1)
-> Index Scan Backward using idx_posts_channel_id_delete_at_create_at on posts q2 (cost=0.56..8032.81 rows=164 width=374) (actual time=0.121..30.124 rows=100 loops=1)
Index Cond: (((channelid)::text = 'h1x9asxr7idjpqfmg8q67us49h'::text) AND (deleteat = 0))
Filter: (((type)::text !~~ 'system_%'::text) AND (to_tsvector('english'::regconfig, (message)::text) @@ '''sapient'''::tsquery))
Rows Removed by Filter: 1088
-> Materialize (cost=0.98..17.04 rows=1 width=27) (actual time=0.001..0.001 rows=1 loops=100)
-> Nested Loop (cost=0.98..17.03 rows=1 width=27) (actual time=0.074..0.075 rows=1 loops=1)
-> Index Scan using channels_pkey on channels (cost=0.42..8.45 rows=1 width=27) (actual time=0.032..0.032 rows=1 loops=1)
Index Cond: ((id)::text = 'h1x9asxr7idjpqfmg8q67us49h'::text)
Filter: ((deleteat = 0) AND (((teamid)::text = '8ywxyw9ocp8smxrmjzrkqhrdwe'::text) OR ((teamid)::text = ''::text)))
-> Index Only Scan using idx_channelmembers_user_id_channel_id_last_viewed_at on channelmembers (cost=0.56..8.58 rows=1 width=27) (actual time=0.041..0.041 rows=1 loops=1)
Index Cond: ((userid = 'tc3p1yqw67d8idcp3g98awexqe'::text) AND (channelid = 'h1x9asxr7idjpqfmg8q67us49h'::text))
Heap Fetches: 1
SubPlan 1
-> Aggregate (cost=57.68..57.69 rows=1 width=8) (actual time=0.010..0.010 rows=1 loops=100)
-> Index Only Scan using idx_posts_root_id_delete_at on posts (cost=0.56..54.44 rows=1294 width=0) (actual time=0.009..0.009 rows=3 loops=100)
Index Cond: ((rootid = (CASE WHEN ((q2.rootid)::text = ''::text) THEN q2.id ELSE q2.rootid END)::text) AND (deleteat = 0))
Heap Fetches: 0
Planning Time: 0.401 ms
Execution Time: 31.466 ms
```
https://mattermost.atlassian.net/browse/MM-25554
```release-note
NONE
```
Just a quick POC to move fast :P
We use a search pointer to keep track of
the next row to inesrt to. For every new search
we increment the pointer and do modulo 5.
This means that the value will always remain
between 0-4. And that way, we will always overwrite
the oldest entry on every search.
And while getting the results, we get
all results for that user.
The search parameters are json marshalled
and stored as a JSON blob. This is because
there is no need to search/filter them
in the DB.
Pending items:
Tests obviously.
To improve:
The client needs to send the channel ids
instead of channel names.
Old versions of the Mattermost server did not qualify queries scanning both `Posts` and `Threads`, and choke on the ambiguity in deciding between the new `DeleteAt` on `Threads` and the `DeleteAt` on `Posts` in existing queries.
While this problem is transient only while running multiple server versions, it effectively makes our backwards compatibility guarantee void, not to mention complicating cloud deployments.
Work around this by renaming `Threads.DeleteAt` to `Threads.ThreadDeleteAt`. The old migration is nulled out, but remains, since some test servers have already upgraded and manually fixing each affected instance would be problematic. Thew new migration takes care of removing the old column -- if it ever existed.
Fixes: https://mattermost.atlassian.net/browse/MM-43770
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
The older method used to reply completely on timestamps
to take batches of items in a timestamp range and then
just incrementing the timestamp. This led to handling
edge-cases such as more items than the batch count, all
having the same timestamp.
Additionally, relying on timestamp as the page cursor
meant that indexing was not very efficient if you had
several items spread out across large spans of time.
To get away from all of that we use a proper cursor-based
approach consisting of createAt+Id. With this, we move
completely to a constant page size where we can fetch
a given number of objects irrespective of when they
were created. This makes indexing much more faster and
efficient.
https://mattermost.atlassian.net/browse/MM-41260
```release-note
Elasticsearch and Bleve indexing have been revamped to be much
more efficient and faster. The config parameter BulkIndexingTimeWindowSeconds
for both elasticsearch and bleve have been removed.
A new config parameter called BatchSize has been introduced instead.
This parameter controls the number of objects that
can be indexed in a single batch. This makes things
more efficient and maintains a constant workload.
```
We implement a cursor based pagination model
to page through the posts in a given thread.
The cursor is a combination of the post.CreateAt+
post.Id to differentiate multiple posts in a given
timestamp.
Some additional parameters like direction, fromPost,
fromCreateAt and perPage were introduced to implement
this.
```release-note
NONE
```
Before gorp, to use sqlx we had to resort to using
internal structs without the `db:"-"` tag so that
it doesn't interfere with the gorp table generation.
Now that it's gone, we can use the original model structs.
```release-note
NONE
```
* MM-40799: Improve ReplyCount query
This PR optimizes the reply count query to use only
the indexed column to count the rows rather than using
the ID column. This results in an index-only scan
rather than an index scan. As a result, the query
can be satisfied directly by scanning the index
and there is no need to touch the heap.
Count(*) also gives the same result, but is not recommended
as it also verifies non-null columns and can take slightly
longer.
Before:
```
explain analyze SELECT p.*, (SELECT COUNT(Posts.Id) FROM Posts WHERE Posts.RootId = (CASE WHEN p.RootId = '' THEN p.Id ELSE p.RootId END) AND Posts.DeleteAt = 0) AS ReplyCount FROM Posts p WHERE (CreateAt < (SELECT CreateAt FROM Posts WHERE Id = 'bguxt6zzcjytpffgza3dc7xody') AND p.ChannelId = '5xmynf36cinrzksuzhq7my1nbc' AND DeleteAt = 0) ORDER BY p.ChannelId, DeleteAt, CreateAt DESC LIMIT 30 OFFSET 0;
QUERY PLAN
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Limit (cost=9.14..130616.04 rows=30 width=922) (actual time=19.208..19.454 rows=30 loops=1)
InitPlan 2 (returns $2)
-> Index Scan using posts_pkey on posts posts_1 (cost=0.56..8.58 rows=1 width=8) (actual time=0.028..0.030 rows=1 loops=1)
Index Cond: ((id)::text = 'bguxt6zzcjytpffgza3dc7xody'::text)
-> Index Scan Backward using idx_posts_channel_id_delete_at_create_at on posts p (cost=0.56..2720977.61 rows=625 width=922) (actual time=0.075..0.317 rows=30 loops=1)
Index Cond: (((channelid)::text = '5xmynf36cinrzksuzhq7my1nbc'::text) AND (deleteat = 0) AND (createat < $2))
SubPlan 1
-> Aggregate (cost=4349.49..4349.50 rows=1 width=8) (actual time=0.007..0.007 rows=1 loops=30)
-> Index Scan using idx_posts_root_id_delete_at on posts (cost=0.56..4346.36 rows=1253 width=27) (actual time=0.007..0.007 rows=0 loops=30)
Index Cond: (((rootid)::text = (CASE WHEN ((p.rootid)::text = ''::text) THEN p.id ELSE p.rootid END)::text) AND (deleteat = 0))
```
After:
```
explain analyze SELECT p.*, (SELECT COUNT(Posts.RootId) FROM Posts WHERE Posts.RootId = (CASE WHEN p.RootId = '' THEN p.Id ELSE p.RootId END) AND Posts.DeleteAt = 0) AS ReplyCount FROM Posts p WHERE (CreateAt < (SELECT CreateAt FROM Posts WHERE Id = 'bguxt6zzcjytpffgza3dc7xody') AND p.ChannelId = '5xmynf36cinrzksuzhq7my1nbc' AND DeleteAt = 0) ORDER BY p.ChannelId, DeleteAt, CreateAt DESC LIMIT 30 OFFSET 0;
QUERY PLAN
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Limit (cost=9.14..130616.04 rows=30 width=922) (actual time=12.936..13.188 rows=30 loops=1)
InitPlan 2 (returns $2)
-> Index Scan using posts_pkey on posts posts_1 (cost=0.56..8.58 rows=1 width=8) (actual time=0.029..0.030 rows=1 loops=1)
Index Cond: ((id)::text = 'bguxt6zzcjytpffgza3dc7xody'::text)
-> Index Scan Backward using idx_posts_channel_id_delete_at_create_at on posts p (cost=0.56..2720977.61 rows=625 width=922) (actual time=0.076..0.324 rows=30 loops=1)
Index Cond: (((channelid)::text = '5xmynf36cinrzksuzhq7my1nbc'::text) AND (deleteat = 0) AND (createat < $2))
SubPlan 1
-> Aggregate (cost=4349.49..4349.50 rows=1 width=8) (actual time=0.008..0.008 rows=1 loops=30)
-> Index Only Scan using idx_posts_root_id_delete_at on posts (cost=0.56..4346.36 rows=1253 width=8) (actual time=0.007..0.007 rows=0 loops=30)
Index Cond: ((rootid = (CASE WHEN ((p.rootid)::text = ''::text) THEN p.id ELSE p.rootid END)::text) AND (deleteat = 0))
Heap Fetches: 14
```
https://mattermost.atlassian.net/browse/MM-40799
```release-note
NONE
```
* Use COUNT(*) for faster
```release-note
NONE
```
This query deals with posts already created,
and it was originally already querying replica.
In recent performance investigations, this came up
a lot of times. I believe the change to query master
was unintentional.
```release-note
NONE
```
