GetPostsSince is used when loading posts for a channel.
An opportunity for optimization is that the primary SQL query
is repeated twice and then a UNION is constructed for the results.
```
SELECT
*
FROM
Posts
WHERE
UpdateAt > :Time AND ChannelId = :ChannelId
LIMIT 1000
```
But we can use a CTE for this which caches the results to be reused later.
This leads to the main query being executed once rather than twice. And from
Postgres 12 onwards, CTEs can be inlined which opens the door to further optimizations.
From the docs (https://www.postgresql.org/docs/10/queries-with.html)
> A useful property of WITH queries is that they are evaluated only once per
execution of the parent query, even if they are referred to more than once
by the parent query or sibling WITH queries. Thus, expensive calculations
that are needed in multiple places can be placed within a WITH query to avoid redundant work.
Another possible application is to prevent unwanted multiple evaluations of functions
with side-effects. However, the other side of this coin is that the optimizer is
less able to push restrictions from the parent query down into a WITH query than an ordinary subquery.
In our case, the caveat does not apply because we are only filtering columns and not rows,
so we can safely use it.
Following are the query plan comparisons:
Old: http://tatiyants.com/pev/#/plans/plan_1599394993105
New: http://tatiyants.com/pev/#/plans/plan_1599395970886
As we can see, in old bitmap index scan+heap scan happens twice, but in the new one,
it happens only once.
This has been load tested with a large dataset and confirmed to exhibit good improvements.
Even after specifying all the 3 columns to coerce MySQL into choosing
the right multi-column index, it sometimes uses 2 separate indices
and does an index_merge of them.
This creates problems because the DeleteAt=0 search is essentially
the entire Posts table, and causes a disastrously bad performance
than even choosing the wrong index (idx_create_at).
The problem with this approach is that we hardcode the decision
to a specific index when MySQL was free to choose the right index
depending on table statistics. However, there does not appear to be
a case where this index can cause regressions than using some other index.
Another option here was to set optimizer_switch="index_merge_intersection=off"
at a session level for a transaction and then switch it back on after the
query is done.
However, this can cause some unintentional consequences because this setting
is only available at a session level and not at a query level.
There is no need to set something at the session level when an index hint suffices.
https://mattermost.atlassian.net/browse/MM-27575
* [MM-25714] Keeps track of the import lines of the posts while importing to report the right line on error
* Adding review comments
* Reverse the order of the error and error line params
* Fixing reply count on new posts
* Fixing tests
* Fixing post reply count on getPostsAround
Co-authored-by: mattermod <mattermod@users.noreply.github.com>
We use the same optimization used in MM-23369 to prevent
mysql from using the index in the sort query.
Co-authored-by: mattermod <mattermod@users.noreply.github.com>
* MM-23369: Allow mysql to choose a better index
When the ORDER BY clause contains a column which is in the WHERE
clause and also part of an index, mysql tries to use that specific
index to avoid sorting. This is inspite of the fact that
there may be other indices which are better for scanning the table
and then doing a sort.
Essentially, mysql becomes dumb and scans a lot of rows to avoid
sorting. Whereas, it could have scanned a lot less rows and do the
sorting in no time.
To fix this, we use the other columns in the ORDER BY clause as well
which are part of the index. This causes no change in the results
because the other columns are an EQUAL condition check, but this
lets mysql use the right index. Because now mysql sees that it has
to order by other columns too, so it better use the other index
to scan and then do the sorting.
This does not affect tables of smaller size because the LIMIT of
rows is always 1. And mysql will stop sorting the moment it gets
the first row. So sorting is not the overhead at all.
Therefore, this seems like an optimal fix.
References:
https://dev.mysql.com/doc/refman/5.7/en/table-scan-avoidance.htmlhttps://code.openark.org/blog/mysql/7-ways-to-convince-mysql-to-use-the-right-indexhttps://dev.mysql.com/doc/refman/5.7/en/limit-optimization.html
* Added a comment to clarify things in code
* Incorporating review comments
* WIP
* Adding bleve to go modules
* WIP
* Adding missing files from searchengine implementation
* WIP
* WIP
* WIP
* WIP
* WIP
* WIP
* User and channel indexing and searches implemented
* Make bleve tests run with in-memory indexes
* Implement post index and deletion tests
* Initial commits for the search layer
* Removing unnecesary indexing
* WIP
* WIP
* More fixes for tests
* Adding the search layer
* Finishing the migration of searchers to the layer
* Removing unnecesary code
* Allowing multiple engines active at the same time
* WIP
* Add simple post search
* Print information when using bleve
* Adding some debugging to understand better how the searches are working
* Making more dynamic config of search engines
* Add post search basics
* Adding the Purge API endpoint
* Fixing bleve config updates
* Adding missed file
* Regenerating search engine mocks
* Adding missed v5 to modules imports
* fixing i18n
* Fixing some test around search engine
* Removing all bleve traces
* Cleaning up the vendors directory and go.mod/go.sum files
* Regenerating timer layer
* Adding properly the license
* Fixing govet shadow error
* Fixing some tests
* Fixing TestSearchPostsFromUser
* Fixing another test
* Fixing more tests
* Fixing more tests
* Removing SearchEngine redundant text from searchengine module code
* Fixing some reindexing problems in members updates
* Fixing tests
* Addressing PR comments
* Reverting go.mod and go.sum
* Addressing PR comments
* Fixing tests compilation
* Fixing govet
* Adding search engine stop method
* Being more explicit on where we use includeDeleted
* Adding GetSqlSupplier test helper method
* Mocking elasticsearch start function
* Fixing tests
Co-authored-by: Miguel de la Cruz <miguel@mcrx.me>
Co-authored-by: mattermod <mattermod@users.noreply.github.com>
* Adding SaveMultiple to posts
* Improving tests
* fixing i18n
* Fixing tests
* Improving testing on top of Save and SaveMultiple
* Fixing shadow variables
* Addressing some PR comments
* More clear update post test
* Addressing some PR comments
* Addressing some PR comments and simplifying the code
* Improting replies in bulk too
* Fixing reply count and processing last imported replies
* Adding OverwriteMultiple to posts aggregating everything in the same transaction
* Adding 2 pending tests to implement
* Adding tests for overwrite multiple posts
* Adding tests for TeamStore.GetByNames method
* Fixing shadow variables
* Addressing PR comments
* Extracting i18n strings
* Fixing tests
* Fixing tests
* Adding more test cases
* Using a variable instead of a fake timestamp
* Consistent license message for all the go files
* Fixing the last set of unconsistencies with the license headers
* Addressing PR review comments
* Fixing busy.go and busy_test.go license header
* Some advances
* Partial advances
* Cache moved
* Tests finished
* Removed all test for PostStore (store/sqlstore) related with Cache. This is tested in the cache layer now
This change modifies the database query used by the getPostsAround()
method in order to improve the performance of loading posts when
scrolling through a channel.
* [MM-11210] Add API GET 'api/v4/channels/{channel_id:[A-Za-z0-9]+}/posts/unread' for scrolling overhaul (#9108)
* Add API GET 'api/v4/channels/{channel_id:[A-Za-z0-9]+}/posts/unread'
* add constants
* refactor GetPostSince and added more tests
* move constants to app package
* [MM-11528 && MM-11583] Add userId to in the "posts/unread" path and update test with time delay to fix intermittent failure (#9229)
* add userId to in the "posts/unread" path and update test with time delay to fix intermittent failure
* add limit before and after to query
* remove time delay on test and put pretermined value of Post.CreateAt
* Fix conflict
* [MM-11876] Add cursor to posts list such as next_post_id and previous_post_id (#9707)
* add cursor to posts list such as next_post_id and previous_post_id
add publish previous_post_id on WEBSOCKET_EVENT_POSTED and only get next or previous post IDs if necessary
revert change on adding previous_post_id in WEBSOCKET_EVENT_POSTED
add missing strings import
fix merge conflicts
* update per comment
* update per feedback
* corrected the logic in getting the next and previous post ID
* fix logic to determine next and post IDs, and rename function to have suffix of "Time"
* rearrange logics and add mote tests
* fix merge conflict
* fix missing message when using unread API (#10233)
* MM-15569 Fixes failing test on TestGetPostsForChannelAroundLastUnread (#11039)
* Fix missing posts when getting posts since
* revert changes to GetPostsSince
* migrate Post.GetPostAfterTime and Post.GetPostBeforeTime to sync by default
* revert change to cacheItem
* Fix post ID validation, build query on squirrel and only return post ID as necessary
* Add ability to get bot counts per day
Modify tests using AnalyticsPostCountsByDay to include extra input
(botsOnly)
Correctly generate the mock file with 'make store-mocks' target. Didn't
see these comments earlier
* Initial commit for calculating total posts previous day and total posts
from bots previous day
* - Refactor to use asserts instead of if statements in tests
- Capture inputs to AnalyticsPostCountsByDay() function into an
options struct
- Remove bot creation from diagnostics_test.go.
* Remove utils library
* create AnalyticsPostCountsOptions struct which is accepted as an input
to AnalyticsPostCountsByDay method
* Go vet fixes
* Migrate "Post.GetPostsSince" to Sync by default #10976
* Update GetPostsSince to Sync by default #10976
* Update GetPostsSince to Sync by default #10976
* Update GetPostsSince to Sync by default #10976
* MM-15841: migrate post save to sync by default #10987
* MM-15841: remove variable shadowing #10987
* MM-15841: log error on post save #10987
* MM-15841: nil check post save errors #10987
* MM-15841: update error message on post save #10987
* MM-15841: add nil check on post save in user store test #10987
