The older method used to reply completely on timestamps
to take batches of items in a timestamp range and then
just incrementing the timestamp. This led to handling
edge-cases such as more items than the batch count, all
having the same timestamp.
Additionally, relying on timestamp as the page cursor
meant that indexing was not very efficient if you had
several items spread out across large spans of time.
To get away from all of that we use a proper cursor-based
approach consisting of createAt+Id. With this, we move
completely to a constant page size where we can fetch
a given number of objects irrespective of when they
were created. This makes indexing much more faster and
efficient.
https://mattermost.atlassian.net/browse/MM-41260
```release-note
Elasticsearch and Bleve indexing have been revamped to be much
more efficient and faster. The config parameter BulkIndexingTimeWindowSeconds
for both elasticsearch and bleve have been removed.
A new config parameter called BatchSize has been introduced instead.
This parameter controls the number of objects that
can be indexed in a single batch. This makes things
more efficient and maintains a constant workload.
```
* MM-39351: fixes LastRootPostAt being null
After upgrading to v6.1 there is a chance for LastRootPostAt to be NULL.
This might be due to a faulty migration when upgrading to v5.35.
v5.35 had `create_msg_root_count` where LastRootPostAt was used as a
temporary column which at the end was deleted.
Then on v6.1 we are adding that column again, this time to keep.
If the migration to v5.35 had failed and it didn't drop the column
the `upgrade_cte` (v6.1) migration would leave some channels with `LastRootPostAt`
being NULL.
This resulted in crashing the app.
This commit fixes that by running again a migration to again set
LastRootPostAt only when it's NULL this time and then set it to 0
for channels that might still have LastRootPostAt being NULL
(channels with no posts).
PS: the whole issue arose when upon creating the `upgrade_cte` migration
the create_msg_root_count migration was manually edited to change
the column name from LastRootPostAt to LastRootAt.
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* migrations: add if statements for some queries
* add better checks to upgrade posts v6 migration
* use "is null" instead of "= null"
* fix down script for users v6
* reflect review comments
* Create basic make commands for configuring golang-migrate
* Showcase full flow with new migrations
* Apply PR suggestions
* Migrate over team members
* Update mocks
* Fix specs
* Move columns that added after table creation onto separate stmts
* Put back gorp table definitions
* Fix issues with golang-migrate that not tracks underlying db driver
* Help prompt after new migration and consistent checksum for bindata
* Put gorp mapping back
* Apply PR suggestiong
* Close migrations after they run
* Add migration file to bindata check
* Updates needed
* Reset store_test
* Add copyright
* Apply PR suggestions
* Fix new circleci check
* Put back upgrade step for backwards comp
* Add store test to test migration directions
* Apply PR suggestions
* Add go-bindata to tools
* Apply PR suggestios
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
