* refactor: added is_pinned attribute to post and direct_post objects when doing bulk import
* fix: handling IsPinned nil value in post and direct_post objects import
* refactor: added pinned post tests for post and direct_post objects import
* refactor: TestImportimportMultiplePostLines code enhancements + added pinned post tests in TestImportImportDirectPost
* fix: test message text
* fix: typo Post => DirectPost
* fix: typo Post => DirectPost
* refactor: remove redundant test
* refactor: test enhancements
* refactor: remove print statements.
* refactor: using require instead of assert
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* MM-40148: threadsOnly query param for user threads
Currently we always calculate counts when fetching user threads.
Those counts include total, unread replies, and unread mentions,
and are potentially expensive to calculate.
This commit adds a new query param 'threadsOnly' which won't calculate
any counts and just return threads.
Co-authored-by: koox00 <3829551+koox00@users.noreply.github.com>
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
The older method used to reply completely on timestamps
to take batches of items in a timestamp range and then
just incrementing the timestamp. This led to handling
edge-cases such as more items than the batch count, all
having the same timestamp.
Additionally, relying on timestamp as the page cursor
meant that indexing was not very efficient if you had
several items spread out across large spans of time.
To get away from all of that we use a proper cursor-based
approach consisting of createAt+Id. With this, we move
completely to a constant page size where we can fetch
a given number of objects irrespective of when they
were created. This makes indexing much more faster and
efficient.
https://mattermost.atlassian.net/browse/MM-41260
```release-note
Elasticsearch and Bleve indexing have been revamped to be much
more efficient and faster. The config parameter BulkIndexingTimeWindowSeconds
for both elasticsearch and bleve have been removed.
A new config parameter called BatchSize has been introduced instead.
This parameter controls the number of objects that
can be indexed in a single batch. This makes things
more efficient and maintains a constant workload.
```
When the test fails, it is not clear whether the method
returned an error or it returned a channel with DeleteAt
set to 0. The if condition was also incorrect because
if the error was not-nil the condition would evaluate
to true and the test would pass.
We split the condition into 2 separate cases which will
let us know exactly what is failing.
https://mattermost.atlassian.net/browse/MM-42092
```release-note
NONE
```
* Running CodeQL everyday twice
* Updated scorecard-check to run everyday
* Running Scorecard job once a day
* Disabling Scorecard Job on every PR
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
We implement a cursor based pagination model
to page through the posts in a given thread.
The cursor is a combination of the post.CreateAt+
post.Id to differentiate multiple posts in a given
timestamp.
Some additional parameters like direction, fromPost,
fromCreateAt and perPage were introduced to implement
this.
```release-note
NONE
```
Summary
A number of performance improvements shipped in v6.5 to support CRT with the ThreadAutoFollow bookkeeping enabled but the frontend features disabled. To prepare for a later enabling of the frontend features by default, we want to encourage customers to re-enable ThreadAutoFollow, and plan to enable this by default in v6.6.
Ticket Link
https://mattermost.atlassian.net/browse/MM-42535
Due to the way our community deployment is done. The job server
is restarted every day. This means that whenever there is a job
that takes more than 24 hours, it will always get cancelled
when the server restarts and therefore will never finish.
This PR adds ability to resume any stopped jobs, by storing
intermediate progress in the job metadata and setting
the job to pending instead of cancelled when everything is
shut down.
The user can still cancel a job explicitly by clicking on
the cross button in the system console. That functionality
hasn't changed. Only server stop or stopping/starting
job server via config will pause/resume jobs.
```release-note
The elasticsearch indexing job is resumable now. Stopping a
server while the job is running will put the job in pending status
and will resume the job when the server starts.
The job can still be explicitly cancelled via the system console UI.
```
This fixes a classic concurrency bug where the stop channel
was being listened in 2 places. In the Run method and DoJob
method.
The problem with that is that one of them will consume
the signal, and if the DoJob method does, then the parent
for-select loop never exits, because the return statement
only runs when the stop signal is consumed.
This would lead to server hanging on shutdown.
To fix this, we close the channel rather than sending a
message. This exits all listeners listening on the channel
and lets the job exit gracefully.
This is a sample stack dump of a stuck goroutine
```
goroutine 1 [chan receive]:
runtime.gopark(0x235be40?, 0xc000faa530?, 0xa8?, 0xf3?, 0x27a1170?)
/home/agniva/gosource/go/src/runtime/proc.go:366 +0xd6 fp=0xc002e77510 sp=0xc002e774f0 pc=0x43c856
runtime.chanrecv(0xc000faa7e0, 0x0, 0x1)
/home/agniva/gosource/go/src/runtime/chan.go:577 +0x56c fp=0xc002e775a0 sp=0xc002e77510 pc=0x40868c
runtime.chanrecv1(0x27cab4c?, 0xf?)
/home/agniva/gosource/go/src/runtime/chan.go:440 +0x18 fp=0xc002e775c8 sp=0xc002e775a0 pc=0x4080b8
github.com/mattermost/mattermost-server/v6/jobs/migrations.(*Worker).Stop(0xc001954280)
/home/agniva/mattermost/mattermost-server/jobs/migrations/worker.go:66 +0x171 fp=0xc002e77698 sp=0xc002e775c8 pc=0x187f8d1
github.com/mattermost/mattermost-server/v6/jobs.(*Workers).Stop(0xc0019df2c0)
/home/agniva/mattermost/mattermost-server/jobs/workers.go:84 +0x12c fp=0xc002e77748 sp=0xc002e77698 pc=0x17b04ec
github.com/mattermost/mattermost-server/v6/jobs.(*JobServer).StopWorkers(0xc00021f180?)
/home/agniva/mattermost/mattermost-server/jobs/server.go:104 +0x9c fp=0xc002e777a0 sp=0xc002e77748 pc=0x17af8bc
github.com/mattermost/mattermost-server/v6/app.(*Server).Shutdown(0xc00021f180)
/home/agniva/mattermost/mattermost-server/app/server.go:1019 +0x965 fp=0xc002e77c48 sp=0xc002e777a0 pc=0x1dd6865
github.com/mattermost/mattermost-server/v6/cmd/mattermost/commands.runServer.func3()
/home/agniva/mattermost/mattermost-server/cmd/mattermost/commands/server.go:79 +0x26 fp=0xc002e77c60 sp=0xc002e77c48 pc=0x214bbe6
github.com/mattermost/mattermost-server/v6/cmd/mattermost/commands.runServer(0xc000e1e000, 0xb?)
/home/agniva/mattermost/mattermost-server/cmd/mattermost/commands/server.go:122 +0x239 fp=0xc002e77d50 sp=0xc002e77c60 pc=0x214ba39
github.com/mattermost/mattermost-server/v6/cmd/mattermost/commands.serverCmdF(0x4207ac0?, {0x279d9d6?, 0x0?, 0x0?})
/home/agniva/mattermost/mattermost-server/cmd/mattermost/commands/server.go:58 +0x11d fp=0xc002e77dd0 sp=0xc002e77d50 pc=0x214b37d
github.com/spf13/cobra.(*Command).execute(0x4207ac0, {0xc00003c200, 0x0, 0x0})
/home/agniva/mattermost/mattermost-server/vendor/github.com/spf13/cobra/command.go:856 +0x67c fp=0xc002e77ea8 sp=0xc002e77dd0 pc=0x211a21c
github.com/spf13/cobra.(*Command).ExecuteC(0x4207ac0)
/home/agniva/mattermost/mattermost-server/vendor/github.com/spf13/cobra/command.go:974 +0x3b4 fp=0xc002e77f60 sp=0xc002e77ea8 pc=0x211a894
github.com/spf13/cobra.(*Command).Execute(...)
/home/agniva/mattermost/mattermost-server/vendor/github.com/spf13/cobra/command.go:902
github.com/mattermost/mattermost-server/v6/cmd/mattermost/commands.Run(...)
/home/agniva/mattermost/mattermost-server/cmd/mattermost/commands/root.go:14
main.main()
/home/agniva/mattermost/mattermost-server/cmd/mattermost/main.go:31 +0x86 fp=0xc002e77f80 sp=0xc002e77f60 pc=0x21f9ee6
runtime.main()
/home/agniva/gosource/go/src/runtime/proc.go:255 +0x227 fp=0xc002e77fe0 sp=0xc002e77f80 pc=0x43c487
runtime.goexit()
/home/agniva/gosource/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc002e77fe8 sp=0xc002e77fe0 pc=0x46d361
```
This is not strictly related to the JIRA ticket but
a general refactor
https://mattermost.atlassian.net/browse/MM-40272
```release-note
NONE
```
