* Running CodeQL everyday twice
* Updated scorecard-check to run everyday
* Running Scorecard job once a day
* Disabling Scorecard Job on every PR
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
We implement a cursor based pagination model
to page through the posts in a given thread.
The cursor is a combination of the post.CreateAt+
post.Id to differentiate multiple posts in a given
timestamp.
Some additional parameters like direction, fromPost,
fromCreateAt and perPage were introduced to implement
this.
```release-note
NONE
```
Summary
A number of performance improvements shipped in v6.5 to support CRT with the ThreadAutoFollow bookkeeping enabled but the frontend features disabled. To prepare for a later enabling of the frontend features by default, we want to encourage customers to re-enable ThreadAutoFollow, and plan to enable this by default in v6.6.
Ticket Link
https://mattermost.atlassian.net/browse/MM-42535
Due to the way our community deployment is done. The job server
is restarted every day. This means that whenever there is a job
that takes more than 24 hours, it will always get cancelled
when the server restarts and therefore will never finish.
This PR adds ability to resume any stopped jobs, by storing
intermediate progress in the job metadata and setting
the job to pending instead of cancelled when everything is
shut down.
The user can still cancel a job explicitly by clicking on
the cross button in the system console. That functionality
hasn't changed. Only server stop or stopping/starting
job server via config will pause/resume jobs.
```release-note
The elasticsearch indexing job is resumable now. Stopping a
server while the job is running will put the job in pending status
and will resume the job when the server starts.
The job can still be explicitly cancelled via the system console UI.
```
This fixes a classic concurrency bug where the stop channel
was being listened in 2 places. In the Run method and DoJob
method.
The problem with that is that one of them will consume
the signal, and if the DoJob method does, then the parent
for-select loop never exits, because the return statement
only runs when the stop signal is consumed.
This would lead to server hanging on shutdown.
To fix this, we close the channel rather than sending a
message. This exits all listeners listening on the channel
and lets the job exit gracefully.
This is a sample stack dump of a stuck goroutine
```
goroutine 1 [chan receive]:
runtime.gopark(0x235be40?, 0xc000faa530?, 0xa8?, 0xf3?, 0x27a1170?)
/home/agniva/gosource/go/src/runtime/proc.go:366 +0xd6 fp=0xc002e77510 sp=0xc002e774f0 pc=0x43c856
runtime.chanrecv(0xc000faa7e0, 0x0, 0x1)
/home/agniva/gosource/go/src/runtime/chan.go:577 +0x56c fp=0xc002e775a0 sp=0xc002e77510 pc=0x40868c
runtime.chanrecv1(0x27cab4c?, 0xf?)
/home/agniva/gosource/go/src/runtime/chan.go:440 +0x18 fp=0xc002e775c8 sp=0xc002e775a0 pc=0x4080b8
github.com/mattermost/mattermost-server/v6/jobs/migrations.(*Worker).Stop(0xc001954280)
/home/agniva/mattermost/mattermost-server/jobs/migrations/worker.go:66 +0x171 fp=0xc002e77698 sp=0xc002e775c8 pc=0x187f8d1
github.com/mattermost/mattermost-server/v6/jobs.(*Workers).Stop(0xc0019df2c0)
/home/agniva/mattermost/mattermost-server/jobs/workers.go:84 +0x12c fp=0xc002e77748 sp=0xc002e77698 pc=0x17b04ec
github.com/mattermost/mattermost-server/v6/jobs.(*JobServer).StopWorkers(0xc00021f180?)
/home/agniva/mattermost/mattermost-server/jobs/server.go:104 +0x9c fp=0xc002e777a0 sp=0xc002e77748 pc=0x17af8bc
github.com/mattermost/mattermost-server/v6/app.(*Server).Shutdown(0xc00021f180)
/home/agniva/mattermost/mattermost-server/app/server.go:1019 +0x965 fp=0xc002e77c48 sp=0xc002e777a0 pc=0x1dd6865
github.com/mattermost/mattermost-server/v6/cmd/mattermost/commands.runServer.func3()
/home/agniva/mattermost/mattermost-server/cmd/mattermost/commands/server.go:79 +0x26 fp=0xc002e77c60 sp=0xc002e77c48 pc=0x214bbe6
github.com/mattermost/mattermost-server/v6/cmd/mattermost/commands.runServer(0xc000e1e000, 0xb?)
/home/agniva/mattermost/mattermost-server/cmd/mattermost/commands/server.go:122 +0x239 fp=0xc002e77d50 sp=0xc002e77c60 pc=0x214ba39
github.com/mattermost/mattermost-server/v6/cmd/mattermost/commands.serverCmdF(0x4207ac0?, {0x279d9d6?, 0x0?, 0x0?})
/home/agniva/mattermost/mattermost-server/cmd/mattermost/commands/server.go:58 +0x11d fp=0xc002e77dd0 sp=0xc002e77d50 pc=0x214b37d
github.com/spf13/cobra.(*Command).execute(0x4207ac0, {0xc00003c200, 0x0, 0x0})
/home/agniva/mattermost/mattermost-server/vendor/github.com/spf13/cobra/command.go:856 +0x67c fp=0xc002e77ea8 sp=0xc002e77dd0 pc=0x211a21c
github.com/spf13/cobra.(*Command).ExecuteC(0x4207ac0)
/home/agniva/mattermost/mattermost-server/vendor/github.com/spf13/cobra/command.go:974 +0x3b4 fp=0xc002e77f60 sp=0xc002e77ea8 pc=0x211a894
github.com/spf13/cobra.(*Command).Execute(...)
/home/agniva/mattermost/mattermost-server/vendor/github.com/spf13/cobra/command.go:902
github.com/mattermost/mattermost-server/v6/cmd/mattermost/commands.Run(...)
/home/agniva/mattermost/mattermost-server/cmd/mattermost/commands/root.go:14
main.main()
/home/agniva/mattermost/mattermost-server/cmd/mattermost/main.go:31 +0x86 fp=0xc002e77f80 sp=0xc002e77f60 pc=0x21f9ee6
runtime.main()
/home/agniva/gosource/go/src/runtime/proc.go:255 +0x227 fp=0xc002e77fe0 sp=0xc002e77f80 pc=0x43c487
runtime.goexit()
/home/agniva/gosource/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc002e77fe8 sp=0xc002e77fe0 pc=0x46d361
```
This is not strictly related to the JIRA ticket but
a general refactor
https://mattermost.atlassian.net/browse/MM-40272
```release-note
NONE
```
* Updated links to legacy domain about.mm.com
* Legacy link updates
* Reverting - only need to change en.json
* Reverting - only need to change en.json
* Reverting - only need to change en.json
* Reverting - only need to change en.json
* Reverting - only need to change en.json
* Reverting - only need to change en.json
* Reverting - only need to change en.json
* Reverting - only need to change en.json
* Reverting - only need to change en.json
* Reverting - only need to change en.json
* Reverting - only need to change en.json
* Reverting - only need to change en.json
* Reverting - only need to change en.json
* Reverting - only need to change en.json
* Reverting - only need to change en.json
* Reverting - only need to change en.json
* Reverting - only need to change en.json
* Reverting - only need to change en.json
* Reverting - only need to change en.json
* about.mm.com URL updates
* mattermost.org URL updates
* forum.mm.org URL update
* Update .github/ISSUE_TEMPLATE.md
* Update .github/ISSUE_TEMPLATE.md
* Un-deleted language files
* Update README.md
* Update tests/test-config.json
* fix some test due to url updating (#19787)
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
Co-authored-by: Harrison Healey <harrisonmhealey@gmail.com>
Co-authored-by: Ibrahim Serdar Acikgoz <serdaracikgoz86@gmail.com>
* revamp db version and add applied migrations endpoint
* replace old schema version with new
* add db version subcommand
* add to local api
* reflect review comments
* log errors
* remove setting the version from model.CurrentVersion
* fix a test
* use different field for schema version
* add build hash and current version to the support packet
* add tests
* update test to use new assets
* MM-42282: handle teamId parameter correctly
As per https://community-daily.mattermost.com/core/pl/ugs7ue6e4j8a7cgegk1bxje8to, `ThreadStore.GetThreadsForUser` accepts a `teamId` parameter, but incorrectly handles an empty value of `""` as looking only for channels with an empty `teamId` (aka DMs and GMs) instead of finding all channels and effectively ignoring the team property.
Fixes: https://mattermost.atlassian.net/browse/MM-42282
* break up getThreadsForUser, leverage errgroup
This change breaks up `GetThreadsForUser` in the `ThreadStore` into its constituent `GetTotalUnreadThreads`, `GetTotalThreads`, `GetTotalUnreadMentions`, and the original `GetThreadsForUser` but now solely returning the thread structures. Instead of a monolithic method at the store level, the application layer now handles calling bulk requests, leveraging `errgroup` for simpler parallelization.
This change brings with it a few benefits:
* Simpler code, including more idiomatic usage of squirrel
* Simpler SQL, joining tables only when configured conditions require same. (No performance benefit here, since an unused LEFT JOIN generally has no overhead.)
* Discrete Grafana metrics for each store method, giving us better insight into the performance characteristics in play.
* **Performance boost**: reduced overhead when clearing push notifications.
This last point is what prompted the re-re-reactoring in this PR. As I broke things up, I realized that `clearPushNotificationSync` only used the `TotalUnreadMentions`, but asked for the count of total threads and total unread threads. By exposing the discrete methods, this code path avoids two aggregate queries. We clear notifications when marking a thread as read, and when marking a channel with unread mentions as viewed, so I expect we'll see at least a modest boost to performance from simply not wasting these cycles anymore.
No performance improvements are expected from this PR for the general case of using `GetThreadsForUser` to populate the threads view.
* never discard errors from building queries
* no MustSql
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* MM-40179 traverse directories when trying to export
* fix test
* fix s3 to behabe the same way
* add test for paths way too deep
* test recursion only on localstorage
* make list directories non recursively the default approach
* fix linting error
* fix non-recursive case
Co-authored-by: = <=>
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
