* ci: shard server Postgres tests into 4 parallel runners (#35739)
* ci: add test sharding plumbing to server CI
Add infrastructure for upcoming test sharding without changing behavior:
- Add shard-index and shard-total inputs to server-test-template.yml
(defaults preserve existing single-runner behavior)
- Add timing cache restore step (activates only when shard-total > 1)
- Add merge-postgres-test-results job to server-ci.yml that:
- Merges JUnit XML reports from shard artifacts
- Saves timing data cache for future shard balancing
- Handles both single-artifact and multi-shard scenarios
- Add .gitignore entries for timing cache and shard work files
Co-authored-by: Claude <claude@anthropic.com>
* ci: shard server Postgres tests into 4 parallel runners
Extract sharding logic into standalone, tested scripts and enable
4-shard parallel test execution for server Postgres CI:
Scripts:
- server/scripts/shard-split.js: Node.js bin-packing solver that
assigns test packages to shards using timing data from previous runs.
Two-tier strategy: light packages (<2min) whole, heavy packages
(api4, app) split at individual test level.
- server/scripts/run-shard-tests.sh: Multi-run wrapper that calls
gotestsum directly for each package group with -run regex filters.
- server/scripts/shard-split.test.js: 8 test cases covering round-robin
fallback, timing-based balancing, heavy package splitting, JUnit XML
fallback, and enterprise package separation.
Workflow changes:
- server-test-template.yml: Add shard splitting step that discovers test
packages and runs the solver. Modified Run Tests step to use wrapper
script when sharding is active.
- server-ci.yml: Add 4-shard matrix to test-postgres-normal. Update
merge job artifact patterns for shard-specific names.
Performance: 7.2 min with timing cache vs 62.5 min baseline = 88%
wall-time improvement. First run without cache uses JUnit XML fallback
or round-robin, then populates the cache for subsequent runs.
Co-authored-by: Claude <claude@anthropic.com>
* fix: raise heavy package threshold to 5 min to preserve test isolation
sqlstore integrity tests scan the entire database and fail when other
packages' test data is present. At 182s, sqlstore was just over the
120s threshold and getting split at test level. Raising to 300s keeps
only api4 (~38 min) and app (~15 min) as heavy — where the real
sharding gains are — while sqlstore, elasticsearch, etc. stay whole
and maintain their test isolation guarantees.
Co-authored-by: Claude <claude@anthropic.com>
* ci: only save test timing cache on default branch
PR branches always restore from master's timing cache via restore-keys
prefix matching. Timing data is stable day-to-day so this eliminates
cache misses on first PR runs and reduces cache storage.
Co-authored-by: Claude <claude@anthropic.com>
* ci: skip FIPS tests on PRs (enterprise CI handles compile check)
Per review feedback: the enterprise CI already runs a FIPS compile
check on every PR. Running the full FIPS test suite on PRs is redundant
since it uses the identical test suite as non-FIPS — the only
FIPS-specific failure mode is a build failure from non-approved crypto
imports, which the enterprise compile check catches.
Full FIPS tests continue to run on every push to master.
Co-authored-by: Claude <claude@anthropic.com>
* fix: address review feedback on run-shard-tests.sh
- Remove set -e so all test runs execute even if earlier ones fail;
track failures and exit with error at the end (wiggin77)
- Remove unused top-level COVERAGE_FLAG variable (wiggin77)
- Fix RUN_IDX increment position so report, json, and coverage files
share the same index (wiggin77)
- Update workflow comment: heavy threshold is 5 min, not 2 min (wiggin77)
Co-authored-by: Claude <claude@anthropic.com>
* style: use node: prefix for built-in fs module in shard-split.js
Co-authored-by: Claude <claude@anthropic.com>
* fix: avoid interpolating file paths into generated shell script
Read shard package lists from files at runtime instead of interpolating
them into the generated script via printf. This prevents theoretical
shell metacharacter injection from directory names, as flagged by
DryRun Security.
Co-authored-by: Claude <claude@anthropic.com>
* fix(ci): rename merged artifact to match server-ci-report glob
The merged artifact was named postgres-server-test-logs-merged which
does not match the *-test-logs pattern in server-ci-report.yml,
causing Postgres test results to be missing from PR/commit reports.
Also pins junit-report-merger to exact version 7.0.0 for supply chain
safety.
Co-authored-by: Claude <claude@anthropic.com>
* fix(ci): pass RACE_MODE env into Docker container
RACE_MODE was set on the host runner but never included in the docker
run --env list. The light-package path worked because the heredoc
expanded on the host, but run-shard-tests.sh reads RACE_MODE at
runtime inside the container where it was unset. This caused heavy
packages (api4, app) to silently lose -race detection.
Co-authored-by: Claude <claude@anthropic.com>
* fix(ci): discover new tests in heavy packages not in timing cache
Tests not present in the timing cache (newly added or renamed) would
not appear in any shard -run regex, causing them to silently skip.
After building items from the cache, run go test -list to discover
current test names and assign any cache-missing tests to shards via
the normal bin-packing algorithm with a small default duration.
Co-authored-by: Claude <claude@anthropic.com>
* fix(ci): add missing line continuation backslash in docker run
The previous --env FIPS_ENABLED line was missing a trailing backslash
after adding --env RACE_MODE, causing docker run to see a truncated
command and fail with "requires at least 1 argument".
Co-authored-by: Claude <claude@anthropic.com>
* fix(ci): add setup-go step for shard test discovery
go test -list in shard-split.js runs on the host runner via execSync,
but Go is only available inside the Docker container. Without this
step, every invocation fails silently and new-test discovery is a
no-op. Adding actions/setup-go before the shard split step ensures
the Go toolchain is available on the host.
Co-authored-by: Claude <claude@anthropic.com>
---------
Co-authored-by: Claude <claude@anthropic.com>
* updated go to version 1.25.8 (#35817)
* updated go to version 1.25.8
* updated gotestsum version to work with go 1.25.8
go 1.25 does not work with indirect tools 0.11 dependency pulled by
gotestsum.
* Use sync.WaitGroup.Go to simplify goroutine creation
Replace the wg.Add(1) + go func() { defer wg.Done() }() pattern with
wg.Go(), which was introduced in Go 1.25.
* pushes fips image on workflow dispatch to allow fips test to run on go version update
* fix new requirements for FIPS compliance imposed on updating to go 1.25.8
* updates openssl symbol check for library shipped with FIPS new versions
go-openssl v2 shipped with FIPS versions starting from 1.25 uses mkcgo to generate
bindings causing symbol names to be different.
* removes temp workflow-dispatch condition
* keep versions out of agents md file
* upgrade golangci-lint (#35845)
* test: clean up channel store data after TestChannelStore (#36066)
TestChannelStore sub-tests create channels, members, and team members
using fake TeamIds and UserIds (model.NewId() for non-existent rows).
These records are left in the database and cause integrity tests
(TestCheck*) running in the same binary to fail their full-table scans.
Register a t.Cleanup on TestChannelStore that purges the affected
tables entirely. A blanket purge is safe: the schema enforces no FK
constraints, and every test suite creates its own data independently.
* Fix command injection in server-test-template workflow (#36080)
Replace the unquoted heredoc (which embedded GITHUB_HEAD_REF into a
generated script) with a cp of the existing run-shard-tests.sh, which
already handles the light-only case. Pass BUILD_NUMBER and TEST_TARGET
as explicit docker env vars instead of interpolating them into script
content.
* fix(ci): restore testname format in sharded gotestsum runs (#36078)
run-shard-tests.sh called gotestsum directly without --format, so it
fell back to gotestsum's default (pkgname) instead of the testname
format set by the Makefile. Pass --format "${GOTESTSUM_FORMAT:-testname}"
to match the Makefile default.
Co-authored-by: Mattermost Build <build@mattermost.com>
* fix(lint): fix pre-existing golangci-lint v2.11.4 issues
Fix misspelling in comment and redundant nil check flagged by the
upgraded linter.
* ci: use golang image for test runner on release-10.11
mattermost-build-server images are not built for release branches.
Use the official golang image which is always available for any Go version.
* ci: use mattermost/mattermost-build-server for release-10.11
The mattermostdevelopment/ images are only built for master.
The production mattermost/ images are built for release branches.
* ci: use mattermost/mattermost-build-server in mmctl test template
The mattermostdevelopment/ images are only built for master.
The production mattermost/ images are built for release branches.
---------
Co-authored-by: Pavel Zeman <pavel.zeman@mattermost.com>
Co-authored-by: Claude <claude@anthropic.com>
Co-authored-by: Carlos Garcia <carlos.garcia@mattermost.com>
Co-authored-by: Mattermost Build <build@mattermost.com>
We've recently spent some effort improving websocket reconnection logic. With this commit, I've augmented the websocket reconnect metric to include a disconnect reason. This will help us measure the impact of these changes in production.
When we added iteration by channelID, this was a known tradeoff during that.
However, it has been observed that the regular connection removal function
creates considerable blocking of the processing loop, leading to high
CPU usage and API latencies.
To fix this, we add a reverse mapping of channelIDs to connections
and their positions in the slice. This helps us to remove the connection
from the slice without iteration.
Unfortunately, this still needs to iterate through all channelIDs
during invalidation of the channel member cache. However, the user
cache invalidation is not a regular activity. So it should be an acceptable
tradeoff to make.
https://mattermost.atlassian.net/browse/MM-62960
```release-note
A new config knob ServiceSettings.EnableWebHubChannelIteration which allows a user to control the performance of websocket broadcasting. By default, this setting is turned off. If it is turned on, it improves the websocket broadcasting performance at the expense of poor performance when users join/leave a channel. It is not recommended to turn it on unless you have atleast 200,000 concurrent users actively using MM.
```
Co-authored-by: Mattermost Build <build@mattermost.com>
We do a cluster request to get the active and dead queues
from other nodes in the cluster to sync any missing
information.
We check the dead queue in the other nodes to see
if there's been any message loss or not. Accordingly,
we send just the active queue or both active and dead queues.
There's still an edge case that is left out where
a client could have potentially connected and reconnected
to multiple nodes leaving multiple active queues
in multiple nodes. We don't handle this scenario
because then potentially we need to create
a slice of sendQueueSize * number_of_nodes. And then
this can happen again, leading to an infinite increase
in sendQueueSize.
We leave this edge-case to Redis, acknowledging
a limitation in our architecture.
In this PR, when there's no message loss, we just
take the active queue from the last node it connected
to.
And if there's message loss where the client's
seqNum is within the last node's dead queue, we also
handle that.
But if there's severe message loss where the client's
seqNum falls within the dead queue of another node, then
we just send the data from that node to reconstruct the
data as much as possible. It could be possible to set
a new connection ID in this case, but this involves
more data transfer always from all nodes and recomputing
the state in the requestor node.
https://mattermost.atlassian.net/browse/MM-61904
```release-note
NONE
```
Co-authored-by: Mattermost Build <build@mattermost.com>
This introduces a new entry in the `Main Menu -> About` modal with the hostname of the currently connected websocket. This will be used to aid debugging issues in clustered environments by showing which node in the cluster is servicing requests for a particular websocket.
This information is only visible in self-managed instances. It will not be visible on cloud instances.
Tests at very high scale indicates that the iteration
of all connections during websocket broadcast starts
to become a bottleneck.
To optimize this, we move the channelMember cache from
inside web_conn.go to the hubConnectionIndex.
This involves adding a new map keyed by the channelID
and containing all webConns where the user is a member
of that channel. Subsequently, a new method needed to
be added to invalidate the cache which previously
used to happen in web_conn.
And as a last step, we remove the cache from web_conn
to reduce SQL queries to the DB.
https://mattermost.atlassian.net/browse/MM-61130
```release-note
NONE
```
This originated from https://github.com/mattermost/mattermost/issues/15249.
However, the original idea was discarded https://github.com/mattermost/mattermost/issues/15249#issuecomment-709713065
as being too complicated to implement. Then I had another
idea to implement it just for session objects.
My thinking was that since every single request allocates a new
session struct, it would be good to use a sync.Pool for that.
However, 4 years later, now we know that the primary bottleneck
in app performance comes from websocket event marshalling.
Therefore, while it would be good to do this, it is difficult
to do it correctly (as shown by the numerous racy tests).
Hence, reverting this.
```release-note
NONE
```
* [MM-58355] Send invalidate cache message across the cluster so that websocket connections on other instances are invalidated correctly
* Add suggestion to clear the session cache on the local node as well
* Force read from master DB when gettting channel members for websocket to avoid any DB sync issues
* PR feedback
* Missed generated files
We were setting the user status to offline without
checking for connections on other nodes in a cluster.
Now we implement a request-response mechanism for the whole
cluster and we check that before setting a user to offline.
https://mattermost.atlassian.net/browse/MM-57153
```release-note
Fix a bug where the user status would incorrectly be set to offline
without checking for connections in other nodes in an HA cluster.
```
Co-authored-by: Ibrahim Serdar Acikgoz <serdaracikgoz86@gmail.com>
Co-authored-by: Mattermost Build <build@mattermost.com>
Co-authored-by: Ben Schumacher <ben.schumacher@mattermost.com>
* Explicitly have the client tell the server when it should expect an ACK
* Don't count missing profile errors for your own posts, added comment
* Fix test
* Make postedAck a parameter in WebSocketClient
* Snapshot fixes
---------
Co-authored-by: Mattermost Build <build@mattermost.com>
We can have 2 types of threads open at the same time. One from
the thread view, and another from RHS.
We add another variable to distinguish between the 2 states.
In future, if we have the ability for more than 2 threads, then
we would need to track by threadID.
https://mattermost.atlassian.net/browse/MM-56071
```release-note
NONE
```
We also track the channelID of the thread opened.
Additionally on every connection create or re-connect
with an existing queue, we reset the active state to empty
to avoid any edge-cases.
https://mattermost.atlassian.net/browse/MM-56071
```release-note
NONE
```
We create a new websocket action called "presence" which
can contain the active_channel and the active_team for a given
client connection.
On the client side, for every channel or team switch, we send
out this message.
https://mattermost.atlassian.net/browse/MM-56060
```release-note
NONE
```
Marshalling a json.RawMessage is not zero overhead. Instead,
it compacts the raw message which starts to have an overhead
at scale.
https://github.com/golang/go/issues/33422
Since we have full control over the message constructed, we
can simply write the byte slice into the network stream.
This gives considerable performance boost.
```
goos: linux
goarch: amd64
pkg: github.com/mattermost/mattermost/server/public/model
cpu: Intel(R) Core(TM) i5-8265U CPU @ 1.60GHz
│ old.txt │ new_2.txt │
│ sec/op │ sec/op vs base │
EncodeJSON-8 1640.5n ± 2% 289.6n ± 1% -82.35% (p=0.000 n=10)
│ old.txt │ new_2.txt │
│ B/op │ B/op vs base │
EncodeJSON-8 528.0 ± 0% 503.0 ± 0% -4.73% (p=0.000 n=10)
│ old.txt │ new_2.txt │
│ allocs/op │ allocs/op vs base │
EncodeJSON-8 5.000 ± 0% 4.000 ± 0% -20.00% (p=0.000 n=10)
```
P.S. No concerns over changing the model API because we are
still using 0.x
https://mattermost.atlassian.net/browse/MM-54998
```release-note
Improve websocket event marshalling performance
```
I was looking at the high number of user_updated ws events
and ran across this piece of code. Had a hard time digesting it,
thought of adding a comment for people to understand it more
clearly.
```release-note
NONE
```
After reliable websockets were introduced,
we would keep pushing to a webConn even after
the client has disconnected. This would lead
to the websocket.slow/full and eventually
the disconnect messages which unnecessarily
confuse the admin.
We don't log them if the connection is inactive.
We use the active field and convert it to an atomic
to use more widely.
```release-note
NONE
```
* Mark category as read
* Fix lint and test
* Fix tests
* Fix test and remove wrong aria
* Address server issues and add mark as read for unreads
* Missing changes
* Fix tests
* fix tests
* Add confirmation popup to mark as read category
* Always use viewMultipleChannels and other fixes
* Remove unneeded code
* Fix test
* Address feedback
* Address feedback
* Fix tests
* Fix test
* Fix tests
* Update aria-haspopup depending on the number of channels to mark as viewed
---------
Co-authored-by: Mattermost Build <build@mattermost.com>
Sometimes a broad distinction of just a channelID or a userID
is not enough to efficiently send a websocket event to users.
In several cases, depending on the user and channel, we might
need to modify the message. Therefore, we introduce the
concept of a channel hook that will get executed if the scope
is set to a channel. This hook can be populated at the app layer
to perform any application specific logic to the event.
Care must be taken to avoid race conditions as the passed event
is not deep copied. It is left to the user to treat it carefully.
For this issue, the main problem was that since we don't know
which users have permissions to which channels, we had to go through
_all_ members of a channel to figure that out. This was redundant
since a large portion of those users might not even be connected
at that time.
We solve this with the channel hook where we push this check
to be performed later while actually sending the event. This
reduces the computation to be done only for _connected_ users
rather than _all_ users of a channel.
The next iteration of this should be to use websocket subscriptions
to monitor exactly which users are on that channel to even
trim down that list. That is a larger initiative to be taken later.
Tested locally with a channel of 50 users. Here are rough results:
```
With PR:
patchPost 97ms
createPost 90ms
Master:
patchPost 306ms
createPost - 298ms
```
https://mattermost.atlassian.net/browse/MM-42810
```release-note
Improve performance while sending messages
with permalinks to channels with large number of users.
```
It was a good decision in hindsight to keep the public module as 0.x
because this would have been a breaking change again.
https://mattermost.atlassian.net/browse/MM-53032
```release-note
Changed the Go module path from github.com/mattermost/mattermost-server/server/v8 to github.com/mattermost/mattermost/server/v8.
For the public facing module, it's path is also changed from github.com/mattermost/mattermost-server/server/public to github.com/mattermost/mattermost/server/public
```
* platform/web_conn: fix a possible data race issue
* reflect review comments
---------
Co-authored-by: Ibrahim Serdar Acikgoz <ibrahim@ibrahims-mac.local>
Co-authored-by: Mattermost Build <build@mattermost.com>
We log warnings whenever our websocket buffer sizes exceed
certain thresholds. The problem with that is, when this happens,
the logs are completely spammed with these lines
making it annoying for the customer.
To improve the situation, we use a timer that only gets reset
every minute.
https://mattermost.atlassian.net/browse/MM-51700
```release-note
NONE
```
