The test at channels/app/platform/web_conn_test.go was added by #35988
without a defer th.TearDown() call. Setup(t) creates a PlatformService
that opens its own SqlStore pool and runs morph migrations against the
shared temp database. Without TearDown, that pool stays alive until
TestMain exits, so the morph-held idle connections are still attached
to the database when MainHelper.Close runs DROP DATABASE. Postgres
refuses the drop with "is being accessed by other users" and the whole
package panics in teardown.
On master and release-11.4 and later, setupTestHelper registers the
shutdown via tb.Cleanup automatically, so the same test does not leak
there. release-10.11 still uses the manual TearDown pattern, which is
why the fix is scoped to this branch.
* ci: shard server Postgres tests into 4 parallel runners (#35739)
* ci: add test sharding plumbing to server CI
Add infrastructure for upcoming test sharding without changing behavior:
- Add shard-index and shard-total inputs to server-test-template.yml
(defaults preserve existing single-runner behavior)
- Add timing cache restore step (activates only when shard-total > 1)
- Add merge-postgres-test-results job to server-ci.yml that:
- Merges JUnit XML reports from shard artifacts
- Saves timing data cache for future shard balancing
- Handles both single-artifact and multi-shard scenarios
- Add .gitignore entries for timing cache and shard work files
Co-authored-by: Claude <claude@anthropic.com>
* ci: shard server Postgres tests into 4 parallel runners
Extract sharding logic into standalone, tested scripts and enable
4-shard parallel test execution for server Postgres CI:
Scripts:
- server/scripts/shard-split.js: Node.js bin-packing solver that
assigns test packages to shards using timing data from previous runs.
Two-tier strategy: light packages (<2min) whole, heavy packages
(api4, app) split at individual test level.
- server/scripts/run-shard-tests.sh: Multi-run wrapper that calls
gotestsum directly for each package group with -run regex filters.
- server/scripts/shard-split.test.js: 8 test cases covering round-robin
fallback, timing-based balancing, heavy package splitting, JUnit XML
fallback, and enterprise package separation.
Workflow changes:
- server-test-template.yml: Add shard splitting step that discovers test
packages and runs the solver. Modified Run Tests step to use wrapper
script when sharding is active.
- server-ci.yml: Add 4-shard matrix to test-postgres-normal. Update
merge job artifact patterns for shard-specific names.
Performance: 7.2 min with timing cache vs 62.5 min baseline = 88%
wall-time improvement. First run without cache uses JUnit XML fallback
or round-robin, then populates the cache for subsequent runs.
Co-authored-by: Claude <claude@anthropic.com>
* fix: raise heavy package threshold to 5 min to preserve test isolation
sqlstore integrity tests scan the entire database and fail when other
packages' test data is present. At 182s, sqlstore was just over the
120s threshold and getting split at test level. Raising to 300s keeps
only api4 (~38 min) and app (~15 min) as heavy — where the real
sharding gains are — while sqlstore, elasticsearch, etc. stay whole
and maintain their test isolation guarantees.
Co-authored-by: Claude <claude@anthropic.com>
* ci: only save test timing cache on default branch
PR branches always restore from master's timing cache via restore-keys
prefix matching. Timing data is stable day-to-day so this eliminates
cache misses on first PR runs and reduces cache storage.
Co-authored-by: Claude <claude@anthropic.com>
* ci: skip FIPS tests on PRs (enterprise CI handles compile check)
Per review feedback: the enterprise CI already runs a FIPS compile
check on every PR. Running the full FIPS test suite on PRs is redundant
since it uses the identical test suite as non-FIPS — the only
FIPS-specific failure mode is a build failure from non-approved crypto
imports, which the enterprise compile check catches.
Full FIPS tests continue to run on every push to master.
Co-authored-by: Claude <claude@anthropic.com>
* fix: address review feedback on run-shard-tests.sh
- Remove set -e so all test runs execute even if earlier ones fail;
track failures and exit with error at the end (wiggin77)
- Remove unused top-level COVERAGE_FLAG variable (wiggin77)
- Fix RUN_IDX increment position so report, json, and coverage files
share the same index (wiggin77)
- Update workflow comment: heavy threshold is 5 min, not 2 min (wiggin77)
Co-authored-by: Claude <claude@anthropic.com>
* style: use node: prefix for built-in fs module in shard-split.js
Co-authored-by: Claude <claude@anthropic.com>
* fix: avoid interpolating file paths into generated shell script
Read shard package lists from files at runtime instead of interpolating
them into the generated script via printf. This prevents theoretical
shell metacharacter injection from directory names, as flagged by
DryRun Security.
Co-authored-by: Claude <claude@anthropic.com>
* fix(ci): rename merged artifact to match server-ci-report glob
The merged artifact was named postgres-server-test-logs-merged which
does not match the *-test-logs pattern in server-ci-report.yml,
causing Postgres test results to be missing from PR/commit reports.
Also pins junit-report-merger to exact version 7.0.0 for supply chain
safety.
Co-authored-by: Claude <claude@anthropic.com>
* fix(ci): pass RACE_MODE env into Docker container
RACE_MODE was set on the host runner but never included in the docker
run --env list. The light-package path worked because the heredoc
expanded on the host, but run-shard-tests.sh reads RACE_MODE at
runtime inside the container where it was unset. This caused heavy
packages (api4, app) to silently lose -race detection.
Co-authored-by: Claude <claude@anthropic.com>
* fix(ci): discover new tests in heavy packages not in timing cache
Tests not present in the timing cache (newly added or renamed) would
not appear in any shard -run regex, causing them to silently skip.
After building items from the cache, run go test -list to discover
current test names and assign any cache-missing tests to shards via
the normal bin-packing algorithm with a small default duration.
Co-authored-by: Claude <claude@anthropic.com>
* fix(ci): add missing line continuation backslash in docker run
The previous --env FIPS_ENABLED line was missing a trailing backslash
after adding --env RACE_MODE, causing docker run to see a truncated
command and fail with "requires at least 1 argument".
Co-authored-by: Claude <claude@anthropic.com>
* fix(ci): add setup-go step for shard test discovery
go test -list in shard-split.js runs on the host runner via execSync,
but Go is only available inside the Docker container. Without this
step, every invocation fails silently and new-test discovery is a
no-op. Adding actions/setup-go before the shard split step ensures
the Go toolchain is available on the host.
Co-authored-by: Claude <claude@anthropic.com>
---------
Co-authored-by: Claude <claude@anthropic.com>
* updated go to version 1.25.8 (#35817)
* updated go to version 1.25.8
* updated gotestsum version to work with go 1.25.8
go 1.25 does not work with indirect tools 0.11 dependency pulled by
gotestsum.
* Use sync.WaitGroup.Go to simplify goroutine creation
Replace the wg.Add(1) + go func() { defer wg.Done() }() pattern with
wg.Go(), which was introduced in Go 1.25.
* pushes fips image on workflow dispatch to allow fips test to run on go version update
* fix new requirements for FIPS compliance imposed on updating to go 1.25.8
* updates openssl symbol check for library shipped with FIPS new versions
go-openssl v2 shipped with FIPS versions starting from 1.25 uses mkcgo to generate
bindings causing symbol names to be different.
* removes temp workflow-dispatch condition
* keep versions out of agents md file
* upgrade golangci-lint (#35845)
* test: clean up channel store data after TestChannelStore (#36066)
TestChannelStore sub-tests create channels, members, and team members
using fake TeamIds and UserIds (model.NewId() for non-existent rows).
These records are left in the database and cause integrity tests
(TestCheck*) running in the same binary to fail their full-table scans.
Register a t.Cleanup on TestChannelStore that purges the affected
tables entirely. A blanket purge is safe: the schema enforces no FK
constraints, and every test suite creates its own data independently.
* Fix command injection in server-test-template workflow (#36080)
Replace the unquoted heredoc (which embedded GITHUB_HEAD_REF into a
generated script) with a cp of the existing run-shard-tests.sh, which
already handles the light-only case. Pass BUILD_NUMBER and TEST_TARGET
as explicit docker env vars instead of interpolating them into script
content.
* fix(ci): restore testname format in sharded gotestsum runs (#36078)
run-shard-tests.sh called gotestsum directly without --format, so it
fell back to gotestsum's default (pkgname) instead of the testname
format set by the Makefile. Pass --format "${GOTESTSUM_FORMAT:-testname}"
to match the Makefile default.
Co-authored-by: Mattermost Build <build@mattermost.com>
* fix(lint): fix pre-existing golangci-lint v2.11.4 issues
Fix misspelling in comment and redundant nil check flagged by the
upgraded linter.
* ci: use golang image for test runner on release-10.11
mattermost-build-server images are not built for release branches.
Use the official golang image which is always available for any Go version.
* ci: use mattermost/mattermost-build-server for release-10.11
The mattermostdevelopment/ images are only built for master.
The production mattermost/ images are built for release branches.
* ci: use mattermost/mattermost-build-server in mmctl test template
The mattermostdevelopment/ images are only built for master.
The production mattermost/ images are built for release branches.
---------
Co-authored-by: Pavel Zeman <pavel.zeman@mattermost.com>
Co-authored-by: Claude <claude@anthropic.com>
Co-authored-by: Carlos Garcia <carlos.garcia@mattermost.com>
Co-authored-by: Mattermost Build <build@mattermost.com>
When a user disconnects from the hub, we would spawn off a goroutine
which would make a cluster request, and then update the user status
as offline in the DB.
This was another case of unbounded concurrency where the number of
goroutines spawned was user controlled. Therefore, we would see
a clear spike in DB connections on master when a lot of users
would suddenly disconnect.
To fix this, we implement concurrency control in two areas:
1. In making the cluster request. We implement a counting semaphore
per-hub to avoid making unbounded cluster requests.
2. We use a buffered channel with a periodic flusher to process
status updates.
We also add a new store method to upsert multiple statuses
in a single query. The statusUpdateThreshold is set to 32, which means
no more than 32 rows will be upserted at one time, keeping the
SQL query load reasonable.
https://mattermost.atlassian.net/browse/MM-64298
```release-note
We improve DB connection spikes on user disconnect
by processing status updates in batches.
```
* Adds support for GMs in shared channels
* Fix linter
* Remove creatorID from slack call
---------
Co-authored-by: Miguel de la Cruz <miguel@ctrlz.es>
* Upgrade Go to 1.24.3
Updates the following files:
- server/.go-version: 1.23.9 → 1.24.3
- server/build/Dockerfile.buildenv: golang:1.23.9-bullseye → golang:1.24.3-bullseye
- server/go.mod: go 1.23.0 → go 1.24.3, toolchain go1.23.9 → go1.24.3
- server/public/go.mod: go 1.23.0 → go 1.24.3, toolchain go1.23.9 → go1.24.3
Also fixes non-constant format string errors introduced by Go 1.24.3's stricter format string checking:
- Added response() helper function in slashcommands/util.go for simple string responses
- Removed unused responsef() function from slashcommands/util.go
- Replaced responsef() with response() for translated strings that don't need formatting
- Fixed fmt.Errorf and fmt.Fprintf calls to use proper format verbs instead of string concatenation
- Updated marketplace buildURL to handle format strings conditionally
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
* Update generated mocks for Go 1.24.3
Regenerated mocks using mockery v2.53.4 to ensure compatibility with Go 1.24.3.
This addresses mock generation failures that occurred with the Go upgrade.
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
* Update to bookworm and fix non-existent sha
Signed-off-by: Stavros Foteinopoulos <stafot@gmail.com>
* fix non-constant format string
---------
Signed-off-by: Stavros Foteinopoulos <stafot@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Stavros Foteinopoulos <stafot@gmail.com>
We've recently spent some effort improving websocket reconnection logic. With this commit, I've augmented the websocket reconnect metric to include a disconnect reason. This will help us measure the impact of these changes in production.
* TestPool
* Store infra
* Store tests updates
* Bump maximum concurrent postgres connections
* More infra
* channels/jobs
* channels/app
* channels/api4
* Protect i18n from concurrent access
* Replace some use of os.Setenv
* Remove debug
* Lint fixes
* Fix more linting
* Fix test
* Remove use of Setenv in drafts tests
* Fix flaky TestWebHubCloseConnOnDBFail
* Fix merge
* [MM-62408] Add CI job to generate test coverage (#30284)
* Add CI job to generate test coverage
* Remove use of Setenv in drafts tests
* Fix flaky TestWebHubCloseConnOnDBFail
* Fix more Setenv usage
* Fix more potential flakyness
* Remove parallelism from flaky test
* Remove conflicting env var
* Fix
* Disable parallelism
* Test atomic covermode
* Disable parallelism
* Enable parallelism
* Add upload coverage step
* Fix codecov.yml
* Add codecov.yml
* Remove redundant workspace field
* Add Parallel() util methods and refactor
* Fix formatting
* More formatting fixes
* Fix reporting
* reject MySQL with the enterprise advanced license
If a user attempts to set an Enterprise Advanced License while
configured with MySQL, reject the license. This SKU is not compatible
with MYSQL.
* fix trial typo
* suppress trial banner if MySQL
* Update server/channels/app/platform/license_test.go
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* fix types
* suppress mysql from show start trial modal
* Skip MySQL-incompatible tests for access control and channel banner features
Skip the following tests when running with MySQL database:
- Access control policy tests (create, get, delete, check/test expressions, search, assign/unassign, get channels)
- Channel banner tests in TestPatchChannel and TestCanEditChannelBanner
These features are not supported on MySQL and the tests would fail.
Tests will continue to run normally on PostgreSQL.
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
* Skip TestSearchChannelsForAccessControlPolicy subtest for MySQL
Add MySQL skip logic to the "SearchChannelsForAccessControlPolicy with regular user"
subtest as this access control feature is not supported on MySQL.
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
* reject trial license requests for MySQL
* return false on sku + mysql match, even if logger is nil
* Fix MySQL trial license tests to skip appropriately based on database driver
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
---------
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Claude <noreply@anthropic.com>
* rm "No TEST_DATABASE... override" log message
Let's only log if this value is actually overridden.
* rm "(Created|Dropped) temporary database" message
* only log "Pinging SQL" on subsequent attempts
* disable morph logging from TestMain
* Fix style issues in store test files
- Add missing parameter to migrate() function calls in tests
- Remove unused log function in settings.go
- Fix formatting with go fmt
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
* within sqlstore, use "enable" vs "disable" for clarity
* remove trailing newline from morph logs
---------
Co-authored-by: Claude <noreply@anthropic.com>
A recent change to the enterprise cluster code introduced a change to the enterprise API interface. GetClusterInfos() can now return an error. This commit introduces code to handle that error.
We switch to using iterators introduced in Go 1.23
to make iteration alloc-free and fast. And since
element removal is allowed while iterating a map,
this also means we don't need to even copy the slice
any more.
While here, we also address the comment https://github.com/mattermost/mattermost/pull/30178#discussion_r1954862151.
I have simply gone back to using []string as the map
entry rather than a type alias or a redirection with
a struct.
https://mattermost.atlassian.net/browse/MM-63130
```release-note
NONE
```
* Changed back nil to len
```release-note
NONE
```
* fixing unused assignment
```release-note
NONE
```
* add benchmark
```
goos: linux
goarch: amd64
pkg: github.com/mattermost/mattermost/server/v8/channels/app/platform
cpu: Intel(R) Core(TM) i5-8265U CPU @ 1.60GHz
│ old.txt │ new.txt │
│ sec/op │ sec/op vs base │
HubConnIndexIterator/2_users-8 93.53n ± 1% 38.09n ± 1% -59.27% (p=0.000 n=10)
HubConnIndexIterator/3_users-8 106.30n ± 0% 38.41n ± 1% -63.86% (p=0.000 n=10)
HubConnIndexIterator/4_users-8 111.30n ± 1% 38.66n ± 1% -65.27% (p=0.000 n=10)
geomean 103.4n 38.39n -62.89%
│ old.txt │ new.txt │
│ B/op │ B/op vs base │
HubConnIndexIterator/2_users-8 16.00 ± 0% 24.00 ± 0% +50.00% (p=0.000 n=10)
HubConnIndexIterator/3_users-8 24.00 ± 0% 24.00 ± 0% ~ (p=1.000 n=10) ¹
HubConnIndexIterator/4_users-8 32.00 ± 0% 24.00 ± 0% -25.00% (p=0.000 n=10)
geomean 23.08 24.00 +4.00%
¹ all samples are equal
│ old.txt │ new.txt │
│ allocs/op │ allocs/op vs base │
HubConnIndexIterator/2_users-8 1.000 ± 0% 1.000 ± 0% ~ (p=1.000 n=10) ¹
HubConnIndexIterator/3_users-8 1.000 ± 0% 1.000 ± 0% ~ (p=1.000 n=10) ¹
HubConnIndexIterator/4_users-8 1.000 ± 0% 1.000 ± 0% ~ (p=1.000 n=10) ¹
geomean 1.000 1.000 +0.00%
¹ all samples are equal
```
```release-note
NONE
```
* ForChannel test as well
```release-note
NONE
```
* review comments
```release-note
NONE
```
* fix lint errors
```release-note
NONE
```
---------
Co-authored-by: Mattermost Build <build@mattermost.com>
* fix: hanlde error from InvalidateAllCaches in slack.go
* change signature of InvalidateAllCaches to *model.AppError
* return err from InvalidateAllCaches everywhere
* Formatting
---------
Co-authored-by: Ben Schumacher <ben.schumacher@mattermost.com>
* Adds direct participants to the channel invite
The channel invite now contains the sanitized users that are local to
the node that is sending the invite. In the event that the receiving
server doesn't have those users in its local database, it can create
them from the invite and correctly generate the DM or GM with them as
members.
* Use IsRemote instead of directly checking user attributes
---------
Co-authored-by: Miguel de la Cruz <miguel@ctrlz.es>
Co-authored-by: Mattermost Build <build@mattermost.com>
If the deadQueue was full, we would be incorrectly sending
the wrong index for dqPtr. Fixed that and added a test case.
https://mattermost.atlassian.net/browse/MM-63304
```release-note
NONE
```
* MM-61947 Run DND expiry job more often and round expiry time to match interval
* Move comment to make it godoc-compatible
* Change truncateDNDEndTime to work with seconds
---------
Co-authored-by: Mattermost Build <build@mattermost.com>
When Channel iteration mode is enabled, we need to ensure that
channel scoped events do not fall through to the connIndex.All()
condition. This is possible because there are multiple hubs
in a given system. So a single event will flow through all of them
and in some hubs, a channel scoped event might not have any connections.
In that case, we need to stop processing further.
https://mattermost.atlassian.net/browse/MM-62960
```release-note
NONE
```
When we added iteration by channelID, this was a known tradeoff during that.
However, it has been observed that the regular connection removal function
creates considerable blocking of the processing loop, leading to high
CPU usage and API latencies.
To fix this, we add a reverse mapping of channelIDs to connections
and their positions in the slice. This helps us to remove the connection
from the slice without iteration.
Unfortunately, this still needs to iterate through all channelIDs
during invalidation of the channel member cache. However, the user
cache invalidation is not a regular activity. So it should be an acceptable
tradeoff to make.
https://mattermost.atlassian.net/browse/MM-62960
```release-note
A new config knob ServiceSettings.EnableWebHubChannelIteration which allows a user to control the performance of websocket broadcasting. By default, this setting is turned off. If it is turned on, it improves the websocket broadcasting performance at the expense of poor performance when users join/leave a channel. It is not recommended to turn it on unless you have atleast 200,000 concurrent users actively using MM.
```
Co-authored-by: Mattermost Build <build@mattermost.com>
We were redirecting any logging happening via the Go standard logger
to our own mlog instance. The issue is that all those logs were happening
at LvlStdLog level which is higher than LvlDebug.
LvlStdLog is at 10, and LvlDebug is at 5. So unless a customer
sets the log level specifically to "stdlog" or sets up advanced
logging to specifically log for that level, no standard logs will
ever get logged.
We fix this by logging it at warn level. The reason for warn
is that mostly external libraries use this facility to log
out-of-band errors that could not be returned using standard
error returns.
I saw another plugin error which was logged similarly. This
was never surfaced before due to this bug.
https://mattermost.atlassian.net/browse/MM-62926
```release-note
NONE
```
We do a cluster request to get the active and dead queues
from other nodes in the cluster to sync any missing
information.
We check the dead queue in the other nodes to see
if there's been any message loss or not. Accordingly,
we send just the active queue or both active and dead queues.
There's still an edge case that is left out where
a client could have potentially connected and reconnected
to multiple nodes leaving multiple active queues
in multiple nodes. We don't handle this scenario
because then potentially we need to create
a slice of sendQueueSize * number_of_nodes. And then
this can happen again, leading to an infinite increase
in sendQueueSize.
We leave this edge-case to Redis, acknowledging
a limitation in our architecture.
In this PR, when there's no message loss, we just
take the active queue from the last node it connected
to.
And if there's message loss where the client's
seqNum is within the last node's dead queue, we also
handle that.
But if there's severe message loss where the client's
seqNum falls within the dead queue of another node, then
we just send the data from that node to reconstruct the
data as much as possible. It could be possible to set
a new connection ID in this case, but this involves
more data transfer always from all nodes and recomputing
the state in the requestor node.
https://mattermost.atlassian.net/browse/MM-61904
```release-note
NONE
```
Co-authored-by: Mattermost Build <build@mattermost.com>
* MM-62378: Initialize status and session cache before loading license
ps.LoadLicense could end up calling InvalidateAllCaches. Therefore,
we need to intialize those caches before calling LoadLicense
to prevent a panic from happening.
While here, we also remove some unused code.
https://mattermost.atlassian.net/browse/MM-62378
```release-note
NONE
```
* fix lint errors
```release-note
NONE
```
* fix test
```release-note
NONE
```
