* ci: shard server Postgres tests into 4 parallel runners (#35739)
* ci: add test sharding plumbing to server CI
Add infrastructure for upcoming test sharding without changing behavior:
- Add shard-index and shard-total inputs to server-test-template.yml
(defaults preserve existing single-runner behavior)
- Add timing cache restore step (activates only when shard-total > 1)
- Add merge-postgres-test-results job to server-ci.yml that:
- Merges JUnit XML reports from shard artifacts
- Saves timing data cache for future shard balancing
- Handles both single-artifact and multi-shard scenarios
- Add .gitignore entries for timing cache and shard work files
Co-authored-by: Claude <claude@anthropic.com>
* ci: shard server Postgres tests into 4 parallel runners
Extract sharding logic into standalone, tested scripts and enable
4-shard parallel test execution for server Postgres CI:
Scripts:
- server/scripts/shard-split.js: Node.js bin-packing solver that
assigns test packages to shards using timing data from previous runs.
Two-tier strategy: light packages (<2min) whole, heavy packages
(api4, app) split at individual test level.
- server/scripts/run-shard-tests.sh: Multi-run wrapper that calls
gotestsum directly for each package group with -run regex filters.
- server/scripts/shard-split.test.js: 8 test cases covering round-robin
fallback, timing-based balancing, heavy package splitting, JUnit XML
fallback, and enterprise package separation.
Workflow changes:
- server-test-template.yml: Add shard splitting step that discovers test
packages and runs the solver. Modified Run Tests step to use wrapper
script when sharding is active.
- server-ci.yml: Add 4-shard matrix to test-postgres-normal. Update
merge job artifact patterns for shard-specific names.
Performance: 7.2 min with timing cache vs 62.5 min baseline = 88%
wall-time improvement. First run without cache uses JUnit XML fallback
or round-robin, then populates the cache for subsequent runs.
Co-authored-by: Claude <claude@anthropic.com>
* fix: raise heavy package threshold to 5 min to preserve test isolation
sqlstore integrity tests scan the entire database and fail when other
packages' test data is present. At 182s, sqlstore was just over the
120s threshold and getting split at test level. Raising to 300s keeps
only api4 (~38 min) and app (~15 min) as heavy — where the real
sharding gains are — while sqlstore, elasticsearch, etc. stay whole
and maintain their test isolation guarantees.
Co-authored-by: Claude <claude@anthropic.com>
* ci: only save test timing cache on default branch
PR branches always restore from master's timing cache via restore-keys
prefix matching. Timing data is stable day-to-day so this eliminates
cache misses on first PR runs and reduces cache storage.
Co-authored-by: Claude <claude@anthropic.com>
* ci: skip FIPS tests on PRs (enterprise CI handles compile check)
Per review feedback: the enterprise CI already runs a FIPS compile
check on every PR. Running the full FIPS test suite on PRs is redundant
since it uses the identical test suite as non-FIPS — the only
FIPS-specific failure mode is a build failure from non-approved crypto
imports, which the enterprise compile check catches.
Full FIPS tests continue to run on every push to master.
Co-authored-by: Claude <claude@anthropic.com>
* fix: address review feedback on run-shard-tests.sh
- Remove set -e so all test runs execute even if earlier ones fail;
track failures and exit with error at the end (wiggin77)
- Remove unused top-level COVERAGE_FLAG variable (wiggin77)
- Fix RUN_IDX increment position so report, json, and coverage files
share the same index (wiggin77)
- Update workflow comment: heavy threshold is 5 min, not 2 min (wiggin77)
Co-authored-by: Claude <claude@anthropic.com>
* style: use node: prefix for built-in fs module in shard-split.js
Co-authored-by: Claude <claude@anthropic.com>
* fix: avoid interpolating file paths into generated shell script
Read shard package lists from files at runtime instead of interpolating
them into the generated script via printf. This prevents theoretical
shell metacharacter injection from directory names, as flagged by
DryRun Security.
Co-authored-by: Claude <claude@anthropic.com>
* fix(ci): rename merged artifact to match server-ci-report glob
The merged artifact was named postgres-server-test-logs-merged which
does not match the *-test-logs pattern in server-ci-report.yml,
causing Postgres test results to be missing from PR/commit reports.
Also pins junit-report-merger to exact version 7.0.0 for supply chain
safety.
Co-authored-by: Claude <claude@anthropic.com>
* fix(ci): pass RACE_MODE env into Docker container
RACE_MODE was set on the host runner but never included in the docker
run --env list. The light-package path worked because the heredoc
expanded on the host, but run-shard-tests.sh reads RACE_MODE at
runtime inside the container where it was unset. This caused heavy
packages (api4, app) to silently lose -race detection.
Co-authored-by: Claude <claude@anthropic.com>
* fix(ci): discover new tests in heavy packages not in timing cache
Tests not present in the timing cache (newly added or renamed) would
not appear in any shard -run regex, causing them to silently skip.
After building items from the cache, run go test -list to discover
current test names and assign any cache-missing tests to shards via
the normal bin-packing algorithm with a small default duration.
Co-authored-by: Claude <claude@anthropic.com>
* fix(ci): add missing line continuation backslash in docker run
The previous --env FIPS_ENABLED line was missing a trailing backslash
after adding --env RACE_MODE, causing docker run to see a truncated
command and fail with "requires at least 1 argument".
Co-authored-by: Claude <claude@anthropic.com>
* fix(ci): add setup-go step for shard test discovery
go test -list in shard-split.js runs on the host runner via execSync,
but Go is only available inside the Docker container. Without this
step, every invocation fails silently and new-test discovery is a
no-op. Adding actions/setup-go before the shard split step ensures
the Go toolchain is available on the host.
Co-authored-by: Claude <claude@anthropic.com>
---------
Co-authored-by: Claude <claude@anthropic.com>
* updated go to version 1.25.8 (#35817)
* updated go to version 1.25.8
* updated gotestsum version to work with go 1.25.8
go 1.25 does not work with indirect tools 0.11 dependency pulled by
gotestsum.
* Use sync.WaitGroup.Go to simplify goroutine creation
Replace the wg.Add(1) + go func() { defer wg.Done() }() pattern with
wg.Go(), which was introduced in Go 1.25.
* pushes fips image on workflow dispatch to allow fips test to run on go version update
* fix new requirements for FIPS compliance imposed on updating to go 1.25.8
* updates openssl symbol check for library shipped with FIPS new versions
go-openssl v2 shipped with FIPS versions starting from 1.25 uses mkcgo to generate
bindings causing symbol names to be different.
* removes temp workflow-dispatch condition
* keep versions out of agents md file
* upgrade golangci-lint (#35845)
* test: clean up channel store data after TestChannelStore (#36066)
TestChannelStore sub-tests create channels, members, and team members
using fake TeamIds and UserIds (model.NewId() for non-existent rows).
These records are left in the database and cause integrity tests
(TestCheck*) running in the same binary to fail their full-table scans.
Register a t.Cleanup on TestChannelStore that purges the affected
tables entirely. A blanket purge is safe: the schema enforces no FK
constraints, and every test suite creates its own data independently.
* Fix command injection in server-test-template workflow (#36080)
Replace the unquoted heredoc (which embedded GITHUB_HEAD_REF into a
generated script) with a cp of the existing run-shard-tests.sh, which
already handles the light-only case. Pass BUILD_NUMBER and TEST_TARGET
as explicit docker env vars instead of interpolating them into script
content.
* fix(ci): restore testname format in sharded gotestsum runs (#36078)
run-shard-tests.sh called gotestsum directly without --format, so it
fell back to gotestsum's default (pkgname) instead of the testname
format set by the Makefile. Pass --format "${GOTESTSUM_FORMAT:-testname}"
to match the Makefile default.
Co-authored-by: Mattermost Build <build@mattermost.com>
* fix(lint): fix pre-existing golangci-lint v2.11.4 issues
Fix misspelling in comment and redundant nil check flagged by the
upgraded linter.
* ci: use golang image for test runner on release-10.11
mattermost-build-server images are not built for release branches.
Use the official golang image which is always available for any Go version.
* ci: use mattermost/mattermost-build-server for release-10.11
The mattermostdevelopment/ images are only built for master.
The production mattermost/ images are built for release branches.
* ci: use mattermost/mattermost-build-server in mmctl test template
The mattermostdevelopment/ images are only built for master.
The production mattermost/ images are built for release branches.
---------
Co-authored-by: Pavel Zeman <pavel.zeman@mattermost.com>
Co-authored-by: Claude <claude@anthropic.com>
Co-authored-by: Carlos Garcia <carlos.garcia@mattermost.com>
Co-authored-by: Mattermost Build <build@mattermost.com>
This commit exposes audit logging functionality to plugins via the plugin API, allowing plugins to create and log audit records. Additionally, it addresses a gob encoding issue that could cause plugin crashes when audit data contains nil pointers or unregistered types.
On page load, we load ALL channels and channel members from all teams.
But then, on team_switch, we would again load channels and channel
members from that team. This was redundant and mainly kept
because previously the websocket events were considered unreliable.
Now with reliable websockets, and client-side pings, we can detect
broken connections faster and recover without loss.
Additionally, the getAllChannelMembers call would page through
all responses on the client side. This was inefficient and incur
extra latency. To optimize for this, we introduce server-side
streaming of the full response if page is set to -1.
This optimizes the intial response as well.
https://mattermost.atlassian.net/browse/MM-56906
```release-note
Optimize team switch operation by removing calls to get channels
and channel members.
```
Co-authored-by: Mattermost Build <build@mattermost.com>
* Adds the main Property System Architecture components
This change adds the necessary migrations for the Property Groups,
Fields and Values tables to be created, the store layer and a Property
Service that can be used from the app layer.
* Adds Custom Profile Attributes endpoints and app layer
* implement get and patch cpa values
* run i18n-extract
* Update property field type to use user instead of person
* Update PropertyFields to allow for unique nondeleted fields and remove redundant indexes
* Update PropertyValues to allow for unique nondeleted fields and remove redundant indexes
* Use StringMap instead of the map[string]any on property fields
* Add i18n strings
* Revert "Use StringMap instead of the map[string]any on property fields"
This reverts commit e2735ab0f8589d2524d636419ca0cb144575c4d6.
* Cast JSON binary data to string and add todo note for StringMap use
* Add mocks to the retrylayer tests
* Cast JSON binary data to string in property value store
* Check for binary parameter instead of casting to string for JSON data
* Fix bad merge
* Check property field type is one of the allowed ones
* Avoid reusing err variable to be explicit about the returned value
* Merge Property System Migrations into one file
* Adds NOT NULL to timestamps at the DB level
* Update stores to use tableSelectQuery instead of a slice var
* Update PropertyField model translations to be more explicit and avoid repetition
* Update PropertyValue model translations to be more explicit and avoid repetition
* Use ExecBuilder instead of ToSql&Exec
* Update property field errors to add context
* Ensure PerPage is greater than zero
* Update store errors to give more context
* Use ExecBuilder in the property stores where possible
* Add an on conflict suffix to the group register to avoid race conditions
* Remove user profile API documentation changes
* Update patchCPAValues endpoint and docs to return the updated information
* Merge two similar error conditions
* Use a route function for ListCPAValues
* Remove badly used translation string
* Remove unused get in register group method
* Adds input sanitization and validation to the CPA API endpoints
* Takes login outside of one test case to make it clear it affects multiple t.Runs
* Fix wrap error and return code when property field has been deleted
* Fix receiver name
* Adds comment to move the CPA group ID to the db cache
* Set the PerPage of CPA fields to the fields limit
* Update server/channels/app/custom_profile_attributes_test.go
Co-authored-by: Alejandro García Montoro <alejandro.garciamontoro@gmail.com>
* Standardize group ID access
* Avoid polluting the state between tests
* Use specific errors for the retrieval of CPA group
---------
Co-authored-by: Scott Bishel <scott.bishel@mattermost.com>
Co-authored-by: Mattermost Build <build@mattermost.com>
Co-authored-by: Alejandro García Montoro <alejandro.garciamontoro@gmail.com>
* add webhook id to logs
* Move hook_id context to api level from app level.
* pass errCtx to other references of same app-error-id
---------
Co-authored-by: Mattermost Build <build@mattermost.com>
Essentially applying the fix for https://github.com/mattermost/mattermost/pull/29214
to the web package. It seems like the app.StoreOverrideWithCache
function was never used in this package.
Just a small highlight of how much boilerplate is repeated
in multiple packages
```release-note
NONE
```
Co-authored-by: Mattermost Build <build@mattermost.com>
This originated from https://github.com/mattermost/mattermost/issues/15249.
However, the original idea was discarded https://github.com/mattermost/mattermost/issues/15249#issuecomment-709713065
as being too complicated to implement. Then I had another
idea to implement it just for session objects.
My thinking was that since every single request allocates a new
session struct, it would be good to use a sync.Pool for that.
However, 4 years later, now we know that the primary bottleneck
in app performance comes from websocket event marshalling.
Therefore, while it would be good to do this, it is difficult
to do it correctly (as shown by the numerous racy tests).
Hence, reverting this.
```release-note
NONE
```
* Adds Shared Channel management API endpoints
New endpoints for the following routes are added:
- Get Shared Channel Remotes by Remote Cluster at `GET
/api/v4/remotecluster/{remote_id}/sharedchannelremotes`
- Invite Remote Cluster to Channel at `POST
/api/v4/remotecluster/{remote_id}/channels/invite`
- Uninvite Remote Cluster to Channel at `POST
/api/v4/remotecluster/{remote_id}/channels/uninvite`
These endpoints are planned to be used from the system console, and
gated through the `manage_secure_connections` permission.
* Adds i18n messages for API errors
* Fix pagination flaky test
* Fix linter
* Adds the posibility of filtering shared channel remotes by home
---------
Co-authored-by: Mattermost Build <build@mattermost.com>
* removed an incorretc return that caused missing API timing metrics for 4xx and 5xx errors
* Added tests and recorded metric for URL length limit as well
* Status code fix
* on login add UserId and SessionId to audit's Actor field to match logout
* lint
* simplify to add only userId and sessionId
* AddToEventActor -> AddUser/SessionToEventActor
* fill in missing session data when logging the audit record
* why did it bump that? reverting.
* make modules-tidy
* trigger build
* add more context to the comment
* Adds Remote Cluster related API endpoints
New endpoints for the following routes are added:
- Get Remote Clusters at `GET /api/v4/remotecluster`
- Create Remote Cluster at `POST /api/v4/remotecluster`
- Accept Remote Cluster invite at `POST
/api/v4/remotecluster/accept_invite`
- Generate Remote Cluster invite at `POST
/api/v4/remotecluster/{remote_id}/generate_invite`
- Get Remote Cluster at `GET /api/v4/remotecluster/{remote_id}`
- Patch Remote Cluster at `PATCH /api/v4/remotecluster/{remote_id}`
- Delete Remote Cluster at `DELETE /api/v4/remotecluster/{remote_id}`
These endpoints are planned to be used from the system console, and
gated through the `manage_secure_connections` permission.
* Update server/channels/api4/remote_cluster_test.go
Co-authored-by: Doug Lauder <wiggin77@warpmail.net>
* Fix AppError names
---------
Co-authored-by: Doug Lauder <wiggin77@warpmail.net>
Co-authored-by: Mattermost Build <build@mattermost.com>
* Deprecate Self Serve: First Pass
* Fix ci
* Fix more ci
* Remmove outdated server tests
* Fix a missed spot opening purchase modal in Self Hosted
* Fix i18n
* Clean up some more server code, fix webapp test
* Fix alignment of button
* Fix linter
* Fix i18n server side
* Deprecate in product true up
* Add back translation
* Remove client functions
* Put back client functions
* webapp deprecation
* Deprecate Self Serve: Second Pass
* Fix various pipeline issues
* Fix linter
* Fix pipelines
* Fix handlers_test.go
* Fix console.error around hostedCustomer in reducer
* PICKY LINTER PLEASE
* Fix webapp tests, various other fixes for the CI pipelines
* Fix i18n
* Updates to accomadate enterprise code removal
* Fix mocks
* More removal
* Fix
* Adjustments from PR
* Fixes for QA Feedback
* Update
* Add migrations to remove true up review history
* Fix migrations check
---------
Co-authored-by: Mattermost Build <build@mattermost.com>
Co-authored-by: maria.nunez <maria.nunez@mattermost.com>
