* Support Elasticsearch v9 alongside v8
* Add CI workflow changes for Elasticsearch v8/v9 testing
* Support Elasticsearch v7 in addition to v8/v9, add v7 CI test job
Lowers the minimum supported ES version from 8 to 7 to avoid dropping
v7 support in a dot release. Adds a dedicated CI job to verify v7
compatibility alongside the existing v8 and v9 (default) jobs.
* Fix ES7 plugin install crash on cgroup v2 hosts
* Fix ES 7 container startup on cgroup v2 Linux (GitHub Actions)
ES 7 bundles JDK 11, which crashes with a NullPointerException in
CgroupV2Subsystem.getMountPoint() on modern Linux kernels that use
cgroup v2 (including GitHub Actions ubuntu-latest runners). The flag
was already set during the Dockerfile RUN step, but not at runtime.
Adding -XX:-UseContainerSupport to ES_JAVA_OPTS in docker-compose
fixes the crash. The flag is harmless on ES 8/9 which ship JDK 17+
where the cgroup v2 bug is fixed (it simply opts out of container-aware
JVM sizing).
* Capture docker compose logs in CI test artifact
* Use ES 7.17.29 for v7 CI test; remove cgroup v2 workarounds
ES 7.17.0 bundled JDK 17.0.1 which had a cgroup v2 bug
(CgroupV2Subsystem NPE) not fixable via -XX:-UseContainerSupport.
ES 7.17.29 bundles JDK 22 where the bug is long fixed.
Reverts the -XX:-UseContainerSupport workarounds added in the
previous two commits as they were based on a wrong diagnosis
and are no longer needed.
---------
Co-authored-by: Mattermost Build <build@mattermost.com>
* ci: shard server Postgres tests into 4 parallel runners (#35739)
* ci: add test sharding plumbing to server CI
Add infrastructure for upcoming test sharding without changing behavior:
- Add shard-index and shard-total inputs to server-test-template.yml
(defaults preserve existing single-runner behavior)
- Add timing cache restore step (activates only when shard-total > 1)
- Add merge-postgres-test-results job to server-ci.yml that:
- Merges JUnit XML reports from shard artifacts
- Saves timing data cache for future shard balancing
- Handles both single-artifact and multi-shard scenarios
- Add .gitignore entries for timing cache and shard work files
Co-authored-by: Claude <claude@anthropic.com>
* ci: shard server Postgres tests into 4 parallel runners
Extract sharding logic into standalone, tested scripts and enable
4-shard parallel test execution for server Postgres CI:
Scripts:
- server/scripts/shard-split.js: Node.js bin-packing solver that
assigns test packages to shards using timing data from previous runs.
Two-tier strategy: light packages (<2min) whole, heavy packages
(api4, app) split at individual test level.
- server/scripts/run-shard-tests.sh: Multi-run wrapper that calls
gotestsum directly for each package group with -run regex filters.
- server/scripts/shard-split.test.js: 8 test cases covering round-robin
fallback, timing-based balancing, heavy package splitting, JUnit XML
fallback, and enterprise package separation.
Workflow changes:
- server-test-template.yml: Add shard splitting step that discovers test
packages and runs the solver. Modified Run Tests step to use wrapper
script when sharding is active.
- server-ci.yml: Add 4-shard matrix to test-postgres-normal. Update
merge job artifact patterns for shard-specific names.
Performance: 7.2 min with timing cache vs 62.5 min baseline = 88%
wall-time improvement. First run without cache uses JUnit XML fallback
or round-robin, then populates the cache for subsequent runs.
Co-authored-by: Claude <claude@anthropic.com>
* fix: raise heavy package threshold to 5 min to preserve test isolation
sqlstore integrity tests scan the entire database and fail when other
packages' test data is present. At 182s, sqlstore was just over the
120s threshold and getting split at test level. Raising to 300s keeps
only api4 (~38 min) and app (~15 min) as heavy — where the real
sharding gains are — while sqlstore, elasticsearch, etc. stay whole
and maintain their test isolation guarantees.
Co-authored-by: Claude <claude@anthropic.com>
* ci: only save test timing cache on default branch
PR branches always restore from master's timing cache via restore-keys
prefix matching. Timing data is stable day-to-day so this eliminates
cache misses on first PR runs and reduces cache storage.
Co-authored-by: Claude <claude@anthropic.com>
* ci: skip FIPS tests on PRs (enterprise CI handles compile check)
Per review feedback: the enterprise CI already runs a FIPS compile
check on every PR. Running the full FIPS test suite on PRs is redundant
since it uses the identical test suite as non-FIPS — the only
FIPS-specific failure mode is a build failure from non-approved crypto
imports, which the enterprise compile check catches.
Full FIPS tests continue to run on every push to master.
Co-authored-by: Claude <claude@anthropic.com>
* fix: address review feedback on run-shard-tests.sh
- Remove set -e so all test runs execute even if earlier ones fail;
track failures and exit with error at the end (wiggin77)
- Remove unused top-level COVERAGE_FLAG variable (wiggin77)
- Fix RUN_IDX increment position so report, json, and coverage files
share the same index (wiggin77)
- Update workflow comment: heavy threshold is 5 min, not 2 min (wiggin77)
Co-authored-by: Claude <claude@anthropic.com>
* style: use node: prefix for built-in fs module in shard-split.js
Co-authored-by: Claude <claude@anthropic.com>
* fix: avoid interpolating file paths into generated shell script
Read shard package lists from files at runtime instead of interpolating
them into the generated script via printf. This prevents theoretical
shell metacharacter injection from directory names, as flagged by
DryRun Security.
Co-authored-by: Claude <claude@anthropic.com>
* fix(ci): rename merged artifact to match server-ci-report glob
The merged artifact was named postgres-server-test-logs-merged which
does not match the *-test-logs pattern in server-ci-report.yml,
causing Postgres test results to be missing from PR/commit reports.
Also pins junit-report-merger to exact version 7.0.0 for supply chain
safety.
Co-authored-by: Claude <claude@anthropic.com>
* fix(ci): pass RACE_MODE env into Docker container
RACE_MODE was set on the host runner but never included in the docker
run --env list. The light-package path worked because the heredoc
expanded on the host, but run-shard-tests.sh reads RACE_MODE at
runtime inside the container where it was unset. This caused heavy
packages (api4, app) to silently lose -race detection.
Co-authored-by: Claude <claude@anthropic.com>
* fix(ci): discover new tests in heavy packages not in timing cache
Tests not present in the timing cache (newly added or renamed) would
not appear in any shard -run regex, causing them to silently skip.
After building items from the cache, run go test -list to discover
current test names and assign any cache-missing tests to shards via
the normal bin-packing algorithm with a small default duration.
Co-authored-by: Claude <claude@anthropic.com>
* fix(ci): add missing line continuation backslash in docker run
The previous --env FIPS_ENABLED line was missing a trailing backslash
after adding --env RACE_MODE, causing docker run to see a truncated
command and fail with "requires at least 1 argument".
Co-authored-by: Claude <claude@anthropic.com>
* fix(ci): add setup-go step for shard test discovery
go test -list in shard-split.js runs on the host runner via execSync,
but Go is only available inside the Docker container. Without this
step, every invocation fails silently and new-test discovery is a
no-op. Adding actions/setup-go before the shard split step ensures
the Go toolchain is available on the host.
Co-authored-by: Claude <claude@anthropic.com>
---------
Co-authored-by: Claude <claude@anthropic.com>
* updated go to version 1.25.8 (#35817)
* updated go to version 1.25.8
* updated gotestsum version to work with go 1.25.8
go 1.25 does not work with indirect tools 0.11 dependency pulled by
gotestsum.
* Use sync.WaitGroup.Go to simplify goroutine creation
Replace the wg.Add(1) + go func() { defer wg.Done() }() pattern with
wg.Go(), which was introduced in Go 1.25.
* pushes fips image on workflow dispatch to allow fips test to run on go version update
* fix new requirements for FIPS compliance imposed on updating to go 1.25.8
* updates openssl symbol check for library shipped with FIPS new versions
go-openssl v2 shipped with FIPS versions starting from 1.25 uses mkcgo to generate
bindings causing symbol names to be different.
* removes temp workflow-dispatch condition
* keep versions out of agents md file
* upgrade golangci-lint (#35845)
* test: clean up channel store data after TestChannelStore (#36066)
TestChannelStore sub-tests create channels, members, and team members
using fake TeamIds and UserIds (model.NewId() for non-existent rows).
These records are left in the database and cause integrity tests
(TestCheck*) running in the same binary to fail their full-table scans.
Register a t.Cleanup on TestChannelStore that purges the affected
tables entirely. A blanket purge is safe: the schema enforces no FK
constraints, and every test suite creates its own data independently.
* Fix command injection in server-test-template workflow (#36080)
Replace the unquoted heredoc (which embedded GITHUB_HEAD_REF into a
generated script) with a cp of the existing run-shard-tests.sh, which
already handles the light-only case. Pass BUILD_NUMBER and TEST_TARGET
as explicit docker env vars instead of interpolating them into script
content.
* fix(ci): restore testname format in sharded gotestsum runs (#36078)
run-shard-tests.sh called gotestsum directly without --format, so it
fell back to gotestsum's default (pkgname) instead of the testname
format set by the Makefile. Pass --format "${GOTESTSUM_FORMAT:-testname}"
to match the Makefile default.
Co-authored-by: Mattermost Build <build@mattermost.com>
* fix(lint): fix pre-existing golangci-lint v2.11.4 issues
Fix misspelling in comment and redundant nil check flagged by the
upgraded linter.
* ci: use golang image for test runner on release-10.11
mattermost-build-server images are not built for release branches.
Use the official golang image which is always available for any Go version.
* ci: use mattermost/mattermost-build-server for release-10.11
The mattermostdevelopment/ images are only built for master.
The production mattermost/ images are built for release branches.
* ci: use mattermost/mattermost-build-server in mmctl test template
The mattermostdevelopment/ images are only built for master.
The production mattermost/ images are built for release branches.
---------
Co-authored-by: Pavel Zeman <pavel.zeman@mattermost.com>
Co-authored-by: Claude <claude@anthropic.com>
Co-authored-by: Carlos Garcia <carlos.garcia@mattermost.com>
Co-authored-by: Mattermost Build <build@mattermost.com>
* Automate setup-go-work as a dependency for Make targets (#35476)
* automate setup-go-work
It's all to easy to forget to `make setup-go-work`, only to run into mysterious build failures. Let's default to doing this automatically, unless `SKIP_SETUP_GO_WORK` is true (or the legacy `IGNORE_GO_WORK_IF_EXISTS`, which was oddly named, since we can't actually ignore it.)
* Make setup-go-work recipe fail-fast with set -e
* ci: post success to required e2e status contexts when no relevant changes (#35880)
* ci: post correct skip status from within cypress/playwright reusable workflows
The 'Required Status Checks' ruleset requires e2e-test/cypress-full/enterprise
and e2e-test/playwright-full/enterprise on master and release-*.* branches.
When a PR has no E2E-relevant changes, the jobs were silently skipped, leaving
required statuses unset and the PR permanently blocked.
Architecture fix: instead of a separate skip-e2e job in the caller that
hardcodes status context names, the skip logic now lives inside the reusable
workflows that already own and compute those context names.
Changes:
- e2e-tests-cypress.yml: add should_run input (default 'true') + skip job
that uses the dynamically-computed context_name when should_run == 'false'
- e2e-tests-playwright.yml: same pattern
- e2e-tests-ci.yml: change e2e-cypress/e2e-playwright job conditions from
should_run == 'true' to PR_NUMBER != '' (always run when there's a PR),
pass should_run as input to both reusable workflows
* Add E2E template workflows for Cypress and Playwright
* Add check-e2e-test-only action for E2E workflow
* Fix: Remove circular E2E workflow file check - skip tests when only CI files change
* Add pull_request trigger to E2E workflow - run automatically on PR events
* Fix resolve-pr to use github.event context for automatic pull_request trigger
* Fix checkout condition to work with pull_request events
* Fix: Remove orphaned fi statement in check-changes script
---------
Co-authored-by: yasser khan <attitude3cena.yf@gmail.com>
With glibc 2.34 and the [removal of libpthread](https://developers.redhat.com/articles/2021/12/17/why-glibc-234-removed-libpthread), binaries built using [Debian bookworm](https://www.debian.org/releases/bookworm/) aren't compatible with older but still supported operating systems like RHEL8. In those environments, Mattermost fails to start with errors like:
```
mattermost/bin/mattermost: /lib64/libc.so.6: version `GLIBC_2.32' not found (required by mattermost/bin/mattermost)
mattermost/bin/mattermost: /lib64/libc.so.6: version `GLIBC_2.34' not found (required by mattermost/bin/mattermost)
```
One option might be to generate a static build and avoid the glibc dependency, but this kind of change is out of scope for now. Let's just revert back to using [Debian bullseye](https://www.debian.org/releases/bullseye/), which remains supported until at least August 2026.
(cherry picked from commit c2120b7224ce8d2f8dd34f17fbeba06296b733db)
Co-authored-by: Jesse Hallam <jesse.hallam@gmail.com>
* Upgrade Go to 1.24.3
Updates the following files:
- server/.go-version: 1.23.9 → 1.24.3
- server/build/Dockerfile.buildenv: golang:1.23.9-bullseye → golang:1.24.3-bullseye
- server/go.mod: go 1.23.0 → go 1.24.3, toolchain go1.23.9 → go1.24.3
- server/public/go.mod: go 1.23.0 → go 1.24.3, toolchain go1.23.9 → go1.24.3
Also fixes non-constant format string errors introduced by Go 1.24.3's stricter format string checking:
- Added response() helper function in slashcommands/util.go for simple string responses
- Removed unused responsef() function from slashcommands/util.go
- Replaced responsef() with response() for translated strings that don't need formatting
- Fixed fmt.Errorf and fmt.Fprintf calls to use proper format verbs instead of string concatenation
- Updated marketplace buildURL to handle format strings conditionally
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
* Update generated mocks for Go 1.24.3
Regenerated mocks using mockery v2.53.4 to ensure compatibility with Go 1.24.3.
This addresses mock generation failures that occurred with the Go upgrade.
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
* Update to bookworm and fix non-existent sha
Signed-off-by: Stavros Foteinopoulos <stafot@gmail.com>
* fix non-constant format string
---------
Signed-off-by: Stavros Foteinopoulos <stafot@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Stavros Foteinopoulos <stafot@gmail.com>
* Move from ubuntu to distroless image
* Update updated ubunty image for first stage
* Update server/build/Dockerfile
Co-authored-by: Daniel Schalla <daniel@mattermost.com>
* Update server/build/Dockerfile
Co-authored-by: Daniel Schalla <daniel@mattermost.com>
* Add mising env var for health check
* Add mattermost user in /etc/passwd
* Fix e2e userId
* use cypress in place of server. Wget and not curl
* Simplify e2e changes
---------
Co-authored-by: Daniel Schalla <daniel@mattermost.com>
* TestPool
* Store infra
* Store tests updates
* Bump maximum concurrent postgres connections
* More infra
* channels/jobs
* channels/app
* channels/api4
* Protect i18n from concurrent access
* Replace some use of os.Setenv
* Remove debug
* Lint fixes
* Fix more linting
* Fix test
* Remove use of Setenv in drafts tests
* Fix flaky TestWebHubCloseConnOnDBFail
* Fix merge
* [MM-62408] Add CI job to generate test coverage (#30284)
* Add CI job to generate test coverage
* Remove use of Setenv in drafts tests
* Fix flaky TestWebHubCloseConnOnDBFail
* Fix more Setenv usage
* Fix more potential flakyness
* Remove parallelism from flaky test
* Remove conflicting env var
* Fix
* Disable parallelism
* Test atomic covermode
* Disable parallelism
* Enable parallelism
* Add upload coverage step
* Fix codecov.yml
* Add codecov.yml
* Remove redundant workspace field
* Add Parallel() util methods and refactor
* Fix formatting
* More formatting fixes
* Fix reporting
As part of https://github.com/mattermost/mattermost/pull/29932, we
stopped packaging Windows for releases. Let's go one step further and
stop building it too (saves build time!). And while we're in here, stop
doing this for OSX as well.
Both these targets remain buildable on demand, but we don't support
these platforms for production deployments.
* Update Go version to v1.23.7
* Bump golangci-lint to a version supporting Go 1.23
* Fix golangci-lint warnings
Several rules from gosimple, revive and staticcheck linters were
failing:
- Redefinition of built-in identifiers (max, min, new, recover...)
- Use of printf-like functions with simple strings
- Check for nil slices, when len already takes it into account
* Trigger Build
* Trigger Build
---------
Co-authored-by: Mattermost Build <build@mattermost.com>
* Bump Go version to 1.23.6
* Update CodeQL Github action as well
* Use server's Go version for CodeQL action
Co-authored-by: Antonis Stamatiou <stamatiou.antonis@gmail.com>
* Empty commit to trigger CI
* Bump golangci-lint to a version supporting Go 1.23
* Fix golangci-lint warnings
Several rules from gosimple, revive and staticcheck linters were
failing:
- Redefinition of built-in identifiers (max, min, new, recover...)
- Use of printf-like functions with simple strings
- Check for nil slices, when len already takes it into account
---------
Co-authored-by: Antonis Stamatiou <stamatiou.antonis@gmail.com>
Co-authored-by: Mattermost Build <build@mattermost.com>
* Remove uneeded hack in docker-compose now that problem is solved
* Remove obsolete attribute `version` in docker compose configuration files
---------
Co-authored-by: Mattermost Build <build@mattermost.com>
This PR allows to run a different Mattermost instance in port :8066
using the `make run-node` target. This instance reuses the frontend
files compiled for the main development instance, has its own database
and local mode enabled through environment variables, and for the rest
of the configuration simply takes whatever properties are set in the
`config.json` file.
This target is specially useful to run a Shared Channels development
environment, compiling first the frontend and then running `make
run-server` and `make run-node` side by side, later connecting both
and testing Shared Channels features.
- Update library version.
- Added MaxFlush delay to help reduce CPU usage.
- Fall back to LRU cache for the caches which use SCAN.
- Added mattermost-redis and running for all api layer
tests in Postgres.
https://mattermost.atlassian.net/browse/MM-59934
```release-note
NONE
```
* Utilize free runners for E2E tests, report back to PR on run completion
* Increase test stability on smaller instances
* Merge worker reports
* merge start+prepare steps
* upgrade keycloak
* increase test timeouts
* Add video recording to report artifacts
Co-authored-by: Saturnino Abril <5334504+saturninoabril@users.noreply.github.com>
---------
Co-authored-by: Saturnino Abril <5334504+saturninoabril@users.noreply.github.com>
* update keycloak docker image
* update realm file with a compatible realm
* import realm on start-docker command
Since bitnami's image does not support importing directly, the import of the test realm is done in the make file start-docker action
* Use official image from quay
* updated realm keycloak config
* final note about nickname attrib for saml
* add admin user
* update realm
* Updated from master
* Updated docs
* local typo
* use jq for ldap and saml
* updated readme
* configure Loki for local development
Extend our docker compose setup to include Loki (along with promtail), scraping `logs/*.log` to ingest development logs as well as all logs from running containers.
While we're in here, teach Prometheus to scrape metrics from Docker containers too.
* tweak promtail labels
* document extra services in config.mk
* update home.json
* Expose metrics under a source available license
* do not assume Cluster()
* allow metrics if licensed or dev
* temporary vet override
* simplify BULID_TAGS handling
* auto clean old imports.go
* fix license listener
* e2e test metrics & license semantics
* update from enterprise
* switch back to mattermost-govet/v2@new now
* update metrics from upstream
* Update license_spec.js
Co-authored-by: Saturnino Abril <saturnino.abril@gmail.com>
* Update license_spec.js
Co-authored-by: Saturnino Abril <saturnino.abril@gmail.com>
* Update e2e-tests/cypress/tests/integration/channels/enterprise/metrics/license_spec.js
Co-authored-by: Saturnino Abril <saturnino.abril@gmail.com>
* Update e2e-tests/cypress/tests/integration/channels/enterprise/metrics/license_spec.js
* split up specs
* require/delete license earlier in e2e test
* expanded expect to debug failures
* more logging
* Revert "more logging"
This reverts commit 0bc513fd926c07a0c6050f4b4c742e856d8dc435.
* e2e: try deleting license first
* update from enterprise
* toggleMetricsOn to work around license delete
* eslint
* ensure admin before deleting license
* update from enterprise
* updates from enterprise
* fix cypress logging
* temp: log at DEBUG for Cypress tests
---------
Co-authored-by: Saturnino Abril <saturnino.abril@gmail.com>
Co-authored-by: Mattermost Build <build@mattermost.com>
