50 Коммитов

Автор SHA1 Сообщение Дата
Vishal
9966d111d2 [MM-69242] prevent Global Relay export panic on attachment read failure (release-10.11) (#37106)
Automatic Merge
2026-06-25 07:35:16 +02:00
Jesse Hallam
f6760151c4 Support Elasticsearch v9 (for v10.11) (#35925)
* Support Elasticsearch v9 alongside v8

* Add CI workflow changes for Elasticsearch v8/v9 testing

* Support Elasticsearch v7 in addition to v8/v9, add v7 CI test job

Lowers the minimum supported ES version from 8 to 7 to avoid dropping
v7 support in a dot release. Adds a dedicated CI job to verify v7
compatibility alongside the existing v8 and v9 (default) jobs.

* Fix ES7 plugin install crash on cgroup v2 hosts

* Fix ES 7 container startup on cgroup v2 Linux (GitHub Actions)

ES 7 bundles JDK 11, which crashes with a NullPointerException in
CgroupV2Subsystem.getMountPoint() on modern Linux kernels that use
cgroup v2 (including GitHub Actions ubuntu-latest runners). The flag
was already set during the Dockerfile RUN step, but not at runtime.

Adding -XX:-UseContainerSupport to ES_JAVA_OPTS in docker-compose
fixes the crash. The flag is harmless on ES 8/9 which ship JDK 17+
where the cgroup v2 bug is fixed (it simply opts out of container-aware
JVM sizing).

* Capture docker compose logs in CI test artifact

* Use ES 7.17.29 for v7 CI test; remove cgroup v2 workarounds

ES 7.17.0 bundled JDK 17.0.1 which had a cgroup v2 bug
(CgroupV2Subsystem NPE) not fixable via -XX:-UseContainerSupport.
ES 7.17.29 bundles JDK 22 where the bug is long fixed.

Reverts the -XX:-UseContainerSupport workarounds added in the
previous two commits as they were based on a wrong diagnosis
and are no longer needed.

---------

Co-authored-by: Mattermost Build <build@mattermost.com>
2026-04-20 20:44:31 +02:00
Jesse Hallam
73d5f507a4 Cherry-pick Go 1.25.8 upgrade for release-10.11 (#36125)
* ci: shard server Postgres tests into 4 parallel runners (#35739)

* ci: add test sharding plumbing to server CI

Add infrastructure for upcoming test sharding without changing behavior:

- Add shard-index and shard-total inputs to server-test-template.yml
  (defaults preserve existing single-runner behavior)
- Add timing cache restore step (activates only when shard-total > 1)
- Add merge-postgres-test-results job to server-ci.yml that:
  - Merges JUnit XML reports from shard artifacts
  - Saves timing data cache for future shard balancing
  - Handles both single-artifact and multi-shard scenarios
- Add .gitignore entries for timing cache and shard work files

Co-authored-by: Claude <claude@anthropic.com>

* ci: shard server Postgres tests into 4 parallel runners

Extract sharding logic into standalone, tested scripts and enable
4-shard parallel test execution for server Postgres CI:

Scripts:
- server/scripts/shard-split.js: Node.js bin-packing solver that
  assigns test packages to shards using timing data from previous runs.
  Two-tier strategy: light packages (<2min) whole, heavy packages
  (api4, app) split at individual test level.
- server/scripts/run-shard-tests.sh: Multi-run wrapper that calls
  gotestsum directly for each package group with -run regex filters.
- server/scripts/shard-split.test.js: 8 test cases covering round-robin
  fallback, timing-based balancing, heavy package splitting, JUnit XML
  fallback, and enterprise package separation.

Workflow changes:
- server-test-template.yml: Add shard splitting step that discovers test
  packages and runs the solver. Modified Run Tests step to use wrapper
  script when sharding is active.
- server-ci.yml: Add 4-shard matrix to test-postgres-normal. Update
  merge job artifact patterns for shard-specific names.

Performance: 7.2 min with timing cache vs 62.5 min baseline = 88%
wall-time improvement. First run without cache uses JUnit XML fallback
or round-robin, then populates the cache for subsequent runs.

Co-authored-by: Claude <claude@anthropic.com>

* fix: raise heavy package threshold to 5 min to preserve test isolation

sqlstore integrity tests scan the entire database and fail when other
packages' test data is present. At 182s, sqlstore was just over the
120s threshold and getting split at test level. Raising to 300s keeps
only api4 (~38 min) and app (~15 min) as heavy — where the real
sharding gains are — while sqlstore, elasticsearch, etc. stay whole
and maintain their test isolation guarantees.

Co-authored-by: Claude <claude@anthropic.com>

* ci: only save test timing cache on default branch

PR branches always restore from master's timing cache via restore-keys
prefix matching. Timing data is stable day-to-day so this eliminates
cache misses on first PR runs and reduces cache storage.

Co-authored-by: Claude <claude@anthropic.com>

* ci: skip FIPS tests on PRs (enterprise CI handles compile check)

Per review feedback: the enterprise CI already runs a FIPS compile
check on every PR. Running the full FIPS test suite on PRs is redundant
since it uses the identical test suite as non-FIPS — the only
FIPS-specific failure mode is a build failure from non-approved crypto
imports, which the enterprise compile check catches.

Full FIPS tests continue to run on every push to master.

Co-authored-by: Claude <claude@anthropic.com>

* fix: address review feedback on run-shard-tests.sh

- Remove set -e so all test runs execute even if earlier ones fail;
  track failures and exit with error at the end (wiggin77)
- Remove unused top-level COVERAGE_FLAG variable (wiggin77)
- Fix RUN_IDX increment position so report, json, and coverage files
  share the same index (wiggin77)
- Update workflow comment: heavy threshold is 5 min, not 2 min (wiggin77)

Co-authored-by: Claude <claude@anthropic.com>

* style: use node: prefix for built-in fs module in shard-split.js

Co-authored-by: Claude <claude@anthropic.com>

* fix: avoid interpolating file paths into generated shell script

Read shard package lists from files at runtime instead of interpolating
them into the generated script via printf. This prevents theoretical
shell metacharacter injection from directory names, as flagged by
DryRun Security.

Co-authored-by: Claude <claude@anthropic.com>

* fix(ci): rename merged artifact to match server-ci-report glob

The merged artifact was named postgres-server-test-logs-merged which
does not match the *-test-logs pattern in server-ci-report.yml,
causing Postgres test results to be missing from PR/commit reports.

Also pins junit-report-merger to exact version 7.0.0 for supply chain
safety.

Co-authored-by: Claude <claude@anthropic.com>

* fix(ci): pass RACE_MODE env into Docker container

RACE_MODE was set on the host runner but never included in the docker
run --env list. The light-package path worked because the heredoc
expanded on the host, but run-shard-tests.sh reads RACE_MODE at
runtime inside the container where it was unset. This caused heavy
packages (api4, app) to silently lose -race detection.

Co-authored-by: Claude <claude@anthropic.com>

* fix(ci): discover new tests in heavy packages not in timing cache

Tests not present in the timing cache (newly added or renamed) would
not appear in any shard -run regex, causing them to silently skip.
After building items from the cache, run go test -list to discover
current test names and assign any cache-missing tests to shards via
the normal bin-packing algorithm with a small default duration.

Co-authored-by: Claude <claude@anthropic.com>

* fix(ci): add missing line continuation backslash in docker run

The previous --env FIPS_ENABLED line was missing a trailing backslash
after adding --env RACE_MODE, causing docker run to see a truncated
command and fail with "requires at least 1 argument".

Co-authored-by: Claude <claude@anthropic.com>

* fix(ci): add setup-go step for shard test discovery

go test -list in shard-split.js runs on the host runner via execSync,
but Go is only available inside the Docker container. Without this
step, every invocation fails silently and new-test discovery is a
no-op. Adding actions/setup-go before the shard split step ensures
the Go toolchain is available on the host.

Co-authored-by: Claude <claude@anthropic.com>

---------

Co-authored-by: Claude <claude@anthropic.com>

* updated go to version 1.25.8 (#35817)

* updated go to version 1.25.8

* updated gotestsum version to work with go 1.25.8

go 1.25 does not work with indirect tools 0.11 dependency pulled by
gotestsum.

* Use sync.WaitGroup.Go to simplify goroutine creation

Replace the wg.Add(1) + go func() { defer wg.Done() }() pattern with
wg.Go(), which was introduced in Go 1.25.

* pushes fips image on workflow dispatch to allow fips test to run on go version update

* fix new requirements for FIPS compliance imposed on updating to go 1.25.8

* updates openssl symbol check for library shipped with FIPS new versions

go-openssl v2 shipped with FIPS versions starting from 1.25 uses mkcgo to generate
bindings causing symbol names to be different.

* removes temp workflow-dispatch condition

* keep versions out of agents md file

* upgrade golangci-lint (#35845)

* test: clean up channel store data after TestChannelStore (#36066)

TestChannelStore sub-tests create channels, members, and team members
using fake TeamIds and UserIds (model.NewId() for non-existent rows).
These records are left in the database and cause integrity tests
(TestCheck*) running in the same binary to fail their full-table scans.

Register a t.Cleanup on TestChannelStore that purges the affected
tables entirely. A blanket purge is safe: the schema enforces no FK
constraints, and every test suite creates its own data independently.

* Fix command injection in server-test-template workflow (#36080)

Replace the unquoted heredoc (which embedded GITHUB_HEAD_REF into a
generated script) with a cp of the existing run-shard-tests.sh, which
already handles the light-only case. Pass BUILD_NUMBER and TEST_TARGET
as explicit docker env vars instead of interpolating them into script
content.

* fix(ci): restore testname format in sharded gotestsum runs (#36078)

run-shard-tests.sh called gotestsum directly without --format, so it
fell back to gotestsum's default (pkgname) instead of the testname
format set by the Makefile. Pass --format "${GOTESTSUM_FORMAT:-testname}"
to match the Makefile default.

Co-authored-by: Mattermost Build <build@mattermost.com>

* fix(lint): fix pre-existing golangci-lint v2.11.4 issues

Fix misspelling in comment and redundant nil check flagged by the
upgraded linter.

* ci: use golang image for test runner on release-10.11

mattermost-build-server images are not built for release branches.
Use the official golang image which is always available for any Go version.

* ci: use mattermost/mattermost-build-server for release-10.11

The mattermostdevelopment/ images are only built for master.
The production mattermost/ images are built for release branches.

* ci: use mattermost/mattermost-build-server in mmctl test template

The mattermostdevelopment/ images are only built for master.
The production mattermost/ images are built for release branches.

---------

Co-authored-by: Pavel Zeman <pavel.zeman@mattermost.com>
Co-authored-by: Claude <claude@anthropic.com>
Co-authored-by: Carlos Garcia <carlos.garcia@mattermost.com>
Co-authored-by: Mattermost Build <build@mattermost.com>
2026-04-16 09:14:15 -03:00
Christopher Poile
9b1e03a6b8 [MM-63557] mmctl: Add compliance export create cmd (#30594)
* Refactor job retrieval to support multiple statuses & multiple types

- Updated job retrieval functions to handle multiple job statuses.
- Renamed `GetJobsByTypeAndStatus` to `GetJobsByTypesAndStatuses` for consistency across the codebase.
- Adjusted related function signatures and implementations in the job store and retry layer to accommodate the new method.
- Updated tests to reflect changes in job retrieval logic and ensure proper functionality.

* Add compliance export create command and tests

- Introduced `ComplianceExportCreateCmd` to facilitate the creation of compliance export jobs with options for date, start, and end timestamps.
- Added unit tests for the new command, covering various scenarios including valid and invalid inputs.
- Updated documentation to include usage examples and options for the new command.
- Enhanced existing tests to ensure proper functionality of compliance export job handling.

* update docs

* update tests for new logic

* Refactor message export job tests to use DefaultPreviousJobPageSize

- Updated all test cases in worker_test.go to replace hardcoded page size of 100 with DefaultPreviousJobPageSize for consistency.
- Adjusted the worker.go file to define DefaultPreviousJobPageSize and use it in job retrieval logic.
- Ensured that the changes maintain the functionality of job data initialization and retrieval tests.

* PR comments

* PR comments, simplifications, clarifications, formatting

* prefer hypen over underscore in command names

* merge conflict

* update mmctl docs
2025-06-24 21:38:30 +00:00
David Krauser
761584c040 [MM-64244] Add websocket disconnect reason metric (#31032)
We've recently spent some effort improving websocket reconnection logic. With this commit, I've augmented the websocket reconnect metric to include a disconnect reason. This will help us measure the impact of these changes in production.
2025-05-30 08:15:20 -04:00
Agniva De Sarker
79bf1c34db MM-63217: Bump dependencies (#31021)
* MM-63217: Bump dependencies

Skip-Enterprise-PR: true

https://mattermost.atlassian.net/browse/MM-63217

```release-note
NONE
```

* fix go.mod

```release-note
NONE
```

* fix test

```release-note
NONE
```
2025-05-29 13:08:00 +05:30
Ibrahim Serdar Acikgoz
a344b3225b [MM-61756] Attribute Based Access Control - Phase 1 (#30785)
Attribute Based Access Control - Base
* MM-63662

* MM-63919

* MM-63954

* MM-63955 

* MM-63425

* MM-63426

* MM-63458

* MM-63459

* MM-63603

* MM-63845

* MM-64146

* MM-64199

* MM-64201

* MM-64233

* MM-64247

* MM-64268

---------

Co-authored-by: Harshil Sharma <harshilsharma63@gmail.com>
Co-authored-by: Pablo Andrés Vélez Vidal <pablovv2012@gmail.com>
Co-authored-by: abhijit-singh <abhijitsingh0702@gmail.com>
Co-authored-by: Harrison Healey <harrisonmhealey@gmail.com>
2025-05-15 11:33:08 +02:00
Ibrahim Serdar Acikgoz
d452f4f043 [MM-63661] add access control metrics (#30680) 2025-05-11 22:11:42 +02:00
Agniva De Sarker
271d928e87 MM-63467: Refactor constLabels to a common initializer (#30779)
This is useful to avoid mistakes while adding new metrics.

https://mattermost.atlassian.net/browse/MM-63467

```release-note
NONE
```

Co-authored-by: Mattermost Build <build@mattermost.com>
2025-04-23 21:59:51 +05:30
Agniva De Sarker
09488558a0 MM-63298: [AI assisted] Elasticsearch add a global search prefix (#30417)
This PR adds functionality to search by a global search prefix.
This allows Mattermost to be used across multiple data centers
with multiple Elasticsearch instances synchronized using
cross-cluster replication.

While here, we also add tests cases to cover for some missing
search interface methods.

For now, no system console setting is exposed. Because IndexPrefix
is also not exposed. It can be added later if a need arises.

https://mattermost.atlassian.net/browse/MM-63298

```release-note
A new config setting ElasticsearchSettings.GlobalSearchPrefix is added
which can be used to search across multiple indices having a common prefix.
This is useful in a scenario with multiple Elasticsearch instances, where
multiple instances are writing to different indices with different prefixes
using the ElasticsearchSettings.IndexPrefix setting.
```

Co-authored-by: Mattermost Build <build@mattermost.com>
2025-04-04 11:42:12 +05:30
Ben Schumacher
9b5d8d52bf [MM-62427] Add message attachments validation (#30180)
* Add message attachments validation

* Add props validation

* Validate slack attachment fields

* Update tests and library usage

* Improve interactive dialog error for length checks

* Allow predefined colors for slack attachments

* Fix TestPostAction

* Use const for data source

* Add tests

* Cleanup unused props

* Add happy path tests

* lint fixes

* Add validation for PostActionOptions
2025-03-20 12:53:50 +01:00
Christopher Poile
c049748b88 [MM-63314] Fix ClaimJob in HA environments (#30383)
* ClaimJob now returns newly claimed job

* internal code affected by change

* test changes required

* two branches: for mysql, use transaction; for postgres, use returning

* two branches: for mysql, use transaction; for postgres, use returning

* use same millis value for LastActivityAt and StartAt

* blank commit

---------

Co-authored-by: Mattermost Build <build@mattermost.com>
2025-03-14 10:24:26 -04:00
Agniva De Sarker
1a58f923e0 [aider assisted] MM-61888: Add ClientSideUserIds field to MetricsSettings (#30127)
We add a new config setting to allow the admin to set a fixed
list of userIDs to track for all client side webapp metrics.

This gives the admin to get a deeper look at how the application
is behaving for a single user.

A new section in the system console is also added for the user
to edit this setting from the UI.

https://mattermost.atlassian.net/browse/MM-61888

```release-note
A new config setting MetricsSettings.ClientSideUserIds is added
where you can set the user ids you want to track for client side webapp
metrics.
```

* fix lint errors

```release-note
NONE
```

* fixing tests

```release-note
NONE
```
2025-02-13 21:10:34 +05:30
Agniva De Sarker
d34dcc2dda MM-62901: Apply constLabels on all metrics (#30125)
When the client side metrics were added constLabels
got missed out. Without this, we cannot do the high
level grouping by rings.

https://mattermost.atlassian.net/browse/MM-62901

```release-note
NONE
```
2025-02-06 21:45:57 +05:30
Harshil Sharma
35e776d805 Removed Elasticsearch and Opensearch channel index schedma check (#30102) 2025-02-06 11:50:37 +05:30
Christopher Poile
8b65771a31 [MM-62897] Compliance export backwards compatibility (#30120)
* add backwards compatibility for <10.5

* test: backwards compatibility for <10.5

* spelling
2025-02-05 21:20:28 +00:00
Agniva De Sarker
afbd9d64c3 [AI assisted]: MM-62295: Search and index archived channels as well. (#29796)
We add a new field delete_at in the channels template.

This field is then searched in the SearchChannels function.

Also added tests to verify that archived channels are searched
properly, and also indexed correctly.

https://mattermost.atlassian.net/browse/MM-62295

```release-note
- Now archived channels are searchable with ES/OS if TeamSettings.ExperimentalViewArchivedChannels is enabled.
- If there are old channels which were archived before a bulk index was run, users would need to purge indexes, and do bulk index again. Because those old archived channels are removed from the index when a bulk index is run.
```


Co-authored-by: Mattermost Build <build@mattermost.com>
2025-01-28 19:44:55 +05:30
Christopher Poile
737bed311c MM-60115, MM-62436, MM-62493: Compliance export downloads from local and s3; e2e tests (#29806)
* add ZipReader method to filestore and file/s3 backends

to merge

to merge

* add WriteStreamResponse as an alternative to WriteFileResponse

* add generated layers

* enable download link; download job endpoint now streams export dir zips

* fix MM-62493

* re-enable e2e tests--we have download links, folks

* Add tests for ZipReader in filestore and s3store

* remove unnecessary error return on ZipReader

* little cleanup

* improve tests; some refactoring: s.Nil(err) -> s.NoError(err)

* blank commit

* backwards compatability for pre-10.5 job downloads

* compress file response; better errors; better comments; PR comments

* update generated app layers

* improve/widen tests; improve comments; simplify localstore ZipReader

* regenerate layers

* follow GoDoc conventions

* update generated layers

* remove unnecessary comment

* in jobs/job-id/download, clean exportDir before sending to ZipReader

* better comments; add an error return on ZipReader

* improve file permissions

* adjust tests for new error returns

* linting

* i18n
2025-01-27 09:28:07 +05:30
Rahim Rahman
540408545a feat(MM-61865): Add mobile client content load network metrics (#29601)
* MM-61865: Add mobile client content load network metrics

* added new common label

* renaming from MobileClientContentLoad to MobileClientNetworkRequests
* content_load_group => network_request_group

* refactor more NetworkRequest-* changes

* replace contentLoadGroup to networkRequestGroup

* new metrics elapsedTime

* Refactor urlCount to totalRequests

* add averageSpeed metric
* replace contentLoadGroup with networkRequestGroup
* use h.Labels vs commonLabels for network_request_group

* add agent

* add effective latency metrics

* add total parallel requests & total sequential

* mocks generated by mockery

* did a bit of cleanup and sorting

* formatting

* updated the AcceptedNetworkRequestGroups

* cleanup and sorting
2025-01-16 11:03:41 -07:00
Christopher Poile
aba4434dab MM-59966 - Compliance Export overhaul - feature branch (#29789)
* [MM-59089] Add a compliance export constant (#27919)

* add a useful constant

* i18n

* another constant

* another i18n

* [MM-60422] Add GetChannelsWithActivityDuring (#28301)

* modify GetUsersInChannelDuring to accept a slice of channelIds

* add GetChannelsWithActivityDuring

* add compliance export progress message; remove unused custom status

* linting

* tests running too fast

* add batch size config settings

* add store tests

* linting

* empty commit

* i18n changes

* fix i18n ordering

* MM-60570 - Server-side changes consolidating the export CLI with server/ent code (#28640)

* add an i18n field; add the CLI's export directory

* int64 -> int

* Add UntilUpdateAt for MessageExport and AnalyticsPostCount

to merge

* remove now-unused i18n strings

* add TranslationsPreInitFromBuffer to allow CLI to use i18n

* use GetBuilder to simplify; rename TranslationsPreInitFromFileBytes

* [MM-59089] Improve compliance export timings (#1733 - Enterprise repo)

* MM-60422 - Performance and logic fixes for Compliance Exports (#1757 - Enterprise repo)

* MM-60570 - Enterprise-side changes consolidating the export CLI with server/ent code (#1769 - Enterprise repo)

* merge conflicts; missed file from ent branch

* MM-61038 - Add an option to sqlstore.New (#28702)

remove useless comment

add test

add an option to sqlstore.New

* MM-60976: Remove RunExport command from Mattermost binary (#28805)

* remove RunExport command from mattermost binary

* remove the code it was calling

* fix i18n

* remove test (was only testing license, not functionality)

* empty commit

* fix flaky GetChannelsWithActivityDuring test

* MM-60063: Dedicated Export Filestore fix, redo of #1772 (enterprise) (#28803)

* redo filestore fix #1772 (enterprise repo) on top of MM-59966 feature

* add new e2e tests for export filestore

* golint

* ok, note to self: shadowing bad, actually (when there's a defer)

* empty commit

* MM-61137 - Message export: Support 7.8.11 era dbs (#28824)

* support 7.8.11 era dbs by wrapping the store using only what we need

* fix flaky GetChannelsWithActivityDuring test

* add a comment

* only need to define the MEFileInfoStore (the one that'll be overridden)

* blank commit

* MM-60974 - Message Export: Add performance metrics (#28836)

* support 7.8.11 era dbs by wrapping the store using only what we need

* fix flaky GetChannelsWithActivityDuring test

* add a comment

* only need to define the MEFileInfoStore (the one that'll be overridden)

* performance metrics

* cleanup unneeded named returns

* blank commit

* MM-60975 - Message export: Add startTime and endTime to export folder name (#28840)

* support 7.8.11 era dbs by wrapping the store using only what we need

* fix flaky GetChannelsWithActivityDuring test

* add a comment

* only need to define the MEFileInfoStore (the one that'll be overridden)

* performance metrics

* output startTime and endTime in export folder

* empty commit

* merge conflict

* MM-60978 - Message export: Improve xml fields; fix delete semantics (#28873)

* support 7.8.11 era dbs by wrapping the store using only what we need

* fix flaky GetChannelsWithActivityDuring test

* add a comment

* only need to define the MEFileInfoStore (the one that'll be overridden)

* performance metrics

* output startTime and endTime in export folder

* empty commit

* add xml fields, omit when empty, tests

* fix delete semantics; test (and test for update semantics)

* clarify comments

* simplify edited post detection, now there's no edge case.

* add some spacing to help fast running tests

* merge conflicts/updates needed for new deleted post semantics

* linting; fixing tests from upstream merge

* use SafeDereference

* linting

* stronger typing; better wrapped errors; better formatting

* blank commit

* goimports formatting

* fix merge mistake

* minor fixes due to changes in master

* MM-61755 - Simplifying and Support reporting to the db from the CLI (#29281)

* finally clean up JobData struct and stringMap; prep for CLI using db

* and now simplify using StringMapToJobDataWithZeroValues

* remove unused fn

* create JobDataExported; clean up errors

* MM-60176 - Message Export: Global relay cleanup (#29168)

* move global relay logic into global_relay_export

* blank commit

* blank commit

* improve errors

* MM-60693 - Refactor CSV to use same codepath as Actiance (#29191)

* move global relay logic into global_relay_export

* blank commit

* refactor (and simplify) ExportParams into shared

* blank commit

* remove unused fn

* csv now uses pre-calculated joins/leaves like actiance

* improve errors

* remove nil post check; remove ignoredPosts metric

* remove unneeded copy

* MM-61696 - Refactor GlobalRelay to use same codepath as Actiance (#29225)

* move global relay logic into global_relay_export

* blank commit

* refactor (and simplify) ExportParams into shared

* blank commit

* remove unused fn

* csv now uses pre-calculated joins/leaves like actiance

* remove newly unneeded function and its test. goodbye.

* refactor GetPostAttachments for csv + global relay to share

* refactor global_relay_export and fix tests (no changes to output)

* improve errors

* remove nil post check; remove ignoredPosts metric

* remove unneeded copy

* remove unneeded nil check

* PR comments

* MM-61715 - Generalize e2e to all export types 🤖  (#29369)

* move global relay logic into global_relay_export

* blank commit

* refactor (and simplify) ExportParams into shared

* blank commit

* remove unused fn

* csv now uses pre-calculated joins/leaves like actiance

* remove newly unneeded function and its test. goodbye.

* refactor GetPostAttachments for csv + global relay to share

* refactor global_relay_export and fix tests (no changes to output)

* improve errors

* remove nil post check; remove ignoredPosts metric

* remove unneeded copy

* remove unneeded nil check

* PR comments

* refactor isDeletedMsg for all export types

* fix start and endtime, nasty csv createAt bug; bring closer to Actiance

* align unit tests with new logic (e.g. starttime / endtime)

* refactor a TimestampConvert fn for code + tests

* bug: pass templates to global relay (hurray for e2e tests, otherwise...)

* add global relay zip to allowed list (only for tests)

* test helpers

* new templates for e2e tests

* e2e tests... phew.

* linting

* merge conflicts

* unexport PostToRow; add test helper marker

* cleanup, shortening, thanks to PR comments

* MM-61972 - Generalize export data path - Actiance (#29399)

* extract and generalize the export data generation functions

* finish moving test (bc of previous extraction)

* lift a function from common -> shared (to break an import cycle)

* actiance now takes general export data, processes it into actiance data

* bring tests in line with correct sorting rules (upadateAt, messageId)

* fixups, PR comments

* turn strings.Repeat into a more descriptive const

amended: one letter fix; bad rebase

* MM-62009 - e2e clock heisenbug (#29434)

* consolidate assertions; output debuggable diffs (keeping for future)

* refactor test output generator to generators file

* waitUntilZeroPosts + pass through until to job = fix all clock issues

* simplify messages to model.NewId(); remove unneeded waitUntilZeroPosts

* model.NewId() -> storetest.NewTestID()

* MM-61980 - Generalize export data path - CSV (#29482)

* simple refactoring

* increase sleep times for (very) rare test failures

* add extra information to the generic export for CSV

* adj Actiance to handle new generic export (no difference in its output)

* no longer need mergePosts (yay), move getJoinLeavePosts for everyone

* adjust tests for new csv semantics (detailed in summary)

* and need to add the new exported data to the export_data_tests

* rearrange csv writing to happen after data export (more logical)

* linting

* remove debug statements

* figured out what was wrong with global relay e2e test 3; solid now

* PR comments

* MM-61718 - Generalize export data path - Global Relay (#29508)

* move global relay over to using the generalized export data

* performance pass -- not much can be done

* Update server/enterprise/message_export/global_relay_export/global_relay_export.go

Co-authored-by: Claudio Costa <cstcld91@gmail.com>

---------

Co-authored-by: Claudio Costa <cstcld91@gmail.com>

* MM-62058 - Align CSV with Actiance (#29551)

* refactoring actiance files and var names for clarity

* bug found in exported attachments (we used to miss some start/ends)

* changes needed for actiance due to new generic exports

* bringing CSV up to actiance standards

* fixing global relay b/c of new semantics (adding a note on an edge case)

* aligning e2e tests, adding comments to clarify what is expected/tested

* necessary changes; 1 more test for added functionality (ignoreDeleted)

* comment style

* MM-62059 - Align Global Relay with Actiance/CSV; many fixes (#29665)

* core logic changes to general export_data and the specific export paths

* unit tests and e2e tests, covering all new edge cases and all logic

* linting

* better var naming, const value, and cleaning up functions calls

* MM-62436 - Temporarily skip cypress tests that require download link (#29772)

---------

Co-authored-by: Claudio Costa <cstcld91@gmail.com>
2025-01-10 16:56:02 -05:00
Agniva De Sarker
647c9b6808 [MM-62293]: Remove Elasticsearch from translation strings (#29728)
We cannot simply use Elasticsearch any more because
we support Opensearch as well.

Therefore changing Elasticsearch to just say Search.

Note: There are still some cases where Elasticsearch
is used. That is intentional and necessary.

```release-note
NONE
```
2025-01-08 11:11:59 +05:30
Agniva De Sarker
42ff2a2639 [Aider assisted]: Templatize rest of the elasticsearch translation strings (#29582)
After https://github.com/mattermost/mattermost/pull/29209, there were some more
strings which were left out. Sending another PR to fix all of them.

The config validation strings are made generic for simplicity, because they
are called before the search backend is started, and even though you could
check for the config setting, it forces the code order. Therefore, changed
the words to just say "Search" for simplicity.

I tried to use Aider initially for a little while, but ended up fighting
a lot with the search/replace and having to repeat myself over and over again
to get it to do the refactoring. Eventually gave up and did rest of the changes
myself.

```release-note
NONE
```
2024-12-13 19:25:08 +05:30
Agniva De Sarker
59b8532a89 metrics/histogram: Fixing incorrect label_name (#29492)
We were incorrectly overwriting the same key - label_name.
```release-note
NONE
```
2024-12-06 19:34:46 +05:30
Agniva De Sarker
a6d37fa14c MM-61887: Log the userID if a metric exceeds the last histogram bucket (#29448)
We create a custom histogram metric that logs the userID
when the observed value is greater or equal to the last bucket value.

This allows us to start tracking the slowest users of a system
while at the same time not polluting the Prometheus metrics
by storing a userID for every observation.

https://mattermost.atlassian.net/browse/MM-61887

```release-note
NONE
```
2024-12-05 09:12:54 +05:30
Agniva De Sarker
4ec4b4d525 MM-61886: Add actionable page navigation metrics (#29332)
Page load is one of the metrics that we track and present
to MLT. However, in its current form, it is not very
actionable because it also contains the network latency.

We split the whole metric into these parts:
startTime
|
responseStart = TTFB
|
responseEnd = TTLB
|
domInteractive = Start of processing phase
|
loadEventEnd = Load complete

This gives us better visibility into exactly
which phase in the load process is slow.

I have experimented with other metrics like
- domContentLoadedEventStart
- domContentLoadedEventEnd
- domComplete

and observed that they do not have sufficient
gaps in the timespan to have any relevance.

Additionally, I have moved TTFB from being a
web vitals metric to being tracked from the performance
metrics to remain consistent with the other navigation
metrics measured.

Lastly, I took this chance to improve some of the
validation errors that we threw to include more
context into the input that was passed and why
does it fail.

This also meant that I had to change the tests
to check for error strings rather than direct
errors which is a bad thing, but I don't think
it's worth the effort trying to have named error
variables for all of them.

https://mattermost.atlassian.net/browse/MM-61886

```release-note
NONE
```
2024-11-29 11:24:35 +05:30
Agniva De Sarker
9fed818200 Update ES template strings (#29209)
Use a template variable to distinguish between Elasticsearch
and Opensearch
```release-note
NONE
```
2024-11-13 15:22:41 +05:30
Agniva De Sarker
65ed87bda0 Move Elasticsearch to source available 🎉 🎉 (#29015)
* Move Elasticsearch to source available
2024-11-06 09:26:54 +05:30
Devin Binnie
0c90b0363b [MM-60609][MM-60612] Include Desktop App metrics in PerformanceReporter, add metrics in Prometheus for CPU/Memory usage (#28825)
* [MM-60609][MM-60612] Include Desktop App metrics in PerformanceReporter, add metrics in Prometheus for CPU/Memory usage

* Fix mocks

* PR feedback
2024-10-22 12:16:20 -04:00
Christopher Poile
b8e462e198 MM-60987: Move message_export code to monorepo (#28674)
* move message_export code to monorepo

* update license headers

* moving over `dedicated filestore` (#1772) from enterprise master
2024-10-15 13:03:31 -04:00
Daniel Espino García
040838b056 Add metrics for mobile versions snapshots (#28191)
* Add metrics for mobile versions snapshots

* Add notifications disabled and fix lint

* Address feedback

* Verify all references to JobTypeActiveUsers

* Fix typos

* Improve platform values

* Add test and MySQL support
2024-09-24 12:02:19 +02:00
Agniva De Sarker
b9debc75a0 MM-60478: Do not invalidate session cache on user added to channel (#28142)
PR: https://github.com/mattermost/mattermost/pull/27204 fixed the issue
where `invalidateWebConnSessionCacheForUser` was not broadcasted across
the cluster. However, we became too conservative and started to invalidate
the session cache as well.

But the session cache only contains the session details for the user
and the team membership. There is no channel membership related info.

Therefore, we avoid that and improve performance as well.

https://mattermost.atlassian.net/browse/MM-60478

```release-note
NONE
```

Co-authored-by: Mattermost Build <build@mattermost.com>
2024-09-11 20:42:49 +05:30
Harrison Healey
8c5f00da86 MM-60285 Add fresh label to channel and team switch metrics (#28100)
* Changed measureAndReport to take an object as parameters

* MM-60285 Add fresh label to channel and team switch metrics
2024-09-05 16:30:24 -04:00
Agniva De Sarker
540febd866 MM-56876: Redis: first introduction (#27752)
```release-note
NONE
```

---------

Co-authored-by: Jesús Espino <jespinog@gmail.com>
Co-authored-by: Mattermost Build <build@mattermost.com>
2024-08-06 09:28:41 +05:30
Agniva De Sarker
c72e14f73d MM-59826: Add more invalidation event types (#27719)
This came up as part of the Redis work. I noticed that a lot of new
invalidation events were missing. This is important to understand
where invalidations are coming from.

https://mattermost.atlassian.net/browse/MM-59826

```release-note
NONE
```

Co-authored-by: Mattermost Build <build@mattermost.com>
2024-07-26 19:09:09 +05:30
Harrison Healey
e3b2b13292 MM-58535 Add more information to LCP and INP metrics (#27484)
* Improve mocking of imported resources in unit tests

We have Webpack configured so that, when code imports an image or other resource, the code gets the URL of that image. Jest now matches that behaviour which is needed because React Testing Library would previously throw an error.

* Polyfill ResizeObserver in all unit tests

* Ensure haveIChannelPermission always returns a boolean value

The previous code could sometimes return undefined. While that should behave the same in practice, it can cause React to print prop type warnings

* MM-58535 Add region label to LCP metrics

* MM-58535 Upgrade web-vitals and add INP attribution

* Change new labels to use snake_case

* Remove replaceGlobalStore option from renderWithContext

I was going to add this in case any tests failed with this option set to false, but after running those tests, that's not the case. I'm going to remove this as an option since it seems more likely than not that anyone using RTL would prefer to have this on.
2024-07-09 15:06:08 -04:00
Daniel Espino García
e5a3dd7fea Add platform information for push notification metrics (#27460)
* Add platform information for push notification metrics

* Address feedback

* Add the client platform returned by the devices to the normalize function

* Add "no platform" platform label to distinguish from unknown
2024-07-05 09:34:01 +02:00
Harrison Healey
6e5c741a7b MM-59033 Increase maximum bucket size for Load Event End metric 2024-06-21 17:47:39 -04:00
Daniel Espino García
1ec2de4a95 Add mobile metrics (#27045)
* Add mobile metrics

* Fix mocks

* Add tests

* Fix lint

* Address feedback

* Fix lint

* Fix test

* Fix CI

---------

Co-authored-by: Mattermost Build <build@mattermost.com>
2024-06-12 11:33:04 +02:00
Harrison Healey
beb8d5e8e0 MM-58367 Increase maximum bucket size for LCP metrics (#27147)
* MM-58367 Increase maximum bucket size for LCP metrics

* Don't potentially mutate DefBuckets

* Increase maximum bucket for FCP metrics too
2024-05-27 16:46:24 -04:00
Harrison Healey
1bd7b6f4c2 MM-58359 Add page_load to Prometheus metrics (#27159)
Co-authored-by: Mattermost Build <build@mattermost.com>
2024-05-27 14:06:18 -04:00
Harrison Healey
441f5657c8 MM-57882 Add metric for the time it takes to open the Threads list (#26983)
* MM-57882 Add metric for the time it takes to open the Threads list

* Clean up mark because the starting mark may be missing

* Pass global threads load duration to Prometheus

* Update mocks
2024-05-21 18:04:12 -04:00
Harrison Healey
b29c211ee2 MM-58303 Report client metrics to Prometheus in seconds (#27033) 2024-05-21 13:47:44 -04:00
Ibrahim Serdar Acikgoz
f0121d4f23 Change client metrics api path to client_perf (#26985) 2024-05-09 22:19:08 +02:00
Ibrahim Serdar Acikgoz
5590e1604a Add new Metrics API (#26919) 2024-05-09 20:49:02 +02:00
Devin Binnie
9cf5a4bccb Notification metrics fixes (#26854)
* Ensure your own posts are never ACKed

* Don't ACK mobile websocket notifications

* Add counter for the unsupported Desktop Apps

* Count only push messages when checking for acks

* Fix generated

* Add tests, fix comment

* Fix help string

* Check for nil session
2024-04-23 21:31:17 +00:00
Devin Binnie
02e23a3275 [MM-57066][MM-57329] Added metrics for all notification stopping points, consolidated categories between metrics and logging (#26799)
* [MM-57066] Add metric counters for notification events

* Some small changes

* Account for Metrics() sometimes being nil

* Fix test (again)

* Fix more tests

* A few changes from testing - added success counter

* Missed a mock

* Lint

* Add feature flag for notification monitoring
2024-04-18 10:30:08 -04:00
Claudio Costa
4b508eed46 [MM-57010] Include client type in websocket connections metric (#26763)
* Include client type in websocket connections metric

* Unexport field
2024-04-16 10:49:49 -06:00
Doug Lauder
38bbf04e48 Metrics for shared channels (#26199)
* add metrics definitions for shared channels
2024-02-21 17:21:35 -05:00
Michael Kochell
4e071e861c Webapp - Outgoing OAuth Connections (#25507)
* added store

* make generated

* add missing license headers

* fix receiver name

* i18n

* i18n sorting

* update migrations from master

* make migrations-extract

* update retrylayer tests

* replaced sql query with id pagination

* fixed flaky tests

* missing columns

* missing columns on save/update

* typo

* improved tests

* remove enum from mysql colum

* add password credentials to store

* license changes

* OAuthOutgoingConnectionInterface

* Oauth -> OAuth

* make generated

* copied over installed_oauth_apps component and renamed things to installed_outgoing_oauth_connections

* merge migrations

* renamed migrations

* model change suggestions

* refactor test functionsn

* migration typo

* refactor store table names

* updated sanitize test

* cleanup merge

* refactor symbol

* "installed outgoing oauth connections" page works

* move things into a nested folder

* add and edit page stubs work

* list endpoint

* oauthoutgoingconnection -> outgoingoauthconnection

* signature change

* i18n update

* granttype typo

* naming

* api list

* uppercase typo

* i18n

* missing license header

* fixed path in comments

* updated openapi definitions

* changes to support selecting command request url

* sanitize connections

* make generated

* test license and no feature flag

* removed t.fatal

* updated testhelper calls

* yaml schema fixes

* switched interface name

* suggested translation

* missing i18n translation

* management permission

* moved permission initalization to proper place

* endpoints

* put tests

* error check typo

* fixed specific enttity urls

* tests

* read permission check

* updated openapi definitions

* i18n

* GetConnectionByAudience method

* notes

* replaced GetConnectionsByAudience with a filter

* added custom oauth token object

* updated interface and usage

* properly set enterprise interface

* move retrieval logic to impl

* webhook tests

* translations

* i18n: updates

* address comments

* endpoint and tests

* i18n

* api docs

* fixed endpoint path

* sq.like

* use filter object instead of parameters

* set url values if not empty

* typos

* converted some components to function components, and move around files

* correctly check token url

* restore flag to previous value

* added command oauth handler

* update enterprise imports

* migrate last component to function component

* Added enterprise import

* refactor permissions and add necessary webapp code

* Check correct flag in permission tree

* allow partial updates

* sort i18n webapp

* missing test modification

* fixed webapp i18n sorting

* allow validating stored connections

* added missing translation

* fix finished adding connection link and text on result page

* added missing permission to smoke tests

* missing role in smoke test

* updated translations

* updated translations

* support editing client secret on existing connection

* fix some i18n strings

* updated translations

* better error messages

* progress on using react select for command request url while maintaining typed in value

* remove writeheader, test

* HasValidGrantType

* end early to avoid nil pointer errors

* move slash command request url input box into its own component

* wrap components related to oauth connections in config check

* fix tests

* i18n-extract

* change some i18n strings to say "Outgoing OAuth 2.0 Connections"

* remove debug code

* fixed i18n

* updated i18n file

* feature configuration backend

* typo

* add system console setting

* Revert "typo"

This reverts commit 669da23e8ee47525ccaa6f59cbbd20bf8a121191.

* Revert "updated i18n file"

This reverts commit d0882c0dd7587533f0d0f7a7b7b190684186158a.

* Revert "fixed i18n"

This reverts commit 3108866bc19139182dfd094921c56cdefc4695ea.

* fixed i18n

* updated i18n file

* typo

* updated i18n

* updated i18n

* updated i18n

* updated version to 9.6

* replace feature flag with system console configuration

* i18n

* updated tests

* pr feedback

* fix styling of disabled text box

* fix styling of action links in integration console

* server changes for validation feature

* webapp changes for validation feature

* pencil icon styling

* styling fixes for oauth audience correct configuration message

* fix sanitize test

* remove max lengths from outgoing oauth connection form

* use config var in webapp instead of feature flag

* change asterisks to bullets

* update api docs for validate endpoint

* feedback from ux review

* fix lint, types, tests

* fix stylelint

* implement validation button under the token url input

* support wildcard for matching audience urls

* updates for styling

* update snapshots

* add doc links for the outgoing oauth connections feature

* change doc links to use permalink

* add docs link to system console

* fix: use limitedreader in json decoding

* fix: form error in validation

* management permission can read now

* updated api documentation

* doc typo

* require one permission to read only

* fix api connection list audience filter

* fix audience matching and add loading indicator

* fix team permissions on outgoing oauth connection api calls

* fix api doc and test, for adding team id to query params

* handle read permissions by adding a team in the payload

* missing teamid query parameter in test

* change validate button logic to not require audience urls to be filled out

* fix redux type

---------

Co-authored-by: Felipe Martin <me@fmartingr.com>
2024-02-09 14:49:49 -05:00
Jesse Hallam
b05093d508 Source available metrics (#24879)
* Expose metrics under a source available license

* do not assume Cluster()

* allow metrics if licensed or dev

* temporary vet override

* simplify BULID_TAGS handling

* auto clean old imports.go

* fix license listener

* e2e test metrics & license semantics

* update from enterprise

* switch back to mattermost-govet/v2@new now

* update metrics from upstream

* Update license_spec.js

Co-authored-by: Saturnino Abril <saturnino.abril@gmail.com>

* Update license_spec.js

Co-authored-by: Saturnino Abril <saturnino.abril@gmail.com>

* Update e2e-tests/cypress/tests/integration/channels/enterprise/metrics/license_spec.js

Co-authored-by: Saturnino Abril <saturnino.abril@gmail.com>

* Update e2e-tests/cypress/tests/integration/channels/enterprise/metrics/license_spec.js

* split up specs

* require/delete license earlier in e2e test

* expanded expect to debug failures

* more logging

* Revert "more logging"

This reverts commit 0bc513fd926c07a0c6050f4b4c742e856d8dc435.

* e2e: try deleting license first

* update from enterprise

* toggleMetricsOn to work around license delete

* eslint

* ensure admin before deleting license

* update from enterprise

* updates from enterprise

* fix cypress logging

* temp: log at DEBUG for Cypress tests

---------

Co-authored-by: Saturnino Abril <saturnino.abril@gmail.com>
Co-authored-by: Mattermost Build <build@mattermost.com>
2024-01-08 10:47:24 -04:00