213 Коммитов

Автор SHA1 Сообщение Дата
Jesse Hallam
73d5f507a4 Cherry-pick Go 1.25.8 upgrade for release-10.11 (#36125)
* ci: shard server Postgres tests into 4 parallel runners (#35739)

* ci: add test sharding plumbing to server CI

Add infrastructure for upcoming test sharding without changing behavior:

- Add shard-index and shard-total inputs to server-test-template.yml
  (defaults preserve existing single-runner behavior)
- Add timing cache restore step (activates only when shard-total > 1)
- Add merge-postgres-test-results job to server-ci.yml that:
  - Merges JUnit XML reports from shard artifacts
  - Saves timing data cache for future shard balancing
  - Handles both single-artifact and multi-shard scenarios
- Add .gitignore entries for timing cache and shard work files

Co-authored-by: Claude <claude@anthropic.com>

* ci: shard server Postgres tests into 4 parallel runners

Extract sharding logic into standalone, tested scripts and enable
4-shard parallel test execution for server Postgres CI:

Scripts:
- server/scripts/shard-split.js: Node.js bin-packing solver that
  assigns test packages to shards using timing data from previous runs.
  Two-tier strategy: light packages (<2min) whole, heavy packages
  (api4, app) split at individual test level.
- server/scripts/run-shard-tests.sh: Multi-run wrapper that calls
  gotestsum directly for each package group with -run regex filters.
- server/scripts/shard-split.test.js: 8 test cases covering round-robin
  fallback, timing-based balancing, heavy package splitting, JUnit XML
  fallback, and enterprise package separation.

Workflow changes:
- server-test-template.yml: Add shard splitting step that discovers test
  packages and runs the solver. Modified Run Tests step to use wrapper
  script when sharding is active.
- server-ci.yml: Add 4-shard matrix to test-postgres-normal. Update
  merge job artifact patterns for shard-specific names.

Performance: 7.2 min with timing cache vs 62.5 min baseline = 88%
wall-time improvement. First run without cache uses JUnit XML fallback
or round-robin, then populates the cache for subsequent runs.

Co-authored-by: Claude <claude@anthropic.com>

* fix: raise heavy package threshold to 5 min to preserve test isolation

sqlstore integrity tests scan the entire database and fail when other
packages' test data is present. At 182s, sqlstore was just over the
120s threshold and getting split at test level. Raising to 300s keeps
only api4 (~38 min) and app (~15 min) as heavy — where the real
sharding gains are — while sqlstore, elasticsearch, etc. stay whole
and maintain their test isolation guarantees.

Co-authored-by: Claude <claude@anthropic.com>

* ci: only save test timing cache on default branch

PR branches always restore from master's timing cache via restore-keys
prefix matching. Timing data is stable day-to-day so this eliminates
cache misses on first PR runs and reduces cache storage.

Co-authored-by: Claude <claude@anthropic.com>

* ci: skip FIPS tests on PRs (enterprise CI handles compile check)

Per review feedback: the enterprise CI already runs a FIPS compile
check on every PR. Running the full FIPS test suite on PRs is redundant
since it uses the identical test suite as non-FIPS — the only
FIPS-specific failure mode is a build failure from non-approved crypto
imports, which the enterprise compile check catches.

Full FIPS tests continue to run on every push to master.

Co-authored-by: Claude <claude@anthropic.com>

* fix: address review feedback on run-shard-tests.sh

- Remove set -e so all test runs execute even if earlier ones fail;
  track failures and exit with error at the end (wiggin77)
- Remove unused top-level COVERAGE_FLAG variable (wiggin77)
- Fix RUN_IDX increment position so report, json, and coverage files
  share the same index (wiggin77)
- Update workflow comment: heavy threshold is 5 min, not 2 min (wiggin77)

Co-authored-by: Claude <claude@anthropic.com>

* style: use node: prefix for built-in fs module in shard-split.js

Co-authored-by: Claude <claude@anthropic.com>

* fix: avoid interpolating file paths into generated shell script

Read shard package lists from files at runtime instead of interpolating
them into the generated script via printf. This prevents theoretical
shell metacharacter injection from directory names, as flagged by
DryRun Security.

Co-authored-by: Claude <claude@anthropic.com>

* fix(ci): rename merged artifact to match server-ci-report glob

The merged artifact was named postgres-server-test-logs-merged which
does not match the *-test-logs pattern in server-ci-report.yml,
causing Postgres test results to be missing from PR/commit reports.

Also pins junit-report-merger to exact version 7.0.0 for supply chain
safety.

Co-authored-by: Claude <claude@anthropic.com>

* fix(ci): pass RACE_MODE env into Docker container

RACE_MODE was set on the host runner but never included in the docker
run --env list. The light-package path worked because the heredoc
expanded on the host, but run-shard-tests.sh reads RACE_MODE at
runtime inside the container where it was unset. This caused heavy
packages (api4, app) to silently lose -race detection.

Co-authored-by: Claude <claude@anthropic.com>

* fix(ci): discover new tests in heavy packages not in timing cache

Tests not present in the timing cache (newly added or renamed) would
not appear in any shard -run regex, causing them to silently skip.
After building items from the cache, run go test -list to discover
current test names and assign any cache-missing tests to shards via
the normal bin-packing algorithm with a small default duration.

Co-authored-by: Claude <claude@anthropic.com>

* fix(ci): add missing line continuation backslash in docker run

The previous --env FIPS_ENABLED line was missing a trailing backslash
after adding --env RACE_MODE, causing docker run to see a truncated
command and fail with "requires at least 1 argument".

Co-authored-by: Claude <claude@anthropic.com>

* fix(ci): add setup-go step for shard test discovery

go test -list in shard-split.js runs on the host runner via execSync,
but Go is only available inside the Docker container. Without this
step, every invocation fails silently and new-test discovery is a
no-op. Adding actions/setup-go before the shard split step ensures
the Go toolchain is available on the host.

Co-authored-by: Claude <claude@anthropic.com>

---------

Co-authored-by: Claude <claude@anthropic.com>

* updated go to version 1.25.8 (#35817)

* updated go to version 1.25.8

* updated gotestsum version to work with go 1.25.8

go 1.25 does not work with indirect tools 0.11 dependency pulled by
gotestsum.

* Use sync.WaitGroup.Go to simplify goroutine creation

Replace the wg.Add(1) + go func() { defer wg.Done() }() pattern with
wg.Go(), which was introduced in Go 1.25.

* pushes fips image on workflow dispatch to allow fips test to run on go version update

* fix new requirements for FIPS compliance imposed on updating to go 1.25.8

* updates openssl symbol check for library shipped with FIPS new versions

go-openssl v2 shipped with FIPS versions starting from 1.25 uses mkcgo to generate
bindings causing symbol names to be different.

* removes temp workflow-dispatch condition

* keep versions out of agents md file

* upgrade golangci-lint (#35845)

* test: clean up channel store data after TestChannelStore (#36066)

TestChannelStore sub-tests create channels, members, and team members
using fake TeamIds and UserIds (model.NewId() for non-existent rows).
These records are left in the database and cause integrity tests
(TestCheck*) running in the same binary to fail their full-table scans.

Register a t.Cleanup on TestChannelStore that purges the affected
tables entirely. A blanket purge is safe: the schema enforces no FK
constraints, and every test suite creates its own data independently.

* Fix command injection in server-test-template workflow (#36080)

Replace the unquoted heredoc (which embedded GITHUB_HEAD_REF into a
generated script) with a cp of the existing run-shard-tests.sh, which
already handles the light-only case. Pass BUILD_NUMBER and TEST_TARGET
as explicit docker env vars instead of interpolating them into script
content.

* fix(ci): restore testname format in sharded gotestsum runs (#36078)

run-shard-tests.sh called gotestsum directly without --format, so it
fell back to gotestsum's default (pkgname) instead of the testname
format set by the Makefile. Pass --format "${GOTESTSUM_FORMAT:-testname}"
to match the Makefile default.

Co-authored-by: Mattermost Build <build@mattermost.com>

* fix(lint): fix pre-existing golangci-lint v2.11.4 issues

Fix misspelling in comment and redundant nil check flagged by the
upgraded linter.

* ci: use golang image for test runner on release-10.11

mattermost-build-server images are not built for release branches.
Use the official golang image which is always available for any Go version.

* ci: use mattermost/mattermost-build-server for release-10.11

The mattermostdevelopment/ images are only built for master.
The production mattermost/ images are built for release branches.

* ci: use mattermost/mattermost-build-server in mmctl test template

The mattermostdevelopment/ images are only built for master.
The production mattermost/ images are built for release branches.

---------

Co-authored-by: Pavel Zeman <pavel.zeman@mattermost.com>
Co-authored-by: Claude <claude@anthropic.com>
Co-authored-by: Carlos Garcia <carlos.garcia@mattermost.com>
Co-authored-by: Mattermost Build <build@mattermost.com>
2026-04-16 09:14:15 -03:00
Mattermost Build
17635d33d8 MM-67049: Fix unauthorized access to public channels in private teams (#34886) (#34994)
Automatic Merge
2026-01-20 13:54:38 +02:00
Mattermost Build
f361e7d75a Automated cherry pick of #34247 (#34257)
Automatic Merge
2025-10-27 12:59:15 +02:00
JG Heithcock
375ce229f4 MM 65084 server-side (#33861) (#34006) (#34044)
* MM 65084 server-side (#33861) (#34006)

Automatic Merge

* Add ConsumeOnce method to store layers

---------

Co-authored-by: Mattermost Build <build@mattermost.com>
2025-10-06 15:20:08 -07:00
Mattermost Build
f21005e359 Add URL validation to LinkMetadata cache and store (#31814) (#33453)
Automatic Merge
2025-07-17 20:58:42 +03:00
Jesse Hallam
ebe03c1d45 Channel Store: No SELECT * (#32167)
* Replace SELECT * with explicit column lists in channel store

Migrates channel_store.go away from SELECT * patterns to explicit column
lists for better performance, maintainability, and schema safety.

- Replace GetPinnedPosts raw SQL with query builder using postSliceColumns()
- Replace "cc.*" in group channel search with channelSliceColumns()
- Replace GetChannelsBatchForIndexing raw SQL with query builder
- Replace channel member and team queries with respective column helpers
- Use SelectBuilder helper instead of manual ToSql() calls

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>

* Replace SELECT * with COUNT(*) in user_test.go

Replaces unnecessary SELECT * queries with SELECT COUNT(*) in
TestPermanentDeleteUser bot count verification. Only needs to check
the count of bots, not retrieve full bot records.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>

---------

Co-authored-by: Claude <noreply@anthropic.com>
2025-07-02 15:35:54 +00:00
Christopher Poile
9b1e03a6b8 [MM-63557] mmctl: Add compliance export create cmd (#30594)
* Refactor job retrieval to support multiple statuses & multiple types

- Updated job retrieval functions to handle multiple job statuses.
- Renamed `GetJobsByTypeAndStatus` to `GetJobsByTypesAndStatuses` for consistency across the codebase.
- Adjusted related function signatures and implementations in the job store and retry layer to accommodate the new method.
- Updated tests to reflect changes in job retrieval logic and ensure proper functionality.

* Add compliance export create command and tests

- Introduced `ComplianceExportCreateCmd` to facilitate the creation of compliance export jobs with options for date, start, and end timestamps.
- Added unit tests for the new command, covering various scenarios including valid and invalid inputs.
- Updated documentation to include usage examples and options for the new command.
- Enhanced existing tests to ensure proper functionality of compliance export job handling.

* update docs

* update tests for new logic

* Refactor message export job tests to use DefaultPreviousJobPageSize

- Updated all test cases in worker_test.go to replace hardcoded page size of 100 with DefaultPreviousJobPageSize for consistency.
- Adjusted the worker.go file to define DefaultPreviousJobPageSize and use it in job retrieval logic.
- Ensured that the changes maintain the functionality of job data initialization and retrieval tests.

* PR comments

* PR comments, simplifications, clarifications, formatting

* prefer hypen over underscore in command names

* merge conflict

* update mmctl docs
2025-06-24 21:38:30 +00:00
Jesse Hallam
dcc72c4c61 MM-63728: simplify category store with graphql gone (#30848)
* move category permissions to api

In https://github.com/mattermost/mattermost/pull/21038, we changed the
behaviour of the channel category store to filter out deleted teams and
teams for which the user was not a member. This was necessary in part
due to querying multiple teams via GraphQL.

With GraphQL no longer supported, let's move the permissions to the
API instead and remove the `JOIN` to filter out teams in the store.

Note that we /don't/ prevent access to deleted teams. For better or
worse, deleted teams remain largely accessible via other API endpoints
anyway.

* remove ExcludeTeam / GraphQL support

As part of https://github.com/mattermost/mattermost/pull/20353, we added
`ExcludeTeam` and the associated logic to support a GraphQL API.

With GraphQL no longer supported, let's simplify this logic and remove
the filtering and associated complexity.

* Fix shadow variable declaration in channel_store_categories.go

Fixed golangci-lint error by reusing existing err variable rather than shadowing it.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>

* fix build issue

* Remove SidebarCategorySearchOpts and simplify API to use teamID string

Per code review feedback, this change removes the SidebarCategorySearchOpts
struct entirely since the Type field was never used in the store implementation.
All methods now accept a simple teamID string parameter instead of the struct,
which simplifies the API and makes the code clearer.

Changes:
- Remove SidebarCategorySearchOpts struct from store.go
- Update CreateInitialSidebarCategories and GetSidebarCategories signatures
- Update all implementations (sqlstore, retrylayer, timerlayer, mocks)
- Update all callers to pass teamID string directly
- Clean up unused imports

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>

---------

Co-authored-by: Claude <noreply@anthropic.com>
2025-06-20 00:48:12 +00:00
Harrison Healey
a3f60f797b MM-63725 Populate multiple sidebar categories at once whenever possible (#31064)
* Remove redundant sidebar tests from TestChannelStore

* MM-63725 Refactor to split out getOrphanedSidebarChannels

* MM-63725 Populate multiple sidebar categories at once whenever possible

* Fix shadowing
2025-06-19 10:09:36 -04:00
Ben Schumacher
cfc1503d62 [MM-63355] Add AuthData to mmctl user search output (#30478) 2025-06-19 11:52:16 +02:00
Ben Schumacher
04a60b6609 [MM-57693] Add schema dump to Support Packet (#31162)
Co-authored-by: Claude <noreply@anthropic.com>
2025-06-19 11:33:55 +02:00
Daniel Espino García
fc14fa0e87 Fix getAllChannels api not returning DMs/GMs (#31362)
Automatic Merge
2025-06-17 09:16:06 +03:00
Agniva De Sarker
761bc7549b [AI assisted] MM-64298: Process setting status offline in batches (#31065)
When a user disconnects from the hub, we would spawn off a goroutine
which would make a cluster request, and then update the user status
as offline in the DB.

This was another case of unbounded concurrency where the number of
goroutines spawned was user controlled. Therefore, we would see
a clear spike in DB connections on master when a lot of users
would suddenly disconnect.

To fix this, we implement concurrency control in two areas:
1. In making the cluster request. We implement a counting semaphore
per-hub to avoid making unbounded cluster requests.
2. We use a buffered channel with a periodic flusher to process
status updates.

We also add a new store method to upsert multiple statuses
in a single query. The statusUpdateThreshold is set to 32, which means
no more than 32 rows will be upserted at one time, keeping the
SQL query load reasonable.

https://mattermost.atlassian.net/browse/MM-64298

```release-note
We improve DB connection spikes on user disconnect
by processing status updates in batches.
```
2025-06-17 09:20:34 +05:30
catalintomai
85391de22a MM-57326: [Shared Channels] Message priority, acknowledgement and persistent notifications need to be synced (#30736) 2025-06-16 02:30:21 +02:00
catalintomai
fa1c77d9b0 MM-52600: [Shared Channels] Shared channels do not sync channel membership (#30976) 2025-06-15 10:07:56 +02:00
catalintomai
c46ed6c681 MM-62751: [Shared Channels] Allow remote users to be discoverable in the create DM/GM modal (#30918) 2025-06-13 16:51:12 +02:00
Miguel de la Cruz
43018759e5 Adds support for GMs in shared channels (#31403)
* Adds support for GMs in shared channels

* Fix linter

* Remove creatorID from slack call

---------

Co-authored-by: Miguel de la Cruz <miguel@ctrlz.es>
2025-06-13 10:43:30 +00:00
Jesse Hallam
e6d8bf5835 Upgrade Go to 1.24.3 (#31220)
* Upgrade Go to 1.24.3

Updates the following files:
- server/.go-version: 1.23.9 → 1.24.3
- server/build/Dockerfile.buildenv: golang:1.23.9-bullseye → golang:1.24.3-bullseye
- server/go.mod: go 1.23.0 → go 1.24.3, toolchain go1.23.9 → go1.24.3
- server/public/go.mod: go 1.23.0 → go 1.24.3, toolchain go1.23.9 → go1.24.3

Also fixes non-constant format string errors introduced by Go 1.24.3's stricter format string checking:
- Added response() helper function in slashcommands/util.go for simple string responses
- Removed unused responsef() function from slashcommands/util.go
- Replaced responsef() with response() for translated strings that don't need formatting
- Fixed fmt.Errorf and fmt.Fprintf calls to use proper format verbs instead of string concatenation
- Updated marketplace buildURL to handle format strings conditionally

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>

* Update generated mocks for Go 1.24.3

Regenerated mocks using mockery v2.53.4 to ensure compatibility with Go 1.24.3.
This addresses mock generation failures that occurred with the Go upgrade.

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>

* Update to bookworm and fix non-existent sha

Signed-off-by: Stavros Foteinopoulos <stafot@gmail.com>

* fix non-constant format string

---------

Signed-off-by: Stavros Foteinopoulos <stafot@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Stavros Foteinopoulos <stafot@gmail.com>
2025-06-10 15:04:57 -03:00
Ben Cooke
aac34f6db4 [MM-64360] Data retention: optionally preserve pinned posts (#31165)
* add new config to preserve pinned posts during data retention

* more graceful error if the pinned post was a reply in a deleted thread
2025-06-09 15:41:07 -04:00
Ibrahim Serdar Acikgoz
6f26ad5cec [ABAC - Table Editor] Improvements on table editor and review feedback (#31125)
* reflect review comments

* update table editor

* adjust test limits

* reflect review comments

* MM-64376

* resolve conflicts

* address review comments

* fix merge conflict error
2025-06-01 12:05:57 +02:00
Devin Binnie
e1c0c57d31 [MM-63933] Don't include channels from an archived team unless specified in GetChannelsForUser (#31163)
* [MM-63933] Don't include channels from an archived team unless specified in GetChannelsForUser

* PR feedback

---------

Co-authored-by: Mattermost Build <build@mattermost.com>
2025-05-29 13:35:38 +00:00
Jesse Hallam
7a61d498ea MM-64438: skip flaky: testGetNthRecentPostTime (#31223) 2025-05-28 17:47:43 -03:00
catalintomai
e6ed3436fb MM-54023: [Shared Channels] Filter out system posts for channel update info that is not synced (#30735)
* initial checkin

* simplify tests, logic

* fix tests

* remove unneeded test

* fix spacing

* updates

* update logic

* fix logic

* move filtering to DB

* remove comment

---------

Co-authored-by: Catalin Tomai <catalintomai@catalins-macbook-pro-2.home>
2025-05-27 13:15:49 +00:00
Miguel de la Cruz
e51ea025db Deletes CPA values on CPA field type change (#31122)
* Deletes CPA values on CPA field type change

* Fix error method name reference

* Cleans the state when a CPA field's type is updated

* Fix types

* Fix linter

* Webapp no longer makes a decision on the change and server sents a flag in the WS message

* Fix linter

---------

Co-authored-by: Miguel de la Cruz <miguel@ctrlz.es>
2025-05-27 13:38:05 +02:00
Jesse Hallam
31a8047973 Disable morph logging during TestMain (#30948)
* rm "No TEST_DATABASE... override" log message

Let's only log if this value is actually overridden.

* rm "(Created|Dropped) temporary database" message

* only log "Pinging SQL" on subsequent attempts

* disable morph logging from TestMain

* Fix style issues in store test files

- Add missing parameter to migrate() function calls in tests
- Remove unused log function in settings.go
- Fix formatting with go fmt

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>

* within sqlstore, use "enable" vs "disable" for clarity

* remove trailing newline from morph logs

---------

Co-authored-by: Claude <noreply@anthropic.com>
2025-05-21 17:31:18 +00:00
Harrison Healey
b70f1d859d MM-63923/MM-63924/MM-63925 Prevent deadlocks and constraint errors in UpdateSidebarCategories (#30965)
* MM-63925 Remove most nested transactions from channel_store_categories.go

There's one place which still has a nested transaction in
CreateInitialSidebarCategories, but that's because it's calling out to a
different part of the store. The only way to avoid that would be to
break the extraction like UpdateSidebarCategories does to update
preferences, but I chose not to follow that pattern here and leave it
as-is.

* MM-63923 Prevent deadlocks caused by updating multiple categories in a different order

* MM-63923 Prevent deadlocks while deleting from SidebarChannels

This could also have been resolved by sorting the categories, but
combining the queries seems a bit more elegant.

* MM-63924 Ensure adding SidebarChannels rows is idempotent

* Add additional test to cause deadlocks

* Prevent channels from appearing in a single category multiple times

* Other review feedback
2025-05-20 16:02:32 -04:00
Ben Schumacher
c2d08b7540 [MM-63772] Add LDAP setting to re-add removed members (#30787) 2025-05-20 11:15:25 +02:00
Arya Khochare
f54d5b41c1 [MM-58163] Jobs paging/offset refactor (#30343)
Co-authored-by: Ben Schumacher <ben.schumacher@mattermost.com>
2025-05-19 12:31:25 +02:00
Ibrahim Serdar Acikgoz
a344b3225b [MM-61756] Attribute Based Access Control - Phase 1 (#30785)
Attribute Based Access Control - Base
* MM-63662

* MM-63919

* MM-63954

* MM-63955 

* MM-63425

* MM-63426

* MM-63458

* MM-63459

* MM-63603

* MM-63845

* MM-64146

* MM-64199

* MM-64201

* MM-64233

* MM-64247

* MM-64268

---------

Co-authored-by: Harshil Sharma <harshilsharma63@gmail.com>
Co-authored-by: Pablo Andrés Vélez Vidal <pablovv2012@gmail.com>
Co-authored-by: abhijit-singh <abhijitsingh0702@gmail.com>
Co-authored-by: Harrison Healey <harrisonmhealey@gmail.com>
2025-05-15 11:33:08 +02:00
Miguel de la Cruz
6ab6a008e6 Adds the capability to retrieve a property field by name (#30859)
* Adds the capability to retrieve a property field by name

Allows to retrieve a property field by name and groupID. As the name
is only unique within the context of a group, and we can have multiple
fields with the same name in the store, for this method the groupID is
directly included in the query instead of being an optional field.

* Adds the targetID parameter to correctly filter fields

* Ensure the method only retrieves non-deleted fields

---------

Co-authored-by: Miguel de la Cruz <miguel@ctrlz.es>
Co-authored-by: Mattermost Build <build@mattermost.com>
2025-05-13 12:45:35 +02:00
Agniva De Sarker
4803892492 MM-56906: Remove redundant calls on team switch (#30771)
On page load, we load ALL channels and channel members from all teams.
But then, on team_switch, we would again load channels and channel
members from that team. This was redundant and mainly kept
because previously the websocket events were considered unreliable.

Now with reliable websockets, and client-side pings, we can detect
broken connections faster and recover without loss.

Additionally, the getAllChannelMembers call would page through
all responses on the client side. This was inefficient and incur
extra latency. To optimize for this, we introduce server-side
streaming of the full response if page is set to -1.

This optimizes the intial response as well.

https://mattermost.atlassian.net/browse/MM-56906

```release-note
Optimize team switch operation by removing calls to get channels
and channel members.
```


Co-authored-by: Mattermost Build <build@mattermost.com>
2025-05-12 20:05:46 +05:30
Jesse Hallam
e1f47e22e7 MM-62158: group store no SELECT * (Part 1) (#30276)
* improved test coverage

* initial pass on removing SELECT * from group store
2025-05-01 09:39:05 -03:00
Jesse Hallam
011f179831 avoid SELECT * in preference store (#30835) 2025-04-23 15:06:04 -04:00
Jesse Hallam
981d1d869a avoid SELECT * in channel member history store (#30828) 2025-04-22 15:33:32 -04:00
Agniva De Sarker
d8dbb6cc22 MM-56548: [AI assisted]Add support for incremental thread loading using UpdateAt timestamp (#30486)
Every time we load the RHS, we used to load the FULL thread always. Although
the actual ThreadViewer React component is virtualized, and the server side
API call is paginated, we still went through all the pages, to get the full
thread and passed it on to the ThreadViewer. This would be for first loads,
and subsequent loads of the same thread.

This was a bug originally, but then it was a necessity after we applied websocket event scope because
now we won't get emoji reactions of a thread if the user is not on the thread.

To fix that, we enhance the thread loading functionality by adding support for fetching
thread updates based on the UpdateAt timestamp. Now, for subsequent loads,
we only get the changed posts in a thread. The implementation:

- Adds new API parameters: fromUpdateAt and updatesOnly to the GetPostThread endpoint
- Updates database queries to support sorting and filtering by UpdateAt
- Implements thread state management to track the last update timestamp
- Adds client-side support to use incremental loading for improved performance
- Ensures proper validation for parameter combinations and error handling

This change enables more efficient thread loading, particularly for long threads
with frequent updates, by only fetching posts that have been updated since the
last view.

Caveats: For delta updates, the SQL query won't use the best index possible
because we have an index for (CreateAt, Id), but no index for (UpdateAt, Id).
However, from my tests, it is not as bad as it looks:

```
[loadtest] # EXPLAIN (ANALYZE, BUFFERS) SELECT * FROM Posts WHERE Posts.DeleteAt = 0 AND Posts.RootId = 'qbr5gctu9iyg8c36hpcq6f3w8e' AND Posts.UpdateAt > 1623445795824 ORDER BY UpdateAt ASC, Id ASC LIMIT 61;
                                                                   QUERY PLAN
-------------------------------------------------------------------------------------------------------------------------------------------------
 Limit  (cost=8.31..8.31 rows=1 width=216) (actual time=0.047..0.049 rows=0 loops=1)
   Buffers: shared hit=2
   ->  Sort  (cost=8.31..8.31 rows=1 width=216) (actual time=0.044..0.045 rows=0 loops=1)
         Sort Key: updateat, id
         Sort Method: quicksort  Memory: 25kB
         Buffers: shared hit=2
         ->  Index Scan using idx_posts_root_id_delete_at on posts  (cost=0.28..8.30 rows=1 width=216) (actual time=0.031..0.032 rows=0 loops=1)
               Index Cond: (((rootid)::text = 'qbr5gctu9iyg8c36hpcq6f3w8e'::text) AND (deleteat = 0))
               Filter: (updateat > '1623445795824'::bigint)
               Buffers: shared hit=2
 Planning:
   Buffers: shared hit=3
 Planning Time: 0.508 ms
 Execution Time: 0.106 ms
(14 rows)
```

We still get an index scan with index cond. Although there's a filter element, but atleast we get the whole thread with the index.
My thinking is that while the whole thread might be large, but after that, updates on a thread should be incremental.
Therefore, we should be okay without adding yet another index on the posts table.

This is just the first step in what could be potentially improved further.

1. We shouldn't even be loading the full thread always. But rather let the virtualized viewer
load more posts on demand.
2. If a post has been just reacted to, then we need not send the whole post down, but just the
reaction. This further saves bandwidth.

https://mattermost.atlassian.net/browse/MM-56548

TBD: Add load-test coverage to update the thread loading code

```release-note
NONE
```
---------

Co-authored-by: Mattermost Build <build@mattermost.com>
2025-04-22 10:43:13 +05:30
Jesse Hallam
b59f10cbbd MM-62158: Group Store, explicit aliases (#30741) 2025-04-17 17:37:28 -03:00
Miguel de la Cruz
ca9fd45408 Adds a mechanism to delete CPA values for a given user (#30330)
* Adds a mechanism to delete CPA values for a given user

This requires improving the Property Value service to enable delete
all values for a given target, so a new method was created that allows
to delete filtering by targetType and targetID (required) and
optionally for a specific groupID in case the caller wants to affect
all values for a target (useful in case you remove a post for example
and want to delete all values pointing to that post regardless of the
feature they belong to) or only those that belong to a specific
feature.

* Fix property value tests

* Fix after merge and update method name

* Fix linter

---------

Co-authored-by: Miguel de la Cruz <miguel@ctrlz.es>
Co-authored-by: Mattermost Build <build@mattermost.com>
2025-04-10 19:22:05 +02:00
Ibrahim Serdar Acikgoz
10b1f4c5ac [MM-63428] add access control policy store (#30597) 2025-04-02 13:39:28 +02:00
Ben Cooke
ce9632cca3 MM-63311 (#30387)
* allow reference group changes
2025-03-31 15:49:55 -04:00
Miguel de la Cruz
1ca6f6d6fb Adds a groupID filter to the property service methods (#30420)
* Adds a groupID filter to the property service methods

This allows the property service caller to directly ensure that a
given call is only going to affect a field or value that belongs to a
given group, instead of (for example) retrieving a property value
before deleting it by id to ensure that the value belongs to a
specific property group. The groupID filter is optional and has no
effect if called with the empty string value.

The changes also remove references to input sanitization on trimming
the whitespace for the CPA field names and validate at the API level
the input for the field patch endpoint.

* Fix linter

---------

Co-authored-by: Miguel de la Cruz <miguel@ctrlz.es>
Co-authored-by: Mattermost Build <build@mattermost.com>
2025-03-31 16:51:16 +00:00
Jesse Hallam
eb851684e9 MM-62162: Replace SELECT * in user_access_token_store.go (#30421)
* MM-62162: Replace SELECT * in user_access_token_store.go

- Replaced all SELECT * queries with explicit column selection
- Used query builder instead of raw SQL strings for all queries
- Added reusable userAccessTokensSelectQuery in the store constructor
- Added comprehensive test for pagination and IsActive flag

🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>

* Fix code formatting in user_access_token_store test file

🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>

* MM-62162: Update UserAccessToken Search method to use explicit column names

- Fixed "ambiguous column" errors by explicitly naming columns with table qualifiers
- Added test for the Search functionality

🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>

* MM-62162: Update UserAccessToken Search method to use explicit column names

- Fixed "ambiguous column" errors by explicitly naming columns with table qualifiers
- Maintained exact semantics of original query's LIKE filters
- Refactored to use query builder instead of raw SQL

🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>

* MM-62162: Update UserAccessToken Search method to use reusable query

- Modified userAccessTokensSelectQuery to use fully qualified column names
- Updated Search method to use the existing query builder
- Fixed "ambiguous column" errors by using full column qualifiers

🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>

* MM-62162: Remove unnecessary comment in user_access_token_store.go

Removed redundant comment in Search method explaining the use of qualified column names, as the code is self-documenting.

🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>

* MM-62162: Use t.Cleanup() for token cleanup in userAccessTokenPagination test

* MM-62162: Handle error in Cleanup function

* linting

---------

Co-authored-by: Mattermost Build <build@mattermost.com>
2025-03-31 13:47:43 -03:00
Chris Gibson
65256843f9 [GH-30056] Refactor SELECT statements in compliance_store.go (#30124)
* Change queries to use builder

* extract to tableSelectQuery

* Apply suggestions from code review

* update tests to check errors, lengths

* linting

* leverage s.toReserveCase

---------

Co-authored-by: Jesse Hallam <jesse@thehallams.ca>
Co-authored-by: Jesse Hallam <jesse.hallam@gmail.com>
Co-authored-by: Mattermost Build <build@mattermost.com>
2025-03-26 10:23:08 -03:00
Agniva De Sarker
8faa8b2e56 MM-63545: Fix post reminder off-by-one error (#30553)
We used < which meant we missed sending
reminders created for that timestamp, because
the job runs on that exact time.

Using <= to fix that.

https://mattermost.atlassian.net/browse/MM-63545
```release-note
NONE
```
2025-03-25 11:09:42 +05:30
Ben Schumacher
9b5d8d52bf [MM-62427] Add message attachments validation (#30180)
* Add message attachments validation

* Add props validation

* Validate slack attachment fields

* Update tests and library usage

* Improve interactive dialog error for length checks

* Allow predefined colors for slack attachments

* Fix TestPostAction

* Use const for data source

* Add tests

* Cleanup unused props

* Add happy path tests

* lint fixes

* Add validation for PostActionOptions
2025-03-20 12:53:50 +01:00
Christopher Poile
c049748b88 [MM-63314] Fix ClaimJob in HA environments (#30383)
* ClaimJob now returns newly claimed job

* internal code affected by change

* test changes required

* two branches: for mysql, use transaction; for postgres, use returning

* two branches: for mysql, use transaction; for postgres, use returning

* use same millis value for LastActivityAt and StartAt

* blank commit

---------

Co-authored-by: Mattermost Build <build@mattermost.com>
2025-03-14 10:24:26 -04:00
Ben Cooke
ccd8a60168 Plugin groups (#30320)
* add new pluginapi methods

* SAML login hook

* set ReAddRemovedMembers to true for plugin groups

* change to DoLogin signature for SAML
2025-03-13 12:00:15 -04:00
Arya Khochare
4fbfd84957 [MM-61691] Deleting drafts when permanently deleting a user (#30233)
* deleting drafts on permanently deleting user

* verified count of drafts before deleting

* i18n fix

* use ExecBuilder

* changed error message to specify user

---------

Co-authored-by: Mattermost Build <build@mattermost.com>
Co-authored-by: Harshil Sharma <harshilsharma63@gmail.com>
2025-03-13 19:19:38 +05:30
Jesse Hallam
9fc83f24b5 MM-62161: Replace SELECT * in session_store.go (#30422)
* MM-62161: Replace SELECT * in session_store.go

- Replaced all SELECT * queries with explicit column selection
- Used query builder instead of raw SQL strings where possible
- Added reusable sessionSelectQuery in the store constructor
- Added comprehensive test for GetSessionsWithActiveDeviceIds

🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>

* Fix variable shadowing issues in session_store.go

Resolved variable shadowing by reassigning to the existing error variables instead of declaring new ones in scoped blocks.

🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>

* Fix code formatting in session_store test file

🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
2025-03-12 19:35:33 +00:00
Miguel de la Cruz
c5e6d9f570 Updates the property service and store method signatures (#30103)
* Updates the property service and store method signatures

Getters can now receive a `groupID` that narrows down the query if
present, so it's not necessary to check for the group ID on the
returning values from the outside layers.

The Search methods now receive the `groupID` and the `targetID`
explicitly as parameters, incentivizing the use of the indexes that
the underlying tables have on the searches.

* Fix tests

---------

Co-authored-by: Miguel de la Cruz <miguel@ctrlz.es>
Co-authored-by: Mattermost Build <build@mattermost.com>
2025-03-10 14:26:47 +00:00
Harshil Sharma
6e738f489f Channel banner sql migrations (#30274)
* Adde MySQL and Postgres migrations

* Replaced select * with column names

* removed all * from channel SQL store

* cleanup

* Fixed a duplicate column

* cleanup

* Added migrations and store support

* WIP

* used channelname slice in a missed place

* Handled patch

* Added app level tests

* Added API layer tests

* Added API layer tests

* WIP

* converted to query builder

* cleanupo

* added not null and default constraints

* Fixed test

* fixed file name

* review fixes

* review fixes

* updated migration file

* fixed text

* Review fixes
2025-02-25 14:52:15 +05:30