With some neat generics, I was able to refactor
the scanning to a util function. I used it to
refactor 3 places and also removed an unnecessary method.
Claude was quite good here.
https://mattermost.atlassian.net/browse/MM-62755
```release-note
NONE
```
INSERT IGNORE will ignore ALL errors in the INSERT statement.
This is not what was intended. The right way is to do a
redundant update on duplicate key.
It's not great, but that's how MySQL wants us to do it.
https://mattermost.atlassian.net/browse/MM-63878
```release-note
NONE
```
* avoid SELECT * in command stores
* Define column names once and reuse as columns list in selects and inserts
* Define column names once and reuse with ExecBuilder
Every time we load the RHS, we used to load the FULL thread always. Although
the actual ThreadViewer React component is virtualized, and the server side
API call is paginated, we still went through all the pages, to get the full
thread and passed it on to the ThreadViewer. This would be for first loads,
and subsequent loads of the same thread.
This was a bug originally, but then it was a necessity after we applied websocket event scope because
now we won't get emoji reactions of a thread if the user is not on the thread.
To fix that, we enhance the thread loading functionality by adding support for fetching
thread updates based on the UpdateAt timestamp. Now, for subsequent loads,
we only get the changed posts in a thread. The implementation:
- Adds new API parameters: fromUpdateAt and updatesOnly to the GetPostThread endpoint
- Updates database queries to support sorting and filtering by UpdateAt
- Implements thread state management to track the last update timestamp
- Adds client-side support to use incremental loading for improved performance
- Ensures proper validation for parameter combinations and error handling
This change enables more efficient thread loading, particularly for long threads
with frequent updates, by only fetching posts that have been updated since the
last view.
Caveats: For delta updates, the SQL query won't use the best index possible
because we have an index for (CreateAt, Id), but no index for (UpdateAt, Id).
However, from my tests, it is not as bad as it looks:
```
[loadtest] # EXPLAIN (ANALYZE, BUFFERS) SELECT * FROM Posts WHERE Posts.DeleteAt = 0 AND Posts.RootId = 'qbr5gctu9iyg8c36hpcq6f3w8e' AND Posts.UpdateAt > 1623445795824 ORDER BY UpdateAt ASC, Id ASC LIMIT 61;
QUERY PLAN
-------------------------------------------------------------------------------------------------------------------------------------------------
Limit (cost=8.31..8.31 rows=1 width=216) (actual time=0.047..0.049 rows=0 loops=1)
Buffers: shared hit=2
-> Sort (cost=8.31..8.31 rows=1 width=216) (actual time=0.044..0.045 rows=0 loops=1)
Sort Key: updateat, id
Sort Method: quicksort Memory: 25kB
Buffers: shared hit=2
-> Index Scan using idx_posts_root_id_delete_at on posts (cost=0.28..8.30 rows=1 width=216) (actual time=0.031..0.032 rows=0 loops=1)
Index Cond: (((rootid)::text = 'qbr5gctu9iyg8c36hpcq6f3w8e'::text) AND (deleteat = 0))
Filter: (updateat > '1623445795824'::bigint)
Buffers: shared hit=2
Planning:
Buffers: shared hit=3
Planning Time: 0.508 ms
Execution Time: 0.106 ms
(14 rows)
```
We still get an index scan with index cond. Although there's a filter element, but atleast we get the whole thread with the index.
My thinking is that while the whole thread might be large, but after that, updates on a thread should be incremental.
Therefore, we should be okay without adding yet another index on the posts table.
This is just the first step in what could be potentially improved further.
1. We shouldn't even be loading the full thread always. But rather let the virtualized viewer
load more posts on demand.
2. If a post has been just reacted to, then we need not send the whole post down, but just the
reaction. This further saves bandwidth.
https://mattermost.atlassian.net/browse/MM-56548
TBD: Add load-test coverage to update the thread loading code
```release-note
NONE
```
---------
Co-authored-by: Mattermost Build <build@mattermost.com>
It was a mistake to return ANY error as NewErrNotFound. Because this
can be a DB timeout, or any other network error.
Returning NewErrNotFound would categorize it as 404, eventually printing
the error in the DEBUG level. Changing it to normal error fixes this.
https://mattermost.atlassian.net/browse/MM-63739
```release-note
NONE
```
* MM-62156: Avoid SELECT * in retention_policy_store.go
- Modified subQueryIN function to use specific column name instead of SELECT *
- Improved code comments to explain the change
- Maintained same functionality while avoiding SELECT *
Fixes: https://mattermost.atlassian.net/browse/MM-62156🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
* simplify subQueryIN comments
* inline part of subQueryIN for greater clarity
---------
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Mattermost Build <build@mattermost.com>
* Adds a mechanism to delete CPA values for a given user
This requires improving the Property Value service to enable delete
all values for a given target, so a new method was created that allows
to delete filtering by targetType and targetID (required) and
optionally for a specific groupID in case the caller wants to affect
all values for a target (useful in case you remove a post for example
and want to delete all values pointing to that post regardless of the
feature they belong to) or only those that belong to a specific
feature.
* Fix property value tests
* Fix after merge and update method name
* Fix linter
---------
Co-authored-by: Miguel de la Cruz <miguel@ctrlz.es>
Co-authored-by: Mattermost Build <build@mattermost.com>
* MM-62157: Avoid SELECT * in user_store.go
- Replace raw SQL query "SELECT * FROM Users" with query builder in Update method
- Replace raw SQL query "SELECT * from Users" with query builder in ClearAllCustomRoleAssignments
- Change Select("*") to Select("data.*") in GetUserReport method
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
* MM-62157: Avoid more instances of SELECT * in user_store.go
- Replace raw SQL query "Users.*, cm.ChannelId" with usersQuery in getUsersInGroupChannels
- Replace "Users.*" with getUsersColumns() in GetUserReport method
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
---------
Co-authored-by: Claude <noreply@anthropic.com>
* Adds a groupID filter to the property service methods
This allows the property service caller to directly ensure that a
given call is only going to affect a field or value that belongs to a
given group, instead of (for example) retrieving a property value
before deleting it by id to ensure that the value belongs to a
specific property group. The groupID filter is optional and has no
effect if called with the empty string value.
The changes also remove references to input sanitization on trimming
the whitespace for the CPA field names and validate at the API level
the input for the field patch endpoint.
* Fix linter
---------
Co-authored-by: Miguel de la Cruz <miguel@ctrlz.es>
Co-authored-by: Mattermost Build <build@mattermost.com>
* MM-62162: Replace SELECT * in user_access_token_store.go
- Replaced all SELECT * queries with explicit column selection
- Used query builder instead of raw SQL strings for all queries
- Added reusable userAccessTokensSelectQuery in the store constructor
- Added comprehensive test for pagination and IsActive flag
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
* Fix code formatting in user_access_token_store test file
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
* MM-62162: Update UserAccessToken Search method to use explicit column names
- Fixed "ambiguous column" errors by explicitly naming columns with table qualifiers
- Added test for the Search functionality
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
* MM-62162: Update UserAccessToken Search method to use explicit column names
- Fixed "ambiguous column" errors by explicitly naming columns with table qualifiers
- Maintained exact semantics of original query's LIKE filters
- Refactored to use query builder instead of raw SQL
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
* MM-62162: Update UserAccessToken Search method to use reusable query
- Modified userAccessTokensSelectQuery to use fully qualified column names
- Updated Search method to use the existing query builder
- Fixed "ambiguous column" errors by using full column qualifiers
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
* MM-62162: Remove unnecessary comment in user_access_token_store.go
Removed redundant comment in Search method explaining the use of qualified column names, as the code is self-documenting.
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
* MM-62162: Use t.Cleanup() for token cleanup in userAccessTokenPagination test
* MM-62162: Handle error in Cleanup function
* linting
---------
Co-authored-by: Mattermost Build <build@mattermost.com>
* MM-62154: Avoid SELECT * in webhook_store.go
- Defined column list functions for both IncomingWebhooks and OutgoingWebhooks tables
- Converted all uses of SELECT * to use the column list functions
- Used the QueryBuilder pattern for SQL queries for better consistency and readability
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
* MM-62154: Improve webhook store implementation using query builders
- Refactored webhook store to follow session_store.go pattern
- Added select query builders in the constructor
- Used pre-built queries throughout the store implementation for better maintainability and performance
- Removed redundant incomingWebhookColumns() and outgoingWebhookColumns() functions
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
* MM-62154: Fix whitespace style issues in webhook_store.go
- Ran gofmt to remove trailing whitespace
- Ensures consistent code formatting
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
* MM-62154: Remove unnecessary int(0) casting
- Simplified DeleteAt=0 condition by removing redundant int casting
- Maintains same behavior while making code cleaner
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
---------
Co-authored-by: Mattermost Build <build@mattermost.com>
* MM-62160: Avoid SELECT * in user_terms_of_service.go
- Added userTermsOfServiceSelectQuery in the constructor
- Replaced raw SQL query with query builder pattern
- Used explicit column selection instead of SELECT *
- Made the implementation more resilient to schema changes
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
* MM-62160: Also avoid SELECT * in terms_of_service_store.go
- Added termsOfServiceSelectQuery field to store struct
- Replaced SELECT * with explicit column selection
- Updated GetLatest and Get methods to use the query builder pattern
- Made the implementation more resilient to schema changes
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
---------
Co-authored-by: Mattermost Build <build@mattermost.com>
* MM-62155: Avoid SELECT * in team_store.go
- Added explicit column lists for all SELECT queries
- Created teamSelectQuery for reused queries
- Replaced raw SQL queries with query builder pattern
* Refactor getTeamMembersWithSchemeSelectQuery to use teamMembersQuery builder
Instead of duplicating TeamMembers columns, use the existing builder to avoid redundancy.
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
* Remove redundant comments from getTeamMembersWithSchemeSelectQuery
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
---------
Co-authored-by: Claude <noreply@anthropic.com>
We used < which meant we missed sending
reminders created for that timestamp, because
the job runs on that exact time.
Using <= to fix that.
https://mattermost.atlassian.net/browse/MM-63545
```release-note
NONE
```
* MM-62159: Avoid SELECT * in channel_store_categories.go
- Added sidebarCategorySelectQuery field to SqlChannelStore struct
- Replaced SELECT * with explicit column selection in GetSidebarCategory and getSidebarCategoriesT functions
- Updated raw SQL query in addChannelToFavoritesCategoryT to use explicit column selection
- Made the implementation more resilient to schema changes
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
* use sc alias, simplify
* MM-62159: Fix ambiguous ID column in sidebar category queries
- Modified sidebarCategorySelectQuery initialization to explicitly use "sc" table alias for all columns
- Prevents "Column 'Id' in field list is ambiguous" error when joining with other tables
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
* MM-62159: Consistently use table aliases in sidebar categories queries
- Added 'sc' table alias to all sidebar category queries
- Ensures consistency and avoids ambiguous column errors in future joins
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
* MM-62159: Replace 'sc' alias with full 'SidebarCategories' table name
- Replaced all instances of the 'sc' alias with the full table name 'SidebarCategories'
- Updated the SidebarCategories query builder to use the full table name
- Removed commented-out debug printf statement
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
When a search term is a fully quoted string, we want to avoid
stemming if it is supported by the search backend.
For Postgres, it does provide a feature by which if we use the
"simple" search config, then no stemming is performed and an exact
match with the word is done without having to resort
to LIKE queries.
Unfortunately, for ES/OS this is not an option because
the message field is a text field, which means ES/OS will analyze it,
stem it and store it in its root form. Therefore, no exact match
can be possible with ES/OS.
The only solution here is to have yet another keyword field
for message which will store it in its raw form. But this
will effectively double the disk storage for post indices
and not a good design choice.
Ref: https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-term-query.html#avoid-term-query-text-fieldshttps://mattermost.atlassian.net/browse/MM-60790
```release-note
NONE
```
* re-arrange the tests to run only on DB
```release-note
NONE
```
---------
Co-authored-by: Mattermost Build <build@mattermost.com>
* Update Go version to v1.23.7
* Bump golangci-lint to a version supporting Go 1.23
* Fix golangci-lint warnings
Several rules from gosimple, revive and staticcheck linters were
failing:
- Redefinition of built-in identifiers (max, min, new, recover...)
- Use of printf-like functions with simple strings
- Check for nil slices, when len already takes it into account
* Trigger Build
* Trigger Build
---------
Co-authored-by: Mattermost Build <build@mattermost.com>
* ClaimJob now returns newly claimed job
* internal code affected by change
* test changes required
* two branches: for mysql, use transaction; for postgres, use returning
* two branches: for mysql, use transaction; for postgres, use returning
* use same millis value for LastActivityAt and StartAt
* blank commit
---------
Co-authored-by: Mattermost Build <build@mattermost.com>
* MM-62161: Replace SELECT * in session_store.go
- Replaced all SELECT * queries with explicit column selection
- Used query builder instead of raw SQL strings where possible
- Added reusable sessionSelectQuery in the store constructor
- Added comprehensive test for GetSessionsWithActiveDeviceIds
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
* Fix variable shadowing issues in session_store.go
Resolved variable shadowing by reassigning to the existing error variables instead of declaring new ones in scoped blocks.
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
* Fix code formatting in session_store test file
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
* Updates the property service and store method signatures
Getters can now receive a `groupID` that narrows down the query if
present, so it's not necessary to check for the group ID on the
returning values from the outside layers.
The Search methods now receive the `groupID` and the `targetID`
explicitly as parameters, incentivizing the use of the indexes that
the underlying tables have on the searches.
* Fix tests
---------
Co-authored-by: Miguel de la Cruz <miguel@ctrlz.es>
Co-authored-by: Mattermost Build <build@mattermost.com>
* MM-63205 Revert emoji store methods to return empty arrays instead of nil
* Update tests for getEmojiList
* Add case to TestGetEmojisByNames
* Update tests for searchEmoji and autocompleteEmoji
* Bump Go version to 1.23.6
* Update CodeQL Github action as well
* Use server's Go version for CodeQL action
Co-authored-by: Antonis Stamatiou <stamatiou.antonis@gmail.com>
* Empty commit to trigger CI
* Bump golangci-lint to a version supporting Go 1.23
* Fix golangci-lint warnings
Several rules from gosimple, revive and staticcheck linters were
failing:
- Redefinition of built-in identifiers (max, min, new, recover...)
- Use of printf-like functions with simple strings
- Check for nil slices, when len already takes it into account
---------
Co-authored-by: Antonis Stamatiou <stamatiou.antonis@gmail.com>
Co-authored-by: Mattermost Build <build@mattermost.com>
* Adde MySQL and Postgres migrations
* Replaced select * with column names
* removed all * from channel SQL store
* cleanup
* Fixed a duplicate column
* cleanup
* Added migrations and store support
* WIP
* used channelname slice in a missed place
* Handled patch
* Added app level tests
* Added API layer tests
* Added API layer tests
* WIP
* converted to query builder
* cleanupo
* added not null and default constraints
* Fixed test
* fixed file name
* review fixes
* review fixes
* updated migration file
* fixed text
* Review fixes
* Adde MySQL and Postgres migrations
* Replaced select * with column names
* removed all * from channel SQL store
* cleanup
* Fixed a duplicate column
* cleanup
* used channelname slice in a missed place
* WIP
* converted to query builder
* cleanupo
* Review fixes
* refactor: Replace pagination with cursor-based pagination for custom profile attributes
* remove pagination loop on property value retrieval for CPA
* add migrations to optimize pagination on property fields and values
* adapt test to remove pagination check
* update migrations list
* postgres: drop index concurrently
* concurrent index manipulation must be done outside of a Tx
* fix: Correct SQL index drop syntax from "OM" to "ON" in migration files
* test: Add CountForGroup test cases for property field store
* refactor: Add CountForGroup method to PropertyFieldStore interface and implementations
* Fix style and i18n
* feat: Add optional deleted property field filtering to CountForGroup method
* refactor: Update CountForGroup to support optional deleted property fields
* test: Add comprehensive tests for CountForGroup with includeDeleted parameter
* adapt test + gen layers
* rename property service method and set the includeDelete to false
* refactor: Remove redundant constant and use CustomProfileAttributesFieldLimit directly
* fix tests
---------
Co-authored-by: Mattermost Build <build@mattermost.com>
* Adds websocket messages to Custom Profile Attributes
The app layer now fires a websocket event as part of the operations
over Custom Profile Attribute fields and values. It updates as well
the Patch method for CPA values so all the changes are commited as
part of the same transaction.
To be able to do this last operation, the change adds methods to
upsert CPA values in both the store and the property service.
* Fix i18n strings
---------
Co-authored-by: Mattermost Build <build@mattermost.com>
Co-authored-by: Miguel de la Cruz <miguel@ctrlz.es>
* Revert "Revert "[MM-62142] Avoid SELECT * in status_store.go (#29610)" (#29985)"
This reverts commit d345e92136.
* add tests for StatusGet and StatusGetByIds
* handle NULL columns in the Status Store
* simplify status store
* more builder simplifications and tests
* expose GetQueryPlaceholder
---------
Co-authored-by: Mattermost Build <build@mattermost.com>
