For our MySQL customers, we have discovered that the query is not
able to choose the right plan by itself without adequate hints.
This is only for MySQL as we have confirmed from multiple customers
that Postgres takes the right index idx_sidebarcategories_userid_teamid
for the sidebarCategories table. And if it doesn't, then a VACUUM ANALYZE
fixes it.
But for MySQL, we have to do two things:
- Pass an index hint to let it use idx_sidebarcategories_userid_teamid.
- Pass an optimizer hint to materialize the sub-query. This is used
to materialize the doesNotHaveSidebarChannel sub-query into a temporary
table, letting MySQL reuse the contents of the table for further processing
in the parent sections of the query.
I have confirmed both locally and in the customer environment
that it gives a clear benefit.
*LOCAL*
OLD:
```
| -> Nested loop antijoin (cost=2889.85 rows=19767) (actual time=3.355..38.033 rows=15 loops=1)
-> Nested loop inner join (cost=66.65 rows=110) (actual time=0.410..1.689 rows=220 loops=1)
-> Filter: ((Channels.DeleteAt = 0) and (Channels.`Type` in ('O','P'))) (cost=25.25 rows=110) (actual time=0.394..0.886 rows=220 loops=1)
-> Index lookup on Channels using idx_channels_team_id_display_name (TeamId='team01'), with index condition: (Channels.Id is not null) (cost=25.25 rows=220) (actual time=0.389..0.793 rows=220 loops=1)
-> Single-row covering index lookup on ChannelMembers using PRIMARY (ChannelId=Channels.Id, UserId='user000') (cost=0.28 rows=1) (actual time=0.003..0.003 rows=1 loops=220)
-> Nested loop inner join (cost=4967.50 rows=180) (actual time=0.165..0.165 rows=1 loops=220)
-> Covering index lookup on SidebarChannels using PRIMARY (ChannelId=Channels.Id) (cost=7.86 rows=180) (actual time=0.055..0.062 rows=13 loops=220)
-> Filter: ((SidebarCategories.TeamId = 'team01') and (SidebarCategories.UserId = 'user000')) (cost=44.93 rows=1) (actual time=0.008..0.008 rows=0 loops=2881)
-> Single-row index lookup on SidebarCategories using PRIMARY (Id=SidebarChannels.CategoryId) (cost=44.93 rows=1) (actual time=0.006..0.006 rows=1 loops=2881)
|
```
NEW:
```
| -> Nested loop antijoin (cost=5879.73 rows=58021) (actual time=1.544..3.135 rows=15 loops=1)
-> Nested loop inner join (cost=66.65 rows=110) (actual time=0.421..1.778 rows=220 loops=1)
-> Filter: ((Channels.DeleteAt = 0) and (Channels.`Type` in ('O','P'))) (cost=25.25 rows=110) (actual time=0.405..0.945 rows=220 loops=1)
-> Index lookup on Channels using idx_channels_team_id_display_name (TeamId='team01'), with index condition: (Channels.Id is not null) (cost=25.25 rows=220) (actual time=0.400..0.859 rows=220 loops=1)
-> Single-row covering index lookup on ChannelMembers using PRIMARY (ChannelId=Channels.Id, UserId='user000') (cost=0.28 rows=1) (actual time=0.003..0.004 rows=1 loops=220)
-> Single-row index lookup on <subquery2> using <auto_distinct_key> (ChannelId=Channels.Id) (cost=130.37..130.37 rows=1) (actual time=0.006..0.006 rows=1 loops=220)
-> Materialize with deduplication (cost=130.35..130.35 rows=527) (actual time=1.118..1.118 rows=205 loops=1)
-> Filter: (SidebarChannels.ChannelId is not null) (cost=77.61 rows=527) (actual time=0.059..0.851 rows=523 loops=1)
-> Nested loop inner join (cost=77.61 rows=527) (actual time=0.058..0.786 rows=523 loops=1)
-> Covering index lookup on SidebarCategories using idx_sidebarcategories_userid_teamid (UserId='user000', TeamId='team01') (cost=2.81 rows=15) (actual time=0.025..0.031 rows=15 loops=1)
-> Covering index lookup on SidebarChannels using idx_sidebarchannels_categoryid (CategoryId=SidebarCategories.Id) (cost=1.70 rows=35) (actual time=0.032..0.046 rows=35 loops=15)
```
Performance improvement from 38ms to 3ms.
*CUSTOMER ENV* (with sensitive data wiped off)
OLD:
```
| -> Sort: channels.DisplayName (actual time=512..512 rows=5 loops=1)
-> Stream results (cost=3.28 rows=1.44) (actual time=223..512 rows=5 loops=1)
-> Nested loop antijoin (cost=3.28 rows=1.44) (actual time=223..512 rows=5 loops=1)
-> Nested loop inner join (cost=3.02 rows=0.3) (actual time=0.025..0.0878 rows=5 loops=1)
-> Covering index lookup on ChannelMembers using idx_channelmembers_user_id_channel_id_last_viewed_at (UserId='') (cost=0.916 rows=6) (actual time=0.0146..0.023 rows=6 loops=1)
-> Filter: ((channels.DeleteAt = 0) and (channels.TeamId = '') and (channels.`Type` in ('O','P'))) (cost=0.251 rows=0.05) (actual time=0.00999..0.0102 rows=0.833 loops=6)
-> Single-row index lookup on Channels using PRIMARY (Id=channelmembers.ChannelId) (cost=0.251 rows=1) (actual time=0.00778..0.00785 rows=1 loops=6)
-> Nested loop inner join (cost=2.85 rows=4.81) (actual time=102..102 rows=0 loops=5)
-> Covering index lookup on SidebarChannels using PRIMARY (ChannelId=channelmembers.ChannelId) (cost=2.01 rows=4.81) (actual time=0.0125..13.8 rows=24134 loops=5)
-> Filter: ((sidebarcategories.TeamId = '') and (sidebarcategories.UserId = '')) (cost=1.54 rows=1) (actual time=0.00359..0.00359 rows=0 loops=120671)
-> Single-row index lookup on SidebarCategories using PRIMARY (Id=sidebarchannels.CategoryId) (cost=1.54 rows=1) (actual time=0.00316..0.00319 rows=1 loops=120671)
```
NEW:
```
Here is the output
| -> Sort: channels.DisplayName (actual time=0.12..0.12 rows=5 loops=1)
-> Stream results (cost=3.45 rows=4.01) (actual time=0.0797..0.11 rows=5 loops=1)
-> Nested loop antijoin (cost=3.45 rows=4.01) (actual time=0.0769..0.106 rows=5 loops=1)
-> Nested loop inner join (cost=3.02 rows=0.3) (actual time=0.0291..0.0555 rows=5 loops=1)
-> Covering index lookup on ChannelMembers using idx_channelmembers_user_id_channel_id_last_viewed_at (UserId='') (cost=0.916 rows=6) (actual time=0.0145..0.0162 rows=6 loops=1)
-> Filter: ((channels.DeleteAt = 0) and (channels.TeamId = '') and (channels.`Type` in ('O','P'))) (cost=0.251 rows=0.05) (actual time=0.00611..0.00619 rows=0.833 loops=6)
-> Single-row index lookup on Channels using PRIMARY (Id=channelmembers.ChannelId) (cost=0.251 rows=1) (actual time=0.0053..0.00534 rows=1 loops=6)
-> Single-row index lookup on <subquery2> using <auto_distinct_key> (ChannelId=channelmembers.ChannelId) (cost=7.01..7.01 rows=1) (actual time=0.00956..0.00956 rows=0 loops=5)
-> Materialize with deduplication (cost=7..7 rows=13.4) (actual time=0.0451..0.0451 rows=0 loops=1)
-> Filter: (sidebarchannels.ChannelId is not null) (cost=5.66 rows=13.4) (actual time=0.0441..0.0441 rows=0 loops=1)
-> Nested loop inner join (cost=5.66 rows=13.4) (actual time=0.0439..0.0439 rows=0 loops=1)
-> Covering index lookup on SidebarCategories using idx_sidebarcategories_userid_teamid (UserId='', TeamId='') (cost=0.592 rows=3) (actual time=0.0105..0.0134 rows=3 loops=1)
-> Covering index lookup on SidebarChannels using idx_sidebarchannels_categoryid (CategoryId=sidebarcategories.Id) (cost=1.39 rows=4.46) (actual time=0.00999..0.00999 rows=0 loops=3)
```
Performance improvement from 512ms to 0.12ms.
https://mattermost.atlassian.net/browse/MM-64209
```release-note
NONE
```
* Prevents the API from updating synced CPA values
The patch functions for CPA values now accept a parameter that checks
if they should allow for synced values to be updated, and prevent
those updates if necessary
* Fix linter
* Fix parameter after merge
---------
Co-authored-by: Miguel de la Cruz <miguel@ctrlz.es>
Co-authored-by: Scott Bishel <scott.bishel@mattermost.com>
Co-authored-by: Mattermost Build <build@mattermost.com>
With some neat generics, I was able to refactor
the scanning to a util function. I used it to
refactor 3 places and also removed an unnecessary method.
Claude was quite good here.
https://mattermost.atlassian.net/browse/MM-62755
```release-note
NONE
```
INSERT IGNORE will ignore ALL errors in the INSERT statement.
This is not what was intended. The right way is to do a
redundant update on duplicate key.
It's not great, but that's how MySQL wants us to do it.
https://mattermost.atlassian.net/browse/MM-63878
```release-note
NONE
```
* MM-61173 - channel settings modal: base modal, initial commit, file creation and base component
* new enhancements to the base modal creation
* revert changes on textbox_links and edit channel header
* fix types and add back unintentioned deleted value
* add the preview textbox component
* extract logic for info tab into its own component
* add the purpose input to the window
* move other component logic to its own component and code clean up
* ability to update channel type
* more advances on the archive channel tab
* fix unit test in textbox
* fix translations
* do not show the archive modal in default channel
* fix issue with url editor not being resetted on undo action
* adjust text and styling for the header and purpose inputs
* remove textboxlinks and use button eye icon
* adjust test and preview button style
* add unit test to channel patch
* move logic from parent modal to info tab component
* fix border issues and focus back to preview textareas
* prevent saving changes when pressing enter when selecting an icon
* enhance input component to cover limits validations and enhances tests
* set default error message for save changes panel
* add props to provide custom value to the buttons
* remove channel input errors on reset button click
* create new component settings textbox
* rename component to advanced textbox and add unit tests
* styling of the info tab and add error state to advanced textbox
* add logic to prevent tab switch with unsaved changes
* adjust url error logic and code clean up
* code clean up and enhance comments
* add char min length to advanced textbox logic
* add the channel settings modal to the new menu
* add new test files and fix reset error
* remove unused error variables
* adjust translations and remove unncesary import
* enhance permissions for archive channels and manage channel settings
* Adjust permission tree so channel admins can convert from private to public
* enhance the test suit around channel conversion type
* fix some e2e tests and solve channel input name issue
* fix unit test by interacting first with the input element
* adjust e2e tests to channel settings modal changes
* remove commented tests and implement pr feedback
* adjust more pr feedback to the code
* more pr feedback enhancements
* further enhancements to tab navigation, and adjust more e2e tests
* remove unused components and fix e2e tests
* revert unnecessary permissions changes
* Add name label to textboxes
* adjust e2e and unit tests
* revert min lenght change value and adjust tests and snapshots
* Channel banner settings (#30721)
* Added channel banner setting header
* Updated section styling
* handled animation
* handled min and max lengths
* cleanup
* color change fix
* general improvements
* Fixed API test
* removed unused param className
* added e2e tests
* test: add channel settings configuration tab test file
* Based on the context, here's a concise commit message for this change:
feat: Add comprehensive tests for ChannelSettingsConfigurationTab
* added some more tests
* CI
* reverted package-lock.json changes in Playwright
* remove extra border from advaced textbox
* adjust styling for name label in advance texbox and restart preview state on modal close
* sync package.lock in playwright
---------
Co-authored-by: Mattermost Build <build@mattermost.com>
Co-authored-by: Harshil Sharma <harshilsharma63@gmail.com>
Co-authored-by: Harshil Sharma <18575143+harshilsharma63@users.noreply.github.com>
* avoid SELECT * in command stores
* Define column names once and reuse as columns list in selects and inserts
* Define column names once and reuse with ExecBuilder
Every time we load the RHS, we used to load the FULL thread always. Although
the actual ThreadViewer React component is virtualized, and the server side
API call is paginated, we still went through all the pages, to get the full
thread and passed it on to the ThreadViewer. This would be for first loads,
and subsequent loads of the same thread.
This was a bug originally, but then it was a necessity after we applied websocket event scope because
now we won't get emoji reactions of a thread if the user is not on the thread.
To fix that, we enhance the thread loading functionality by adding support for fetching
thread updates based on the UpdateAt timestamp. Now, for subsequent loads,
we only get the changed posts in a thread. The implementation:
- Adds new API parameters: fromUpdateAt and updatesOnly to the GetPostThread endpoint
- Updates database queries to support sorting and filtering by UpdateAt
- Implements thread state management to track the last update timestamp
- Adds client-side support to use incremental loading for improved performance
- Ensures proper validation for parameter combinations and error handling
This change enables more efficient thread loading, particularly for long threads
with frequent updates, by only fetching posts that have been updated since the
last view.
Caveats: For delta updates, the SQL query won't use the best index possible
because we have an index for (CreateAt, Id), but no index for (UpdateAt, Id).
However, from my tests, it is not as bad as it looks:
```
[loadtest] # EXPLAIN (ANALYZE, BUFFERS) SELECT * FROM Posts WHERE Posts.DeleteAt = 0 AND Posts.RootId = 'qbr5gctu9iyg8c36hpcq6f3w8e' AND Posts.UpdateAt > 1623445795824 ORDER BY UpdateAt ASC, Id ASC LIMIT 61;
QUERY PLAN
-------------------------------------------------------------------------------------------------------------------------------------------------
Limit (cost=8.31..8.31 rows=1 width=216) (actual time=0.047..0.049 rows=0 loops=1)
Buffers: shared hit=2
-> Sort (cost=8.31..8.31 rows=1 width=216) (actual time=0.044..0.045 rows=0 loops=1)
Sort Key: updateat, id
Sort Method: quicksort Memory: 25kB
Buffers: shared hit=2
-> Index Scan using idx_posts_root_id_delete_at on posts (cost=0.28..8.30 rows=1 width=216) (actual time=0.031..0.032 rows=0 loops=1)
Index Cond: (((rootid)::text = 'qbr5gctu9iyg8c36hpcq6f3w8e'::text) AND (deleteat = 0))
Filter: (updateat > '1623445795824'::bigint)
Buffers: shared hit=2
Planning:
Buffers: shared hit=3
Planning Time: 0.508 ms
Execution Time: 0.106 ms
(14 rows)
```
We still get an index scan with index cond. Although there's a filter element, but atleast we get the whole thread with the index.
My thinking is that while the whole thread might be large, but after that, updates on a thread should be incremental.
Therefore, we should be okay without adding yet another index on the posts table.
This is just the first step in what could be potentially improved further.
1. We shouldn't even be loading the full thread always. But rather let the virtualized viewer
load more posts on demand.
2. If a post has been just reacted to, then we need not send the whole post down, but just the
reaction. This further saves bandwidth.
https://mattermost.atlassian.net/browse/MM-56548
TBD: Add load-test coverage to update the thread loading code
```release-note
NONE
```
---------
Co-authored-by: Mattermost Build <build@mattermost.com>
* Add license load metric endpoint and UI indicator
Adds an API endpoint to calculate and return license usage as a load metric, and displays this metric in the About dialog. The metric is calculated as (MAU/licensed users)*100.
Additionally:
- Renamed function to be consistent with API endpoint name
- Added proper i18n strings for error messages and UI elements
* Fix TypeScript null check in about_build_modal.tsx
* MM-63728: Update OpenAPI documentation for license load metric
Update the OpenAPI documentation and code comments to correctly describe the license load metric calculation as using a multiplier of 1000 instead of percentage.
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
* MM-63728: Use float for license load metric calculation
Modify the license load metric calculation to use floats throughout the computation process while still returning an integer result. This maintains the existing API but improves the precision of the calculation.
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
* improve tests manually
* Update server/channels/api4/license_test.go
Co-authored-by: Doug Lauder <wiggin77@warpmail.net>
* Update server/channels/api4/license_test.go
Co-authored-by: Doug Lauder <wiggin77@warpmail.net>
---------
Co-authored-by: Mattermost Build <build@mattermost.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Doug Lauder <wiggin77@warpmail.net>
This change automatically removes options and sync attributes when
sanitizing fields that don't support them. As per values, it returns
an error when the value for a text type field is longer than the 64
characters limit we're currently applying.
The PR fixes a bug on the create CPA field endpoint that was causing
the attrs of the CPAField not to be decoded correctly.
Co-authored-by: Miguel de la Cruz <miguel@ctrlz.es>
* feat: Add certificate upload option for audit logging settings
* Commit current changes
* Additions
* MM-62944 Fix fileupload settings not being clickable
* Support for uploading a cert for experimental audit logging cert. Pre cloud implementation in the backend
* Forgot to add new hook
* Add support for setting custom audit log certifcates in Cloud
* Permissions
* I18n
* Change order
* Linter fixes
* Linter fixes, add openapi spec
* additions for openapi
* More openapi fixes because it won't run locally
* Undo, cursor went rogue
* newline fix
* Align types properly
* Fix i18n
* Fix i18n AGAIN
* Fix error
* Update api/v4/source/audit_logging.yaml
---------
Co-authored-by: Harrison Healey <harrisonmhealey@gmail.com>
Co-authored-by: Mattermost Build <build@mattermost.com>
It was a mistake to return ANY error as NewErrNotFound. Because this
can be a DB timeout, or any other network error.
Returning NewErrNotFound would categorize it as 404, eventually printing
the error in the DEBUG level. Changing it to normal error fixes this.
https://mattermost.atlassian.net/browse/MM-63739
```release-note
NONE
```
This avoids a non-zero chance we append "pi" from the `model.NewId` and
end up with an invalid prefix "Api".
Co-authored-by: Mattermost Build <build@mattermost.com>
* MM-62156: Avoid SELECT * in retention_policy_store.go
- Modified subQueryIN function to use specific column name instead of SELECT *
- Improved code comments to explain the change
- Maintained same functionality while avoiding SELECT *
Fixes: https://mattermost.atlassian.net/browse/MM-62156🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
* simplify subQueryIN comments
* inline part of subQueryIN for greater clarity
---------
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Mattermost Build <build@mattermost.com>
* fix: hanlde error from InvalidateAllCaches in slack.go
* change signature of InvalidateAllCaches to *model.AppError
* return err from InvalidateAllCaches everywhere
* Formatting
---------
Co-authored-by: Ben Schumacher <ben.schumacher@mattermost.com>
