* Added materialized view migration
* Renamed mat view
* Added channel membership mat view and indexes
* Added channel membership mat view and indexes
* Added new index
* WIP
* Simplifying user reporting code
* Created app and API layer for cahnnel reporting, reporting refactoring in general
* New router
* Remobved channel reporting meanwhile
* Upodated autogenerated stuff
* Lint fix
* Fixed typo
* api vet
* i18n fix
* Fixed API vetting and removed channel reporting constants
* yaml
* removed app pagination tests
While loading a thread, we were unnecessarily re-counting the replies
for all posts in a thread even if they would be the same number.
While this is needed in other queries where the posts can be from different
threads or they can be random ids, but to load a single post thread, there
is no need to recompute it again and again.
Therefore, we use a CTE to precompute the replycount and then just plug in
the value in the subsequent query. This gives an improvement in the query
plan as well:
OLD:
```
explain (analyze, buffers) SELECT p.id, p.rootid, p.createat, (SELECT count(*) FROM Posts WHERE Posts.RootId = (CASE WHEN p.RootId = '' THEN p.Id ELSE p.RootId END) AND Posts.DeleteAt = 0) as ReplyC
ount FROM Posts p WHERE (p.Id = 'h3cer597jb8abbcbitpghpomua' OR p.RootId = 'h3cer597jb8abbcbitpghpomua') AND p.DeleteAt = 0;
QUERY PLAN
------------------------------------------------------------------------------------------------------------------------------------------------------------------
Bitmap Heap Scan on posts p (cost=45.39..1042149.61 rows=2748 width=49) (actual time=3.156..7906.215 rows=5353 loops=1)
Recheck Cond: (((id)::text = 'h3cer597jb8abbcbitpghpomua'::text) OR (((rootid)::text = 'h3cer597jb8abbcbitpghpomua'::text) AND (deleteat = 0)))
Filter: (deleteat = 0)
Heap Blocks: exact=5308
Buffers: shared hit=610244
-> BitmapOr (cost=45.39..45.39 rows=2748 width=0) (actual time=0.918..0.920 rows=0 loops=1)
Buffers: shared hit=47
-> Bitmap Index Scan on posts_pkey (cost=0.00..1.68 rows=1 width=0) (actual time=0.028..0.028 rows=1 loops=1)
Index Cond: ((id)::text = 'h3cer597jb8abbcbitpghpomua'::text)
Buffers: shared hit=4
-> Bitmap Index Scan on idx_posts_root_id_delete_at (cost=0.00..42.34 rows=2747 width=0) (actual time=0.889..0.890 rows=5352 loops=1)
Index Cond: (((rootid)::text = 'h3cer597jb8abbcbitpghpomua'::text) AND (deleteat = 0))
Buffers: shared hit=43
SubPlan 1
-> Aggregate (cost=378.10..378.11 rows=1 width=8) (actual time=1.474..1.474 rows=1 loops=5353)
Buffers: shared hit=604889
-> Index Only Scan using idx_posts_root_id_delete_at on posts (cost=0.57..343.85 rows=13699 width=0) (actual time=0.016..1.039 rows=5352 loops=5353)
Index Cond: ((rootid = (CASE WHEN ((p.rootid)::text = ''::text) THEN p.id ELSE p.rootid END)::text) AND (deleteat = 0))
Heap Fetches: 0
Buffers: shared hit=604889
Planning Time: 0.194 ms
Execution Time: 7906.846 ms
```
NEW:
```
explain analyze with replycount as (select count(*) as num from posts where rootid='h3cer597jb8abbcbitpghpomua' and deleteat=0)
select id, rootid, createat, replycount.num from posts, replycount where id='h3cer597jb8abbcbitpghpomua' or rootid='h3cer597jb8abbcbitpghpomua' and deleteat=0;
-------------------------------------------------------------------------------------------------------------------------------------------------------------------
Nested Loop (cost=123.16..3215.48 rows=2748 width=49) (actual time=2.960..9.775 rows=5353 loops=1)
-> Aggregate (cost=77.78..77.79 rows=1 width=8) (actual time=1.455..1.456 rows=1 loops=1)
-> Index Only Scan using idx_posts_root_id_delete_at on posts posts_1 (cost=0.57..70.91 rows=2747 width=0) (actual time=0.056..1.145 rows=5352 loops=1)
Index Cond: ((rootid = 'h3cer597jb8abbcbitpghpomua'::text) AND (deleteat = 0))
Heap Fetches: 0
-> Bitmap Heap Scan on posts (cost=45.39..3110.20 rows=2748 width=41) (actual time=1.501..7.747 rows=5353 loops=1)
Recheck Cond: (((id)::text = 'h3cer597jb8abbcbitpghpomua'::text) OR (((rootid)::text = 'h3cer597jb8abbcbitpghpomua'::text) AND (deleteat = 0)))
Heap Blocks: exact=5308
-> BitmapOr (cost=45.39..45.39 rows=2748 width=0) (actual time=0.797..0.798 rows=0 loops=1)
-> Bitmap Index Scan on posts_pkey (cost=0.00..1.68 rows=1 width=0) (actual time=0.014..0.014 rows=1 loops=1)
Index Cond: ((id)::text = 'h3cer597jb8abbcbitpghpomua'::text)
-> Bitmap Index Scan on idx_posts_root_id_delete_at (cost=0.00..42.34 rows=2747 width=0) (actual time=0.782..0.782 rows=5352 loops=1)
Index Cond: (((rootid)::text = 'h3cer597jb8abbcbitpghpomua'::text) AND (deleteat = 0))
Planning Time: 0.220 ms
Execution Time: 10.052 ms
(15 rows)
```
Observe the `loops=5353` in the first query, and `loops=1` in the next.
https://mattermost.atlassian.net/browse/MM-55476
```release-note
Optimize createPost performance
```
Co-authored-by: Mattermost Build <build@mattermost.com>
We just throw a warn log for now. Support will be completely
removed next ESR.
https://mattermost.atlassian.net/browse/MM-55589
```release-note
MySQL 5.7 is at EOL. We recommend all customers to upgrade to atleast 8.x. For now, we are logging a warning. From version 9.5 onwards, which is the next ESR, we will stop supporting 5.7 altogether.
```
Co-authored-by: Neil B <93996140+nab-77@users.noreply.github.com>
Co-authored-by: Mattermost Build <build@mattermost.com>
* Add store method to get reporting data
* Some store changes
* Added app layer
* Added API call, some miscellaneous fixes
* Fix lint
* Fix serialized check
* Add API docs
* Fix user store tests leaking users
* Fix test
* PR feedback
* Add filtering for role/team/activated user, filter out bot users
* Fix mock
* Fix test
* Oops
* Switch to using struct filter
* More PR feedback
* Fix gen
* Fix test
* Fix API docs
* Fix test
* Fix possible SQL injection, some query optimization
* Fix migrations
* Oops
* Add role to API
* Fix check
* Add Client4 API call for load testing
* Fix test
* Update server/channels/store/storetest/user_store.go
Co-authored-by: Ibrahim Serdar Acikgoz <serdaracikgoz86@gmail.com>
* PR feedback
---------
Co-authored-by: Mattermost Build <build@mattermost.com>
Co-authored-by: Ibrahim Serdar Acikgoz <serdaracikgoz86@gmail.com>
* - columns added to ShareChannelRemotes: lastpostcreateat, lastpostupdateat
- SyncMsg and SyncResponse moved to `model` package
- field added to RemoteCluster struct: PluginID
* sync new posts before updated posts to ensure post order in MS Teams
* add plugid to remoteclusters table and store
* don't sync history by default
* [MM-55143] Disallow reacting with an emoji that does not exist
* WIP for server limit on emoji reactions
* WIP
* Implement default limit of 25 unique emoji reactions
* Add modal for reaction limit
* Fix test
* PR feedback
* Fix i18n
* Update admin string
* Merge'd
* Fixing some issues, check limits correctly based on other users reactions
* Fix typos
* Fix lint/test
* Add tests, fix other tests
---------
Co-authored-by: Mattermost Build <build@mattermost.com>
Load tests show that channelstore.GetMember and
channelstore.GetMembersForUser are among the chief
queries that take up CPU in the DB.
In this PR, we attempt some strategic optimizations to
reduce/optimize calls to channelstore.GetMember
1. Optimize `(a *App) HasPermissionToChannel`
We replace GetChannelMember with GetAllChannelMembersForUser
because it's cache backed. So although it gets more data,
it does not hit the DB and saves some latency.
2. Optimize getPostsForChannelAroundLastUnread
We repace getChannelMember with getChannelMemberOnly
which is a lite version of the store call which queries
just the ChannelMembers table. This is because
in the app layer, we just use the LastViewedAt attribute.
Therefore, there is no reason to join with 5 tables when
a single table can do the job.
3. Optimize publishWebsocketEventForPermalinkPost
We use GetAllChannelMembersById instead of GetChannelMembersPage
which again joins with a lot of other tables.
4. Optimize countMentionsFromPost
Again, we replace GetChannelMember which is a costly call joining
multiple tables, with GetAllChannelMembersNotifyPropsForChannel
which is cache-backed and gives us just what we need in the app
layer - notify props.
```release-note
Make small optimizations in several DB calls:
- App.HasPermissionToChannel
- getPostsForChannelAroundLastUnread
- publishWebsocketEventForPermalinkPost
- countMentionsFromPost
```
https://mattermost.atlassian.net/browse/MM-55295
---------
Co-authored-by: Mattermost Build <build@mattermost.com>
* Fix mark as unread on GMs
* Don't count own messages in gms when marking as unread
* Change argument name
* Rename userId
---------
Co-authored-by: Mattermost Build <build@mattermost.com>
* [MM-53428] Delete empty drafts on upsert
* Add migrations to fix existing drafts
* Fix CI
* Delete empty drafts entirely from the DB
* Fix lint
* Implement batch migration for deleting drafts
* Missing store layers
* Add updated mock
* Remove unnecessary test
* PR feedback
* Add check for cluster migration
* Fix MySQL
* Don't check for len<2
* Bit of PR feedback
* Use query builder for parameters
* PR feedback
* More PR feedback
* Merge'd
* unit test GetLastCreateAtAndUserIdValuesForEmptyDraftsMigration
* simplified builder interface
* fix DeleteEmptyDraftsByCreateAtAndUserId for MySQL
* rework as batch migration worker
* fix typo
* log ip address on version mismatches too
* simplify reset semantics
* remove trace log in favour of low spam
* document parameters for clarity
---------
Co-authored-by: Mattermost Build <build@mattermost.com>
Co-authored-by: Jesse Hallam <jesse.hallam@gmail.com>
* pass a logger instead of embedding on job
* leverage mlog.Millis
* use worker logger with HandleJobPanic
* rely on existing LogClone instead
* guard Job.LogClone against nil Job
* s/workername/worker_name
* Revert "rely on existing LogClone instead"
This reverts commit 17303cbac90d4b01815abca1309b78b97de368fb.
* Revert "guard Job.LogClone against nil Job"
This reverts commit f1ae22dee58d76f084582857830ffe8d4c546d7e.
* Added convert to channel menu item
* WIP
* refactored channel name input field and created conversion modal
* style
* style
* WIP
* wip
* Created API to fetch common teams of GM members
* Added UI for all members deactivated
* Fetched common teams in client
* WIP
* Added a required attribute to DropdownInput component
* Fixed a case with dropdown input required flag
* WIP
* API first draft
* Genetayed layers and mocks
* Fixed create channel bug
* WIP
* Added cache invalidation
* Calling API from client
* Updated API to accept name and display name as well
* WIP
* Moved converted GM to correct category
* Style fixes
* Added logic to move user to new team/channel after GM conversion
* Prevented guest user from performing action
* Added loading indicator
* Added smoother height transistion when loading finishes
* UI imporvements
* WIP
* Formatted GM conversion message on client side
* lint fix
* Moved convert option from sidebar menu to channel header menu
* Some cleanup
* Updated server layers
* Fixed i18n
* Fixed types
* Fix server i18n
* Fixed channel creation bug
* Added store test for GetCommonTeamIDsForMultipleUsers
* Server tests done
* Updated snapshots
* Updated layers
* lint fix
* Update tests
* For CI
* lint
* restored debug code
* Used user ID instead of username in channel conversion post
* WIP
* Review fixes
* LInt fixes
* Test fix
* WIP
* WIP
* WIP
* wip
* Review fixes, lots of them
* Review fix
* Disabled WIP test
* test
* Cleanup
* Test fix
* removed testing line
* Fixed incorrect default message
* Review fixes
* Fixes
* lint and i18n fix
* Setting category on server side
* updated i18n
* Updated tests
* Added tests
* Refs cleanup
* added test
---------
Co-authored-by: Harshil Sharma <harshilsharma@Harshils-MacBook-Pro.local>
* Use builder in GetMember
* Use query builder in GetByUser
* Use query builder in GetMemberUsers
* Use query builder in GetNonMemberUsersPage
* Use query builder in PermanentDeleteMembersByUser
---------
Co-authored-by: Mattermost Build <build@mattermost.com>
* adding new migration for RetentionIdsForDeletion, changing logic for deleting orphaned reactions. Updating delete user and channel endpoints to remove respective reactions
* Refactor: Use generic to change func StringInSlice to Contains
* Refactor: Use generic to change func stringNotInSlice to Contains
Make utils.go (dir server/public/utils) and func Contains
* Refactor: Move func Contains from channels/utils to public/utils
Move func Contains from channels/utils to public/utils
Fix import declarations line
* Docs: Add a description of the Contains function
* Test: add TestContains
Add a test code for a Contain function
* add ability to restore groups from the user group modal
* factory selector for groups to reduce number of renders across the app
* react window and infinite scroll for user groups
* adding archive groups to dropdown
* restore user group from the view modal
* component cleanup
* lint
* adding websocket for archiveGroup
* updating tests
* adding some tests and fixing types
* lint
* fixing broken test
* fixing snapshot
* fixing infinitescroll
* lint
* increasing max-height and updating snapshots
* fixing PR comments
* fixing case for button
* snapshot and translation
* fixing PR comments
* tiding up repition and creating new hook
* fixing tests
* add additional parammeter for call to getGroups()
* make sure popup is visible for all rows
* update text for admin console
* update css for lint
* fix edge cases found in review
* revert package-lock.json
* revert adding query to GetGroupsParam
* fixing lint
* change include_archived to false in team_controller
---------
Co-authored-by: Benjamin Cooke <benjamincooke@Benjamins-MacBook-Pro.local>
Co-authored-by: Scott Bishel <scott.bishel@mattermost.com>
Co-authored-by: Mattermost Build <build@mattermost.com>
* [MM-37984] Allow Desktop App to authenticate via external providers outside of the app on supported servers
* PR feedback
* Add support for mattermost-dev protocol for development use
* Update server/channels/db/migrations/postgres/000110_create_desktop_tokens.up.sql
* Fix silly typo
* Update server/channels/db/migrations/postgres/000110_create_desktop_tokens.up.sql
* Remove storage of client token, only validate it on the client
* Update migrations
* Add concurrently create index
* Remove CONCURRENTLY for now
* Fix issue with changing history
* Remove old migration
* Use idempotent statement to drop old index
* Remove reference to old table
