- Bump timeouts for some cases
- Improve sleep with require/assert.EventuallyT
for better performance and reliability.
https://mattermost.atlassian.net/browse/MM-57532
```release-note
NONE
```
Improvements to logging start-up.
- comments have been added
- order of configuration is adjusted to allow the main logger to capture errors in notification logger
- redundant logger initialization removed; no need to configure the logger just to replace the configuration immediately after
Previous log lines just had generic errors
which made things difficult to debug because
if a system bot is disabled, it's hard to know
whether the error is coming while trying to send
post reminders or from some other part of the system.
```release-note
NONE
```
* Added hard limits when creating user
* Added check to user activation
* Added missing check for licensed servers
* Fix i18n
* Fixed style order
* Added a separate hard limit along with existing 10k user soft limit
* For CI
* Fixing flaky test, hopefully
* Added tests
* add ratelimit to /sessions; cap userAgent version length; tests
* add MaxSessionsLimit; remove oldest session first; tests
* can't use slices in 1.20; improve test
* nits
* add GetLRUSessions; move limiting to CreateSession; remove rate limiting
* use queryBuilder
* mysql needs a limit when using offset
* update i18n
* refactor into limitNumberOfSessions; protect createSessionForUserAccessToken
* add comment to GetLRUSessions
* add limit to oauth path; PR comments
* Use a row constructor comparison to avoid a filter
The original query had a WHERE condition that looked like:
(x > a) OR (x = a AND y < b)
This commit changes it to a compound comparison that, given the
lexicographical order it uses, is semantically the same:
(x, y) > (a, b)
This makes the plan in Postgres use an Index Cond instead of a Filter,
having two main performance improvements:
1. The query's time is more or less constant, not increasing every time
this query is executed. This is important because the query is executed
repeatedly until all posts are indexed.
2. The query's time is much shorter: while the original query eventually
takes 30s, hitting the timeout, the new one takes ~30ms.
Notice the difference in the plans below:
- the original query uses a `Filter` step, while the new one uses an
`IndexCond` + `Filter` step.
- the original query has a shared hit count of 40557115 (109GiB), while
the new one has a shared hit count of 13951 (109MiBA). This does not
show exactly how much data is read from disk, given it's a nested loop
node, but it gives us an idea of the amount of data processed in the
server.
Original plan:
agnivaltdb=> EXPLAIN (ANALYZE, BUFFERS) SELECT Posts.*, Channels.TeamId FROM Posts LEFT JOIN Channels ON Posts.ChannelId = Channels.Id WHERE Posts.CreateAt > '1687424888405' OR (Posts.CreateAt = '1687424888405' AND Posts.Id > 'tpomh9yu1tffmdp6dopobwuc9h') ORDER BY Posts.CreateAt ASC, Posts.Id ASC LIMIT 10000;
QUERY PLAN
------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Limit (cost=1.22..2287.94 rows=10000 width=464) (actual time=23921.236..23954.229 rows=10000 loops=1)
Buffers: shared hit=40557115
-> Incremental Sort (cost=1.22..13998881.93 rows=61217938 width=464) (actual time=23921.235..23953.205 rows=10000 loops=1)
Sort Key: posts.createat, posts.id
Presorted Key: posts.createat
Full-sort Groups: 311 Sort Method: quicksort Average Memory: 45kB Peak Memory: 45kB
Buffers: shared hit=40557115
-> Nested Loop Left Join (cost=1.00..11421751.31 rows=61217938 width=464) (actual time=23920.970..23947.067 rows=10001 loops=1)
Buffers: shared hit=40557115
-> Index Scan using idx_posts_create_at on posts (cost=0.57..9889434.54 rows=61217938 width=461) (actual time=23920.930..23931.063 rows=10001 loops=1)
Filter: ((createat > '1687424888405'::bigint) OR ((createat = '1687424888405'::bigint) AND ((id)::text > 'tpomh9yu1tffmdp6dopobwuc9h'::text)))
Rows Removed by Filter: 40920000
Buffers: shared hit=40553119
-> Memoize (cost=0.43..0.70 rows=1 width=30) (actual time=0.001..0.001 rows=1 loops=10001)
Cache Key: posts.channelid
Cache Mode: logical
Hits: 9002 Misses: 999 Evictions: 0 Overflows: 0 Memory Usage: 151kB
Buffers: shared hit=3996
-> Index Scan using channels_pkey on channels (cost=0.42..0.69 rows=1 width=30) (actual time=0.007..0.007 rows=1 loops=999)
Index Cond: ((id)::text = (posts.channelid)::text)
Buffers: shared hit=3996
Planning:
Buffers: shared hit=112
Planning Time: 0.501 ms
Execution Time: 23954.974 ms
(25 rows)
New plan:
agnivaltdb=> EXPLAIN (ANALYZE, BUFFERS) SELECT Posts.*, Channels.TeamId FROM Posts LEFT JOIN Channels ON Posts.ChannelId = Channels.Id WHERE (Posts.CreateAt, Posts.Id) > ('1687424888405', 'tpomh9yu1tffmdp6dopobwuc9h') ORDER BY Posts.CreateAt ASC, Posts.Id ASC LIMIT 10000;
QUERY PLAN
-----------------------------------------------------------------------------------------------------------------------------------------------------------------
Limit (cost=1.15..1666.83 rows=10000 width=464) (actual time=0.366..34.053 rows=10000 loops=1)
Buffers: shared hit=13951
-> Incremental Sort (cost=1.15..10196977.36 rows=61217938 width=464) (actual time=0.365..33.031 rows=10000 loops=1)
Sort Key: posts.createat, posts.id
Presorted Key: posts.createat
Full-sort Groups: 311 Sort Method: quicksort Average Memory: 45kB Peak Memory: 45kB
Buffers: shared hit=13951
-> Nested Loop Left Join (cost=1.00..7619846.74 rows=61217938 width=464) (actual time=0.059..26.840 rows=10001 loops=1)
Buffers: shared hit=13951
-> Index Scan using idx_posts_create_at on posts (cost=0.57..6087529.97 rows=61217938 width=461) (actual time=0.040..10.548 rows=10001 loops=1)
Index Cond: (createat >= '1687424888405'::bigint)
Filter: (ROW(createat, (id)::text) > ROW('1687424888405'::bigint, 'tpomh9yu1tffmdp6dopobwuc9h'::text))
Rows Removed by Filter: 2
Buffers: shared hit=9955
-> Memoize (cost=0.43..0.70 rows=1 width=30) (actual time=0.001..0.001 rows=1 loops=10001)
Cache Key: posts.channelid
Cache Mode: logical
Hits: 9002 Misses: 999 Evictions: 0 Overflows: 0 Memory Usage: 151kB
Buffers: shared hit=3996
-> Index Scan using channels_pkey on channels (cost=0.42..0.69 rows=1 width=30) (actual time=0.007..0.007 rows=1 loops=999)
Index Cond: ((id)::text = (posts.channelid)::text)
Buffers: shared hit=3996
Planning:
Buffers: shared hit=112
Planning Time: 0.471 ms
Execution Time: 34.716 ms
(26 rows)
* Go back to the old query for MySQL
As one could expect, the two databases have completely opposite
behaviours, and for MySQL, the old query is significantly faster than
the new one, exactly for the same reason than in Postgres, but the other
way around: the old query uses an Index range scan while the new one
would use a Filter.
Old query (0.18s):
mysql> EXPLAIN ANALYZE SELECT Posts.*, Channels.TeamId FROM Posts USE INDEX(idx_posts_create_at_id) LEFT JOIN Channels ON Posts.ChannelId = Channels.Id WHERE Posts.CreateAt > 1557752415221 OR (Posts.CreateAt = 1557752415221 AND Posts.Id > 'ad59ire57tfwmjr5r8xqxc75qw') ORDER BY Posts.CreateAt ASC, Posts.Id ASC LIMIT 10000;
--------------
EXPLAIN ANALYZE SELECT Posts.*, Channels.TeamId FROM Posts USE INDEX(idx_posts_create_at_id) LEFT JOIN Channels ON Posts.ChannelId = Channels.Id WHERE Posts.CreateAt > 1557752415221 OR (Posts.CreateAt = 1557752415221 AND Posts.Id > 'ad59ire57tfwmjr5r8xqxc75qw') ORDER BY Posts.CreateAt ASC, Posts.Id ASC LIMIT 10000
--------------
+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| EXPLAIN |
+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| -> Limit: 10000 row(s) (cost=7472068.91 rows=10000) (actual time=0.063..164.174 rows=10000 loops=1)
-> Nested loop left join (cost=7472068.91 rows=5559093) (actual time=0.062..163.450 rows=10000 loops=1)
-> Index range scan on Posts using idx_posts_create_at_id over (CreateAt = 1557752415221 AND 'ad59ire57tfwmjr5r8xqxc75qw' < Id) OR (1557752415221 < CreateAt), with index condition: ((Posts.CreateAt > 1557752415221) or ((Posts.CreateAt = 1557752415221) and (Posts.Id > 'ad59ire57tfwmjr5r8xqxc75qw'))) (cost=1357066.29 rows=5559093) (actual time=0.043..97.358 rows=10000 loops=1)
-> Single-row index lookup on Channels using PRIMARY (Id=Posts.ChannelId) (cost=1.00 rows=1) (actual time=0.006..0.006 rows=1 loops=10000)
|
+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.18 sec)
New query (5s):
mysql> EXPLAIN ANALYZE SELECT Posts.*, Channels.TeamId FROM Posts USE INDEX(idx_posts_create_at_id) LEFT JOIN Channels ON Posts.ChannelId = Channels.Id WHERE (Posts.CreateAt, Posts.Id) > (1557752415221, 'ad
59ire57tfwmjr5r8xqxc75qw') ORDER BY Posts.CreateAt ASC, Posts.Id ASC LIMIT 10000;
--------------
EXPLAIN ANALYZE SELECT Posts.*, Channels.TeamId FROM Posts USE INDEX(idx_posts_create_at_id) LEFT JOIN Channels ON Posts.ChannelId = Channels.Id WHERE (Posts.CreateAt, Posts.Id) > (1557752415221, 'ad59ire57tfwmjr5r8xqxc75qw') ORDER BY Posts.CreateAt ASC, Posts.Id ASC LIMIT 10000
--------------
+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| EXPLAIN |
+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| -> Limit: 10000 row(s) (cost=11119405.48 rows=10000) (actual time=5295.106..5455.285 rows=10000 loops=1)
-> Nested loop left join (cost=11119405.48 rows=10000) (actual time=5295.105..5454.572 rows=10000 loops=1)
-> Filter: ((Posts.CreateAt,Posts.Id) > (1557752415221,'ad59ire57tfwmjr5r8xqxc75qw')) (cost=221.48 rows=10000) (actual time=5295.078..5388.668 rows=10000 loops=1)
-> Index scan on Posts using idx_posts_create_at_id (cost=221.48 rows=10000) (actual time=0.055..5314.753 rows=600000 loops=1)
-> Single-row index lookup on Channels using PRIMARY (Id=Posts.ChannelId) (cost=1.00 rows=1) (actual time=0.006..0.006 rows=1 loops=10000)
|
+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (5.46 sec)
```
* Add comment explaining the difference between DBs
---------
Co-authored-by: Mattermost Build <build@mattermost.com>
We use the same name for hit/miss/invalidation metrics.
The downside is that now we have conflated a full cache purge
and a single cache key invalidation into a single label, but
the advantage is that it's easier to monitor the usage of a single
cache.
And moreover, duplicating the same string across multiple places
increases the chances of making a mistake. Centralizing the cache
name only at instantiation simplifies the code.
After this, we would be ready to use Redis.
https://mattermost.atlassian.net/browse/MM-57193
```release-note
NONE
```
* WIP
* Added rebuild channels index functionality
* Added rough logic to send message to all sysadmins
* WIP
* WIP
* WIP
* Cleanup
* i18n fix
* reading through all pages of system admins
* Fixed webapp style
* i18n fix
* Added help text
* i18n fix
* i18n update
* Updated system console button action
* Updated snapshots
* some cleanup
* Updated snapshot
* Update server/channels/app/server.go
Co-authored-by: Daniel Espino García <larkox@gmail.com>
* fixed typo
* Refactoring to improve readibility
* moved index check to API later during config update
* Added some docs
* Updated get system bot
---------
Co-authored-by: Daniel Espino García <larkox@gmail.com>
* create ChannelBookmarks table
* ChannelBookmark model
* channel bookamrks Store layer
* add GetBookmarksForAllChannelByIdSince
* add channel bookmarks to test store
* Add channel bookmarks to app layer
* remove index for createAt in channel bookmarks migrations
* remove createAt from select channel bookmark query and enable store delete bookmark test
* update reponse of UpdateBookmark
* rename db migration files
* channel bookmarks store update sort order
* channel bookmarks app layer update sort order
* fix lint & tests
* Fix lint and introduce util functions to insert / remove from slice
* remove model etag
* i18n
* defer remove file info after test run
* Fix tests passing the request context
* fix migrations
* fix TestRetry
* Add bookmark permissions (#25560)
* Adds channel bookmarks permissions
* Fix linter
* Remove unnecessary empty lines
* Remove scss change as it's not necessary anymore
* Fix mock store
* Fix mock store and add role entry
* Fix test
* Adds cypress test and update permissions migration to update admin roles
* Adds channel bookmarks roles to default admin roles
* Adds bookmark permissions to default role permissions constant in webapp
* Update mmctl test
* Update permission test after normalising the roles
* fix store tests
* fix app layer tests
* Add new bookmark endpoint (#25624)
* Adds channel bookmarks api scaffold and create endpoint
* Applies review comments to the API docs
* Adds websocket test to create channel bookmark
---------
Co-authored-by: Mattermost Build <build@mattermost.com>
* MM-54426 exclude Channel Bookmarks files from data retention (#25656)
* Augment channel APIs to include bookmarks (#25567)
* update files docs for server 9.4
* Adds update channel bookmark endpoint (#25653)
* Adds update channel bookmark sort order endpoint (#25686)
* Adds update channel bookmark endpoint
* Updates edit app method to return the right deleted bookmark and adds tests
* Adds the update channel bookmark sort order endpoint
* Fix repeated test after merge
* Assign right permissions to each test
* Update store and app layer to return specific errors and add tests
* Adds delete channel bookmark endpoint (#25693)
* Updates edit app method to return the right deleted bookmark and adds tests
* Fix repeated test after merge
* Updates edit app method to return the right deleted bookmark and adds tests
* Adds delete channel bookmark endpoint
* Adds list channel bookmarks endpoint (#25700)
* Add channel moderation to bookmarks (#25716)
* fix migrations index
* fix getChannelsForTeamForUser
* fix getChannelsForTeamForUser
* fix bad merge client4
* fix file api with bookmark permission
* add ChannelBookmarks feature flag
* add missing translations
* Set DB column for type as enum
* use custom type for bookmark query using sqlx
* use transaction when saving bookmark
* return NewErrNotFound instead of Sql.ErrNoRows
* use squirrel for IN query
* add a limit of 1K for records in GetBookmarksForAllChannelByIdSince
* UpdateSortOrder with one single query instead of multiple updates
* fix shadow declaration
* fix channel bookmarks permission string definition in admin console
* fix another shadow declaration
* Fix model conversion
* add SplitSliceInChunks
* remove include bookmarks in channels api
* Cap amount of bookmarks per channel
* add etag back to get channels
* feedback review
* update file info when replacing a bookmark file
* return 501 not implemented when the license is not available
* add detail message when getting channel member on bookmark api
* start audit before permission check on create bookmark api
* use require.Eventuallyf for testing WS events
* remove unnecessary log in app layer
* use require instead of assert to avoid panics
* enforce limit when querying bookmarks since
* prevent to create/update bookmark if file is already attached
* fix lint
* delete file when a bookmark is deleted
* Dot allow to set a fileId and a url at the same time to a bookmark
* fix query to delete a file that belongs to a bookmark
* do not patch the bookmark type
* Server side FeatureFlag check (#26145)
* use ff in server, set ff to false
* turn on FF for unit tests
* defer unset FF for unit tests
* turn ff on for testing
* only allow attaching files that were uploaded for bookmark
* Set feature flag off as default
* fix lint
* update email templates as PR failed
* revert templates
* force the assignment of ID when creating a bookmark
* Fix unit tests
---------
Co-authored-by: Miguel de la Cruz <miguel@mcrx.me>
Co-authored-by: Mattermost Build <build@mattermost.com>
Co-authored-by: Caleb Roseland <caleb@calebroseland.com>
Co-authored-by: Scott Bishel <scott.bishel@mattermost.com>
UserStore.GetAllProfiles is a very frequent call. We cache it
when there are no options passed which is the default case.
https://mattermost.atlassian.net/browse/MM-57084
```release-note
NONE
```
* Moved some common SQL function tu public utls as they are used in plugins
* goimported file
* Added tests
* Created sub-package
* MOved SetupConnection to public sql utils
* don't allow guest to be set only on channels.
* fix bad merge, add jira ticket link
---------
Co-authored-by: Mattermost Build <build@mattermost.com>
* dont allow guests to be set via team API
* comment out invalid test
* Update import_functions_test.go
---------
Co-authored-by: Mattermost Build <build@mattermost.com>
This is in preparation to make the codebase
ready to use Redis. In Redis, iterating all
the keys at once is an expensive operation.
It is recommended to work on batches of keys.
Remove the Len method as it was unused.
I tried to repurpose the Keys method to iterate
on keys rather than returning all keys at once, but
it has other complicacies because the code calls
other cache functions on those keys, so to handle
the LRU cache properly, it becomes slightly more
painful.
For now, we keep it like this and rather collect
all keys from Redis and then return.
https://mattermost.atlassian.net/browse/MM-56987
```release-note
NONE
```
* [MM-56757] Expand NotificationsLog to include websocket and email, adding much more varied logging across the entire process
* Rework the status/notify prop calls
* Avoid some repetition in the logging calls
* Fix one log
* Wrap error
---------
Co-authored-by: Mattermost Build <build@mattermost.com>
* refactor: migrate getFormattedPostTime to utils
Move app.getFormattedPostTime to utils and export along with its struct
* Set batch notification post times to user TZ
* default useMilitaryTime to false in batched email
If there is an error reading the user's preference for useMilitaryTime,
default to false, as that should be the default value if the user never
sets it.
---------
Co-authored-by: Mattermost Build <build@mattermost.com>
1. For file endpoints, a recent optimization added the ChannelID column
to be part of the fileinfo table. Therefore, we can skip the postID
and directly use the channelID.
2. For post endpoints, we reorder the sequence of calls such that
we get the channelID first, and use it to check the permission of the channel
rather than query the long way around by joining with the posts table
in the permissions query.
The benefit is that SessionHasPermissionToChannel is cache-backed.
So in the happy path, we save a DB call. Because GetSinglePost anyways
needed to be called.
And in the bad path, we replace it with a more efficient call. Because
SessionHasPermissionToChannel is cache-backed, so effectively we
are replacing SessionHasPermissionToChannelByPost with GetSinglePost.
3. And then for the calls that don't have the channelID available,
we change the implementation itself to get the channelID by querying
the posts table first, and then calling SessionHasPermissionToChannel.
This creates the happy path as mentioned earlier.
While here, we also do some other optimizations:
4. Pre-populate the channelID while saving the reaction, so that
we don't need to query the posts table for every single reaction save.
5. Remove unnecessary goroutine spawning for publishing reaction events,
because anyways those are asynchronous.
https://mattermost.atlassian.net/browse/MM-56877
```release-note
NONE
```
Co-authored-by: Mattermost Build <build@mattermost.com>
* Add 'Disable' config to CloudSettings to prevent the CWS backend from making calls to the Customer Portal
* Add custom error when disabled
* Make Disable setting cloud_restrictable
* Return 422 instead of 400
---------
Co-authored-by: Mattermost Build <build@mattermost.com>
* Use correct error in doInviteRemote
A Sentry crash report identified this nil pointer dereference:
https://mattermost-mr.sentry.io/issues/4930233067/events/eae438652c7b4335be2bbe19c977f680
Interestingly, the stack trace shows the actual crash happening in the
function deferred above this line, which makes sense, given that it's
accessing the returned value, which is now nil as well. However, the
origin of the crash is here, since it's using the previous appErr, which
at this point is ensured to be nil, instead of the error returned by
InviteRemoteToChannel. This makes the returned value that should have
been generated by responsef to no longer exist.
* Test inviting a remote to a channel shared with us
* Improve error when InviteRemoteToChannel fails
* Clarify that channel has remoteID
Co-authored-by: Doug Lauder <wiggin77@warpmail.net>
* gofmt
---------
Co-authored-by: Doug Lauder <wiggin77@warpmail.net>
