* MM-45120: Starts tracking team join time. Adds API to retrieve team members who joined after a given time.
* MM-45120: Updates json casing to match model.User.
We use * instead of a column name to use
index-only scan always even when other column filters
are applied.
Right now, index-only scan will only get applied
in the basic query of "select count(p.id) as value
from posts p". But it won't get applied if the query
is "select count(p.id) as value from posts p where
p.deleteat=0".
So this is a minor optimization which improves
some corner cases.
This was found from the slow query monitoring.
```release-note
NONE
```
* MM-43939: fixes lastreplyat when deleting the last reply
Currently we are not updating the Threads.LastReplyAt when the last
reply gets deleted. This can lead to threads appearing unread when
actually there is no unread thread.
This commit updates the value of Threads.LastReplyAt when a reply gets
deleted, to the most recent post's timestamp in the thread.
* Updates ReplyCount to current value on post delete
* Addresses review comments
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
This PR adds the post reminder backend work.
We add a new API endpoint via which a user can set a reminder for a post. An ephemeral message will be sent down the line to let the user know about the action. And then after the time is over, the system admin bot will send a DM message to the user about the reminder post.
We replace expected/received with the source of the collation.
This makes it more clear as to from where the values
were retrieved.
```release-note
NONE
```
* Denormalize Reactions to add ChannelId for top reactions insights query
* Remove hardcoded timestamps
* Fix store, api4 tests for reactions
* Fix tests
* Fix integrity tests, allow reaction to have ChannelId populated before calling store function
* Lint fixes
* Add ChannelId field to BulkGetForPosts, Delete store handlers
* Add index to mysql migration, add not null characteristic without a separate command
* Select channelId instead of fetching post via store.GetPost, add if exists to drop column
* Make updating of Reactions conditional to support pre-migration
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* MM-45395: Exclude bot and webhook posts from Top Team Channels. Exclude webhook posts from My Top Channels.
* MM-45395: Adds missing error test.
* MM-45395: Adds missing whitespace.
* Add new Job to keep updating the last_accessible_post time
* Filter out posts for funcs returning PostList model
* Separate methods to get and compute cache
* filter pinned posts
* For posts with sorted CreateAt order, support a faster form of filtering.
* Add inaccessible header for getPost and getPostsByIDs APIs
* replace manual binary search with the std. library
* in-place filter posts
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
Co-authored-by: Nathaniel Allred <neallred@protonmail.com>
* Add route endpoints, model, store functions, and tests for top threads
* Run make store-layers
* Make the following changes
- Fix top user threads query
- Fix passing parameters in api4/insights.go to handler in app
- Add top user threads test
* Add post-message, user_id, participants information to insights results
* model.TopThread.UserID -> model.TopThread.UserId, for compatibility with MySQL
* Rename name -> channel_name
* Add user information to response
* Link post in response, filter out deleted root posts from top threads
* Handle thread delete cases, add app tests for threads insights
* lint: fix typo
* lint: rename asserts
* lint: require.nil -> require.NoError
* Add integration tests for thread insights
* Add embeds and images to top posts
* Add license checks for top threads endpoints
* Query users in batch to populate post-creator
* Make the following changes
- Add license to test server in api4/
- Add tests for threads insights
- top team threads shouldn't include threads from other teams, DMs
- Test duration constraint
- Pagination testing for top threads in model/insights_test.go
* Add i18n-extract
* i18n fixes
* Add username, nickname to user_information
* Hide message, user_id, post_id, reply_count in depth=1 of top threads response
* Fix tests using response.reply_count to use response.post.reply_count
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
We use the default_text_search_config param from the DB
and pass that as the search config instead of hardcoding
it to english.
The docs do say that if no param is passed, by default
it will automatically fall back to default_text_search_config.
But I have seen different query plans being created
when the parameter is not passed versus when it is passed.
I am not sure why this is happening, but to be safe,
I have taken the search config during startup and stored
it in the SqlStore struct.
For reference, this is what happens without a search config
passed:
```
[bigdb] # explain analyze SELECT *, (SELECT COUNT(*) FROM Posts WHERE Posts.RootId = (CASE WHEN q2.RootId = '' THEN q2.Id ELSE q2.RootId END) AND Posts.DeleteAt = 0) as ReplyCount FROM Posts q2 WHERE q2.DeleteAt = 0 AND q2.Type NOT LIKE 'system_%' AND to_tsvector(Message) @@ to_tsquery('sapiente') AND ChannelId IN (SELECT Id FROM Channels, ChannelMembers WHERE Id = ChannelId AND Channels.DeleteAt = 0 AND ChannelMembe
rs.UserId = 'tc3p1yqw67d8idcp3g98awexqe' AND (TeamId = '8ywxyw9ocp8smxrmjzrkqhrdwe' OR TeamId = '') AND Id IN ('h1x9asxr7idjpqfmg8q67us49h')) ORDER BY q2.CreateAt DESC LIMIT 100;
QUERY PLAN
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Limit (cost=8200.93..8720.23 rows=9 width=382) (actual time=177.277..177.948 rows=100 loops=1)
-> Result (cost=8200.93..8720.23 rows=9 width=382) (actual time=177.274..177.939 rows=100 loops=1)
-> Sort (cost=8200.93..8200.95 rows=9 width=374) (actual time=177.245..177.271 rows=100 loops=1)
Sort Key: q2.createat DESC
Sort Method: top-N heapsort Memory: 161kB
-> Nested Loop Semi Join (cost=71.53..8200.79 rows=9 width=374) (actual time=5.910..176.815 rows=532 loops=1)
-> Bitmap Heap Scan on posts q2 (cost=70.56..8183.64 rows=9 width=374) (actual time=5.815..176.457 rows=532 loops=1)
Recheck Cond: ((channelid)::text = 'h1x9asxr7idjpqfmg8q67us49h'::text)
Filter: (((type)::text !~~ 'system_%'::text) AND (deleteat = 0) AND (to_tsvector((message)::text) @@ to_tsquery('sapiente'::text)))
Rows Removed by Filter: 5244
Heap Blocks: exact=944
-> Bitmap Index Scan on idx_posts_channel_id_update_at (cost=0.00..70.56 rows=1866 width=0) (actual time=1.883..1.883 rows=5776 loops=1)
Index Cond: ((channelid)::text = 'h1x9asxr7idjpqfmg8q67us49h'::text)
-> Materialize (cost=0.98..17.04 rows=1 width=27) (actual time=0.000..0.000 rows=1 loops=532)
-> Nested Loop (cost=0.98..17.03 rows=1 width=27) (actual time=0.080..0.080 rows=1 loops=1)
-> Index Scan using channels_pkey on channels (cost=0.42..8.45 rows=1 width=27) (actual time=0.041..0.042 rows=1 loops=1)
Index Cond: ((id)::text = 'h1x9asxr7idjpqfmg8q67us49h'::text)
Filter: ((deleteat = 0) AND (((teamid)::text = '8ywxyw9ocp8smxrmjzrkqhrdwe'::text) OR ((teamid)::text = ''::text)))
-> Index Only Scan using idx_channelmembers_user_id_channel_id_last_viewed_at on channelmembers (cost=0.56..8.58 rows=1 width=27) (actual time=0.034..0.034 rows=1 loops=1)
Index Cond: ((userid = 'tc3p1yqw67d8idcp3g98awexqe'::text) AND (channelid = 'h1x9asxr7idjpqfmg8q67us49h'::text))
Heap Fetches: 1
SubPlan 1
-> Aggregate (cost=57.68..57.69 rows=1 width=8) (actual time=0.006..0.006 rows=1 loops=100)
-> Index Only Scan using idx_posts_root_id_delete_at on posts (cost=0.56..54.44 rows=1294 width=0) (actual time=0.005..0.005 rows=3 loops=100)
Index Cond: ((rootid = (CASE WHEN ((q2.rootid)::text = ''::text) THEN q2.id ELSE q2.rootid END)::text) AND (deleteat = 0))
Heap Fetches: 0
Planning Time: 2.178 ms
Execution Time: 178.155 ms
```
We can see it using a top-N heapsort.
And this is what happens when an explicit search config
is passed:
```
[bigdb] # explain analyze SELECT *, (SELECT COUNT(*) FROM Posts WHERE Posts.RootId = (CASE WHEN q2.RootId = '' THEN q2.Id ELSE q2.RootId END) AND Posts.DeleteAt = 0) as ReplyCount FROM Posts q2 WHERE q2.DeleteAt = 0 AND q2.Type NOT LIKE 'system_%' AND to_tsvector('pg_catalog.english', Message) @@ to_tsquery('pg_catalog.english', 'sapiente') AND ChannelId IN (SELECT Id FROM Channels, ChannelMembers WHERE Id = ChannelId AND Channels.DeleteAt = 0 AND ChannelMembers.UserId = 'tc3p1yqw67d8idcp3g98awexqe' AND (TeamId = '8ywxyw9ocp8smxrmjzrkqhrdwe' OR TeamId = '') AND Id IN ('h1x9asxr7idjpqfmg8q67us49h')) ORDER BY q2.CreateAt DESC LIMIT 100;
QUERY PLAN
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Limit (cost=1.54..10679.04 rows=100 width=382) (actual time=0.237..31.405 rows=100 loops=1)
-> Nested Loop Semi Join (cost=1.54..17512.64 rows=164 width=382) (actual time=0.236..31.387 rows=100 loops=1)
-> Index Scan Backward using idx_posts_channel_id_delete_at_create_at on posts q2 (cost=0.56..8032.81 rows=164 width=374) (actual time=0.121..30.124 rows=100 loops=1)
Index Cond: (((channelid)::text = 'h1x9asxr7idjpqfmg8q67us49h'::text) AND (deleteat = 0))
Filter: (((type)::text !~~ 'system_%'::text) AND (to_tsvector('english'::regconfig, (message)::text) @@ '''sapient'''::tsquery))
Rows Removed by Filter: 1088
-> Materialize (cost=0.98..17.04 rows=1 width=27) (actual time=0.001..0.001 rows=1 loops=100)
-> Nested Loop (cost=0.98..17.03 rows=1 width=27) (actual time=0.074..0.075 rows=1 loops=1)
-> Index Scan using channels_pkey on channels (cost=0.42..8.45 rows=1 width=27) (actual time=0.032..0.032 rows=1 loops=1)
Index Cond: ((id)::text = 'h1x9asxr7idjpqfmg8q67us49h'::text)
Filter: ((deleteat = 0) AND (((teamid)::text = '8ywxyw9ocp8smxrmjzrkqhrdwe'::text) OR ((teamid)::text = ''::text)))
-> Index Only Scan using idx_channelmembers_user_id_channel_id_last_viewed_at on channelmembers (cost=0.56..8.58 rows=1 width=27) (actual time=0.041..0.041 rows=1 loops=1)
Index Cond: ((userid = 'tc3p1yqw67d8idcp3g98awexqe'::text) AND (channelid = 'h1x9asxr7idjpqfmg8q67us49h'::text))
Heap Fetches: 1
SubPlan 1
-> Aggregate (cost=57.68..57.69 rows=1 width=8) (actual time=0.010..0.010 rows=1 loops=100)
-> Index Only Scan using idx_posts_root_id_delete_at on posts (cost=0.56..54.44 rows=1294 width=0) (actual time=0.009..0.009 rows=3 loops=100)
Index Cond: ((rootid = (CASE WHEN ((q2.rootid)::text = ''::text) THEN q2.id ELSE q2.rootid END)::text) AND (deleteat = 0))
Heap Fetches: 0
Planning Time: 0.401 ms
Execution Time: 31.466 ms
```
https://mattermost.atlassian.net/browse/MM-25554
```release-note
NONE
```
* Introduced search for channel id's
* Small fix to test case as it is not looking at count
* Introduced searching channels by id as a ChannelOpt. Defaulting to true for system console for the time being
* go fmt
* Modified whether to include search by id by passing it in as prop to search endpoint. Modified webapp to pass through search by id as well
* Fixed issue with full text search, as there is not always necessarily an index on Id.
* Modified test to verify count, and moved code inside conditional
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* Improves groups membership telemetry.
* Removes unnecessary where clause (it's added by default).
* Cleans up data.
* Testing the relative group counts.
* Adds missing test mock.
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* MM-43956: Adds post counts by day.
* MM-43956: Test data cleanup. Switch from Unix to UnixMilli.
* MM-43956: Changes from selecting date data types to strings in SQL.
* MM-43956: Fixes date format key.
* MM-43956: Adds missing user id scope for 'my' top channels graph.
* MM-43956: Adds the ability to group post counts by hour.
* MM-43956: Require enterprise or professional license. Reject guests.
* MM-43956: Adds license for tests.
* MM-43956: Renames function.
* MM-43956: Omits future hours from post counts by hour.
* MM-43956: Adjust API response grouping to users timezone.
* MM-43956: Adds translation.
* MM-43956: Adds user's timezone to the data tier for the grouping by day and hour.
* MM-43956: Fixes layers.
* MM-43956: Lint fix.
* MM-43956: Fix store layers.
* MM-43956: Switches to default name for time package; changes parameter names to avoid naming conflict.
* MM-43956: Updates mocks.
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* [MM-44329] Added methods to sqlstore.Store
* [MM-44329] Added methods to the store.Store interface
* [MM-44329] Regenerated mocks
* [MM-44329] Added comments for the store.Store interface
Co-authored-by: Tim Scheuermann <tim.scheuermann@mattermost.com>
* store: guests will only receive results of which channels they are in
* api4/channel_test: add test case for guest accounts channel autocomplete
* apply to searchengine and also for AutocompleteInTeam
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
A deadlock that has been found on our Community servers is:
```
------------------------
LATEST DETECTED DEADLOCK
------------------------
2022-03-03 06:14:37 0x2aec58887700
*** (1) TRANSACTION:
TRANSACTION 1043681684, ACTIVE 0 sec starting index read
mysql tables in use 1, locked 1
LOCK WAIT 23 lock struct(s), heap size 1136, 137 row lock(s), undo log entries 26
MySQL thread id 448011, OS thread handle 47195956565760, query id 752406931 10.128.147.211 mmcloud updating
DELETE FROM SidebarChannels WHERE ((ChannelId IN (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?) OR ChannelId IN (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)) AND CategoryId = ?)
*** (1) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 907 page no 368 n bits 0 index PRIMARY of table `mattermost`.`SidebarChannels` trx id 1043681684 lock_mode X waiting
Record lock, heap no 121 PHYSICAL RECORD: n_fields 6; compact format; info bits 0
0: len=26; bufptr=0x2aebbd36b75c; hex= 3363776f6a6a6d36316a62663967657036706872316533696a77; asc 3cwojjm61jbf9gep6phr1e3ijw;;
1: len=26; bufptr=0x2aebbd36b776; hex= 3568616279726d36746a6265747061656538786973786d6b7177; asc 5habyrm6tjbetpaee8xisxmkqw;;
2: len=26; bufptr=0x2aebbd36b790; hex= 373568726a7a68676169726370653477696877727866716f7463; asc 75hrjzhgaircpe4wihwrxfqotc;;
3: len=6; bufptr=0x2aebbd36b7aa; hex= 00003e355176; asc >5Qv;;
4: len=7; bufptr=0x2aebbd36b7b0; hex= 39000352420fce; asc 9 RB ;;
5: len=8; bufptr=0x2aebbd36b7b7; hex= 80000000000000fa; asc ;;
*** (2) TRANSACTION:
TRANSACTION 1043681679, ACTIVE 0 sec starting index read
mysql tables in use 1, locked 1
LOCK WAIT 76 lock struct(s), heap size 1136, 294 row lock(s), undo log entries 72
MySQL thread id 447990, OS thread handle 47194812454656, query id 752406930 10.128.147.211 mmcloud updating
DELETE FROM SidebarChannels WHERE ((ChannelId IN (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?) OR ChannelId IN (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)) AND CategoryId = ?)
*** (2) HOLDS THE LOCK(S):
RECORD LOCKS space id 907 page no 368 n bits 0 index PRIMARY of table `mattermost`.`SidebarChannels` trx id 1043681679 lock_mode X
Record lock, heap no 121 PHYSICAL RECORD: n_fields 6; compact format; info bits 0
0: len=26; bufptr=0x2aebbd36b75c; hex= 3363776f6a6a6d36316a62663967657036706872316533696a77; asc 3cwojjm61jbf9gep6phr1e3ijw;;
1: len=26; bufptr=0x2aebbd36b776; hex= 3568616279726d36746a6265747061656538786973786d6b7177; asc 5habyrm6tjbetpaee8xisxmkqw;;
2: len=26; bufptr=0x2aebbd36b790; hex= 373568726a7a68676169726370653477696877727866716f7463; asc 75hrjzhgaircpe4wihwrxfqotc;;
3: len=6; bufptr=0x2aebbd36b7aa; hex= 00003e355176; asc >5Qv;;
4: len=7; bufptr=0x2aebbd36b7b0; hex= 39000352420fce; asc 9 RB ;;
5: len=8; bufptr=0x2aebbd36b7b7; hex= 80000000000000fa; asc ;;
[bitmap of 256 bytes in hex: 00 00 00 00 00 00 00 f0 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ]
*** (2) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 907 page no 67 n bits 0 index PRIMARY of table `mattermost`.`SidebarChannels` trx id 1043681679 lock_mode X waiting
Record lock, heap no 103 PHYSICAL RECORD: n_fields 6; compact format; info bits 0
0: len=26; bufptr=0x2aeb4515ec1b; hex= 6431376b35706f71786979776d6b6d616b6a78623539356b366f; asc d17k5poqxiywmkmakjxb595k6o;;
1: len=26; bufptr=0x2aeb4515ec35; hex= 7067653939686a356562676b38636f6f70637a6e7871677a6568; asc pge99hj5ebgk8coopcznxqgzeh;;
2: len=26; bufptr=0x2aeb4515ec4f; hex= 336e3731636a73656f667931397267746664337962786d713777; asc 3n71cjseofy19rgtfd3ybxmq7w;;
3: len=6; bufptr=0x2aeb4515ec69; hex= 00003e34fa4d; asc >4 M;;
4: len=7; bufptr=0x2aeb4515ec6f; hex= 4600026d940e4c; asc F m L;;
5: len=8; bufptr=0x2aeb4515ec76; hex= 8000000000000082; asc ;;
*** WE ROLL BACK TRANSACTION (1)
```
It seems like 2 DELETE queries are somehow deadlocking on the same row.
On looking at it deeply, we can see that in an updateSidebarCategory scenario,
we run 2 delete queries in a single transaction. In that delete query, we not only delete the existing channels for a category,
but also delete the new channels that are going to be inserted in the new category.
The potential issue with this is that since we are updating both the source and destination category in a single transaction,
the category which is moved appears in the destination section for the first category, and in the source section for the second category.
Which means in a single transaction, you have 2 queries possibly interlocking due to the same channels.
Therefore, if we run 2 separate updateCategories with an inverted order of categories, then the same set of delete queries
can lock on the opposite order of channels and cause a deadlock.
I haven't been able to reproduce this, but just a hunch. And in any case, this will reduce the number of DB updates.
```release-note
NONE
```
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* Added Archived column to FileInfo
* fixed typo
* Added new column to store column list
* Fixing tests
* Updated migration number to syncup with master
Just a quick POC to move fast :P
We use a search pointer to keep track of
the next row to inesrt to. For every new search
we increment the pointer and do modulo 5.
This means that the value will always remain
between 0-4. And that way, we will always overwrite
the oldest entry on every search.
And while getting the results, we get
all results for that user.
The search parameters are json marshalled
and stored as a JSON blob. This is because
there is no need to search/filter them
in the DB.
Pending items:
Tests obviously.
To improve:
The client needs to send the channel ids
instead of channel names.
We add 2 new params to channel members query.
1. Filter by teamId.
2. Negate that filter.
We include some more optimizations like:
- Moved the team role checks inside the dataloader.
- Moved the channel pretty name computation inside the loader.
Now that we load less data on initial load, we can reduce
the concurrency requirement to be a bit on the safer side.
```release-note
NONE
```
Old versions of the Mattermost server did not qualify queries scanning both `Posts` and `Threads`, and choke on the ambiguity in deciding between the new `DeleteAt` on `Threads` and the `DeleteAt` on `Posts` in existing queries.
While this problem is transient only while running multiple server versions, it effectively makes our backwards compatibility guarantee void, not to mention complicating cloud deployments.
Work around this by renaming `Threads.DeleteAt` to `Threads.ThreadDeleteAt`. The old migration is nulled out, but remains, since some test servers have already upgraded and manually fixing each affected instance would be problematic. Thew new migration takes care of removing the old column -- if it ever existed.
Fixes: https://mattermost.atlassian.net/browse/MM-43770
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* [MM-42739] Initial setup for top channels for team
* [MM-42739] Add initial tests
* [MM-42739] Update tests
* [MM-42739] Add top channels for user
* [MM-42739] Fix query
* [MM-42739] Update query
* [MM-42739] Improve query performance
* [MM-42739] Remove rank
* [MM-42739] Fix tests to use new time range today
* [MM-42739] Add tests for top channels for user
* [MM-42739] Add test for pagination
* Remove top channels by time struct
* [MM-42739] Update test names
* [MM-42739] Remove rank from top reactions
* [MM-42739] Return empty array instead of nil when result is empty
* [MM-42739] Add additional tests and update permissions check for teams
* [MM-42739] Add excluded channel tests for top reactions
* [MM-42739] Move insights to api4/insights and keep time range as string until required
* [MM-42739] Update queries only check DeleteAt after union
* [MM-42739] Improve query performance by using publicchannels table
* [MM-42739] Fix broken query after merge
* removed appending the root post to the posts list
also changed `UpdateAt` to `CreateAt` in thread_store.go in accordance with kyriakos
* fixed failing test after latest change
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
