mirrors/mostlymatter
1
0
Форкнуть 0
Код Задачи Запросы на слияние Действия Пакеты Проекты Релизы Вики Активность
15 496 коммитов 3 Ветки 0 Теги
df62328291756d443be588d29b5d2691e809bedf
Граф коммитов

1364 Коммитов

Автор SHA1 Сообщение Дата
Agniva De Sarker
10a627fac1 MM-45723: Include DM/GM channels for user indexing (#21391) 
There were 2 separate issues here:

1. We were excluding DM/GM channels for user indexing.
Not sure why. This prevented users to be shown as
channel members in autocomplete results in both
Bleve and Elasticsearch.

2. We were incorrectly deciding to search across
the team based on if the current channel
is in the list of allowed channels or not. But we
were incorrectly wiping the list of allowed channels
even if a channelid was passed. This makes it work
same as a DB search.

One catch is that for the correct results to appear,
a re-indexing is required.

```release-note
Autocompletion results using Elasticsearch or Bleve will
correctly show a user as a channel member in DM/GM channels.
To force this change to appear, a re-indexing will be
necessary.
```
 2022-10-19 20:51:00 +05:30
Shivashis Padhi
28698abb9f Apply role filters for /get/users in channel (#21251) 2022-10-19 09:50:41 +05:30
Julien Tant
3747d99806 [MM-46694] A/B Test: welcome post (#20926) 
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
 2022-10-13 14:23:22 -07:00
Rob Bierbooms
66b7c45e69 [MM-47367] Make sure mini preview is saved to database when updating database (#21304) 2022-10-12 15:19:00 +02:00
Daniel Espino García
79651874ea Do not send categories for teams deleted or not member of (#21038) 
* Do not send categories for teams deleted or not member of

* Fix tests and add test for deleted teams and memberships
 2022-10-11 15:33:29 +02:00
Vishal
6e9b808efd [MM-46644] Auto respond message (#20900) 
Automatic Merge
 2022-10-07 16:02:11 +03:00
Alejandro García Montoro
70c74fc10d MM-47374: Revert full text search (#21233) 
* Revert "MM-46871: Add remining search parameters to be escaped (#20963)"

This reverts commit 8797bfcde7.

* Revert "MM-46503: Escape incorrect pg user search query (#20863)"

This reverts commit 8fd1762c3b.

* Revert "MM-44576 autocomplete names including utf 8 chars (#20367)"

This reverts commit 8444c45959.
 2022-10-06 17:02:20 +02:00
Ibrahim Serdar Acikgoz
5e69c6b02f Move cluster, webhub and store out of Server (#20899) 2022-10-06 11:04:21 +03:00
Alejandro García Montoro
635cea19c7 MM-46105: Simplify posts batch retrieval query (#21149) 
In MySQL, tested in Community, times for the old and new queries are
mostly the same:
- Original : 6 s 250 ms (execution: 258 ms, fetching: 5 s 992 ms)
- New      : 6 s 148 ms (execution: 148 ms, fetching: 6 s)

The output from EXPLAIN in Community is also similar for both queries:

- Original:

| id | select_type | table      | partitions | type   | possible_keys          | key                    | key_len | ref                  | rows    | filtered | Extra                 |
|----|-------------|------------|------------|--------|------------------------|------------------------|---------|----------------------|---------|----------|-----------------------|
|  1 | PRIMARY     | <derived2> |            | ALL    |                        |                        |         |                      |   10000 |   100.00 | Using filesort        |
|  1 | PRIMARY     | Channels   |            | eq_ref | PRIMARY                | PRIMARY                | 106     | PostsQuery.ChannelId |       1 |   100.00 |                       |
|  2 | DERIVED     | Posts      |            | range  | idx_posts_create_at_id | idx_posts_create_at_id | 115     |                      | 3108048 |   100.00 | Using index condition |

- New:

| id | select_type | table    | partitions | type   | possible_keys          | key                    | key_len | ref                        | rows    | filtered | Extra                 |
|----|-------------|----------|------------|--------|------------------------|------------------------|---------|----------------------------|---------|----------|-----------------------|
|  1 | SIMPLE      | Posts    |            | range  | idx_posts_create_at_id | idx_posts_create_at_id | 115     |                            | 3108050 |   100.00 | Using index condition |
|  1 | SIMPLE      | Channels |            | eq_ref | PRIMARY                | PRIMARY                | 106     | mattermost.Posts.ChannelId |       1 |   100.00 |                       |

In Postgres 12, this was tested locally with a 12M posts database, in a
machine with the following specs:
- CPU: i7-11800H (8C / 16T, 2.3 / 4.6GHz, 24MB)
- Memory: 32GB
- Storage: 1TB SSD M.2 2280 PCIe 4.0x4 Performance NVMe Opal2

The times of the queries in this setup are:
- Original query : 118.080 ms
- New query      : 94.454 ms

The complete output from EXPLAIN ANALYZE in Postgres 12:

- Original query:

mattermost=> EXPLAIN ANALYZE
SELECT PostsQuery.*, Channels.TeamId
FROM (
    SELECT *
    FROM
    Posts
    WHERE Posts.CreateAt > 1629244800000
        OR (Posts.CreateAt = 1629244800000 AND Posts.Id > '')
    ORDER BY CreateAt ASC, Id ASC
    LIMIT 10000
) AS PostsQuery
LEFT JOIN Channels ON PostsQuery.ChannelId = Channels.Id
ORDER BY CreateAt ASC, Id ASC;
                                                                                QUERY PLAN
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 Sort  (cost=156636.28..156661.28 rows=10000 width=375) (actual time=114.093..114.833 rows=10000 loops=1)
   Sort Key: posts.createat, posts.id
   Sort Method: external merge  Disk: 3128kB
   ->  Hash Right Join  (cost=136335.48..155971.89 rows=10000 width=375) (actual time=67.919..107.330 rows=10000 loops=1)
         Hash Cond: ((channels.id)::text = (posts.channelid)::text)
         ->  Seq Scan on channels  (cost=0.00..10902.68 rows=272168 width=28) (actual time=0.018..28.826 rows=272168 loops=1)
         ->  Hash  (cost=135721.48..135721.48 rows=10000 width=374) (actual time=40.141..40.144 rows=10000 loops=1)
               Buckets: 16384  Batches: 2  Memory Usage: 2244kB
               ->  Limit  (cost=135596.48..135621.48 rows=10000 width=374) (actual time=36.799..38.169 rows=10000 loops=1)
                     ->  Sort  (cost=135596.48..135704.93 rows=43380 width=374) (actual time=30.641..31.699 rows=10000 loops=1)
                           Sort Key: posts.createat, posts.id
                           Sort Method: external merge  Disk: 18840kB
                           ->  Bitmap Heap Scan on posts  (cost=827.91..132497.48 rows=43380 width=374) (actual time=1.856..8.630 rows=54398 loops=1)
                                 Recheck Cond: ((createat > '1629244800000'::bigint) OR (createat = '1629244800000'::bigint))
                                 Filter: ((createat > '1629244800000'::bigint) OR ((createat = '1629244800000'::bigint) AND ((id)::text > ''::text)))
                                 Heap Blocks: exact=2674
                                 ->  BitmapOr  (cost=827.91..827.91 rows=43380 width=0) (actual time=1.660..1.661 rows=0 loops=1)
                                       ->  Bitmap Index Scan on idx_posts_create_at  (cost=0.00..801.78 rows=43379 width=0) (actual time=1.657..1.657 rows=54398 loops=1)
                                             Index Cond: (createat > '1629244800000'::bigint)
                                       ->  Bitmap Index Scan on idx_posts_create_at  (cost=0.00..4.44 rows=1 width=0) (actual time=0.002..0.002 rows=0 loops=1)
                                             Index Cond: (createat = '1629244800000'::bigint)
 Planning Time: 0.230 ms
 JIT:
   Functions: 14
   Options: Inlining false, Optimization false, Expressions true, Deforming true
   Timing: Generation 0.562 ms, Inlining 0.000 ms, Optimization 0.320 ms, Emission 5.863 ms, Total 6.745 ms
 Execution Time: 118.080 ms
(27 rows)

- New query:

mattermost=> EXPLAIN ANALYZE
SELECT Posts.*, Channels.TeamId
FROM
Posts
LEFT JOIN Channels ON Posts.ChannelId = Channels.Id
WHERE Posts.CreateAt > 1629244800000
    OR (Posts.CreateAt = 1629244800000 AND Posts.Id > '')
ORDER BY Posts.CreateAt ASC, Posts.Id ASC
LIMIT 10000;
                                                                             QUERY PLAN
--------------------------------------------------------------------------------------------------------------------------------------------------------------------
 Limit  (cost=148443.93..149610.68 rows=10000 width=375) (actual time=83.610..92.492 rows=10000 loops=1)
   ->  Gather Merge  (cost=148443.93..152661.73 rows=36150 width=375) (actual time=74.409..82.975 rows=10000 loops=1)
         Workers Planned: 2
         Workers Launched: 2
         ->  Sort  (cost=147443.91..147489.10 rows=18075 width=375) (actual time=64.325..64.629 rows=3436 loops=3)
               Sort Key: posts.createat, posts.id
               Sort Method: external merge  Disk: 5992kB
               Worker 0:  Sort Method: external merge  Disk: 7144kB
               Worker 1:  Sort Method: external merge  Disk: 7152kB
               ->  Parallel Hash Left Join  (cost=12336.48..146152.66 rows=18075 width=375) (actual time=42.908..51.116 rows=18133 loops=3)
                     Hash Cond: ((posts.channelid)::text = (channels.id)::text)
                     ->  Parallel Bitmap Heap Scan on posts  (cost=827.91..132054.64 rows=18075 width=374) (actual time=2.448..5.417 rows=18133 loops=3)
                           Recheck Cond: ((createat > '1629244800000'::bigint) OR (createat = '1629244800000'::bigint))
                           Filter: ((createat > '1629244800000'::bigint) OR ((createat = '1629244800000'::bigint) AND ((id)::text > ''::text)))
                           Heap Blocks: exact=902
                           ->  BitmapOr  (cost=827.91..827.91 rows=43380 width=0) (actual time=2.111..2.112 rows=0 loops=1)
                                 ->  Bitmap Index Scan on idx_posts_create_at  (cost=0.00..801.78 rows=43379 width=0) (actual time=2.105..2.105 rows=54398 loops=1)
                                       Index Cond: (createat > '1629244800000'::bigint)
                                 ->  Bitmap Index Scan on idx_posts_create_at  (cost=0.00..4.44 rows=1 width=0) (actual time=0.004..0.005 rows=0 loops=1)
                                       Index Cond: (createat = '1629244800000'::bigint)
                     ->  Parallel Hash  (cost=9315.03..9315.03 rows=113403 width=28) (actual time=29.928..29.929 rows=90723 loops=3)
                           Buckets: 65536  Batches: 8  Memory Usage: 2688kB
                           ->  Parallel Seq Scan on channels  (cost=0.00..9315.03 rows=113403 width=28) (actual time=5.378..16.495 rows=90723 loops=3)
 Planning Time: 0.460 ms
 JIT:
   Functions: 46
   Options: Inlining false, Optimization false, Expressions true, Deforming true
   Timing: Generation 2.291 ms, Inlining 0.000 ms, Optimization 1.518 ms, Emission 23.827 ms, Total 27.636 ms
 Execution Time: 94.454 ms
(29 rows)
 2022-09-30 18:18:26 +02:00
Michael Kochell
15b5b1c191 Avoid counting top channel posts for posts made by plugins and OAuth apps (#20943) 
* add from_integration prop to oauth posts to:
- oauth app posts
- plugin posts
- slash command responses
- incoming webhook posts

* tests

* include check for bot posts

* use from_plugin and from_oauth_app props

* fix test

* avoid counting top channel posts for posts made by plugins and oauth apps
 2022-09-30 04:12:15 -04:00
Vishal
f5f036d94b [MM-44489] Cloud limits: enforcing files (#20703) 
* Update last accessible file time

* Filter fileInfos

* Set inaccessible header

* Fix lint issue

* Fix lint issue

* Fix i18n

* add nil check

* Fix merge conflicts

* Add helper functions to clear out inaccessible files content

* Remove content for inaccessible files

* Fix typo

* wip

* Remove InaccessibleContent field, instead use Archived

* Add store tests

* Add tests

* Add separate funcs to ignore cloud limits

* Use separate query for MySql

* Use GetReplicaX

Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
 2022-09-28 12:52:53 -04:00
Orlando Romo
b9834a2fc2 [MM-42191]: Include deleted posts (#19985) 
* MM-42191: Include deleted posts: Add includeDeleted query parameter for getPostsForChannel

* MM-42191: Fix error typo for includeDeleted query parameter

* MM-42191: Include deleted posts: Set permission error when deleted posts are requested by non system admins

* MM-42191: Include deleted posts: Refactor replyCountSubQuery and conditions when includeDeleted is not presented, refactor getRootPosts

* MM-42191: Include deleted posts: Refactor getRootPosts function along with skipFetchThreads and includeDeleted

* MM-42191: Include deleted posts: Rename includeDeleted to include_deleted param

* MM-42191: Include deleted posts: Fix failed posts unit tests

* MM-42191: Include deleted posts: Add missing include deleted option in multiple queries

* MM-42191: Include deleted posts: Add tests for include deleted option in TestGetPostsForChannel, TestGetPostsBefore, TestGetPostsAfter

* MM-42191: include deleted posts: Add tests cases for post store test

* MM-42191: Include deleted posts: Add extra unit test to ensure not returning deleted posts when IncludeDelete is false

Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
 2022-09-27 14:00:42 -04:00
Shivashis Padhi
2ea14ef395 [MM-47002] Fix new private channels not showing up in least active channels insights (#21031) 
Automatic Merge
 2022-09-23 17:12:24 +03:00
Anurag Shivarathri
203c6a5013 Badge count fix for push notifications when CRT is enabled (#20898) 
* Fix

* Fixed other cases and tests

Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
 2022-09-12 14:51:33 +05:30
Shivashis Padhi
38aaa9e3d3 [MM-46603] P1 - Improvements in handling '0 posts' channels (#20905) 
Automatic Merge
 2022-09-09 20:34:00 +03:00
Agniva De Sarker
8797bfcde7 MM-46871: Add remining search parameters to be escaped (#20963) 
The first try wasn't exhaustive. I was planning to use
1f933263e7/store/sqlstore/post_store.go (L1738-L1748) for this
but it also contained `@` which we don't want.

In the end, I just used a separate slice for all
the characters to be escaped.

https://mattermost.atlassian.net/browse/MM-46871

```release-note
NONE
```

Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
 2022-09-09 21:21:34 +05:30
Shivashis Padhi
8b328386c5 MM-46911: P1 - Fix MySQL query to filter bots out of TopDM (#20965) 
Automatic Merge
 2022-09-09 18:34:00 +03:00
Vishal
4bc2cc4b0f Add Limit (#20762) 
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
 2022-09-08 19:29:13 +05:30
Shivashis Padhi
4ffdf6b859 [MM-46667] P1 Fix top DM insights pagination (#20896) 
Automatic Merge
 2022-09-07 23:34:01 +03:00
Shivashis Padhi
8f743c37b6 [MM-46516] P1 - Fix total message counting for top DM (#20858) 
Automatic Merge
 2022-09-07 01:34:01 +03:00
Allan Guwatudde
bd42f0cd8c [MM-45564] - Notify Admin v2 (#20777) 
* [MM-45564] - Notify Admin v2

* add dummy data

* update dummy data

* add store methods

* experiment with recurring task

* complete saving of the notification

* make improvements

* make improvements

* add store layer tests

* fix lint

* update store layer tests

* add app layer unit tests

* add store layers

* add app layer tests

* fix lint

* fix lint

* fix tests

* fix tests lint

* fix lint

* fix lint

* fix retry layer test

* add notifications manual trigger

* filter notifications based on current plan

* add test case

* temp change

* feedback impl

* fix translations

* change job scheduler

* refactor job

* fix store layer tests

* extract i18n

* fix lint

* fix translations

* fix translations

* add license statement for new file

* feedback impl-2

* fix lint

* update make file

* add intl ids

* improve

* fix lint

* feedback impl

* move code and rename files

* fix lint

* feedback impl

* add config for trigger notifications api

* fix tests

* tmp change

* undo temp changes

Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
 2022-09-02 13:52:48 +03:00
Julien Tant
a5ea445bf9 [MM-46097] Change FileInfo count query (#20871) 
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
 2022-08-30 09:49:39 -07:00
Tim Scheuermann
eb37139f16 MM-45994 ensure database operations return their errors (#20857) 2022-08-26 11:12:59 +02:00
Shivashis Padhi
6adbcc5d05 MM-45899: Insights: least active channels (#20796) 
* Add api endpoints, app layers for top inactive channels with dummy store calls

* Add store functions for top inactive channels

* Add model, store, app tests.

* Add client function and api tests

* Add participants information to TopInactiveChannel

* Translation fix

* Style fix while writing response

* Return channelmember IDs instead of profiles, query in batch avoiding inside the loop

* Make the following changes

 - move DeleteAt to subqueries, to avoid select, group by
 - Remove TeamId from response
 - Count bots and webhook posts

* SQL query lint fix, store test fix to include bot messages

* make app-layers

* Fix empty participant lists being sent as [""]

* Track channel joins, to distinguish 0 activity channels vs new channels

Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
 2022-08-24 23:14:56 +05:30
Agniva De Sarker
8fd1762c3b MM-46503: Escape incorrect pg user search query (#20863) 
There were 2 main problems after https://github.com/mattermost/mattermost-server/pull/20367.

1. The : wasn't escaped.
2. Empty search didn't work.

For 1, we escape the ':'. For 2, we just use
pattern search.

```release-note
NONE
```
 2022-08-24 21:45:52 +05:30
Agniva De Sarker
492a3c0669 Warn on MariaDB servers (#20862) 
```release-note
NONE
```
 2022-08-23 14:17:45 +05:30
Shivashis Padhi
582812f1fc Merge branch 'master' of github.com:mattermost/mattermost-server into top-dms-clean 2022-08-12 17:00:55 +05:30
Martin Kraft
55b3961b98 MM-45120: Track team join times. Add API to retrieve new team members since a given time. (#20708) 
* MM-45120: Starts tracking team join time. Adds API to retrieve team members who joined after a given time.

* MM-45120: Updates json casing to match model.User.
 2022-08-11 10:38:52 -04:00
Shivashis Padhi
4ec3eade3b Merge branch 'master' of github.com:mattermost/mattermost-server into top-dms-clean 2022-08-10 21:00:57 +05:30
Mattermod
4fc8ef0125 Merge branch 'master' into MM-45118_my_top_dms 2022-08-05 09:56:52 +03:00
Shivashis Padhi
3e36894ac7 Avoid joining with Posts table while getting top user reactions 2022-08-03 15:43:09 +05:30
Shivashis Padhi
45e434cc26 Fix issue with pagination where has_next is false for every case 2022-08-03 13:28:03 +05:30
Agniva De Sarker
334b199e7a Optimize AnalyticsPostCount (#20727) 
We use * instead of a column name to use
index-only scan always even when other column filters
are applied.

Right now, index-only scan will only get applied
in the basic query of "select count(p.id) as value
from posts p". But it won't get applied if the query
is "select count(p.id) as value from posts p where
p.deleteat=0".

So this is a minor optimization which improves
some corner cases.

This was found from the slow query monitoring.

```release-note
NONE
```
 2022-08-03 10:31:18 +05:30
Vishal
fa768e0fa7 [MM-45726] Fix Sentry crash: nil deference in app/post_helpers.go:85 (#20654) 
* Avoid adding duplicate post
 2022-08-02 12:10:26 +05:30
Dimitris Oik
8444c45959 MM-44576 autocomplete names including utf 8 chars (#20367) 
* Added unaccent extension

* Changed comment

* Deletes migrations, changes query method

* Mionr changes

* Creates test for multilingual queries

* Minor formatting changes

* Adds two more tests

* Changes name in test function

* Minor change

* Retriggers tests

* Removes % from Postgres query

* Lint fix

* Changes variable name

* Removes SetDefaultTextSearchConfig method

* Removes mocks

* Adds error handling on test raw queries

* Lint fix

* Deletes unused generated file

Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
 2022-08-01 06:58:31 -04:00
Shivashis Padhi
61a716a98c Fix Posts query while populating top DMs, add OutgoingMessageCount check to tests 2022-08-01 13:58:50 +05:30
Kyriakos Z
562388501e MM-43939: fixes LastReplyAt when deleting the last reply (#20615) 
* MM-43939: fixes lastreplyat when deleting the last reply

Currently we are not updating the Threads.LastReplyAt when the last
reply gets deleted. This can lead to threads appearing unread when
actually there is no unread thread.

This commit updates the value of Threads.LastReplyAt when a reply gets
deleted, to the most recent post's timestamp in the thread.

* Updates ReplyCount to current value on post delete

* Addresses review comments

Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
 2022-07-29 10:03:52 -04:00
Shivashis Padhi
f84a6ae7ac Add TotalMsgCount to GROUP BY fields 2022-07-29 13:05:05 +05:30
Shivashis Padhi
9f468cf011 Select total message count from Channel instead of counting posts 2022-07-28 17:22:14 +05:30
Shivashis Padhi
57e25b5d8f Add information on individual message count 2022-07-28 16:36:37 +05:30
Shivashis Padhi
f007d94155 Ignore self DMs 2022-07-28 12:46:18 +05:30
Shivashis Padhi
55bf7458d7 Temporarily add second participant information for self dms 2022-07-28 12:35:17 +05:30
Shivashis Padhi
e94532c862 SecondParticipant now has extended user object 2022-07-27 18:41:18 +05:30
Shivashis Padhi
849aea452c Merge branch 'master' of github.com:mattermost/mattermost-server into top-dms-clean 2022-07-27 17:03:34 +05:30
Agniva De Sarker
20cb042362 Hackathon: Post Reminders (#20555) 
This PR adds the post reminder backend work.

We add a new API endpoint via which a user can set a reminder for a post. An ephemeral message will be sent down the line to let the user know about the action. And then after the time is over, the system admin bot will send a DM message to the user about the reminder post.
 2022-07-26 16:12:56 +05:30
Shivashis Padhi
870d86ae1d Save users to db in storetests, lint fix 2022-07-25 16:38:46 +05:30
Shivashis Padhi
b3330b1eb0 Fix boolean check of user.IsBot 2022-07-25 16:09:44 +05:30
Shivashis Padhi
5f56d43d80 Make the following changes 
- Use db userId instead of referring to session
 - Filter out bot DM channels, and add relevant test
 2022-07-25 14:07:08 +05:30
Shivashis Padhi
2b66652f69 Fix top dms query, add storetests 2022-07-21 18:41:41 +05:30
Shivashis Padhi
a8fa09f946 Fix counting of posts, remove redundant limit offset for queries 2022-07-21 18:39:23 +05:30