* MM-48614: saves priority for drafts
Adds a new column in the drafts table, "priority".
This way we can save post's priority in the draft.
Fixes OmitConnectionId, which when you published a ws event for a user
was getting bypassed.
Fixes Get for drafts returns deleted ones as well, which is needed for
upsert.
* Adds test case for the OmitConnectionId
* Addresses review comments, removes DeleteAt
* Vets
* Adds missing translation
* Re-instates DeleteAt column
* Adds separate case to get draft including deleted
* Fixes Update Draft
* Empty
* MM-23881: global drafts endpoints and ws events
Adds endpoints:
- create/update drafts
- delete draft
- get drafts
Adds WS events:
- draft_updated
- draft_created
- draft_deleted
* Ordering and WS event name fixes
* Adds PostID to the drafts table
In the future the drafts will include edited posts, this commit adds the
post id in the combined pkey of the table.
* Fixes route for deleting a thread draft
* Fixes failed checks
* Fixes migrations
* Fixes migration
* Extract translation strings
* Removes PostID since we won't sync editing posts
* Fixes tests
* Fixes i18n
* Update migrations for global drafts
* update branch with latest master changes
* Add feature flag for global drafts
* Set global drafts feature flag default to true
* Added support for files in drafts
* Fix failing i18n check
* Added support for deleting files in drafts
* Revert "Added support for deleting files in drafts"
This reverts commit 45dfd04a760359de2e8814d652c9ef46daf994f6.
* Triggering new test server
* Add config setting 'AllowSyncedDrafts' for syncing drafts with server
* Triggering new test server
* Triggering new test server
* Add guard for config setting and add initial tests
* Fix i18n and lint errors
* Triggering new test server
* Add tests for drafts
* fix lint issues
* Add tests for model/draft
* Triggering new test server
* Triggering new test server
* Trigger new test server
* Address PR comments
* Change left join to regular join in GetDraftsForUser
* Fix broken test
Maybe consider adding an inclDeleted field if we want to get deleted drafts in the future
* fix translations
* Add store tests for drafts
* fix test naming
* remove comment
* update migrations
* set feature flag default to false
* update migrations
Co-authored-by: Mylon Suren <mylonsuren@gmail.com>
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* MM-46410: adds urgency on mention counts
We have introduced priority for posts in
https://github.com/mattermost/mattermost-webapp/pull/10951.
We do need to color the mention badges in the webapp with a prominent
color when a mention is posted in an urgent message.
A thread has urgent mentions if the root post is marked as urgent, and
the replies contain mentions to the user viewing the thread.
This PR adds two columns, urgentmentioncount, and isurgent, in
channelmembers, and threads tables respectively.
Furthermore when asking for team/thread mention counts, we also return
urgent mention counts for the user.
* Fixes method in tests
* empty commit
* Fixes method call
* Fixes single thread response is_urgent
* Fixes errors
* Fixes mysql migration and adds graphql schema
* Fixes tests
* Refactors IsUrgent and Adds PostsPriority table
Changes:
- removes is_urgent from the threads table
- adds a new table to hold posts priorities
- refactors priority out of the props and into the new table
* Fixes
* Adds translation strings
* Fixes migrations and tests
* Fixes tests
* empty
* Adds Priority to Copy
* empty
* Fixes priority not saved when boards is enabled
We are nilifying Metadata when post.ForPlugin(), which didn't save Priority
for a post when Boards was enabled.
This commit copies metadata again to the post, so metadata are
reinstated.
* Fixes tests
* Adding store tests and fixes syntax error
* Uses threads.ThreadTeamId
* Fixes error
* Adds UrgentMentionCount in graphql api test
* Fetches post priority in batches
* Addresses review comments
* Restore only priority on create post
* Fixes tests
* Nits
* Some refactoring
* Fixes get thread options when post priority enabled
* Adds missing translation
* Use the constant instead of "urgent" string
* Renames urgent constant
* MM-47750: Adds PostAcknowledgements table and apis
- Adds post acknowledgement api/app/store methods to be able to save and
delete post acknowledgements by users.
- Adds wesbsocket events for acknowledgement created/deleted
- Returns post acknowledgements in the post's metadata
* Empty
* Fixes incorrect urgent count when marking a post as unread
* Adds license
* Fixes ACK api, and adds tests
* Fixes vet
* Fixes tests
* Addresses review comments
* Remove unnecessary lines
* Adds config option and changes return of delete ack
* Empty
* Empty
* Enable config by default
* Fixes intl
* Fixes test after setting config default true
* Changes endpoints to PostForUser
* Avoids replica lag
* Fixes error in merge
* Fixes RetryLayer tests due to merge
* Empty
* Empty
* Empty
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* MM-46410: adds urgency on mention counts
We have introduced priority for posts in
https://github.com/mattermost/mattermost-webapp/pull/10951.
We do need to color the mention badges in the webapp with a prominent
color when a mention is posted in an urgent message.
A thread has urgent mentions if the root post is marked as urgent, and
the replies contain mentions to the user viewing the thread.
This PR adds a column, urgentmentioncount, in channelmembers.
Furthermore when asking for team/thread mention counts, we also return
urgent mention counts for the user.
Adds a new table to hold posts priorities
Refactors priority out of the props and into the new table
We are nilifying Metadata when post.ForPlugin(), which didn't save Priority
for a post when Boards was enabled.
This commit copies metadata again to the post, so metadata are
reinstated.
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
Co-authored-by: Vishal Choudhary <vish9812@gmail.com>
* Add api endpoints, app layers for top inactive channels with dummy store calls
* Add store functions for top inactive channels
* Add model, store, app tests.
* Add client function and api tests
* Add participants information to TopInactiveChannel
* Translation fix
* Style fix while writing response
* Return channelmember IDs instead of profiles, query in batch avoiding inside the loop
* Make the following changes
- move DeleteAt to subqueries, to avoid select, group by
- Remove TeamId from response
- Count bots and webhook posts
* SQL query lint fix, store test fix to include bot messages
* make app-layers
* Fix empty participant lists being sent as [""]
* Track channel joins, to distinguish 0 activity channels vs new channels
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* MM-45120: Starts tracking team join time. Adds API to retrieve team members who joined after a given time.
* MM-45120: Updates json casing to match model.User.
This PR adds the post reminder backend work.
We add a new API endpoint via which a user can set a reminder for a post. An ephemeral message will be sent down the line to let the user know about the action. And then after the time is over, the system admin bot will send a DM message to the user about the reminder post.
* Add new Job to keep updating the last_accessible_post time
* Filter out posts for funcs returning PostList model
* Separate methods to get and compute cache
* filter pinned posts
* For posts with sorted CreateAt order, support a faster form of filtering.
* Add inaccessible header for getPost and getPostsByIDs APIs
* replace manual binary search with the std. library
* in-place filter posts
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
Co-authored-by: Nathaniel Allred <neallred@protonmail.com>
* Add route endpoints, model, store functions, and tests for top threads
* Run make store-layers
* Make the following changes
- Fix top user threads query
- Fix passing parameters in api4/insights.go to handler in app
- Add top user threads test
* Add post-message, user_id, participants information to insights results
* model.TopThread.UserID -> model.TopThread.UserId, for compatibility with MySQL
* Rename name -> channel_name
* Add user information to response
* Link post in response, filter out deleted root posts from top threads
* Handle thread delete cases, add app tests for threads insights
* lint: fix typo
* lint: rename asserts
* lint: require.nil -> require.NoError
* Add integration tests for thread insights
* Add embeds and images to top posts
* Add license checks for top threads endpoints
* Query users in batch to populate post-creator
* Make the following changes
- Add license to test server in api4/
- Add tests for threads insights
- top team threads shouldn't include threads from other teams, DMs
- Test duration constraint
- Pagination testing for top threads in model/insights_test.go
* Add i18n-extract
* i18n fixes
* Add username, nickname to user_information
* Hide message, user_id, post_id, reply_count in depth=1 of top threads response
* Fix tests using response.reply_count to use response.post.reply_count
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* Improves groups membership telemetry.
* Removes unnecessary where clause (it's added by default).
* Cleans up data.
* Testing the relative group counts.
* Adds missing test mock.
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* MM-43956: Adds post counts by day.
* MM-43956: Test data cleanup. Switch from Unix to UnixMilli.
* MM-43956: Changes from selecting date data types to strings in SQL.
* MM-43956: Fixes date format key.
* MM-43956: Adds missing user id scope for 'my' top channels graph.
* MM-43956: Adds the ability to group post counts by hour.
* MM-43956: Require enterprise or professional license. Reject guests.
* MM-43956: Adds license for tests.
* MM-43956: Renames function.
* MM-43956: Omits future hours from post counts by hour.
* MM-43956: Adjust API response grouping to users timezone.
* MM-43956: Adds translation.
* MM-43956: Adds user's timezone to the data tier for the grouping by day and hour.
* MM-43956: Fixes layers.
* MM-43956: Lint fix.
* MM-43956: Fix store layers.
* MM-43956: Switches to default name for time package; changes parameter names to avoid naming conflict.
* MM-43956: Updates mocks.
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* store: guests will only receive results of which channels they are in
* api4/channel_test: add test case for guest accounts channel autocomplete
* apply to searchengine and also for AutocompleteInTeam
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
Just a quick POC to move fast :P
We use a search pointer to keep track of
the next row to inesrt to. For every new search
we increment the pointer and do modulo 5.
This means that the value will always remain
between 0-4. And that way, we will always overwrite
the oldest entry on every search.
And while getting the results, we get
all results for that user.
The search parameters are json marshalled
and stored as a JSON blob. This is because
there is no need to search/filter them
in the DB.
Pending items:
Tests obviously.
To improve:
The client needs to send the channel ids
instead of channel names.
We add 2 new params to channel members query.
1. Filter by teamId.
2. Negate that filter.
We include some more optimizations like:
- Moved the team role checks inside the dataloader.
- Moved the channel pretty name computation inside the loader.
Now that we load less data on initial load, we can reduce
the concurrency requirement to be a bit on the safer side.
```release-note
NONE
```
* [MM-42739] Initial setup for top channels for team
* [MM-42739] Add initial tests
* [MM-42739] Update tests
* [MM-42739] Add top channels for user
* [MM-42739] Fix query
* [MM-42739] Update query
* [MM-42739] Improve query performance
* [MM-42739] Remove rank
* [MM-42739] Fix tests to use new time range today
* [MM-42739] Add tests for top channels for user
* [MM-42739] Add test for pagination
* Remove top channels by time struct
* [MM-42739] Update test names
* [MM-42739] Remove rank from top reactions
* [MM-42739] Return empty array instead of nil when result is empty
* [MM-42739] Add additional tests and update permissions check for teams
* [MM-42739] Add excluded channel tests for top reactions
* [MM-42739] Move insights to api4/insights and keep time range as string until required
* [MM-42739] Update queries only check DeleteAt after union
* [MM-42739] Improve query performance by using publicchannels table
* [MM-42739] Fix broken query after merge
The older method used to reply completely on timestamps
to take batches of items in a timestamp range and then
just incrementing the timestamp. This led to handling
edge-cases such as more items than the batch count, all
having the same timestamp.
Additionally, relying on timestamp as the page cursor
meant that indexing was not very efficient if you had
several items spread out across large spans of time.
To get away from all of that we use a proper cursor-based
approach consisting of createAt+Id. With this, we move
completely to a constant page size where we can fetch
a given number of objects irrespective of when they
were created. This makes indexing much more faster and
efficient.
https://mattermost.atlassian.net/browse/MM-41260
```release-note
Elasticsearch and Bleve indexing have been revamped to be much
more efficient and faster. The config parameter BulkIndexingTimeWindowSeconds
for both elasticsearch and bleve have been removed.
A new config parameter called BatchSize has been introduced instead.
This parameter controls the number of objects that
can be indexed in a single batch. This makes things
more efficient and maintains a constant workload.
```
We implement a cursor based pagination model
to page through the posts in a given thread.
The cursor is a combination of the post.CreateAt+
post.Id to differentiate multiple posts in a given
timestamp.
Some additional parameters like direction, fromPost,
fromCreateAt and perPage were introduced to implement
this.
```release-note
NONE
```
* revamp db version and add applied migrations endpoint
* replace old schema version with new
* add db version subcommand
* add to local api
* reflect review comments
* log errors
* remove setting the version from model.CurrentVersion
* fix a test
* use different field for schema version
* add build hash and current version to the support packet
* add tests
* update test to use new assets
* MM-42282: handle teamId parameter correctly
As per https://community-daily.mattermost.com/core/pl/ugs7ue6e4j8a7cgegk1bxje8to, `ThreadStore.GetThreadsForUser` accepts a `teamId` parameter, but incorrectly handles an empty value of `""` as looking only for channels with an empty `teamId` (aka DMs and GMs) instead of finding all channels and effectively ignoring the team property.
Fixes: https://mattermost.atlassian.net/browse/MM-42282
* break up getThreadsForUser, leverage errgroup
This change breaks up `GetThreadsForUser` in the `ThreadStore` into its constituent `GetTotalUnreadThreads`, `GetTotalThreads`, `GetTotalUnreadMentions`, and the original `GetThreadsForUser` but now solely returning the thread structures. Instead of a monolithic method at the store level, the application layer now handles calling bulk requests, leveraging `errgroup` for simpler parallelization.
This change brings with it a few benefits:
* Simpler code, including more idiomatic usage of squirrel
* Simpler SQL, joining tables only when configured conditions require same. (No performance benefit here, since an unused LEFT JOIN generally has no overhead.)
* Discrete Grafana metrics for each store method, giving us better insight into the performance characteristics in play.
* **Performance boost**: reduced overhead when clearing push notifications.
This last point is what prompted the re-re-reactoring in this PR. As I broke things up, I realized that `clearPushNotificationSync` only used the `TotalUnreadMentions`, but asked for the count of total threads and total unread threads. By exposing the discrete methods, this code path avoids two aggregate queries. We clear notifications when marking a thread as read, and when marking a channel with unread mentions as viewed, so I expect we'll see at least a modest boost to performance from simply not wasting these cycles anymore.
No performance improvements are expected from this PR for the general case of using `GetThreadsForUser` to populate the threads view.
* never discard errors from building queries
* no MustSql
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* Revert "[MM-41576] Revamp database schema version (#19586)"
This reverts commit 645fee3fe3.
* Revert "MM-42049 - license endpoint not working (#19686)"
This reverts commit 4fe89e5847.
* revamp db version and add applied migrations endpoint
* replace old schema version with new
* add db version subcommand
* add to local api
* reflect review comments
* log errors
* remove setting the version from model.CurrentVersion
* fix a test
* deadcode: remove UpdateChannelLastViewedAt
* deadcode: remove ThreadStore.(Save(Multiple)|Update|Delete)
* deadcode: followThead in App.MarkChannelAsUnreadFromPost
* document ThreadMembership, Thread structs
* maintain LastUpdated consistently
Whenever we touch a `ThreadMembership` record, we should be setting `LastUpdated` to the current timestamp. The mobile client relies on this to detect changes to these records.
* simplify: never updateThreads from `App.MarkChannelAsUnreadFromPost`
Change all invocations of `ChannelStore.UpdateLastViewedAtPost` from `App.MarkChannelAsUnreadFromPost` to pass `updateThreads` as `false`. When `ChannelStore.UpdateLastViewedAtPost` was invoked with `updateThreads` as `true`, it would in turn call `ThreadStore.UpdateUnreadsByChannel` but pass `updateViewedTimestamp` as `false`. This effectively updated the `LastUpdated` field of the corresponding thread memberships but never touched any of the actual data (such as `LastViewed`).
The overall CRT feature continued to work, because `App.MarkChannelAsUnreadFromPost` directly updates the relevant thread memberships via `ThreadStore.MaintainMembership`.
* deadcode: updateThreads in ChannelStore.UpdateLastViewedAtPost
* simplify: never updateThreads from App.SendNotifications
Change all invocations of `ChannelStore.IncrementMentionCount` from
`App.SendNotifications` to pass `updateThreads` as `false`. When `ChannelStore.IncrementMentionCount` was invoked with `updateThreads` as `true`, it would in turn call `ThreadStore.UpdateUnreadsByChannel` but pass `updateViewedTimestamp` as `false`. This effectively updated the `LastUpdated` field of the corresponding thread memberships but never touched any of the actual data (such as `UnreadMentions`).
The overall CRT feature continued to work, because `App.SendNotifications` directly updates the relevant thread memberships mention counts via `ThreadStore.MaintainMembership`.
* deadcode: updateThreads in ChannelStore.IncrementMentionCount
* fix & rename ThreadStore.UpdateUnreadsByChannel
Rename `ThreadStore.UpdateUnreadsByChannel` to `ThreadStore.UpdateLastViewedByThreadIds`, making it unconditionally set the `LastViewed` for the given threads (as well as `LastUpdated`).
All previous invocations of this method that passed `updateViewedTimestamp` have been previously removed.
* unrelated gofmt -w -s changes to satisfy linter
* always set LastUpdated to model.GetMillis()
* deadcode: ThreadStore.SaveMembership
* fix TestMarkUnreadWithThreads
* MM-40302: CRT, use updateThreads param vs. MarkAllAsReadInChannels
`MarkAllAsReadInChannels` was the subject of a significant performance regression in v5.37 and is known to be very inefficient, by virtue of always writing to an ever increasing number of rows, and doing so on common events like simply viewing a channel.
Fortunately, `ChannelStore.UpdateLastViewedAt` already supported an `updateThreads` parameter that implemented the start of an improved algorithm: query the set of threads with newer posts, and then update only /those/. Missing was the need to reset the `UnreadMentions`, but thanks to the previous simplifications in #19523, we can make this change largely without impacting other semantics.
Fixes: https://mattermost.atlassian.net/browse/MM-40302
* fix MySQL
* remove another JOIN
* remove outdated comment
* unit tests
