We implement a cursor based pagination model
to page through the posts in a given thread.
The cursor is a combination of the post.CreateAt+
post.Id to differentiate multiple posts in a given
timestamp.
Some additional parameters like direction, fromPost,
fromCreateAt and perPage were introduced to implement
this.
```release-note
NONE
```
* revamp db version and add applied migrations endpoint
* replace old schema version with new
* add db version subcommand
* add to local api
* reflect review comments
* log errors
* remove setting the version from model.CurrentVersion
* fix a test
* use different field for schema version
* add build hash and current version to the support packet
* add tests
* update test to use new assets
* MM-42282: handle teamId parameter correctly
As per https://community-daily.mattermost.com/core/pl/ugs7ue6e4j8a7cgegk1bxje8to, `ThreadStore.GetThreadsForUser` accepts a `teamId` parameter, but incorrectly handles an empty value of `""` as looking only for channels with an empty `teamId` (aka DMs and GMs) instead of finding all channels and effectively ignoring the team property.
Fixes: https://mattermost.atlassian.net/browse/MM-42282
* break up getThreadsForUser, leverage errgroup
This change breaks up `GetThreadsForUser` in the `ThreadStore` into its constituent `GetTotalUnreadThreads`, `GetTotalThreads`, `GetTotalUnreadMentions`, and the original `GetThreadsForUser` but now solely returning the thread structures. Instead of a monolithic method at the store level, the application layer now handles calling bulk requests, leveraging `errgroup` for simpler parallelization.
This change brings with it a few benefits:
* Simpler code, including more idiomatic usage of squirrel
* Simpler SQL, joining tables only when configured conditions require same. (No performance benefit here, since an unused LEFT JOIN generally has no overhead.)
* Discrete Grafana metrics for each store method, giving us better insight into the performance characteristics in play.
* **Performance boost**: reduced overhead when clearing push notifications.
This last point is what prompted the re-re-reactoring in this PR. As I broke things up, I realized that `clearPushNotificationSync` only used the `TotalUnreadMentions`, but asked for the count of total threads and total unread threads. By exposing the discrete methods, this code path avoids two aggregate queries. We clear notifications when marking a thread as read, and when marking a channel with unread mentions as viewed, so I expect we'll see at least a modest boost to performance from simply not wasting these cycles anymore.
No performance improvements are expected from this PR for the general case of using `GetThreadsForUser` to populate the threads view.
* never discard errors from building queries
* no MustSql
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* Revert "[MM-41576] Revamp database schema version (#19586)"
This reverts commit 645fee3fe3.
* Revert "MM-42049 - license endpoint not working (#19686)"
This reverts commit 4fe89e5847.
* revamp db version and add applied migrations endpoint
* replace old schema version with new
* add db version subcommand
* add to local api
* reflect review comments
* log errors
* remove setting the version from model.CurrentVersion
* fix a test
* deadcode: remove UpdateChannelLastViewedAt
* deadcode: remove ThreadStore.(Save(Multiple)|Update|Delete)
* deadcode: followThead in App.MarkChannelAsUnreadFromPost
* document ThreadMembership, Thread structs
* maintain LastUpdated consistently
Whenever we touch a `ThreadMembership` record, we should be setting `LastUpdated` to the current timestamp. The mobile client relies on this to detect changes to these records.
* simplify: never updateThreads from `App.MarkChannelAsUnreadFromPost`
Change all invocations of `ChannelStore.UpdateLastViewedAtPost` from `App.MarkChannelAsUnreadFromPost` to pass `updateThreads` as `false`. When `ChannelStore.UpdateLastViewedAtPost` was invoked with `updateThreads` as `true`, it would in turn call `ThreadStore.UpdateUnreadsByChannel` but pass `updateViewedTimestamp` as `false`. This effectively updated the `LastUpdated` field of the corresponding thread memberships but never touched any of the actual data (such as `LastViewed`).
The overall CRT feature continued to work, because `App.MarkChannelAsUnreadFromPost` directly updates the relevant thread memberships via `ThreadStore.MaintainMembership`.
* deadcode: updateThreads in ChannelStore.UpdateLastViewedAtPost
* simplify: never updateThreads from App.SendNotifications
Change all invocations of `ChannelStore.IncrementMentionCount` from
`App.SendNotifications` to pass `updateThreads` as `false`. When `ChannelStore.IncrementMentionCount` was invoked with `updateThreads` as `true`, it would in turn call `ThreadStore.UpdateUnreadsByChannel` but pass `updateViewedTimestamp` as `false`. This effectively updated the `LastUpdated` field of the corresponding thread memberships but never touched any of the actual data (such as `UnreadMentions`).
The overall CRT feature continued to work, because `App.SendNotifications` directly updates the relevant thread memberships mention counts via `ThreadStore.MaintainMembership`.
* deadcode: updateThreads in ChannelStore.IncrementMentionCount
* fix & rename ThreadStore.UpdateUnreadsByChannel
Rename `ThreadStore.UpdateUnreadsByChannel` to `ThreadStore.UpdateLastViewedByThreadIds`, making it unconditionally set the `LastViewed` for the given threads (as well as `LastUpdated`).
All previous invocations of this method that passed `updateViewedTimestamp` have been previously removed.
* unrelated gofmt -w -s changes to satisfy linter
* always set LastUpdated to model.GetMillis()
* deadcode: ThreadStore.SaveMembership
* fix TestMarkUnreadWithThreads
* MM-40302: CRT, use updateThreads param vs. MarkAllAsReadInChannels
`MarkAllAsReadInChannels` was the subject of a significant performance regression in v5.37 and is known to be very inefficient, by virtue of always writing to an ever increasing number of rows, and doing so on common events like simply viewing a channel.
Fortunately, `ChannelStore.UpdateLastViewedAt` already supported an `updateThreads` parameter that implemented the start of an improved algorithm: query the set of threads with newer posts, and then update only /those/. Missing was the need to reset the `UnreadMentions`, but thanks to the previous simplifications in #19523, we can make this change largely without impacting other semantics.
Fixes: https://mattermost.atlassian.net/browse/MM-40302
* fix MySQL
* remove another JOIN
* remove outdated comment
* unit tests
Previously, we were incrementing mentions one-by-one
all concurrently in an unbounded fashion.
This would cause a big spike in memory usage if there
were an `@all` mention in a large channel.
We fix this by changing the SQL query to take all userIDs
at once.
https://mattermost.atlassian.net/browse/MM-41752
```release-note
NONE
```
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* deadcode: remove UpdateChannelLastViewedAt
* deadcode: remove ThreadStore.(Save(Multiple)|Update|Delete)
* deadcode: followThead in App.MarkChannelAsUnreadFromPost
* document ThreadMembership, Thread structs
* maintain LastUpdated consistently
Whenever we touch a `ThreadMembership` record, we should be setting `LastUpdated` to the current timestamp. The mobile client relies on this to detect changes to these records.
* simplify: never updateThreads from `App.MarkChannelAsUnreadFromPost`
Change all invocations of `ChannelStore.UpdateLastViewedAtPost` from `App.MarkChannelAsUnreadFromPost` to pass `updateThreads` as `false`. When `ChannelStore.UpdateLastViewedAtPost` was invoked with `updateThreads` as `true`, it would in turn call `ThreadStore.UpdateUnreadsByChannel` but pass `updateViewedTimestamp` as `false`. This effectively updated the `LastUpdated` field of the corresponding thread memberships but never touched any of the actual data (such as `LastViewed`).
The overall CRT feature continued to work, because `App.MarkChannelAsUnreadFromPost` directly updates the relevant thread memberships via `ThreadStore.MaintainMembership`.
* deadcode: updateThreads in ChannelStore.UpdateLastViewedAtPost
* simplify: never updateThreads from App.SendNotifications
Change all invocations of `ChannelStore.IncrementMentionCount` from
`App.SendNotifications` to pass `updateThreads` as `false`. When `ChannelStore.IncrementMentionCount` was invoked with `updateThreads` as `true`, it would in turn call `ThreadStore.UpdateUnreadsByChannel` but pass `updateViewedTimestamp` as `false`. This effectively updated the `LastUpdated` field of the corresponding thread memberships but never touched any of the actual data (such as `UnreadMentions`).
The overall CRT feature continued to work, because `App.SendNotifications` directly updates the relevant thread memberships mention counts via `ThreadStore.MaintainMembership`.
* deadcode: updateThreads in ChannelStore.IncrementMentionCount
* fix & rename ThreadStore.UpdateUnreadsByChannel
Rename `ThreadStore.UpdateUnreadsByChannel` to `ThreadStore.UpdateLastViewedByThreadIds`, making it unconditionally set the `LastViewed` for the given threads (as well as `LastUpdated`).
All previous invocations of this method that passed `updateViewedTimestamp` have been previously removed.
* unrelated gofmt -w -s changes to satisfy linter
* always set LastUpdated to model.GetMillis()
* deadcode: ThreadStore.SaveMembership
* fix TestMarkUnreadWithThreads
* GetMasterX
* WIP
* adding initial creategroup endpoint
* fetching by group source
* fixing startup error
* updating create endpoint to take an array of user_ids, this will allow us to create the group with one request
* adding delete group endpoint and appropriate test
* adding source param for getGroups
* adding add members and delete members endpoints
* locking down crud endpoints to only be allowed for custom groups
* user search stuff
* allowing remoteid be null by changing field to pointer
* code cleanup and store level tests
* adding new tests and removing unused endpoint
* resolving conflicts
* Adds authz check for group.
* Adds authz checks to groups APIs.
* Updated create group authz tests.
* Updates delete group tests.
* Tests create group.
* Adds some tests and validations.
* adding new parameter so I can get users not in a group
* Fixed all lint warnings.
* Fix type.
* fixing search users not in group
* Fixes some lint errors.
* Moves entry in JSON array.
* Fixed SQL query.
* Fixes permission migration test.
* Fixes migration test.
* Fixes some group store tests.
* Fix test.
* Fix test.
* Revert lint change.
* Migrated CreateWithUserIds to sqlx.
* Adds tests for GetMember; migrates implementation to sqlx.
* Tests GetNonMemberUsersPage and hanles wrong group id.
* Fixes test.
* Switches GetMaster to GetMasterX.
* Switches GetReplica to GetReplicaX.
* Fixes logic.
* Fixes shadow declaration.
* Adds include_member_count to get group API endpoint.
* Adds filter_has_member param to getGroups.
* Fixes.
* Removes array of group sources.
* fixing error
* Testing reverting CreateWithUserIds back to gorp.
* Added websocket event for CreateGroupWithUserIds.
* Changed a few response status codes. Switched to correct permission.
* Added member count to ws payload for group when updating or creating.
* Adds feature flag checks for custom groups.
* Added middleware function to require license. Added config to disable custom groups.
* Change for function signature change of executePossiblyEmptyQuery.
* Lint fixes.
* Adds telemetry none comment.
* Adds translations.
* Migrated to sqlx.
* Temp. removal of translation.
* Fixed typo.
* Added an intermediary model to query with a field that is now ignored by sqlx on read queries.
* Re-used existing store struct.
* Inludes member count.
* Fix for merge error.'
* Require license for group endpoints.
* Updates translations.
* Fix shadow declaration.
* Renames permissions. Switches to new method to retrieve remoteid.
* Added WS events for upsert and delete member(s).
* Added new store error type ErrUniqueConstraint.
* Added EnableCustonGroups to the client config.
* Sanitized some user records.
* Added parameter to include_total_count for listing groups.
* Added translations.
* adding deleteAt field to getByUsers query
* Revert sanitize.
* Added uniqueness constraint error to UpdateGroup.
* Removed the FutureFeatures flag so that the feature is not enabled on old Enterprise licenses.
* Renamed function.
* Updates authz check for user search related to groups.
* Removed debug statement.
* Removed unused app method.
* Added telemetry for enable_custom_groups.
* Returns early from nil license.
* Updates test.
* Returned early to avoid nesting in (*SqlGroupStore).checkUserExist. Switched to reading from replica in (*SqlGroupStore).GetMember. Handled JSON marshal error in (*Client4).UpsertGroupMembers
* Switched to SanitizeProfile.
* Switched to model.NewInt.
* Switched from status NotImplemented to Forbidden for missing license.
* Removed deactivated users from 'exists' set.
* Revert gotool update.
* Ignored lint error that I think is invalid.
* Added the approprate access tag for disabling custom groups.
* Revert change to response status.
* Fixed refactor mistake.
* Limited the group member WS events to individual users.
* Removed WS event of deleted groups.
* Updated license check for searchUsers endpoint.
* Switched from license feature to license sku.
* Update app/group.go
Co-authored-by: Claudio Costa <cstcld91@gmail.com>
* Update app/group.go
Co-authored-by: Claudio Costa <cstcld91@gmail.com>
* Remove linter ignore comment.
* Added function to create sku-specific license.
* Fixed typo. Removed comment.
* Fixed for wrong type.
* Added missing param to client. Removed unnecessary props setting. Added test for retrieving groups by source.
* Updated some tests now that we're validating group membership not created for deactivated user.
* Fix for groups endpoint returning all group types by default.
* Changes constant names. Adds migration for all users to manage custom group members.
* Removes requirement for manage_system permission to filter user search by group.
* Added migration mock.
* Removes default permissions from custom_group_user role.
* Fixes migration.
* Fixes emoji migration test.
* fixing issue with member counts
* fixing search issue for deleted members
Co-authored-by: Benjamin Cooke <benjamincooke@Benjamins-MacBook-Pro.local>
Co-authored-by: Benjamin Cooke <benjamincooke@Benjamins-MBP.ht.home>
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
Co-authored-by: Benjamin Cooke <benjamincooke@Benjamins-MacBook-Pro.fritz.box>
Co-authored-by: Claudio Costa <cstcld91@gmail.com>
The GetTeamsUnreadForUser call would be called for every team switch.
In CRT mode, it would make a separate store call for every team, which
would run the 3 aggregate SQL queries in GetThreadsForUser.
This is suboptimal because the complexity is linearly proportional
to the number of teams.
We make the following optimizations:
1. Change the query to a single one which aggregates all teams.
2. The query originally used just 2 out of the 3 queries, so one
query was fully redundant. We remove that query in the new one.
3. Further analysis was done whether it makes sense to run the 2
queries synchronously or not. The load-tests didn't show any degradation
in running them concurrently, so we keep the same behavior.
```release-note
NONE
```
* tools updates
* Revert "tools updates"
This reverts commit 6293297b55803c5a263e200ebd80192899666ae9.
* new endpoint to get users that should potentially be guests
* checking authservice to ensure they were an email signup
* adding tests for new endpoint
* fixing translation issue
* permissions for new endpoint
* fixing tests
* fixing when domain array is empty
* fixing when domain array is empty
* removing bots from request
Co-authored-by: Benjamin Cooke <benjamincooke@Benjamins-MacBook-Pro.local>
Co-authored-by: Benjamin Cooke <benjamincooke@Benjamins-MBP.ht.home>
Co-authored-by: mkraft <martinkraft@gmail.com>
* MM-36589: provide previous values for unreads
To successfully figure out the new counts of mentions or unread replies
for CRT we need to provide previous values alongside with the new.
This is needed so we'll know how many to subtract from the total.
This commit provides those numbers upon publishing websocket ThreadUpdated
and ThreadReadChanged events.
* Fixes errors
* Removes unneeded lines
* Adds GetThreadUnreadReplyCount store method
Uses the new store method to get unread replies instead of
GetThreadForUser.
* Tests, and some changes
- Adds api4 tests to test ws events
- Uses sqlx instead of gorp for new store method
- Fixes case where previous_unread_replies could be a negative value
* Refactors tests and adds more cases
* Fixes previous and current unread counts for commenter
When a user posts a reply to a thread the unread counts had a couple of
issues.
UnreadMentions where not zeroed out, and previous unread counts where
not set correctly.
This commit tries to fix that by marking the thread as read for the
current poster after we set previous unread counts to the websocket
event data.
Also MaintainMembership should zero out UnreadMentions when we are
setting the thread as read.
* Oops
* Fixes tests by updating when maintaining membership
OK, so some tests broke because we zero UnreadMentions in the membership
when we UpdateViewedTimestamp, since the new timestamp is always now.
Some tests broke because MaintainMembership for the commenter so that
the thread is read each time commenter posts moved further down in the
SendNotification method.
BOTH those test cases are fixed with this commit. To be sincere though I
don't understand why the second one is fixed by this.
* SystemAdminUser was not part of the channel
Some tests are failing because SystemAdminUser is not part of the
team and channels.
This commit adds user to team and channels, in an effort to fix
api4/user_tests
* Fixes tests
* Fixes tests
* Addresses review comments
* Fix if clause
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
Summary
The addition of the TotalMsgCountRoot and MsgCountRoot columns to support CRT caused several issues with previously read threads and channels being marked as unread. Previously we attempted to fix this purely in a SQL migration [MM-35345][MM-35494] fixes for incorrect mentions and unreads for threads and channels #17803 but that turned out to be too heavy and it was decided to break up some of the fixes into async jobs.
This PR implements an async job to mark channels as read if there are no user posts since the last time the user viewed the channel.
Ticket Link
https://mattermost.atlassian.net/browse/MM-37013
* Show private channels in autocomplete
This is supported in all Engines:
MySQL, Postgres, Bleve, Elasticsearch.
https://mattermost.atlassian.net/browse/MM-18496
```release-note
Private channels will now appear in channel autocomplete.
If you are using Bleve or ElasticSearch, you will have to reindex
the channels again to populate them with the new attributes.
```
A large chunk of this work has been based on the earlier
effort at https://github.com/mattermost/mattermost-server/pull/17804.
Full credit goes to https://github.com/arvinDarmawan.
* Add comment
```release-note
NONE
```
* Adding more tests
```release-note
NONE
```
* fix more tests
```release-note
NONE
```
* tmp
```release-note
NONE
```
* more fixes
```release-note
NONE
```
* add tests
```release-note
NONE
```
* Add review comments from previous PR
```release-note
NONE
```
* Add API to return all channels from all team
```release-note
NONE
```
* Added support for bleve and ES
```release-note
NONE
```
* Streaming response for GetAllChannels
```release-note
NONE
```
* Fix tests
```release-note
NONE
```
* Trigger CI
```release-note
NONE
```
* fix tests
```release-note
NONE
```
* Addressing review comments
```release-note
NONE
```
* Fix lint
```release-note
NONE
```
* Removing flaky test
```release-note
NONE
```
* Address comments
```release-note
NONE
```
* Trigger CI
```release-note
NONE
```
* Added /users/<userid>/channel_members endpoint
```release-note
NONE
```
* Minor edit
```release-note
NONE
```
* Improve embedding
```release-note
NONE
```
* Fix lint error
```release-note
NONE
```
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* Add functionality to cleanup old jobs
Historically, we never cleaned up old jobs from the DB
leading to them being accumulated forever.
This PR introduces functionality to cleanup old jobs
older than a defined threshold.
The functionality is set to false by default and has
to be enabled for it to work.
```release-note
2 new config settings were added.
JobSettings.CleanupOldJobs: This indicates whether to clean up old jobs
from the DB or not. Default is false.
JobSettings.CleanupJobsThresholdHours: This defines the time gap in hours beyond
which older jobs will be removed. This has no effect if the above config
setting is set to false. Default is -1
```
* fix copy pasta
```release-note
NONE
```
* address review comments
```release-note
NONE
```
* Fix lint
```release-note
NONE
```
* Use single config option for everything
```release-note
NONE
```
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* Add API endpoint and adapt search to allow multi-team search
* Refactor handler, refactor sql query to use squirrel, rename app and store functions and add tests
* Fix lint
* Fix search engines and remove unneeded comments
* Fix test
* Remove user from channel after test
* CRT: desktop thread notifications
* Fixes go lint
* Adds default for desktop CRT notifications
* Adds email and push notifications for CRT threads
Adds user ids of thread followers with CRT to crtMentions so they will get
notified appropriately.
* Minor change
* Refactor a bit
CRTMentions.addMention had a bug on the return and de-duplication.
This commit fixes duplicate notifications by looking up if the user is to be
notified on CRT on both email and push notifications.
* Minor refactor
* Changes according to review comments
- Fixes adding to followers a user that had explicitly unfollowed a
thread.
- Simplified send email according to email_threads option
- Send mentions and followers in separate arrays via the websocket
- Fixes push notifications message for push_threads
* Adds a comment on a buggy use case
* Updates comment to correct ticket link
* Fixes when user notifications is set to all
There was a bug where if user had set notifications to all
then they would receive desktop notifications even for non following threads.
A similar bug existed in push notifications, where if a user has set it
to all the threads setting would still be considered.
This commit fixes that by adding users to notificationsForCRT
StringArray when they have the non thread setting to 'all'.
* Fixes notifications to users unfollowing threads
Users which had previously explicitly unfollowed a thread
should not receive notifications about those threads.
* Update store mocks
* Fixes push notifications for CRT
Push notification about replies for CRT users should have a title of
"Reply to Thread".
CRT users with global user setting to 'UserNotifyAll' should not get
notifications for unfollowed threads.
This commit fixes those issues.
* Fixes i18n error
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
- We move logging statements to the upper layer.
Store functions are low-level methods and should return error
upwards rather than logging.
- Used IN instead of any (array ()) which is equivalent.
- Made the delay to be of type time.Duration and un-exported it.
- Unexported the batch size constant.
```release-note
NONE
```
* MM-21357: Use typed constant for channel types
https://mattermost.atlassian.net/browse/MM-21357
```release-note
- Introduced a new type ChannelType for all channel types.
- Updated the Client4.UpdateChannelPrivacy method to ChannelType.
```
* Address review comments
```release-note
NONE
```
* telemetry fix
```release-note
NONE
```
* pre-checkout commit
* add API endpoints for retention policies
* allow deleting multiple teams/channels from a policy in a single request
* pre-checkout commit
* add auditing in API functions
* add permission checks
* update the store layers
* update storetest
* add check constraint on PostDuration column
* pre-checkout commit
* add query to delete posts under the scope of a granular retention policy
* add suggestions from sbishel
* allow clients to specify channels/teams when creating a new policy
* remove foreign keys referencing Channels and Teams tables
* add checks for whether teams and channels exist
* pre-checkout commit
* remove data referencing the Posts table
* pre-checkout commit
* write data store tests
* sort results of buildGetPoliciesQuery
* add missing test cases for teams
* pre-checkout commit
* add Client4 methods for data retention policy endpoints
* add uint and uint64 to app/layer_generators
* make granular policies override global policies
* fix lint errors
* pre-checkout commit
* add license to top of files
* add tests for data store layer
* add missing test cases for store layer
* run make i18n-extract
* add query to delete ChannelMemberHistory
* work in progress
* add test for old reply to old post
* fix lint error
* use COALESCE on each Posts column
* begin implementing orphaned rows worker
* split PR
* pre-checkout commit
* use RetentionPolicyWithTeamAndChannelCounts
* update app and api layers
* run make i18n-extract
* add RetentionPolicy to retrylayer_test.go
* Revert "split PR"
This reverts commit b316f03dd307a30deae931944ca7e4a1cc904605.
* fix errors caused by revert
* add suggestions from sbishel
* fix copy-paste error
* fix lint errors
* pre-checkout commit
* add function to delete orphaned rows
* use -1 for infinite retention
* remove check constraint
* copy i18n entries from master
* re-run tests with newer enterprise branch
* add team data to channel list
* add search for channels and teams in a policy
* add store tests for channel and team search
* add suggestions from mkraft
* run make einterfaces-mocks
* fix lint errors
* add suggestions from mkraft
* move removeOrphanedRows method to wireup branch
* Revert "move removeOrphanedRows method to wireup branch"
This reverts commit 94605c9b4a5378ffa44a3dec4d3f8e3306b9d33e.
* use DeleteOrphanedRows where possible
* run make i18n-extract
* use COMPLIANCE permissions
* run make migrations-bindadta
* clean up teams before test
* fix tests for TestRetentionPolicyStore
* add API endpoints for mobile
* fix lint error
* fix some of the lint errors
* move user/data_retention endpoints to data_retention.go
* Revert "fix some of the lint errors"
This reverts commit b5b2dc27566c427187db942c5c0afe319e7679c4.
* add exclude_policy_constrained parameter for /channels and /teams
* fix lint errors
* add policy_id field to GET endpoints for channels and teams
* use PolicyWithTeamID in RetentionPolicy layer
* fix lint errors
* run make i18n-extract
* update mock call in telemetry_test.go
* return status:OK in JSON instead of 204
* pre-checkout commit
* add policy_id field on channels/teams
* fix lint errors
* use sq.Eq instead of '?'
* use new subsection permissions
* update channels and teams endpoints to use new subsection permissions
* add extra search opts for channels in a policy
* fix lint errors
* allow negative post duration in patch
* remove DELETE FROM query in retention policy tests
* use *int64 for PostDuration
* re-run CI tests
* use 3-step deletion strategy for each table
* fix lint errors
* run make store-layers
* re-run CI tests
* add test with channel, team and global policies
* use common function for SQL queries
* add pagination test
* use struct for args to common SQL function
* fix lint errors
* run make i18n-extract
* check if Channels.TeamId is "" or nil
* use three OR clauses
* write separate genericRetentionPoliciesDeletion function
* add config setting for BatchSize
* add telemetry for BatchSize
* use feature flag
* add old i18n messages back in
* re-run CI tests
* update call signature in storetest
* MM-30831: Adds constant for retention default batch size.
* MM-30831: Removes comment re: optimization.
* MM-30831: Converts days to milliseconds.
* MM-30831: Reverts change to test.
* Revert "MM-30831: Reverts change to test."
This reverts commit 6d14275a1ceae682bb9e17ec69b39252b44e0c0c.
* Revert "MM-30831: Converts days to milliseconds."
This reverts commit a0cb6ec09d854a05194c1daee1c5333f260231c3.
* MM-30831: Fixes tests.
* MM-30381: Fix for change to method sig.
Co-authored-by: Max Erenberg <max.erenberg@mattermost.com>
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
Co-authored-by: Martin Kraft <martin@upspin.org>
* MM-34434: Added 'is_trial' boolean to all trial license requests and to the License struct.
* MM-34434: Generalized the concept of a license request.
* MM-34434: Verifies JSON field of license instance is set.
* MM-34434: Added missing client param.
* MM-34434: Added some tests of the request trial API endpoint.
* MM-34434: Removed comment.
* fix broken test (#17348)
* Add missing wrapped errors (#17339)
* Improve document extraction and including a document extraction command (#17183)
* Add extract documents content command
* Adding the extraction command and making the pure go pdf library as secondary option
* Improving the memory usage and docextractor interface
* Enable content extraction by default in all the instances
* Tiny improvement on archive indexing
* Adding App interface generation and the opentracing layer
* Fixing linter errors
* Addressing PR review comments
* Addressing PR review comments
* Update en.json (#17356)
Automatic Merge
* adding new feature flag (#17308)
Co-authored-by: Benjamin Cooke <benjamincooke@Benjamins-MacBook-Pro.local>
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* Bump no_output_timeout to 2 hours (#17358)
* log invalid username (#17345)
Automatic Merge
* MM-34434: Added missing client param.
MM-34434: Added some tests of the request trial API endpoint.
MM-34434: Removed comment.
* MM-34434: Switched to a hard-coded true value.
* MM-34434: Reverts test change.
* MM-34434: Removes unnecessary field.
* MM-34434: Tests that is_trial is hard-coded by TrialLicenseRequest.
* MM-34434: Removed accidental commit.
* MM-34434: Removes unnecessary is_trial key from JSON payload.
* MM-34434: Reverts to old pointer receiver variable name.
* MM-34434: Removes test.
* #MM-34437 Initialized license service
* ##MM-34437 Verified at all points if server is trial elligible
* WIp
* #MM-34437 removed unused commented code
* MM-34437 make a log less severe
* #MM-34437 generated einterface mocks
* #MM-34437 added license on new file
* #MM-34437 removed unused translation
* #MM-34437 some refactoring
* Update api4/license.go
* Update api4/license.go
* #MM-34437 made a variable name consistent
* #MM-34437 Added mocks for lince validator
* #M--34437 Added license validator test framework
* #MM-34437 Renamed isTrial method to isTrialLicense to avoid conflict with newlya dded field
* #M--34437 Allowed sales-sanctioned trials
* #MM-34437 fixed trial license API tests
* Added tests for add license API
* #MM-34437 fixed ValidateLicense test
* #MM-34437 Added util tests
* #MM-34437 using NoError for checking no error
* #MM-34437 using NoError for checking no error
* Added dummy piblic key for testing
* Fixed tests
* #MM-34437 udpaetd trial license URL for testing
* #MM-34437 adjusted times for licences generated through admin portal
* Reverted test-only changes
Co-authored-by: Martin Kraft <martin@upspin.org>
Co-authored-by: Hossein <hahmadia@users.noreply.github.com>
Co-authored-by: Ben Schumacher <ben.schumacher@mattermost.com>
Co-authored-by: Jesús Espino <jespinog@gmail.com>
Co-authored-by: Amy Blais <amy_blais@hotmail.com>
Co-authored-by: Ben Cooke <benkcooke@gmail.com>
Co-authored-by: Benjamin Cooke <benjamincooke@Benjamins-MacBook-Pro.local>
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
Co-authored-by: Agniva De Sarker <agnivade@yahoo.co.in>
Co-authored-by: Max Erenberg <max.erenberg@mattermost.com>
* Revert "Revert "[MM-8497] Ability to set Do Not Disturb for a specified period of time (#16067)" (#17657)"
This reverts commit ff383990f8.
* add debug log for recurring function
* add feature flag for dnd timed status
* refactoring changes
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* Update store function GetThreadForUser to use master DB to fix replica lag
* Refactor GetThreadForUser store func to take membership as argument to prevent replica lag issues and reduce joins in query
* Add translation
* Fix test
* Updates per feedback
* Minor clean-up per feedback
* restrict creation of direct channels to team members
* run make i18n-extract
* add suggestions from hahmadia
* place common-team-check logic in app layer
* use flat SQL query
* show more specific error message to user
* MM-7968: Fmt file.
* MM-7968: Fix for moved session field.
Co-authored-by: Max Erenberg <max.erenberg@mattermost.com>
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
Co-authored-by: Martin Kraft <martinkraft@gmail.com>
Co-authored-by: Martin Kraft <martin@upspin.org>
* Add support for timed DND status
- accept a date time value in api query when dnd mode for user needs to be unset
- Create a new function to handle SetDNDStatus calls
- Create a scheduled task to unset dnd mode to wahtever mode was before setting it to DND
* update schema version
* Model changes to make fields more intuitive
- move dndendtime to status model
- add new field prev status in status to keep track of previous status of user
- update db migration function
- make use of prevstatus and dndendtime from status model
* set prev status and dndendtime appropriately after unsetting dnd mode
* add json tag for dndendtime
* unset dnd status only if not changed manually by user
* update dnd statuses after server restart
* make app-layers
* fix failing tests
* don't create sched task when setting status to DND
* get only expired statuses from db
- convert end time from any timezone to utc
- store dnd end time in unix format for usability reasons
* run update dnd status only on leader
* make mocks
* fix tests
* run UpdateDNDStatusOfUsers as recurring task
* save all statuses at once in db and update UpdateDNDStatusOfUsers logic
* add app method to get timezone of user
* store dnd end time in context.Params
* set max size of prevstatus
* update status model to take endtime input as string and store in db as unix time(int64)
* Add tests for SetStatusDoNotDisturbTimed
* if dnd_end_time is not passed the call old api to set dnd mode
* fix tests
* new plugin api to use new timed dnd mode
* get and update rows in a single db query
* dnd end time will be stored in request body and not route param
* exclude statuses which has dndendtimeunix < 0
* update and get the updated dnd statuses in single db query
* add updated status to cache
* DNDEndTimeUnix and PrevStatus need not to be visible to users
* update db schema version for migration
* Keep Status and PrevStatus varchar size same
* add test to verify status is restored after dnd end time expires
* expect endtime in utc from client
- remove store method GetTimezone as no longer needed
- add documentation for SetStatusDoNotDisturbTimed
* reduce sleep time for dnd timed restore test
* more appropriate name for new api to update user status
* update db migration function
* parse and validate time before potentially triggering db query to get status of user
* add migration changes in to existing upgrade function
* not supporting un-timed dnd status via api
* don't call Srv.Store directly, call via app layer
* rename dndendtime to statuscleartime to make it suitable for custom status usage as well
* Revert "rename dndendtime to statuscleartime to make it suitable for custom status usage as well"
This reverts commit fa69152d9a3db18f1c59b34c878fb7ce494440b5.
* mysql doesn't support RETURNING clause so add tx to get and update statuses
* add UpdateDNDStatusOfUsers mock in tests
* update store mock import path
* add mock in storelib
* Add status mocks to empty store
* Close the task during server shutdown
* Do not cancel a nil task
* update squirrel queries
* remove untimed dnd test
* start recurring task to unset statuses on leadership change
* set dndTask to nil after cancelling it upon server shutdown
* new recurring task which starts at nearest rounded time of the interval
* mock Get() call for status
* return updated statuses in case of mysql
* remove unneccessary code
* add Get() mock to empty store
* fix mocking for once and all
* address review comments
fix mysql updateStatus fn
protect dndTask with mutex
minor refactors
* move runDNDStatusExpireJob to server.go and pass App as arg instead of method receiver
* frontend will send endtime in unix epoch format so get rid of double representation
* scan for all fields and not just two
* add some tests and fix review comments
* remove extra sql query and create needed result in go
* add storetest for UpdateExpiredDNDStatuses
* add migrations to latest version
* update min supported version
* add comment to fix a bug in future
* update test to expect 1 status in return
* rename UpdateUserStatusWithDNDTimeout to SetUserStatusTimedDND
* rename DNDEndTimeUnix to DNDEndTime
* cast int to int64 for equality
* fix tests and error handling
* move updating values to retrieved statuses fields outside sql transaction
* move migrations to 5.36
Co-authored-by: Agniva De Sarker <agnivade@yahoo.co.in>
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* add includeRemovedMembers flag
* fix API call in client4.go
* remove check for 'since'
* add comments
* run make app-layers
* re-run CI tests
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* add MessageExportCursor
* sort by PostUpdateAt and PostId
* re-run CI tests
* remove panic debugging line
Co-authored-by: Martin Kraft <martin@upspin.org>
* don't send auto response if already responded today
* update query to get posts from channel for given user and Updatetime requires value in milli seconds
* regenerate mocks and layers
* update function to return true/false on existence of auto responded post in channel and add tests
* add store tests
* bubble up error and propagate upstream
* fix error handling logic
* use require instead of assert
Co-authored-by: Ibrahim Serdar Acikgoz <serdaracikgoz86@gmail.com>
* rename variable for better redability and logging fixes
* update comment explaining function
* use new function to generate test ids
* add comments to clarify NewTestId copies
* add translations for error id
* fix translation
Co-authored-by: Ibrahim Serdar Acikgoz <serdaracikgoz86@gmail.com>
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
Co-authored-by: Saturnino Abril <saturnino.abril@gmail.com>
