* WIP
* adding initial creategroup endpoint
* fetching by group source
* fixing startup error
* updating create endpoint to take an array of user_ids, this will allow us to create the group with one request
* adding delete group endpoint and appropriate test
* adding source param for getGroups
* adding add members and delete members endpoints
* locking down crud endpoints to only be allowed for custom groups
* user search stuff
* allowing remoteid be null by changing field to pointer
* code cleanup and store level tests
* adding new tests and removing unused endpoint
* resolving conflicts
* Adds authz check for group.
* Adds authz checks to groups APIs.
* Updated create group authz tests.
* Updates delete group tests.
* Tests create group.
* Adds some tests and validations.
* adding new parameter so I can get users not in a group
* Fixed all lint warnings.
* Fix type.
* fixing search users not in group
* Fixes some lint errors.
* Moves entry in JSON array.
* Fixed SQL query.
* Fixes permission migration test.
* Fixes migration test.
* Fixes some group store tests.
* Fix test.
* Fix test.
* Revert lint change.
* Migrated CreateWithUserIds to sqlx.
* Adds tests for GetMember; migrates implementation to sqlx.
* Tests GetNonMemberUsersPage and hanles wrong group id.
* Fixes test.
* Switches GetMaster to GetMasterX.
* Switches GetReplica to GetReplicaX.
* Fixes logic.
* Fixes shadow declaration.
* Adds include_member_count to get group API endpoint.
* Adds filter_has_member param to getGroups.
* Fixes.
* Removes array of group sources.
* fixing error
* Testing reverting CreateWithUserIds back to gorp.
* Added websocket event for CreateGroupWithUserIds.
* Changed a few response status codes. Switched to correct permission.
* Added member count to ws payload for group when updating or creating.
* Adds feature flag checks for custom groups.
* Added middleware function to require license. Added config to disable custom groups.
* Change for function signature change of executePossiblyEmptyQuery.
* Lint fixes.
* Adds telemetry none comment.
* Adds translations.
* Migrated to sqlx.
* Temp. removal of translation.
* Fixed typo.
* Added an intermediary model to query with a field that is now ignored by sqlx on read queries.
* Re-used existing store struct.
* Inludes member count.
* Fix for merge error.'
* Require license for group endpoints.
* Updates translations.
* Fix shadow declaration.
* Renames permissions. Switches to new method to retrieve remoteid.
* Added WS events for upsert and delete member(s).
* Added new store error type ErrUniqueConstraint.
* Added EnableCustonGroups to the client config.
* Sanitized some user records.
* Added parameter to include_total_count for listing groups.
* Added translations.
* adding deleteAt field to getByUsers query
* Revert sanitize.
* Added uniqueness constraint error to UpdateGroup.
* Removed the FutureFeatures flag so that the feature is not enabled on old Enterprise licenses.
* Renamed function.
* Updates authz check for user search related to groups.
* Removed debug statement.
* Removed unused app method.
* Added telemetry for enable_custom_groups.
* Returns early from nil license.
* Updates test.
* Returned early to avoid nesting in (*SqlGroupStore).checkUserExist. Switched to reading from replica in (*SqlGroupStore).GetMember. Handled JSON marshal error in (*Client4).UpsertGroupMembers
* Switched to SanitizeProfile.
* Switched to model.NewInt.
* Switched from status NotImplemented to Forbidden for missing license.
* Removed deactivated users from 'exists' set.
* Revert gotool update.
* Ignored lint error that I think is invalid.
* Added the approprate access tag for disabling custom groups.
* Revert change to response status.
* Fixed refactor mistake.
* Limited the group member WS events to individual users.
* Removed WS event of deleted groups.
* Updated license check for searchUsers endpoint.
* Switched from license feature to license sku.
* Update app/group.go
Co-authored-by: Claudio Costa <cstcld91@gmail.com>
* Update app/group.go
Co-authored-by: Claudio Costa <cstcld91@gmail.com>
* Remove linter ignore comment.
* Added function to create sku-specific license.
* Fixed typo. Removed comment.
* Fixed for wrong type.
* Added missing param to client. Removed unnecessary props setting. Added test for retrieving groups by source.
* Updated some tests now that we're validating group membership not created for deactivated user.
* Fix for groups endpoint returning all group types by default.
* Changes constant names. Adds migration for all users to manage custom group members.
* Removes requirement for manage_system permission to filter user search by group.
* Added migration mock.
* Removes default permissions from custom_group_user role.
* Fixes migration.
* Fixes emoji migration test.
* fixing issue with member counts
* fixing search issue for deleted members
Co-authored-by: Benjamin Cooke <benjamincooke@Benjamins-MacBook-Pro.local>
Co-authored-by: Benjamin Cooke <benjamincooke@Benjamins-MBP.ht.home>
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
Co-authored-by: Benjamin Cooke <benjamincooke@Benjamins-MacBook-Pro.fritz.box>
Co-authored-by: Claudio Costa <cstcld91@gmail.com>
* Working on refactoring jobs service
* Making more consistent with the previous existing code
* Remove no longer needed functions
* Making a base PeridicScheduler to use it in most of the schedulers implementations
* Removing accidental complexity from on of the jobs
* Removing accidental complexity from expirynotify
* Fixing compilation from previous commit
* Remove accidental complexity from the export_delete job
* Simplifying the workers by making a reusable worker
* Using simple worker for export_delete job
* Simpliying export process job
* Simpliying extract content job
* Simpliying import delete job
* Simpliying import process job
* Simpliying product noticies job
* Simpliying fix crt channel unreads job (only removing the uneeded register function)
* Simpliying migrations job (only removing the uneeded register function)
* fixup
* Simpliying plugins job (only removing the uneeded register function)
* Simpliying bleve indexing job (only removing the uneeded register function)
* Simpliying resend invitation email job (only removing the uneeded register function)
* Fixing tests
* Simplifying migration tests infrastructure
* Adding missed license to files
* Adding an empty file to imports package to ensure this package exist even without enterprise repo
* Regenerating einterfaces mocks
* Adding missed license to files
* Updating i18n/en.json file
* help fixing enterprise tests compilation
* Adding new DailyScheduler
* Fixing typo and changing the waitTime type for periodic sechduler
* Making the daily scheduler more generic
* Adding comments to clarify not used parameters in interface scheduler interface implementations
* Using merror to handle multiple errors in jobs workers
* Fixing linter errors
* Addressing PR review comments
* Reverting go.tools.mod changes
* Removing the static check for worker type in the model (moving it to the insertion of new jobs
* Moving migrations job to the jobs directory
* Fixing (and improving a bit) tests
* Apply suggestions from code review
Co-authored-by: Doug Lauder <wiggin77@warpmail.net>
* Fixing enterprise tests
* Removing unneeded InitWorkers/InitSchedulers calls
* Fix expirenotify job when error happens
* Fixing govet errors
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
Co-authored-by: Doug Lauder <wiggin77@warpmail.net>
This PR adds the necessary resolvers to support
the graphQL front-end. Some additional methods
need to be added to the model structs to support
this. Comments have been made to clarify their purpose.
There is a slight duplication of code in the api layer.
But eventually that's going to go away when we move
away from the REST API entirely.
The schema is in api4/schema.graphqls and gets compiled
into the binary as an asset.
The GraphiQL editor url is at GET /api/v5/graphql.
Everything is behind the feature flag MM_FEATUREFLAGS_GRAPHQL.
```release-note
NONE
```
* MM-36589: provide previous values for unreads
To successfully figure out the new counts of mentions or unread replies
for CRT we need to provide previous values alongside with the new.
This is needed so we'll know how many to subtract from the total.
This commit provides those numbers upon publishing websocket ThreadUpdated
and ThreadReadChanged events.
* Fixes errors
* Removes unneeded lines
* Adds GetThreadUnreadReplyCount store method
Uses the new store method to get unread replies instead of
GetThreadForUser.
* Tests, and some changes
- Adds api4 tests to test ws events
- Uses sqlx instead of gorp for new store method
- Fixes case where previous_unread_replies could be a negative value
* Refactors tests and adds more cases
* Fixes previous and current unread counts for commenter
When a user posts a reply to a thread the unread counts had a couple of
issues.
UnreadMentions where not zeroed out, and previous unread counts where
not set correctly.
This commit tries to fix that by marking the thread as read for the
current poster after we set previous unread counts to the websocket
event data.
Also MaintainMembership should zero out UnreadMentions when we are
setting the thread as read.
* Oops
* Fixes tests by updating when maintaining membership
OK, so some tests broke because we zero UnreadMentions in the membership
when we UpdateViewedTimestamp, since the new timestamp is always now.
Some tests broke because MaintainMembership for the commenter so that
the thread is read each time commenter posts moved further down in the
SendNotification method.
BOTH those test cases are fixed with this commit. To be sincere though I
don't understand why the second one is fixed by this.
* SystemAdminUser was not part of the channel
Some tests are failing because SystemAdminUser is not part of the
team and channels.
This commit adds user to team and channels, in an effort to fix
api4/user_tests
* Fixes tests
* Fixes tests
* Addresses review comments
* Fix if clause
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* MM-39612: Make acquiring and removing connections atomic
The reconnect phase of a websocket was split into two parts:
one where we check if a connection with a given connectionID
exists or not. And second, where we remove that connection
and insert the new connection again in the index.
This would lead to a race where it would be possible
for 2 concurrent requests for the same connectionID to go through
which would lead to separate goroutines working on the same dead queue.
We simplify this by removing the connection from the index
in the check connection stage itself. And then just add that
during register phase.
And to distinguish between a fresh and an old connection, we add
a new field called reuseCount.
While here, we also cleanup some old comments and add more
in some places.
https://mattermost.atlassian.net/browse/MM-39612
```release-note
NONE
```
* remove unused method
```release-note
NONE
```
* race test
```release-note
NONE
```
* Improve API4 initialization
- Refactored openGraphDataCache to be inside app layer.
- Moved the cache instance from global variable to be inside server.
- Moved out the app instantiation from the global commands package
to be instantiated on every call. Only the server instance is passed.
- Moved InitLocal to be called from inside Init.
```release-note
NONE
```
* Remove commented line
```release-note
NONE
```
* Move Channels into App
In this PR, we make Channels as part of App
instead of Server. This is part of the transition period
of moving fields from Server to Channels.
For now, Channels contains Server. So the hierarchy is
App -> Channels -> Server.
And as a first step, we also move httpService to Channels.
```release-note
NONE
```
* Fixing another test
```release-note
NONE
```
* new method
```release-note
NONE
```
Fix an issue where context fields logged by server on behalf of plugins didn't contain correct name/value pairs:
- bump Logr version to add key/value methods to sugar logger
- expose factory args when configuring logging with custom target types (needed for FocalBoard to create log target adapter that converts typed fields into slices of interface{} as per plugin logging API)
* Return an error seperately from Response
* Remove BuildErrorResponse
* Drop Response.Error from model/client4.go
* Migrate require.Nil checks
* Migrate require.NotNil checks
* More manual fixes
* Move error check out of CheckOKStatus and CheckCreatedStatus
* Move error check out of CheckForbiddenStatus
* Move error check out of CheckUnauthorizedStatus
* Move error check out of CheckNotFoundStatus
* Move error check out of CheckBadRequestStatus
* Move error check out of CheckNotImplementedStatus and CheckRequestEntityTooLargeStatus
* Move error check out of CheckInternalErrorStatus
* Move error check out of CheckServiceUnavailableStatus
* Remove error check from checkHTTPStatus
* Remove remaining references to Response.Error
* Check previously unchecked errors
* Manually fix compile and linter errors
* Return error in CreateWebSocket methods
* Return error instead of *AppError in DoApi methods
* Manually fix bad replacments
* Conistently return Response and error
* Use err instead of seperate bool return value to indicate success
* Reduce ussage of model.AppError in web/oauth_test.go
* Remove client4.Must
* Check error in buf.ReadFrom
* Fix failing tests
* MM-21357: Use typed constant for channel types
https://mattermost.atlassian.net/browse/MM-21357
```release-note
- Introduced a new type ChannelType for all channel types.
- Updated the Client4.UpdateChannelPrivacy method to ChannelType.
```
* Address review comments
```release-note
NONE
```
* telemetry fix
```release-note
NONE
```
* Revert "Revert "[MM-8497] Ability to set Do Not Disturb for a specified period of time (#16067)" (#17657)"
This reverts commit ff383990f8.
* add debug log for recurring function
* add feature flag for dnd timed status
* refactoring changes
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* Replace config generator
* Cleanup
* Some renaming and docs additions to add clarity
* Cleanup logging related methods
* Cleanup emitter
* Fix TestDefaultsGenerator
* Move feature flags synchronization logic out of config package
* Remove unnecessary util functions
* Simplify load/set logic
* Refine semantics and add some test to cover them
* Remove unnecessary deep copies
* Improve logic further
* Fix license header
* Review file store tests
* Fix test
* Fix test
* Avoid additional write during initialization
* More consistent naming
* Update app/feature_flags.go
Co-authored-by: Christopher Speller <crspeller@gmail.com>
* Update config/store.go
Co-authored-by: Christopher Speller <crspeller@gmail.com>
* Update config/store.go
Co-authored-by: Christopher Speller <crspeller@gmail.com>
* Update config/store.go
Co-authored-by: Ibrahim Serdar Acikgoz <serdaracikgoz86@gmail.com>
* Make ConfigStore.Set() return both old and new configs
* Implement config diff function
* Make app.SaveConfig return previous and current configs
* Add config diff to audit record
* Fix returned configs
* Include high level test
* Move FF synchronizer to its own package
* Remove unidiomatic use of sync.Once
* Add some comments
* Rename function
* More comment
* Save config diff in audit record for local endpoints
* Enable audit for config set/reset commands
* Improve tests output
Co-authored-by: Christopher Speller <crspeller@gmail.com>
Co-authored-by: Ibrahim Serdar Acikgoz <serdaracikgoz86@gmail.com>
* add request context
* move initialialization to server
* use app interface instead of global app functions
* remove app context from webconn
* cleanup
* remove duplicated services
* move context to separate package
* remove finalize init method and move content to NewServer function
* restart workers and schedulers after adding license for tests
* reflect review comments
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* Add support for timed DND status
- accept a date time value in api query when dnd mode for user needs to be unset
- Create a new function to handle SetDNDStatus calls
- Create a scheduled task to unset dnd mode to wahtever mode was before setting it to DND
* update schema version
* Model changes to make fields more intuitive
- move dndendtime to status model
- add new field prev status in status to keep track of previous status of user
- update db migration function
- make use of prevstatus and dndendtime from status model
* set prev status and dndendtime appropriately after unsetting dnd mode
* add json tag for dndendtime
* unset dnd status only if not changed manually by user
* update dnd statuses after server restart
* make app-layers
* fix failing tests
* don't create sched task when setting status to DND
* get only expired statuses from db
- convert end time from any timezone to utc
- store dnd end time in unix format for usability reasons
* run update dnd status only on leader
* make mocks
* fix tests
* run UpdateDNDStatusOfUsers as recurring task
* save all statuses at once in db and update UpdateDNDStatusOfUsers logic
* add app method to get timezone of user
* store dnd end time in context.Params
* set max size of prevstatus
* update status model to take endtime input as string and store in db as unix time(int64)
* Add tests for SetStatusDoNotDisturbTimed
* if dnd_end_time is not passed the call old api to set dnd mode
* fix tests
* new plugin api to use new timed dnd mode
* get and update rows in a single db query
* dnd end time will be stored in request body and not route param
* exclude statuses which has dndendtimeunix < 0
* update and get the updated dnd statuses in single db query
* add updated status to cache
* DNDEndTimeUnix and PrevStatus need not to be visible to users
* update db schema version for migration
* Keep Status and PrevStatus varchar size same
* add test to verify status is restored after dnd end time expires
* expect endtime in utc from client
- remove store method GetTimezone as no longer needed
- add documentation for SetStatusDoNotDisturbTimed
* reduce sleep time for dnd timed restore test
* more appropriate name for new api to update user status
* update db migration function
* parse and validate time before potentially triggering db query to get status of user
* add migration changes in to existing upgrade function
* not supporting un-timed dnd status via api
* don't call Srv.Store directly, call via app layer
* rename dndendtime to statuscleartime to make it suitable for custom status usage as well
* Revert "rename dndendtime to statuscleartime to make it suitable for custom status usage as well"
This reverts commit fa69152d9a3db18f1c59b34c878fb7ce494440b5.
* mysql doesn't support RETURNING clause so add tx to get and update statuses
* add UpdateDNDStatusOfUsers mock in tests
* update store mock import path
* add mock in storelib
* Add status mocks to empty store
* Close the task during server shutdown
* Do not cancel a nil task
* update squirrel queries
* remove untimed dnd test
* start recurring task to unset statuses on leadership change
* set dndTask to nil after cancelling it upon server shutdown
* new recurring task which starts at nearest rounded time of the interval
* mock Get() call for status
* return updated statuses in case of mysql
* remove unneccessary code
* add Get() mock to empty store
* fix mocking for once and all
* address review comments
fix mysql updateStatus fn
protect dndTask with mutex
minor refactors
* move runDNDStatusExpireJob to server.go and pass App as arg instead of method receiver
* frontend will send endtime in unix epoch format so get rid of double representation
* scan for all fields and not just two
* add some tests and fix review comments
* remove extra sql query and create needed result in go
* add storetest for UpdateExpiredDNDStatuses
* add migrations to latest version
* update min supported version
* add comment to fix a bug in future
* update test to expect 1 status in return
* rename UpdateUserStatusWithDNDTimeout to SetUserStatusTimedDND
* rename DNDEndTimeUnix to DNDEndTime
* cast int to int64 for equality
* fix tests and error handling
* move updating values to retrieved statuses fields outside sql transaction
* move migrations to 5.36
Co-authored-by: Agniva De Sarker <agnivade@yahoo.co.in>
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* MM-34002: Improve AddUserToChannel
When we would add a user to a channel, we would
check whether the user is removed from that team or not.
During LDAP sync, this check is not required because the
team member would have just been created. Hence, we
pass a boolean flag to bypass the check.
And with that done, we can freely query the replica.
https://mattermost.atlassian.net/browse/MM-34002
```release-note
NONE
```
* Refactor code
* Rename a struct field
* fix double negative
During LDAP sync, we would call AddTeamMember which had a read-after-write issue
where we would create a team member but then immediately after that
query the team member.
The same pattern was found in:
AddTeamMember
AddTeamMembers
AddTeamMemberByToken
To fix this, we just return the inserted team member from AddUserToTeam and use that
instead of query GetTeamMember again.
```release-note
NONE
```
https://mattermost.atlassian.net/browse/MM-33913
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* MM-31094: Adds tooling to develop and test using a MySQL instance with replication lag. Adds some lazy lookups to fallback to master if results are not found.
* MM-31094: Removes mysql-read-replica from default docker services.
* MM-31094: Switches (store..SessionStore).Get and (store.TeamStore).GetMember to using context.Context.
* MM-31094: Updates (store.UsersStore).Get to use context.
* MM-31094: Updates (store.PostStore).Get to use context.
* MM-31094: Removes feature flag and config setting.
* MM-31094: Rolls back some master reads.
* MM-31094: Rolls a non-cache read.
* MM-31094: Removes feature flag from the store.
* MM-31094: Removes unused constant and struct field.
* MM-31094: Removes some old feature flag references.
* MM-31094: Fixes some tests.
* MM-31094: App layers fix.
* MM-31094: Fixes mocks.
* MM-31094: Don't reparse flag.
* MM-31094: No reparse.
* MM-31094: Removed unused FeatureFlags field.
* MM-31094: Removes unnecessary feature flags variable declarations.
* MM-31094: Fixes copy-paste error.
* MM-31094: Fixes logical error.
* MM-30194: Removes test method from store.
* Revert "MM-30194: Removes test method from store."
This reverts commit d5a6e8529bd5f4d993824c828e239d009b05e567.
* MM-31094: Conforming to make's strange syntax.
* MM-31094: Configures helper for read replica with option.
* MM-31094: Adds some missing ctx's.
* MM-31094: WIP
* MM-31094: Updates test names.
* MM-31094: WIP
* MM-31094: Removes unnecessary master reads.
* MM-31094: ID case changes out of scope.
* MM-31094: Removes unused context.
* MM-31094: Switches to a helper. Removes some var naming changes. Fixes a merge error.
* MM-31094: Removes SQLITE db driver ref.
* MM-31094: Layer generate fix.
* MM-31094: Removes unnecessary changes.
* MM-31094: Moves test method.
* MM-31094: Re-add previous fix.
* MM-31094: Removes make command for dev.
* MM-31094: Fix for login.
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* format using `goimports -local github.com/mattermost/mattermost-server/v5 -w`
* added goimports lint check to .golangci.yml
* format using `goimports -local github.com/mattermost/mattermost-server/v5 -w` for a corner case
* make app-layers, *-mocks and store-layers for ci check
Co-authored-by: Mahmudul Haque <mahmudulhaque@protonmail.com>
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* Installing plugins specified by feature flags using the marketplace.
* Switch back to using getplugins client.
* Respect disabling automatic installation of pluings.
* pluginid -> plugin_id
* Debug logs for enable plugin error
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* Removing supplier concept from the sql store
* Removing other metions to supplier
* Fixing gofmt
* Fixing gofmt
* Renaming NewSqlStore to New
* Fixing tests
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
