Custom groups (#18839)

* WIP

* adding initial creategroup endpoint

* fetching by group source

* fixing startup error

* updating create endpoint to take an array of user_ids, this will allow us to create the group with one request

* adding delete group endpoint and appropriate test

* adding source param for getGroups

* adding add members and delete members endpoints

* locking down crud endpoints to only be allowed for custom groups

* user search stuff

* allowing remoteid be null by changing field to pointer

* code cleanup and store level tests

* adding new tests and removing unused endpoint

* resolving conflicts

* Adds authz check for group.

* Adds authz checks to groups APIs.

* Updated create group authz tests.

* Updates delete group tests.

* Tests create group.

* Adds some tests and validations.

* adding new parameter so I can get users not in a group

* Fixed all lint warnings.

* Fix type.

* fixing search users not in group

* Fixes some lint errors.

* Moves entry in JSON array.

* Fixed SQL query.

* Fixes permission migration test.

* Fixes migration test.

* Fixes some group store tests.

* Fix test.

* Fix test.

* Revert lint change.

* Migrated CreateWithUserIds to sqlx.

* Adds tests for GetMember; migrates implementation to sqlx.

* Tests GetNonMemberUsersPage and hanles wrong group id.

* Fixes test.

* Switches GetMaster to GetMasterX.

* Switches GetReplica to GetReplicaX.

* Fixes logic.

* Fixes shadow declaration.

* Adds include_member_count to get group API endpoint.

* Adds filter_has_member param to getGroups.

* Fixes.

* Removes array of group sources.

* fixing error

* Testing reverting CreateWithUserIds back to gorp.

* Added websocket event for CreateGroupWithUserIds.

* Changed a few response status codes. Switched to correct permission.

* Added member count to ws payload for group when updating or creating.

* Adds feature flag checks for custom groups.

* Added middleware function to require license. Added config to disable custom groups.

* Change for function signature change of executePossiblyEmptyQuery.

* Lint fixes.

* Adds telemetry none comment.

* Adds translations.

* Migrated to sqlx.

* Temp. removal of translation.

* Fixed typo.

* Added an intermediary model to query with a field that is now ignored by sqlx on read queries.

* Re-used existing store struct.

* Inludes member count.

* Fix for merge error.'

* Require license for group endpoints.

* Updates translations.

* Fix shadow declaration.

* Renames permissions. Switches to new method to retrieve remoteid.

* Added WS events for upsert and delete member(s).

* Added new store error type ErrUniqueConstraint.

* Added EnableCustonGroups to the client config.

* Sanitized some user records.

* Added parameter to include_total_count for listing groups.

* Added translations.

* adding deleteAt field to getByUsers query

* Revert sanitize.

* Added uniqueness constraint error to UpdateGroup.

* Removed the FutureFeatures flag so that the feature is not enabled on old Enterprise licenses.

* Renamed function.

* Updates authz check for user search related to groups.

* Removed debug statement.

* Removed unused app method.

* Added telemetry for enable_custom_groups.

* Returns early from nil license.

* Updates test.

* Returned early to avoid nesting in (*SqlGroupStore).checkUserExist. Switched to reading from replica in (*SqlGroupStore).GetMember. Handled JSON marshal error in (*Client4).UpsertGroupMembers

* Switched to SanitizeProfile.

* Switched to model.NewInt.

* Switched from status NotImplemented to Forbidden for missing license.

* Removed deactivated users from 'exists' set.

* Revert gotool update.

* Ignored lint error that I think is invalid.

* Added the approprate access tag for disabling custom groups.

* Revert change to response status.

* Fixed refactor mistake.

* Limited the group member WS events to individual users.

* Removed WS event of deleted groups.

* Updated license check for searchUsers endpoint.

* Switched from license feature to license sku.

* Update app/group.go

Co-authored-by: Claudio Costa <cstcld91@gmail.com>

* Update app/group.go

Co-authored-by: Claudio Costa <cstcld91@gmail.com>

* Remove linter ignore comment.

* Added function to create sku-specific license.

* Fixed typo. Removed comment.

* Fixed for wrong type.

* Added missing param to client. Removed unnecessary props setting. Added test for retrieving groups by source.

* Updated some tests now that we're validating group membership not created for deactivated user.

* Fix for groups endpoint returning all group types by default.

* Changes constant names. Adds migration for all users to manage custom group members.

* Removes requirement for manage_system permission to filter user search by group.

* Added migration mock.

* Removes default permissions from custom_group_user role.

* Fixes migration.

* Fixes emoji migration test.

* fixing issue with member counts

* fixing search issue for deleted members

Co-authored-by: Benjamin Cooke <benjamincooke@Benjamins-MacBook-Pro.local>
Co-authored-by: Benjamin Cooke <benjamincooke@Benjamins-MBP.ht.home>
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
Co-authored-by: Benjamin Cooke <benjamincooke@Benjamins-MacBook-Pro.fritz.box>
Co-authored-by: Claudio Costa <cstcld91@gmail.com>

model/group.go

@@ -9,7 +9,8 @@ import (
 )
 
 const (
-	GroupSourceLdap GroupSource = "ldap"
+	GroupSourceLdap   GroupSource = "ldap"
+	GroupSourceCustom GroupSource = "custom"
 
 	GroupNameMaxLength        = 64
 	GroupSourceMaxLength      = 64
@@ -22,6 +23,7 @@ type GroupSource string
 
 var allGroupSources = []GroupSource{
 	GroupSourceLdap,
+	GroupSourceCustom,
 }
 
 var groupSourcesRequiringRemoteID = []GroupSource{
@@ -34,7 +36,7 @@ type Group struct {
 	DisplayName    string      `json:"display_name"`
 	Description    string      `json:"description"`
 	Source         GroupSource `json:"source"`
-	RemoteId       string      `json:"remote_id"`
+	RemoteId       *string     `json:"remote_id"`
 	CreateAt       int64       `json:"create_at"`
 	UpdateAt       int64       `json:"update_at"`
 	DeleteAt       int64       `json:"delete_at"`
@@ -43,6 +45,11 @@ type Group struct {
 	AllowReference bool        `json:"allow_reference"`
 }
 
+type GroupWithUserIds struct {
+	Group
+	UserIds []string `json:"user_ids"`
+}
+
 type GroupWithSchemeAdmin struct {
 	Group
 	SchemeAdmin *bool `db:"SyncableSchemeAdmin" json:"scheme_admin,omitempty"`
@@ -63,6 +70,8 @@ type GroupPatch struct {
 	DisplayName    *string `json:"display_name"`
 	Description    *string `json:"description"`
 	AllowReference *bool   `json:"allow_reference"`
+	// For security reasons (including preventing unintended LDAP group synchronization) do no allow a Group's RemoteId or Source field to be
+	// included in patches.
 }
 
 type LdapGroupSearchOpts struct {
@@ -79,12 +88,21 @@ type GroupSearchOpts struct {
 	FilterAllowReference   bool
 	PageOpts               *PageOpts
 	Since                  int64
+	Source                 GroupSource
 
 	// FilterParentTeamPermitted filters the groups to the intersect of the
 	// set associated to the parent team and those returned by the query.
 	// If the parent team is not group-constrained or if NotAssociatedToChannel
 	// is not set then this option is ignored.
 	FilterParentTeamPermitted bool
+
+	// FilterHasMember filters the groups to the intersect of the
+	// set returned by the query and those that have the given user as a member.
+	FilterHasMember string
+}
+
+type GetGroupOpts struct {
+	IncludeMemberCount bool
 }
 
 type PageOpts struct {
@@ -97,6 +115,10 @@ type GroupStats struct {
 	TotalMemberCount int64  `json:"total_member_count"`
 }
 
+type GroupModifyMembers struct {
+	UserIds []string `json:"user_ids"`
+}
+
 func (group *Group) Patch(patch *GroupPatch) {
 	if patch.Name != nil {
 		group.Name = patch.Name
@@ -137,7 +159,7 @@ func (group *Group) IsValidForCreate() *AppError {
 		return NewAppError("Group.IsValidForCreate", "model.group.source.app_error", nil, "", http.StatusBadRequest)
 	}
 
-	if len(group.RemoteId) > GroupRemoteIDMaxLength || (group.RemoteId == "" && group.requiresRemoteId()) {
+	if (group.GetRemoteId() == "" && group.requiresRemoteId()) || len(group.GetRemoteId()) > GroupRemoteIDMaxLength {
 		return NewAppError("Group.IsValidForCreate", "model.group.remote_id.app_error", nil, "", http.StatusBadRequest)
 	}
 
@@ -188,3 +210,15 @@ func (group *Group) IsValidName() *AppError {
 	}
 	return nil
 }
+
+func (group *Group) GetRemoteId() string {
+	if group.RemoteId == nil {
+		return ""
+	}
+	return *group.RemoteId
+}
+
+type GroupsWithCount struct {
+	Groups     []*Group `json:"groups"`
+	TotalCount int64    `json:"total_count"`
+}