Custom groups (#18839)

* WIP

* adding initial creategroup endpoint

* fetching by group source

* fixing startup error

* updating create endpoint to take an array of user_ids, this will allow us to create the group with one request

* adding delete group endpoint and appropriate test

* adding source param for getGroups

* adding add members and delete members endpoints

* locking down crud endpoints to only be allowed for custom groups

* user search stuff

* allowing remoteid be null by changing field to pointer

* code cleanup and store level tests

* adding new tests and removing unused endpoint

* resolving conflicts

* Adds authz check for group.

* Adds authz checks to groups APIs.

* Updated create group authz tests.

* Updates delete group tests.

* Tests create group.

* Adds some tests and validations.

* adding new parameter so I can get users not in a group

* Fixed all lint warnings.

* Fix type.

* fixing search users not in group

* Fixes some lint errors.

* Moves entry in JSON array.

* Fixed SQL query.

* Fixes permission migration test.

* Fixes migration test.

* Fixes some group store tests.

* Fix test.

* Fix test.

* Revert lint change.

* Migrated CreateWithUserIds to sqlx.

* Adds tests for GetMember; migrates implementation to sqlx.

* Tests GetNonMemberUsersPage and hanles wrong group id.

* Fixes test.

* Switches GetMaster to GetMasterX.

* Switches GetReplica to GetReplicaX.

* Fixes logic.

* Fixes shadow declaration.

* Adds include_member_count to get group API endpoint.

* Adds filter_has_member param to getGroups.

* Fixes.

* Removes array of group sources.

* fixing error

* Testing reverting CreateWithUserIds back to gorp.

* Added websocket event for CreateGroupWithUserIds.

* Changed a few response status codes. Switched to correct permission.

* Added member count to ws payload for group when updating or creating.

* Adds feature flag checks for custom groups.

* Added middleware function to require license. Added config to disable custom groups.

* Change for function signature change of executePossiblyEmptyQuery.

* Lint fixes.

* Adds telemetry none comment.

* Adds translations.

* Migrated to sqlx.

* Temp. removal of translation.

* Fixed typo.

* Added an intermediary model to query with a field that is now ignored by sqlx on read queries.

* Re-used existing store struct.

* Inludes member count.

* Fix for merge error.'

* Require license for group endpoints.

* Updates translations.

* Fix shadow declaration.

* Renames permissions. Switches to new method to retrieve remoteid.

* Added WS events for upsert and delete member(s).

* Added new store error type ErrUniqueConstraint.

* Added EnableCustonGroups to the client config.

* Sanitized some user records.

* Added parameter to include_total_count for listing groups.

* Added translations.

* adding deleteAt field to getByUsers query

* Revert sanitize.

* Added uniqueness constraint error to UpdateGroup.

* Removed the FutureFeatures flag so that the feature is not enabled on old Enterprise licenses.

* Renamed function.

* Updates authz check for user search related to groups.

* Removed debug statement.

* Removed unused app method.

* Added telemetry for enable_custom_groups.

* Returns early from nil license.

* Updates test.

* Returned early to avoid nesting in (*SqlGroupStore).checkUserExist. Switched to reading from replica in (*SqlGroupStore).GetMember. Handled JSON marshal error in (*Client4).UpsertGroupMembers

* Switched to SanitizeProfile.

* Switched to model.NewInt.

* Switched from status NotImplemented to Forbidden for missing license.

* Removed deactivated users from 'exists' set.

* Revert gotool update.

* Ignored lint error that I think is invalid.

* Added the approprate access tag for disabling custom groups.

* Revert change to response status.

* Fixed refactor mistake.

* Limited the group member WS events to individual users.

* Removed WS event of deleted groups.

* Updated license check for searchUsers endpoint.

* Switched from license feature to license sku.

* Update app/group.go

Co-authored-by: Claudio Costa <cstcld91@gmail.com>

* Update app/group.go

Co-authored-by: Claudio Costa <cstcld91@gmail.com>

* Remove linter ignore comment.

* Added function to create sku-specific license.

* Fixed typo. Removed comment.

* Fixed for wrong type.

* Added missing param to client. Removed unnecessary props setting. Added test for retrieving groups by source.

* Updated some tests now that we're validating group membership not created for deactivated user.

* Fix for groups endpoint returning all group types by default.

* Changes constant names. Adds migration for all users to manage custom group members.

* Removes requirement for manage_system permission to filter user search by group.

* Added migration mock.

* Removes default permissions from custom_group_user role.

* Fixes migration.

* Fixes emoji migration test.

* fixing issue with member counts

* fixing search issue for deleted members

Co-authored-by: Benjamin Cooke <benjamincooke@Benjamins-MacBook-Pro.local>
Co-authored-by: Benjamin Cooke <benjamincooke@Benjamins-MBP.ht.home>
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
Co-authored-by: Benjamin Cooke <benjamincooke@Benjamins-MacBook-Pro.fritz.box>
Co-authored-by: Claudio Costa <cstcld91@gmail.com>

app/group.go

@@ -13,7 +13,7 @@ import (
 	"github.com/mattermost/mattermost-server/v6/store"
 )
 
-func (a *App) GetGroup(id string) (*model.Group, *model.AppError) {
+func (a *App) GetGroup(id string, opts *model.GetGroupOpts) (*model.Group, *model.AppError) {
 	group, err := a.Srv().Store.Group().Get(id)
 	if err != nil {
 		var nfErr *store.ErrNotFound
@@ -25,6 +25,14 @@ func (a *App) GetGroup(id string) (*model.Group, *model.AppError) {
 		}
 	}
 
+	if opts != nil && opts.IncludeMemberCount {
+		memberCount, err := a.Srv().Store.Group().GetMemberCount(id)
+		if err != nil {
+			return nil, model.NewAppError("GetGroup", "app.member_count", nil, err.Error(), http.StatusInternalServerError)
+		}
+		group.MemberCount = model.NewInt(int(memberCount))
+	}
+
 	return group, nil
 }
 
@@ -94,10 +102,49 @@ func (a *App) CreateGroup(group *model.Group) (*model.Group, *model.AppError) {
 	return group, nil
 }
 
+func (a *App) CreateGroupWithUserIds(group *model.GroupWithUserIds) (*model.Group, *model.AppError) {
+	newGroup, err := a.Srv().Store.Group().CreateWithUserIds(group)
+	if err != nil {
+		var invErr *store.ErrInvalidInput
+		var appErr *model.AppError
+		var dupKey *store.ErrUniqueConstraint
+		switch {
+		case errors.As(err, &appErr):
+			return nil, appErr
+		case errors.As(err, &invErr):
+			return nil, model.NewAppError("CreateGroupWithUserIds", "app.group.id.app_error", nil, invErr.Error(), http.StatusBadRequest)
+		case errors.As(err, &dupKey):
+			return nil, model.NewAppError("CreateGroup", "app.custom_group.unique_name", nil, dupKey.Error(), http.StatusBadRequest)
+		default:
+			return nil, model.NewAppError("CreateGroupWithUserIds", "app.insert_error", nil, err.Error(), http.StatusInternalServerError)
+		}
+	}
+
+	messageWs := model.NewWebSocketEvent(model.WebsocketEventReceivedGroup, "", "", "", nil)
+	count, err := a.Srv().Store.Group().GetMemberCount(newGroup.Id)
+	if err != nil {
+		return nil, model.NewAppError("CreateGroupWithUserIds", "app.group.id.app_error", nil, err.Error(), http.StatusBadRequest)
+	}
+	group.MemberCount = model.NewInt(int(count))
+	groupJSON, jsonErr := json.Marshal(newGroup)
+	if jsonErr != nil {
+		mlog.Warn("Failed to encode group to JSON", mlog.Err(jsonErr))
+	}
+	messageWs.Add("group", string(groupJSON))
+	a.Publish(messageWs)
+
+	return newGroup, nil
+}
+
 func (a *App) UpdateGroup(group *model.Group) (*model.Group, *model.AppError) {
 	updatedGroup, err := a.Srv().Store.Group().Update(group)
 
 	if err == nil {
+		count, countErr := a.Srv().Store.Group().GetMemberCount(updatedGroup.Id)
+		if countErr != nil {
+			return nil, model.NewAppError("CreateGroupWithUserIds", "app.group.id.app_error", nil, countErr.Error(), http.StatusBadRequest)
+		}
+		updatedGroup.MemberCount = model.NewInt(int(count))
 		messageWs := model.NewWebSocketEvent(model.WebsocketEventReceivedGroup, "", "", "", nil)
 		groupJSON, jsonErr := json.Marshal(updatedGroup)
 		if jsonErr != nil {
@@ -110,11 +157,14 @@ func (a *App) UpdateGroup(group *model.Group) (*model.Group, *model.AppError) {
 	if err != nil {
 		var nfErr *store.ErrNotFound
 		var appErr *model.AppError
+		var dupKey *store.ErrUniqueConstraint
 		switch {
 		case errors.As(err, &appErr):
 			return nil, appErr
 		case errors.As(err, &nfErr):
 			return nil, model.NewAppError("UpdateGroup", "app.group.no_rows", nil, nfErr.Error(), http.StatusNotFound)
+		case errors.As(err, &dupKey):
+			return nil, model.NewAppError("CreateGroup", "app.custom_group.unique_name", nil, dupKey.Error(), http.StatusBadRequest)
 		default:
 			return nil, model.NewAppError("UpdateGroup", "app.select_error", nil, err.Error(), http.StatusInternalServerError)
 		}
@@ -125,17 +175,6 @@ func (a *App) UpdateGroup(group *model.Group) (*model.Group, *model.AppError) {
 
 func (a *App) DeleteGroup(groupID string) (*model.Group, *model.AppError) {
 	deletedGroup, err := a.Srv().Store.Group().Delete(groupID)
-
-	if err == nil {
-		messageWs := model.NewWebSocketEvent(model.WebsocketEventReceivedGroup, "", "", "", nil)
-		groupJSON, jsonErr := json.Marshal(deletedGroup)
-		if jsonErr != nil {
-			mlog.Warn("Failed to encode group to JSON", mlog.Err(jsonErr))
-		}
-		messageWs.Add("group", string(groupJSON))
-		a.Publish(messageWs)
-	}
-
 	if err != nil {
 		var nfErr *store.ErrNotFound
 		switch {
@@ -177,7 +216,15 @@ func (a *App) GetGroupMemberUsersPage(groupID string, page int, perPage int) ([]
 	if appErr != nil {
 		return nil, 0, appErr
 	}
-	return members, int(count), nil
+	return a.sanitizeProfiles(members, false), int(count), nil
+}
+func (a *App) GetUsersNotInGroupPage(groupID string, page int, perPage int) ([]*model.User, *model.AppError) {
+	members, err := a.Srv().Store.Group().GetNonMemberUsersPage(groupID, page, perPage)
+	if err != nil {
+		return nil, model.NewAppError("GetUsersNotInGroupPage", "app.select_error", nil, err.Error(), http.StatusInternalServerError)
+	}
+
+	return a.sanitizeProfiles(members, false), nil
 }
 
 func (a *App) UpsertGroupMember(groupID string, userID string) (*model.GroupMember, *model.AppError) {
@@ -195,6 +242,8 @@ func (a *App) UpsertGroupMember(groupID string, userID string) (*model.GroupMemb
 		}
 	}
 
+	a.publishGroupMemberEvent(model.WebsocketEventGroupMemberAdd, groupMember)
+
 	return groupMember, nil
 }
 
@@ -210,6 +259,8 @@ func (a *App) DeleteGroupMember(groupID string, userID string) (*model.GroupMemb
 		}
 	}
 
+	a.publishGroupMemberEvent(model.WebsocketEventGroupMemberDelete, groupMember)
+
 	return groupMember, nil
 }
 
@@ -515,6 +566,10 @@ func (a *App) TeamMembersMinusGroupMembers(teamID string, groupIDs []string, pag
 		return nil, 0, model.NewAppError("TeamMembersMinusGroupMembers", "app.select_error", nil, err.Error(), http.StatusInternalServerError)
 	}
 
+	for _, u := range users {
+		a.SanitizeProfile(&u.User, false)
+	}
+
 	// parse all group ids of all users
 	allUsersGroupIDMap := map[string]bool{}
 	for _, user := range users {
@@ -579,6 +634,10 @@ func (a *App) ChannelMembersMinusGroupMembers(channelID string, groupIDs []strin
 		return nil, 0, model.NewAppError("ChannelMembersMinusGroupMembers", "app.select_error", nil, err.Error(), http.StatusInternalServerError)
 	}
 
+	for _, u := range users {
+		a.SanitizeProfile(&u.User, false)
+	}
+
 	// parse all group ids of all users
 	allUsersGroupIDMap := map[string]bool{}
 	for _, user := range users {
@@ -637,3 +696,57 @@ func (a *App) UserIsInAdminRoleGroup(userID, syncableID string, syncableType mod
 
 	return true, nil
 }
+
+func (a *App) UpsertGroupMembers(groupID string, userIDs []string) ([]*model.GroupMember, *model.AppError) {
+	members, err := a.Srv().Store.Group().UpsertMembers(groupID, userIDs)
+	if err != nil {
+		var invErr *store.ErrInvalidInput
+		var appErr *model.AppError
+		switch {
+		case errors.As(err, &appErr):
+			return nil, appErr
+		case errors.As(err, &invErr):
+			return nil, model.NewAppError("UpsertGroupMembers", "app.group.uniqueness_error", nil, invErr.Error(), http.StatusBadRequest)
+		default:
+			return nil, model.NewAppError("UpsertGroupMembers", "app.update_error", nil, err.Error(), http.StatusInternalServerError)
+		}
+	}
+
+	for _, groupMember := range members {
+		a.publishGroupMemberEvent(model.WebsocketEventGroupMemberAdd, groupMember)
+	}
+
+	return members, nil
+}
+
+func (a *App) DeleteGroupMembers(groupID string, userIDs []string) ([]*model.GroupMember, *model.AppError) {
+	members, err := a.Srv().Store.Group().DeleteMembers(groupID, userIDs)
+	if err != nil {
+		var invErr *store.ErrInvalidInput
+		var appErr *model.AppError
+		switch {
+		case errors.As(err, &appErr):
+			return nil, appErr
+		case errors.As(err, &invErr):
+			return nil, model.NewAppError("DeleteGroupMember", "app.group.uniqueness_error", nil, invErr.Error(), http.StatusBadRequest)
+		default:
+			return nil, model.NewAppError("DeleteGroupMember", "app.update_error", nil, err.Error(), http.StatusInternalServerError)
+		}
+	}
+
+	for _, groupMember := range members {
+		a.publishGroupMemberEvent(model.WebsocketEventGroupMemberDelete, groupMember)
+	}
+
+	return members, nil
+}
+
+func (a *App) publishGroupMemberEvent(eventName string, groupMember *model.GroupMember) {
+	messageWs := model.NewWebSocketEvent(eventName, "", "", groupMember.UserId, nil)
+	groupMemberJSON, jsonErr := json.Marshal(groupMember)
+	if jsonErr != nil {
+		mlog.Warn("failed to encode group member to JSON", mlog.Err(jsonErr))
+	}
+	messageWs.Add("group_member", string(groupMemberJSON))
+	a.Publish(messageWs)
+}