Custom groups (#18839)

* WIP * adding initial creategroup endpoint * fetching by group source * fixing startup error * updating create endpoint to take an array of user_ids, this will allow us to create the group with one request * adding delete group endpoint and appropriate test * adding source param for getGroups * adding add members and delete members endpoints * locking down crud endpoints to only be allowed for custom groups * user search stuff * allowing remoteid be null by changing field to pointer * code cleanup and store level tests * adding new tests and removing unused endpoint * resolving conflicts * Adds authz check for group. * Adds authz checks to groups APIs. * Updated create group authz tests. * Updates delete group tests. * Tests create group. * Adds some tests and validations. * adding new parameter so I can get users not in a group * Fixed all lint warnings. * Fix type. * fixing search users not in group * Fixes some lint errors. * Moves entry in JSON array. * Fixed SQL query. * Fixes permission migration test. * Fixes migration test. * Fixes some group store tests. * Fix test. * Fix test. * Revert lint change. * Migrated CreateWithUserIds to sqlx. * Adds tests for GetMember; migrates implementation to sqlx. * Tests GetNonMemberUsersPage and hanles wrong group id. * Fixes test. * Switches GetMaster to GetMasterX. * Switches GetReplica to GetReplicaX. * Fixes logic. * Fixes shadow declaration. * Adds include_member_count to get group API endpoint. * Adds filter_has_member param to getGroups. * Fixes. * Removes array of group sources. * fixing error * Testing reverting CreateWithUserIds back to gorp. * Added websocket event for CreateGroupWithUserIds. * Changed a few response status codes. Switched to correct permission. * Added member count to ws payload for group when updating or creating. * Adds feature flag checks for custom groups. * Added middleware function to require license. Added config to disable custom groups. * Change for function signature change of executePossiblyEmptyQuery. * Lint fixes. * Adds telemetry none comment. * Adds translations. * Migrated to sqlx. * Temp. removal of translation. * Fixed typo. * Added an intermediary model to query with a field that is now ignored by sqlx on read queries. * Re-used existing store struct. * Inludes member count. * Fix for merge error.' * Require license for group endpoints. * Updates translations. * Fix shadow declaration. * Renames permissions. Switches to new method to retrieve remoteid. * Added WS events for upsert and delete member(s). * Added new store error type ErrUniqueConstraint. * Added EnableCustonGroups to the client config. * Sanitized some user records. * Added parameter to include_total_count for listing groups. * Added translations. * adding deleteAt field to getByUsers query * Revert sanitize. * Added uniqueness constraint error to UpdateGroup. * Removed the FutureFeatures flag so that the feature is not enabled on old Enterprise licenses. * Renamed function. * Updates authz check for user search related to groups. * Removed debug statement. * Removed unused app method. * Added telemetry for enable_custom_groups. * Returns early from nil license. * Updates test. * Returned early to avoid nesting in (*SqlGroupStore).checkUserExist. Switched to reading from replica in (*SqlGroupStore).GetMember. Handled JSON marshal error in (*Client4).UpsertGroupMembers * Switched to SanitizeProfile. * Switched to model.NewInt. * Switched from status NotImplemented to Forbidden for missing license. * Removed deactivated users from 'exists' set. * Revert gotool update. * Ignored lint error that I think is invalid. * Added the approprate access tag for disabling custom groups. * Revert change to response status. * Fixed refactor mistake. * Limited the group member WS events to individual users. * Removed WS event of deleted groups. * Updated license check for searchUsers endpoint. * Switched from license feature to license sku. * Update app/group.go Co-authored-by: Claudio Costa <cstcld91@gmail.com> * Update app/group.go Co-authored-by: Claudio Costa <cstcld91@gmail.com> * Remove linter ignore comment. * Added function to create sku-specific license. * Fixed typo. Removed comment. * Fixed for wrong type. * Added missing param to client. Removed unnecessary props setting. Added test for retrieving groups by source. * Updated some tests now that we're validating group membership not created for deactivated user. * Fix for groups endpoint returning all group types by default. * Changes constant names. Adds migration for all users to manage custom group members. * Removes requirement for manage_system permission to filter user search by group. * Added migration mock. * Removes default permissions from custom_group_user role. * Fixes migration. * Fixes emoji migration test. * fixing issue with member counts * fixing search issue for deleted members Co-authored-by: Benjamin Cooke <benjamincooke@Benjamins-MacBook-Pro.local> Co-authored-by: Benjamin Cooke <benjamincooke@Benjamins-MBP.ht.home> Co-authored-by: Mattermod <mattermod@users.noreply.github.com> Co-authored-by: Benjamin Cooke <benjamincooke@Benjamins-MacBook-Pro.fritz.box> Co-authored-by: Claudio Costa <cstcld91@gmail.com>
2022-02-17 12:34:39 -05:00
--- a/model/client4.go
+++ b/model/client4.go
@@ -5253,7 +5253,7 @@ func (c *Client4) GetGroupsAssociatedToChannelsByTeam(teamId string, opts GroupS
 // GetGroups retrieves Mattermost Groups
 func (c *Client4) GetGroups(opts GroupSearchOpts) ([]*Group, *Response, error) {
 	path := fmt.Sprintf(
-		"%s?include_member_count=%v&not_associated_to_team=%v&not_associated_to_channel=%v&filter_allow_reference=%v&q=%v&filter_parent_team_permitted=%v",
+		"%s?include_member_count=%v&not_associated_to_team=%v&not_associated_to_channel=%v&filter_allow_reference=%v&q=%v&filter_parent_team_permitted=%v&group_source=%v",
 		c.groupsRoute(),
 		opts.IncludeMemberCount,
 		opts.NotAssociatedToTeam,
@@ -5261,6 +5261,7 @@ func (c *Client4) GetGroups(opts GroupSearchOpts) ([]*Group, *Response, error) {
 		opts.FilterAllowReference,
 		opts.Q,
 		opts.FilterParentTeamPermitted,
+		opts.Source,
 	)
 	if opts.Since > 0 {
 		path = fmt.Sprintf("%s&since=%v", path, opts.Since)
@@ -7072,6 +7073,36 @@ func (c *Client4) GetGroup(groupID, etag string) (*Group, *Response, error) {
 	return &g, BuildResponse(r), nil
 }

+func (c *Client4) CreateGroup(group *Group) (*Group, *Response, error) {
+	groupJSON, jsonErr := json.Marshal(group)
+	if jsonErr != nil {
+		return nil, nil, NewAppError("CreateGroup", "api.marshal_error", nil, jsonErr.Error(), http.StatusInternalServerError)
+	}
+	r, err := c.DoAPIPostBytes("/groups", groupJSON)
+	if err != nil {
+		return nil, BuildResponse(r), err
+	}
+	defer closeBody(r)
+	var p Group
+	if jsonErr := json.NewDecoder(r.Body).Decode(&p); jsonErr != nil {
+		return nil, nil, NewAppError("CreateGroup", "api.unmarshal_error", nil, jsonErr.Error(), http.StatusInternalServerError)
+	}
+	return &p, BuildResponse(r), nil
+}
+
+func (c *Client4) DeleteGroup(groupID string) (*Group, *Response, error) {
+	r, err := c.DoAPIDelete(c.groupRoute(groupID))
+	if err != nil {
+		return nil, BuildResponse(r), err
+	}
+	defer closeBody(r)
+	var p Group
+	if jsonErr := json.NewDecoder(r.Body).Decode(&p); jsonErr != nil {
+		return nil, nil, NewAppError("DeleteGroup", "api.unmarshal_error", nil, jsonErr.Error(), http.StatusInternalServerError)
+	}
+	return &p, BuildResponse(r), nil
+}
+
 func (c *Client4) PatchGroup(groupID string, patch *GroupPatch) (*Group, *Response, error) {
 	payload, _ := json.Marshal(patch)
 	r, err := c.DoAPIPut(c.groupRoute(groupID)+"/patch", string(payload))
@@ -7086,6 +7117,40 @@ func (c *Client4) PatchGroup(groupID string, patch *GroupPatch) (*Group, *Respon
 	return &g, BuildResponse(r), nil
 }

+func (c *Client4) UpsertGroupMembers(groupID string, userIds *GroupModifyMembers) ([]*GroupMember, *Response, error) {
+	payload, jsonErr := json.Marshal(userIds)
+	if jsonErr != nil {
+		return nil, nil, NewAppError("UpsertGroupMembers", "api.marshal_error", nil, jsonErr.Error(), http.StatusInternalServerError)
+	}
+	r, err := c.DoAPIPostBytes(c.groupRoute(groupID)+"/members", payload)
+	if err != nil {
+		return nil, BuildResponse(r), err
+	}
+	defer closeBody(r)
+	var g []*GroupMember
+	if jsonErr := json.NewDecoder(r.Body).Decode(&g); jsonErr != nil {
+		return nil, nil, NewAppError("UpsertGroupMembers", "api.unmarshal_error", nil, jsonErr.Error(), http.StatusInternalServerError)
+	}
+	return g, BuildResponse(r), nil
+}
+
+func (c *Client4) DeleteGroupMembers(groupID string, userIds *GroupModifyMembers) ([]*GroupMember, *Response, error) {
+	payload, jsonErr := json.Marshal(userIds)
+	if jsonErr != nil {
+		return nil, nil, NewAppError("DeleteGroupMembers", "api.marshal_error", nil, jsonErr.Error(), http.StatusInternalServerError)
+	}
+	r, err := c.DoAPIDeleteBytes(c.groupRoute(groupID)+"/members", payload)
+	if err != nil {
+		return nil, BuildResponse(r), err
+	}
+	defer closeBody(r)
+	var g []*GroupMember
+	if jsonErr := json.NewDecoder(r.Body).Decode(&g); jsonErr != nil {
+		return nil, nil, NewAppError("DeleteGroupMembers", "api.unmarshal_error", nil, jsonErr.Error(), http.StatusInternalServerError)
+	}
+	return g, BuildResponse(r), nil
+}
+
 func (c *Client4) LinkGroupSyncable(groupID, syncableID string, syncableType GroupSyncableType, patch *GroupSyncablePatch) (*GroupSyncable, *Response, error) {
 	payload, _ := json.Marshal(patch)
 	url := fmt.Sprintf("%s/link", c.groupSyncableRoute(groupID, syncableID, syncableType))
--- a/model/config.go
+++ b/model/config.go
@@ -367,6 +367,7 @@ type ServiceSettings struct {
 	ThreadAutoFollow                                  *bool   `access:"experimental_features"`
 	CollapsedThreads                                  *string `access:"experimental_features"`
 	ManagedResourcePaths                              *string `access:"environment_web_server,write_restrictable,cloud_restrictable"`
+	EnableCustomGroups                                *bool   `access:"site_users_and_teams"`
 }

 func (s *ServiceSettings) SetDefaults(isUpdate bool) {
@@ -787,6 +788,10 @@ func (s *ServiceSettings) SetDefaults(isUpdate bool) {
 	if s.ManagedResourcePaths == nil {
 		s.ManagedResourcePaths = NewString("")
 	}
+
+	if s.EnableCustomGroups == nil {
+		s.EnableCustomGroups = NewBool(true)
+	}
 }

 type ClusterSettings struct {
--- a/model/feature_flags.go
+++ b/model/feature_flags.go
@@ -59,6 +59,11 @@ type FeatureFlags struct {
 	// Determine after which duration in hours to send a second invitation to someone that didn't join after the initial invite, possible values = ("48", "72")
 	ResendInviteEmailInterval string

+	// A/B test for whether radio buttons or toggle button is more effective in in-screen invite to team modal ("none", "toggle")
+	InviteToTeam string
+
+	CustomGroups bool
+
 	// Enable inline post editing
 	InlinePostEditing bool

@@ -95,6 +100,8 @@ func (f *FeatureFlags) SetDefaults() {
 	f.AddMembersToChannel = "top"
 	f.GuidedChannelCreation = false
 	f.ResendInviteEmailInterval = ""
+	f.InviteToTeam = "none"
+	f.CustomGroups = true
 	f.InlinePostEditing = false
 	f.BoardsDataRetention = false
 	f.NormalizeLdapDNs = false
@@ -102,7 +109,6 @@ func (f *FeatureFlags) SetDefaults() {
 	f.WorkspaceOptimizationDashboard = false
 	f.GraphQL = false
 }
-
 func (f *FeatureFlags) Plugins() map[string]string {
 	rFFVal := reflect.ValueOf(f).Elem()
 	rFFType := reflect.TypeOf(f).Elem()
--- a/model/group.go
+++ b/model/group.go
@@ -9,7 +9,8 @@ import (
 )

 const (
-	GroupSourceLdap GroupSource = "ldap"
+	GroupSourceLdap   GroupSource = "ldap"
+	GroupSourceCustom GroupSource = "custom"

 	GroupNameMaxLength        = 64
 	GroupSourceMaxLength      = 64
@@ -22,6 +23,7 @@ type GroupSource string

 var allGroupSources = []GroupSource{
 	GroupSourceLdap,
+	GroupSourceCustom,
 }

 var groupSourcesRequiringRemoteID = []GroupSource{
@@ -34,7 +36,7 @@ type Group struct {
 	DisplayName    string      `json:"display_name"`
 	Description    string      `json:"description"`
 	Source         GroupSource `json:"source"`
-	RemoteId       string      `json:"remote_id"`
+	RemoteId       *string     `json:"remote_id"`
 	CreateAt       int64       `json:"create_at"`
 	UpdateAt       int64       `json:"update_at"`
 	DeleteAt       int64       `json:"delete_at"`
@@ -43,6 +45,11 @@ type Group struct {
 	AllowReference bool        `json:"allow_reference"`
 }

+type GroupWithUserIds struct {
+	Group
+	UserIds []string `json:"user_ids"`
+}
+
 type GroupWithSchemeAdmin struct {
 	Group
 	SchemeAdmin *bool `db:"SyncableSchemeAdmin" json:"scheme_admin,omitempty"`
@@ -63,6 +70,8 @@ type GroupPatch struct {
 	DisplayName    *string `json:"display_name"`
 	Description    *string `json:"description"`
 	AllowReference *bool   `json:"allow_reference"`
+	// For security reasons (including preventing unintended LDAP group synchronization) do no allow a Group's RemoteId or Source field to be
+	// included in patches.
 }

 type LdapGroupSearchOpts struct {
@@ -79,12 +88,21 @@ type GroupSearchOpts struct {
 	FilterAllowReference   bool
 	PageOpts               *PageOpts
 	Since                  int64
+	Source                 GroupSource

 	// FilterParentTeamPermitted filters the groups to the intersect of the
 	// set associated to the parent team and those returned by the query.
 	// If the parent team is not group-constrained or if NotAssociatedToChannel
 	// is not set then this option is ignored.
 	FilterParentTeamPermitted bool
+
+	// FilterHasMember filters the groups to the intersect of the
+	// set returned by the query and those that have the given user as a member.
+	FilterHasMember string
+}
+
+type GetGroupOpts struct {
+	IncludeMemberCount bool
 }

 type PageOpts struct {
@@ -97,6 +115,10 @@ type GroupStats struct {
 	TotalMemberCount int64  `json:"total_member_count"`
 }

+type GroupModifyMembers struct {
+	UserIds []string `json:"user_ids"`
+}
+
 func (group *Group) Patch(patch *GroupPatch) {
 	if patch.Name != nil {
 		group.Name = patch.Name
@@ -137,7 +159,7 @@ func (group *Group) IsValidForCreate() *AppError {
 		return NewAppError("Group.IsValidForCreate", "model.group.source.app_error", nil, "", http.StatusBadRequest)
 	}

-	if len(group.RemoteId) > GroupRemoteIDMaxLength || (group.RemoteId == "" && group.requiresRemoteId()) {
+	if (group.GetRemoteId() == "" && group.requiresRemoteId()) || len(group.GetRemoteId()) > GroupRemoteIDMaxLength {
 		return NewAppError("Group.IsValidForCreate", "model.group.remote_id.app_error", nil, "", http.StatusBadRequest)
 	}

@@ -188,3 +210,15 @@ func (group *Group) IsValidName() *AppError {
 	}
 	return nil
 }
+
+func (group *Group) GetRemoteId() string {
+	if group.RemoteId == nil {
+		return ""
+	}
+	return *group.RemoteId
+}
+
+type GroupsWithCount struct {
+	Groups     []*Group `json:"groups"`
+	TotalCount int64    `json:"total_count"`
+}
--- a/model/license.go
+++ b/model/license.go
@@ -323,6 +323,12 @@ func NewTestLicense(features ...string) *License {
 	return ret
 }

+func NewTestLicenseSKU(skuShortName string, features ...string) *License {
+	lic := NewTestLicense(features...)
+	lic.SkuShortName = skuShortName
+	return lic
+}
+
 func (lr *LicenseRecord) IsValid() *AppError {
 	if !IsValidId(lr.Id) {
 		return NewAppError("LicenseRecord.IsValid", "model.license_record.is_valid.id.app_error", nil, "", http.StatusBadRequest)
--- a/model/migration.go
+++ b/model/migration.go
@@ -36,4 +36,5 @@ const (
 	MigrationKeyAddAboutSubsectionPermissions          = "about_subsection_permissions"
 	MigrationKeyAddIntegrationsSubsectionPermissions   = "integrations_subsection_permissions"
 	MigrationKeyAddPlaybooksPermissions                = "playbooks_permissions"
+	MigrationKeyAddCustomUserGroupsPermissions         = "custom_groups_permissions"
 )
--- a/model/permission.go
+++ b/model/permission.go
@@ -7,6 +7,7 @@ const (
 	PermissionScopeSystem   = "system_scope"
 	PermissionScopeTeam     = "team_scope"
 	PermissionScopeChannel  = "channel_scope"
+	PermissionScopeGroup    = "group_scope"
 	PermissionScopePlaybook = "playbook_scope"
 	PermissionScopeRun      = "run_scope"
 )
@@ -355,6 +356,11 @@ var PermissionRunView *Permission
 // admin functions but not others
 var PermissionManageSystem *Permission

+var PermissionCreateCustomGroup *Permission
+var PermissionManageCustomGroupMembers *Permission
+var PermissionEditCustomGroup *Permission
+var PermissionDeleteCustomGroup *Permission
+
 var AllPermissions []*Permission
 var DeprecatedPermissions []*Permission

@@ -1914,6 +1920,34 @@ func initializePermissions() {
 		PermissionScopeSystem,
 	}

+	PermissionCreateCustomGroup = &Permission{
+		"create_custom_group",
+		"authentication.permissions.create_custom_group.name",
+		"authentication.permissions.create_custom_group.description",
+		PermissionScopeSystem,
+	}
+
+	PermissionManageCustomGroupMembers = &Permission{
+		"manage_custom_group_members",
+		"authentication.permissions.manage_custom_group_members.name",
+		"authentication.permissions.manage_custom_group_members.description",
+		PermissionScopeGroup,
+	}
+
+	PermissionEditCustomGroup = &Permission{
+		"edit_custom_group",
+		"authentication.permissions.edit_custom_group.name",
+		"authentication.permissions.edit_custom_group.description",
+		PermissionScopeGroup,
+	}
+
+	PermissionDeleteCustomGroup = &Permission{
+		"delete_custom_group",
+		"authentication.permissions.delete_custom_group.name",
+		"authentication.permissions.delete_custom_group.description",
+		PermissionScopeGroup,
+	}
+
 	// Playbooks
 	PermissionPublicPlaybookCreate = &Permission{
 		"playbook_public_create",
@@ -2200,6 +2234,7 @@ func initializePermissions() {
 		PermissionGetLogs,
 		PermissionReadLicenseInformation,
 		PermissionManageLicenseInformation,
+		PermissionCreateCustomGroup,
 	}

 	TeamScopedPermissions := []*Permission{
@@ -2259,6 +2294,12 @@ func initializePermissions() {
 		PermissionUseGroupMentions,
 	}

+	GroupScopedPermissions := []*Permission{
+		PermissionManageCustomGroupMembers,
+		PermissionEditCustomGroup,
+		PermissionDeleteCustomGroup,
+	}
+
 	DeprecatedPermissions = []*Permission{
 		PermissionPermanentDeleteUser,
 		PermissionManageWebhooks,
@@ -2307,6 +2348,7 @@ func initializePermissions() {
 	AllPermissions = append(AllPermissions, ChannelScopedPermissions...)
 	AllPermissions = append(AllPermissions, SysconsoleReadPermissions...)
 	AllPermissions = append(AllPermissions, SysconsoleWritePermissions...)
+	AllPermissions = append(AllPermissions, GroupScopedPermissions...)
 	AllPermissions = append(AllPermissions, PlaybookScopedPermissions...)
 	AllPermissions = append(AllPermissions, RunScopedPermissions...)

--- a/model/role.go
+++ b/model/role.go
@@ -4,6 +4,7 @@
 package model

 import (
+	"fmt"
 	"strings"
 )

@@ -42,6 +43,8 @@ func init() {
 		ChannelUserRoleId,
 		ChannelAdminRoleId,

+		CustomGroupUserRoleId,
+
 		PlaybookAdminRoleId,
 		PlaybookMemberRoleId,
 		RunAdminRoleId,
@@ -367,6 +370,8 @@ const (
 	ChannelUserRoleId  = "channel_user"
 	ChannelAdminRoleId = "channel_admin"

+	CustomGroupUserRoleId = "custom_group_user"
+
 	PlaybookAdminRoleId  = "playbook_admin"
 	PlaybookMemberRoleId = "playbook_member"
 	RunAdminRoleId       = "run_admin"
@@ -379,6 +384,7 @@ const (
 	RoleScopeSystem  RoleScope = "System"
 	RoleScopeTeam    RoleScope = "Team"
 	RoleScopeChannel RoleScope = "Channel"
+	RoleScopeGroup   RoleScope = "Group"

 	RoleTypeGuest RoleType = "Guest"
 	RoleTypeUser  RoleType = "User"
@@ -683,6 +689,13 @@ func IsValidRoleName(roleName string) bool {
 func MakeDefaultRoles() map[string]*Role {
 	roles := make(map[string]*Role)

+	roles[CustomGroupUserRoleId] = &Role{
+		Name:        CustomGroupUserRoleId,
+		DisplayName: fmt.Sprintf("authentication.roles.%s.name", CustomGroupUserRoleId),
+		Description: fmt.Sprintf("authentication.roles.%s.description", CustomGroupUserRoleId),
+		Permissions: []string{},
+	}
+
 	roles[ChannelGuestRoleId] = &Role{
 		Name:        "channel_guest",
 		DisplayName: "authentication.roles.channel_guest.name",
@@ -895,6 +908,10 @@ func MakeDefaultRoles() map[string]*Role {
 			PermissionCreateGroupChannel.Id,
 			PermissionViewMembers.Id,
 			PermissionCreateTeam.Id,
+			PermissionCreateCustomGroup.Id,
+			PermissionEditCustomGroup.Id,
+			PermissionDeleteCustomGroup.Id,
+			PermissionManageCustomGroupMembers.Id,
 		},
 		SchemeManaged: true,
 		BuiltIn:       true,
--- a/model/user_get.go
+++ b/model/user_get.go
@@ -14,6 +14,8 @@ type UserGetOptions struct {
 	NotInChannelId string
 	// Filters the users in the group
 	InGroupId string
+	// Filters the users not in the group
+	NotInGroupId string
 	// Filters the users group constrained
 	GroupConstrained bool
 	// Filters the users without a team
--- a/model/user_search.go
+++ b/model/user_search.go
@@ -22,6 +22,7 @@ type UserSearch struct {
 	Roles            []string `json:"roles"`
 	ChannelRoles     []string `json:"channel_roles"`
 	TeamRoles        []string `json:"team_roles"`
+	NotInGroupId     string   `json:"not_in_group_id"`
 }

 // UserSearchOptions captures internal parameters derived from the user's permissions and a
--- a/model/websocket_message.go
+++ b/model/websocket_message.go
@@ -62,6 +62,8 @@ const (
 	WebsocketEventReceivedGroupNotAssociatedToTeam    = "received_group_not_associated_to_team"
 	WebsocketEventReceivedGroupAssociatedToChannel    = "received_group_associated_to_channel"
 	WebsocketEventReceivedGroupNotAssociatedToChannel = "received_group_not_associated_to_channel"
+	WebsocketEventGroupMemberDelete                   = "group_member_deleted"
+	WebsocketEventGroupMemberAdd                      = "group_member_add"
 	WebsocketEventSidebarCategoryCreated              = "sidebar_category_created"
 	WebsocketEventSidebarCategoryUpdated              = "sidebar_category_updated"
 	WebsocketEventSidebarCategoryDeleted              = "sidebar_category_deleted"