* MM-47250: Add new system console section
We add a new section in system console
for products.
```release-note
Added a new section in system console
for products. For now, it only contains
boards specific settings.
```
* fix unit tests
```release-note
NONE
```
* change comment
```release-note
NONE
```
This PR makes 2 main changes to the tests:
1. User logins were a huge part of the total CPU time because of the constant hashing of the password. There were 3 logins every time for every `Setup()` call. We remove the `LoginSystemManager` from the Setup call as it wasn't used in a lot of tests, and call that as needed.
2. We add the new app migrations that got added after the last preload optimization was done.
This shaves off around 3 mins from the test runtime. They are down from ~18m to ~15m.
```release-note
NONE
```
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* MM-43753: Module workspaces
Move to module workspaces
We make the following changes:
- Remove the vendor directory.
- Dynamically create the go.work file if it doesn't exist.
- Set the MM_SERVER_PATH for enterprise tests.
https://mattermost.atlassian.net/browse/MM-43753
```release-note
NONE
```
* Fix missing reference to MM_SERVER_PATH
```release-note
NONE
```
* Update test to add another nested dir
```release-note
NONE
```
* Have separate script to create go.work file
```release-note
NONE
```
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* revamp db version and add applied migrations endpoint
* replace old schema version with new
* add db version subcommand
* add to local api
* reflect review comments
* log errors
* remove setting the version from model.CurrentVersion
* fix a test
* use different field for schema version
* add build hash and current version to the support packet
* add tests
* update test to use new assets
* Revert "[MM-41576] Revamp database schema version (#19586)"
This reverts commit 645fee3fe3.
* Revert "MM-42049 - license endpoint not working (#19686)"
This reverts commit 4fe89e5847.
* revamp db version and add applied migrations endpoint
* replace old schema version with new
* add db version subcommand
* add to local api
* reflect review comments
* log errors
* remove setting the version from model.CurrentVersion
* fix a test
* WIP
* adding initial creategroup endpoint
* fetching by group source
* fixing startup error
* updating create endpoint to take an array of user_ids, this will allow us to create the group with one request
* adding delete group endpoint and appropriate test
* adding source param for getGroups
* adding add members and delete members endpoints
* locking down crud endpoints to only be allowed for custom groups
* user search stuff
* allowing remoteid be null by changing field to pointer
* code cleanup and store level tests
* adding new tests and removing unused endpoint
* resolving conflicts
* Adds authz check for group.
* Adds authz checks to groups APIs.
* Updated create group authz tests.
* Updates delete group tests.
* Tests create group.
* Adds some tests and validations.
* adding new parameter so I can get users not in a group
* Fixed all lint warnings.
* Fix type.
* fixing search users not in group
* Fixes some lint errors.
* Moves entry in JSON array.
* Fixed SQL query.
* Fixes permission migration test.
* Fixes migration test.
* Fixes some group store tests.
* Fix test.
* Fix test.
* Revert lint change.
* Migrated CreateWithUserIds to sqlx.
* Adds tests for GetMember; migrates implementation to sqlx.
* Tests GetNonMemberUsersPage and hanles wrong group id.
* Fixes test.
* Switches GetMaster to GetMasterX.
* Switches GetReplica to GetReplicaX.
* Fixes logic.
* Fixes shadow declaration.
* Adds include_member_count to get group API endpoint.
* Adds filter_has_member param to getGroups.
* Fixes.
* Removes array of group sources.
* fixing error
* Testing reverting CreateWithUserIds back to gorp.
* Added websocket event for CreateGroupWithUserIds.
* Changed a few response status codes. Switched to correct permission.
* Added member count to ws payload for group when updating or creating.
* Adds feature flag checks for custom groups.
* Added middleware function to require license. Added config to disable custom groups.
* Change for function signature change of executePossiblyEmptyQuery.
* Lint fixes.
* Adds telemetry none comment.
* Adds translations.
* Migrated to sqlx.
* Temp. removal of translation.
* Fixed typo.
* Added an intermediary model to query with a field that is now ignored by sqlx on read queries.
* Re-used existing store struct.
* Inludes member count.
* Fix for merge error.'
* Require license for group endpoints.
* Updates translations.
* Fix shadow declaration.
* Renames permissions. Switches to new method to retrieve remoteid.
* Added WS events for upsert and delete member(s).
* Added new store error type ErrUniqueConstraint.
* Added EnableCustonGroups to the client config.
* Sanitized some user records.
* Added parameter to include_total_count for listing groups.
* Added translations.
* adding deleteAt field to getByUsers query
* Revert sanitize.
* Added uniqueness constraint error to UpdateGroup.
* Removed the FutureFeatures flag so that the feature is not enabled on old Enterprise licenses.
* Renamed function.
* Updates authz check for user search related to groups.
* Removed debug statement.
* Removed unused app method.
* Added telemetry for enable_custom_groups.
* Returns early from nil license.
* Updates test.
* Returned early to avoid nesting in (*SqlGroupStore).checkUserExist. Switched to reading from replica in (*SqlGroupStore).GetMember. Handled JSON marshal error in (*Client4).UpsertGroupMembers
* Switched to SanitizeProfile.
* Switched to model.NewInt.
* Switched from status NotImplemented to Forbidden for missing license.
* Removed deactivated users from 'exists' set.
* Revert gotool update.
* Ignored lint error that I think is invalid.
* Added the approprate access tag for disabling custom groups.
* Revert change to response status.
* Fixed refactor mistake.
* Limited the group member WS events to individual users.
* Removed WS event of deleted groups.
* Updated license check for searchUsers endpoint.
* Switched from license feature to license sku.
* Update app/group.go
Co-authored-by: Claudio Costa <cstcld91@gmail.com>
* Update app/group.go
Co-authored-by: Claudio Costa <cstcld91@gmail.com>
* Remove linter ignore comment.
* Added function to create sku-specific license.
* Fixed typo. Removed comment.
* Fixed for wrong type.
* Added missing param to client. Removed unnecessary props setting. Added test for retrieving groups by source.
* Updated some tests now that we're validating group membership not created for deactivated user.
* Fix for groups endpoint returning all group types by default.
* Changes constant names. Adds migration for all users to manage custom group members.
* Removes requirement for manage_system permission to filter user search by group.
* Added migration mock.
* Removes default permissions from custom_group_user role.
* Fixes migration.
* Fixes emoji migration test.
* fixing issue with member counts
* fixing search issue for deleted members
Co-authored-by: Benjamin Cooke <benjamincooke@Benjamins-MacBook-Pro.local>
Co-authored-by: Benjamin Cooke <benjamincooke@Benjamins-MBP.ht.home>
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
Co-authored-by: Benjamin Cooke <benjamincooke@Benjamins-MacBook-Pro.fritz.box>
Co-authored-by: Claudio Costa <cstcld91@gmail.com>
* Add playbooks related permissions.
* Add RolesGrantPermission to pluginapi
* Fixing scopes.
* New defaults.
* Fix defaults
* Fix tests.
* Fix migration.
* More test and migration fixes.
* Need to add everything to system admin too.
* Move to 63
* Feedback fixes.
* Fix system manager editing playbook permissions.
https://mattermost.atlassian.net/browse/MM-22051
```release-note
Removed the following methods/functions:
(ad *AccessData) ToJson()
(ar *AccessResponse) ToJson()
(ar *AnalyticsRow) ToJson()
(ar AnalyticsRows) ToJson()
(o *Audit) ToJson()
(o Audits) ToJson()
(ad *AuthData) ToJson()
(ar *AuthorizeRequest) ToJson()
(o *ChannelPatch) ToJson()
(o *ChannelsWithCount) ToJson()
(o *ChannelCounts) ToJson()
(o *ChannelData) ToJson()
(o *ChannelMembers) ToJson()
(o *ChannelUnread) ToJson()
(o *ChannelUnreadAt) ToJson()
(o *ChannelStats) ToJson()
(o *ChannelView) ToJson()
(o *ChannelViewResponse) ToJson()
(o *ClusterDiscovery) ToJson()
(ci *ClusterInfo) ToJson()
(cs *ClusterStats) ToJson()
(o *Command) ToJson()
CommandListToJson(l []*Command) string
(o *CommandArgs) ToJson()
(cmr *CommandMoveRequest) ToJson()
(o *CommandResponse) ToJson()
(c *Compliance) ToJson()
(c Compliances) ToJson()
(o *Config) ToJson()
EmojiListToJson(emojiList []*Emoji)
```
* Removing FilesSearch feature flag
* Fixing tests
* Adding an improvement on plain text extraction
* Adding tests for plain text extraction
* Removed unneeded conversion
* Adding missed license
* Remove the feature flag from the migration
* Fixing some tests
* Updating i18n/en.json file
* Add basic support for plugin intra-cluster communication
* Some renaming for added clarity
* Allow sending cluster event to specific nodes
* Improve naming and documentation
* Improve logging
Remote Cluster Service
- provides ability for multiple Mattermost cluster instances to create a trusted connection with each other and exchange messages
- trusted connections are managed via slash commands (for now)
- facilitates features requiring inter-cluster communication, such as Shared Channels
Shared Channels Service
- provides ability to shared channels between one or more Mattermost cluster instances (using trusted connection)
- sharing/unsharing of channels is managed via slash commands (for now)
* Extracting html templates into a library
* Moving tests to the right place
* Fixing tests
* Addressing PR review comments
* Addressing PR review comments
* Replacing attomic with RWMutex
* Returning errors as channel for Templates watcher
* Address PR review comments
* Other small fixes
* Simplifying NewWithWatcher
* Addressing PR review comments
* Making error handling on rendering templates more robust
* Fixing tests
* Changing how we return errors
* Fixing shadow variables
* Addressing PR review comments
* Logging errors from the outside of sendNotificationEmail
* Fixing lock in shutdown
* Fixing the resource copy for commands tests temporary directories
* Removing unused import
* A couple of tiny fixes
* MM-31094: Adds tooling to develop and test using a MySQL instance with replication lag. Adds some lazy lookups to fallback to master if results are not found.
* MM-31094: Removes mysql-read-replica from default docker services.
* MM-31094: Switches (store..SessionStore).Get and (store.TeamStore).GetMember to using context.Context.
* MM-31094: Updates (store.UsersStore).Get to use context.
* MM-31094: Updates (store.PostStore).Get to use context.
* MM-31094: Removes feature flag and config setting.
* MM-31094: Rolls back some master reads.
* MM-31094: Rolls a non-cache read.
* MM-31094: Removes feature flag from the store.
* MM-31094: Removes unused constant and struct field.
* MM-31094: Removes some old feature flag references.
* MM-31094: Fixes some tests.
* MM-31094: App layers fix.
* MM-31094: Fixes mocks.
* MM-31094: Don't reparse flag.
* MM-31094: No reparse.
* MM-31094: Removed unused FeatureFlags field.
* MM-31094: Removes unnecessary feature flags variable declarations.
* MM-31094: Fixes copy-paste error.
* MM-31094: Fixes logical error.
* MM-30194: Removes test method from store.
* Revert "MM-30194: Removes test method from store."
This reverts commit d5a6e8529bd5f4d993824c828e239d009b05e567.
* MM-31094: Conforming to make's strange syntax.
* MM-31094: Configures helper for read replica with option.
* MM-31094: Adds some missing ctx's.
* MM-31094: WIP
* MM-31094: Updates test names.
* MM-31094: WIP
* MM-31094: Removes unnecessary master reads.
* MM-31094: ID case changes out of scope.
* MM-31094: Removes unused context.
* MM-31094: Switches to a helper. Removes some var naming changes. Fixes a merge error.
* MM-31094: Removes SQLITE db driver ref.
* MM-31094: Layer generate fix.
* MM-31094: Removes unnecessary changes.
* MM-31094: Moves test method.
* MM-31094: Re-add previous fix.
* MM-31094: Removes make command for dev.
* MM-31094: Fix for login.
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* Adding search files api
* Fixing golangci-lint
* Adding bulk-indexing and improving a bit the name indexing for bleve and elasticsearch
* Add content extraction config migration
* Fixing a problem with document extraction
* Unapplying certain changes moved to other PR
* Fixing tests
* Making extract content app migration a private method
* Addressing PR review comments
* Addressing PR review comments
* Adding feature flag
* Removing debug string
* Fixing imports
* Fixing linting errors
* Do not migrate the config if the feature flag is not enabled
* Fix tests
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* Reducing the filestore dependencies from the rest of the source code
* Making more generic config conversion to FileBackendSettings
* Fixing usage of the NewFileBackend function
* Fixing more usages of the NewFileBackend function
* Fix some linter errors
* Fix more linter errors
* Fixing some unit tests
* Fixing linter problem
* Addressing PR review comments
* Simplifing the CopyFile for tests
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* init commit
* clean up the code
* make mocks
* fix translations
* mocks and lint fixes
* add tests
* little fixes
* Update i18n/en.json
Co-authored-by: Scott Bishel <scott.bishel@mattermost.com>
* Update i18n/en.json
Co-authored-by: Scott Bishel <scott.bishel@mattermost.com>
* Update i18n/en.json
Co-authored-by: Scott Bishel <scott.bishel@mattermost.com>
* Update i18n/en.json
Co-authored-by: Scott Bishel <scott.bishel@mattermost.com>
* Update i18n/en.json
Co-authored-by: Scott Bishel <scott.bishel@mattermost.com>
* Address Comments
* fix i18n
* update api endpoint
* add enable file and file level for conditional show of banner
* Address Comments
* Make it more clear about returns
* Create zip file utility function
* update en.json
* address comments
* write tests
* check for data in test
* remove warning string
* Correct expected and actual
* set database through environment variables
* reset environment variable at end of test
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
Co-authored-by: Scott Bishel <scott.bishel@mattermost.com>
* format using `goimports -local github.com/mattermost/mattermost-server/v5 -w`
* added goimports lint check to .golangci.yml
* format using `goimports -local github.com/mattermost/mattermost-server/v5 -w` for a corner case
* make app-layers, *-mocks and store-layers for ci check
Co-authored-by: Mahmudul Haque <mahmudulhaque@protonmail.com>
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* MM-31082: Use a base path for test assets
Tests can be run from other tools using a different base path.
We need to respect the MM_SERVER_PATH variable for that.
https://mattermost.atlassian.net/browse/MM-31082
```release-note
NONE
```
* do not append mattermost-server if env var is set
* fix typo
* Removing supplier concept from the sql store
* Removing other metions to supplier
* Fixing gofmt
* Fixing gofmt
* Renaming NewSqlStore to New
* Fixing tests
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* MM-28067: Optimize app server startup in tests
For every test, we would wipe out the database and then start
the server again. This would run through all the migrations
and create new rows in Systems and Roles table every time.
As a result, this was consuming a lot of setup time for every test.
We optimize this by preloading the DB with dummy data in the roles
and systems table so that the server code just skips over those migrations.
This is completely forward compatible and adding new migrations does not
need to generate the sql files again. Only in case of schema changes to the
roles or systems table, this would need to be done.
It is unlikely the Systems table schema will get changed. The Roles table
might change in future but it's a comparatively rare event. Given the reduction
in CI runtime we are seeing, it's a worthy optimization.
We also apply some more optimizations:
- Coalesce multiple UpdateConfig calls into a single one. Each UpdateConfig call
has to do a json marshal which would take precious CPU cycles. It's much more
efficient to do everything in a single call.
- Remove unnecessary debug.FreeOSMemory in reload config. This was an artifact
from old days and is no longer required.
Numbers:
Results show a full **2 minutes** shaved off the test runtime. Earlier, tests
would take around 16 minutes. Now they take 14 minutes.
```release-note
NONE
```
* fix app package
* Refactor apply multi role filters and add role filters to get all profiles
* Add some tests
* Fix tests
* Fix lint
* Trigger CI
* Rename param to make more sense
* Tie get filtered user stats to usermanagement read users
* Dont filter out other system roles when searching for team members or team admins only filter out system admins
* add new permissions
* add migration
* fix test
* remove system roles as default permissions
* implement changes discussed with dennis
* add read only and fix i18n
* use model consts instead of strings
* turn the permissions into pseudo constants
* Update read only default permissions
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
Co-authored-by: Hossein Ahmadian-Yazdi <hyazdi1997@gmail.com>
* MM-27909: Add manage_shared_channels permission
We add a new permission to manage shared channels.
It's a channel scoped permission and only the system admin
has that by default.
https://mattermost.atlassian.net/browse/MM-27909
* change to system scoped
* Trigger CI
* Trigger CI
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
