* Improves groups membership telemetry.
* Removes unnecessary where clause (it's added by default).
* Cleans up data.
* Testing the relative group counts.
* Adds missing test mock.
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
```release-note
We create a new config option MaxImageDecoderConcurrency which
indicates how many images can be decoded concurrently at once.
The default is -1 which means number of CPUs present.
This affects the total memory consumption of the server. The
maximum memory of a single image is dictated by MaxImageResolution * 24 bytes.
Therefore, a good rule of thumb to follow is that MaxImageResolution
* MaxImageDecoderConcurrency * 24 should be less then the allocated memory for
image decoding.
```
https://mattermost.atlassian.net/browse/MM-43733
* MM-41211: Replaces SessionLength[Web|Mobile|SSO]InDays with SessionLength[Web|Mobile|SSO]InHours.
* MM-41211: Clear the value of the old config settings.
The older method used to reply completely on timestamps
to take batches of items in a timestamp range and then
just incrementing the timestamp. This led to handling
edge-cases such as more items than the batch count, all
having the same timestamp.
Additionally, relying on timestamp as the page cursor
meant that indexing was not very efficient if you had
several items spread out across large spans of time.
To get away from all of that we use a proper cursor-based
approach consisting of createAt+Id. With this, we move
completely to a constant page size where we can fetch
a given number of objects irrespective of when they
were created. This makes indexing much more faster and
efficient.
https://mattermost.atlassian.net/browse/MM-41260
```release-note
Elasticsearch and Bleve indexing have been revamped to be much
more efficient and faster. The config parameter BulkIndexingTimeWindowSeconds
for both elasticsearch and bleve have been removed.
A new config parameter called BatchSize has been introduced instead.
This parameter controls the number of objects that
can be indexed in a single batch. This makes things
more efficient and maintains a constant workload.
```
* WIP
* adding initial creategroup endpoint
* fetching by group source
* fixing startup error
* updating create endpoint to take an array of user_ids, this will allow us to create the group with one request
* adding delete group endpoint and appropriate test
* adding source param for getGroups
* adding add members and delete members endpoints
* locking down crud endpoints to only be allowed for custom groups
* user search stuff
* allowing remoteid be null by changing field to pointer
* code cleanup and store level tests
* adding new tests and removing unused endpoint
* resolving conflicts
* Adds authz check for group.
* Adds authz checks to groups APIs.
* Updated create group authz tests.
* Updates delete group tests.
* Tests create group.
* Adds some tests and validations.
* adding new parameter so I can get users not in a group
* Fixed all lint warnings.
* Fix type.
* fixing search users not in group
* Fixes some lint errors.
* Moves entry in JSON array.
* Fixed SQL query.
* Fixes permission migration test.
* Fixes migration test.
* Fixes some group store tests.
* Fix test.
* Fix test.
* Revert lint change.
* Migrated CreateWithUserIds to sqlx.
* Adds tests for GetMember; migrates implementation to sqlx.
* Tests GetNonMemberUsersPage and hanles wrong group id.
* Fixes test.
* Switches GetMaster to GetMasterX.
* Switches GetReplica to GetReplicaX.
* Fixes logic.
* Fixes shadow declaration.
* Adds include_member_count to get group API endpoint.
* Adds filter_has_member param to getGroups.
* Fixes.
* Removes array of group sources.
* fixing error
* Testing reverting CreateWithUserIds back to gorp.
* Added websocket event for CreateGroupWithUserIds.
* Changed a few response status codes. Switched to correct permission.
* Added member count to ws payload for group when updating or creating.
* Adds feature flag checks for custom groups.
* Added middleware function to require license. Added config to disable custom groups.
* Change for function signature change of executePossiblyEmptyQuery.
* Lint fixes.
* Adds telemetry none comment.
* Adds translations.
* Migrated to sqlx.
* Temp. removal of translation.
* Fixed typo.
* Added an intermediary model to query with a field that is now ignored by sqlx on read queries.
* Re-used existing store struct.
* Inludes member count.
* Fix for merge error.'
* Require license for group endpoints.
* Updates translations.
* Fix shadow declaration.
* Renames permissions. Switches to new method to retrieve remoteid.
* Added WS events for upsert and delete member(s).
* Added new store error type ErrUniqueConstraint.
* Added EnableCustonGroups to the client config.
* Sanitized some user records.
* Added parameter to include_total_count for listing groups.
* Added translations.
* adding deleteAt field to getByUsers query
* Revert sanitize.
* Added uniqueness constraint error to UpdateGroup.
* Removed the FutureFeatures flag so that the feature is not enabled on old Enterprise licenses.
* Renamed function.
* Updates authz check for user search related to groups.
* Removed debug statement.
* Removed unused app method.
* Added telemetry for enable_custom_groups.
* Returns early from nil license.
* Updates test.
* Returned early to avoid nesting in (*SqlGroupStore).checkUserExist. Switched to reading from replica in (*SqlGroupStore).GetMember. Handled JSON marshal error in (*Client4).UpsertGroupMembers
* Switched to SanitizeProfile.
* Switched to model.NewInt.
* Switched from status NotImplemented to Forbidden for missing license.
* Removed deactivated users from 'exists' set.
* Revert gotool update.
* Ignored lint error that I think is invalid.
* Added the approprate access tag for disabling custom groups.
* Revert change to response status.
* Fixed refactor mistake.
* Limited the group member WS events to individual users.
* Removed WS event of deleted groups.
* Updated license check for searchUsers endpoint.
* Switched from license feature to license sku.
* Update app/group.go
Co-authored-by: Claudio Costa <cstcld91@gmail.com>
* Update app/group.go
Co-authored-by: Claudio Costa <cstcld91@gmail.com>
* Remove linter ignore comment.
* Added function to create sku-specific license.
* Fixed typo. Removed comment.
* Fixed for wrong type.
* Added missing param to client. Removed unnecessary props setting. Added test for retrieving groups by source.
* Updated some tests now that we're validating group membership not created for deactivated user.
* Fix for groups endpoint returning all group types by default.
* Changes constant names. Adds migration for all users to manage custom group members.
* Removes requirement for manage_system permission to filter user search by group.
* Added migration mock.
* Removes default permissions from custom_group_user role.
* Fixes migration.
* Fixes emoji migration test.
* fixing issue with member counts
* fixing search issue for deleted members
Co-authored-by: Benjamin Cooke <benjamincooke@Benjamins-MacBook-Pro.local>
Co-authored-by: Benjamin Cooke <benjamincooke@Benjamins-MBP.ht.home>
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
Co-authored-by: Benjamin Cooke <benjamincooke@Benjamins-MacBook-Pro.fritz.box>
Co-authored-by: Claudio Costa <cstcld91@gmail.com>
* add Boards to DataRetention, add hook for data retention
* remove replaces
* update hook to remove parameter
* add boards data retention to telemetry
* fix unit test
* update test, update hooks
* update RunDataRetention server version
* put behind a feature flag
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* GH-18288: Allow configuring unsafe-eval and unsafe-inline
* Add developer flags to telemetry
* wip
* wip
* Refactor based on review
* Refactor based on review
* fix expected vs. actual in assert.Equal
* Add unit tests, rework to check only supported flags.
* Update model/config.go
Co-authored-by: Jesse Hallam <jesse@thehallams.ca>
* Fix failing tests
* Refactor based on review
* Refactor based on review
* Refactor based on review
Co-authored-by: Jesse Hallam <jesse.hallam@gmail.com>
Co-authored-by: Jesse Hallam <jesse@thehallams.ca>
* Add JobSettings.CleanupJobsThresholdDays to telemetry
```release-note
NONE
```
* Moved telemetry to more specific fields
```release-note
NONE
```
* Add missing space
```release-note
NONE
```
* removing deprecated stuff
* removing deprecations and marking others for deprecations
* removing some of the configs
* partially remove ExperimentalChannelOrganization
* fixing merge issues
* moving default permissions to defaultroles function
* removing experimentaltownsquare
* removing old tests and remove legacysidebar config
* fixing test
* removing log
* removing unused variables
* MM-36444: Fix for renamed functions.
* MM-36444: go mod tidy and go mod vendor.
* MM-36444: Vendored go modules.
* MM-36444: Removes UseExperimentalGossip config field from model.
* Fix vendors
* fixing test
Co-authored-by: Benjamin Cooke <benjamincooke@Benjamins-MacBook-Pro.local>
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
Co-authored-by: Martin Kraft <martin@upspin.org>
Co-authored-by: Claudio Costa <cstcld91@gmail.com>
Fix an issue where context fields logged by server on behalf of plugins didn't contain correct name/value pairs:
- bump Logr version to add key/value methods to sugar logger
- expose factory args when configuring logging with custom target types (needed for FocalBoard to create log target adapter that converts typed fields into slices of interface{} as per plugin logging API)
* MM-22845: Added support for permalink previews.
* MM-22845: Adds license to new file.
* MM-22845: Adds endpoint to retrieve multiple posts by id.
* MM-22845: Fix for deleted post.
* MM-22845: Adds config setting for permalink previews.
* MM-22845: Adds API test for new endpoint.
* MM-22845: Fix typo.
* MM-22845: Tests that post create or updated via App get the previewed_post prop.
* MM-22845: Tests for matching permalinks.
* MM-22845: Adds PreparePostForClient test for permalink previews.
* MM-22845: Embeds entire post in permalink metadata.
* MM-22845: Filter WS message payload of created and edited post based on permissions.
* MM-22845: Runs app layer generator.
* MM-22845: Lint check fix.
* MM-22845: Adds feature flag.
* MM-22845: Clones WS message.
* MM-22845: Removes knowledge of permalink from LinkMetadata table. Removes knowledge of user id from post embedding methods in favour of a 'sanitize' method/step.
* MM-22845: Handle nil post metadata.
* MM-22845: Switch to cloning post.
* MM-22845: Removes unused code.
* MM-22845: Refactor.
* MM-22845: Reverts whitespace change.
* MM-22845: Removes unnecessary code.
* MM-22845: Removes unnecessary function.
* MM-22845: Warn but don't error if permalinked referenced post or channel is not found.
* MM-22845: Fix for clone method.
* MM-22845: Fix for clone method.
* MM-22845: Updates translations.
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* MM-21357: Use typed constant for channel types
https://mattermost.atlassian.net/browse/MM-21357
```release-note
- Introduced a new type ChannelType for all channel types.
- Updated the Client4.UpdateChannelPrivacy method to ChannelType.
```
* Address review comments
```release-note
NONE
```
* telemetry fix
```release-note
NONE
```
* pre-checkout commit
* add API endpoints for retention policies
* allow deleting multiple teams/channels from a policy in a single request
* pre-checkout commit
* add auditing in API functions
* add permission checks
* update the store layers
* update storetest
* add check constraint on PostDuration column
* pre-checkout commit
* add query to delete posts under the scope of a granular retention policy
* add suggestions from sbishel
* allow clients to specify channels/teams when creating a new policy
* remove foreign keys referencing Channels and Teams tables
* add checks for whether teams and channels exist
* pre-checkout commit
* remove data referencing the Posts table
* pre-checkout commit
* write data store tests
* sort results of buildGetPoliciesQuery
* add missing test cases for teams
* pre-checkout commit
* add Client4 methods for data retention policy endpoints
* add uint and uint64 to app/layer_generators
* make granular policies override global policies
* fix lint errors
* pre-checkout commit
* add license to top of files
* add tests for data store layer
* add missing test cases for store layer
* run make i18n-extract
* add query to delete ChannelMemberHistory
* work in progress
* add test for old reply to old post
* fix lint error
* use COALESCE on each Posts column
* begin implementing orphaned rows worker
* split PR
* pre-checkout commit
* use RetentionPolicyWithTeamAndChannelCounts
* update app and api layers
* run make i18n-extract
* add RetentionPolicy to retrylayer_test.go
* Revert "split PR"
This reverts commit b316f03dd307a30deae931944ca7e4a1cc904605.
* fix errors caused by revert
* add suggestions from sbishel
* fix copy-paste error
* fix lint errors
* pre-checkout commit
* add function to delete orphaned rows
* use -1 for infinite retention
* remove check constraint
* copy i18n entries from master
* re-run tests with newer enterprise branch
* add team data to channel list
* add search for channels and teams in a policy
* add store tests for channel and team search
* add suggestions from mkraft
* run make einterfaces-mocks
* fix lint errors
* add suggestions from mkraft
* move removeOrphanedRows method to wireup branch
* Revert "move removeOrphanedRows method to wireup branch"
This reverts commit 94605c9b4a5378ffa44a3dec4d3f8e3306b9d33e.
* use DeleteOrphanedRows where possible
* run make i18n-extract
* use COMPLIANCE permissions
* run make migrations-bindadta
* clean up teams before test
* fix tests for TestRetentionPolicyStore
* add API endpoints for mobile
* fix lint error
* fix some of the lint errors
* move user/data_retention endpoints to data_retention.go
* Revert "fix some of the lint errors"
This reverts commit b5b2dc27566c427187db942c5c0afe319e7679c4.
* add exclude_policy_constrained parameter for /channels and /teams
* fix lint errors
* add policy_id field to GET endpoints for channels and teams
* use PolicyWithTeamID in RetentionPolicy layer
* fix lint errors
* run make i18n-extract
* update mock call in telemetry_test.go
* return status:OK in JSON instead of 204
* pre-checkout commit
* add policy_id field on channels/teams
* fix lint errors
* use sq.Eq instead of '?'
* use new subsection permissions
* update channels and teams endpoints to use new subsection permissions
* add extra search opts for channels in a policy
* fix lint errors
* allow negative post duration in patch
* remove DELETE FROM query in retention policy tests
* use *int64 for PostDuration
* re-run CI tests
* use 3-step deletion strategy for each table
* fix lint errors
* run make store-layers
* re-run CI tests
* add test with channel, team and global policies
* use common function for SQL queries
* add pagination test
* use struct for args to common SQL function
* fix lint errors
* run make i18n-extract
* check if Channels.TeamId is "" or nil
* use three OR clauses
* write separate genericRetentionPoliciesDeletion function
* add config setting for BatchSize
* add telemetry for BatchSize
* use feature flag
* add old i18n messages back in
* re-run CI tests
* update call signature in storetest
* MM-30831: Adds constant for retention default batch size.
* MM-30831: Removes comment re: optimization.
* MM-30831: Converts days to milliseconds.
* MM-30831: Reverts change to test.
* Revert "MM-30831: Reverts change to test."
This reverts commit 6d14275a1ceae682bb9e17ec69b39252b44e0c0c.
* Revert "MM-30831: Converts days to milliseconds."
This reverts commit a0cb6ec09d854a05194c1daee1c5333f260231c3.
* MM-30831: Fixes tests.
* MM-30381: Fix for change to method sig.
Co-authored-by: Max Erenberg <max.erenberg@mattermost.com>
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
Co-authored-by: Martin Kraft <martin@upspin.org>
This PR builds up on the pass-through DB driver to a fully functioning DB driver implementation via our RPC layer.
To keep things separate from the plugin RPC API, and have the ability to move fast with changes, a separate field Driver is added to MattermostPlugin. Typically the field which is required to be compatible are the API and Helpers. It would be well-documented that Driver is purely for internal use by Mattermost plugins.
A new Driver interface was created which would have a client and server implementation. Every object (connection, statement, etc.) is created and added to a map on the server side. On the client side, the wrapper structs hold the object id, and communicate via the RPC API using this id.
When the server gets the object id, it picks up the appropriate object from its map and performs the operation, and sends back the data.
Some things that need to be handled are errors. Typical error types like pq.Error and mysql.MySQLError are registered with encoding/gob. But for error variables like sql.ErrNoRows, a special integer is encoded with the ErrorString struct. And on the cilent side, the integer is checked, and the appropriate error variable is returned.
Some pending things:
- Context support. This is tricky. Since context.Context is an interface, it's not possible to marshal it. We have to find a way to get the timeout value from the context and pass it.
- RowsColumnScanType(rowsID string, index int) reflect.Type API. Again, reflect.Type is an interface.
- Master/Replica API support.
* MM-31717: Remove HTTP clustering
https://mattermost.atlassian.net/browse/MM-31717
```release-note
Gossip clustering mode is now GA and is no longer
available as a choice now. All cluster traffic will always
use the gossip protocol.
The config setting UseExperimentalGossip has no effect
and has only been kept for compatibility purposes.
The setting to use gossip has been removed from admin console.
Upgrade note:
If an existing system is not using gossip, then one cannot
upgrade one node in a cluster and keep the other one to the older
version, if an HA upgrade is intended. All nodes
in the cluster must use a single protocol.
Either the customer must use gossip if they want to do an HA
upgrade. Or, shut down all nodes and bring up all of them
after doing an upgrade.
```
* Trigger CI
* update translations
