fmt.Sprintf was badly optimized and in cases of heavy traffic,
it was in the hot path and caused a lot of allocations.
We sacrifice readability for performance here.
```
10:46:40-~/mattermost/mattermost-server/model] benchstat base.txt new.txt
goos: linux
goarch: amd64
pkg: github.com/mattermost/mattermost-server/v6/model
cpu: Intel(R) Core(TM) i5-8265U CPU @ 1.60GHz
│ base.txt │ new.txt │
│ sec/op │ sec/op vs base │
EncodeJSON-8 2.061µ ± 1% 1.588µ ± 2% -22.93% (p=0.000 n=10)
│ base.txt │ new.txt │
│ B/op │ B/op vs base │
EncodeJSON-8 545.0 ± 0% 472.0 ± 0% -13.39% (p=0.000 n=10)
│ base.txt │ new.txt │
│ allocs/op │ allocs/op vs base │
EncodeJSON-8 6.000 ± 0% 3.000 ± 0% -50.00% (p=0.000 n=10)
```
```release-note
Writes to websocket now take 13% less memory
and also 22% faster per message.
```
* MM-49048: do not omit needed wsbroadcast vars
It seems that we are omitting ContainsSanitizedData, and
ContainsSensitiveData variables in WebsocketBroadcast.
This created a bug in cluster mode, in which we were sending duplicate
events to users whereas normally they would have access to only one of those.
Co-authored-by: Mattermost Build <build@mattermost.com>
* Self-hosted admins can purchase licenses in-app when `ServiceSettings,SelfHostedPurchase` is true (the default)
* Content Security Policy enables loading assets from `js.stripe.com/v3` when `ServiceSettings.SelfHostedPurchase` is true (the default).
* Add `hosted_customer` API subpath
* Add status of SelfHostedPurchase to telemetry config report.
* Support showing admins self-hosted invoices when `ServiceSettings.SelfHostedPurchase` is true (the default)
Co-authored-by: Conor Macpherson <116016004+ConorMacpherson@users.noreply.github.com>
* MM-23881: global drafts endpoints and ws events
Adds endpoints:
- create/update drafts
- delete draft
- get drafts
Adds WS events:
- draft_updated
- draft_created
- draft_deleted
* Ordering and WS event name fixes
* Adds PostID to the drafts table
In the future the drafts will include edited posts, this commit adds the
post id in the combined pkey of the table.
* Fixes route for deleting a thread draft
* Fixes failed checks
* Fixes migrations
* Fixes migration
* Extract translation strings
* Removes PostID since we won't sync editing posts
* Fixes tests
* Fixes i18n
* Update migrations for global drafts
* update branch with latest master changes
* Add feature flag for global drafts
* Set global drafts feature flag default to true
* Added support for files in drafts
* Fix failing i18n check
* Added support for deleting files in drafts
* Revert "Added support for deleting files in drafts"
This reverts commit 45dfd04a760359de2e8814d652c9ef46daf994f6.
* Triggering new test server
* Add config setting 'AllowSyncedDrafts' for syncing drafts with server
* Triggering new test server
* Triggering new test server
* Add guard for config setting and add initial tests
* Fix i18n and lint errors
* Triggering new test server
* Add tests for drafts
* fix lint issues
* Add tests for model/draft
* Triggering new test server
* Triggering new test server
* Trigger new test server
* Address PR comments
* Change left join to regular join in GetDraftsForUser
* Fix broken test
Maybe consider adding an inclDeleted field if we want to get deleted drafts in the future
* fix translations
* Add store tests for drafts
* fix test naming
* remove comment
* update migrations
* set feature flag default to false
* update migrations
Co-authored-by: Mylon Suren <mylonsuren@gmail.com>
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* MM-46410: adds urgency on mention counts
We have introduced priority for posts in
https://github.com/mattermost/mattermost-webapp/pull/10951.
We do need to color the mention badges in the webapp with a prominent
color when a mention is posted in an urgent message.
A thread has urgent mentions if the root post is marked as urgent, and
the replies contain mentions to the user viewing the thread.
This PR adds two columns, urgentmentioncount, and isurgent, in
channelmembers, and threads tables respectively.
Furthermore when asking for team/thread mention counts, we also return
urgent mention counts for the user.
* Fixes method in tests
* empty commit
* Fixes method call
* Fixes single thread response is_urgent
* Fixes errors
* Fixes mysql migration and adds graphql schema
* Fixes tests
* Refactors IsUrgent and Adds PostsPriority table
Changes:
- removes is_urgent from the threads table
- adds a new table to hold posts priorities
- refactors priority out of the props and into the new table
* Fixes
* Adds translation strings
* Fixes migrations and tests
* Fixes tests
* empty
* Adds Priority to Copy
* empty
* Fixes priority not saved when boards is enabled
We are nilifying Metadata when post.ForPlugin(), which didn't save Priority
for a post when Boards was enabled.
This commit copies metadata again to the post, so metadata are
reinstated.
* Fixes tests
* Adding store tests and fixes syntax error
* Uses threads.ThreadTeamId
* Fixes error
* Adds UrgentMentionCount in graphql api test
* Fetches post priority in batches
* Addresses review comments
* Restore only priority on create post
* Fixes tests
* Nits
* Some refactoring
* Fixes get thread options when post priority enabled
* Adds missing translation
* Use the constant instead of "urgent" string
* Renames urgent constant
* MM-47750: Adds PostAcknowledgements table and apis
- Adds post acknowledgement api/app/store methods to be able to save and
delete post acknowledgements by users.
- Adds wesbsocket events for acknowledgement created/deleted
- Returns post acknowledgements in the post's metadata
* Empty
* Fixes incorrect urgent count when marking a post as unread
* Adds license
* Fixes ACK api, and adds tests
* Fixes vet
* Fixes tests
* Addresses review comments
* Remove unnecessary lines
* Adds config option and changes return of delete ack
* Empty
* Empty
* Enable config by default
* Fixes intl
* Fixes test after setting config default true
* Changes endpoints to PostForUser
* Avoids replica lag
* Fixes error in merge
* Fixes RetryLayer tests due to merge
* Empty
* Empty
* Empty
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* Adds OmitConnection parameter to broadcast
Currently we have no means to omit sending a websocket event to a
specific connection id.
This is needed mainly so that the initiator won't receive an event for
the action it just initiated.
Will be used for the global drafts feature, so that we won't update
drafts through ws when a user is typing.
This commit adds OmitConnection to the Broadcast struct and to the
NewWebSocketEvent function signature.
shouldSendEvent should return false for that specific connection.
* Return early only if connection id matches the omitted
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* WIP
* adding initial creategroup endpoint
* fetching by group source
* fixing startup error
* updating create endpoint to take an array of user_ids, this will allow us to create the group with one request
* adding delete group endpoint and appropriate test
* adding source param for getGroups
* adding add members and delete members endpoints
* locking down crud endpoints to only be allowed for custom groups
* user search stuff
* allowing remoteid be null by changing field to pointer
* code cleanup and store level tests
* adding new tests and removing unused endpoint
* resolving conflicts
* Adds authz check for group.
* Adds authz checks to groups APIs.
* Updated create group authz tests.
* Updates delete group tests.
* Tests create group.
* Adds some tests and validations.
* adding new parameter so I can get users not in a group
* Fixed all lint warnings.
* Fix type.
* fixing search users not in group
* Fixes some lint errors.
* Moves entry in JSON array.
* Fixed SQL query.
* Fixes permission migration test.
* Fixes migration test.
* Fixes some group store tests.
* Fix test.
* Fix test.
* Revert lint change.
* Migrated CreateWithUserIds to sqlx.
* Adds tests for GetMember; migrates implementation to sqlx.
* Tests GetNonMemberUsersPage and hanles wrong group id.
* Fixes test.
* Switches GetMaster to GetMasterX.
* Switches GetReplica to GetReplicaX.
* Fixes logic.
* Fixes shadow declaration.
* Adds include_member_count to get group API endpoint.
* Adds filter_has_member param to getGroups.
* Fixes.
* Removes array of group sources.
* fixing error
* Testing reverting CreateWithUserIds back to gorp.
* Added websocket event for CreateGroupWithUserIds.
* Changed a few response status codes. Switched to correct permission.
* Added member count to ws payload for group when updating or creating.
* Adds feature flag checks for custom groups.
* Added middleware function to require license. Added config to disable custom groups.
* Change for function signature change of executePossiblyEmptyQuery.
* Lint fixes.
* Adds telemetry none comment.
* Adds translations.
* Migrated to sqlx.
* Temp. removal of translation.
* Fixed typo.
* Added an intermediary model to query with a field that is now ignored by sqlx on read queries.
* Re-used existing store struct.
* Inludes member count.
* Fix for merge error.'
* Require license for group endpoints.
* Updates translations.
* Fix shadow declaration.
* Renames permissions. Switches to new method to retrieve remoteid.
* Added WS events for upsert and delete member(s).
* Added new store error type ErrUniqueConstraint.
* Added EnableCustonGroups to the client config.
* Sanitized some user records.
* Added parameter to include_total_count for listing groups.
* Added translations.
* adding deleteAt field to getByUsers query
* Revert sanitize.
* Added uniqueness constraint error to UpdateGroup.
* Removed the FutureFeatures flag so that the feature is not enabled on old Enterprise licenses.
* Renamed function.
* Updates authz check for user search related to groups.
* Removed debug statement.
* Removed unused app method.
* Added telemetry for enable_custom_groups.
* Returns early from nil license.
* Updates test.
* Returned early to avoid nesting in (*SqlGroupStore).checkUserExist. Switched to reading from replica in (*SqlGroupStore).GetMember. Handled JSON marshal error in (*Client4).UpsertGroupMembers
* Switched to SanitizeProfile.
* Switched to model.NewInt.
* Switched from status NotImplemented to Forbidden for missing license.
* Removed deactivated users from 'exists' set.
* Revert gotool update.
* Ignored lint error that I think is invalid.
* Added the approprate access tag for disabling custom groups.
* Revert change to response status.
* Fixed refactor mistake.
* Limited the group member WS events to individual users.
* Removed WS event of deleted groups.
* Updated license check for searchUsers endpoint.
* Switched from license feature to license sku.
* Update app/group.go
Co-authored-by: Claudio Costa <cstcld91@gmail.com>
* Update app/group.go
Co-authored-by: Claudio Costa <cstcld91@gmail.com>
* Remove linter ignore comment.
* Added function to create sku-specific license.
* Fixed typo. Removed comment.
* Fixed for wrong type.
* Added missing param to client. Removed unnecessary props setting. Added test for retrieving groups by source.
* Updated some tests now that we're validating group membership not created for deactivated user.
* Fix for groups endpoint returning all group types by default.
* Changes constant names. Adds migration for all users to manage custom group members.
* Removes requirement for manage_system permission to filter user search by group.
* Added migration mock.
* Removes default permissions from custom_group_user role.
* Fixes migration.
* Fixes emoji migration test.
* fixing issue with member counts
* fixing search issue for deleted members
Co-authored-by: Benjamin Cooke <benjamincooke@Benjamins-MacBook-Pro.local>
Co-authored-by: Benjamin Cooke <benjamincooke@Benjamins-MBP.ht.home>
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
Co-authored-by: Benjamin Cooke <benjamincooke@Benjamins-MacBook-Pro.fritz.box>
Co-authored-by: Claudio Costa <cstcld91@gmail.com>
* MM-25516: Changed to byte slice instead of string for cluster messages
https://mattermost.atlassian.net/browse/MM-25116
Testing:
Manually tested.
Load-tested with Cluster Controller.
I looked into changing the serialization method to use msgpack,
but the ClusterMessage struct was mainly used for only 3 fields
which didn't lead to much of a CPU time improvement, whereas
actually led to more allocations using msgpack. Hence, I chose
to remain with JSON.
```
name old time/op new time/op delta
ClusterMarshal-8 3.51µs ± 1% 3.10µs ± 2% -11.59% (p=0.000 n=9+10)
name old alloc/op new alloc/op delta
ClusterMarshal-8 776B ± 0% 1000B ± 0% +28.87% (p=0.000 n=10+10)
name old allocs/op new allocs/op delta
ClusterMarshal-8 12.0 ± 0% 13.0 ± 0% +8.33% (p=0.000 n=10+10)
```
```release-note
Changed the field type of Data in model.ClusterMessage to []byte from string.
```
* Trigger CI
```release-note
NONE
```
* parent 48256721c4 (#14358)
author Eli Yukelzon <reflog@gmail.com> 1585814774 +0300
committer Eli Yukelzon <reflog@gmail.com> 1589111022 +0300
Sidebar caregories implemented
Apply suggestions from code review
Co-authored-by: Ibrahim Serdar Acikgoz <serdaracikgoz86@gmail.com>
Update store/sqlstore/channel_store.go
Co-authored-by: Ibrahim Serdar Acikgoz <serdaracikgoz86@gmail.com>
Update store/sqlstore/channel_store.go
Co-authored-by: Ibrahim Serdar Acikgoz <serdaracikgoz86@gmail.com>
code review suggestions
status messages
edge case
bugs...
timeout reverse
* MM-25126 Add a separate default sorting method for categories (#14575)
* MM-25158 Add user to initial sidebar categories when joining team (#14570)
* MM-25281 Place new categories in the correct position (#14609)
* MM-25277 Return channels that aren't in a category as part of the Channels/Direct Messages categories (#14601)
* MM-25276 Remove categories when leaving a team (#14600)
* Remove categories when leaving a team
* layers
* corrected cleanup function
* lint
* .
* corrected errors in postgres
* .
* MM-25280 Ensure that the "update category order" API call only contains real category IDs and isn't missing any IDs (#14626)
* Ensure that the "update category order" API call only contains real category IDs and isn't missing any IDs
* tests
* correct status code
* MM-25278 Change "update category" API to return 400 when changing unsupported fields (#14599)
* MM-25279 Change category migration to only populate channels in Favorites (#14627)
* MM-25157 Add API to delete custom categories (#14574)
* MM-25157 Add API to delete custom categories
* get categories fix
* maxorder fix
* Use correct websocket event when deleting category
* Fix tests and remove debug code
* Actually use the right websocket event this time
* test cleanup
* Update test for new category order
Co-authored-by: Eli Yukelzon <reflog@gmail.com>
* MM-24914 Various fixes for sidebar channel handling (#14756)
* Fix checking for channel membership when reordering channels
* Remove unique constraint on SidebarCategories
* Set column sizes for SidebarCategories and SidebarChannels tables
* Allow changing the sorting method for non-DM categories
* Fix nil pointers in error handling
* Fix orphaned channels from other team being returned in Channels category
* Fix non-orphaned channels being duplicated in the Channels category
* Remove unique constraint on SidebarChannels
* Fix category/name of favorites preferences
* Fix testSidebarChannelsMigration
* Rename err to nErr and appErr to err
* Fix channel order returned by GetSidebarCategories on MySQL
* Fix adding/removing favorites preferences
* Remove leftover TODO
* Change SidebarCategoryType enums to use full names (#14786)
* Change SidebarCategoryType enums to use full names
* Fix Channels constant
* Remove leftover debug code
* MM-24914 Fix updateCategory endpoint returning the wrong type (#14795)
* MM-24914 Make some changes to UpdateSidebarCategories (#14806)
* Fix orphaned DMs not always being returned
* MM-24914 Make some changes to UpdateSidebarCategories
* Run updateSidebarCategoryOrderT in a transaction
* Fix deleting SidebarChannels based on order of arguments to UpdateSidebarCategories
* bump for api testing
* bump for api testing
* Change CreateInitialSidebarCategories to return a plain error
* Change MigrateSidebarCategories to return a plain error
* Remove usage of UpdateColumns when updating sidebar categories (#14843)
* Remove usage of UpdateColumns when changing category order
* Add a random test case
* Remove usage of UpdateColumns when updating sidebar categories (#14843)
* Remove usage of UpdateColumns when changing category order
* Add a random test case
* Remove usage of UpdateColumns when updating sidebar categories (#14843)
* Remove usage of UpdateColumns when changing category order
* Add a random test case
* MM-26343 Make CreateInitialSidebarCategories idempotent (#14870)
* Fix bad merge
* Fix another bad merge
* Fix unintentionally removed i18n string
Co-authored-by: Eli Yukelzon <reflog@gmail.com>
* MM-22212: Read non-moderated permissions from higher-scoped scheme.
* MM-2212: Corrects test count in comment.
* MM-22212: Adds godoc comment.
* MM-2212: Switches to the channel roles check in a few more places.
* MM-22212: Refactors and fixes.
* MM-22212: Reverts change, no longer required.
* MM-22212: Removes translation.
* MM-22212: Un-comments merged new permission.
* MM-22212: Un-comments merged new permission.
* MM-22212: Performance tweak.
* MM-22212: Fixes some fmting.
* MM-22212: Add unit test for newly-added store methods.
* MM-22212: Renames app method.
* MM-22212: Re-uses existing function to find string in slice.
* MM-22212: Keeps 'higher-scoped' terminology for consistency.
* MM-22212: Refactors based on PR feedback.
* MM-22212: Fix for some bad merging.
* MM-22212: Renamed some things.
* MM-22212: Use an 'else' instead of a 'continue' for readability.
* MM-22212: Caches (*SqlRoleStore).ChannelRolesUnderTeamRole.
* MM-22212: Adds mock to new cache store.
* MM-22212: Adds missing open tracing app layer methods.
* MM-22212: Adds migration to add moderated permissions to channel_admin if present on channel_user.
* MM-22212: Migrates team schemes. Removes unused AppError.
* MM-22212: Fix for for if.
* MM-22212: Fixes iterator.
* MM-22212: Updates open tracing generated methods.
* MM-22212: Fix mocks.
* MM-22212: Change migration key name.
* MM-22212: Switched to data structure from other branch.
* MM-22212: Fixes tests after adding 'use_channel_mentions' to the channel_admin role.
* MM-22212: Adds tracking of channel moderation.
* Revert "MM-22212: Adds tracking of channel moderation."
This reverts commit 23689efa22c112e4ba37f6a212535dd7ebfb63db.
* MM-22212: Switch some functions to methods and vice versa.
* MM-22212: Fix for refactor bug not notifiying websocket about changed role.
* MM-22212: Adds test for public/private 'manage_members' handling.
* MM-22122 Fix manage channel members edge case for public and private channels (#14049)
* MM-22212: Adds moderated permission to team_admin.
* MM-22212: Updates migration.
* MM-22212: Revert unnecessary update to default roles.
* Add channel scheme updated event when channel scheme is deleted or created (#14057)
* MM-22212: Adds newline.
* MM-22212: Migration fix.
* MM-22212: Fix for migration.
* MM-22212: Test fix.
Co-authored-by: Farhan Munshi <3207297+fm2munsh@users.noreply.github.com>
* MM-21481: Fixed several issues from user marshalling
- Fixed the root cause for panic by properly converting
the map to a User struct.
- Added a check for type conversion for extra safety.
- Fixed a somewhat unrelated issue of a pointer to pointer reference.
* Fix tests
* Make WebSocketEvent type immutable
* Update code to use updated immutable WebSocketEvent type
* Export WebSocketEvent fields and mark them as deprecated
* Consistent license message for all the go files
* Fixing the last set of unconsistencies with the license headers
* Addressing PR review comments
* Fixing busy.go and busy_test.go license header
* Add interactive dialogs
* Fix unit test
* Updates per feedback
* Fix typo
* Updates per feedback, add icon_url and error returns
* Updates per feedback
* Update per feedback
* send websocket event whenever the channel has changed it's type (public|private)
* updated per comment
Signed-off-by: Saturnino Abril <saturnino.abril@gmail.com>
* add channel_converted websocket event
Signed-off-by: Saturnino Abril <saturnino.abril@gmail.com>
* only send channel_id via websocket message for "channel_converted" event
Signed-off-by: Saturnino Abril <saturnino.abril@gmail.com>
* allow `Wait()`ing on the supervisor
In the event the plugin supervisor shuts down a plugin for crashing too
many times, the new `Wait()` interface allows the `ActivatePlugin` to
accept a callback function to trigger when `supervisor.Wait()` returns.
If the supervisor shuts down normally, this callback is invoked with
a nil error, otherwise any error reported by the supervisor is passed
along.
* improve plugin activation/deactivation logic
Avoid triggering activation of previously failed-to-start plugins just
becase something in the configuration changed. Now, intelligently
compare the global enable bit as well as the each individual plugin's
enabled bit.
* expose store to manipulate PluginStatuses
* expose API to fetch plugin statuses
* keep track of whether or not plugin sandboxing is supported
* transition plugin statuses
* restore error on plugin activation if already active
* don't initialize test plugins until successfully loaded
* emit websocket events when plugin statuses change
* skip pruning if already initialized
* MM-8622: maintain plugin statuses in memory
Switch away from persisting plugin statuses to the database, and
maintain in memory instead. This will be followed by a cluster interface
to query the in-memory status of plugin statuses from all cluster nodes.
At the same time, rename `cluster_discovery_id` on the `PluginStatus`
model object to `cluster_id`.
* MM-8622: aggregate plugin statuses across cluster
* fetch cluster plugin statuses when emitting websocket notification
* address unit test fixes after rebasing
* relax (poor) racey unit test re: supervisor.Wait()
* make store-mocks
* Add user.DeepCopy() function
* Add omit admins/non-admins to WS broadcast and use for updating users
* Updates per feedback and adding unit test for ShouldSendEvent
* Add command and store changes to allow mute toggling
* Change channel muting to use ChannelMember notification structure
* Suppress email and push notifications for a muted channel
* Make i18n keys issue-compliant
* Add notification-cache handling for channel-muting
* Add channel handle for channel-muting slash-command
* Add unit test for mute command
* Merge branch 'master' into PLT-4340
# Conflicts:
# app/notification.go
* Fix issue that command_mute responses will be overwritten
* Fix i18n key for channel muting
* Apply new Provider Interface to MuteCommand
* Migrate mute notification property to mark_unread
PLT-4340
* Make some i18n improvements for command_mute
PLT-4340
* Remove de.json translations
* Prevent push notifications when channel is muted
* Treat Group messages like Direct messages
* Fix unit test
* Send WS event when the channel member notify props changed
