* MM-38982: fixes SVG uploading
Uploading SVG files was "almost" treated like an image in some places,
resulting in not showing the SVG on the client.
This is happening because when we fail to generate a mini preview for an
image, we prepend "invalid-" to the MimeType.
This commit adds a check to guard against SVG files in methods that make
no sense for SVGs.
* Reverts invalid-{mimetype} fix
* Uses fileInfo.IsSvg() where it can
* MM-41211: Replaces SessionLength[Web|Mobile|SSO]InDays with SessionLength[Web|Mobile|SSO]InHours.
* MM-41211: Clear the value of the old config settings.
The older method used to reply completely on timestamps
to take batches of items in a timestamp range and then
just incrementing the timestamp. This led to handling
edge-cases such as more items than the batch count, all
having the same timestamp.
Additionally, relying on timestamp as the page cursor
meant that indexing was not very efficient if you had
several items spread out across large spans of time.
To get away from all of that we use a proper cursor-based
approach consisting of createAt+Id. With this, we move
completely to a constant page size where we can fetch
a given number of objects irrespective of when they
were created. This makes indexing much more faster and
efficient.
https://mattermost.atlassian.net/browse/MM-41260
```release-note
Elasticsearch and Bleve indexing have been revamped to be much
more efficient and faster. The config parameter BulkIndexingTimeWindowSeconds
for both elasticsearch and bleve have been removed.
A new config parameter called BatchSize has been introduced instead.
This parameter controls the number of objects that
can be indexed in a single batch. This makes things
more efficient and maintains a constant workload.
```
We implement a cursor based pagination model
to page through the posts in a given thread.
The cursor is a combination of the post.CreateAt+
post.Id to differentiate multiple posts in a given
timestamp.
Some additional parameters like direction, fromPost,
fromCreateAt and perPage were introduced to implement
this.
```release-note
NONE
```
This fixes a classic concurrency bug where the stop channel
was being listened in 2 places. In the Run method and DoJob
method.
The problem with that is that one of them will consume
the signal, and if the DoJob method does, then the parent
for-select loop never exits, because the return statement
only runs when the stop signal is consumed.
This would lead to server hanging on shutdown.
To fix this, we close the channel rather than sending a
message. This exits all listeners listening on the channel
and lets the job exit gracefully.
This is a sample stack dump of a stuck goroutine
```
goroutine 1 [chan receive]:
runtime.gopark(0x235be40?, 0xc000faa530?, 0xa8?, 0xf3?, 0x27a1170?)
/home/agniva/gosource/go/src/runtime/proc.go:366 +0xd6 fp=0xc002e77510 sp=0xc002e774f0 pc=0x43c856
runtime.chanrecv(0xc000faa7e0, 0x0, 0x1)
/home/agniva/gosource/go/src/runtime/chan.go:577 +0x56c fp=0xc002e775a0 sp=0xc002e77510 pc=0x40868c
runtime.chanrecv1(0x27cab4c?, 0xf?)
/home/agniva/gosource/go/src/runtime/chan.go:440 +0x18 fp=0xc002e775c8 sp=0xc002e775a0 pc=0x4080b8
github.com/mattermost/mattermost-server/v6/jobs/migrations.(*Worker).Stop(0xc001954280)
/home/agniva/mattermost/mattermost-server/jobs/migrations/worker.go:66 +0x171 fp=0xc002e77698 sp=0xc002e775c8 pc=0x187f8d1
github.com/mattermost/mattermost-server/v6/jobs.(*Workers).Stop(0xc0019df2c0)
/home/agniva/mattermost/mattermost-server/jobs/workers.go:84 +0x12c fp=0xc002e77748 sp=0xc002e77698 pc=0x17b04ec
github.com/mattermost/mattermost-server/v6/jobs.(*JobServer).StopWorkers(0xc00021f180?)
/home/agniva/mattermost/mattermost-server/jobs/server.go:104 +0x9c fp=0xc002e777a0 sp=0xc002e77748 pc=0x17af8bc
github.com/mattermost/mattermost-server/v6/app.(*Server).Shutdown(0xc00021f180)
/home/agniva/mattermost/mattermost-server/app/server.go:1019 +0x965 fp=0xc002e77c48 sp=0xc002e777a0 pc=0x1dd6865
github.com/mattermost/mattermost-server/v6/cmd/mattermost/commands.runServer.func3()
/home/agniva/mattermost/mattermost-server/cmd/mattermost/commands/server.go:79 +0x26 fp=0xc002e77c60 sp=0xc002e77c48 pc=0x214bbe6
github.com/mattermost/mattermost-server/v6/cmd/mattermost/commands.runServer(0xc000e1e000, 0xb?)
/home/agniva/mattermost/mattermost-server/cmd/mattermost/commands/server.go:122 +0x239 fp=0xc002e77d50 sp=0xc002e77c60 pc=0x214ba39
github.com/mattermost/mattermost-server/v6/cmd/mattermost/commands.serverCmdF(0x4207ac0?, {0x279d9d6?, 0x0?, 0x0?})
/home/agniva/mattermost/mattermost-server/cmd/mattermost/commands/server.go:58 +0x11d fp=0xc002e77dd0 sp=0xc002e77d50 pc=0x214b37d
github.com/spf13/cobra.(*Command).execute(0x4207ac0, {0xc00003c200, 0x0, 0x0})
/home/agniva/mattermost/mattermost-server/vendor/github.com/spf13/cobra/command.go:856 +0x67c fp=0xc002e77ea8 sp=0xc002e77dd0 pc=0x211a21c
github.com/spf13/cobra.(*Command).ExecuteC(0x4207ac0)
/home/agniva/mattermost/mattermost-server/vendor/github.com/spf13/cobra/command.go:974 +0x3b4 fp=0xc002e77f60 sp=0xc002e77ea8 pc=0x211a894
github.com/spf13/cobra.(*Command).Execute(...)
/home/agniva/mattermost/mattermost-server/vendor/github.com/spf13/cobra/command.go:902
github.com/mattermost/mattermost-server/v6/cmd/mattermost/commands.Run(...)
/home/agniva/mattermost/mattermost-server/cmd/mattermost/commands/root.go:14
main.main()
/home/agniva/mattermost/mattermost-server/cmd/mattermost/main.go:31 +0x86 fp=0xc002e77f80 sp=0xc002e77f60 pc=0x21f9ee6
runtime.main()
/home/agniva/gosource/go/src/runtime/proc.go:255 +0x227 fp=0xc002e77fe0 sp=0xc002e77f80 pc=0x43c487
runtime.goexit()
/home/agniva/gosource/go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc002e77fe8 sp=0xc002e77fe0 pc=0x46d361
```
This is not strictly related to the JIRA ticket but
a general refactor
https://mattermost.atlassian.net/browse/MM-40272
```release-note
NONE
```
* Updated links to legacy domain about.mm.com
* Legacy link updates
* Reverting - only need to change en.json
* Reverting - only need to change en.json
* Reverting - only need to change en.json
* Reverting - only need to change en.json
* Reverting - only need to change en.json
* Reverting - only need to change en.json
* Reverting - only need to change en.json
* Reverting - only need to change en.json
* Reverting - only need to change en.json
* Reverting - only need to change en.json
* Reverting - only need to change en.json
* Reverting - only need to change en.json
* Reverting - only need to change en.json
* Reverting - only need to change en.json
* Reverting - only need to change en.json
* Reverting - only need to change en.json
* Reverting - only need to change en.json
* Reverting - only need to change en.json
* Reverting - only need to change en.json
* about.mm.com URL updates
* mattermost.org URL updates
* forum.mm.org URL update
* Update .github/ISSUE_TEMPLATE.md
* Update .github/ISSUE_TEMPLATE.md
* Un-deleted language files
* Update README.md
* Update tests/test-config.json
* fix some test due to url updating (#19787)
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
Co-authored-by: Harrison Healey <harrisonmhealey@gmail.com>
Co-authored-by: Ibrahim Serdar Acikgoz <serdaracikgoz86@gmail.com>
* WIP
* adding initial creategroup endpoint
* fetching by group source
* fixing startup error
* updating create endpoint to take an array of user_ids, this will allow us to create the group with one request
* adding delete group endpoint and appropriate test
* adding source param for getGroups
* adding add members and delete members endpoints
* locking down crud endpoints to only be allowed for custom groups
* user search stuff
* allowing remoteid be null by changing field to pointer
* code cleanup and store level tests
* adding new tests and removing unused endpoint
* resolving conflicts
* Adds authz check for group.
* Adds authz checks to groups APIs.
* Updated create group authz tests.
* Updates delete group tests.
* Tests create group.
* Adds some tests and validations.
* adding new parameter so I can get users not in a group
* Fixed all lint warnings.
* Fix type.
* fixing search users not in group
* Fixes some lint errors.
* Moves entry in JSON array.
* Fixed SQL query.
* Fixes permission migration test.
* Fixes migration test.
* Fixes some group store tests.
* Fix test.
* Fix test.
* Revert lint change.
* Migrated CreateWithUserIds to sqlx.
* Adds tests for GetMember; migrates implementation to sqlx.
* Tests GetNonMemberUsersPage and hanles wrong group id.
* Fixes test.
* Switches GetMaster to GetMasterX.
* Switches GetReplica to GetReplicaX.
* Fixes logic.
* Fixes shadow declaration.
* Adds include_member_count to get group API endpoint.
* Adds filter_has_member param to getGroups.
* Fixes.
* Removes array of group sources.
* fixing error
* Testing reverting CreateWithUserIds back to gorp.
* Added websocket event for CreateGroupWithUserIds.
* Changed a few response status codes. Switched to correct permission.
* Added member count to ws payload for group when updating or creating.
* Adds feature flag checks for custom groups.
* Added middleware function to require license. Added config to disable custom groups.
* Change for function signature change of executePossiblyEmptyQuery.
* Lint fixes.
* Adds telemetry none comment.
* Adds translations.
* Migrated to sqlx.
* Temp. removal of translation.
* Fixed typo.
* Added an intermediary model to query with a field that is now ignored by sqlx on read queries.
* Re-used existing store struct.
* Inludes member count.
* Fix for merge error.'
* Require license for group endpoints.
* Updates translations.
* Fix shadow declaration.
* Renames permissions. Switches to new method to retrieve remoteid.
* Added WS events for upsert and delete member(s).
* Added new store error type ErrUniqueConstraint.
* Added EnableCustonGroups to the client config.
* Sanitized some user records.
* Added parameter to include_total_count for listing groups.
* Added translations.
* adding deleteAt field to getByUsers query
* Revert sanitize.
* Added uniqueness constraint error to UpdateGroup.
* Removed the FutureFeatures flag so that the feature is not enabled on old Enterprise licenses.
* Renamed function.
* Updates authz check for user search related to groups.
* Removed debug statement.
* Removed unused app method.
* Added telemetry for enable_custom_groups.
* Returns early from nil license.
* Updates test.
* Returned early to avoid nesting in (*SqlGroupStore).checkUserExist. Switched to reading from replica in (*SqlGroupStore).GetMember. Handled JSON marshal error in (*Client4).UpsertGroupMembers
* Switched to SanitizeProfile.
* Switched to model.NewInt.
* Switched from status NotImplemented to Forbidden for missing license.
* Removed deactivated users from 'exists' set.
* Revert gotool update.
* Ignored lint error that I think is invalid.
* Added the approprate access tag for disabling custom groups.
* Revert change to response status.
* Fixed refactor mistake.
* Limited the group member WS events to individual users.
* Removed WS event of deleted groups.
* Updated license check for searchUsers endpoint.
* Switched from license feature to license sku.
* Update app/group.go
Co-authored-by: Claudio Costa <cstcld91@gmail.com>
* Update app/group.go
Co-authored-by: Claudio Costa <cstcld91@gmail.com>
* Remove linter ignore comment.
* Added function to create sku-specific license.
* Fixed typo. Removed comment.
* Fixed for wrong type.
* Added missing param to client. Removed unnecessary props setting. Added test for retrieving groups by source.
* Updated some tests now that we're validating group membership not created for deactivated user.
* Fix for groups endpoint returning all group types by default.
* Changes constant names. Adds migration for all users to manage custom group members.
* Removes requirement for manage_system permission to filter user search by group.
* Added migration mock.
* Removes default permissions from custom_group_user role.
* Fixes migration.
* Fixes emoji migration test.
* fixing issue with member counts
* fixing search issue for deleted members
Co-authored-by: Benjamin Cooke <benjamincooke@Benjamins-MacBook-Pro.local>
Co-authored-by: Benjamin Cooke <benjamincooke@Benjamins-MBP.ht.home>
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
Co-authored-by: Benjamin Cooke <benjamincooke@Benjamins-MacBook-Pro.fritz.box>
Co-authored-by: Claudio Costa <cstcld91@gmail.com>
* Working on refactoring jobs service
* Making more consistent with the previous existing code
* Remove no longer needed functions
* Making a base PeridicScheduler to use it in most of the schedulers implementations
* Removing accidental complexity from on of the jobs
* Removing accidental complexity from expirynotify
* Fixing compilation from previous commit
* Remove accidental complexity from the export_delete job
* Simplifying the workers by making a reusable worker
* Using simple worker for export_delete job
* Simpliying export process job
* Simpliying extract content job
* Simpliying import delete job
* Simpliying import process job
* Simpliying product noticies job
* Simpliying fix crt channel unreads job (only removing the uneeded register function)
* Simpliying migrations job (only removing the uneeded register function)
* fixup
* Simpliying plugins job (only removing the uneeded register function)
* Simpliying bleve indexing job (only removing the uneeded register function)
* Simpliying resend invitation email job (only removing the uneeded register function)
* Fixing tests
* Simplifying migration tests infrastructure
* Adding missed license to files
* Adding an empty file to imports package to ensure this package exist even without enterprise repo
* Regenerating einterfaces mocks
* Adding missed license to files
* Updating i18n/en.json file
* help fixing enterprise tests compilation
* Adding new DailyScheduler
* Fixing typo and changing the waitTime type for periodic sechduler
* Making the daily scheduler more generic
* Adding comments to clarify not used parameters in interface scheduler interface implementations
* Using merror to handle multiple errors in jobs workers
* Fixing linter errors
* Addressing PR review comments
* Reverting go.tools.mod changes
* Removing the static check for worker type in the model (moving it to the insertion of new jobs
* Moving migrations job to the jobs directory
* Fixing (and improving a bit) tests
* Apply suggestions from code review
Co-authored-by: Doug Lauder <wiggin77@warpmail.net>
* Fixing enterprise tests
* Removing unneeded InitWorkers/InitSchedulers calls
* Fix expirenotify job when error happens
* Fixing govet errors
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
Co-authored-by: Doug Lauder <wiggin77@warpmail.net>
* MM-41211: Accepts a decimal for the SSO session length.
* MM-41211: Fixes test and logic error.
* MM-41211: Validates session length is minimum 1 hour.
* MM-41211: Tricking the CI to allow an empty string.
* Fixes a test error and some lint changes.
* Fix error.
* Reverting IDE change.
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* add Boards to DataRetention, add hook for data retention
* remove replaces
* update hook to remove parameter
* add boards data retention to telemetry
* fix unit test
* update test, update hooks
* update RunDataRetention server version
* put behind a feature flag
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* GH-18288: Allow configuring unsafe-eval and unsafe-inline
* Add developer flags to telemetry
* wip
* wip
* Refactor based on review
* Refactor based on review
* fix expected vs. actual in assert.Equal
* Add unit tests, rework to check only supported flags.
* Update model/config.go
Co-authored-by: Jesse Hallam <jesse@thehallams.ca>
* Fix failing tests
* Refactor based on review
* Refactor based on review
* Refactor based on review
Co-authored-by: Jesse Hallam <jesse.hallam@gmail.com>
Co-authored-by: Jesse Hallam <jesse@thehallams.ca>
* Add JobSettings.CleanupJobsThresholdDays to telemetry
```release-note
NONE
```
* Moved telemetry to more specific fields
```release-note
NONE
```
* Add missing space
```release-note
NONE
```
* Show private channels in autocomplete
This is supported in all Engines:
MySQL, Postgres, Bleve, Elasticsearch.
https://mattermost.atlassian.net/browse/MM-18496
```release-note
Private channels will now appear in channel autocomplete.
If you are using Bleve or ElasticSearch, you will have to reindex
the channels again to populate them with the new attributes.
```
A large chunk of this work has been based on the earlier
effort at https://github.com/mattermost/mattermost-server/pull/17804.
Full credit goes to https://github.com/arvinDarmawan.
* Add comment
```release-note
NONE
```
* Adding more tests
```release-note
NONE
```
* fix more tests
```release-note
NONE
```
* tmp
```release-note
NONE
```
* more fixes
```release-note
NONE
```
* add tests
```release-note
NONE
```
* Add review comments from previous PR
```release-note
NONE
```
* Add API to return all channels from all team
```release-note
NONE
```
* Added support for bleve and ES
```release-note
NONE
```
* Streaming response for GetAllChannels
```release-note
NONE
```
* Fix tests
```release-note
NONE
```
* Trigger CI
```release-note
NONE
```
* fix tests
```release-note
NONE
```
* Addressing review comments
```release-note
NONE
```
* Fix lint
```release-note
NONE
```
* Removing flaky test
```release-note
NONE
```
* Address comments
```release-note
NONE
```
* Trigger CI
```release-note
NONE
```
* Added /users/<userid>/channel_members endpoint
```release-note
NONE
```
* Minor edit
```release-note
NONE
```
* Improve embedding
```release-note
NONE
```
* Fix lint error
```release-note
NONE
```
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* Move some more atomic values under Channels
The following fields were moved:
```
asymmetricSigningKey atomic.Value
clientConfig atomic.Value
clientConfigHash atomic.Value
limitedClientConfig atomic.Value
```
And also moved the initialization order from NewServer
to Channels.Start to better reflect the order of things.
Removed AsymmetricSigningKey from ConfigService
as it was no longer used.
Removed calling regenerateClientConfig during startup explicitly
because it was anyways called from ensureAsymmetricSigningKey.
https://community-daily.mattermost.com/boards/workspace/zyoahc9uapdn3xdptac6jb69ic/285b80a3-257d-41f6-8cf4-ed80ca9d92e5/495cdb4d-c13a-4992-8eb9-80cfee2819a4?c=87df1e15-588e-49ff-8bd1-ffa9651b8c82
```release-note
NONE
```
* Fix lint error
```release-note
NONE
```
* removing deprecated stuff
* removing deprecations and marking others for deprecations
* removing some of the configs
* partially remove ExperimentalChannelOrganization
* fixing merge issues
* moving default permissions to defaultroles function
* removing experimentaltownsquare
* removing old tests and remove legacysidebar config
* fixing test
* removing log
* removing unused variables
* MM-36444: Fix for renamed functions.
* MM-36444: go mod tidy and go mod vendor.
* MM-36444: Vendored go modules.
* MM-36444: Removes UseExperimentalGossip config field from model.
* Fix vendors
* fixing test
Co-authored-by: Benjamin Cooke <benjamincooke@Benjamins-MacBook-Pro.local>
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
Co-authored-by: Martin Kraft <martin@upspin.org>
Co-authored-by: Claudio Costa <cstcld91@gmail.com>
Fix an issue where context fields logged by server on behalf of plugins didn't contain correct name/value pairs:
- bump Logr version to add key/value methods to sugar logger
- expose factory args when configuring logging with custom target types (needed for FocalBoard to create log target adapter that converts typed fields into slices of interface{} as per plugin logging API)
