* Add extract documents content command
* Adding the extraction command and making the pure go pdf library as secondary option
* Improving the memory usage and docextractor interface
* Enable content extraction by default in all the instances
* Tiny improvement on archive indexing
* Adding App interface generation and the opentracing layer
* Fixing linter errors
* Addressing PR review comments
* Addressing PR review comments
* always add FeatureFlags to config
* update const to TitleCase
* gofmt
* update method to use 'access:all'
* update method to use 'access:all'
* fix lint
* gofmt
* added comments
* update name of access tag for any
* fix bad save
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
Remote Cluster Service
- provides ability for multiple Mattermost cluster instances to create a trusted connection with each other and exchange messages
- trusted connections are managed via slash commands (for now)
- facilitates features requiring inter-cluster communication, such as Shared Channels
Shared Channels Service
- provides ability to shared channels between one or more Mattermost cluster instances (using trusted connection)
- sharing/unsharing of channels is managed via slash commands (for now)
* MM-34389: Reliable websockets: First commit
This is the first commit which makes some basic changes
to get it ready for the actual implementation.
Changes include:
- A config field to conditionally enable it.
- Refactoring the WriteMessage along with setting the deadline
to a separate method.
The basic idea is that the client sends the connection_id
and sequence_number either during the handshake (via query params),
or during challenge_auth (via added parameters in the map).
If the conn_id is empty, then we create a new one and set it.
Otherwise, we get the queues from a connection manager (TBD)
and attach them to WebConn.
```release-note
NONE
```
https://mattermost.atlassian.net/browse/MM-34389
* Incorporate review comments
* Trigger CI
* removing telemetry
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* MM-33818: Add replica lag metric
We add two new metrics for monitoring replica lag:
- Monitor absolute lag based on binlog distance/transaction queue length.
- Monitor time taken for the replica to catch up.
To achieve this, we add a config setting to run a user defined SQL query
on the database.
We need to specify a separate datasource field as part of the config because
in some databases, querying the replica lag value requires elevated credentials
which are not needed for usual running of the application, and can even be a security risk.
Arguably, a peculiar part of the design is the requirement of the query output to be in a (node, value)
format. But since from the application, the SQL query is a black box and the user can set any query
they want, we cannot, in any way templatize this.
And as an extra note, freno also does it in a similar way.
The last bit is because we need to have a separate datasources, now we consume one extra connection
rather than sharing it with the pool. This is an unfortunate result of the design, and while one extra
connection doesn't make much of a difference in a single-tenant scenario. It does make so, in a multi-tenant scenario.
But in a multi-tenant scenario, the expectation would already be to use a connection pool. So this
is not a big concern.
https://mattermost.atlassian.net/browse/MM-33818
```release-note
Two new gauge metrics were added:
mattermost_db_replica_lag_abs and mattermost_db_replica_lag_time, both
containing a label of "node", signifying which db host is the metric from.
These metrics signify the replica lag in absolute terms and in the time dimension
capturing the whole picture of replica lag.
To use these metrics, a separate config section ReplicaLagSettings was added
under SqlSettings. This is an array of maps which contain three keys: DataSource,
QueryAbsoluteLag, and QueryTimeLag. Each map entry is for a single replica instance.
DataSource contains the DB credentials to connect to the replica instance.
QueryAbsoluteLag is a plain SQL query that must return a single row of which the first column
must be the node value of the Prometheus metric, and the second column must be the value of the lag.
QueryTimeLag is the same as above, but used to measure the time lag.
As an example, for AWS Aurora instances, the QueryAbsoluteLag can be:
select server_id, highest_lsn_rcvd-durable_lsn as bindiff from aurora_global_db_instance_status() where server_id=<>
and QueryTimeLag can be:
select server_id, visibility_lag_in_msec from aurora_global_db_instance_status() where server_id=<>
For MySQL Group Replication, the absolute lag can be measured from the number of pending transactions
in the applier queue:
select member_id, count_transaction_remote_in_applier_queue FROM performance_schema.replication_group_member_stats where member_id=<>
Overall, what query to choose is left to the administrator, and depending on the database and need, an appropriate
query can be chosen.
```
* Trigger CI
* Fix tests
* address review comments
* Remove t.Parallel
It was spawning too many connections,
and overloading the docker container.
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* Add feature flag for apps
* Update default to false
* Add plugin version Feature Flag
* Fix typo
* Only force shutdown, and leave the enable status dependant on the user (defaulting to enable)
* Remove unneeded tracking of status
* Handle plugin init on startup for locally installed plugin
* MM-33893: Disable TCP_NO_DELAY for websocket connections
In very large installations, websocket messages cause too much
traffic congestion by sending too small packets and thereby cause
a drop in throughput.
To counter this, we disable the TCP_NO_DELAY flag for websocket
connections. This has shown to give noticeable improvements in
load tests.
We wrap this in a feature flag for now to let it soak in Community
first.
```release-note
NONE
```
https://mattermost.atlassian.net/browse/MM-33893
* fix gorilla specific conn
* Add field to config model
Config option for disabling link previews for given domains.
* Refactor functions and corresponding tests
* Expand logic for link preview
Newly added isLinkAllowedForPreview function determines whether a link
should display a preview. It gets corresponding config values consisting
of comma separated domain values, normalizes them and checks for
matches.
* Create tests for link preview restriction
* Fix formatting issue
* Add test cases where images are expected
* Apply suggestions from code review
Co-authored-by: Ibrahim Serdar Acikgoz <serdaracikgoz86@gmail.com>
* Apply remaining code suggestions
* Add RestrictLinkPreviews value to telemetries
Co-authored-by: Ibrahim Serdar Acikgoz <serdaracikgoz86@gmail.com>
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
Co-authored-by: Harrison Healey <harrisonmhealey@gmail.com>
* Adding search files api
* Fixing golangci-lint
* Adding bulk-indexing and improving a bit the name indexing for bleve and elasticsearch
* Add content extraction config migration
* Fixing a problem with document extraction
* Unapplying certain changes moved to other PR
* Fixing tests
* Making extract content app migration a private method
* Addressing PR review comments
* Addressing PR review comments
* Adding feature flag
* Removing debug string
* Fixing imports
* Fixing linting errors
* Do not migrate the config if the feature flag is not enabled
* Fix tests
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* [MM-32691] - Send email when the users are not able to join a workspace
* Revert "[MM-32691] - Send email when the users are not able to join a workspace"
This reverts commit 3c11643c7c6867d992743a15675f0099df4f4e4d.
* Feeback impl-1
* use date as key for rate limiting
* Fix tests
* Translations
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* Adding bulk-indexing and improving a bit the name indexing for bleve and elasticsearch
* Update services/searchengine/bleveengine/bleve.go
Co-authored-by: Mario de Frutos Dieguez <mario@defrutos.org>
* Update store/sqlstore/file_info_store.go
Co-authored-by: Mario de Frutos Dieguez <mario@defrutos.org>
* Update store/sqlstore/file_info_store.go
Co-authored-by: Mario de Frutos Dieguez <mario@defrutos.org>
* Adding tests requested in the PR review
* fixing tests
* Adding a feature flag to avoid indexing files before the feature is released
* Fixing i18n
Co-authored-by: Mario de Frutos Dieguez <mario@defrutos.org>
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* Isolate more the mfa service
* Simplifing the mfa service
* Removing channels and adding waitgroup
* Migrating mfa service to regular errors
* Fixing tests
* i18n extract
* Addressing PR review comments
* Removing unneeded struct
* Create the system console setting and send to webapp
* MI-1145: Add custom status APIs
* MI-1145 Add slash commands to set and clear status
* Add validation for custom status API
* Trim custom status message
* Code refactoring
- Run gofmt
- Rename constants
* Remove sendUserUpdated webhook event
* Fix recent custom status length
* Update error conditions
* Disable /status slash command when config setting is off
* MI-1155: Create the feature flag for custom status APIs and slash commands
* Move recent custom statuses to user preferences (#7)
* Move recent custom statuses to user preferences
* Code refactoring and feedback changes
* Update slash command text and emoji regex
* Make the custom status feature flag off by default
* Update SetCustomStatus, handle recents not set better
* Update status codes
* Update slash command handling
* Add telementry settings
* Fix i18n order
* Revert "Fix i18n order"
This reverts commit 499f7eaca8180336f5bcca360cc0133365899e08.
* Update i18n strings
