https://mattermost.atlassian.net/browse/MM-39215
```release-note
We now have Grafana metrics for DB connection metrics.
They are:
max_open_connections
open_connections
in_use_connections
idle_connections
wait_count_total
wait_duration_seconds_total
max_idle_closed_total
max_idle_time_closed_total
max_lifetime_closed_total
```
* [MM-47648] Handler for check if a subscription is expandable
* Mock cloud regenerate
* PR Webapp test purpose only
* [MM-47648] Refactor and tests for expand stats for a subscription
* Fix typos
* style fix
* has to be revert
* CWS URL to Spinwick
* new spinwick url
* target portal test
* Generate token instead of send licenseId
* Rename
* Rename methods
* Revert public key and cws url
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* Self-hosted admins can purchase licenses in-app when `ServiceSettings,SelfHostedPurchase` is true (the default)
* Content Security Policy enables loading assets from `js.stripe.com/v3` when `ServiceSettings.SelfHostedPurchase` is true (the default).
* Add `hosted_customer` API subpath
* Add status of SelfHostedPurchase to telemetry config report.
* Support showing admins self-hosted invoices when `ServiceSettings.SelfHostedPurchase` is true (the default)
Co-authored-by: Conor Macpherson <116016004+ConorMacpherson@users.noreply.github.com>
* created selfhosted endpoint
* make autogenerated mocks
* add test
* add user id
* remove user id requirement
* add tests
* send user id with request
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* Call Cloud HandleLicenseChange when license is changed in a cloud context
* Remove UpdateSubscriptionFromHook as its no longer necessary
* Update mocks
* Remove another reference
* Remove translation
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* MM-43666 - request bussiness email
* add the newValidEmail information
* revert ff model changes
* remove unnecessary pointer and fix test spelling
* fix unit tests and add a new one for when passing a valid email
* fix go lint
Co-authored-by: Pablo Velez Vidal <pablo.velez@mattermost.com>
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* MM-43529 - start freemium trial
* change the method to put
* Add unit testing for the request-trial-endpoint
* use correct require
* add the translation texts
* fix texts
* remove unnecessary log
* change response status code to forbidden when non cloud
Co-authored-by: Pablo Velez Vidal <pablo.velez@mattermost.com>
* Working on refactoring jobs service
* Making more consistent with the previous existing code
* Remove no longer needed functions
* Making a base PeridicScheduler to use it in most of the schedulers implementations
* Removing accidental complexity from on of the jobs
* Removing accidental complexity from expirynotify
* Fixing compilation from previous commit
* Remove accidental complexity from the export_delete job
* Simplifying the workers by making a reusable worker
* Using simple worker for export_delete job
* Simpliying export process job
* Simpliying extract content job
* Simpliying import delete job
* Simpliying import process job
* Simpliying product noticies job
* Simpliying fix crt channel unreads job (only removing the uneeded register function)
* Simpliying migrations job (only removing the uneeded register function)
* fixup
* Simpliying plugins job (only removing the uneeded register function)
* Simpliying bleve indexing job (only removing the uneeded register function)
* Simpliying resend invitation email job (only removing the uneeded register function)
* Fixing tests
* Simplifying migration tests infrastructure
* Adding missed license to files
* Adding an empty file to imports package to ensure this package exist even without enterprise repo
* Regenerating einterfaces mocks
* Adding missed license to files
* Updating i18n/en.json file
* help fixing enterprise tests compilation
* Adding new DailyScheduler
* Fixing typo and changing the waitTime type for periodic sechduler
* Making the daily scheduler more generic
* Adding comments to clarify not used parameters in interface scheduler interface implementations
* Using merror to handle multiple errors in jobs workers
* Fixing linter errors
* Addressing PR review comments
* Reverting go.tools.mod changes
* Removing the static check for worker type in the model (moving it to the insertion of new jobs
* Moving migrations job to the jobs directory
* Fixing (and improving a bit) tests
* Apply suggestions from code review
Co-authored-by: Doug Lauder <wiggin77@warpmail.net>
* Fixing enterprise tests
* Removing unneeded InitWorkers/InitSchedulers calls
* Fix expirenotify job when error happens
* Fixing govet errors
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
Co-authored-by: Doug Lauder <wiggin77@warpmail.net>
* [MM-40407] - Do not show the Renew Now if the license id does not exist in the portal
* improvements
* impl mock
* fix translations
* feedback impl
* fix typo
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* Change internal getProducts to filter on a set of families instead of a single family.
* Extend /products endpoint to be able to return any of the current four sets of product families.
* Use include_legacy query param to decide which cloud products to request.
* MM-34434: Added 'is_trial' boolean to all trial license requests and to the License struct.
* MM-34434: Generalized the concept of a license request.
* MM-34434: Verifies JSON field of license instance is set.
* MM-34434: Added missing client param.
* MM-34434: Added some tests of the request trial API endpoint.
* MM-34434: Removed comment.
* fix broken test (#17348)
* Add missing wrapped errors (#17339)
* Improve document extraction and including a document extraction command (#17183)
* Add extract documents content command
* Adding the extraction command and making the pure go pdf library as secondary option
* Improving the memory usage and docextractor interface
* Enable content extraction by default in all the instances
* Tiny improvement on archive indexing
* Adding App interface generation and the opentracing layer
* Fixing linter errors
* Addressing PR review comments
* Addressing PR review comments
* Update en.json (#17356)
Automatic Merge
* adding new feature flag (#17308)
Co-authored-by: Benjamin Cooke <benjamincooke@Benjamins-MacBook-Pro.local>
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* Bump no_output_timeout to 2 hours (#17358)
* log invalid username (#17345)
Automatic Merge
* MM-34434: Added missing client param.
MM-34434: Added some tests of the request trial API endpoint.
MM-34434: Removed comment.
* MM-34434: Switched to a hard-coded true value.
* MM-34434: Reverts test change.
* MM-34434: Removes unnecessary field.
* MM-34434: Tests that is_trial is hard-coded by TrialLicenseRequest.
* MM-34434: Removed accidental commit.
* MM-34434: Removes unnecessary is_trial key from JSON payload.
* MM-34434: Reverts to old pointer receiver variable name.
* MM-34434: Removes test.
* #MM-34437 Initialized license service
* ##MM-34437 Verified at all points if server is trial elligible
* WIp
* #MM-34437 removed unused commented code
* MM-34437 make a log less severe
* #MM-34437 generated einterface mocks
* #MM-34437 added license on new file
* #MM-34437 removed unused translation
* #MM-34437 some refactoring
* Update api4/license.go
* Update api4/license.go
* #MM-34437 made a variable name consistent
* #MM-34437 Added mocks for lince validator
* #M--34437 Added license validator test framework
* #MM-34437 Renamed isTrial method to isTrialLicense to avoid conflict with newlya dded field
* #M--34437 Allowed sales-sanctioned trials
* #MM-34437 fixed trial license API tests
* Added tests for add license API
* #MM-34437 fixed ValidateLicense test
* #MM-34437 Added util tests
* #MM-34437 using NoError for checking no error
* #MM-34437 using NoError for checking no error
* Added dummy piblic key for testing
* Fixed tests
* #MM-34437 udpaetd trial license URL for testing
* #MM-34437 adjusted times for licences generated through admin portal
* Reverted test-only changes
Co-authored-by: Martin Kraft <martin@upspin.org>
Co-authored-by: Hossein <hahmadia@users.noreply.github.com>
Co-authored-by: Ben Schumacher <ben.schumacher@mattermost.com>
Co-authored-by: Jesús Espino <jespinog@gmail.com>
Co-authored-by: Amy Blais <amy_blais@hotmail.com>
Co-authored-by: Ben Cooke <benkcooke@gmail.com>
Co-authored-by: Benjamin Cooke <benjamincooke@Benjamins-MacBook-Pro.local>
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
Co-authored-by: Agniva De Sarker <agnivade@yahoo.co.in>
Co-authored-by: Max Erenberg <max.erenberg@mattermost.com>
* Add a controller for changing subscription
* Changes for PR
* Another change
* Fix pipeline
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* check authdata for different format
* add mocks, fix lint
* rename function
* update auth-data with new format
* update assert
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* add request context
* move initialialization to server
* use app interface instead of global app functions
* remove app context from webconn
* cleanup
* remove duplicated services
* move context to separate package
* remove finalize init method and move content to NewServer function
* restart workers and schedulers after adding license for tests
* reflect review comments
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* add includeRemovedMembers flag
* fix API call in client4.go
* remove check for 'since'
* add comments
* run make app-layers
* re-run CI tests
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* MM-34878: Add metrics for websocket reconnects
Added two new metrics to track successful websocket drain
and dead queue misses.
```release-note
NONE
```
https://mattermost.atlassian.net/browse/MM-34878
* remove TODO line
* Add basic support for plugin intra-cluster communication
* Some renaming for added clarity
* Allow sending cluster event to specific nodes
* Improve naming and documentation
* Improve logging
Remote Cluster Service
- provides ability for multiple Mattermost cluster instances to create a trusted connection with each other and exchange messages
- trusted connections are managed via slash commands (for now)
- facilitates features requiring inter-cluster communication, such as Shared Channels
Shared Channels Service
- provides ability to shared channels between one or more Mattermost cluster instances (using trusted connection)
- sharing/unsharing of channels is managed via slash commands (for now)
* MM-33818: Add replica lag metric
We add two new metrics for monitoring replica lag:
- Monitor absolute lag based on binlog distance/transaction queue length.
- Monitor time taken for the replica to catch up.
To achieve this, we add a config setting to run a user defined SQL query
on the database.
We need to specify a separate datasource field as part of the config because
in some databases, querying the replica lag value requires elevated credentials
which are not needed for usual running of the application, and can even be a security risk.
Arguably, a peculiar part of the design is the requirement of the query output to be in a (node, value)
format. But since from the application, the SQL query is a black box and the user can set any query
they want, we cannot, in any way templatize this.
And as an extra note, freno also does it in a similar way.
The last bit is because we need to have a separate datasources, now we consume one extra connection
rather than sharing it with the pool. This is an unfortunate result of the design, and while one extra
connection doesn't make much of a difference in a single-tenant scenario. It does make so, in a multi-tenant scenario.
But in a multi-tenant scenario, the expectation would already be to use a connection pool. So this
is not a big concern.
https://mattermost.atlassian.net/browse/MM-33818
```release-note
Two new gauge metrics were added:
mattermost_db_replica_lag_abs and mattermost_db_replica_lag_time, both
containing a label of "node", signifying which db host is the metric from.
These metrics signify the replica lag in absolute terms and in the time dimension
capturing the whole picture of replica lag.
To use these metrics, a separate config section ReplicaLagSettings was added
under SqlSettings. This is an array of maps which contain three keys: DataSource,
QueryAbsoluteLag, and QueryTimeLag. Each map entry is for a single replica instance.
DataSource contains the DB credentials to connect to the replica instance.
QueryAbsoluteLag is a plain SQL query that must return a single row of which the first column
must be the node value of the Prometheus metric, and the second column must be the value of the lag.
QueryTimeLag is the same as above, but used to measure the time lag.
As an example, for AWS Aurora instances, the QueryAbsoluteLag can be:
select server_id, highest_lsn_rcvd-durable_lsn as bindiff from aurora_global_db_instance_status() where server_id=<>
and QueryTimeLag can be:
select server_id, visibility_lag_in_msec from aurora_global_db_instance_status() where server_id=<>
For MySQL Group Replication, the absolute lag can be measured from the number of pending transactions
in the applier queue:
select member_id, count_transaction_remote_in_applier_queue FROM performance_schema.replication_group_member_stats where member_id=<>
Overall, what query to choose is left to the administrator, and depending on the database and need, an appropriate
query can be chosen.
```
* Trigger CI
* Fix tests
* address review comments
* Remove t.Parallel
It was spawning too many connections,
and overloading the docker container.
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* app/server: add pprof endpoint
* reflect review comments
* make metrics link conditional
* app/server: add metrics server setup to licence listeners
* refactor a bit
* trigger CI
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* Include user data in the cloud endpoints
Those headers will include the user ID and Email so we can use them
in CWS
* Removed AppError from enterprise/cloud
We're removing the AppError from all the places that don't belong
to the app or api4 packages.
* Remove unused i18n strings
* Move it to the server init of enterprise
Also moved the initialization of the enterprise part in the server after the store is initialized
* Initialize after the store is set in NewServer
The ideal way to do it should be to move the initEnterprise call after
the store is set but that would lead to undesired side-effects so we
initialize the cloud part alone.
Signed-off-by: Mario de Frutos <mario@defrutos.org>
* init commit
* clean up the code
* make mocks
* fix translations
* mocks and lint fixes
* add tests
* little fixes
* Update i18n/en.json
Co-authored-by: Scott Bishel <scott.bishel@mattermost.com>
* Update i18n/en.json
Co-authored-by: Scott Bishel <scott.bishel@mattermost.com>
* Update i18n/en.json
Co-authored-by: Scott Bishel <scott.bishel@mattermost.com>
* Update i18n/en.json
Co-authored-by: Scott Bishel <scott.bishel@mattermost.com>
* Update i18n/en.json
Co-authored-by: Scott Bishel <scott.bishel@mattermost.com>
* Address Comments
* fix i18n
* update api endpoint
* add enable file and file level for conditional show of banner
* Address Comments
* Make it more clear about returns
* Create zip file utility function
* update en.json
* address comments
* write tests
* check for data in test
* remove warning string
* Correct expected and actual
* set database through environment variables
* reset environment variable at end of test
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
Co-authored-by: Scott Bishel <scott.bishel@mattermost.com>
