* MM-33818: Add replica lag metric
We add two new metrics for monitoring replica lag:
- Monitor absolute lag based on binlog distance/transaction queue length.
- Monitor time taken for the replica to catch up.
To achieve this, we add a config setting to run a user defined SQL query
on the database.
We need to specify a separate datasource field as part of the config because
in some databases, querying the replica lag value requires elevated credentials
which are not needed for usual running of the application, and can even be a security risk.
Arguably, a peculiar part of the design is the requirement of the query output to be in a (node, value)
format. But since from the application, the SQL query is a black box and the user can set any query
they want, we cannot, in any way templatize this.
And as an extra note, freno also does it in a similar way.
The last bit is because we need to have a separate datasources, now we consume one extra connection
rather than sharing it with the pool. This is an unfortunate result of the design, and while one extra
connection doesn't make much of a difference in a single-tenant scenario. It does make so, in a multi-tenant scenario.
But in a multi-tenant scenario, the expectation would already be to use a connection pool. So this
is not a big concern.
https://mattermost.atlassian.net/browse/MM-33818
```release-note
Two new gauge metrics were added:
mattermost_db_replica_lag_abs and mattermost_db_replica_lag_time, both
containing a label of "node", signifying which db host is the metric from.
These metrics signify the replica lag in absolute terms and in the time dimension
capturing the whole picture of replica lag.
To use these metrics, a separate config section ReplicaLagSettings was added
under SqlSettings. This is an array of maps which contain three keys: DataSource,
QueryAbsoluteLag, and QueryTimeLag. Each map entry is for a single replica instance.
DataSource contains the DB credentials to connect to the replica instance.
QueryAbsoluteLag is a plain SQL query that must return a single row of which the first column
must be the node value of the Prometheus metric, and the second column must be the value of the lag.
QueryTimeLag is the same as above, but used to measure the time lag.
As an example, for AWS Aurora instances, the QueryAbsoluteLag can be:
select server_id, highest_lsn_rcvd-durable_lsn as bindiff from aurora_global_db_instance_status() where server_id=<>
and QueryTimeLag can be:
select server_id, visibility_lag_in_msec from aurora_global_db_instance_status() where server_id=<>
For MySQL Group Replication, the absolute lag can be measured from the number of pending transactions
in the applier queue:
select member_id, count_transaction_remote_in_applier_queue FROM performance_schema.replication_group_member_stats where member_id=<>
Overall, what query to choose is left to the administrator, and depending on the database and need, an appropriate
query can be chosen.
```
* Trigger CI
* Fix tests
* address review comments
* Remove t.Parallel
It was spawning too many connections,
and overloading the docker container.
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* Create basic make commands for configuring golang-migrate
* Showcase full flow with new migrations
* Apply PR suggestions
* Migrate over team members
* Update mocks
* Fix specs
* Move columns that added after table creation onto separate stmts
* Put back gorp table definitions
* Fix issues with golang-migrate that not tracks underlying db driver
* Help prompt after new migration and consistent checksum for bindata
* Put gorp mapping back
* Apply PR suggestiong
* Close migrations after they run
* Add migration file to bindata check
* Updates needed
* Reset store_test
* Add copyright
* Apply PR suggestions
* Fix new circleci check
* Put back upgrade step for backwards comp
* Add store test to test migration directions
* Apply PR suggestions
* Add go-bindata to tools
* Apply PR suggestios
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* Update config.go
* Update telemetry.go
* Update store and store test
* Update settings.go
* use new error code
* Trigger CI
Co-authored-by: Haardik Dharma <dharmahaardik08@gmail.com>
* MM-31063: Change constants to use CamelCase
* store package
* change allcaps to camel case (#16615)
* New tools.mod
Co-authored-by: Ibrahim Serdar Acikgoz <serdaracikgoz86@gmail.com>
* MM-29573: Reuse s3 client amongst all requests
An http.Client is meant as a long-term object to be reused between
consecutive requests. This reduces burden on HTTP servers by reusing TCP connections
and also makes things faster by avoiding the TCP handshake for every single request.
https://mattermost.atlassian.net/browse/MM-29573
* Trigger CI
* Fix i18n
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* Add functionality to update password with password hash
This can be used to update a user's password with a direct password
hash instead of providing the password in plaintext.
* Use test helper for local mode
- Update reference to mmgotool in go.tools.mod
This would automatically be bumped while running i18n-extract.
- We also remove some other redundant lines for mmgotool that went in.
This has to be removed manually because go mod tidy would automatically scan the entire
codebase which is not possible with this file.
* Add a config for MM User Limit
* Adding graceful errors for if an administrator invites people passed their user limit
* Including changed vendor files
* Adding unit test
* Fix a bug
* Push up working tests (Thanks Joram)
* Add more cases, clean up logs in code
* One more case
* Refactoring based on PR comments
* Updating i18n
* Some changes based on PR review
* Remove a comment
* Bring back some translations that were somehow removed
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* Refactor team store to use squirrel
* Fixing or expression
* Refactor team store to use squirrel
* Refactor team store to use squirrel
* Refactor team store to use squirrel
* Refactor team store to use squirrel
* Removing shadow declaration
* Fix error messages
* Improving Update() and Delete() calls
* Update store/sqlstore/team_store.go
Using squirrel to build 'in' clause
Co-authored-by: Miguel de la Cruz <mgdelacroix@gmail.com>
* Add missing translation
Co-authored-by: Dante Pippi <dantepippi@gmai.com>
Co-authored-by: Miguel de la Cruz <mgdelacroix@gmail.com>
Co-authored-by: mattermod <mattermod@users.noreply.github.com>
Co-authored-by: Claudio Costa <cstcld91@gmail.com>
Go 1.14 introduces usage of alternate go.mod files to track
other dependencies which are not related to the main app.
We use this to track all tool dependencies so that everybody uses
the same version of all tools, including CI.
This will prevent version conflicts due to everybody using different
versions of the tools.
And it will not try to upgrade the tool version, every single time,
one runs the tool command.
While here, we also re-generate some filestore mocks which weren't up to date.
Fixes#13088
Co-authored-by: mattermod <mattermod@users.noreply.github.com>
