MM-50427: Make MM survive DB replica outage (#22888)

We monitor the health of DB replicas, and on a fatal error,
take them out of the pool.

On a separate goroutine, we keep pinging the unhealthy replicas,
and on getting a good response back, we add them back to the pool.

https://mattermost.atlassian.net/browse/MM-50427

```release-note
Mattermost is now resilient against DB replica outages and will
dynamically choose a replica if it's alive.

Also added a config parameter ReplicaMonitorIntervalSeconds
whose default value is 5. This controls how frequently unhealthy
replicas will be monitored for liveness check.
```

Co-authored-by: Mattermost Build <build@mattermost.com>
Этот коммит содержится в:
Agniva De Sarker
2023-04-19 17:03:18 +05:30
коммит произвёл yasserfaraazkhan
родитель bab2620b59
Коммит 7216dcf367
13 изменённых файлов: 245 добавлений и 121 удалений

Просмотреть файл

@@ -522,6 +522,7 @@ func (ts *TelemetryService) trackConfig() {
"query_timeout": *cfg.SqlSettings.QueryTimeout,
"disable_database_search": *cfg.SqlSettings.DisableDatabaseSearch,
"migrations_statement_timeout_seconds": *cfg.SqlSettings.MigrationsStatementTimeoutSeconds,
"replica_monitor_interval_seconds": *cfg.SqlSettings.ReplicaMonitorIntervalSeconds,
})
ts.SendTelemetry(TrackConfigLog, map[string]any{