MM-50427: Make MM survive DB replica outage (#22888)

We monitor the health of DB replicas, and on a fatal error,
take them out of the pool.

On a separate goroutine, we keep pinging the unhealthy replicas,
and on getting a good response back, we add them back to the pool.

https://mattermost.atlassian.net/browse/MM-50427

```release-note
Mattermost is now resilient against DB replica outages and will
dynamically choose a replica if it's alive.

Also added a config parameter ReplicaMonitorIntervalSeconds
whose default value is 5. This controls how frequently unhealthy
replicas will be monitored for liveness check.
```

Co-authored-by: Mattermost Build <build@mattermost.com>
Этот коммит содержится в:
Agniva De Sarker
2023-04-19 17:03:18 +05:30
коммит произвёл GitHub
родитель 5d5c1d90bf
Коммит c34a50a6c7
13 изменённых файлов: 245 добавлений и 121 удалений

Просмотреть файл

@@ -331,7 +331,7 @@ func (h *MainHelper) SetReplicationLagForTesting(seconds int) error {
func (h *MainHelper) execOnEachReplica(query string, args ...any) error {
for _, replica := range h.SQLStore.ReplicaXs {
_, err := replica.Exec(query, args...)
_, err := replica.Load().Exec(query, args...)
if err != nil {
return err
}