MM-50427: Make MM survive DB replica outage (#22888)

We monitor the health of DB replicas, and on a fatal error,
take them out of the pool.

On a separate goroutine, we keep pinging the unhealthy replicas,
and on getting a good response back, we add them back to the pool.

https://mattermost.atlassian.net/browse/MM-50427

```release-note
Mattermost is now resilient against DB replica outages and will
dynamically choose a replica if it's alive.

Also added a config parameter ReplicaMonitorIntervalSeconds
whose default value is 5. This controls how frequently unhealthy
replicas will be monitored for liveness check.
```

Co-authored-by: Mattermost Build <build@mattermost.com>

server/channels/einterfaces/metrics.go

@@ -13,6 +13,7 @@ import (
 type MetricsInterface interface {
 	Register()
 	RegisterDBCollector(db *sql.DB, name string)
+	UnregisterDBCollector(db *sql.DB, name string)
 
 	IncrementPostCreate()
 	IncrementWebhookPost()

server/channels/einterfaces/mocks/MetricsInterface.go

@@ -319,6 +319,11 @@ func (_m *MetricsInterface) SetReplicaLagTime(node string, value float64) {
 	_m.Called(node, value)
 }
 
+// UnregisterDBCollector provides a mock function with given fields: db, name
+func (_m *MetricsInterface) UnregisterDBCollector(db *sql.DB, name string) {
+	_m.Called(db, name)
+}
+
 type mockConstructorTestingTNewMetricsInterface interface {
 	mock.TestingT
 	Cleanup(func())