MM-8622: Improved plugin error reporting (#8737)

* allow `Wait()`ing on the supervisor

In the event the plugin supervisor shuts down a plugin for crashing too
many times, the new `Wait()` interface allows the `ActivatePlugin` to
accept a callback function to trigger when `supervisor.Wait()` returns.
If the supervisor shuts down normally, this callback is invoked with
a nil error, otherwise any error reported by the supervisor is passed
along.

* improve plugin activation/deactivation logic

Avoid triggering activation of previously failed-to-start plugins just
becase something in the configuration changed. Now, intelligently
compare the global enable bit as well as the each individual plugin's
enabled bit.

* expose store to manipulate PluginStatuses

* expose API to fetch plugin statuses

* keep track of whether or not plugin sandboxing is supported

* transition plugin statuses

* restore error on plugin activation if already active

* don't initialize test plugins until successfully loaded

* emit websocket events when plugin statuses change

* skip pruning if already initialized

* MM-8622: maintain plugin statuses in memory

Switch away from persisting plugin statuses to the database, and
maintain in memory instead. This will be followed by a cluster interface
to query the in-memory status of plugin statuses from all cluster nodes.

At the same time, rename `cluster_discovery_id` on the `PluginStatus`
model object to `cluster_id`.

* MM-8622: aggregate plugin statuses across cluster

* fetch cluster plugin statuses when emitting websocket notification

* address unit test fixes after rebasing

* relax (poor) racey unit test re: supervisor.Wait()

* make store-mocks

model/plugin_status.go

@@ -0,0 +1,44 @@
+// Copyright (c) 2015-present Mattermost, Inc. All Rights Reserved.
+// See LICENSE.txt for license information.
+
+package model
+
+import (
+	"encoding/json"
+	"io"
+)
+
+const (
+	PluginStateNotRunning          = 0
+	PluginStateStarting            = 1
+	PluginStateRunning             = 2
+	PluginStateFailedToStart       = 3
+	PluginStateFailedToStayRunning = 4
+	PluginStateStopping            = 5
+)
+
+// PluginStatus provides a cluster-aware view of installed plugins.
+type PluginStatus struct {
+	PluginId      string `json:"plugin_id"`
+	ClusterId     string `json:"cluster_id"`
+	PluginPath    string `json:"plugin_path"`
+	State         int    `json:"state"`
+	IsSandboxed   bool   `json:"is_sandboxed"`
+	IsPrepackaged bool   `json:"is_prepackaged"`
+	Name          string `json:"name"`
+	Description   string `json:"description"`
+	Version       string `json:"version"`
+}
+
+type PluginStatuses []*PluginStatus
+
+func (m *PluginStatuses) ToJson() string {
+	b, _ := json.Marshal(m)
+	return string(b)
+}
+
+func PluginStatusesFromJson(data io.Reader) PluginStatuses {
+	var m PluginStatuses
+	json.NewDecoder(data).Decode(&m)
+	return m
+}