We create the basic boilerplate to pass services
to each product. The basic idea is to have a map
containing service names and the services.
The server will pass all services to all products.
But it is on the product, to choose what services
they will actually consume.
Each product will cast the received service to
an interface with the methods that product requires.
https://mattermost.atlassian.net/browse/MM-40813
```release-note
NONE
```
* Move pluginCommands into Channels
We move pluginCommands, pluginCommandsLock
into Channels.
We also move the plugin related route handlers
under Channels and move the init code under
NewChannels. To achieve this, the router initialization
is bumped up.
Along with it, we clean up some App methods
which were just wrappers over Channel methods.
Instead, we call the Channel method directly
to make things more readable and easy to understand.
```release-note
NONE
```
* fix tests
```release-note
NONE
```
* Move plugins under Channels
We move all plugin related fields under *Channels.
This essentially migrates several methods from being under
*Server to under *Channels.
We also move the plugin startup and shutdown code
to be under Channels.Start and Channels.Shutdown.
While here, we remove the getPluginPublicKeyFiles
method which was a one-line method which uselessly
returned an error.
Lastly, we fix the product initialization order which
was incorrect previously. Products are dependent on
the main server.
So startup should be products -> server.
And shutdown should be server -> products.
```release-note
NONE
```
* Added app layer
```release-note
NONE
```
* Incorporate suggestions
```release-note
NONE
```
* Fix PluginHealthCheckJob
We were incorrectly calling to start PluginHealthCheckJob
without initializing the plugins environment.
To fix this, we move the code inside initPlugins right after
the environment is initialized.
To respond to config changes, we call it again from pluginsEnvironment.Shutdown
which gets called from ConfigListener when plugins are disabled. And initPlugins
is anyways called again from ConfigListener which plugins are enabled,
so we can avoid checking for PluginSettings.Enable again in the call.
We also rename the method to better indicate its nature.
During this, we also uncover and fix another bug where disabling
plugins would not shut down plugins at all because we were calling
s.GetPluginsEnvironment() directly which returns nil if plugins
were disabled. The approach we follow is to manually acquire the lock
whenever we need access to the struct ignoring config setting.
We fix that as well.
https://community-daily.mattermost.com/boards/workspace/zyoahc9uapdn3xdptac6jb69ic/285b80a3-257d-41f6-8cf4-ed80ca9d92e5/495cdb4d-c13a-4992-8eb9-80cfee2819a4?c=6ef6178c-3512-4e57-8edd-1d2b66a09c9e
```release-note
NONE
```
* Fix test
```release-note
NONE
```
* Move Channels into App
In this PR, we make Channels as part of App
instead of Server. This is part of the transition period
of moving fields from Server to Channels.
For now, Channels contains Server. So the hierarchy is
App -> Channels -> Server.
And as a first step, we also move httpService to Channels.
```release-note
NONE
```
* Fixing another test
```release-note
NONE
```
* new method
```release-note
NONE
```
This PR builds up on the pass-through DB driver to a fully functioning DB driver implementation via our RPC layer.
To keep things separate from the plugin RPC API, and have the ability to move fast with changes, a separate field Driver is added to MattermostPlugin. Typically the field which is required to be compatible are the API and Helpers. It would be well-documented that Driver is purely for internal use by Mattermost plugins.
A new Driver interface was created which would have a client and server implementation. Every object (connection, statement, etc.) is created and added to a map on the server side. On the client side, the wrapper structs hold the object id, and communicate via the RPC API using this id.
When the server gets the object id, it picks up the appropriate object from its map and performs the operation, and sends back the data.
Some things that need to be handled are errors. Typical error types like pq.Error and mysql.MySQLError are registered with encoding/gob. But for error variables like sql.ErrNoRows, a special integer is encoded with the ErrorString struct. And on the cilent side, the integer is checked, and the appropriate error variable is returned.
Some pending things:
- Context support. This is tricky. Since context.Context is an interface, it's not possible to marshal it. We have to find a way to get the timeout value from the context and pass it.
- RowsColumnScanType(rowsID string, index int) reflect.Type API. Again, reflect.Type is an interface.
- Master/Replica API support.
* Replace config generator
* Cleanup
* Some renaming and docs additions to add clarity
* Cleanup logging related methods
* Cleanup emitter
* Fix TestDefaultsGenerator
* Move feature flags synchronization logic out of config package
* Remove unnecessary util functions
* Simplify load/set logic
* Refine semantics and add some test to cover them
* Remove unnecessary deep copies
* Improve logic further
* Fix license header
* Review file store tests
* Fix test
* Fix test
* Avoid additional write during initialization
* More consistent naming
* Update app/feature_flags.go
Co-authored-by: Christopher Speller <crspeller@gmail.com>
* Update config/store.go
Co-authored-by: Christopher Speller <crspeller@gmail.com>
* Update config/store.go
Co-authored-by: Christopher Speller <crspeller@gmail.com>
* Update config/store.go
Co-authored-by: Ibrahim Serdar Acikgoz <serdaracikgoz86@gmail.com>
* Make ConfigStore.Set() return both old and new configs
* Implement config diff function
* Make app.SaveConfig return previous and current configs
* Add config diff to audit record
* Fix returned configs
* Include high level test
* Move FF synchronizer to its own package
* Remove unidiomatic use of sync.Once
* Add some comments
* Rename function
* More comment
* Save config diff in audit record for local endpoints
* Enable audit for config set/reset commands
* Improve tests output
Co-authored-by: Christopher Speller <crspeller@gmail.com>
Co-authored-by: Ibrahim Serdar Acikgoz <serdaracikgoz86@gmail.com>
* add request context
* move initialialization to server
* use app interface instead of global app functions
* remove app context from webconn
* cleanup
* remove duplicated services
* move context to separate package
* remove finalize init method and move content to NewServer function
* restart workers and schedulers after adding license for tests
* reflect review comments
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
A race happens when we try to enable plugins
in an HA environment. There is no loss in functionality
here, but it's a timing bug.
There are two bugs here.
The first one is a case of nested config listeners. a.InitPlugins
is called in InitServer, and as well as a config listener. And that
calls a.SyncPluginsActiveState(). But inside a.InitPlugins,
there is yet another config listener which again calls
a.SyncPluginsActiveState(). The first fix is to simply not call
the method again from the nested listener.
The second bug happens because the config changed message is sent
across the cluster only after saving the config locally
in the store. To fix this, we simply change GetPluginStatus
to not fail and return empty status when other nodes don't have plugins
enabled.
https://mattermost.atlassian.net/browse/MM-34419
```release-note
Fix a race condition where enabling plugins would result
in spurious errors in the logs.
```
* Add feature flag for apps
* Update default to false
* Add plugin version Feature Flag
* Fix typo
* Only force shutdown, and leave the enable status dependant on the user (defaulting to enable)
* Remove unneeded tracking of status
* Handle plugin init on startup for locally installed plugin
* Don't downgrade plugins with feature flag for on-prem
* Update app/plugin.go
Co-authored-by: Christopher Poile <cpoile@gmail.com>
* Short circut for plugin doesn't exist.
Co-authored-by: Christopher Poile <cpoile@gmail.com>
* format using `goimports -local github.com/mattermost/mattermost-server/v5 -w`
* added goimports lint check to .golangci.yml
* format using `goimports -local github.com/mattermost/mattermost-server/v5 -w` for a corner case
* make app-layers, *-mocks and store-layers for ci check
Co-authored-by: Mahmudul Haque <mahmudulhaque@protonmail.com>
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* Installing plugins specified by feature flags using the marketplace.
* Switch back to using getplugins client.
* Respect disabling automatic installation of pluings.
* pluginid -> plugin_id
* Debug logs for enable plugin error
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* Start plugins concurrently
* Determine which plugins need to be activated/deactivated before changing their state
* Test activation/deactivation of plugins in TestSyncPluginsActiveState
* Change test comments
* remove unneeded temp var
* Removing some other fake apps
* More FakeApp removed
* Removing entirely FakeApp
* Fixing some tests
* Fixing get Cluster id from get plugin status
* Fixing failing tests
* Fixing tests
* Fixing test initialization for web
* Fixing InitServer for server tests
* Fixing InitServer for server tests
* Reverting go.sum and go.mod
* Removing unneded HTMLTemplates function in App layer
* Moving back some functions to its old place to easy the review
* Moving back some functions to its old place to easy the review
* Using the last struct2interface version
* Generating store layers
* Fixing merge problems
* Addressing PR comments
* Small fix
* Fixing app tests build
* Fixing tests
* fixing tests
* Fix tests
* Fixing tests
* Fixing tests
* Fixing tests
* Moving license to server struct
* Adding some fixes to the test compilation
* Fixing cluster and some jobs initialization
* Fixing some license tests compilation problems
* Fixing recursive cache invalidation
* Regenerating app layers
* Fix test compilation
Co-authored-by: mattermod <mattermod@users.noreply.github.com>
There was a race in (*App).SyncPlugins where if the same plugin
existed in availablePlugins and pluginSignaturePathMap, then
we would try to add/remove at the same time.
This would lead to a possible removal / addition of a plugin directory
or even unable to remove a directory because it was already in use.
We fix this by first finishing the removal of availablePlugins
before syncing it with the file store.
* [MM-20941] Parallelize plugin loading on startup
* [MM-20941] Parallelize plugin SyncPlugins
* [MM-20941] Make activation and deactivation of plugin concurrent
* [MM-20941] Remove uncessary optimization
This reverts commit aad2680aec843125bf72d8d2406869f4948dba42.
[MM-20941] Remove uncessary wait in SyncPlugins
[MM-20941] Parallelize processing of prepackaged plugins
* [MM-20941] Fix lint issue
* [MM-20941] Fix bug for getting the number of plugins to process
* [MM-20941] Use buffered channel and fix return issue in prepackaged plugin processing
* [MM-20941] Implement feedback
* [MM-20941] Fix looping over channel for plugins
* [MM-20941] Retain signature of process plugin
* [MM-20941] Fix function documentation
* [MM-20941] Fix bug to pass the plugin path in anonymous function
* [MM-20941] Rename variable
* [MM-20941] Make minor changes in variable naming
Co-authored-by: mattermod <mattermod@users.noreply.github.com>
* MM-21103: change plugin signature path
Save as `<plugin_id>.tar.gz.sig` instead of `<plugin_id>.sig`. The latter was a relic of the previous design to support multiple plugin signatures, but now creates an inconsistency with how the original source files were supplied as `<some_name>.tar.gz` and `<some_name>.tar.gz.sig`.
Fixes: https://mattermost.atlassian.net/browse/MM-21103
* relax signature matches to avoid assuming signatures always exist
