This PR builds up on the pass-through DB driver to a fully functioning DB driver implementation via our RPC layer.
To keep things separate from the plugin RPC API, and have the ability to move fast with changes, a separate field Driver is added to MattermostPlugin. Typically the field which is required to be compatible are the API and Helpers. It would be well-documented that Driver is purely for internal use by Mattermost plugins.
A new Driver interface was created which would have a client and server implementation. Every object (connection, statement, etc.) is created and added to a map on the server side. On the client side, the wrapper structs hold the object id, and communicate via the RPC API using this id.
When the server gets the object id, it picks up the appropriate object from its map and performs the operation, and sends back the data.
Some things that need to be handled are errors. Typical error types like pq.Error and mysql.MySQLError are registered with encoding/gob. But for error variables like sql.ErrNoRows, a special integer is encoded with the ErrorString struct. And on the cilent side, the integer is checked, and the appropriate error variable is returned.
Some pending things:
- Context support. This is tricky. Since context.Context is an interface, it's not possible to marshal it. We have to find a way to get the timeout value from the context and pass it.
- RowsColumnScanType(rowsID string, index int) reflect.Type API. Again, reflect.Type is an interface.
- Master/Replica API support.
* Replace config generator
* Cleanup
* Some renaming and docs additions to add clarity
* Cleanup logging related methods
* Cleanup emitter
* Fix TestDefaultsGenerator
* Move feature flags synchronization logic out of config package
* Remove unnecessary util functions
* Simplify load/set logic
* Refine semantics and add some test to cover them
* Remove unnecessary deep copies
* Improve logic further
* Fix license header
* Review file store tests
* Fix test
* Fix test
* Avoid additional write during initialization
* More consistent naming
* Update app/feature_flags.go
Co-authored-by: Christopher Speller <crspeller@gmail.com>
* Update config/store.go
Co-authored-by: Christopher Speller <crspeller@gmail.com>
* Update config/store.go
Co-authored-by: Christopher Speller <crspeller@gmail.com>
* Update config/store.go
Co-authored-by: Ibrahim Serdar Acikgoz <serdaracikgoz86@gmail.com>
* Make ConfigStore.Set() return both old and new configs
* Implement config diff function
* Make app.SaveConfig return previous and current configs
* Add config diff to audit record
* Fix returned configs
* Include high level test
* Move FF synchronizer to its own package
* Remove unidiomatic use of sync.Once
* Add some comments
* Rename function
* More comment
* Save config diff in audit record for local endpoints
* Enable audit for config set/reset commands
* Improve tests output
Co-authored-by: Christopher Speller <crspeller@gmail.com>
Co-authored-by: Ibrahim Serdar Acikgoz <serdaracikgoz86@gmail.com>
* add request context
* move initialialization to server
* use app interface instead of global app functions
* remove app context from webconn
* cleanup
* remove duplicated services
* move context to separate package
* remove finalize init method and move content to NewServer function
* restart workers and schedulers after adding license for tests
* reflect review comments
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
A race happens when we try to enable plugins
in an HA environment. There is no loss in functionality
here, but it's a timing bug.
There are two bugs here.
The first one is a case of nested config listeners. a.InitPlugins
is called in InitServer, and as well as a config listener. And that
calls a.SyncPluginsActiveState(). But inside a.InitPlugins,
there is yet another config listener which again calls
a.SyncPluginsActiveState(). The first fix is to simply not call
the method again from the nested listener.
The second bug happens because the config changed message is sent
across the cluster only after saving the config locally
in the store. To fix this, we simply change GetPluginStatus
to not fail and return empty status when other nodes don't have plugins
enabled.
https://mattermost.atlassian.net/browse/MM-34419
```release-note
Fix a race condition where enabling plugins would result
in spurious errors in the logs.
```
* Add feature flag for apps
* Update default to false
* Add plugin version Feature Flag
* Fix typo
* Only force shutdown, and leave the enable status dependant on the user (defaulting to enable)
* Remove unneeded tracking of status
* Handle plugin init on startup for locally installed plugin
* Don't downgrade plugins with feature flag for on-prem
* Update app/plugin.go
Co-authored-by: Christopher Poile <cpoile@gmail.com>
* Short circut for plugin doesn't exist.
Co-authored-by: Christopher Poile <cpoile@gmail.com>
* format using `goimports -local github.com/mattermost/mattermost-server/v5 -w`
* added goimports lint check to .golangci.yml
* format using `goimports -local github.com/mattermost/mattermost-server/v5 -w` for a corner case
* make app-layers, *-mocks and store-layers for ci check
Co-authored-by: Mahmudul Haque <mahmudulhaque@protonmail.com>
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* Installing plugins specified by feature flags using the marketplace.
* Switch back to using getplugins client.
* Respect disabling automatic installation of pluings.
* pluginid -> plugin_id
* Debug logs for enable plugin error
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* Start plugins concurrently
* Determine which plugins need to be activated/deactivated before changing their state
* Test activation/deactivation of plugins in TestSyncPluginsActiveState
* Change test comments
* remove unneeded temp var
* Removing some other fake apps
* More FakeApp removed
* Removing entirely FakeApp
* Fixing some tests
* Fixing get Cluster id from get plugin status
* Fixing failing tests
* Fixing tests
* Fixing test initialization for web
* Fixing InitServer for server tests
* Fixing InitServer for server tests
* Reverting go.sum and go.mod
* Removing unneded HTMLTemplates function in App layer
* Moving back some functions to its old place to easy the review
* Moving back some functions to its old place to easy the review
* Using the last struct2interface version
* Generating store layers
* Fixing merge problems
* Addressing PR comments
* Small fix
* Fixing app tests build
* Fixing tests
* fixing tests
* Fix tests
* Fixing tests
* Fixing tests
* Fixing tests
* Moving license to server struct
* Adding some fixes to the test compilation
* Fixing cluster and some jobs initialization
* Fixing some license tests compilation problems
* Fixing recursive cache invalidation
* Regenerating app layers
* Fix test compilation
Co-authored-by: mattermod <mattermod@users.noreply.github.com>
There was a race in (*App).SyncPlugins where if the same plugin
existed in availablePlugins and pluginSignaturePathMap, then
we would try to add/remove at the same time.
This would lead to a possible removal / addition of a plugin directory
or even unable to remove a directory because it was already in use.
We fix this by first finishing the removal of availablePlugins
before syncing it with the file store.
* [MM-20941] Parallelize plugin loading on startup
* [MM-20941] Parallelize plugin SyncPlugins
* [MM-20941] Make activation and deactivation of plugin concurrent
* [MM-20941] Remove uncessary optimization
This reverts commit aad2680aec843125bf72d8d2406869f4948dba42.
[MM-20941] Remove uncessary wait in SyncPlugins
[MM-20941] Parallelize processing of prepackaged plugins
* [MM-20941] Fix lint issue
* [MM-20941] Fix bug for getting the number of plugins to process
* [MM-20941] Use buffered channel and fix return issue in prepackaged plugin processing
* [MM-20941] Implement feedback
* [MM-20941] Fix looping over channel for plugins
* [MM-20941] Retain signature of process plugin
* [MM-20941] Fix function documentation
* [MM-20941] Fix bug to pass the plugin path in anonymous function
* [MM-20941] Rename variable
* [MM-20941] Make minor changes in variable naming
Co-authored-by: mattermod <mattermod@users.noreply.github.com>
* MM-21103: change plugin signature path
Save as `<plugin_id>.tar.gz.sig` instead of `<plugin_id>.sig`. The latter was a relic of the previous design to support multiple plugin signatures, but now creates an inconsistency with how the original source files were supplied as `<some_name>.tar.gz` and `<some_name>.tar.gz.sig`.
Fixes: https://mattermost.atlassian.net/browse/MM-21103
* relax signature matches to avoid assuming signatures always exist
* Consistent license message for all the go files
* Fixing the last set of unconsistencies with the license headers
* Addressing PR review comments
* Fixing busy.go and busy_test.go license header
* MM-17549: use StatusNotFound when deleting plugin (#12983)
Trying to delete a plugin that does not exist should fail with a 404, not a 400.
Relates-to: https://mattermost.atlassian.net/browse/MM-17549
* MM-19630: marketplace: model ReleaseNotesURL (#13083)
Automatic Merge
* MM-20065: allow prepackaged plugin upgrade (#13076)
* MM-20065: allow prepackaged plugin upgrade
When locally installing prepackaged plugins, skip if a plugin exists with the same id and is the same or a newer version.
This is effectively a "poor man's" rework of prepackaged plugins to allow upgrade of prepackaged plugins via the marketplace. The larger plan to rework prepackaged plugins was deferred from v5.18.
Fixes: https://mattermost.atlassian.net/browse/MM-20065
* eliminate unnecessary installPlugin
* fix TestPluginSync defaults to match minio
* cleanExistingBundles
* close prepackaged filereader
* simplify
* MM-18540 - Demo plugin crashes on server shutdown
* MM-18540 - Removed unncessary lock statements
* Added comments, logging a warning if PluginsEnv is updated while shutting down plugins
* Don't clean up PluginsEnv if another Env is detected
* Changing warn to debug
* Revert "Changing warn to debug"
This reverts commit 46f20ab21eeabb01d07f53e02d850cd6d9b2b837.
* MM-17149 - Extend config.json for marketplace settings (#11933)
* MM-17149 - Extend config.json for marketplace settings
* Renamed MarketplaceUrl, tracking default marketplace url
* Added EnableMarketplace to the client config
* Revert "Added EnableMarketplace to the client config"
This reverts commit 0f982c4c661c2cd9bb96264e9a01a2363c40d9c5.
* MM-17149 - Added EnableMarketplace to the client config (#11958)
* Added EnableMarketplace to the client config
* Moved EnableMarketplace setting out of limited client configuration
* MM-17150, MM-17545, MM-18100 - Implement GET /api/v4/plugins/m… (#11977)
* MM-17150 - Implement GET /api/v4/plugins/marketplace proxying upstream
MM-17545 - Merge locally installed plugins into GET /api/v4/plugins/marketplace
* Replaced MarketplacePluginState with Installed
* Setting InstalledVersion instead of Installed
* marketplace client setting per_page if non zero
* Creating insecure client for marketplace url
* Fixed trailing slash for default marketplace url
* Adding filtering
* Fixed function names
* Renamed Manifest() to GetManifest(), added godoc for BaseMarketplacePlugin
* Handling plugin.ErrNotFound correctly
* Checking err == nil instead when a plugin is installed
* MM-18450 - Local-only plugin search (#12152)
* MM-17846: plugin icons (#12157)
* MM-17846: add support for plugin icons
Extend the model definitions to support plugin icons from the marketplace.
* s/IconURL/IconData
* MM-18475 - Converge on snake_case responses from the marketplace (#12179)
* MM-18520 - MM-Server should forward server version to marketplace server (#12181)
* Renamed request to filter client4.GetMarketplacePlugins
* Renamed request to filter
* Guarding against bad marketplace server response
* MM-17087 - Disable plugin on removal
* Updated documentation
* Got reid of notifyPluginEvents
* Updated documentation
* Added plugin installation/activatoin flow as a toplevel go doc in plugin_install.go
* Generating webapp bundle on plugin installation
* Fixed shadowing issue
* Updated doc to include unguarded race condition
* Renamed GenerateWebappBundle
* Added a debug log when peers are not ready to notify
* Updated docs
* Removed extra line
* MM-16272 - Synchronize plugins across cluster (#11611)
* MM-16272 - Synchronize plugins across cluster
* Adding a test
* MM-16272 - Fixed tests
* MM-16272 - PR feedback
* MM-16270 - Plugin Sync (#11615)
* Initial implementation for plugin synch with file store. WIP
* Removed ListAll implementation. Used ListDirectory and change localstore to be consistent and return all items (files and folders) from directory
* Refactored plugin filestore operations out of main install/remove plugin
* Fixing error handling details
* Changes to use structured logging
* More logging fixes
* Wording and comments improvements
* Error handling and control flow improvements
* Changed managed flag check to use os.stat
* Added file store plugin dir and filename consts
* Replaced FileRead to use a the FileReader in PluginSync
* Minor styling and PR feedback changes
* Minor error handling improvements
* Added unit test for SyncPlugins. Changed SyncPlugins to use plugins environment to list available plugins
* PR Feedback improvements
* Minor err handling fix
* Removing FileStorePath from PluginEventData (#11644)
* Fix plugin path (#11654)
* tweak path, logging
Fix an issue not finding the plugins folder in S3. Tweak logging messages to add additional clarity.
* Removing FileExists check when Syncing plugins. Updated localstore to not return an error when directory does not exist
* PR Feedback
* Install prepackaged plugins locally only (#11656)
* s/uninstall/remove
* Updated ClusterMessage comment
* Updated PluginSync to test against s3 + local storage
