Hubs property has been detected by the race condition detector
as a possible one because we could set the Hubs variable
and also we're spawning goroutines that could ask for that property
This condition is not happening right now because we're setting the
Hubs variable in the server start and after that, we're spawning the
goroutines but the tests are failing because of this and in the future
this could be a problem
* Refactor to use structured logging
* Properly formatted with gofmt
* created interface Cache, but construction of cache is still coupled.
* Implementing cache factories to build caches
* Simple redis implementation without error handling. Keys and values by default are string
* refactor NewLocalCacheLayer to inject cache factory
* Removed redis impl to focus on cache abstraction
* CacheFactory injected on sqlsupplier and saved in Store struct
* remove useless private method
* replace concrete declaration of lru cache to cache abstraction
* discard spaces
* Renamed factory to provider because concrete implementations of factories may hold an state (such as a redis client for example)
* refactor to include all caches in the same package cache and subpackages
* method close cache. This method will be used for concrete implementations that need to release resources (close a connection, etc)
* closing cacheprovider and releasing resources while closing sql store
* fixed merge conflict fail
* gofmt files
* remove unused property from post_store
* naming refactor to avoid stutter. Added godocs on interface
* Store doesnt know anything about the cache and provider. Cache provider will be built after loading config and injected in localCacheLayer
* fixed broken test
* cache provider initialized before RunOldAppInitialization which initializes the localcachelayer
* move statusCache to server to initialize it with the new cache provider
* update terms_service and channel_layer to have new cacheProvider
* gofmt
* Add Connect method to the cache provider
* mock cacheprovider in user_layer_test
Co-authored-by: Ben Schumacher <ben.schumacher@mattermost.com>
Co-authored-by: mattermod <mattermod@users.noreply.github.com>
* MM-8607: Add cluster support for server busy status
MM-8607: Busy.Cluster should be non-exported
MM-8607: rename logging field seconds -> expires_sec
MM-8607: each node clears its own busy flag
MM-8607: ensure unit tests are not sensitive to test machine speed
* MM-8607: chg comment to force CI rebuild
* Consistent license message for all the go files
* Fixing the last set of unconsistencies with the license headers
* Addressing PR review comments
* Fixing busy.go and busy_test.go license header
* MM-8607: add ability to turn off non-critical services under load
* server busy invalid param unit tests
* MM-8607: rename server busy endpoints
* MM-8607: handle case where App not initialized
* MM-8607: additional unit test cases per feedback.
* MM-8607: use decorator to check isbusy when adding endpoint route
* MM-8607: rename endpoints, use struct for json
* Update api4/system.go
Fix misspelled log output
Co-Authored-By: Saturnino Abril <saturnino.abril@gmail.com>
* MM-8607: fix i18n order; max seconds for server busy expiry
* diagnostics_test.go: fix spacing
* diagnostics_test.go: explicitly assert payload
This fails, since the package is currently receiving struct pointers and won't set the MessageId or Timestamp on the corresponding Message.
* MM-18115: fix segment v3 usage
In v5.14, we updated [github.com/segmentio/analytics-go](https://github.com/segmentio/analytics-go) to v3 as part of https://mattermost.atlassian.net/browse/MM-12389. As noted in the [migration guide](https://segment.com/docs/sources/server/go/#migrating-from-v2), the API subtly changed to expect a struct value and not a struct pointer:
```go
// in v2, you would call the `Track` method with a `Track` struct.
client.Track(&track)
// in v3, you would call the `Enqueue` method with a `Track` struct.
// Note that a pointer is not used here.
client.Enqueue(track)
```
Unfortunately, we kept passing a pointer, and the package didn't complain since it only required an interface -- which the pointer to these structs still implemented. Internally, it only checked for the value types, and failed to annotate our payloads with the requisite metadata. Upstream, segment.io accepted the payload, but then discarded it silently.
This has since been reported and fixed in https://github.com/segmentio/analytics-go/pull/146, but isn't yet part of a tagged release of the package.
Fix our code to pass struct values instead.
Fixes: MM-18115
* revert 4e5f6fcfbc
This reverts the configuration setting to disable plugin health checks, preferring instead to retain this functionality for advanced cases. In part, this was driven by the discovery that the health checks were failing on Windows, though that will be addressed separately.
Fixes: MM-16378
* simplify interval handling/logging
Also shutdown the job when plugins are altogether disabled.
The job workers and cluster discovery components require access to the
database during shutdown. To avoid issues, we leave the database open
until the last step of the shutdown process.
Properly handle analytics client life cycle
This change improves the life cycle management of the analytics
client. It does so in two primary ways:
- Move the client from being a shared pointer to belonging to the
Server struct.
- Expose the client.Close() method for proper shutdown of the
client.
* Updated ResetAll function to return object
* Modified the defined interface
* Changed the data type where ResetAll was referrenced
* Refactored code
Changed user's email
* Fixed indentation
* Add config setting to explicitly define which IP headers are trusted
* fix variable shadowing
* Optimize code flow; Add Ratelimit test for header set
* Extend Ratelimit tests
* Add additional unit tests
* Structured logging
* MM-15494 Adding warning log when someone has configured a Mattermost push proxy with HTTP endpoint
* MM-15494 go fmt check
* MM-15494 Making function private
* MM-15494 Updating warning message
* Remove NotificationRegistry table and use structured logging
* Fix ackId for notification sent
* Notification logger at server level
* Remove unused i18n strings
* introduce plugin health check
* implement plugin health check job
* add support for checking pid of plugin process and RPC ping, to determine a plugin's health
* implement restart policy with back-offs
* support "EnableHealthCheck" boolean from config file.
* add tests for supervisor.PerformHealthCheck() and shouldDeactivatePlugin()
* improve error handling. clean up if blocks to be more concise
* [MM-10199] Printing stack traces from http handlers panics in mattermost logs
Previously recovery handlers from gorilla mux was used, which was printing stack traces from panics in stderr
Removing it should print stack traces into logger passed to http.Server, which as of now is created from mlog
Refer #10351
* Logging errors from http server at error level instead of info level
Added test for verifying whether panic gets logged in server logger
* Fixed failing test due to tls error
* Closing temp file for panic log before removing and checking errors for closure and removal
* Added regular log line in panic handler for TestPanicLog to check whether it gets logged
* Added support for PostActions in ephemeral posts
The general approach is that we take all the metadata that DoPostAction
needs to process client DoPostActionRequests, and store it in a
serialized, encrypted Cookie field, in the PostAction struct.
The client then must send it back, and it is then used to process
PostActions as a fallback top the metadata in the database.
This PR adds a new config setting, `ServiceSettings.ActionCookieSecret`.
In a cluster environment it must be the same for all instances.
- Added type PostActionCookie, and a Cookie string to PostAction.
- Added App.AddActionCookiesToPost.
- Use App.AddActionCookiesToPost in api4.createEphemeralPost,
App.SendEphemeralPost, App.UpdateEphemeralPost.
- Added App.DoPostActionWithCookie to process incoming requests with
cookies. For backward compatibility, it prefers the metadata in the
database; falls back to cookie.
- Added plugin.API.UpdateEphemeralPost and plugin.API.DeleteEphemeralPost.
- Added App.encryptActionCookie/App.decryptActionCookie.
* Style
* Fixed an unfortunate typo, tested with matterpoll
* minor PR feedback
* Fixed uninitialized Context
* Fixed another test failure
* Fixed permission check
* Added api test for DoPostActionWithCookie
* Replaced config.ActionCookieSecret with Server.PostActionCookieSecret
Modeled after AsymetricSigningKey
* style
* Set DeleteAt in DeleteEphemeralPost
* PR feedback
* Removed deadwood comment
* Added EXPERIMENTAL comment to the 2 APIs in question
* Rerouted the unit test logs through t.Log
* resolving merge confilvts
* Update testing.go
* Update helper_test.go
* Added godocs for NewTestingLogger
* Added go docs for NewTestingLogger
* Resolving conflicts
* config file store
Introduce an interface and concrete implementation for accessing the config.
This mostly maps 1:1 with the exiting usage in `App`, except for internalizing the watcher. A future change will likely eliminate `App.PersistConfig()` and make this implicit on `Set` or `Patch`
* experimental file test changes
* emoji: move file driver checks from api4 to app
It is no longer possible to app.UpdateConfig and provide an invalid configuration, making it hard to test this case. This check doesn't really belong in the api anyway, since it's a configuration validity check and not a permissions check. Either way, the check now occurs at the App level.
* api4: generate valid public link salts for test
* TestStartServerRateLimiterCriticalError: use mock store to test invalid config
* remove config_test.go
* remove needsSave, and have Load() save to the backing store as necessary
* restore README.md
* move ldap UserFilter check to model isValid checks
* remove databaseStore until ready
* remove unimplemented Patch
* simplify unlockOnce implementation
* revert forgetting to set s.Ldap
* config/file.go: rename ReadOnlyConfigurationError to ErrReadOnlyConfiguration
* config: export FileStore
* add TestFileStoreSave
* improved config/utils test coverage
* restore config/README.md copy
* tweaks
* file store: acquire a write lock on Save/Close to safely close watcher
* fix unmarshal_test.go
* refactor utils/config* to config/
* pull validateLdapFilter into app
* clean up Config/GetConfig/GetSanitizedConfig usage
Eliminate app.GetConfig() in favour of just using app.Config() directly,
but expose app.GetSanitizedConfig() for when the old behaviour was
required.
* web: isolate config setup
* TestInvitePeopleProvider: make config explicit
* regenerateClientConfig: avoid racey map access
* integrate watch flag into app.ConfigFile option
* make app.Option return an error
* release.mk: only cp static files from config/
* release.mk: fix cp static files from config/
* api4: TestPlugin cleanup
* s/c/cfg/ for clarity
* fix merge conflict
* testlib: allow customization of testlib driver name
* MM-10417 Add local image proxy and enable by default
* Remove unused function
* Add dependencies for willnorris/imageproxy
* Fixed compilation errors
* Lock to the master version of willnorris/imageproxy
* Fix atmos/camo proxy when no SiteURL is specified
* Re-add default values for deprecated settings
* Fix unit tests added by merge
* Pass imageproxy to App struct
* Remove unneeded locking when creating the image proxy
* Remove empty test file
* MM-10856: deduplicate posts with the same pending post id
Leverage a fixed size cache with a window of 30 seconds to deduplicate posts received by a single app server. Clients that duplicate the same pending post id will see one request potentially delayed until the first finishes, after which the same payload should be returned by both. Duplicate posts outside the 30 second window will not be de-duplicated.
Note that the cache is not synchronized between app servers. In an HA cluster consisting of more than one app server, sticky load balancing (e.g. hashing by IP or Session ID) is required to route the users to the same app instance for multiple requests.
Other options considered for this feature:
* adding a column to the `Posts` table: rejected as being too heavyweight
* maintaining a `PendingPostIds` table: similarly rejected for the database impact
* using the pending post id as the post id and relying on the unique constraints on the Post table: rejected for being difficult to show that it's safe to use a client-provided value as the row identifier
* utils/lru: simplify to ttl internally and for new methods
* move seenPendingPostIdsCache to App.Server
* just fail concurrent post requests, vs. trying to wait
* add debug log when create post is deduplicated
* guard app plugins with mutex
Shutting down the app could race with a goroutine that uses the plugins environment, since we shut down the plugins first before cleaning up goroutines.
* fix go vet issues
* Moving goroutine pool
* Auto refactor
* Moving plugins.
* Auto refactor
* Moving fields to server
* Auto refactor
* Removing siteurl duplication.
* Moving reset of app fields
* Auto refactor
* Formatting
* Moving niling of Server to after last use
* Fixing unit tests.
Config Checks at StartUp Part1
Config Checks; Tests for TLS Server
HSTS header implementation + tests
make gofmt happy with new go version...
make gofmt happy with new go version #2...
fix logic bug
fix typo
Fix unnecessary code block
