* MM-23800: remove goroutineID and stack printing
Each hub has a goroutineID which is calculated with a known hack.
The FAQ clearly explains why goroutines don't have an id:
https://golang.org/doc/faq#no_goroutine_id.
We only added that because sometimes the hub would be deadlocked and
having the goroutineID would be useful when getting the stack trace.
This is also problematic in stress tests because the hubs would
frequently get overloaded and the logs would unnecessarily have stack traces.
But that was in the past, and we have done extensive testing with
load tests and fuzz testing to smooth any rough edges remaining.
Including adding additional metrics for hub buffer size.
Monitoring the metrics is a better way to approach this problem.
Therefore, we remove these kludges from the code.
* Also remove deadlock checking code
There is no need for that anymore since
we are getting rid of the stack printing anyways.
Let's do a wholesale refactor and clean up the codebase.
Co-authored-by: mattermod <mattermod@users.noreply.github.com>
Before this, for every single request we would create a new http.Client
from scratch. This was costly because every new client has internal
TCP connections which it reuses.
Therefore, originally, beyond a throughput of 800-1200 rps with a
concurrency of 50, beyond which it would run out of file descriptors.
Now it can support upto 8000-10000 rps.
An improvement by a factor of 10.
Throughput numbers:
BenchmarkPushNotification
BenchmarkPushNotification: notification_push_test.go:1478: throughput: 11445.091601 reqs/s
BenchmarkPushNotification: notification_push_test.go:1478: throughput: 9190.761518 reqs/s
BenchmarkPushNotification: notification_push_test.go:1478: throughput: 8334.394764 reqs/s
BenchmarkPushNotification-8 219 5999304 ns/op 3369224 B/op 42127 allocs/op
BenchmarkPushNotification: notification_push_test.go:1478: throughput: 9590.399780 reqs/s
BenchmarkPushNotification: notification_push_test.go:1478: throughput: 9347.137463 reqs/s
BenchmarkPushNotification: notification_push_test.go:1478: throughput: 9053.784250 reqs/s
BenchmarkPushNotification-8 223 5522636 ns/op 3368470 B/op 42034 allocs/op
BenchmarkPushNotification: notification_push_test.go:1478: throughput: 10250.783365 reqs/s
BenchmarkPushNotification: notification_push_test.go:1478: throughput: 9123.292899 reqs/s
BenchmarkPushNotification: notification_push_test.go:1478: throughput: 9173.553180 reqs/s
This is the aggregate count using benchstat.
The number is for a single run of sending 50 requests.
name time/op
PushNotification-8 5.61ms ± 7%
name alloc/op
PushNotification-8 3.36MB ± 1%
name allocs/op
PushNotification-8 42.0k ± 1%
Tested on a machine with ulimit -n = 4096.
* WIP
* Adding bleve to go modules
* WIP
* Adding missing files from searchengine implementation
* WIP
* WIP
* WIP
* WIP
* WIP
* WIP
* User and channel indexing and searches implemented
* Make bleve tests run with in-memory indexes
* Implement post index and deletion tests
* Initial commits for the search layer
* Removing unnecesary indexing
* WIP
* WIP
* More fixes for tests
* Adding the search layer
* Finishing the migration of searchers to the layer
* Removing unnecesary code
* Allowing multiple engines active at the same time
* WIP
* Add simple post search
* Print information when using bleve
* Adding some debugging to understand better how the searches are working
* Making more dynamic config of search engines
* Add post search basics
* Adding the Purge API endpoint
* Fixing bleve config updates
* Adding missed file
* Regenerating search engine mocks
* Adding missed v5 to modules imports
* fixing i18n
* Fixing some test around search engine
* Removing all bleve traces
* Cleaning up the vendors directory and go.mod/go.sum files
* Regenerating timer layer
* Adding properly the license
* Fixing govet shadow error
* Fixing some tests
* Fixing TestSearchPostsFromUser
* Fixing another test
* Fixing more tests
* Fixing more tests
* Removing SearchEngine redundant text from searchengine module code
* Fixing some reindexing problems in members updates
* Fixing tests
* Addressing PR comments
* Reverting go.mod and go.sum
* Addressing PR comments
* Fixing tests compilation
* Fixing govet
* Adding search engine stop method
* Being more explicit on where we use includeDeleted
* Adding GetSqlSupplier test helper method
* Mocking elasticsearch start function
* Fixing tests
Co-authored-by: Miguel de la Cruz <miguel@mcrx.me>
Co-authored-by: mattermod <mattermod@users.noreply.github.com>
* New auditing API outputting to syslog via TLS
* New config section for specifying remote syslog server IP, port, and cert.
* Legacy audit API retained for access history feature
Hubs property has been detected by the race condition detector
as a possible one because we could set the Hubs variable
and also we're spawning goroutines that could ask for that property
This condition is not happening right now because we're setting the
Hubs variable in the server start and after that, we're spawning the
goroutines but the tests are failing because of this and in the future
this could be a problem
* Refactor to use structured logging
* Properly formatted with gofmt
* created interface Cache, but construction of cache is still coupled.
* Implementing cache factories to build caches
* Simple redis implementation without error handling. Keys and values by default are string
* refactor NewLocalCacheLayer to inject cache factory
* Removed redis impl to focus on cache abstraction
* CacheFactory injected on sqlsupplier and saved in Store struct
* remove useless private method
* replace concrete declaration of lru cache to cache abstraction
* discard spaces
* Renamed factory to provider because concrete implementations of factories may hold an state (such as a redis client for example)
* refactor to include all caches in the same package cache and subpackages
* method close cache. This method will be used for concrete implementations that need to release resources (close a connection, etc)
* closing cacheprovider and releasing resources while closing sql store
* fixed merge conflict fail
* gofmt files
* remove unused property from post_store
* naming refactor to avoid stutter. Added godocs on interface
* Store doesnt know anything about the cache and provider. Cache provider will be built after loading config and injected in localCacheLayer
* fixed broken test
* cache provider initialized before RunOldAppInitialization which initializes the localcachelayer
* move statusCache to server to initialize it with the new cache provider
* update terms_service and channel_layer to have new cacheProvider
* gofmt
* Add Connect method to the cache provider
* mock cacheprovider in user_layer_test
Co-authored-by: Ben Schumacher <ben.schumacher@mattermost.com>
Co-authored-by: mattermod <mattermod@users.noreply.github.com>
* MM-8607: Add cluster support for server busy status
MM-8607: Busy.Cluster should be non-exported
MM-8607: rename logging field seconds -> expires_sec
MM-8607: each node clears its own busy flag
MM-8607: ensure unit tests are not sensitive to test machine speed
* MM-8607: chg comment to force CI rebuild
* Consistent license message for all the go files
* Fixing the last set of unconsistencies with the license headers
* Addressing PR review comments
* Fixing busy.go and busy_test.go license header
* MM-8607: add ability to turn off non-critical services under load
* server busy invalid param unit tests
* MM-8607: rename server busy endpoints
* MM-8607: handle case where App not initialized
* MM-8607: additional unit test cases per feedback.
* MM-8607: use decorator to check isbusy when adding endpoint route
* MM-8607: rename endpoints, use struct for json
* Update api4/system.go
Fix misspelled log output
Co-Authored-By: Saturnino Abril <saturnino.abril@gmail.com>
* MM-8607: fix i18n order; max seconds for server busy expiry
* diagnostics_test.go: fix spacing
* diagnostics_test.go: explicitly assert payload
This fails, since the package is currently receiving struct pointers and won't set the MessageId or Timestamp on the corresponding Message.
* MM-18115: fix segment v3 usage
In v5.14, we updated [github.com/segmentio/analytics-go](https://github.com/segmentio/analytics-go) to v3 as part of https://mattermost.atlassian.net/browse/MM-12389. As noted in the [migration guide](https://segment.com/docs/sources/server/go/#migrating-from-v2), the API subtly changed to expect a struct value and not a struct pointer:
```go
// in v2, you would call the `Track` method with a `Track` struct.
client.Track(&track)
// in v3, you would call the `Enqueue` method with a `Track` struct.
// Note that a pointer is not used here.
client.Enqueue(track)
```
Unfortunately, we kept passing a pointer, and the package didn't complain since it only required an interface -- which the pointer to these structs still implemented. Internally, it only checked for the value types, and failed to annotate our payloads with the requisite metadata. Upstream, segment.io accepted the payload, but then discarded it silently.
This has since been reported and fixed in https://github.com/segmentio/analytics-go/pull/146, but isn't yet part of a tagged release of the package.
Fix our code to pass struct values instead.
Fixes: MM-18115
* revert 4e5f6fcfbc
This reverts the configuration setting to disable plugin health checks, preferring instead to retain this functionality for advanced cases. In part, this was driven by the discovery that the health checks were failing on Windows, though that will be addressed separately.
Fixes: MM-16378
* simplify interval handling/logging
Also shutdown the job when plugins are altogether disabled.
The job workers and cluster discovery components require access to the
database during shutdown. To avoid issues, we leave the database open
until the last step of the shutdown process.
Properly handle analytics client life cycle
This change improves the life cycle management of the analytics
client. It does so in two primary ways:
- Move the client from being a shared pointer to belonging to the
Server struct.
- Expose the client.Close() method for proper shutdown of the
client.
* Updated ResetAll function to return object
* Modified the defined interface
* Changed the data type where ResetAll was referrenced
* Refactored code
Changed user's email
* Fixed indentation
* Add config setting to explicitly define which IP headers are trusted
* fix variable shadowing
* Optimize code flow; Add Ratelimit test for header set
* Extend Ratelimit tests
* Add additional unit tests
* Structured logging
* MM-15494 Adding warning log when someone has configured a Mattermost push proxy with HTTP endpoint
* MM-15494 go fmt check
* MM-15494 Making function private
* MM-15494 Updating warning message
* Remove NotificationRegistry table and use structured logging
* Fix ackId for notification sent
* Notification logger at server level
* Remove unused i18n strings
* introduce plugin health check
* implement plugin health check job
* add support for checking pid of plugin process and RPC ping, to determine a plugin's health
* implement restart policy with back-offs
* support "EnableHealthCheck" boolean from config file.
* add tests for supervisor.PerformHealthCheck() and shouldDeactivatePlugin()
* improve error handling. clean up if blocks to be more concise
* [MM-10199] Printing stack traces from http handlers panics in mattermost logs
Previously recovery handlers from gorilla mux was used, which was printing stack traces from panics in stderr
Removing it should print stack traces into logger passed to http.Server, which as of now is created from mlog
Refer #10351
* Logging errors from http server at error level instead of info level
Added test for verifying whether panic gets logged in server logger
* Fixed failing test due to tls error
* Closing temp file for panic log before removing and checking errors for closure and removal
* Added regular log line in panic handler for TestPanicLog to check whether it gets logged
* Added support for PostActions in ephemeral posts
The general approach is that we take all the metadata that DoPostAction
needs to process client DoPostActionRequests, and store it in a
serialized, encrypted Cookie field, in the PostAction struct.
The client then must send it back, and it is then used to process
PostActions as a fallback top the metadata in the database.
This PR adds a new config setting, `ServiceSettings.ActionCookieSecret`.
In a cluster environment it must be the same for all instances.
- Added type PostActionCookie, and a Cookie string to PostAction.
- Added App.AddActionCookiesToPost.
- Use App.AddActionCookiesToPost in api4.createEphemeralPost,
App.SendEphemeralPost, App.UpdateEphemeralPost.
- Added App.DoPostActionWithCookie to process incoming requests with
cookies. For backward compatibility, it prefers the metadata in the
database; falls back to cookie.
- Added plugin.API.UpdateEphemeralPost and plugin.API.DeleteEphemeralPost.
- Added App.encryptActionCookie/App.decryptActionCookie.
* Style
* Fixed an unfortunate typo, tested with matterpoll
* minor PR feedback
* Fixed uninitialized Context
* Fixed another test failure
* Fixed permission check
* Added api test for DoPostActionWithCookie
* Replaced config.ActionCookieSecret with Server.PostActionCookieSecret
Modeled after AsymetricSigningKey
* style
* Set DeleteAt in DeleteEphemeralPost
* PR feedback
* Removed deadwood comment
* Added EXPERIMENTAL comment to the 2 APIs in question
* Rerouted the unit test logs through t.Log
* resolving merge confilvts
* Update testing.go
* Update helper_test.go
* Added godocs for NewTestingLogger
* Added go docs for NewTestingLogger
* Resolving conflicts
* config file store
Introduce an interface and concrete implementation for accessing the config.
This mostly maps 1:1 with the exiting usage in `App`, except for internalizing the watcher. A future change will likely eliminate `App.PersistConfig()` and make this implicit on `Set` or `Patch`
* experimental file test changes
* emoji: move file driver checks from api4 to app
It is no longer possible to app.UpdateConfig and provide an invalid configuration, making it hard to test this case. This check doesn't really belong in the api anyway, since it's a configuration validity check and not a permissions check. Either way, the check now occurs at the App level.
* api4: generate valid public link salts for test
* TestStartServerRateLimiterCriticalError: use mock store to test invalid config
* remove config_test.go
* remove needsSave, and have Load() save to the backing store as necessary
* restore README.md
* move ldap UserFilter check to model isValid checks
* remove databaseStore until ready
* remove unimplemented Patch
* simplify unlockOnce implementation
* revert forgetting to set s.Ldap
* config/file.go: rename ReadOnlyConfigurationError to ErrReadOnlyConfiguration
* config: export FileStore
* add TestFileStoreSave
* improved config/utils test coverage
* restore config/README.md copy
* tweaks
* file store: acquire a write lock on Save/Close to safely close watcher
* fix unmarshal_test.go
* refactor utils/config* to config/
* pull validateLdapFilter into app
* clean up Config/GetConfig/GetSanitizedConfig usage
Eliminate app.GetConfig() in favour of just using app.Config() directly,
but expose app.GetSanitizedConfig() for when the old behaviour was
required.
* web: isolate config setup
* TestInvitePeopleProvider: make config explicit
* regenerateClientConfig: avoid racey map access
* integrate watch flag into app.ConfigFile option
* make app.Option return an error
* release.mk: only cp static files from config/
* release.mk: fix cp static files from config/
* api4: TestPlugin cleanup
* s/c/cfg/ for clarity
* fix merge conflict
* testlib: allow customization of testlib driver name
* MM-10417 Add local image proxy and enable by default
* Remove unused function
* Add dependencies for willnorris/imageproxy
* Fixed compilation errors
* Lock to the master version of willnorris/imageproxy
* Fix atmos/camo proxy when no SiteURL is specified
* Re-add default values for deprecated settings
* Fix unit tests added by merge
* Pass imageproxy to App struct
* Remove unneeded locking when creating the image proxy
* Remove empty test file
* MM-10856: deduplicate posts with the same pending post id
Leverage a fixed size cache with a window of 30 seconds to deduplicate posts received by a single app server. Clients that duplicate the same pending post id will see one request potentially delayed until the first finishes, after which the same payload should be returned by both. Duplicate posts outside the 30 second window will not be de-duplicated.
Note that the cache is not synchronized between app servers. In an HA cluster consisting of more than one app server, sticky load balancing (e.g. hashing by IP or Session ID) is required to route the users to the same app instance for multiple requests.
Other options considered for this feature:
* adding a column to the `Posts` table: rejected as being too heavyweight
* maintaining a `PendingPostIds` table: similarly rejected for the database impact
* using the pending post id as the post id and relying on the unique constraints on the Post table: rejected for being difficult to show that it's safe to use a client-provided value as the row identifier
* utils/lru: simplify to ttl internally and for new methods
* move seenPendingPostIdsCache to App.Server
* just fail concurrent post requests, vs. trying to wait
* add debug log when create post is deduplicated
