* MM-26575: Move app initialization inside Websocket router out of handler
The websocket router struct is shared amongst multiple handlers. Therefore,
there is a race condition while setting the app field and reading from it.
We move the initialization of the app field when the router struct is initialized.
And we also remove the initializations of some app fields which caused race
conditions by being written from multiple goroutines. They are already being
written once inside the AppInitializedOnce.Do method and it's redundant
to set them again to the same value.
* Add missing fields in server connector
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* MM-27116: Drain push notification channel before closing
Without this, pending push notifications will not be sent when shutting down
or restarting the server.
* Fix race
* Re-arrange server shutdown
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
Adds the advanced logging config for audit. Existing support for auditing to a single file remains for E0 and E10 licenses instances, and a new config item ExperimentalAuditSettings.AdvancedLoggingConfig is added that behaves like LogSettings.AdvancedLoggingConfig.
Supported destinations:
- file
- syslog (with out without TLS)
- raw TCP socket (with out without TLS)
ExperimentalAuditSettings.AdvancedLoggingConfig can contain a filespec to a config file, a database DSN, or JSON.
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
Co-authored-by: Claudio Costa <cstcld91@gmail.com>
Adds Advanced Logging to server. Advanced Logging is an optional logging capability that allows customers to send log records to any number of destinations.
Supported destinations:
- file
- syslog (with out without TLS)
- raw TCP socket (with out without TLS)
Allows developers to specify discrete log levels as well as the standard trace, debug, info, ... panic. Existing code and logging API usage is unchanged.
Log records are emitted asynchronously to reduce latency to the caller. Supports hot-reloading of logger config, including adding removing targets.
Advanced Logging is configured within config.json via "LogSettings.AdvancedLoggingConfig" which can contain a filespec to another config file, a database DSN, or JSON.
* Extracting email functions into a service
* Fixing two shadowing errors
* Address PR review comments
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* MM-24972: make GetHubForUserId lock-less and zero-alloc
The hubs aren't meant to be modified after the server starts up.
But the key problem was the SetHubs method which would zero out
the underlying hubs slice.
This wasn't ideally necessary because an hub would only be stopped
once during server shutdown. However, this was required by a test
which would shutdown the hub twice. And since the hub shutdown isn't
idempotent, zeroing the slice would just skip over shutting down the hub again.
To improve the overall situation, we apply several optimizations.
- We use the new hash/maphash package which exposes Go runtime's
internal hash algorithms to be used as a package. This is much faster
than hash/fnv.
- We move around the initialization of the hub to happen before
the metrics server starts. This allows us to initialize the hub
before any of the hub elements are being accessed.
- To make the test run successfully, we do not call th.TearDown.
This is fine for a test, because anyways the test process would eventually stop
and relinquish the resources to the OS.
This allows us to completely remove any mutexes and thereby
we can remove all the methods and any edge-case checks related to
index being out of bounds. As a result, the fast path becomes
very straightforward and zero-alloc.
name old time/op new time/op delta
GetHubForUserId-8 116ns ± 1% 38ns ± 7% -67.22% (p=0.000 n=10+10)
name old alloc/op new alloc/op delta
GetHubForUserId-8 36.0B ± 0% 0.0B -100.00% (p=0.000 n=10+10)
name old allocs/op new allocs/op delta
GetHubForUserId-8 2.00 ± 0% 0.00 -100.00% (p=0.000 n=10+10)
Manually tested with some load testing and running Hub tests in -race mode.
* remove mutex
* incorporate review comments
* MM-25404: Add GetHealthScore metric for the cluster
This PR adds the necessary function to the cluster interface
so that they can be called from App code.
* Add mocks
* Remove fakeapp
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* Removing some other fake apps
* More FakeApp removed
* Removing entirely FakeApp
* Fixing some tests
* Fixing get Cluster id from get plugin status
* Fixing failing tests
* Fixing tests
* Fixing test initialization for web
* Fixing InitServer for server tests
* Fixing InitServer for server tests
* Reverting go.sum and go.mod
* Removing unneded HTMLTemplates function in App layer
* Moving back some functions to its old place to easy the review
* Moving back some functions to its old place to easy the review
* Using the last struct2interface version
* Generating store layers
* Fixing merge problems
* Addressing PR comments
* Small fix
* Fixing app tests build
* Fixing tests
* fixing tests
* Fix tests
* Fixing tests
* Fixing tests
* Fixing tests
* Moving license to server struct
* Adding some fixes to the test compilation
* Fixing cluster and some jobs initialization
* Fixing some license tests compilation problems
* Fixing recursive cache invalidation
* Regenerating app layers
* Fix test compilation
Co-authored-by: mattermod <mattermod@users.noreply.github.com>
There are 3 client config related variables which are set in regenerateClientConfig
but can be read from public methods in the App struct.
We use the same approach as the license fields in the App struct and use atomic.Value
for modifying them.
Co-authored-by: mattermod <mattermod@users.noreply.github.com>
* Fixing behavior of Replicas and SearchReplicas in canary environments
* Trying to fix tests
* Revert "Trying to fix tests"
This reverts commit 3531da961844f5cb8557efcd3b570b06f362c6df.
* Revert "Fixing behavior of Replicas and SearchReplicas in canary environments"
This reverts commit 0c05901c843e4ccd60c8320fb4e0123b1bacf430.
* Revert "Disable read/search db replicas in TE/E0 (#14400)"
This reverts commit ef5ac519d9.
* Making the store aware of the license
* Readding the unit tests
* Fixing sqlstor supplier tests
* Adding mutex to ensure license write consistency and fixing tests
* Fixing tests
* Fixing tests
* Shuting down server properly during tests
* Trying to fix tests
* Trying to fix the tests
* Skipping flaky tests
Co-authored-by: mattermod <mattermod@users.noreply.github.com>
The order of booting up a server should be
Job scheduler -> Cluster.
And shutdown should be the opposite. This is needed because the job scheduler
initializes certain data structures that are later called by ClusterLeaderChanged
event handlers. And the event handlers run in a separate goroutine.
Therefore, if the cluster initialization happens before the job scheduler,
then a race happens between accessing the jobs variable and setting it.
To prevent this race, we fix the order of startup, and add comments in both
places to prevent things from regressing again.
* [MM-20979] Add first implementation of the Bleve search engine
* Fix i18n
* Migrate searchengine utils tests
* Fix linter
* Don't add allTermsQ if both termQueries and notTermQueries are empty
* Fix test that should work if user is system admin
* Modify naming according to review comments
* Abstract getIndexDir function
* Extracting bleve engine name as a constant
* Merge both Indexer interfaces into one
* Add worker stopped message
* Allow worker to be started/stopped with config change
* Use constants for index names
* Modify test order
* Fix linter
* Trying to unlock the CI
* [MM-24146] Add unix socket listener for mmctl local mode (#14296)
* add unix socket listener for mmctl local mode
* add a constant for local-mode socket path
* reflect review comments
* [MM-24401] Base approach for Local Mode (#14333)
* add unix socket listener for mmctl local mode
* First working PoC
* Adds the channel list endpoint
* Add team list endpoint
* Add a LocalClient to the api test helper and start local mode
* Add helper to test with both SystemAdmin and Local clients
* Add some docs
* Adds TestForAllClients test helper
* Incorporating @ashishbhate's proposal for adding test names to the helpers
* Fix init errors after merge
* Adds create channel tests
* Always init local mode to allow for enabling-disabling it via config
* Check the RemoteAddr of the request before marking session as local
* Mark the request as errored if it's local and the origin is remote
* Set the socket permissions to read/write when initialising
* Fix linter
* Replace RemoteAddr check to ditch connections with the IP:PORT shape
Co-authored-by: Ibrahim Serdar Acikgoz <serdaracikgoz86@gmail.com>
* Fix translations order
* [MM-24832] Migrate plugin endpoints to local mode (#14543)
* [MM-24832] Migrate plugin endpoints to local mode
* Fix client reference in helper
* [MM-24776] Migrate config endpoints to local mode (#14544)
* [MM-24776] Migrate get config endpoint to local mode
* [MM-24777] Migrate update config endpoint to local mode
* Fix update config to bypass RestrictSystemAdmin flag
* Add patchConfig endpoint
* MM-24774/MM-24755: local mode for addLicense and removeLicense (#14491)
Automatic Merge
Co-authored-by: Ibrahim Serdar Acikgoz <serdaracikgoz86@gmail.com>
Co-authored-by: Ashish Bhate <bhate.ashish@gmail.com>
* Disable read/search db replicas in TE/E0
* fixing tests
* Removing unnecesary text.
* Updating without-license read-replicas config before store initialization
* Reconnecting to database after remove read replicas
* MM-23800: remove goroutineID and stack printing
Each hub has a goroutineID which is calculated with a known hack.
The FAQ clearly explains why goroutines don't have an id:
https://golang.org/doc/faq#no_goroutine_id.
We only added that because sometimes the hub would be deadlocked and
having the goroutineID would be useful when getting the stack trace.
This is also problematic in stress tests because the hubs would
frequently get overloaded and the logs would unnecessarily have stack traces.
But that was in the past, and we have done extensive testing with
load tests and fuzz testing to smooth any rough edges remaining.
Including adding additional metrics for hub buffer size.
Monitoring the metrics is a better way to approach this problem.
Therefore, we remove these kludges from the code.
* Also remove deadlock checking code
There is no need for that anymore since
we are getting rid of the stack printing anyways.
Let's do a wholesale refactor and clean up the codebase.
Co-authored-by: mattermod <mattermod@users.noreply.github.com>
Before this, for every single request we would create a new http.Client
from scratch. This was costly because every new client has internal
TCP connections which it reuses.
Therefore, originally, beyond a throughput of 800-1200 rps with a
concurrency of 50, beyond which it would run out of file descriptors.
Now it can support upto 8000-10000 rps.
An improvement by a factor of 10.
Throughput numbers:
BenchmarkPushNotification
BenchmarkPushNotification: notification_push_test.go:1478: throughput: 11445.091601 reqs/s
BenchmarkPushNotification: notification_push_test.go:1478: throughput: 9190.761518 reqs/s
BenchmarkPushNotification: notification_push_test.go:1478: throughput: 8334.394764 reqs/s
BenchmarkPushNotification-8 219 5999304 ns/op 3369224 B/op 42127 allocs/op
BenchmarkPushNotification: notification_push_test.go:1478: throughput: 9590.399780 reqs/s
BenchmarkPushNotification: notification_push_test.go:1478: throughput: 9347.137463 reqs/s
BenchmarkPushNotification: notification_push_test.go:1478: throughput: 9053.784250 reqs/s
BenchmarkPushNotification-8 223 5522636 ns/op 3368470 B/op 42034 allocs/op
BenchmarkPushNotification: notification_push_test.go:1478: throughput: 10250.783365 reqs/s
BenchmarkPushNotification: notification_push_test.go:1478: throughput: 9123.292899 reqs/s
BenchmarkPushNotification: notification_push_test.go:1478: throughput: 9173.553180 reqs/s
This is the aggregate count using benchstat.
The number is for a single run of sending 50 requests.
name time/op
PushNotification-8 5.61ms ± 7%
name alloc/op
PushNotification-8 3.36MB ± 1%
name allocs/op
PushNotification-8 42.0k ± 1%
Tested on a machine with ulimit -n = 4096.
* WIP
* Adding bleve to go modules
* WIP
* Adding missing files from searchengine implementation
* WIP
* WIP
* WIP
* WIP
* WIP
* WIP
* User and channel indexing and searches implemented
* Make bleve tests run with in-memory indexes
* Implement post index and deletion tests
* Initial commits for the search layer
* Removing unnecesary indexing
* WIP
* WIP
* More fixes for tests
* Adding the search layer
* Finishing the migration of searchers to the layer
* Removing unnecesary code
* Allowing multiple engines active at the same time
* WIP
* Add simple post search
* Print information when using bleve
* Adding some debugging to understand better how the searches are working
* Making more dynamic config of search engines
* Add post search basics
* Adding the Purge API endpoint
* Fixing bleve config updates
* Adding missed file
* Regenerating search engine mocks
* Adding missed v5 to modules imports
* fixing i18n
* Fixing some test around search engine
* Removing all bleve traces
* Cleaning up the vendors directory and go.mod/go.sum files
* Regenerating timer layer
* Adding properly the license
* Fixing govet shadow error
* Fixing some tests
* Fixing TestSearchPostsFromUser
* Fixing another test
* Fixing more tests
* Fixing more tests
* Removing SearchEngine redundant text from searchengine module code
* Fixing some reindexing problems in members updates
* Fixing tests
* Addressing PR comments
* Reverting go.mod and go.sum
* Addressing PR comments
* Fixing tests compilation
* Fixing govet
* Adding search engine stop method
* Being more explicit on where we use includeDeleted
* Adding GetSqlSupplier test helper method
* Mocking elasticsearch start function
* Fixing tests
Co-authored-by: Miguel de la Cruz <miguel@mcrx.me>
Co-authored-by: mattermod <mattermod@users.noreply.github.com>
* New auditing API outputting to syslog via TLS
* New config section for specifying remote syslog server IP, port, and cert.
* Legacy audit API retained for access history feature
Hubs property has been detected by the race condition detector
as a possible one because we could set the Hubs variable
and also we're spawning goroutines that could ask for that property
This condition is not happening right now because we're setting the
Hubs variable in the server start and after that, we're spawning the
goroutines but the tests are failing because of this and in the future
this could be a problem
* Refactor to use structured logging
* Properly formatted with gofmt
* created interface Cache, but construction of cache is still coupled.
* Implementing cache factories to build caches
* Simple redis implementation without error handling. Keys and values by default are string
* refactor NewLocalCacheLayer to inject cache factory
* Removed redis impl to focus on cache abstraction
* CacheFactory injected on sqlsupplier and saved in Store struct
* remove useless private method
* replace concrete declaration of lru cache to cache abstraction
* discard spaces
* Renamed factory to provider because concrete implementations of factories may hold an state (such as a redis client for example)
* refactor to include all caches in the same package cache and subpackages
* method close cache. This method will be used for concrete implementations that need to release resources (close a connection, etc)
* closing cacheprovider and releasing resources while closing sql store
* fixed merge conflict fail
* gofmt files
* remove unused property from post_store
* naming refactor to avoid stutter. Added godocs on interface
* Store doesnt know anything about the cache and provider. Cache provider will be built after loading config and injected in localCacheLayer
* fixed broken test
* cache provider initialized before RunOldAppInitialization which initializes the localcachelayer
* move statusCache to server to initialize it with the new cache provider
* update terms_service and channel_layer to have new cacheProvider
* gofmt
* Add Connect method to the cache provider
* mock cacheprovider in user_layer_test
Co-authored-by: Ben Schumacher <ben.schumacher@mattermost.com>
Co-authored-by: mattermod <mattermod@users.noreply.github.com>
* MM-8607: Add cluster support for server busy status
MM-8607: Busy.Cluster should be non-exported
MM-8607: rename logging field seconds -> expires_sec
MM-8607: each node clears its own busy flag
MM-8607: ensure unit tests are not sensitive to test machine speed
* MM-8607: chg comment to force CI rebuild
* Consistent license message for all the go files
* Fixing the last set of unconsistencies with the license headers
* Addressing PR review comments
* Fixing busy.go and busy_test.go license header
* MM-8607: add ability to turn off non-critical services under load
* server busy invalid param unit tests
* MM-8607: rename server busy endpoints
* MM-8607: handle case where App not initialized
* MM-8607: additional unit test cases per feedback.
* MM-8607: use decorator to check isbusy when adding endpoint route
* MM-8607: rename endpoints, use struct for json
* Update api4/system.go
Fix misspelled log output
Co-Authored-By: Saturnino Abril <saturnino.abril@gmail.com>
* MM-8607: fix i18n order; max seconds for server busy expiry
