Add auditing to server CLI.
Also:
- simplify auditing in API layer
- reduce number of AddMeta calls
- have models serialize themselves
- more consistent field naming
Filter blacklisted Unicode characters from:
user: first name, last name, nickname, bot description, username
team: name, display name, description, company name
channel: name, display name
With Go 1.14 (https://tip.golang.org/doc/go1.14#go-command),
it now automatically checks for the existence of a vendor directory.
So there is no need to explicitly set the `-mod=vendor` flag.
* MM-23620: Handle error from GetUser
In case of high DB load, the DB will start to throw errors.
Unless we handle the error appropriately, the server will crash.
* Removing unnecessary lines
* store/cache: add cache to user layer
* store/localcache: fix cache data access
* app/login: add invalidate for user
* store/user: move user sanitization to cache
* store/user: remove sanitize checks form the tests
* store/user: remove sanitization from store and cache layer
* store/cache: remove unnecessary error wraps
Co-authored-by: mattermod <mattermod@users.noreply.github.com>
* MM-23770: Fix for blank DefaultChannelGuestRole on team schemes.
* MM-23770: Adds test for an team scheme with a blank DefaultChannelGuestRole field.
* MM-23770: Fix for unexpected Schemes.Get.
* store failed timestamps on health check job instead of on registeredPlugin
Update test
* change EnsurePlugin calls
* Make env.SetPluginState private
* Write test for plugin deactivate and PluginStateFailedToStayRunning
* Add license comment
* adjust comments, use time.Since
* Additional PR feedback:
time.Since cleanup
test cleanup
remove duplicate .Store() call
* PR Feedback
- Add test case for reactivating the failed plugin
- Change `crashed` to `healthy` and `hasPluginCrashed` to `isPluginHealthy`
- remove stale timestamps from health check job
* Keep registeredPlugins in env when plugin is deactivated, so the crashed state of a plugin can be persisted.
* PR feedback
* PR feedback from Jesse
Co-authored-by: mattermod <mattermod@users.noreply.github.com>
- replace all instances of `fmt.Sprintf("whatever")` with simply "whatever".
- replace all instances of `fmt.Println(fmt.Sprintf("whatever %d", i))` with `fmt.Printf("whatever %d\n", i)`
* MM-23484 - add simple validation to GetUsersStatusesByIds
* MM-23484 - return early in validation, add extra test case
* MM-23484 - add test case for empty array
Co-authored-by: Fedor Vitkovskiy <fedorvitkovskiy@pop-os.localdomain>
* MM-22962 Create use_group_mentions permission and give to all non guest roles that can create post
* Add use_group_mentions to team admin role for test
* Trigger CI
* MM-22962 Remove old migration keys
* Document minimum server version required for plugin hooks
* change mins < v5.2 for v5.2
Co-authored-by: Federico Martín Alconada Verzini <fedealconada@gmail.com>
* explicitly assert panic as error log
* Revert "[MM-18150] plugin panic trace should not be lost (#13559)"
This reverts commit 5d928b4f94, while leaving the unit tests intact
and now asserting debug logs instead.
* missing license header
* WIP
* Adding bleve to go modules
* WIP
* Adding missing files from searchengine implementation
* WIP
* WIP
* WIP
* WIP
* WIP
* WIP
* User and channel indexing and searches implemented
* Make bleve tests run with in-memory indexes
* Implement post index and deletion tests
* Initial commits for the search layer
* Removing unnecesary indexing
* WIP
* WIP
* More fixes for tests
* Adding the search layer
* Finishing the migration of searchers to the layer
* Removing unnecesary code
* Allowing multiple engines active at the same time
* WIP
* Add simple post search
* Print information when using bleve
* Adding some debugging to understand better how the searches are working
* Making more dynamic config of search engines
* Add post search basics
* Adding the Purge API endpoint
* Fixing bleve config updates
* Adding missed file
* Regenerating search engine mocks
* Adding missed v5 to modules imports
* fixing i18n
* Fixing some test around search engine
* Removing all bleve traces
* Cleaning up the vendors directory and go.mod/go.sum files
* Regenerating timer layer
* Adding properly the license
* Fixing govet shadow error
* Fixing some tests
* Fixing TestSearchPostsFromUser
* Fixing another test
* Fixing more tests
* Fixing more tests
* Removing SearchEngine redundant text from searchengine module code
* Fixing some reindexing problems in members updates
* Fixing tests
* Addressing PR comments
* Reverting go.mod and go.sum
* Addressing PR comments
* Fixing tests compilation
* Fixing govet
* Adding search engine stop method
* Being more explicit on where we use includeDeleted
* Adding GetSqlSupplier test helper method
* Mocking elasticsearch start function
* Fixing tests
* Search tests
* Fix tests
* Fix mod
* Fixing searchEngine for test helpers with store mocks
* Remove loglines
* Fix i18n strings
* Migrate search posts tests
* Fix linter
* Do not run search tests if -short flag is enabled
* Migrate back store tests that didn't belong to the searchlayer
* Fix scopelint issues
Co-authored-by: Jesús Espino <jespinog@gmail.com>
* MM-23503: Fix race in websocket_client writer
We create a separate writer goroutine where all the writes happen.
* Fixing the case of double close
* Incorporate review comments
* Use CAS
* Fix incorrect comment
* Check if client is closed in pingHandler too
Co-authored-by: mattermod <mattermod@users.noreply.github.com>
* MM-22044: Fix panic on web_conn send hello
(*Hub).Start is the central place for sending all web connection
related traffic. However, there was this one call to (*WebConn).Hello
which tried to send a message to a webconn separately.
This was a rare case, but it did occur under stress conditions generated
from a load test.
When the websocket send SEND_QUEUE_SIZE would get filled up and we would
attempt to make a broadcast, the non-blocking send would close the Send
channel of the web connection. During that time, if a web connection
would try to perform a broadcast, it would try to send to a closed channel
and cause a panic.
The solution is to bring back the sending of hello into the same goroutine
inside (*Hub).Start so that all state is centralised and we avoid
sending to a closed channel by sending the hello message inside the registering
code itself.
* Adding non-blocking send
* Simplify things
* Remove test
* Bring sendHello back
* Improve code further
Co-authored-by: mattermod <mattermod@users.noreply.github.com>
Before this, for every single request we would create a new http.Client
from scratch. This was costly because every new client has internal
TCP connections which it reuses.
Therefore, originally, beyond a throughput of 800-1200 rps with a
concurrency of 50, beyond which it would run out of file descriptors.
Now it can support upto 8000-10000 rps.
An improvement by a factor of 10.
Throughput numbers:
BenchmarkPushNotification
BenchmarkPushNotification: notification_push_test.go:1478: throughput: 11445.091601 reqs/s
BenchmarkPushNotification: notification_push_test.go:1478: throughput: 9190.761518 reqs/s
BenchmarkPushNotification: notification_push_test.go:1478: throughput: 8334.394764 reqs/s
BenchmarkPushNotification-8 219 5999304 ns/op 3369224 B/op 42127 allocs/op
BenchmarkPushNotification: notification_push_test.go:1478: throughput: 9590.399780 reqs/s
BenchmarkPushNotification: notification_push_test.go:1478: throughput: 9347.137463 reqs/s
BenchmarkPushNotification: notification_push_test.go:1478: throughput: 9053.784250 reqs/s
BenchmarkPushNotification-8 223 5522636 ns/op 3368470 B/op 42034 allocs/op
BenchmarkPushNotification: notification_push_test.go:1478: throughput: 10250.783365 reqs/s
BenchmarkPushNotification: notification_push_test.go:1478: throughput: 9123.292899 reqs/s
BenchmarkPushNotification: notification_push_test.go:1478: throughput: 9173.553180 reqs/s
This is the aggregate count using benchstat.
The number is for a single run of sending 50 requests.
name time/op
PushNotification-8 5.61ms ± 7%
name alloc/op
PushNotification-8 3.36MB ± 1%
name allocs/op
PushNotification-8 42.0k ± 1%
Tested on a machine with ulimit -n = 4096.
* Remove goroutines from session_store.go
There isn't much benefit of running these 2 queries concurrently
versus just serially. These queries are not in the hot path anyways.
Maybe just simpler to keep it serial.
* Incorporate review comments
* Members Save Multiple
* Multiple Channel Members implemented
* Working version with member multi save and multi update
* Adding tests to the save multiple team members store methods
* Tests passing for SaveMember
* Adding remove members tests
* Working on channel store tests
* More work on testing properly SaveMember and SaveMultipleMembers on channels
* More work on testing properly SaveMember and SaveMultipleMembers on teams
* Adding tests for update multiple members for team
* Adding tests for update multiple members for channel
* Fixing some tets
* Fixing tests
* Fixing tests
* Fixing i18n
* Addressing govet problems
* Addressing govet problems
* Addressing govet problems
* Addressing govet problems
* Addressing PR comments (and removing TODO)
* Addressing PR comments
* Fixed broken test
* Removing unnecesary i18n translation string
