* MM-29980: Optimize profilesInChannels cache to fast path
We add one more message type to the fast path- profiles in channels. There
are 2 primary reasons for this:
- This is not really a new model type, but just a map of users. And users already use
the fast path. So we can get some more gains without really investing much more code.
- A more important reason is that with the upcoming striped mutex changes, we will get
a higher throughput at the cost of a bit more CPU utilization. The reason being that
since less amount of time will be spent in lock-contention, the CPU is free to do more
stuff. So this change is to counter that increase.
As usual, this gives much better performance than the original decoder.
Micro-benchmark results
```
name old time/op new time/op delta
LRU/UserMap=new-8 16.6µs ± 3% 3.9µs ± 4% -76.15% (p=0.000 n=10+10)
name old alloc/op new alloc/op delta
LRU/UserMap=new-8 4.78kB ± 0% 2.74kB ± 0% -42.65% (p=0.000 n=10+10)
name old allocs/op new allocs/op delta
LRU/UserMap=new-8 38.0 ± 0% 30.0 ± 0% -21.05% (p=0.000 n=10+10)
```
https://mattermost.atlassian.net/browse/MM-29980
Here are some results from a load test. The comparison is done with a 2 node cluster; one running master
and one running with this patch so that it's easier to compare. The total users are 2000.
<See PR>
```release-note
NONE
```
* Fix gofmt
* Trigger CI
* Document extractor service
* Fixing vendor modules
* Addressing PR Review comments
* Some small simplifications
* Fixing a linter complain
* simplifying a bit the code using package variables
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* MM-29525: Strip path_prefix from ListDirectory
An AWS path prefix is meant to be an implementation detail
which the calling application should not be aware of. Hence, we should
strip the path prefix when returning objects in a directory
because they anyways get applied while using the other APIs
https://mattermost.atlassian.net/browse/MM-29525
* simplify
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* MM-29572: Add Bifrost support to codebase
We use a custom transport for minio when an environment variable
indicates that Mattermost is running in cloud.
The transport is used to redirect the request to the S3 endpoint set
in the config. And the scheme is set depending on if S3SSL is set.
A custom credentials provider is needed to return empty credentials
which will be overridden by the service anyways. This is just to allow
the minio client library to work transparently without knowing that
there is something else in the middle intercepting requests.
https://mattermost.atlassian.net/browse/MM-29572
* incorporate review comments
* Update services/filesstore/s3_overrides.go
Co-authored-by: Ibrahim Serdar Acikgoz <serdaracikgoz86@gmail.com>
* Change to new env key
Co-authored-by: Ibrahim Serdar Acikgoz <serdaracikgoz86@gmail.com>
* MM-29573: Reuse s3 client amongst all requests
An http.Client is meant as a long-term object to be reused between
consecutive requests. This reduces burden on HTTP servers by reusing TCP connections
and also makes things faster by avoiding the TCP handshake for every single request.
https://mattermost.atlassian.net/browse/MM-29573
* Trigger CI
* Fix i18n
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* MM-29033: Modify Test S3 connection to handle path prefix
When a IAM user is just allowed access to a specific path prefix,
the bucket exists test will fail because that is applicable to the entire bucket.
We modify that check to just list objects under that path prefix and return
on the first non-nil error.
Another consideration was to do a StatObject, but it is only applicable to actual
objects and not at a directory level. Path prefixes are entirely logical and depends
on the underlying implementation. Therefore, ListObjects is a better choice.
https://mattermost.atlassian.net/browse/MM-29033
* Improve logic
* tmp
* Only receive
* Implement LDAP Certificate
* add diagnostics and translations
* update from code review
* pass pointer to update pict function
* pass object to first function
* remove debug log messages
* update test to add localmode test
* update lint errors
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
MM-27744 disable Zap for unit tests.
Zap has no concept of shutdown or close. Zap is only shutdown when the app exits. Not a problem for console logging, but when creating a new Zap logger that outputs to files on every unit test, that leaves no easy way to clean up until process exit. Depending on what else is running this can exhaust all file handles and cause unit tests to fail.
Zap is now disabled unit tests and uses Logr instead, regardless of config settings. `make test-server` peak file handle usage dropped from ~5K to less than 100.
* Moving diagnostics into a service
* Fixing golint checks
* Fixing tests
* Renaming from diagnostics to telemetry
* Adding missing files
* Initializing telemetry earlier in the server startup
* Fixing tests
* Adding a log for the telemetryID initialization error
* Addressing PR review comments
* Fixing merge problem
* Removing some extra Diagnostics mentions
* Making tests pass
* Adding Upgrade to Enterprise version feature
* Addressing PR review comments, and adding some minor improvements
* Add tests file
* Addressing PR comments
* fix linter checks
* Storing and exposing the upgraded from TE info
* Fix showing errors on mac
* A more appropiate status code for not-supported upgrade
* Fixing tests
* Handling permissions errors
* More server logging around upgrade failures
* Apply text changes suggested from code review
Co-authored-by: Eric Sadur <57730300+esadur@users.noreply.github.com>
* Address PR review comments
* Only allow to restart the system after an upgrade
* Verify file signature before upgrade
* Adding limit to the downloaded file
* Simplifying the upgrade binary process with backup in memory
* Fixing backup/restore mechanism for the binary file
* Improve file permissions handling
* Askin the permissions for the right place (the parent directory)
* Fixing tests
* Addressing PR review comments
* Fix license headers
* Fixing retry layer
* Making it work on windows builds
* Adding license header
* Fixing 2 tests
* Fixing tests that need UpgradeFromTE System key mock
* Extracting i18n translation
* Apply suggestions from code review
Co-authored-by: Eric Sadur <57730300+esadur@users.noreply.github.com>
* Improving how the errors are written
* Fixing another error text
* Removing unneeded translation
* Fixing upgrade status strings
* Update i18n/en.json
Co-authored-by: Eric Sadur <57730300+esadur@users.noreply.github.com>
* Fixing tests
Co-authored-by: Eric Sadur <57730300+esadur@users.noreply.github.com>
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* Implemented AmazonS3PathPrefix
* Remove unecessary method
* fix for test
* fix for test which are failing
* fix for test which are failing
* fix for test
Co-authored-by: Dusan Panic <dusan@salestrekker.com>
* [MM-26397] Take query size and order into account for Bleve
* Add a test to check post search pagination
* Add tests for checking limit when searching users
* Make pagination an independent test to discriminate DB engines
* Two missing methods to add in the channel layer
* Added delete user/channel posts methods
- Created in both search engines but only implemented in ES
- Add those methods in the search layer
- Included the PermanentDeleteByUser/Channel methods
* Two new delete documents are included in the bleve code with this
change:
- DeleteChannelPosts
- DeleteUserPosts
These two new functions delete post documents from the index-based
in the filed value provided
* MM-25710: Use an efficient cache serialization algorithm
We investigate 3 packages for selecting a suitable replacement
for gob encoding. The algorithm chosen was msgpack which gives
a decent boost over the standard gob encoding.
Any external schema dependent algorithms like protobuf, flatbuffers, avro,
capn'proto were not considered as that would entail converting the model structs
into separate schema objects and then code generating the Go structs.
It could be done theoretically at a later stage specifically for structs
which are in the hot path. This is a general solution for now.
The packages considered were:
- github.com/tinylib/msgp
- github.com/ugorji/go/codec
- github.com/vmihailenco/msgpack/v5
msgp uses code generation to generate encoding/decoding code without the reflection overhead.
Theoretically, therefore this is supposed to give the fastest performance. However, a major
flaw in that package is that it works only at a file/directory level, not at a package level.
Therefore, for structs which are spread across multiple files, it becomes near to impossible
to chase down all transitive dependencies to generate the code. Even if that's done, it fails
on some complex type like xml.Name and time.Time. (See: https://github.com/tinylib/msgp/issues/274#issuecomment-643654611)
Therefore, we are left with 2 choices. Both of them use the same underlying algorithm.
But msgpack/v5 wraps the encoders/decoders in a sync.Pool. To make a perfect apples-apples
comparison, I wrote a sync.Pool for ugorji/go/codec too and compared performance.
msgpack/v5 came out to be the fastest by a small margin.
benchstat master.txt ugorji.txt
name old time/op new time/op delta
LRU/simple=new-8 5.62µs ± 3% 3.68µs ± 2% -34.64% (p=0.000 n=10+10)
LRU/complex=new-8 38.4µs ± 2% 9.1µs ± 2% -76.38% (p=0.000 n=9+9)
LRU/User=new-8 75.8µs ± 2% 23.5µs ± 2% -69.01% (p=0.000 n=10+10)
LRU/Post=new-8 125µs ± 2% 21µs ± 3% -82.92% (p=0.000 n=9+10)
LRU/Status=new-8 27.6µs ± 1% 5.4µs ± 4% -80.34% (p=0.000 n=10+10)
name old alloc/op new alloc/op delta
LRU/simple=new-8 3.20kB ± 0% 1.60kB ± 0% -49.97% (p=0.000 n=10+10)
LRU/complex=new-8 15.7kB ± 0% 4.4kB ± 0% -71.89% (p=0.000 n=9+10)
LRU/User=new-8 33.5kB ± 0% 9.2kB ± 0% -72.48% (p=0.000 n=10+8)
LRU/Post=new-8 38.7kB ± 0% 4.8kB ± 0% -87.48% (p=0.000 n=10+10)
LRU/Status=new-8 10.6kB ± 0% 1.7kB ± 0% -83.50% (p=0.000 n=10+10)
name old allocs/op new allocs/op delta
LRU/simple=new-8 46.0 ± 0% 20.0 ± 0% -56.52% (p=0.000 n=10+10)
LRU/complex=new-8 324 ± 0% 48 ± 0% -85.19% (p=0.000 n=10+10)
LRU/User=new-8 622 ± 0% 108 ± 0% -82.64% (p=0.000 n=10+10)
LRU/Post=new-8 902 ± 0% 74 ± 0% -91.80% (p=0.000 n=10+10)
LRU/Status=new-8 242 ± 0% 22 ± 0% -90.91% (p=0.000 n=10+10)
11:31:48-~/mattermost/mattermost-server/services/cache2$benchstat master.txt vmi.txt
name old time/op new time/op delta
LRU/simple=new-8 5.62µs ± 3% 3.68µs ± 3% -34.59% (p=0.000 n=10+10)
LRU/complex=new-8 38.4µs ± 2% 8.7µs ± 3% -77.45% (p=0.000 n=9+10)
LRU/User=new-8 75.8µs ± 2% 20.9µs ± 1% -72.45% (p=0.000 n=10+10)
LRU/Post=new-8 125µs ± 2% 21µs ± 2% -83.08% (p=0.000 n=9+10)
LRU/Status=new-8 27.6µs ± 1% 5.1µs ± 3% -81.66% (p=0.000 n=10+10)
name old alloc/op new alloc/op delta
LRU/simple=new-8 3.20kB ± 0% 1.60kB ± 0% -49.89% (p=0.000 n=10+10)
LRU/complex=new-8 15.7kB ± 0% 4.6kB ± 0% -70.87% (p=0.000 n=9+8)
LRU/User=new-8 33.5kB ± 0% 10.3kB ± 0% -69.40% (p=0.000 n=10+9)
LRU/Post=new-8 38.7kB ± 0% 6.0kB ± 0% -84.62% (p=0.000 n=10+10)
LRU/Status=new-8 10.6kB ± 0% 1.9kB ± 0% -82.41% (p=0.000 n=10+10)
name old allocs/op new allocs/op delta
LRU/simple=new-8 46.0 ± 0% 20.0 ± 0% -56.52% (p=0.000 n=10+10)
LRU/complex=new-8 324 ± 0% 46 ± 0% -85.80% (p=0.000 n=10+10)
LRU/User=new-8 622 ± 0% 106 ± 0% -82.96% (p=0.000 n=10+10)
LRU/Post=new-8 902 ± 0% 89 ± 0% -90.13% (p=0.000 n=10+10)
LRU/Status=new-8 242 ± 0% 23 ± 0% -90.50% (p=0.000 n=10+10)
In general, we can see that the time to marshal/unmarshal pays off as the size of the struct
increases.
We can see that msgpack/v5 is faster for CPU but very slightly heavier on memory.
Since we are interested in fastest speed, we choose msgpack/v5.
As a future optimization, we can use a mix of msgpack and msgp for hot structs.
To do that, we would need to shuffle around some code so that for the hot struct,
all its dependencies are in the same file.
Let's use this in production for some time, watch grafana graphs for the hottest caches
and come back to optimizing this more once we have more data.
Side note: we have to do with micro-benchmarks for the time being, because all the caches
aren't migrated to cache2 interface yet. Once that's in, we can actually run some load tests
and do comparisons.
* Bring back missing import
* Fix tests
* Removing some other fake apps
* More FakeApp removed
* Removing entirely FakeApp
* Fixing some tests
* Fixing get Cluster id from get plugin status
* Fixing failing tests
* Fixing tests
* Fixing test initialization for web
* Fixing InitServer for server tests
* Fixing InitServer for server tests
* Reverting go.sum and go.mod
* Removing unneded HTMLTemplates function in App layer
* Moving back some functions to its old place to easy the review
* Moving back some functions to its old place to easy the review
* Using the last struct2interface version
* Generating store layers
* Fixing merge problems
* Addressing PR comments
* Small fix
* Fixing app tests build
* Fixing tests
* fixing tests
* Fix tests
* Fixing tests
* Fixing tests
* Fixing tests
* Moving license to server struct
* Adding some fixes to the test compilation
* Fixing cluster and some jobs initialization
* Fixing some license tests compilation problems
* Fixing recursive cache invalidation
* Regenerating app layers
* Fix test compilation
Co-authored-by: mattermod <mattermod@users.noreply.github.com>
