* MM-30892: Avoid unnecessary table scan in AnalyticsPostCount.
[AnalyticsPostCount](d9e8402dc2/store/sqlstore/post_store.go (L1513-L1541)) unconditionally joins on the `Channels` table even when not filtering to a team.
This is a proposal to stop doing this, reducing this to a simple index query when no team is specified. Note that this is technically a subtle semantic difference, since orphaned posts (from deleted or invalid channel ids) are currently excluded. But I submit the benefits outweigh the technical differences here.
Fixes: https://mattermost.atlassian.net/browse/MM-30892
* Update store/sqlstore/post_store.go
Co-authored-by: Claudio Costa <cstcld91@gmail.com>
* Update store/sqlstore/post_store.go
Co-authored-by: Claudio Costa <cstcld91@gmail.com>
Co-authored-by: Claudio Costa <cstcld91@gmail.com>
* Add search engine support for files
* Fixing i18n
* Fix golangci-lint
* Fix consistency problem in the Search receiver functio of the SqlFileStore
* Fixing some tests
* Fixing test
* Apply suggestions from code review
Co-authored-by: Mario de Frutos Dieguez <mario@defrutos.org>
* Addressing PR review comments
* Removing some empty lines
* Address PR review comments
* Fixing problem after merge master
* Fixing spelling problem
* Add missed translations
* Fixing certain global variable usages after merge master
* Fixing some constants usage
* Fixing goimports order
Co-authored-by: Mario de Frutos Dieguez <mario@defrutos.org>
* format using `goimports -local github.com/mattermost/mattermost-server/v5 -w`
* added goimports lint check to .golangci.yml
* format using `goimports -local github.com/mattermost/mattermost-server/v5 -w` for a corner case
* make app-layers, *-mocks and store-layers for ci check
Co-authored-by: Mahmudul Haque <mahmudulhaque@protonmail.com>
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* MM-31063: Change constants to use CamelCase
* store package
* change allcaps to camel case (#16615)
* New tools.mod
Co-authored-by: Ibrahim Serdar Acikgoz <serdaracikgoz86@gmail.com>
* MM-31356: Add a minimum required version check for Postgres
To keep conformance with our failing fast and obvious philosophy,
we add a check to prevent Mattermost server from starting
if the postgres version is below 10.0.
This gives customers a chance to upgrade their database before upgrading
their Mattermost version, than to run into weird compatibility issues
after they have finished the upgrade.
https://mattermost.atlassian.net/browse/MM-31356
```release-note
NONE
```
* fix lint errors
* Use a function to pretty-print version string
* rectify comment
* Allow uploading 10 files per post
* Increase Post.Fileids column size to fit 10 files
* Add Post.Fileids column changes to DB v531
* Fix current_schema_version
Co-authored-by: Claudio Costa <cstcld91@gmail.com>
* Update store/sqlstore/upgrade.go
Co-authored-by: Claudio Costa <cstcld91@gmail.com>
* Fix schema version
Co-authored-by: Diogo Lima Nicolau <diogo.lima.nicolau@cern.ch>
Co-authored-by: Harrison Healey <harrisonmhealey@gmail.com>
Co-authored-by: Claudio Costa <cstcld91@gmail.com>
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* Implement unzip function
* Implement FileSize method
* Implement path rewriting for bulk import
* Small improvements
* Add ImportSettings to config
* Implement ListImports API endpoint
* Enable uploading import files
* Implement import process job
* Add missing license headers
* Address reviews
* Make path sanitization a bit smarter
* Clean path before calculating Dir
* [MM-30008] Add mmctl support for file imports (#16301)
* Add mmctl support for import files
* Improve test
* Remove unnecessary handlers
* Use th.TestForSystemAdminAndLocal
* Make nouser id a constant
* Removing supplier concept from the sql store
* Removing other metions to supplier
* Fixing gofmt
* Fixing gofmt
* Renaming NewSqlStore to New
* Fixing tests
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* Syncronize access when updating config on search layer
* Simplify validity of race condition on tests
* Add License header on new file
* Use atomic.Value for config access on search layer
* Apply PR suggestions
* Add logic to skip duplicate table errors when initializing db
* Update store/sqlstore/supplier.go
Co-authored-by: Gabe Jackson <gabe@coffeepowered.co>
* Handle mysql and pg table already exist errors
* Apply PR suggestions
* Fix typo on IsDuplicate
* Update mysql error link to point to the official one
Co-authored-by: Gabe Jackson <gabe@coffeepowered.co>
* Make UpdateSidebarCategories return the original categories
* MM-20897 Add category muting
* Prevent muting the DMs category
* Fix muted state not being stored in the database
* Address feedback
* Address some feedback
* Fix unit tests
* MM-20897 Mute/unmute channels in the database in bulk
* Satisfy golangci-lint
* MM-28067: Optimize app server startup in tests
For every test, we would wipe out the database and then start
the server again. This would run through all the migrations
and create new rows in Systems and Roles table every time.
As a result, this was consuming a lot of setup time for every test.
We optimize this by preloading the DB with dummy data in the roles
and systems table so that the server code just skips over those migrations.
This is completely forward compatible and adding new migrations does not
need to generate the sql files again. Only in case of schema changes to the
roles or systems table, this would need to be done.
It is unlikely the Systems table schema will get changed. The Roles table
might change in future but it's a comparatively rare event. Given the reduction
in CI runtime we are seeing, it's a worthy optimization.
We also apply some more optimizations:
- Coalesce multiple UpdateConfig calls into a single one. Each UpdateConfig call
has to do a json marshal which would take precious CPU cycles. It's much more
efficient to do everything in a single call.
- Remove unnecessary debug.FreeOSMemory in reload config. This was an artifact
from old days and is no longer required.
Numbers:
Results show a full **2 minutes** shaved off the test runtime. Earlier, tests
would take around 16 minutes. Now they take 14 minutes.
```release-note
NONE
```
* fix app package
* Refactor apply multi role filters and add role filters to get all profiles
* Add some tests
* Fix tests
* Fix lint
* Trigger CI
* Rename param to make more sense
* Tie get filtered user stats to usermanagement read users
* Dont filter out other system roles when searching for team members or team admins only filter out system admins
* add new permissions
* add migration
* fix test
* remove system roles as default permissions
* implement changes discussed with dennis
* add read only and fix i18n
* use model consts instead of strings
* turn the permissions into pseudo constants
* Update read only default permissions
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
Co-authored-by: Hossein Ahmadian-Yazdi <hyazdi1997@gmail.com>
* MM-30026: Use DB master when getting team members from a session
A race condition happens when the read-replica isn't updated yet
by the time a session expiry message reaches another node in the cluster.
Here is the sequence of events that can cause it:
- Server1 gets any request which has to wipe session cache.
- The SQL query is written to DB master, and a cluster message is propagated
to clear the session cache for that user.
- Now before the read-replica is updated with the master’s update,
the cluster message reaches Server2. The session cache is wiped out for that user.
- _Any random_ request for that user hits Server2. Does NOT have to be
the update team name request. The request does not find the value
in session cache, because it’s wiped off, and picks it up from the DB.
Surprise surprise, it gets the stale value. Sticks it into the cache.
By now, the read-replica is updated. But guess what, we aren’t going to
ask the DB anymore, because we have it in the cache. And the cache has the stale value.
We use a temporary approach for now by introducing a context in the DB calls so that
the useMaster information can be easily passed. And this has the added advantage of
reusing the same context for future DB calls in case it happens. And we can also
add more context keys as needed.
A proper approach needs some architectural changes. See the issue for more details.
```release-note
Fixed a bug where a session will hold on to a cached value
in an HA setup with read-replicas configured.
```
* incorporate review comments
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* MM-30300: Add coalesce to get null content as empty string on FileInfo store
* Making explicit the fields get from the FileInfo table
* Addressing PR review comments
* Fix the team and channel filtering UT to include empty team or channel
* Fix tests that were failing before this change
Once we've activated the team/channels filter tests for PostgreSQL
and MySQL there are some tests failing so this changes fixes them
* Disable team filtering tests for DBs by now
We have a discrepancy between DB search and ES/Bleve on how to filter
teams when you have users in both teams:
- DB when filtering by one team and searching by another returns users that are in both teams
- Bleve and ES returns empty
Co-authored-by: Mario de Frutos <mario@defrutos.org>
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* MM-29980: Optimize profilesInChannels cache to fast path
We add one more message type to the fast path- profiles in channels. There
are 2 primary reasons for this:
- This is not really a new model type, but just a map of users. And users already use
the fast path. So we can get some more gains without really investing much more code.
- A more important reason is that with the upcoming striped mutex changes, we will get
a higher throughput at the cost of a bit more CPU utilization. The reason being that
since less amount of time will be spent in lock-contention, the CPU is free to do more
stuff. So this change is to counter that increase.
As usual, this gives much better performance than the original decoder.
Micro-benchmark results
```
name old time/op new time/op delta
LRU/UserMap=new-8 16.6µs ± 3% 3.9µs ± 4% -76.15% (p=0.000 n=10+10)
name old alloc/op new alloc/op delta
LRU/UserMap=new-8 4.78kB ± 0% 2.74kB ± 0% -42.65% (p=0.000 n=10+10)
name old allocs/op new allocs/op delta
LRU/UserMap=new-8 38.0 ± 0% 30.0 ± 0% -21.05% (p=0.000 n=10+10)
```
https://mattermost.atlassian.net/browse/MM-29980
Here are some results from a load test. The comparison is done with a 2 node cluster; one running master
and one running with this patch so that it's easier to compare. The total users are 2000.
<See PR>
```release-note
NONE
```
* Fix gofmt
* Trigger CI
* Add the content field to FileInfo
* Fixing the upgrade code
* Trying to fix the text-scheme
* Fixing test-schema
* Fixing test-schema
* Moving the migration to the next version
