Remote Cluster Service
- provides ability for multiple Mattermost cluster instances to create a trusted connection with each other and exchange messages
- trusted connections are managed via slash commands (for now)
- facilitates features requiring inter-cluster communication, such as Shared Channels
Shared Channels Service
- provides ability to shared channels between one or more Mattermost cluster instances (using trusted connection)
- sharing/unsharing of channels is managed via slash commands (for now)
* MM-31094: Adds tooling to develop and test using a MySQL instance with replication lag. Adds some lazy lookups to fallback to master if results are not found.
* MM-31094: Removes mysql-read-replica from default docker services.
* MM-31094: Switches (store..SessionStore).Get and (store.TeamStore).GetMember to using context.Context.
* MM-31094: Updates (store.UsersStore).Get to use context.
* MM-31094: Updates (store.PostStore).Get to use context.
* MM-31094: Removes feature flag and config setting.
* MM-31094: Rolls back some master reads.
* MM-31094: Rolls a non-cache read.
* MM-31094: Removes feature flag from the store.
* MM-31094: Removes unused constant and struct field.
* MM-31094: Removes some old feature flag references.
* MM-31094: Fixes some tests.
* MM-31094: App layers fix.
* MM-31094: Fixes mocks.
* MM-31094: Don't reparse flag.
* MM-31094: No reparse.
* MM-31094: Removed unused FeatureFlags field.
* MM-31094: Removes unnecessary feature flags variable declarations.
* MM-31094: Fixes copy-paste error.
* MM-31094: Fixes logical error.
* MM-30194: Removes test method from store.
* Revert "MM-30194: Removes test method from store."
This reverts commit d5a6e8529bd5f4d993824c828e239d009b05e567.
* MM-31094: Conforming to make's strange syntax.
* MM-31094: Configures helper for read replica with option.
* MM-31094: Adds some missing ctx's.
* MM-31094: WIP
* MM-31094: Updates test names.
* MM-31094: WIP
* MM-31094: Removes unnecessary master reads.
* MM-31094: ID case changes out of scope.
* MM-31094: Removes unused context.
* MM-31094: Switches to a helper. Removes some var naming changes. Fixes a merge error.
* MM-31094: Removes SQLITE db driver ref.
* MM-31094: Layer generate fix.
* MM-31094: Removes unnecessary changes.
* MM-31094: Moves test method.
* MM-31094: Re-add previous fix.
* MM-31094: Removes make command for dev.
* MM-31094: Fix for login.
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* Adding bulk-indexing and improving a bit the name indexing for bleve and elasticsearch
* Update services/searchengine/bleveengine/bleve.go
Co-authored-by: Mario de Frutos Dieguez <mario@defrutos.org>
* Update store/sqlstore/file_info_store.go
Co-authored-by: Mario de Frutos Dieguez <mario@defrutos.org>
* Update store/sqlstore/file_info_store.go
Co-authored-by: Mario de Frutos Dieguez <mario@defrutos.org>
* Adding tests requested in the PR review
* fixing tests
* Adding a feature flag to avoid indexing files before the feature is released
* Fixing i18n
Co-authored-by: Mario de Frutos Dieguez <mario@defrutos.org>
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* MM-30882: Fix read-after-write issue for demoting user
In (*App).DemoteUserToGuest, we would demote a user, and then immediately
read it back to do future operations from the user. This reading back
of the user had the effect of sticking the old value into the cache
after which it would never be updated.
There was another issue along with this, which was when the invalidation
message would broadcast across the cluster, it would hit the cache invalidation
problem where an unrelated store call would miss the cache because
it was invalidated, and then again read from replica and stick the old value.
To fix all these, we return the new value directly from the store method
to avoid having the app to read it again.
And we add a map in the localcache layer which tracks invalidations made,
and then switch to use master if it's true.
The core change is fairly limited, but due to changing the store method signatures,
a lot of code needed to be updated to pass "context.Background". Therefore the PR
just "appears" to be big, but the main changes are limited to app/user.go,
sqlstore/user_store.go and user_layer.go
https://mattermost.atlassian.net/browse/MM-30882
```release-note
Fix an issue where demoting a user to guest would not take effect in
an environment with read replicas.
```
* Fix concurrent map access
* Fixing mistakes
* fix tests
* Optimise creation of dm
* Handle direct channels with the same user
* Cover GetMany with specs and add it on tha cache layer as well
* Fix specs by handling user dming themselves
* Apply PR suggestions
* Apply PR suggestions
* Use require.NoError instead of require.Nil on userstore test
* Improve readability of GetOrCreateDirectChannel
* Apply PR suggestions
* Update layers
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* Add search engine support for files
* Fixing i18n
* Fix golangci-lint
* Fix consistency problem in the Search receiver functio of the SqlFileStore
* Fixing some tests
* Fixing test
* Apply suggestions from code review
Co-authored-by: Mario de Frutos Dieguez <mario@defrutos.org>
* Addressing PR review comments
* Removing some empty lines
* Address PR review comments
* Fixing problem after merge master
* Fixing spelling problem
* Add missed translations
* Fixing certain global variable usages after merge master
* Fixing some constants usage
* Fixing goimports order
Co-authored-by: Mario de Frutos Dieguez <mario@defrutos.org>
* Make UpdateSidebarCategories return the original categories
* MM-20897 Add category muting
* Prevent muting the DMs category
* Fix muted state not being stored in the database
* Address feedback
* Address some feedback
* Fix unit tests
* MM-20897 Mute/unmute channels in the database in bulk
* Satisfy golangci-lint
* MM-30026: Use DB master when getting team members from a session
A race condition happens when the read-replica isn't updated yet
by the time a session expiry message reaches another node in the cluster.
Here is the sequence of events that can cause it:
- Server1 gets any request which has to wipe session cache.
- The SQL query is written to DB master, and a cluster message is propagated
to clear the session cache for that user.
- Now before the read-replica is updated with the master’s update,
the cluster message reaches Server2. The session cache is wiped out for that user.
- _Any random_ request for that user hits Server2. Does NOT have to be
the update team name request. The request does not find the value
in session cache, because it’s wiped off, and picks it up from the DB.
Surprise surprise, it gets the stale value. Sticks it into the cache.
By now, the read-replica is updated. But guess what, we aren’t going to
ask the DB anymore, because we have it in the cache. And the cache has the stale value.
We use a temporary approach for now by introducing a context in the DB calls so that
the useMaster information can be easily passed. And this has the added advantage of
reusing the same context for future DB calls in case it happens. And we can also
add more context keys as needed.
A proper approach needs some architectural changes. See the issue for more details.
```release-note
Fixed a bug where a session will hold on to a cached value
in an HA setup with read-replicas configured.
```
* incorporate review comments
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* Add the content field to FileInfo
* Fixing the upgrade code
* Trying to fix the text-scheme
* Fixing test-schema
* Fixing test-schema
* Moving the migration to the next version
* Migration completed
* Fix tests
* Reduce to one line
* Fix: change to plain error
* Fix imports
* Trigger CI
* Fix i18
* Fix merge with master
* Trigger CI
* add warning as success result
* revert config changes
* add unit test, update unit test
* add a couple more tests
* update store layers
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
