Summary
- Attempt at fixing the AutocompleteInTeam/empty_string test which is flaky on MySQL-5.6. This is likely an index_merge_intersection optimization issue. See https://community.mattermost.com/core/pl/uzgygwa9e3d4pns14ayfs7mobw
for more discussion. The fix here simply modifies the where clause to change the query plan so index_merge_intersection is not used, while also not introducing a temporary table or a filesort.
Ticket Link
- https://mattermost.atlassian.net/browse/MM-26837
* MM-27040: Fix flaky tests due to same UpdateAt timestamp
Update queries are not guaranteed to always update the row.
Since we are only bumping the UpdateAt timestamps, if 2 update
requests hit concurrently within the same millisecond, then one
is bound to fail.
We fix all cases in the codebase where we were updating the UpdateAt
field to not check for count != 1, but rather count > 1.
A similar fix was already done in 3f46cf6f60.
* Remove incorrect test
* MM-26410 Allow moving channels into Favorites when they're favorited in prefs
* MM-26410 Fix management of Favorites category when updating preferences
* MM-26410 Add management of Favorites category when deleting preferences
* Address feedback 1
* Remove WHERE (1=1) from query
* Remove unnecessary sq.Expr
* Rewrite query to use left join
* Remove redundant where statement and add some more tests
* Fix linting issues
* Rename addChannelToFavoritesCategory to addChannelToFavoritesCategory
* Remove use of Gorp in ChannelStore.GetAllChannelMembersForUser
In order to improve performance, we're removing gorp calls in the
most used queries in the platform and this is one of them
* MM-26753 Change CreateInitialSidebarCategories to only take a user ID
* MM-26753 Create initial sidebar categories on demand if migration hasn't ran
* Wait for sidebar categories to be loaded in case of replication lag
* MM-26541 parse channel member counts using like instead of using JSON as mysql versions do not support json extract
* Use a constant for default timezone
* Update wording and comments
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* parent 48256721c4 (#14358)
author Eli Yukelzon <reflog@gmail.com> 1585814774 +0300
committer Eli Yukelzon <reflog@gmail.com> 1589111022 +0300
Sidebar caregories implemented
Apply suggestions from code review
Co-authored-by: Ibrahim Serdar Acikgoz <serdaracikgoz86@gmail.com>
Update store/sqlstore/channel_store.go
Co-authored-by: Ibrahim Serdar Acikgoz <serdaracikgoz86@gmail.com>
Update store/sqlstore/channel_store.go
Co-authored-by: Ibrahim Serdar Acikgoz <serdaracikgoz86@gmail.com>
code review suggestions
status messages
edge case
bugs...
timeout reverse
* MM-25126 Add a separate default sorting method for categories (#14575)
* MM-25158 Add user to initial sidebar categories when joining team (#14570)
* MM-25281 Place new categories in the correct position (#14609)
* MM-25277 Return channels that aren't in a category as part of the Channels/Direct Messages categories (#14601)
* MM-25276 Remove categories when leaving a team (#14600)
* Remove categories when leaving a team
* layers
* corrected cleanup function
* lint
* .
* corrected errors in postgres
* .
* MM-25280 Ensure that the "update category order" API call only contains real category IDs and isn't missing any IDs (#14626)
* Ensure that the "update category order" API call only contains real category IDs and isn't missing any IDs
* tests
* correct status code
* MM-25278 Change "update category" API to return 400 when changing unsupported fields (#14599)
* MM-25279 Change category migration to only populate channels in Favorites (#14627)
* MM-25157 Add API to delete custom categories (#14574)
* MM-25157 Add API to delete custom categories
* get categories fix
* maxorder fix
* Use correct websocket event when deleting category
* Fix tests and remove debug code
* Actually use the right websocket event this time
* test cleanup
* Update test for new category order
Co-authored-by: Eli Yukelzon <reflog@gmail.com>
* MM-24914 Various fixes for sidebar channel handling (#14756)
* Fix checking for channel membership when reordering channels
* Remove unique constraint on SidebarCategories
* Set column sizes for SidebarCategories and SidebarChannels tables
* Allow changing the sorting method for non-DM categories
* Fix nil pointers in error handling
* Fix orphaned channels from other team being returned in Channels category
* Fix non-orphaned channels being duplicated in the Channels category
* Remove unique constraint on SidebarChannels
* Fix category/name of favorites preferences
* Fix testSidebarChannelsMigration
* Rename err to nErr and appErr to err
* Fix channel order returned by GetSidebarCategories on MySQL
* Fix adding/removing favorites preferences
* Remove leftover TODO
* Change SidebarCategoryType enums to use full names (#14786)
* Change SidebarCategoryType enums to use full names
* Fix Channels constant
* Remove leftover debug code
* MM-24914 Fix updateCategory endpoint returning the wrong type (#14795)
* MM-24914 Make some changes to UpdateSidebarCategories (#14806)
* Fix orphaned DMs not always being returned
* MM-24914 Make some changes to UpdateSidebarCategories
* Run updateSidebarCategoryOrderT in a transaction
* Fix deleting SidebarChannels based on order of arguments to UpdateSidebarCategories
* bump for api testing
* bump for api testing
* Change CreateInitialSidebarCategories to return a plain error
* Change MigrateSidebarCategories to return a plain error
* Remove usage of UpdateColumns when updating sidebar categories (#14843)
* Remove usage of UpdateColumns when changing category order
* Add a random test case
* Remove usage of UpdateColumns when updating sidebar categories (#14843)
* Remove usage of UpdateColumns when changing category order
* Add a random test case
* Remove usage of UpdateColumns when updating sidebar categories (#14843)
* Remove usage of UpdateColumns when changing category order
* Add a random test case
* MM-26343 Make CreateInitialSidebarCategories idempotent (#14870)
* Fix bad merge
* Fix another bad merge
* Fix unintentionally removed i18n string
Co-authored-by: Eli Yukelzon <reflog@gmail.com>
* MM-19548: Add a deadlock retry function for SaveChannel
A deadlock has been seen to occur in the upsertPublicChannelT method
during bulk import.
Here is a brief excerpt:
*** (1) TRANSACTION:
TRANSACTION 3141, ACTIVE 1 sec inserting
INSERT INTO
PublicChannels(Id, DeleteAt, TeamId, DisplayName, Name, Header, Purpose)
VALUES
(?, ?, ?, ?, ?, ?, ?)
ON DUPLICATE KEY UPDATE
DeleteAt = ?,
TeamId = ?,
DisplayName = ?,
Name = ?,
Header = ?,
Purpose = ?
*** (1) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 76 page no 4 n bits 104 index Name of table `mydb`.`PublicChannels` trx id 3141 lock_mode X locks gap before rec insert intention waiting
** (2) TRANSACTION:
TRANSACTION 3140, ACTIVE 1 sec inserting
mysql tables in use 1, locked 1
5 lock struct(s), heap size 1136, 3 row lock(s), undo log entries 2
MySQL thread id 50, OS thread handle 140641523848960, query id 3226 172.17.0.1 mmuser update
INSERT INTO
PublicChannels(Id, DeleteAt, TeamId, DisplayName, Name, Header, Purpose)
VALUES
(?, ?, ?, ?, ?, ?, ?)
ON DUPLICATE KEY UPDATE
DeleteAt = ?,
TeamId = ?,
DisplayName = ?,
Name = ?,
Header = ?,
Purpose = ?
*** (2) HOLDS THE LOCK(S):
RECORD LOCKS space id 76 page no 4 n bits 104 index Name of table `mydb`.`PublicChannels` trx id 3140 lock_mode X locks gap before rec
*** (2) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 76 page no 4 n bits 104 index Name of table `mydb`.`PublicChannels` trx id 3140 lock_mode X locks gap before rec insert intention waiting
*** WE ROLL BACK TRANSACTION (1)
Following is my analysis:
From the deadlock output, it can be seen that it's due to a gap lock.
And that's clear because the index is Name which is a multi-column index using Name and TeamId.
But interestingly, both transactions seem to be inserting the same data, which is what is puzzling me.
The multi-column index on Name and TeamId will guarantee that they are always unique. And from looking at the code,
it does not seem possible to me that it will try to insert the same data from 2 different transactions.
But even if they do, why does tx 2 try to acquire the same lock again when it already has that ?
Here is what I think the order of events happening
Tx 2 gets a gap lock.
Tx 1 tries to get the same gap lock.
Tx 2 tries to again get the same gap lock ?
The last step is what is puzzling me. Why does an UPSERT statement acquire 2 gap locks ? From my reading of https://dev.mysql.com/doc/refman/8.0/en/innodb-locks-set.html:
> INSERT ... ON DUPLICATE KEY UPDATE differs from a simple INSERT in that an exclusive lock rather than a shared lock is placed on the row to be updated when a duplicate-key error occurs. An exclusive index-record lock is taken for a duplicate primary key value. An exclusive next-key lock is taken for a duplicate unique key value.
From what I understand, the expectation is that there will be one X lock and one gap lock is taken.
But that's not what the deadlock output seems to say.
The general advice on the internet seems to be that deadlocks will happen and not all of them can be understood.
For now, we add a generic deadlock retry function at the store package which can be reused by other queries too.
P.S.: This is a verbatim copy of my investigation posted at https://dba.stackexchange.com/questions/268652/mysql-deadlock-upsert-query-acquiring-gap-lock-twice
Testing:
This is ofcourse hard to test because it is impossible to reproduce this. I have tested this by manually returning an error
and confirming that it indeed retries.
WARN[2020-06-19T11:18:24.9585676+05:30] A deadlock happened. Retrying. caller="sqlstore/channel_store.go:568" error="Error 1213: mydeadlock"
WARN[2020-06-19T11:18:24.959158+05:30] A deadlock happened. Retrying. caller="sqlstore/channel_store.go:568" error="Error 1213: mydeadlock"
WARN[2020-06-19T11:18:24.9595072+05:30] A deadlock happened. Retrying. caller="sqlstore/channel_store.go:568" error="Error 1213: mydeadlock"
WARN[2020-06-19T11:18:24.9595451+05:30] Deadlock happened 3 times. Giving up caller="sqlstore/channel_store.go:579"
ERRO[2020-06-19T11:18:24.9596426+05:30] Unable to save channel. caller="mlog/log.go:175" err_details="Error 1213: mydeadlock" err_where=CreateChannel http_code=500 ip_addr="::1" method=POST path=/api/v4/channels request_id=745bsj13b7f6mnmsbn3t97grbw user_id=xcof1ipipbrfxpfjf6x4p6kx9e
* Fix tests
* Address review comments
* Address review comments
* Add forgotten test
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* MM-24674 Update channel members by group to look at distinct autoTimezone and manualTimezone.
Instead of just doing a blanket distinct on the column since that returns more entries than expected
* Use JSON extract instead of parsing text
* Use single quotes in mysql query too
* Dont need to prepend users on unambiguous column
* Use json extract instead of shorthand
* Dont count timezone if timezone default length
* CI
Co-authored-by: mattermod <mattermod@users.noreply.github.com>
* Implement prefix/suffix search on teams page
* Make Channel Page prefix/suffix search as well
* address PR comments
* add tests
* fix styling
* take postgres into account
* add more api tests
* update test
* add team store test
* write store test for team and channel
* fix lint
* update description
* revert go mod
Co-authored-by: mattermod <mattermod@users.noreply.github.com>
* Migrate GetDeleted method from ChannelStore to return error interface
* Moving i18n translations to the correct place
Co-authored-by: Jesús Espino <jespinog@gmail.com>
* Migrate GetAllChannels and GetAllChannelsCount from ChannelStore to return plain errors
* Moving i18n translations to the correct place
Co-authored-by: Jesús Espino <jespinog@gmail.com>
* MM-24135: Migrate AppError from SaveChannel/channel_store.go
This is the first POC of migration of store app errors to plain error.
We create a few basic error types in the store package and use
them to return the errors from store methods. In the app layer,
we inspect the error and re-create the exact app errors. This lets
us preserve the same error content, but yet move to plain errors.
Since this is a gradual migration, this means that the error inspection
code will be duplicated across the app layer whenever a store method
is invoked. But all of that should go away once we start propagating
the errors higher up the hierarchy.
There have been a significant amount of changes in the storetest and searchtest
layer, primarily because we have to rename the err variable now that it is of
a different type.
* Addressed review comments
* MM-24132: Migrate AppError from SaveDirectChannel/channel_store.go
This PR migrates 2 new methods SaveDirectChannel and CreateDirectChannel
to return error instead of AppError.
We also need to handle the error internally in SaveMultipleMember for now
until that is migrated too.
* Fix layers
Co-authored-by: mattermod <mattermod@users.noreply.github.com>
* MM-24135: Migrate AppError from SaveChannel/channel_store.go
This is the first POC of migration of store app errors to plain error.
We create a few basic error types in the store package and use
them to return the errors from store methods. In the app layer,
we inspect the error and re-create the exact app errors. This lets
us preserve the same error content, but yet move to plain errors.
Since this is a gradual migration, this means that the error inspection
code will be duplicated across the app layer whenever a store method
is invoked. But all of that should go away once we start propagating
the errors higher up the hierarchy.
There have been a significant amount of changes in the storetest and searchtest
layer, primarily because we have to rename the err variable now that it is of
a different type.
* Addressed review comments
* Made all appError origins to be CreateChannel
* Remove typed internal error
* Fix translations
* fix layer generation
Co-authored-by: mattermod <mattermod@users.noreply.github.com>
* MM-23264 Add api endpoint for get groups with members in channel
Add store tests
Add tests for api func
Gofmt
Apply changes from code review
* MM-23264 Make store layers
* MM-23264 Check read permission on channel member counts
* Trigger CI
* Members Save Multiple
* Multiple Channel Members implemented
* Working version with member multi save and multi update
* Adding tests to the save multiple team members store methods
* Tests passing for SaveMember
* Adding remove members tests
* Working on channel store tests
* More work on testing properly SaveMember and SaveMultipleMembers on channels
* More work on testing properly SaveMember and SaveMultipleMembers on teams
* Adding tests for update multiple members for team
* Adding tests for update multiple members for channel
* Fixing some tets
* Fixing tests
* Fixing tests
* Fixing i18n
* Addressing govet problems
* Addressing govet problems
* Addressing govet problems
* Addressing govet problems
* Addressing PR comments (and removing TODO)
* Addressing PR comments
* Fixed broken test
* Removing unnecesary i18n translation string
* MM-22212: Read non-moderated permissions from higher-scoped scheme.
* MM-2212: Corrects test count in comment.
* MM-22212: Adds godoc comment.
* MM-2212: Switches to the channel roles check in a few more places.
* MM-22212: Refactors and fixes.
* MM-22212: Reverts change, no longer required.
* MM-22212: Removes translation.
* MM-22212: Un-comments merged new permission.
* MM-22212: Un-comments merged new permission.
* MM-22212: Performance tweak.
* MM-22212: Fixes some fmting.
* MM-22212: Add unit test for newly-added store methods.
* MM-22212: Renames app method.
* MM-22212: Re-uses existing function to find string in slice.
* MM-22212: Keeps 'higher-scoped' terminology for consistency.
* MM-22212: Refactors based on PR feedback.
* MM-22212: Fix for some bad merging.
* MM-22212: Renamed some things.
* MM-22212: Use an 'else' instead of a 'continue' for readability.
* MM-22212: Caches (*SqlRoleStore).ChannelRolesUnderTeamRole.
* MM-22212: Adds mock to new cache store.
* MM-22212: Adds missing open tracing app layer methods.
* MM-22212: Adds migration to add moderated permissions to channel_admin if present on channel_user.
* MM-22212: Migrates team schemes. Removes unused AppError.
* MM-22212: Fix for for if.
* MM-22212: Fixes iterator.
* MM-22212: Updates open tracing generated methods.
* MM-22212: Fix mocks.
* MM-22212: Change migration key name.
* MM-22212: Switched to data structure from other branch.
* MM-22212: Fixes tests after adding 'use_channel_mentions' to the channel_admin role.
* MM-22212: Adds tracking of channel moderation.
* Revert "MM-22212: Adds tracking of channel moderation."
This reverts commit 23689efa22c112e4ba37f6a212535dd7ebfb63db.
* MM-22212: Switch some functions to methods and vice versa.
* MM-22212: Fix for refactor bug not notifiying websocket about changed role.
* MM-22212: Adds test for public/private 'manage_members' handling.
* MM-22122 Fix manage channel members edge case for public and private channels (#14049)
* MM-22212: Adds moderated permission to team_admin.
* MM-22212: Updates migration.
* MM-22212: Revert unnecessary update to default roles.
* Add channel scheme updated event when channel scheme is deleted or created (#14057)
* MM-22212: Adds newline.
* MM-22212: Migration fix.
* MM-22212: Fix for migration.
* MM-22212: Test fix.
Co-authored-by: Farhan Munshi <3207297+fm2munsh@users.noreply.github.com>
* WIP
* Adding bleve to go modules
* WIP
* Adding missing files from searchengine implementation
* WIP
* WIP
* WIP
* WIP
* WIP
* WIP
* User and channel indexing and searches implemented
* Make bleve tests run with in-memory indexes
* Implement post index and deletion tests
* Initial commits for the search layer
* Removing unnecesary indexing
* WIP
* WIP
* More fixes for tests
* Adding the search layer
* Finishing the migration of searchers to the layer
* Removing unnecesary code
* Allowing multiple engines active at the same time
* WIP
* Add simple post search
* Print information when using bleve
* Adding some debugging to understand better how the searches are working
* Making more dynamic config of search engines
* Add post search basics
* Adding the Purge API endpoint
* Fixing bleve config updates
* Adding missed file
* Regenerating search engine mocks
* Adding missed v5 to modules imports
* fixing i18n
* Fixing some test around search engine
* Removing all bleve traces
* Cleaning up the vendors directory and go.mod/go.sum files
* Regenerating timer layer
* Adding properly the license
* Fixing govet shadow error
* Fixing some tests
* Fixing TestSearchPostsFromUser
* Fixing another test
* Fixing more tests
* Fixing more tests
* Removing SearchEngine redundant text from searchengine module code
* Fixing some reindexing problems in members updates
* Fixing tests
* Addressing PR comments
* Reverting go.mod and go.sum
* Addressing PR comments
* Fixing tests compilation
* Fixing govet
* Adding search engine stop method
* Being more explicit on where we use includeDeleted
* Adding GetSqlSupplier test helper method
* Mocking elasticsearch start function
* Fixing tests
Co-authored-by: Miguel de la Cruz <miguel@mcrx.me>
Co-authored-by: mattermod <mattermod@users.noreply.github.com>
* MM-20644: Add users to teams as a SchemeAdmin based on a new configuration field on GroupTeams and GroupChannels records.
* MM-20644: Adds SchemeAdmin to mapping of the GroupSyncable struct fields.
* MM-2064: Adds test to ensure SchemeAdmin field value is mapped.
* MM-20644: Adds missing index creation for fresh DBs.
* MM-20644: Duplicates UpdateMembersRole across Team and Channel stores. Adds tests.
* MM-20644: Fixes some old method name references.
* MM-20644: Moves variable declaration; removes Println statement.
* MM-20644: Use a SQL query instead of two to update Team and Channel members.
* MM-20644: Fixes tests; updates query.
* MM-20644: Fix permission check for patching a group syncable.
* MM-20644: Fixes test for change of permissions verification in group patch API request.
* MM-20644: Fix for ORM select vs insert.
* MM-20644: Linting fixes.
* MM-20644: Fixes some tests.
* MM-20644: Skips changing the role of guests.
* MM-20644: Added syncableID filtering
* MM-21211: Optimize Channelstore.UpdateLastViewedAt
For postgres, we use CTEs to do both queries at one
and return the data.
For mysql, CTEs aren't available until version 8. So we just optimize the
existing query.
- We use IN instead of multiple OR conditions for the channel membership check.
According to mysql https://dev.mysql.com/doc/refman/5.7/en/comparison-operators.html#operator_in:
If no type conversion is needed for the values in the IN() list,
they are all constants of the same type, and expr can be compared to each of them
as a value of the same type (possibly after type conversion), an optimization takes place.
The values the list are sorted and the search for expr is done using a binary search,
which makes the IN() operation very quick.
- We use the previous column LastViewedAt to replace the value of LastUpdateAt
instead of doing a switch case again.
https://dev.mysql.com/doc/refman/5.6/en/ansi-diff-update.html
If you access a column from the table to be updated in an expression, UPDATE uses the
current value of the column.
Results (ran with getchannel profile):
Postgres:
| Metric | Baseline | Actual | Delta | Delta % |
| --- | --- | --- | --- | --- |
| Hits | 1594 | 1574 | -20 | -1.25%
| Error Rate | 0.00% | 0.00% | 0% | 0% |
| Mean Response Time | 7.08ms | 6.36ms | -0.72ms | -10.20% |
| Median Response Time | 5.00ms | 4.00ms | -1.00ms | -20.00% |
| 95th Percentile | 9.00ms | 8.00ms | -1.00ms | -11.11% |
Mysql:
| Metric | Baseline | Actual | Delta | Delta % |
| --- | --- | --- | --- | --- |
| Hits | 1600 | 1566 | -34 | -2.12%
| Error Rate | 0.00% | 0.00% | 0% | 0% |
| Mean Response Time | 7.39ms | 7.02ms | -0.37ms | -4.94% |
| Median Response Time | 6.00ms | 6.00ms | 0ms | 0% |
| 95th Percentile | 10.00ms | 10.00ms | 0ms | 0% |
* Add comment on using CTEs for mysql8
* incorporate review comments
Co-authored-by: mattermod <mattermod@users.noreply.github.com>
* Refactor to use structured logging
* Properly formatted with gofmt
* created interface Cache, but construction of cache is still coupled.
* Implementing cache factories to build caches
* Simple redis implementation without error handling. Keys and values by default are string
* refactor NewLocalCacheLayer to inject cache factory
* Removed redis impl to focus on cache abstraction
* CacheFactory injected on sqlsupplier and saved in Store struct
* remove useless private method
* replace concrete declaration of lru cache to cache abstraction
* discard spaces
* Renamed factory to provider because concrete implementations of factories may hold an state (such as a redis client for example)
* refactor to include all caches in the same package cache and subpackages
* method close cache. This method will be used for concrete implementations that need to release resources (close a connection, etc)
* closing cacheprovider and releasing resources while closing sql store
* fixed merge conflict fail
* gofmt files
* remove unused property from post_store
* naming refactor to avoid stutter. Added godocs on interface
* Store doesnt know anything about the cache and provider. Cache provider will be built after loading config and injected in localCacheLayer
* fixed broken test
* cache provider initialized before RunOldAppInitialization which initializes the localcachelayer
* move statusCache to server to initialize it with the new cache provider
* update terms_service and channel_layer to have new cacheProvider
* gofmt
* Add Connect method to the cache provider
* mock cacheprovider in user_layer_test
Co-authored-by: Ben Schumacher <ben.schumacher@mattermost.com>
Co-authored-by: mattermod <mattermod@users.noreply.github.com>
