* Pass device ID
* dont use device id as way of detecting
* fix spelling mistake
* update layers
* fix test
* fix linting
* save schema
* put columns in correct place
* fix linting
* update
* upgrade go change
* use props
* fix stuff
* update session tests
* address PR comments
* address PR comments
This avoids the race of assigning the sema field from the start method
which runs in a goroutine.
The race that happens is
==================
WARNING: DATA RACE
Write at 0x00c003a9d1b0 by goroutine 67:
github.com/mattermost/mattermost-server/v5/app.(*PushNotificationsHub).start()
/home/agniva/mattermost/mattermost-server/app/notification_push.go:264 +0x90
Previous read at 0x00c003a9d1b0 by goroutine 69:
github.com/mattermost/mattermost-server/v5/app.(*Server).createPushNotificationsHub()
/home/agniva/mattermost/mattermost-server/app/notification_push.go:260 +0x1d5
* MM-26206: Add GroupMentions permissions in default channel admin
This permission was skipped from the default permissions function.
Which led to a bug where permissions reset would not bring back
this permission to the different roles.
* Added tests
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* Dont return an error if createDefaultChannelMemberships fails due to email domain restriction on team
* Add test ensuring sync works as intended and only skips over failed users
* Remove unneeded line
* Update wording
* Trigger CI
While resetting permissions, we were not removing old migration state residue
for the EMOJIS_PERMISSIONS_MIGRATION_KEY and GUEST_ROLES_CREATION_MIGRATION_KEY.
Therefore, during their migration, they got skipped because the code checks
if a migration key is already present or not.
To fix this, we remove the migration key just as we do for ADVANCED_PERMISSIONS_MIGRATION_KEY.
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* MM-25404: Add GetHealthScore metric for the cluster
This PR adds the necessary function to the cluster interface
so that they can be called from App code.
* Add mocks
* Remove fakeapp
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* MM-19548: Add a deadlock retry function for SaveChannel
A deadlock has been seen to occur in the upsertPublicChannelT method
during bulk import.
Here is a brief excerpt:
*** (1) TRANSACTION:
TRANSACTION 3141, ACTIVE 1 sec inserting
INSERT INTO
PublicChannels(Id, DeleteAt, TeamId, DisplayName, Name, Header, Purpose)
VALUES
(?, ?, ?, ?, ?, ?, ?)
ON DUPLICATE KEY UPDATE
DeleteAt = ?,
TeamId = ?,
DisplayName = ?,
Name = ?,
Header = ?,
Purpose = ?
*** (1) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 76 page no 4 n bits 104 index Name of table `mydb`.`PublicChannels` trx id 3141 lock_mode X locks gap before rec insert intention waiting
** (2) TRANSACTION:
TRANSACTION 3140, ACTIVE 1 sec inserting
mysql tables in use 1, locked 1
5 lock struct(s), heap size 1136, 3 row lock(s), undo log entries 2
MySQL thread id 50, OS thread handle 140641523848960, query id 3226 172.17.0.1 mmuser update
INSERT INTO
PublicChannels(Id, DeleteAt, TeamId, DisplayName, Name, Header, Purpose)
VALUES
(?, ?, ?, ?, ?, ?, ?)
ON DUPLICATE KEY UPDATE
DeleteAt = ?,
TeamId = ?,
DisplayName = ?,
Name = ?,
Header = ?,
Purpose = ?
*** (2) HOLDS THE LOCK(S):
RECORD LOCKS space id 76 page no 4 n bits 104 index Name of table `mydb`.`PublicChannels` trx id 3140 lock_mode X locks gap before rec
*** (2) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 76 page no 4 n bits 104 index Name of table `mydb`.`PublicChannels` trx id 3140 lock_mode X locks gap before rec insert intention waiting
*** WE ROLL BACK TRANSACTION (1)
Following is my analysis:
From the deadlock output, it can be seen that it's due to a gap lock.
And that's clear because the index is Name which is a multi-column index using Name and TeamId.
But interestingly, both transactions seem to be inserting the same data, which is what is puzzling me.
The multi-column index on Name and TeamId will guarantee that they are always unique. And from looking at the code,
it does not seem possible to me that it will try to insert the same data from 2 different transactions.
But even if they do, why does tx 2 try to acquire the same lock again when it already has that ?
Here is what I think the order of events happening
Tx 2 gets a gap lock.
Tx 1 tries to get the same gap lock.
Tx 2 tries to again get the same gap lock ?
The last step is what is puzzling me. Why does an UPSERT statement acquire 2 gap locks ? From my reading of https://dev.mysql.com/doc/refman/8.0/en/innodb-locks-set.html:
> INSERT ... ON DUPLICATE KEY UPDATE differs from a simple INSERT in that an exclusive lock rather than a shared lock is placed on the row to be updated when a duplicate-key error occurs. An exclusive index-record lock is taken for a duplicate primary key value. An exclusive next-key lock is taken for a duplicate unique key value.
From what I understand, the expectation is that there will be one X lock and one gap lock is taken.
But that's not what the deadlock output seems to say.
The general advice on the internet seems to be that deadlocks will happen and not all of them can be understood.
For now, we add a generic deadlock retry function at the store package which can be reused by other queries too.
P.S.: This is a verbatim copy of my investigation posted at https://dba.stackexchange.com/questions/268652/mysql-deadlock-upsert-query-acquiring-gap-lock-twice
Testing:
This is ofcourse hard to test because it is impossible to reproduce this. I have tested this by manually returning an error
and confirming that it indeed retries.
WARN[2020-06-19T11:18:24.9585676+05:30] A deadlock happened. Retrying. caller="sqlstore/channel_store.go:568" error="Error 1213: mydeadlock"
WARN[2020-06-19T11:18:24.959158+05:30] A deadlock happened. Retrying. caller="sqlstore/channel_store.go:568" error="Error 1213: mydeadlock"
WARN[2020-06-19T11:18:24.9595072+05:30] A deadlock happened. Retrying. caller="sqlstore/channel_store.go:568" error="Error 1213: mydeadlock"
WARN[2020-06-19T11:18:24.9595451+05:30] Deadlock happened 3 times. Giving up caller="sqlstore/channel_store.go:579"
ERRO[2020-06-19T11:18:24.9596426+05:30] Unable to save channel. caller="mlog/log.go:175" err_details="Error 1213: mydeadlock" err_where=CreateChannel http_code=500 ip_addr="::1" method=POST path=/api/v4/channels request_id=745bsj13b7f6mnmsbn3t97grbw user_id=xcof1ipipbrfxpfjf6x4p6kx9e
* Fix tests
* Address review comments
* Address review comments
* Add forgotten test
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* MM-25261: Add 'include_deleted' parameter to get all channels API request.
* MM-25261: Adds test for deleted channels.
* MM-25261: Switches to single liner.
* MM-25261: Adhere to beta config setting for viewing archived channels.
* MM-25261: Test fix.
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* new job type created that checks for expired mobile sessions and pushes notifications.
* only send session expired notifications if ExtendSessionLengthWithActivity is enabled.
* includes schema change: field added to Sessions table
* add unix socket listener for mmctl local mode
* First working PoC
* Adds the channel list endpoint
* Add team list endpoint
* Add a LocalClient to the api test helper and start local mode
* Add helper to test with both SystemAdmin and Local clients
* Add some docs
* Adds TestForAllClients test helper
* Incorporating @ashishbhate's proposal for adding test names to the helpers
* [MM-24146] Add unix socket listener for mmctl local mode (#14296)
* add unix socket listener for mmctl local mode
* add a constant for local-mode socket path
* reflect review comments
* Fix init errors after merge
* Adds create channel tests
* Always init local mode to allow for enabling-disabling it via config
* Check the RemoteAddr of the request before marking session as local
* [MM-24604] Migrate get channel and get channel by name endpoints to local mode
* Remove local test conditional
Co-authored-by: Ibrahim Serdar Acikgoz <serdaracikgoz86@gmail.com>
* Removing some other fake apps
* More FakeApp removed
* Removing entirely FakeApp
* Fixing some tests
* Fixing get Cluster id from get plugin status
* Fixing failing tests
* Fixing tests
* Fixing test initialization for web
* Fixing InitServer for server tests
* Fixing InitServer for server tests
* Reverting go.sum and go.mod
* Removing unneded HTMLTemplates function in App layer
* Moving back some functions to its old place to easy the review
* Moving back some functions to its old place to easy the review
* Using the last struct2interface version
* Generating store layers
* Fixing merge problems
* Addressing PR comments
* Small fix
* Fixing app tests build
* Fixing tests
* fixing tests
* Fix tests
* Fixing tests
* Fixing tests
* Fixing tests
* Moving license to server struct
* Adding some fixes to the test compilation
* Fixing cluster and some jobs initialization
* Fixing some license tests compilation problems
* Fixing recursive cache invalidation
* Regenerating app layers
* Fix test compilation
Co-authored-by: mattermod <mattermod@users.noreply.github.com>
* Added 2 checks for app/file.go
- Check if channel id exist
- Check if user has permission to the upload file to the channel
Also added translations for 2 errors defined in app/file.go
* fixed 1 failing test that was linked with UploadFile
* Fixed small issue, according to the review.
* missed 1 review item. Just updated the code for it.
* fix 1 failing test, assuming that the file upload is required. ignoring nouser idea.
Added the translation for english for 2 newly defined errors.
* removed new line
* trying to fix the translation issue. Added the missing translations from master.
* as per discussion, we need to revert the check for user channel permission. So reverted it.
* Update app/file_test.go
Co-Authored-By: Alejandro García Montoro <alejandro.garciamontoro@gmail.com>
* Update i18n/en.json
Co-Authored-By: Alejandro García Montoro <alejandro.garciamontoro@gmail.com>
* Update file.go
Move the check to the top of the method.
* go fmt
Co-authored-by: mattermod <mattermod@users.noreply.github.com>
Co-authored-by: Alejandro García Montoro <alejandro.garciamontoro@gmail.com>
Co-authored-by: Ben Schumacher <ben.schumacher@mattermost.com>
* Migrate GetDeleted method from ChannelStore to return error interface
* Moving i18n translations to the correct place
Co-authored-by: Jesús Espino <jespinog@gmail.com>
* Migrate GetAllChannels and GetAllChannelsCount from ChannelStore to return plain errors
* Moving i18n translations to the correct place
Co-authored-by: Jesús Espino <jespinog@gmail.com>
