* MM-23369: Allow mysql to choose a better index
When the ORDER BY clause contains a column which is in the WHERE
clause and also part of an index, mysql tries to use that specific
index to avoid sorting. This is inspite of the fact that
there may be other indices which are better for scanning the table
and then doing a sort.
Essentially, mysql becomes dumb and scans a lot of rows to avoid
sorting. Whereas, it could have scanned a lot less rows and do the
sorting in no time.
To fix this, we use the other columns in the ORDER BY clause as well
which are part of the index. This causes no change in the results
because the other columns are an EQUAL condition check, but this
lets mysql use the right index. Because now mysql sees that it has
to order by other columns too, so it better use the other index
to scan and then do the sorting.
This does not affect tables of smaller size because the LIMIT of
rows is always 1. And mysql will stop sorting the moment it gets
the first row. So sorting is not the overhead at all.
Therefore, this seems like an optimal fix.
References:
https://dev.mysql.com/doc/refman/5.7/en/table-scan-avoidance.htmlhttps://code.openark.org/blog/mysql/7-ways-to-convince-mysql-to-use-the-right-indexhttps://dev.mysql.com/doc/refman/5.7/en/limit-optimization.html
* Added a comment to clarify things in code
* Incorporating review comments
* MM-22212: Read non-moderated permissions from higher-scoped scheme.
* MM-2212: Corrects test count in comment.
* MM-22212: Adds godoc comment.
* MM-2212: Switches to the channel roles check in a few more places.
* MM-22212: Refactors and fixes.
* MM-22212: Reverts change, no longer required.
* MM-22212: Removes translation.
* MM-22212: Un-comments merged new permission.
* MM-22212: Un-comments merged new permission.
* MM-22212: Performance tweak.
* MM-22212: Fixes some fmting.
* MM-22212: Add unit test for newly-added store methods.
* MM-22212: Renames app method.
* MM-22212: Re-uses existing function to find string in slice.
* MM-22212: Keeps 'higher-scoped' terminology for consistency.
* MM-22212: Refactors based on PR feedback.
* MM-22212: Fix for some bad merging.
* MM-22212: Renamed some things.
* MM-22212: Use an 'else' instead of a 'continue' for readability.
* MM-22212: Caches (*SqlRoleStore).ChannelRolesUnderTeamRole.
* MM-22212: Adds mock to new cache store.
* MM-22212: Adds missing open tracing app layer methods.
* MM-22212: Adds migration to add moderated permissions to channel_admin if present on channel_user.
* MM-22212: Migrates team schemes. Removes unused AppError.
* MM-22212: Fix for for if.
* MM-22212: Fixes iterator.
* MM-22212: Updates open tracing generated methods.
* MM-22212: Fix mocks.
* MM-22212: Change migration key name.
* MM-22212: Switched to data structure from other branch.
* MM-22212: Fixes tests after adding 'use_channel_mentions' to the channel_admin role.
* MM-22212: Adds tracking of channel moderation.
* Revert "MM-22212: Adds tracking of channel moderation."
This reverts commit 23689efa22c112e4ba37f6a212535dd7ebfb63db.
* MM-22212: Switch some functions to methods and vice versa.
* MM-22212: Fix for refactor bug not notifiying websocket about changed role.
* MM-22212: Adds test for public/private 'manage_members' handling.
* MM-22122 Fix manage channel members edge case for public and private channels (#14049)
* MM-22212: Adds moderated permission to team_admin.
* MM-22212: Updates migration.
* MM-22212: Revert unnecessary update to default roles.
* Add channel scheme updated event when channel scheme is deleted or created (#14057)
* MM-22212: Adds newline.
* MM-22212: Migration fix.
* MM-22212: Fix for migration.
* MM-22212: Test fix.
Co-authored-by: Farhan Munshi <3207297+fm2munsh@users.noreply.github.com>
* change how we are parsing the args.command
* change comment
* atttempt to write unit test
* remove quotes
* use unicode.IsSpace
* MM-21343 fix panic
* use table
* test cases
* bad request, add test for missing '/'
* mispelled err
* Update app/command.go
Co-Authored-By: Jesse Hallam <jesse.hallam@gmail.com>
* Update app/command.go
Co-Authored-By: Jesse Hallam <jesse.hallam@gmail.com>
* use a map
* use subtest and add test case of empty command
* go fmt
* do not use t.fatal
* Update app/command.go
Co-Authored-By: Bot from GolangCI <42910462+golangcibot@users.noreply.github.com>
* fix missing unicode import
Co-authored-by: wiggin77 <wiggin77@warpmail.net>
Co-authored-by: Jesse Hallam <jesse.hallam@gmail.com>
Co-authored-by: mattermod <mattermod@users.noreply.github.com>
Co-authored-by: Bot from GolangCI <42910462+golangcibot@users.noreply.github.com>
* add support for exporting and importing props
* merge master
* add export test for post props
* add check if postData exists
* add import tests for post props
* fix ci bot issues
Co-Authored-By: Bot from GolangCI <42910462+golangcibot@users.noreply.github.com>
* fix gofmt errors
* remove test comment and add name instead
* remove uneccessary changes
Co-authored-by: Bot from GolangCI <42910462+golangcibot@users.noreply.github.com>
* MM-23222 add file target (with rotation) to audit
* MM-23222 mirror syslog audits to local filesystem
* provides config options for file name, max size, max age
* rotates files based on max size and max age; delete as needed based on max backups
* include cluster id in log records
* sort meta data fields
There was a race in (*App).SyncPlugins where if the same plugin
existed in availablePlugins and pluginSignaturePathMap, then
we would try to add/remove at the same time.
This would lead to a possible removal / addition of a plugin directory
or even unable to remove a directory because it was already in use.
We fix this by first finishing the removal of availablePlugins
before syncing it with the file store.
* Remove unncessary recover() calls
A recover which is not called inside a defer will always return nil.
So there's no use of calling it.
And even if we did call it inside a defer, recovering and immediately
panicking does not make sense either. So we just remove it.
* Fixed another instance
* Optimize reactions table
Change reactions primary key to (PostId, UserId, EmojiName) so fetching
reactions for post will use primary key lookup instead of table scan.
* fix db version
* review fixes
* update database schema in scripts/
* MM-22560: Fix crash during slack import
We were not checking for existence of the users map value
before accessing the Id field. We fix it by checking for
nil pointer first.
Also, we check the length of the Members slice to prevent range
panics.
And finally, while we are here, we move to a switch-case
to make the code a little more idiomatic.
* Adding warn logs when user not found
Co-authored-by: mattermod <mattermod@users.noreply.github.com>
* Improving mfa tests coverage and format
* Adding assertions for the specific error types
Co-authored-by: mattermod <mattermod@users.noreply.github.com>
* WIP
* Adding bleve to go modules
* WIP
* Adding missing files from searchengine implementation
* WIP
* WIP
* WIP
* WIP
* WIP
* WIP
* User and channel indexing and searches implemented
* Make bleve tests run with in-memory indexes
* Implement post index and deletion tests
* Initial commits for the search layer
* Removing unnecesary indexing
* WIP
* WIP
* More fixes for tests
* Adding the search layer
* Finishing the migration of searchers to the layer
* Removing unnecesary code
* Allowing multiple engines active at the same time
* WIP
* Add simple post search
* Print information when using bleve
* Adding some debugging to understand better how the searches are working
* Making more dynamic config of search engines
* Add post search basics
* Adding the Purge API endpoint
* Fixing bleve config updates
* Adding missed file
* Regenerating search engine mocks
* Adding missed v5 to modules imports
* fixing i18n
* Fixing some test around search engine
* Removing all bleve traces
* Cleaning up the vendors directory and go.mod/go.sum files
* Regenerating timer layer
* Adding properly the license
* Fixing govet shadow error
* Fixing some tests
* Fixing TestSearchPostsFromUser
* Fixing another test
* Fixing more tests
* Fixing more tests
* Removing SearchEngine redundant text from searchengine module code
* Fixing some reindexing problems in members updates
* Fixing tests
* Addressing PR comments
* Reverting go.mod and go.sum
* Addressing PR comments
* Fixing tests compilation
* Fixing govet
* Adding search engine stop method
* Being more explicit on where we use includeDeleted
* Adding GetSqlSupplier test helper method
* Mocking elasticsearch start function
* Fixing tests
Co-authored-by: Miguel de la Cruz <miguel@mcrx.me>
Co-authored-by: mattermod <mattermod@users.noreply.github.com>
* New auditing API outputting to syslog via TLS
* New config section for specifying remote syslog server IP, port, and cert.
* Legacy audit API retained for access history feature
* Adding SaveMultiple to posts
* Improving tests
* fixing i18n
* Fixing tests
* Improving testing on top of Save and SaveMultiple
* Fixing shadow variables
* Addressing some PR comments
* More clear update post test
* Addressing some PR comments
* Addressing some PR comments and simplifying the code
* Improting replies in bulk too
* Fixing reply count and processing last imported replies
* Adding OverwriteMultiple to posts aggregating everything in the same transaction
* Adding 2 pending tests to implement
* Adding tests for overwrite multiple posts
* Adding tests for TeamStore.GetByNames method
* Fixing shadow variables
* Addressing PR comments
* Extracting i18n strings
* Fixing tests
* Fixing tests
* Adding more test cases
* Using a variable instead of a fake timestamp
* Create infrastructure to manage mentions
Two new files have been added (along with their tests); namely:
- model/at_mentions.go: utilities to parse and manage mentions; for the moment,
it just contains a regex and a couple of functions to parse possible mentions
and to post-process them, but it can be extended in the future.
- model/mention_map.go: it contains two new types (UserMentionMap and
ChannelMentionMap) that both have FromURLValues and ToURLValues. These types
can be used when adding the mentions to the payload of the plugin slash
commands.
* Extend custom commands payload with mentions
Two couples of new fields are added to the payload; namely:
- user_mentions and user_mentions_ids: two aligned arrays of the same length
containing all the different @-mentions found in the command: the i-th element
of user_mentions_ids is the user identifier of the i-th element of
user_mentions.
- channel_mentions and channel_mentions_ids: two aligned arrays of the same
length containing all the different ~-mentions found in the command: the i-th
element of channel_mentions_ids is the channel identifier of the i-th element
of channel_mentions.
* Fix shadowing of variables and redundant return
* Fix shadowing of variable
* Address review comments (HT @lieut-data)
- Improvements in mentionsToTeamMembers and mentionsToPublicChannels:
- Scope implementation details inside the functions.
- Improve goroutines synchronization by using a sync.WaitGroup.
- Retry lookup of username only if the returned error is http.StatusCode,
so we can return early if the error is more severe.
- Invert check in PossibleAtMentions to improve readability.
- Make user and channel mention keys private to the module.
- Allow the specification of an empty map of mentions in
(Channel|User)MentionsFromURLValues when both mentions keys are absent.
- Replace custom functions in tests with require.Equal on maps.
* Test functions to parse mentions from messages
* Extend plugin commands payload with mentions
* Add functions to CommandArgs to add mentions
The functions make sure that the maps are initialized before adding any value.
* Address review comments (HT @lieut-data)
- Adds a mlog.Warn to avoid burying the error when the user is not found.
- Improve readability in loop populating the mention map by moving the
initialization of the map closer to the loop and by iterating over the channel
itself, not over its length.
* File was not gofmt-ed with -s
* Close channel when all goroutines are finished
* Again, all code should be checked with gofmt -s
* Refactor code out of a goroutine
This change helps improve the readability of the code and does not affect its
overall performance. Less complexity is always better.
* Close channel and iterate over its range
Adapt mentionsToPublicChannels to have the same structure in the management
of the mentions channel as in mentionsToTeamMembers.
* Adapt mentionsToTeamMembers to new App
Commit 17523fa changed the App structure, making the *Server field
private, which is now accessed through the Srv() function.
Co-authored-by: mattermod <mattermod@users.noreply.github.com>
* Transit panic from debug to error
* Parse plugin's StdErr and output panic to the mlog.Error
* Add unit tests
* Change log test
* Remove buffer from logger
* Remove 'panic' string filter
* Change *Buffer to io.Writer
Co-authored-by: mattermod <mattermod@users.noreply.github.com>
