* WIP
* Adding bleve to go modules
* WIP
* Adding missing files from searchengine implementation
* WIP
* WIP
* WIP
* WIP
* WIP
* WIP
* User and channel indexing and searches implemented
* Make bleve tests run with in-memory indexes
* Implement post index and deletion tests
* Initial commits for the search layer
* Removing unnecesary indexing
* WIP
* WIP
* More fixes for tests
* Adding the search layer
* Finishing the migration of searchers to the layer
* Removing unnecesary code
* Allowing multiple engines active at the same time
* WIP
* Add simple post search
* Print information when using bleve
* Adding some debugging to understand better how the searches are working
* Making more dynamic config of search engines
* Add post search basics
* Adding the Purge API endpoint
* Fixing bleve config updates
* Adding missed file
* Regenerating search engine mocks
* Adding missed v5 to modules imports
* fixing i18n
* Fixing some test around search engine
* Removing all bleve traces
* Cleaning up the vendors directory and go.mod/go.sum files
* Regenerating timer layer
* Adding properly the license
* Fixing govet shadow error
* Fixing some tests
* Fixing TestSearchPostsFromUser
* Fixing another test
* Fixing more tests
* Fixing more tests
* Removing SearchEngine redundant text from searchengine module code
* Fixing some reindexing problems in members updates
* Fixing tests
* Addressing PR comments
* Reverting go.mod and go.sum
* Addressing PR comments
* Fixing tests compilation
* Fixing govet
* Adding search engine stop method
* Being more explicit on where we use includeDeleted
* Adding GetSqlSupplier test helper method
* Mocking elasticsearch start function
* Fixing tests
* Search tests
* Fix tests
* Fix mod
* Fixing searchEngine for test helpers with store mocks
* Remove loglines
* Fix i18n strings
* Migrate search posts tests
* Fix linter
* Do not run search tests if -short flag is enabled
* Migrate back store tests that didn't belong to the searchlayer
* Fix scopelint issues
Co-authored-by: Jesús Espino <jespinog@gmail.com>
* Remove goroutines from session_store.go
There isn't much benefit of running these 2 queries concurrently
versus just serially. These queries are not in the hot path anyways.
Maybe just simpler to keep it serial.
* Incorporate review comments
* Members Save Multiple
* Multiple Channel Members implemented
* Working version with member multi save and multi update
* Adding tests to the save multiple team members store methods
* Tests passing for SaveMember
* Adding remove members tests
* Working on channel store tests
* More work on testing properly SaveMember and SaveMultipleMembers on channels
* More work on testing properly SaveMember and SaveMultipleMembers on teams
* Adding tests for update multiple members for team
* Adding tests for update multiple members for channel
* Fixing some tets
* Fixing tests
* Fixing tests
* Fixing i18n
* Addressing govet problems
* Addressing govet problems
* Addressing govet problems
* Addressing govet problems
* Addressing PR comments (and removing TODO)
* Addressing PR comments
* Fixed broken test
* Removing unnecesary i18n translation string
* MM-23369: Allow mysql to choose a better index
When the ORDER BY clause contains a column which is in the WHERE
clause and also part of an index, mysql tries to use that specific
index to avoid sorting. This is inspite of the fact that
there may be other indices which are better for scanning the table
and then doing a sort.
Essentially, mysql becomes dumb and scans a lot of rows to avoid
sorting. Whereas, it could have scanned a lot less rows and do the
sorting in no time.
To fix this, we use the other columns in the ORDER BY clause as well
which are part of the index. This causes no change in the results
because the other columns are an EQUAL condition check, but this
lets mysql use the right index. Because now mysql sees that it has
to order by other columns too, so it better use the other index
to scan and then do the sorting.
This does not affect tables of smaller size because the LIMIT of
rows is always 1. And mysql will stop sorting the moment it gets
the first row. So sorting is not the overhead at all.
Therefore, this seems like an optimal fix.
References:
https://dev.mysql.com/doc/refman/5.7/en/table-scan-avoidance.htmlhttps://code.openark.org/blog/mysql/7-ways-to-convince-mysql-to-use-the-right-indexhttps://dev.mysql.com/doc/refman/5.7/en/limit-optimization.html
* Added a comment to clarify things in code
* Incorporating review comments
* MM-22212: Read non-moderated permissions from higher-scoped scheme.
* MM-2212: Corrects test count in comment.
* MM-22212: Adds godoc comment.
* MM-2212: Switches to the channel roles check in a few more places.
* MM-22212: Refactors and fixes.
* MM-22212: Reverts change, no longer required.
* MM-22212: Removes translation.
* MM-22212: Un-comments merged new permission.
* MM-22212: Un-comments merged new permission.
* MM-22212: Performance tweak.
* MM-22212: Fixes some fmting.
* MM-22212: Add unit test for newly-added store methods.
* MM-22212: Renames app method.
* MM-22212: Re-uses existing function to find string in slice.
* MM-22212: Keeps 'higher-scoped' terminology for consistency.
* MM-22212: Refactors based on PR feedback.
* MM-22212: Fix for some bad merging.
* MM-22212: Renamed some things.
* MM-22212: Use an 'else' instead of a 'continue' for readability.
* MM-22212: Caches (*SqlRoleStore).ChannelRolesUnderTeamRole.
* MM-22212: Adds mock to new cache store.
* MM-22212: Adds missing open tracing app layer methods.
* MM-22212: Adds migration to add moderated permissions to channel_admin if present on channel_user.
* MM-22212: Migrates team schemes. Removes unused AppError.
* MM-22212: Fix for for if.
* MM-22212: Fixes iterator.
* MM-22212: Updates open tracing generated methods.
* MM-22212: Fix mocks.
* MM-22212: Change migration key name.
* MM-22212: Switched to data structure from other branch.
* MM-22212: Fixes tests after adding 'use_channel_mentions' to the channel_admin role.
* MM-22212: Adds tracking of channel moderation.
* Revert "MM-22212: Adds tracking of channel moderation."
This reverts commit 23689efa22c112e4ba37f6a212535dd7ebfb63db.
* MM-22212: Switch some functions to methods and vice versa.
* MM-22212: Fix for refactor bug not notifiying websocket about changed role.
* MM-22212: Adds test for public/private 'manage_members' handling.
* MM-22122 Fix manage channel members edge case for public and private channels (#14049)
* MM-22212: Adds moderated permission to team_admin.
* MM-22212: Updates migration.
* MM-22212: Revert unnecessary update to default roles.
* Add channel scheme updated event when channel scheme is deleted or created (#14057)
* MM-22212: Adds newline.
* MM-22212: Migration fix.
* MM-22212: Fix for migration.
* MM-22212: Test fix.
Co-authored-by: Farhan Munshi <3207297+fm2munsh@users.noreply.github.com>
* Optimize reactions table
Change reactions primary key to (PostId, UserId, EmojiName) so fetching
reactions for post will use primary key lookup instead of table scan.
* fix db version
* review fixes
* update database schema in scripts/
* WIP
* Adding bleve to go modules
* WIP
* Adding missing files from searchengine implementation
* WIP
* WIP
* WIP
* WIP
* WIP
* WIP
* User and channel indexing and searches implemented
* Make bleve tests run with in-memory indexes
* Implement post index and deletion tests
* Initial commits for the search layer
* Removing unnecesary indexing
* WIP
* WIP
* More fixes for tests
* Adding the search layer
* Finishing the migration of searchers to the layer
* Removing unnecesary code
* Allowing multiple engines active at the same time
* WIP
* Add simple post search
* Print information when using bleve
* Adding some debugging to understand better how the searches are working
* Making more dynamic config of search engines
* Add post search basics
* Adding the Purge API endpoint
* Fixing bleve config updates
* Adding missed file
* Regenerating search engine mocks
* Adding missed v5 to modules imports
* fixing i18n
* Fixing some test around search engine
* Removing all bleve traces
* Cleaning up the vendors directory and go.mod/go.sum files
* Regenerating timer layer
* Adding properly the license
* Fixing govet shadow error
* Fixing some tests
* Fixing TestSearchPostsFromUser
* Fixing another test
* Fixing more tests
* Fixing more tests
* Removing SearchEngine redundant text from searchengine module code
* Fixing some reindexing problems in members updates
* Fixing tests
* Addressing PR comments
* Reverting go.mod and go.sum
* Addressing PR comments
* Fixing tests compilation
* Fixing govet
* Adding search engine stop method
* Being more explicit on where we use includeDeleted
* Adding GetSqlSupplier test helper method
* Mocking elasticsearch start function
* Fixing tests
Co-authored-by: Miguel de la Cruz <miguel@mcrx.me>
Co-authored-by: mattermod <mattermod@users.noreply.github.com>
* Adding SaveMultiple to posts
* Improving tests
* fixing i18n
* Fixing tests
* Improving testing on top of Save and SaveMultiple
* Fixing shadow variables
* Addressing some PR comments
* More clear update post test
* Addressing some PR comments
* Addressing some PR comments and simplifying the code
* Improting replies in bulk too
* Fixing reply count and processing last imported replies
* Adding OverwriteMultiple to posts aggregating everything in the same transaction
* Adding 2 pending tests to implement
* Adding tests for overwrite multiple posts
* Adding tests for TeamStore.GetByNames method
* Fixing shadow variables
* Addressing PR comments
* Extracting i18n strings
* Fixing tests
* Fixing tests
* Adding more test cases
* Using a variable instead of a fake timestamp
* Introducing unit (not integration) tests for the app layer
* Initial support for unit tests at the API
* Adding unit tests support to the store layer
* Add unit tests support in commands
* Adding last tests needed for run unit tests properly
* Fixing govet
* Removing some duplication
* Fixing tests
* Fixing tests
* Not compiling test helpers with the main module for api
* Revert "Not compiling test helpers with the main module for api"
This reverts commit 36a199bbe0f7503f665f5d6c7b41c53aabc2e54f.
* Fixing tests
* Fixing unit tests
* More consistency between api4/apiteslib.go and app/helper_test.go
* Renaming things to make more obvious the new Setup functions purpose
* Reverting change in go.sum
* Start with empty mock for app layer
* Start with empty mock for api layer
* Start with empty mock for web layer
* Renaming SetupWithStoreMockConfig to SetupConfigWithStoreMock
* Fixing tests on web package
* Removing unnecesary function
* allow ExpireInSeconds < 0
Allow `ExpireInSeconds < 0` for use with `KVSetWithOptions`. While this has no practical use in reality, it's much easier to thoroughly unit tests the underlying functionality if we can match the semantics of CompareAndSet. Strictly speaking, this is a breaking change, but not relative to the advertised semantics. Anyway, it's also not entirely unreasonable to treat a negative `ExpireInSeconds` as having already expired vs. marking it as never expired.
* updated tests, to break apart
* honour expiry in CompareAndSet
* honour expiry in CompareAndDelete
* honour expiry in List
* fail unique constraint exception for SaveOrUpdate
A unique constraint error on a `SaveOrUpdate` should not be ignored: we did not save or update the requested value, as someone else managed to write the record first.
Note this is handled differently in `CompareAndSet`, where we correctly swallow the error and return `false` to indicate we did not successfully save the value.
* unexport DEFAULT_PLUGIN_KEY_FETCH_LIMIT
* s/InternalServerError/BadRequest/ for failed SaveOrUpdate
Co-authored-by: mattermod <mattermod@users.noreply.github.com>
* constants and options for getting files
* Method to get files with options
* Add i18n strings for en
* Add API methods for getting files with options
* gofmt -s file
* explicitly set create at in tests
* use greater than nanosecond time difference for tests
* use gte instead of gt for getting files by created time
* use created at time as default sort order for getting file infos
* use explicit inline strings instead of format strings
* join tables only when required
* use if as secondary sort, and update tests
* update field docs to reflect previous changes
* make page and perPage get options as required
* add json struct tags to GetFileOptions
* bump minimum server versioni
* remove sorting by username and channelname
* use bool for sort order type
* use FileInfo prefix instead of just File
* clearer comments
* use zero-based page numbering
* test filtering by user and channel
* remove unnecessary whitespace
* use int instead of uint for page and perPage arguments
Co-authored-by: mattermod <mattermod@users.noreply.github.com>
* Run all tests fasts in postgres
* Moving postgres config to a file
* Addressing PR review comments
* Testing against mysql in the CI to keep things less changed
* Revert "Testing against mysql in the CI to keep things less changed"
This reverts commit fc940c111db269a821c54e005d76df87cb70e031.
* Fixing a test broken in postgres but working in mysql
* Fixing some tests
* Fixing some config tests with postgres
* Fixing the rest of config tests
* Fixing govet error
* Fixing search tests for postgres
Setting a `nil` value should actually delete the row instead of preserve it with a `nil` value.
The existing tests passed because they only checked the return value of `nil`. When running the tests on Postgres, this value is returned as an empty `[]byte{}` instead of `nil`, so update the test to reflect this semantically equivalent behaviour.
* MM-21946 Ensure deleted groups are not returned from the groups API
* MM-21946 Only append to query at the end
* MM-21946 Add DeleteAt check to the top of the GetGroups function and clean up the formatting a bit
* MM-21946 Move the From statement into the next block
* MM-20644: Add users to teams as a SchemeAdmin based on a new configuration field on GroupTeams and GroupChannels records.
* MM-20644: Adds SchemeAdmin to mapping of the GroupSyncable struct fields.
* MM-2064: Adds test to ensure SchemeAdmin field value is mapped.
* MM-20644: Adds missing index creation for fresh DBs.
* MM-20644: Duplicates UpdateMembersRole across Team and Channel stores. Adds tests.
* MM-20644: Fixes some old method name references.
* MM-20644: Moves variable declaration; removes Println statement.
* MM-21157: Triggers role sync when a syncable is linked, unlinked, or updated.
* MM-20644: Use a SQL query instead of two to update Team and Channel members.
* MM-20644: Fixes tests; updates query.
* MM-21157: Removes second invocation of function because of refactor that performs it all in a single query.
* MM-21157: Switches a few queries to Squirrel.
* MM-21157: SQL-formats some strings.
* MM-21157: Select with list.
* MM-21157: Converts some more sql to squirrel.
* MM-21157: Fix incorrect conflict resolutions.
* MM-21157: Fix incorrect conflict resolutions.
* MM-21157: Adds missing mocks.
* MM-21157: Clears cache upon syncing roles.
Co-authored-by: mattermod <mattermod@users.noreply.github.com>
`KVCompareAndSet(key, sameValue, sameValue)` can fail spuriously on MySQL if the underlying `UPDATE` requires no actual changes. As per the [MySQL documentation](https://dev.mysql.com/doc/refman/8.0/en/mysql-affected-rows.html), we can't rely on rows affected in this case:
> For UPDATE statements, the affected-rows value by default is the number of rows actually changed. If you specify the CLIENT_FOUND_ROWS flag to mysql_real_connect() when connecting to mysqld, the affected-rows value is the number of rows “found”; that is, matched by the WHERE clause.
It's not tenable to change `CLIENT_FOUND_ROWS` for the all connection, so handle this case in the code instead by running a `SELECT` after the fact. Note that `KVCompareAndSet` has no guarantee of atomicity in this case, but neither would `CompareAndSwap` on which this is method was inspired.
Finally, note that no changes are required for Postgres, which has sane semantics as the default.
Fixes: https://mattermost.atlassian.net/browse/MM-21328
Co-authored-by: mattermod <mattermod@users.noreply.github.com>
* MM-20644: Add users to teams as a SchemeAdmin based on a new configuration field on GroupTeams and GroupChannels records.
* MM-20644: Adds SchemeAdmin to mapping of the GroupSyncable struct fields.
* MM-2064: Adds test to ensure SchemeAdmin field value is mapped.
* MM-20644: Adds missing index creation for fresh DBs.
* MM-20644: Duplicates UpdateMembersRole across Team and Channel stores. Adds tests.
* MM-20644: Fixes some old method name references.
* MM-20644: Moves variable declaration; removes Println statement.
* MM-20644: Use a SQL query instead of two to update Team and Channel members.
* MM-20644: Fixes tests; updates query.
* MM-20644: Fix permission check for patching a group syncable.
* MM-20644: Fixes test for change of permissions verification in group patch API request.
* MM-20644: Fix for ORM select vs insert.
* MM-20644: Linting fixes.
* MM-20644: Fixes some tests.
* MM-20644: Skips changing the role of guests.
* MM-20644: Added syncableID filtering
* MM-21211: Optimize Channelstore.UpdateLastViewedAt
For postgres, we use CTEs to do both queries at one
and return the data.
For mysql, CTEs aren't available until version 8. So we just optimize the
existing query.
- We use IN instead of multiple OR conditions for the channel membership check.
According to mysql https://dev.mysql.com/doc/refman/5.7/en/comparison-operators.html#operator_in:
If no type conversion is needed for the values in the IN() list,
they are all constants of the same type, and expr can be compared to each of them
as a value of the same type (possibly after type conversion), an optimization takes place.
The values the list are sorted and the search for expr is done using a binary search,
which makes the IN() operation very quick.
- We use the previous column LastViewedAt to replace the value of LastUpdateAt
instead of doing a switch case again.
https://dev.mysql.com/doc/refman/5.6/en/ansi-diff-update.html
If you access a column from the table to be updated in an expression, UPDATE uses the
current value of the column.
Results (ran with getchannel profile):
Postgres:
| Metric | Baseline | Actual | Delta | Delta % |
| --- | --- | --- | --- | --- |
| Hits | 1594 | 1574 | -20 | -1.25%
| Error Rate | 0.00% | 0.00% | 0% | 0% |
| Mean Response Time | 7.08ms | 6.36ms | -0.72ms | -10.20% |
| Median Response Time | 5.00ms | 4.00ms | -1.00ms | -20.00% |
| 95th Percentile | 9.00ms | 8.00ms | -1.00ms | -11.11% |
Mysql:
| Metric | Baseline | Actual | Delta | Delta % |
| --- | --- | --- | --- | --- |
| Hits | 1600 | 1566 | -34 | -2.12%
| Error Rate | 0.00% | 0.00% | 0% | 0% |
| Mean Response Time | 7.39ms | 7.02ms | -0.37ms | -4.94% |
| Median Response Time | 6.00ms | 6.00ms | 0ms | 0% |
| 95th Percentile | 10.00ms | 10.00ms | 0ms | 0% |
* Add comment on using CTEs for mysql8
* incorporate review comments
Co-authored-by: mattermod <mattermod@users.noreply.github.com>
* Refactor to use structured logging
* Properly formatted with gofmt
* created interface Cache, but construction of cache is still coupled.
* Implementing cache factories to build caches
* Simple redis implementation without error handling. Keys and values by default are string
* refactor NewLocalCacheLayer to inject cache factory
* Removed redis impl to focus on cache abstraction
* CacheFactory injected on sqlsupplier and saved in Store struct
* remove useless private method
* replace concrete declaration of lru cache to cache abstraction
* discard spaces
* Renamed factory to provider because concrete implementations of factories may hold an state (such as a redis client for example)
* refactor to include all caches in the same package cache and subpackages
* method close cache. This method will be used for concrete implementations that need to release resources (close a connection, etc)
* closing cacheprovider and releasing resources while closing sql store
* fixed merge conflict fail
* gofmt files
* remove unused property from post_store
* naming refactor to avoid stutter. Added godocs on interface
* Store doesnt know anything about the cache and provider. Cache provider will be built after loading config and injected in localCacheLayer
* fixed broken test
* cache provider initialized before RunOldAppInitialization which initializes the localcachelayer
* move statusCache to server to initialize it with the new cache provider
* update terms_service and channel_layer to have new cacheProvider
* gofmt
* Add Connect method to the cache provider
* mock cacheprovider in user_layer_test
Co-authored-by: Ben Schumacher <ben.schumacher@mattermost.com>
Co-authored-by: mattermod <mattermod@users.noreply.github.com>
Display LHS bot Icon in web app. As part of mentioned task, Added LastIconUpdate variable in model.bot to store last update time of icon. Also added code to update/delete value of the mentioned variable when setting/deleting bot icon.
