* MM-21211: Optimize Channelstore.UpdateLastViewedAt
For postgres, we use CTEs to do both queries at one
and return the data.
For mysql, CTEs aren't available until version 8. So we just optimize the
existing query.
- We use IN instead of multiple OR conditions for the channel membership check.
According to mysql https://dev.mysql.com/doc/refman/5.7/en/comparison-operators.html#operator_in:
If no type conversion is needed for the values in the IN() list,
they are all constants of the same type, and expr can be compared to each of them
as a value of the same type (possibly after type conversion), an optimization takes place.
The values the list are sorted and the search for expr is done using a binary search,
which makes the IN() operation very quick.
- We use the previous column LastViewedAt to replace the value of LastUpdateAt
instead of doing a switch case again.
https://dev.mysql.com/doc/refman/5.6/en/ansi-diff-update.html
If you access a column from the table to be updated in an expression, UPDATE uses the
current value of the column.
Results (ran with getchannel profile):
Postgres:
| Metric | Baseline | Actual | Delta | Delta % |
| --- | --- | --- | --- | --- |
| Hits | 1594 | 1574 | -20 | -1.25%
| Error Rate | 0.00% | 0.00% | 0% | 0% |
| Mean Response Time | 7.08ms | 6.36ms | -0.72ms | -10.20% |
| Median Response Time | 5.00ms | 4.00ms | -1.00ms | -20.00% |
| 95th Percentile | 9.00ms | 8.00ms | -1.00ms | -11.11% |
Mysql:
| Metric | Baseline | Actual | Delta | Delta % |
| --- | --- | --- | --- | --- |
| Hits | 1600 | 1566 | -34 | -2.12%
| Error Rate | 0.00% | 0.00% | 0% | 0% |
| Mean Response Time | 7.39ms | 7.02ms | -0.37ms | -4.94% |
| Median Response Time | 6.00ms | 6.00ms | 0ms | 0% |
| 95th Percentile | 10.00ms | 10.00ms | 0ms | 0% |
* Add comment on using CTEs for mysql8
* incorporate review comments
Co-authored-by: mattermod <mattermod@users.noreply.github.com>
* Refactor to use structured logging
* Properly formatted with gofmt
* created interface Cache, but construction of cache is still coupled.
* Implementing cache factories to build caches
* Simple redis implementation without error handling. Keys and values by default are string
* refactor NewLocalCacheLayer to inject cache factory
* Removed redis impl to focus on cache abstraction
* CacheFactory injected on sqlsupplier and saved in Store struct
* remove useless private method
* replace concrete declaration of lru cache to cache abstraction
* discard spaces
* Renamed factory to provider because concrete implementations of factories may hold an state (such as a redis client for example)
* refactor to include all caches in the same package cache and subpackages
* method close cache. This method will be used for concrete implementations that need to release resources (close a connection, etc)
* closing cacheprovider and releasing resources while closing sql store
* fixed merge conflict fail
* gofmt files
* remove unused property from post_store
* naming refactor to avoid stutter. Added godocs on interface
* Store doesnt know anything about the cache and provider. Cache provider will be built after loading config and injected in localCacheLayer
* fixed broken test
* cache provider initialized before RunOldAppInitialization which initializes the localcachelayer
* move statusCache to server to initialize it with the new cache provider
* update terms_service and channel_layer to have new cacheProvider
* gofmt
* Add Connect method to the cache provider
* mock cacheprovider in user_layer_test
Co-authored-by: Ben Schumacher <ben.schumacher@mattermost.com>
Co-authored-by: mattermod <mattermod@users.noreply.github.com>
Display LHS bot Icon in web app. As part of mentioned task, Added LastIconUpdate variable in model.bot to store last update time of icon. Also added code to update/delete value of the mentioned variable when setting/deleting bot icon.
* migrated termsOfServiceCache from sqlstore to localcachelayer, and mocked store tests
* revert gitignore local change
* fixed caching in termsOfService Get, and added basic tests for termsOfServiceCache
* added a test for cache save, and fixed call to Store tests
* fixed GeLatest termsOfService from cache test
* added license headers to terms_of_service cache files
* using doStandardReadFromCache and doStandardAddToCache when reading and writing to cache
* removed unused variable, termsOfServiceCacheName
* added special key for the latest termsOfService value in termsOfServiceCache
* updated license information on localcachelayer termsOfServiceCache files
* fixed not updating latest termsOfServiceCache on Get by ID, and invalidating cache cluster on termsOfServiceCache save
Co-authored-by: mattermod <mattermod@users.noreply.github.com>
The `channelByNameCache` in the `GetByNames` query checks each teamId+channelName
combination in the cache and if any of the keys is not present, it queries the DB.
In this case, this is effectively a cache miss from the point of view of user.
But since the cache is at a teamId+channelName level, there may be several hits and some misses.
This is misleading as it does not generate proper metrics and might lead to a
false assumption of how effective the cache is.
We create separate metrics which account for all or none cache hits/misses.
This removes the old metrics which are used by other queries. So they are now separated,
and it will improve both the old and new metric. Because the old metric is not contaminated,
and the new metric is also separate.
* Migrate channelCache cache from store/sqlstore/channel_store.go to the new store/localcachelayer
* fix import for new path
* remove unnecessary code, refactor cache constant
* resolve conflicts
* fix missing variable and import after merging
* refactor code, fix current issue
Co-authored-by: mattermod <mattermod@users.noreply.github.com>
* MM-14675: Upserts GroupTeam when GroupChannel is linked.
MM-14675: Delete all associated GroupChannels when deleting a GroupTeam.
MM-14675: Using replica DB where possible.
MM-14675: Updates create method used in tests.
* MM-14675: Removes unnecessary DB retrieval of GroupSyncable record.
* MM-21118: Use querybuilder instead of fmt.Sprintf in channelStore
We use the querybuilder API instead of just building the query string
manually using fmt.Sprintf
* Replaced comma with colon
* Add MLOG for sql store
* Update printf function
* Update store/sqlstore/supplier.go
Co-Authored-By: Martin Kraft <martin@upspin.org>
* Update go mod
* update go mod and tidy it up
* update vendor folder
* MM-21116: Improve channelSearch SQL query generation
The likeTerm from buildFulltextClause was being built inefficient using strings.
We change it to use squirrel to build the query.
And while at it, we also change the len check with checking with
an empty string which is more idiomatic. Both compile to the same
code, so there is no difference performance wise.
* Use a better variable name
* Consistent license message for all the go files
* Fixing the last set of unconsistencies with the license headers
* Addressing PR review comments
* Fixing busy.go and busy_test.go license header
* Migrate channelPinnedPostCountsCache cache from store/sqlstore/channel_store.go to the new store/localcachelayer
* Remove GetPinnedPostCountFromCache
* Rearrange test to avoid undeclared variable
* Migrate webhookCache from store/sqlstore/webhook_store.go to the new store/localcachelayer
* Remove duplicated statistics and move several functions to the layer
* Put cache size in format minutes * 60seconds
* Move cache functions to layer functions over the caches
* Directly remove and purge caches in handleClusterInvalidate to avoid loops
* Some advances
* Partial advances
* Cache moved
* Tests finished
* Removed all test for PostStore (store/sqlstore) related with Cache. This is tested in the cache layer now
* MM-18356: Adds ability to paginate channel search.
* MM-18356: Minor refactor.
* MM-18356: Adds doc.
* MM-18356: Fixes doc.
* MM-18356: Some commentary, adds the total count to non-paginated responses, and removes a stray fmt.
* MM-18356: Fixes shadowed variable.
* MM-18356: Removes paginate field and API parameter.
* MM-18356: Adds method to check if channel search is a paginated request.
* MM-18356: Vet fix.
* Migrate channelMemberCountsCache cache from store/sqlstore/channel_store.go to the new store/localcachelayer
* Move cache functions to layer functions over the caches
* Directly remove and purge caches in handleClusterInvalidate to avoid loops and prepare details for layer with multiple caches
