We replace expected/received with the source of the collation.
This makes it more clear as to from where the values
were retrieved.
```release-note
NONE
```
We use the default_text_search_config param from the DB
and pass that as the search config instead of hardcoding
it to english.
The docs do say that if no param is passed, by default
it will automatically fall back to default_text_search_config.
But I have seen different query plans being created
when the parameter is not passed versus when it is passed.
I am not sure why this is happening, but to be safe,
I have taken the search config during startup and stored
it in the SqlStore struct.
For reference, this is what happens without a search config
passed:
```
[bigdb] # explain analyze SELECT *, (SELECT COUNT(*) FROM Posts WHERE Posts.RootId = (CASE WHEN q2.RootId = '' THEN q2.Id ELSE q2.RootId END) AND Posts.DeleteAt = 0) as ReplyCount FROM Posts q2 WHERE q2.DeleteAt = 0 AND q2.Type NOT LIKE 'system_%' AND to_tsvector(Message) @@ to_tsquery('sapiente') AND ChannelId IN (SELECT Id FROM Channels, ChannelMembers WHERE Id = ChannelId AND Channels.DeleteAt = 0 AND ChannelMembe
rs.UserId = 'tc3p1yqw67d8idcp3g98awexqe' AND (TeamId = '8ywxyw9ocp8smxrmjzrkqhrdwe' OR TeamId = '') AND Id IN ('h1x9asxr7idjpqfmg8q67us49h')) ORDER BY q2.CreateAt DESC LIMIT 100;
QUERY PLAN
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Limit (cost=8200.93..8720.23 rows=9 width=382) (actual time=177.277..177.948 rows=100 loops=1)
-> Result (cost=8200.93..8720.23 rows=9 width=382) (actual time=177.274..177.939 rows=100 loops=1)
-> Sort (cost=8200.93..8200.95 rows=9 width=374) (actual time=177.245..177.271 rows=100 loops=1)
Sort Key: q2.createat DESC
Sort Method: top-N heapsort Memory: 161kB
-> Nested Loop Semi Join (cost=71.53..8200.79 rows=9 width=374) (actual time=5.910..176.815 rows=532 loops=1)
-> Bitmap Heap Scan on posts q2 (cost=70.56..8183.64 rows=9 width=374) (actual time=5.815..176.457 rows=532 loops=1)
Recheck Cond: ((channelid)::text = 'h1x9asxr7idjpqfmg8q67us49h'::text)
Filter: (((type)::text !~~ 'system_%'::text) AND (deleteat = 0) AND (to_tsvector((message)::text) @@ to_tsquery('sapiente'::text)))
Rows Removed by Filter: 5244
Heap Blocks: exact=944
-> Bitmap Index Scan on idx_posts_channel_id_update_at (cost=0.00..70.56 rows=1866 width=0) (actual time=1.883..1.883 rows=5776 loops=1)
Index Cond: ((channelid)::text = 'h1x9asxr7idjpqfmg8q67us49h'::text)
-> Materialize (cost=0.98..17.04 rows=1 width=27) (actual time=0.000..0.000 rows=1 loops=532)
-> Nested Loop (cost=0.98..17.03 rows=1 width=27) (actual time=0.080..0.080 rows=1 loops=1)
-> Index Scan using channels_pkey on channels (cost=0.42..8.45 rows=1 width=27) (actual time=0.041..0.042 rows=1 loops=1)
Index Cond: ((id)::text = 'h1x9asxr7idjpqfmg8q67us49h'::text)
Filter: ((deleteat = 0) AND (((teamid)::text = '8ywxyw9ocp8smxrmjzrkqhrdwe'::text) OR ((teamid)::text = ''::text)))
-> Index Only Scan using idx_channelmembers_user_id_channel_id_last_viewed_at on channelmembers (cost=0.56..8.58 rows=1 width=27) (actual time=0.034..0.034 rows=1 loops=1)
Index Cond: ((userid = 'tc3p1yqw67d8idcp3g98awexqe'::text) AND (channelid = 'h1x9asxr7idjpqfmg8q67us49h'::text))
Heap Fetches: 1
SubPlan 1
-> Aggregate (cost=57.68..57.69 rows=1 width=8) (actual time=0.006..0.006 rows=1 loops=100)
-> Index Only Scan using idx_posts_root_id_delete_at on posts (cost=0.56..54.44 rows=1294 width=0) (actual time=0.005..0.005 rows=3 loops=100)
Index Cond: ((rootid = (CASE WHEN ((q2.rootid)::text = ''::text) THEN q2.id ELSE q2.rootid END)::text) AND (deleteat = 0))
Heap Fetches: 0
Planning Time: 2.178 ms
Execution Time: 178.155 ms
```
We can see it using a top-N heapsort.
And this is what happens when an explicit search config
is passed:
```
[bigdb] # explain analyze SELECT *, (SELECT COUNT(*) FROM Posts WHERE Posts.RootId = (CASE WHEN q2.RootId = '' THEN q2.Id ELSE q2.RootId END) AND Posts.DeleteAt = 0) as ReplyCount FROM Posts q2 WHERE q2.DeleteAt = 0 AND q2.Type NOT LIKE 'system_%' AND to_tsvector('pg_catalog.english', Message) @@ to_tsquery('pg_catalog.english', 'sapiente') AND ChannelId IN (SELECT Id FROM Channels, ChannelMembers WHERE Id = ChannelId AND Channels.DeleteAt = 0 AND ChannelMembers.UserId = 'tc3p1yqw67d8idcp3g98awexqe' AND (TeamId = '8ywxyw9ocp8smxrmjzrkqhrdwe' OR TeamId = '') AND Id IN ('h1x9asxr7idjpqfmg8q67us49h')) ORDER BY q2.CreateAt DESC LIMIT 100;
QUERY PLAN
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Limit (cost=1.54..10679.04 rows=100 width=382) (actual time=0.237..31.405 rows=100 loops=1)
-> Nested Loop Semi Join (cost=1.54..17512.64 rows=164 width=382) (actual time=0.236..31.387 rows=100 loops=1)
-> Index Scan Backward using idx_posts_channel_id_delete_at_create_at on posts q2 (cost=0.56..8032.81 rows=164 width=374) (actual time=0.121..30.124 rows=100 loops=1)
Index Cond: (((channelid)::text = 'h1x9asxr7idjpqfmg8q67us49h'::text) AND (deleteat = 0))
Filter: (((type)::text !~~ 'system_%'::text) AND (to_tsvector('english'::regconfig, (message)::text) @@ '''sapient'''::tsquery))
Rows Removed by Filter: 1088
-> Materialize (cost=0.98..17.04 rows=1 width=27) (actual time=0.001..0.001 rows=1 loops=100)
-> Nested Loop (cost=0.98..17.03 rows=1 width=27) (actual time=0.074..0.075 rows=1 loops=1)
-> Index Scan using channels_pkey on channels (cost=0.42..8.45 rows=1 width=27) (actual time=0.032..0.032 rows=1 loops=1)
Index Cond: ((id)::text = 'h1x9asxr7idjpqfmg8q67us49h'::text)
Filter: ((deleteat = 0) AND (((teamid)::text = '8ywxyw9ocp8smxrmjzrkqhrdwe'::text) OR ((teamid)::text = ''::text)))
-> Index Only Scan using idx_channelmembers_user_id_channel_id_last_viewed_at on channelmembers (cost=0.56..8.58 rows=1 width=27) (actual time=0.041..0.041 rows=1 loops=1)
Index Cond: ((userid = 'tc3p1yqw67d8idcp3g98awexqe'::text) AND (channelid = 'h1x9asxr7idjpqfmg8q67us49h'::text))
Heap Fetches: 1
SubPlan 1
-> Aggregate (cost=57.68..57.69 rows=1 width=8) (actual time=0.010..0.010 rows=1 loops=100)
-> Index Only Scan using idx_posts_root_id_delete_at on posts (cost=0.56..54.44 rows=1294 width=0) (actual time=0.009..0.009 rows=3 loops=100)
Index Cond: ((rootid = (CASE WHEN ((q2.rootid)::text = ''::text) THEN q2.id ELSE q2.rootid END)::text) AND (deleteat = 0))
Heap Fetches: 0
Planning Time: 0.401 ms
Execution Time: 31.466 ms
```
https://mattermost.atlassian.net/browse/MM-25554
```release-note
NONE
```
* [MM-44329] Added methods to sqlstore.Store
* [MM-44329] Added methods to the store.Store interface
* [MM-44329] Regenerated mocks
* [MM-44329] Added comments for the store.Store interface
Co-authored-by: Tim Scheuermann <tim.scheuermann@mattermost.com>
We check for the presence of binary_parameters
in the DSN and add the 0x01 byte accordingly.
This helps us avoid casting to string
and efficiently use the database.
```release-note
NONE
```
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* revamp db version and add applied migrations endpoint
* replace old schema version with new
* add db version subcommand
* add to local api
* reflect review comments
* log errors
* remove setting the version from model.CurrentVersion
* fix a test
* use different field for schema version
* add build hash and current version to the support packet
* add tests
* update test to use new assets
* Ported GetMembersByIds and GetMembersByChannelIds to Squirrel
* Added checks for empty ID lists
* Updated GetAllChannelMembersById to use Squirrel
* Added a method to get the query placeholder directly
* Migrated UpdateMultipleMembers to use a Squirrel query
* Migrated UpdateMultipleMembers to Squirrel
* Initialize a prepared query builder for channelMembersForTeamWithSchemeSelectQuery
* Migrated GetMembersForUser to Squirrel
* Slight improvement for constructMySQLJSONArgs
* Migrated UpdateMemberNotifyProps to Squirrel
* Migrated GetMembers to Squirrel
* Migrated GetMember to Squirrel
* Avoid shadowing err
* Don't set query builder on copy of SqlChannelStore
* Fixed typo in error message
* Fixed missing elipsis
* Shorter SQL generation for GetAllChannelMembersById
* Don't unnecessarily copy the reference
* Use a function to generate the ChannelMember map
* Avoid shadowing err
* Don't use IN for matching multiple possible values
* Initialize the members lists
* Remove check for empty channelIDs list
* Fixed test and removed check for empty list
* Use err2 rather than eerr
Co-authored-by: Tim Scheuermann <tim@plusmid.dev>
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* Revert "[MM-41576] Revamp database schema version (#19586)"
This reverts commit 645fee3fe3.
* Revert "MM-42049 - license endpoint not working (#19686)"
This reverts commit 4fe89e5847.
* revamp db version and add applied migrations endpoint
* replace old schema version with new
* add db version subcommand
* add to local api
* reflect review comments
* log errors
* remove setting the version from model.CurrentVersion
* fix a test
We create a custom io.Writer to pass to the logger
instance. For now, we keep everything to debug
as all errors are surfaced back to the server via
the API.
```release-note
NONE
```
* Update morph dependency to use a newer version
* remove timeout check for migrations statements
* store/sqlstore: reset timeout for mysql while creating db for migrations
* update morph to v0.2.1
Our store upgrade methods used mlog.Critical and then slept
for a second and triggered os.Exit.
First of all, with the new mlog refactor Critical wasn't a standard
log level so it was never to be logged.
And secondly, this completely missed the fact that there already
existed an API method Fatal which did exactly this.
We mark the Critical API as deprecated, and use Fatal for all
occurences.
The choice should be between either to use Error or Fatal.
While here, we also remove the non-standard exit codes which
just confused things further while debugging.
https://mattermost.atlassian.net/browse/MM-38712
```release-note
NONE
```
* Added a sqlXPassThrough layer
This layer embeds the original sqlx object and wraps some
key methods to avoid some boilerplate.
- Add timeout to all methods from the timeout in settings.
- Added trace functionality.
- Added auto-rebind capabilities so that each store method
doesn't need to call rebind every time.
- Added auto-lowering of query string to support NamedExec
for both MySQL and Postgres
All methods are not wrapped as all are likely not to be called
from our code. As and when they are called, support will be added.
```release-note
NONE
```
* fix lint issues
```release-note
NONE
```
* address review comments
```release-note
NONE
```
* incorporate suggestions
```release-note
NONE
```
* Prepare support for sqlx
Just setting up some basic skeleton to start
replacing gorp queries with sqlx.
There is still an issue with efficiently writing
cross-platform named queries, because :FieldName
needs to be :fieldname for Postgres.
The positional params can be rebound depending on
driver name. But named queries can't from my
investigation. Will look into this.
```release-note
NONE
```
* Refactor mapper
```release-note
NONE
```
* Forgot to init slice
```release-note
NONE
```
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* MM-35030: Hoist GetDBVersion before creating store
The creation of the store included running the migrations
which means that SQL statements would be executed before
we could make the check for DB version.
We perform the DB check before running the migrations.
https://mattermost.atlassian.net/browse/MM-35030
```release-note
NONE
```
* fix lint error
