Граф коммитов

24 Коммитов

Автор SHA1 Сообщение Дата
Rodrigo Villablanca
1d4a7c0552 TeamStore migration part 2 (#15396)
* Migration completed

* Fix tests

* Reduce to one line

* Fix: change to plain error

* Fix imports

* Trigger CI

* Fix i18

* Fix merge with master

* Trigger CI
2020-09-16 21:54:39 -03:00
Rodrigo Villablanca
f0fc51244c TeamStore migration Part 3 (#15397)
* Migration completed

* TODO
2020-09-16 12:12:10 -03:00
Rodrigo Villablanca
9ee9c78412 ChannelStore migration Part 2 (#15405)
* Migration completed

* Several corrections in tests

* Fix imports

* Fix some errors after testing

* Trigger CI

* Fix tests

* Suggestions

* Suggestions

* Add license

Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
2020-09-15 23:18:30 +05:30
Rodrigo Villablanca
2cb655ed67 Migration of ChannelStore Part 1 (#15235)
* Lint: remove unnecessary use of sprintf

* Fix i18n

* Returning the right error (InternalServerError)

* Doing some suggestions

* Fix store layers

* Fix missed translation

Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
2020-09-03 11:24:46 +05:30
Rodrigo Villablanca
54f86e7fb1 Migration of TeamStore Part 1 (#15246)
Automatic Merge
2020-09-03 00:29:57 -04:00
Abdulkadir Poyraz
f12ca27bac [MM-24522] remove duplication in OR and IncludeDeletedChannels params for search (#14573) 2020-08-31 13:40:58 +02:00
Rodrigo Villablanca
6a50106cd9 Filter stop words when searching posts in mysql (#14509) 2020-08-21 17:58:17 +02:00
Jesús Espino
19173ea6ff Migrating Team.SaveMember/SaveMultipleMembers into errors (#14924)
* Migrating Team.SaveMember/SaveMultipleMembers into errors

* Removing copy/paste comments

* Fixing another small thing

* Fixing tests

* Addressing PR review comments

* Fixing some tests

* Fixing golangci-lint

* Fixing linter

* Regenerating store layers

* Fix lint error after merge

* Other lint error fixed

Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
2020-08-12 21:02:41 +02:00
Rodrigo Villablanca
5566395032 First part of PostStore migration (#15123)
Automatic Merge
2020-08-12 13:35:57 -04:00
Agniva De Sarker
77f7a97bee MM-25563: Add endpoint to fetch only archived channels (#15031)
Automatic Merge
2020-07-27 13:31:39 +02:00
Mario de Frutos Dieguez
4c33b7a35d [MM-25767] Quick switch users search is always falling back to the database (#14762)
* Refactor of getListOfAllowedChannelsForTeam

Also, I've fixed some problematic scenarios:

- The quick search doesn't provide team id so it was always failing
- When the teamId was empty and view restrictions too we always
  return all the channels because if we do "strings.Contains("foo", "")
  it always returns true
- There was a case, in quick search with a guest account, where you
  get an empty result because teamId is not provided

* Error if team id is not passed when searching for the channel

If we search users passing the channel id, we must pass the team id
too so we avoid returning all the channels if we remove the empty
team id restriction we have in the getListOfAllowedChannelsForTeam

There is no known reason to search for a channel but not filtering
using the team id. Even guest accounts belong to a team
2020-06-26 20:37:35 +02:00
Mario de Frutos Dieguez
05ec3733c0 [MM-25406] Include missing methods in the search layer (#14799)
* Two missing methods to add in the channel layer

* Added delete user/channel posts methods

- Created in both search engines but only implemented in ES
- Add those methods in the search layer
- Included the PermanentDeleteByUser/Channel methods

* Two new delete documents are included in the bleve code with this
change:

- DeleteChannelPosts
- DeleteUserPosts

These two new functions delete post documents from the index-based
in the filed value provided
2020-06-25 13:45:39 +02:00
Rodrigo Villablanca
41d9c673cf Migrate GetChannels method from ChannelStore to return error interface (#14711)
* Migrate GetChannels method from ChannelStore to return error interface

* Fix testing

* Changed error type: ErrInvalidInput -> ErrNotFound

* Added note about error migrations

* Fix en.json

* Fix i18n

Co-authored-by: Agniva De Sarker <agnivade@yahoo.co.in>
2020-06-11 14:02:03 +05:30
Rodrigo Villablanca
6f28f3526d Migrate PermanentDelete method from ChannelStore to return error inte… (#14706)
Automatic Merge
2020-06-06 11:47:18 +02:00
Rodrigo Villablanca
60cc775cf6 Migrate Update method from ChannelStore to return idiomatic plain error (#14687)
* Partial advances

* Update migrated

* Fix imports

* Suggestions

* Suggestions

* Updating i18n

Co-authored-by: Jesús Espino <jespinog@gmail.com>
Co-authored-by: mattermod <mattermod@users.noreply.github.com>
2020-06-02 20:50:34 +05:30
Mario de Frutos Dieguez
f135e6c1bf [MM-24453] Sanitize user term search (#14494)
* Sanitize user term search

Remove the @ at the beginning of the search term in order to
make it possible for ES to return results
2020-05-24 10:50:07 +02:00
Agniva De Sarker
090743de86 MM-24132: Migrate AppError from SaveDirectChannel/channel_store.go (#14318)
* MM-24135: Migrate AppError from SaveChannel/channel_store.go

This is the first POC of migration of store app errors to plain error.

We create a few basic error types in the store package and use
them to return the errors from store methods. In the app layer,
we inspect the error and re-create the exact app errors. This lets
us preserve the same error content, but yet move to plain errors.

Since this is a gradual migration, this means that the error inspection
code will be duplicated across the app layer whenever a store method
is invoked. But all of that should go away once we start propagating
the errors higher up the hierarchy.

There have been a significant amount of changes in the storetest and searchtest
layer, primarily because we have to rename the err variable now that it is of
a different type.

* Addressed review comments

* MM-24132: Migrate AppError from SaveDirectChannel/channel_store.go

This PR migrates 2 new methods SaveDirectChannel and CreateDirectChannel
to return error instead of AppError.

We also need to handle the error internally in SaveMultipleMember for now
until that is migrated too.

* Fix layers

Co-authored-by: mattermod <mattermod@users.noreply.github.com>
2020-05-14 23:41:05 +05:30
Miguel de la Cruz
4fde004a5b [MM-23535] Add disable database search flag and return empty posts results if set (#14245)
* [MM-23535] Add disable database search flag and return empty posts results if set

* Add UpdateConfig function for the SearchStore and hook it into the app lifecycle

* Add the config listener in the server instance instead of using FakeApp

* Instantiate searchlayer as a pointer to avoid passing around copies of it
2020-05-13 14:00:57 +02:00
Agniva De Sarker
729a84a3e6 MM-24135: Migrate AppError from SaveChannel/channel_store.go (#14299)
* MM-24135: Migrate AppError from SaveChannel/channel_store.go

This is the first POC of migration of store app errors to plain error.

We create a few basic error types in the store package and use
them to return the errors from store methods. In the app layer,
we inspect the error and re-create the exact app errors. This lets
us preserve the same error content, but yet move to plain errors.

Since this is a gradual migration, this means that the error inspection
code will be duplicated across the app layer whenever a store method
is invoked. But all of that should go away once we start propagating
the errors higher up the hierarchy.

There have been a significant amount of changes in the storetest and searchtest
layer, primarily because we have to rename the err variable now that it is of
a different type.

* Addressed review comments

* Made all appError origins to be CreateChannel

* Remove typed internal error

* Fix translations

* fix layer generation

Co-authored-by: mattermod <mattermod@users.noreply.github.com>
2020-05-12 21:23:41 +05:30
Jesús Espino
1e89ef7314 Fixing searching users without team (#14182) 2020-04-02 17:22:08 +02:00
Jesse Hallam
e276a2bf57 avoid panic in search channel store (#14209)
Seen in Kibana logs:
```
http: panic serving 10.128.148.31:47536: runtime error: invalid memory address or nil pointer dereference
goroutine 1063732 [running]:
net/http.(*conn).serve.func1(0xc0042995e0)
	net/http/server.go:1767 +0x139
panic(0x1a4e080, 0x2edab40)
	runtime/panic.go:679 +0x1b2
github.com/mattermost/mattermost-server/v5/model.(*AppError).Error(0x0, 0xc000032000, 0x1a618c0)
	github.com/mattermost/mattermost-server/v5@/model/utils.go:73 +0x26
go.uber.org/zap/zapcore.encodeError(0x1c9c1d0, 0x5, 0x1fd6cc0, 0x0, 0x2028c60, 0xc0089fdfb0, 0xc0089fdfb0, 0x34)
	go.uber.org/zap@v1.13.0/zapcore/error.go:46 +0x3b
go.uber.org/zap/zapcore.Field.AddTo(0x1c9c1d0, 0x5, 0x19, 0x0, 0x0, 0x0, 0x1aa8c00, 0x0, 0x2028c60, 0xc0089fdfb0)
	go.uber.org/zap@v1.13.0/zapcore/field.go:165 +0xc8c
go.uber.org/zap/zapcore.addFields(0x2028c60, 0xc0089fdfb0, 0xc001df0800, 0x2, 0x2)
	go.uber.org/zap@v1.13.0/zapcore/field.go:199 +0xcf
go.uber.org/zap/zapcore.(*jsonEncoder).EncodeEntry(0xc00167d200, 0x2, 0xbf9678532a114ede, 0x3d02c134ae83, 0x2f284e0, 0x0, 0x0, 0x1d0d926, 0x2a, 0x1, ...)
	go.uber.org/zap@v1.13.0/zapcore/json_encoder.go:376 +0x1e4
go.uber.org/zap/zapcore.(*ioCore).Write(0xc00167d230, 0x2, 0xbf9678532a114ede, 0x3d02c134ae83, 0x2f284e0, 0x0, 0x0, 0x1d0d926, 0x2a, 0x1, ...)
	go.uber.org/zap@v1.13.0/zapcore/core.go:86 +0xa9
go.uber.org/zap/zapcore.(*CheckedEntry).Write(0xc004b2e8f0, 0xc001df0800, 0x2, 0x2)
	go.uber.org/zap@v1.13.0/zapcore/entry.go:216 +0x117
go.uber.org/zap.(*Logger).Error(0xc001fc4f00, 0x1d0d926, 0x2a, 0xc001df0800, 0x2, 0x2)
	go.uber.org/zap@v1.13.0/logger.go:203 +0x7f
github.com/mattermost/mattermost-server/v5/mlog.(*Logger).Error(...)
	github.com/mattermost/mattermost-server/v5@/mlog/log.go:184
github.com/mattermost/mattermost-server/v5/store/searchlayer.(*SearchChannelStore).SaveMember(0xc0009749c0, 0xc003e69a40, 0x2f284e0, 0x0)
	github.com/mattermost/mattermost-server/v5@/store/searchlayer/channel_layer.go:84 +0x2ed
github.com/mattermost/mattermost-server/v5/store.(*TimerLayerChannelStore).SaveMember(0xc000974ae0, 0xc003e69a40, 0xc000974ae0, 0xc)
	github.com/mattermost/mattermost-server/v5@/store/timer_layer.go:1559 +0x66
github.com/mattermost/mattermost-server/v5/app.(*App).createGroupChannel(0xc006d54f00, 0xc0085d1740, 0x3, 0x4, 0xc00a7396a0, 0x1a, 0xc008a319a0, 0xc0089fd2c0)
	github.com/mattermost/mattermost-server/v5@/app/channel.go:431 +0x6de
github.com/mattermost/mattermost-server/v5/app.(*App).CreateGroupChannel(0xc006d54f00, 0xc0085d1740, 0x3, 0x4, 0xc00a7396a0, 0x1a, 0x0, 0x1710a4516c7)
	github.com/mattermost/mattermost-server/v5@/app/channel.go:371 +0x81
github.com/mattermost/mattermost-server/v5/api4.createGroupChannel(0xc0085d1640, 0x1ff94a0, 0xc003e698f0, 0xc004dfae00)
	github.com/mattermost/mattermost-server/v5@/api4/channel.go:551 +0x437
github.com/mattermost/mattermost-server/v5/web.Handler.ServeHTTP(0xc0050cb6f0, 0x1da88d8, 0x2b0cb00, 0x12, 0x10001, 0x0, 0x0, 0x1ff94a0, 0xc003e698f0, 0xc004dfae00)
	github.com/mattermost/mattermost-server/v5@/web/handlers.go:212 +0x1e9a
github.com/NYTimes/gziphandler.GzipHandlerWithOpts.func1.1(0x1ff8f20, 0xc0060882a0, 0xc004dfae00)
	github.com/NYTimes/gziphandler@v1.1.1/gzip.go:336 +0x23f
net/http.HandlerFunc.ServeHTTP(0xc004024120, 0x1ff8f20, 0xc0060882a0, 0xc004dfae00)
	net/http/server.go:2007 +0x44
github.com/gorilla/mux.(*Router).ServeHTTP(0xc001680180, 0x1ff8f20, 0xc0060882a0, 0xc004dfac00)
	github.com/gorilla/mux@v1.7.3/mux.go:212 +0xe2
net/http.serverHandler.ServeHTTP(0xc00140c540, 0x1ff8f20, 0xc0060882a0, 0xc004dfac00)
	net/http/server.go:2802 +0xa4
net/http.(*conn).serve(0xc0042995e0, 0x1ffeee0, 0xc0063ee780)
	net/http/server.go:1890 +0x875
created by net/http.(*Server).Serve
	net/http/server.go:2927 +0x38e
```
2020-04-01 16:40:12 -03:00
Miguel de la Cruz
4fe25b1cdd [MM-21551] Add search tests structure to test the search engines (#14031)
* WIP

* Adding bleve to go modules

* WIP

* Adding missing files from searchengine implementation

* WIP

* WIP

* WIP

* WIP

* WIP

* WIP

* User and channel indexing and searches implemented

* Make bleve tests run with in-memory indexes

* Implement post index and deletion tests

* Initial commits for the search layer

* Removing unnecesary indexing

* WIP

* WIP

* More fixes for tests

* Adding the search layer

* Finishing the migration of searchers to the layer

* Removing unnecesary code

* Allowing multiple engines active at the same time

* WIP

* Add simple post search

* Print information when using bleve

* Adding some debugging to understand better how the searches are working

* Making more dynamic config of search engines

* Add post search basics

* Adding the Purge API endpoint

* Fixing bleve config updates

* Adding missed file

* Regenerating search engine mocks

* Adding missed v5 to modules imports

* fixing i18n

* Fixing some test around search engine

* Removing all bleve traces

* Cleaning up the vendors directory and go.mod/go.sum files

* Regenerating timer layer

* Adding properly the license

* Fixing govet shadow error

* Fixing some tests

* Fixing TestSearchPostsFromUser

* Fixing another test

* Fixing more tests

* Fixing more tests

* Removing SearchEngine redundant text from searchengine module code

* Fixing some reindexing problems in members updates

* Fixing tests

* Addressing PR comments

* Reverting go.mod and go.sum

* Addressing PR comments

* Fixing tests compilation

* Fixing govet

* Adding search engine stop method

* Being more explicit on where we use includeDeleted

* Adding GetSqlSupplier test helper method

* Mocking elasticsearch start function

* Fixing tests

* Search tests

* Fix tests

* Fix mod

* Fixing searchEngine for test helpers with store mocks

* Remove loglines

* Fix i18n strings

* Migrate search posts tests

* Fix linter

* Do not run search tests if -short flag is enabled

* Migrate back store tests that didn't belong to the searchlayer

* Fix scopelint issues

Co-authored-by: Jesús Espino <jespinog@gmail.com>
2020-03-30 19:17:40 +02:00
Miguel de la Cruz
11debb55a6 [MM-23510] Avoid race condition searching post's channel by channel id instead of post id (#14154) 2020-03-25 23:50:11 +01:00
Jesús Espino
c66e182b08 Adding the new search engine abstraction (#13304)
* WIP

* Adding bleve to go modules

* WIP

* Adding missing files from searchengine implementation

* WIP

* WIP

* WIP

* WIP

* WIP

* WIP

* User and channel indexing and searches implemented

* Make bleve tests run with in-memory indexes

* Implement post index and deletion tests

* Initial commits for the search layer

* Removing unnecesary indexing

* WIP

* WIP

* More fixes for tests

* Adding the search layer

* Finishing the migration of searchers to the layer

* Removing unnecesary code

* Allowing multiple engines active at the same time

* WIP

* Add simple post search

* Print information when using bleve

* Adding some debugging to understand better how the searches are working

* Making more dynamic config of search engines

* Add post search basics

* Adding the Purge API endpoint

* Fixing bleve config updates

* Adding missed file

* Regenerating search engine mocks

* Adding missed v5 to modules imports

* fixing i18n

* Fixing some test around search engine

* Removing all bleve traces

* Cleaning up the vendors directory and go.mod/go.sum files

* Regenerating timer layer

* Adding properly the license

* Fixing govet shadow error

* Fixing some tests

* Fixing TestSearchPostsFromUser

* Fixing another test

* Fixing more tests

* Fixing more tests

* Removing SearchEngine redundant text from searchengine module code

* Fixing some reindexing problems in members updates

* Fixing tests

* Addressing PR comments

* Reverting go.mod and go.sum

* Addressing PR comments

* Fixing tests compilation

* Fixing govet

* Adding search engine stop method

* Being more explicit on where we use includeDeleted

* Adding GetSqlSupplier test helper method

* Mocking elasticsearch start function

* Fixing tests

Co-authored-by: Miguel de la Cruz <miguel@mcrx.me>
Co-authored-by: mattermod <mattermod@users.noreply.github.com>
2020-03-13 15:33:18 +01:00