* Revert "Revert "MM-43828: Pass object length for some image operations (#20711)" (#20740)"
This reverts commit c7ae090dad.
* Fix to work in bifrost mode
```release-note
NONE
```
For user profile and plugin upload, we use a bytes.Buffer.
In that case, we know the object size and can find
it out from the length of the buffer.
This helps reduce multi-part uploads.
This approach can also be taken in thumbnail and preview
images. However, they use an io.Pipe to directly upload
the image as it is being encoded. We could make the whole
process in separate parts of writing the full image in the
buffer and then upload it. But taking a conservative approach
for now.
Also, while here, removed some unused code.
https://mattermost.atlassian.net/browse/MM-43828
```release-note
NONE
```
This creates faulty requests to Bifrost and results in
errors and warnings in the logs. Even without Bifrost,
this would make unnecessary requests to S3.
We only extract info from documents and therefore we can
safely avoid this.
```release-note
NONE
```
* [MM-44667] Set InsecureSkipVerify for S3 when EnableInsecureOutgoingConnections is set
* Cleanup s3New, customTransport
* Lint fixes
* Cleanup s3store_test.go
* Using default transport
```release-note
NONE
```
Co-authored-by: Jake Gutierrez <jakegut0108@gmail.com>
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* MM-38982: fixes SVG uploading
Uploading SVG files was "almost" treated like an image in some places,
resulting in not showing the SVG on the client.
This is happening because when we fail to generate a mini preview for an
image, we prepend "invalid-" to the MimeType.
This commit adds a check to guard against SVG files in methods that make
no sense for SVGs.
* Reverts invalid-{mimetype} fix
* Uses fileInfo.IsSvg() where it can
Just a quick POC to move fast :P
We use a search pointer to keep track of
the next row to inesrt to. For every new search
we increment the pointer and do modulo 5.
This means that the value will always remain
between 0-4. And that way, we will always overwrite
the oldest entry on every search.
And while getting the results, we get
all results for that user.
The search parameters are json marshalled
and stored as a JSON blob. This is because
there is no need to search/filter them
in the DB.
Pending items:
Tests obviously.
To improve:
The client needs to send the channel ids
instead of channel names.
We implement a cursor based pagination model
to page through the posts in a given thread.
The cursor is a combination of the post.CreateAt+
post.Id to differentiate multiple posts in a given
timestamp.
Some additional parameters like direction, fromPost,
fromCreateAt and perPage were introduced to implement
this.
```release-note
NONE
```
* MM-40179 traverse directories when trying to export
* fix test
* fix s3 to behabe the same way
* add test for paths way too deep
* test recursion only on localstorage
* make list directories non recursively the default approach
* fix linting error
* fix non-recursive case
Co-authored-by: = <=>
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
This is more of a general refactor of the initialization
process which should allow us to pass services more
easily.
The changes are minimal to keep the scope limited.
For now, the objective is to pass the file service
to the Channels product. For that, it was required
to move some of the enterprise interfaces under Channels
from Server.
We also create a filestore field in the server to
avoid creating filestore reference every time
we make a filestore operation. This will be later
passed on to the Channels product.
Also removed an unnecessary test.
The test was working so far because we were creating
the filebackend every time for every request. But
we should go via UpdateConfig call which would fail,
were we to assign an invalid filestore name.
So we were actually testing for a different thing.
Therefore, removed the test.
```release-note
NONE
```
* Move Channels into App
In this PR, we make Channels as part of App
instead of Server. This is part of the transition period
of moving fields from Server to Channels.
For now, Channels contains Server. So the hierarchy is
App -> Channels -> Server.
And as a first step, we also move httpService to Channels.
```release-note
NONE
```
* Fixing another test
```release-note
NONE
```
* new method
```release-note
NONE
```
* prevent invalid images from entering generateMiniPreview in the future
* fixing err, err2, imgErr
* linting
* load actual pngs and gifs in tests
* consistent style
* Add API endpoint and adapt search to allow multi-team search
* Refactor handler, refactor sql query to use squirrel, rename app and store functions and add tests
* Fix lint
* Fix search engines and remove unneeded comments
* Fix test
* Remove user from channel after test
* add debug info to generateMiniPreview error
* linting
* simplify
* Update app/file.go
Co-authored-by: Claudio Costa <cstcld91@gmail.com>
* add channel_id and creator_id to the log statement
Co-authored-by: Jesse Hallam <jesse.hallam@gmail.com>
Co-authored-by: Claudio Costa <cstcld91@gmail.com>
* MM-37491: Fix flaky test on file content extraction
* Changing Error log to Warn log
* Fix uppercase and dot in the log file
* Simplifiying the test case
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* add request context
* move initialialization to server
* use app interface instead of global app functions
* remove app context from webconn
* cleanup
* remove duplicated services
* move context to separate package
* remove finalize init method and move content to NewServer function
* restart workers and schedulers after adding license for tests
* reflect review comments
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* Removing FilesSearch feature flag
* Fixing tests
* Adding an improvement on plain text extraction
* Adding tests for plain text extraction
* Removed unneeded conversion
* Adding missed license
* Remove the feature flag from the migration
* Fixing some tests
* Updating i18n/en.json file
* Add extract documents content command
* Adding the extraction command and making the pure go pdf library as secondary option
* Improving the memory usage and docextractor interface
* Enable content extraction by default in all the instances
* Tiny improvement on archive indexing
* Adding App interface generation and the opentracing layer
* Fixing linter errors
* Addressing PR review comments
* Addressing PR review comments
* MM-31094: Adds tooling to develop and test using a MySQL instance with replication lag. Adds some lazy lookups to fallback to master if results are not found.
* MM-31094: Removes mysql-read-replica from default docker services.
* MM-31094: Switches (store..SessionStore).Get and (store.TeamStore).GetMember to using context.Context.
* MM-31094: Updates (store.UsersStore).Get to use context.
* MM-31094: Updates (store.PostStore).Get to use context.
* MM-31094: Removes feature flag and config setting.
* MM-31094: Rolls back some master reads.
* MM-31094: Rolls a non-cache read.
* MM-31094: Removes feature flag from the store.
* MM-31094: Removes unused constant and struct field.
* MM-31094: Removes some old feature flag references.
* MM-31094: Fixes some tests.
* MM-31094: App layers fix.
* MM-31094: Fixes mocks.
* MM-31094: Don't reparse flag.
* MM-31094: No reparse.
* MM-31094: Removed unused FeatureFlags field.
* MM-31094: Removes unnecessary feature flags variable declarations.
* MM-31094: Fixes copy-paste error.
* MM-31094: Fixes logical error.
* MM-30194: Removes test method from store.
* Revert "MM-30194: Removes test method from store."
This reverts commit d5a6e8529bd5f4d993824c828e239d009b05e567.
* MM-31094: Conforming to make's strange syntax.
* MM-31094: Configures helper for read replica with option.
* MM-31094: Adds some missing ctx's.
* MM-31094: WIP
* MM-31094: Updates test names.
* MM-31094: WIP
* MM-31094: Removes unnecessary master reads.
* MM-31094: ID case changes out of scope.
* MM-31094: Removes unused context.
* MM-31094: Switches to a helper. Removes some var naming changes. Fixes a merge error.
* MM-31094: Removes SQLITE db driver ref.
* MM-31094: Layer generate fix.
* MM-31094: Removes unnecessary changes.
* MM-31094: Moves test method.
* MM-31094: Re-add previous fix.
* MM-31094: Removes make command for dev.
* MM-31094: Fix for login.
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* Adding search files api
* Fixing golangci-lint
* Adding bulk-indexing and improving a bit the name indexing for bleve and elasticsearch
* Add content extraction config migration
* Fixing a problem with document extraction
* Unapplying certain changes moved to other PR
* Fixing tests
* Making extract content app migration a private method
* Addressing PR review comments
* Addressing PR review comments
* Adding feature flag
* Removing debug string
* Fixing imports
* Fixing linting errors
* Do not migrate the config if the feature flag is not enabled
* Fix tests
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* Reducing the filestore dependencies from the rest of the source code
* Making more generic config conversion to FileBackendSettings
* Fixing usage of the NewFileBackend function
* Fixing more usages of the NewFileBackend function
* Fix some linter errors
* Fix more linter errors
* Fixing some unit tests
* Fixing linter problem
* Addressing PR review comments
* Simplifing the CopyFile for tests
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* init commit
* clean up the code
* make mocks
* fix translations
* mocks and lint fixes
* add tests
* little fixes
* Update i18n/en.json
Co-authored-by: Scott Bishel <scott.bishel@mattermost.com>
* Update i18n/en.json
Co-authored-by: Scott Bishel <scott.bishel@mattermost.com>
* Update i18n/en.json
Co-authored-by: Scott Bishel <scott.bishel@mattermost.com>
* Update i18n/en.json
Co-authored-by: Scott Bishel <scott.bishel@mattermost.com>
* Update i18n/en.json
Co-authored-by: Scott Bishel <scott.bishel@mattermost.com>
* Address Comments
* fix i18n
* update api endpoint
* add enable file and file level for conditional show of banner
* Address Comments
* Make it more clear about returns
* Create zip file utility function
* update en.json
* address comments
* write tests
* check for data in test
* remove warning string
* Correct expected and actual
* set database through environment variables
* reset environment variable at end of test
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
Co-authored-by: Scott Bishel <scott.bishel@mattermost.com>
