* Add psd preview support
* Adding required vendor files
* Adding oov/psd license to NOTICE.txt file
* Adding the license notice for gopherjs to the NOTICE.txt
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* Added 2 checks for app/file.go
- Check if channel id exist
- Check if user has permission to the upload file to the channel
Also added translations for 2 errors defined in app/file.go
* fixed 1 failing test that was linked with UploadFile
* Fixed small issue, according to the review.
* missed 1 review item. Just updated the code for it.
* fix 1 failing test, assuming that the file upload is required. ignoring nouser idea.
Added the translation for english for 2 newly defined errors.
* removed new line
* trying to fix the translation issue. Added the missing translations from master.
* as per discussion, we need to revert the check for user channel permission. So reverted it.
* Update app/file_test.go
Co-Authored-By: Alejandro García Montoro <alejandro.garciamontoro@gmail.com>
* Update i18n/en.json
Co-Authored-By: Alejandro García Montoro <alejandro.garciamontoro@gmail.com>
* Update file.go
Move the check to the top of the method.
* go fmt
Co-authored-by: mattermod <mattermod@users.noreply.github.com>
Co-authored-by: Alejandro García Montoro <alejandro.garciamontoro@gmail.com>
Co-authored-by: Ben Schumacher <ben.schumacher@mattermost.com>
* MM-20934: Fixing int overflow in 32 bits on MaxImageSize check
* Adding comments explaining the casting and the bug fixed there
* Apply suggestions from code review
Co-Authored-By: Juho Nurminen <juhonurm@gmail.com>
* Fixing store layers
Co-authored-by: Juho Nurminen <juhonurm@gmail.com>
* constants and options for getting files
* Method to get files with options
* Add i18n strings for en
* Add API methods for getting files with options
* gofmt -s file
* explicitly set create at in tests
* use greater than nanosecond time difference for tests
* use gte instead of gt for getting files by created time
* use created at time as default sort order for getting file infos
* use explicit inline strings instead of format strings
* join tables only when required
* use if as secondary sort, and update tests
* update field docs to reflect previous changes
* make page and perPage get options as required
* add json struct tags to GetFileOptions
* bump minimum server versioni
* remove sorting by username and channelname
* use bool for sort order type
* use FileInfo prefix instead of just File
* clearer comments
* use zero-based page numbering
* test filtering by user and channel
* remove unnecessary whitespace
* use int instead of uint for page and perPage arguments
Co-authored-by: mattermod <mattermod@users.noreply.github.com>
* MM-22057: Limit incoming request bodies
Set the max request body size to be equal to MaxFileSize.
Ideally, non-file request bodies should be smaller than file request bodies,
but we don't have a clean way to identify all file upload handlers.
There shouldn't be any valid request which exceeds the max file upload size.
So this is a safe global limit to apply.
* Fix tests
Co-authored-by: mattermod <mattermod@users.noreply.github.com>
* Consistent license message for all the go files
* Fixing the last set of unconsistencies with the license headers
* Addressing PR review comments
* Fixing busy.go and busy_test.go license header
* MM-16272 - Synchronize plugins across cluster (#11611)
* MM-16272 - Synchronize plugins across cluster
* Adding a test
* MM-16272 - Fixed tests
* MM-16272 - PR feedback
* MM-16270 - Plugin Sync (#11615)
* Initial implementation for plugin synch with file store. WIP
* Removed ListAll implementation. Used ListDirectory and change localstore to be consistent and return all items (files and folders) from directory
* Refactored plugin filestore operations out of main install/remove plugin
* Fixing error handling details
* Changes to use structured logging
* More logging fixes
* Wording and comments improvements
* Error handling and control flow improvements
* Changed managed flag check to use os.stat
* Added file store plugin dir and filename consts
* Replaced FileRead to use a the FileReader in PluginSync
* Minor styling and PR feedback changes
* Minor error handling improvements
* Added unit test for SyncPlugins. Changed SyncPlugins to use plugins environment to list available plugins
* PR Feedback improvements
* Minor err handling fix
* Removing FileStorePath from PluginEventData (#11644)
* Fix plugin path (#11654)
* tweak path, logging
Fix an issue not finding the plugins folder in S3. Tweak logging messages to add additional clarity.
* Removing FileExists check when Syncing plugins. Updated localstore to not return an error when directory does not exist
* PR Feedback
* Install prepackaged plugins locally only (#11656)
* s/uninstall/remove
* Updated ClusterMessage comment
* Updated PluginSync to test against s3 + local storage
* migrating get posts to sync by default
* flow control and style changes
* style changes, error checking
* pulling master down
* counting missed cache, not hit
* fixing bad conflict resolution
* forcing rebuild
* GH-10761: Migrate Post.Get to sync by default
* instantiate storeChannel with make
* fix go vet issue
* fix go vet issue
* use select for checking channel for value
* fix go vet shadow variable issue
* addressing review comments
* call post.Get() in goroutine
* refactor minor values to fix tests
* fix: remove err error type, it should be *model.AppErr
* MM-15198 Migrate Channel.Get/GetFromMaster to Sync by default
* MM-15198 - Update store/storetest/post_store.go
fix error handling in post_store.go test case
Co-Authored-By: andresoro <ao15@my.fsu.edu>
* MM-7633: Optimize memory utilization during file uploads
Refactored the file upload code to reduce redundant buffering and stream
directly to the file store. Added tests.
Benchmark results:
```
levs-mbp:mattermost-server levb$ go test -v -run nothing -bench Upload -benchmem ./app
...
BenchmarkUploadFile/random-5Mb-gif-raw-ish_DoUploadFile-4 10 122598031 ns/op 21211370 B/op 1008 allocs/op
BenchmarkUploadFile/random-5Mb-gif-raw_UploadFileTask-4 100 20211926 ns/op 5678750 B/op 126 allocs/op
BenchmarkUploadFile/random-5Mb-gif-UploadFiles-4 2 1037051184 ns/op 81806360 B/op 3705013 allocs/op
BenchmarkUploadFile/random-5Mb-gif-UploadFileTask-4 2 933644431 ns/op 67015868 B/op 3704410 allocs/op
BenchmarkUploadFile/random-2Mb-jpg-raw-ish_DoUploadFile-4 100 13110509 ns/op 6032614 B/op 8052 allocs/op
BenchmarkUploadFile/random-2Mb-jpg-raw_UploadFileTask-4 100 10729867 ns/op 1738303 B/op 125 allocs/op
BenchmarkUploadFile/random-2Mb-jpg-UploadFiles-4 2 925274912 ns/op 70326352 B/op 3718856 allocs/op
BenchmarkUploadFile/random-2Mb-jpg-UploadFileTask-4 2 995033336 ns/op 58113796 B/op 3710943 allocs/op
BenchmarkUploadFile/zero-10Mb-raw-ish_DoUploadFile-4 30 50777211 ns/op 54791929 B/op 2714 allocs/op
BenchmarkUploadFile/zero-10Mb-raw_UploadFileTask-4 50 36387339 ns/op 10503920 B/op 126 allocs/op
BenchmarkUploadFile/zero-10Mb-UploadFiles-4 30 48657678 ns/op 54791948 B/op 2719 allocs/op
BenchmarkUploadFile/zero-10Mb-UploadFileTask-4 50 37506467 ns/op 31492060 B/op 131 allocs/op
...
```
https://mattermost.atlassian.net/browse/MM-7633https://github.com/mattermost/mattermost-server/issues/7801
[Place an '[x]' (no spaces) in all applicable fields. Please remove unrelated fields.]
- [x] Added or updated unit tests (required for all new features)
- [ ] Added API documentation (required for all new APIs)
- [ ] All new/modified APIs include changes to the drivers
*N/A*???
- [x] Includes text changes and localization file ([.../i18n/en.json](https://github.com/mattermost/mattermost-server/blob/master/i18n/en.json)) updates
Overview of changes:
- api4
- Replaced `uploadFile` handler with `uploadFileStream` that reduces
unnecessary buffering.
- Added/refactored tests for the new API.
- Refactored apitestlib/Check...Status functions.
- app
- Added App.UploadFileTask, a more efficient refactor of UploadFile.
- Consistently set `FileInfo.HasPreviewImage`
- Added benchmarks for the new and prior implementations
- Replaced passing around `*image.Image` with `image.Image` in the
existing code.
- model
- Added a more capable `client4.UploadFiles` API to match the new server
API’s capabilities.
- I18n
- Replaced `api.file.upload_file.bad_parse.app_error` with a more generic
`api.file.upload_file.read_request.app_error`
- plugin
- Removed type `plugin.multiPluginHookRunnerFunc` in favor of using
`func(hooks Hooks) bool` explicitly, to help with testing
- tests
- Added test files for testing images
Still remaining, but can be separate PRs - please let me know the preferred
course of action
- Investigate JS client API - how does it do multipart?
- Performance loss from old code on (small) image processing?
- Deprecate the old functions, change other API implementations to use
UploadFileTask
Definitely separate future PRs - should I file tickets foe these?
- Only invoke t.readAll() if there are indeed applicable plugins to run
- Find a way to leverage goexif buffer rather than re-reading
Suggested long-term improvements - should I file separate tickets for these?
- Actually allow uploading of large (GB-sized) files. This may require a
change in how the file is passed to plugins.
- (Many) api4 tests should probably be subtests and share a server setup -
will be much faster
- Performance improvements in image processing (goexif/thumbnail/preview)
(maybe use https://mattermost.atlassian.net/browse/MM-10188 for this)
Questions:
1. I am commiting MBs of test images, are there better alternatives? I can
probably create much less dense images that would take up considerably less
space, even at pretty large sizes
2. I18n: Do I need to do anything special for the string change? Or just wait
until it gets picked up and translated/updated?
3. The image dimensions are flipped in resulting FileInfo to match the actual
orientation. Is this by design? Should add a test for it, perhaps?
4. What to do in the case of partial success? S3 but not DB, some files but not
others? For now, just doing what the old code did, I think.
5. Make maxUploadDrainBytes configurable? Also, should this be the systemic
behavior of all APIs with non-empty body? Otherwise dropped altogether?
Check all other ioutil.ReadAll() from sockets. Find a way to set a total
byte limit on request Body?
* WIP - Fixed for GetPluginsEnvironment() changes
* WIP - PR feedback
1. Refactored checkHTTPStatus to improve failure messages
2. Use `var name []type` rather than `name := ([]type)(nil)`
3. Replaced single-letter `p` with a more intention-revealing `part`
4. Added tests for full image size, `HasPreviewImage`
* WIP - rebased (c.Session->c.App.Session)
* WIP - PR feedback: eliminated use of Request.MultipartReader
Instead of hacking the request object to use r.MultipartReader now have own
functions `parseMultipartRequestHeader` and `multipartReader` eliminating the
need to hack the request object to use Request.MultipartReader limitations.
* WIP - PR feedback: UploadFileX with functional options
* WIP - PR feedback: style
* WIP - PR feedback: errors cleanup
* WIP - clarified File Upload benchmarks
* WIP - PR feedback: display the value of erroneous formname
* WIP - PR feedback: fixed handling of multiple channel_ids
* WIP - rebased from master - fixed tests
* PR Feedback
* PR feedback - moved client4.UploadFiles to _test for now
* guard app plugins with mutex
Shutting down the app could race with a goroutine that uses the plugins environment, since we shut down the plugins first before cleaning up goroutines.
* fix go vet issues
* Moving goroutine pool
* Auto refactor
* Moving plugins.
* Auto refactor
* Moving fields to server
* Auto refactor
* Removing siteurl duplication.
* Moving reset of app fields
* Auto refactor
* Formatting
* Moving niling of Server to after last use
* Fixing unit tests.
Fix code issues in channel_test.go
Fix Channel Test Issues detected by Megacheck
Fix API Emoji Test Issues detected by Megacheck
Fixed API Issues Reported by Megacheck
Fixed App issues reported by megacheck
Remaining fixes
removed test added by mistake from old HEAD
gofmt
Store Fixes
simplified returns
Fix test for multi member channel delete
revert to delete unused function
* Initial implementation of a CopyFileInfos function that creates new FileInfo objects copied from provided FileIds with the provided user as the creator and not linked to a post yet. This can subsequently be used to copy existing attachments from another post to attach to a new post without having to re-upload the actual files
* added a unit test for the CopyFileInfos function
* resolving pull request suggestions
