* Rework default password requirements
* Update API Test Lib Default User PW
* Remove unused function; Disable password reqs in dev mode
* Disable strict password requirements for unit tests
* Update unit tests
* MM-15021: Adding new builtin system schema for Guests
* Fixing tests
* Setting properly the permissions
* Adding guests to sampledata
* Restrict more roles updates in the app layer for guests
* Adding comment to explain that permissions migration must go at the end
* Setting the default guest role for custom scheme during migration
* Fixing import and export
* Creating scheme guest roles on migration
* Fixing tests
* Fixing tests
* Fixing tests
* Migrating audit store
* Final migration example for the audit store
* async example
* Ending migration
* Removing Async helper
* Fixing tests
* Fixing govet problems with the StoreResult instanstiation
* MM-147753: Verifies that users are allowed to be members of a team or a channel, based on group constraints, prior to allowing the API to add them.
* MM-14753: Allow methods to return meaningful results for deleted teams or channels.
* MM-14753: Renames methods to differentiate from permissions and other team and channel restrictions.
* MM-14753: Only check if users are team/channel members if team/channel is group constrained.
* MM-14753: Updates test function names.
* MM-14753: Changes a few method signatures.
* MM-14753: Small refactor and adds missing returns.
* MM-14753: Changes method names from Get* to Filter* name prefixes.
* MM-14753: Renames error variables.
* MM-14753: Updates method names for consistency with join table names.
* MM-14753: Adds case for non AppError return.
* Update i18n/en.json
* TestGetLicenseFileFromDisk: avoid using fileutils.FindConfigFile
* config: abstract config-related file access, extend memory store
* simplify config validate to avoid file knowledge
* fix relative file tests
* cluster: fix ConfigChanged event
The old and new configurations were swapped when notifying the enterprise code of configuration changes, creating needless instability in propagating config updates across a cluster.
* config/database: ignore duplicates
* test cleanup
* remove unnecessary Save() in test
* Change eMail as a post-verification action
* Fix broken test
* comment for special behavior, tests
* govet
* Check for already existent eMails when require email verification is turned on before accepting update
* vendor github.com/jmoiron/sqlx
* MM-11262: introduce a database store
* revert unnecessary fmt.Errorf
* simplify unit test helper methods
* remote TODO re: retry
* relocate initializeConfigurationsTable for clarity
* factor out a commonStore
* acquire database config lock on close for safety
* add missing header
* fix lock comment
* refactor utils/config* to config/
* pull validateLdapFilter into app
* clean up Config/GetConfig/GetSanitizedConfig usage
Eliminate app.GetConfig() in favour of just using app.Config() directly,
but expose app.GetSanitizedConfig() for when the old behaviour was
required.
* web: isolate config setup
* TestInvitePeopleProvider: make config explicit
* regenerateClientConfig: avoid racey map access
* integrate watch flag into app.ConfigFile option
* make app.Option return an error
* release.mk: only cp static files from config/
* release.mk: fix cp static files from config/
* api4: TestPlugin cleanup
* s/c/cfg/ for clarity
* fix merge conflict
* testlib: allow customization of testlib driver name
* MM 10658 Change config fields to pointers (#8898)
* Change fields of config structs to pointers and set defaults
MM-10658 https://github.com/mattermost/mattermost-server/issues/8841
* Fix tests that go broken during switching config structs to pointers
MM-10658 https://github.com/mattermost/mattermost-server/issues/8841
* Apply changes of current master while switching config structs to pointers
MM-10658 https://github.com/mattermost/mattermost-server/issues/8841
* Fix new config pointer uses
* Fix app tests
* Fix mail test
* remove debugging statement
* fix TestUpdateConfig
* assign config consistently
* initialize AmazonS3Region in TestS3TestConnection
* initialize fields for TestEmailTest
* fix TestCheckMandatoryS3Fields
* MM-7633: Optimize memory utilization during file uploads
Refactored the file upload code to reduce redundant buffering and stream
directly to the file store. Added tests.
Benchmark results:
```
levs-mbp:mattermost-server levb$ go test -v -run nothing -bench Upload -benchmem ./app
...
BenchmarkUploadFile/random-5Mb-gif-raw-ish_DoUploadFile-4 10 122598031 ns/op 21211370 B/op 1008 allocs/op
BenchmarkUploadFile/random-5Mb-gif-raw_UploadFileTask-4 100 20211926 ns/op 5678750 B/op 126 allocs/op
BenchmarkUploadFile/random-5Mb-gif-UploadFiles-4 2 1037051184 ns/op 81806360 B/op 3705013 allocs/op
BenchmarkUploadFile/random-5Mb-gif-UploadFileTask-4 2 933644431 ns/op 67015868 B/op 3704410 allocs/op
BenchmarkUploadFile/random-2Mb-jpg-raw-ish_DoUploadFile-4 100 13110509 ns/op 6032614 B/op 8052 allocs/op
BenchmarkUploadFile/random-2Mb-jpg-raw_UploadFileTask-4 100 10729867 ns/op 1738303 B/op 125 allocs/op
BenchmarkUploadFile/random-2Mb-jpg-UploadFiles-4 2 925274912 ns/op 70326352 B/op 3718856 allocs/op
BenchmarkUploadFile/random-2Mb-jpg-UploadFileTask-4 2 995033336 ns/op 58113796 B/op 3710943 allocs/op
BenchmarkUploadFile/zero-10Mb-raw-ish_DoUploadFile-4 30 50777211 ns/op 54791929 B/op 2714 allocs/op
BenchmarkUploadFile/zero-10Mb-raw_UploadFileTask-4 50 36387339 ns/op 10503920 B/op 126 allocs/op
BenchmarkUploadFile/zero-10Mb-UploadFiles-4 30 48657678 ns/op 54791948 B/op 2719 allocs/op
BenchmarkUploadFile/zero-10Mb-UploadFileTask-4 50 37506467 ns/op 31492060 B/op 131 allocs/op
...
```
https://mattermost.atlassian.net/browse/MM-7633https://github.com/mattermost/mattermost-server/issues/7801
[Place an '[x]' (no spaces) in all applicable fields. Please remove unrelated fields.]
- [x] Added or updated unit tests (required for all new features)
- [ ] Added API documentation (required for all new APIs)
- [ ] All new/modified APIs include changes to the drivers
*N/A*???
- [x] Includes text changes and localization file ([.../i18n/en.json](https://github.com/mattermost/mattermost-server/blob/master/i18n/en.json)) updates
Overview of changes:
- api4
- Replaced `uploadFile` handler with `uploadFileStream` that reduces
unnecessary buffering.
- Added/refactored tests for the new API.
- Refactored apitestlib/Check...Status functions.
- app
- Added App.UploadFileTask, a more efficient refactor of UploadFile.
- Consistently set `FileInfo.HasPreviewImage`
- Added benchmarks for the new and prior implementations
- Replaced passing around `*image.Image` with `image.Image` in the
existing code.
- model
- Added a more capable `client4.UploadFiles` API to match the new server
API’s capabilities.
- I18n
- Replaced `api.file.upload_file.bad_parse.app_error` with a more generic
`api.file.upload_file.read_request.app_error`
- plugin
- Removed type `plugin.multiPluginHookRunnerFunc` in favor of using
`func(hooks Hooks) bool` explicitly, to help with testing
- tests
- Added test files for testing images
Still remaining, but can be separate PRs - please let me know the preferred
course of action
- Investigate JS client API - how does it do multipart?
- Performance loss from old code on (small) image processing?
- Deprecate the old functions, change other API implementations to use
UploadFileTask
Definitely separate future PRs - should I file tickets foe these?
- Only invoke t.readAll() if there are indeed applicable plugins to run
- Find a way to leverage goexif buffer rather than re-reading
Suggested long-term improvements - should I file separate tickets for these?
- Actually allow uploading of large (GB-sized) files. This may require a
change in how the file is passed to plugins.
- (Many) api4 tests should probably be subtests and share a server setup -
will be much faster
- Performance improvements in image processing (goexif/thumbnail/preview)
(maybe use https://mattermost.atlassian.net/browse/MM-10188 for this)
Questions:
1. I am commiting MBs of test images, are there better alternatives? I can
probably create much less dense images that would take up considerably less
space, even at pretty large sizes
2. I18n: Do I need to do anything special for the string change? Or just wait
until it gets picked up and translated/updated?
3. The image dimensions are flipped in resulting FileInfo to match the actual
orientation. Is this by design? Should add a test for it, perhaps?
4. What to do in the case of partial success? S3 but not DB, some files but not
others? For now, just doing what the old code did, I think.
5. Make maxUploadDrainBytes configurable? Also, should this be the systemic
behavior of all APIs with non-empty body? Otherwise dropped altogether?
Check all other ioutil.ReadAll() from sockets. Find a way to set a total
byte limit on request Body?
* WIP - Fixed for GetPluginsEnvironment() changes
* WIP - PR feedback
1. Refactored checkHTTPStatus to improve failure messages
2. Use `var name []type` rather than `name := ([]type)(nil)`
3. Replaced single-letter `p` with a more intention-revealing `part`
4. Added tests for full image size, `HasPreviewImage`
* WIP - rebased (c.Session->c.App.Session)
* WIP - PR feedback: eliminated use of Request.MultipartReader
Instead of hacking the request object to use r.MultipartReader now have own
functions `parseMultipartRequestHeader` and `multipartReader` eliminating the
need to hack the request object to use Request.MultipartReader limitations.
* WIP - PR feedback: UploadFileX with functional options
* WIP - PR feedback: style
* WIP - PR feedback: errors cleanup
* WIP - clarified File Upload benchmarks
* WIP - PR feedback: display the value of erroneous formname
* WIP - PR feedback: fixed handling of multiple channel_ids
* WIP - rebased from master - fixed tests
* PR Feedback
* PR feedback - moved client4.UploadFiles to _test for now
* testlib: introduce and leverage
This doesn't yet factor out the individual test helpers: many packages
still rely on `api4` directly to do this, but now wire up the test store
setup through this package. `app` and `store`, in particular, don't use
`testlib` because of circular dependencies at the moment.
* cmd: command_test.go: use api4 testlib
* cmd: plugin_test.go: remove dependence on test-config.json
* cmd: config_test.go use configured database settings
* ensure test-(te|ee) exit with status code
* test-server: run all tests, deprecating test-te/test-ee
* cmd/mattermost/commands: fix unit tests
Instead of relying on (and modifying) a config.json found in the current path, explicitly create a temporary one from defaults for each test. This was likely the source of various bugs over time, but specifically allows us to override the SqlSettings to point at the configured test database for all tests simultaneously.
* wrap run/check into a test helper
It was insufficient to set a config for each invocation of CheckCommand or RunCommand: some tests relied on the config having changed in a subsequent assertion. Instead, create a new test helper embedding api4.TestHelper. This has the nice advantage of cleaning up all the teardown.
* additional TestConfigGet granularity
* customized config path to avoid default location
* be explicit if the storetest initialization fails
* generate safe coverprofile names in the presence of subtests
* additional TestConfigShow granularity
* fix permission_test.go typo
* fix webhook tests
* actually flag.Parse() to skip database setup on os.Execed tests
* fix recent regression in #9962, not caught by unit tests
* refactor GetDirectChannel and CreateDirectChannel in one function
* remove CreateDirectChannel plugin api and update GetDirectChannel and GetGroupChannel plugin api
* update tests
* api4: improved error handling
* system_store: more logs
* integrate go-junit-report into test-te/test-ee
* add CI_MINIO_HOST and CI_INBUCKET_HOST instead of CI_HOST
* comment re: minio configuration issue
* fix TestStartServerPortUnavailable to pass even when root can bind to :21
* skip TestFindManifest_FolderPermission while running as root
* fix webconn close semantics
Avoid race conditions in WebConn on shutdown by closing channels to guarantee all readers are notified. Wrap this with sync.Once to avoid closing the channel more than once.
* web_hub_test.go
* webhub: fix race condition on shutdown
Ensure that if the webhub shuts down in the process of sending, the caller unblocks given that the webhub will no longer consume incoming events.
* panic if app shutdown takes >30 seconds
* simplify WebConn::Pump channel semantics too
* api4: fix TestGetUsersNotInTeam assertions
This test was relying on data from a previous test run. With the data cleared before each test, the assertions much match reality.
* *testlib: always InitSystemAdmin
Some tests implicitly relied on the basic user having system
administrator privileges because it was the first user created as such.
Eliminate `InitSystemAdmin` and explicitly create the system admin user
instead to avoid this ambiguity going forward.
* *testlib: drop all tables before each test
* api4: split up TestChannelDelete to avoid duplicate InitBasic
* api4: teardown in TestResetPassword, for when this test comes back
* invalidate cache on DropAllTables
This is necessary since the test store persists across tests.
* disable parallel tests
While tests within a package must be explicitly parallelized using `t.Parallel()`, tests across packages are run in parallel by default. This causes problems given that the tests all currently share the same database instance.
Unfortunately, this also means that running the tests is much slower, but we can return to this later.
* Move OpenGraph code into its own file
* Move OpenGraph image proxying to app layer
* Move test file code out of api4 package
* MM-11272 Add OpenGraph and image dimension metadata to posts
* initial implementation of after, before, on search flags allowing to restrict the search to a specific day or a date range
* missed setting beforeDate in SearchParams in one place
* fixed condition when only flags are used for search without any plain terms
* changed date format used for after/before/on flags to be in ISO8601 format as suggested in PR comments, added a helper function to pad month and day with zeroes allowing the user user either format, with or without leading zeroes
* corrected expected compare to date setting for the TestParseDateFilterToTimeISO8601 test
* fixed a bug for the scenario when you only have the date flags without any terms, added a couple of tests for that scenario
* updated the date filter logic to use parameters to construct the query instead of simply appending strings together, as suggested in the pull request comments
* added search unit test using date flags
* added a helper function to create a test post with a createat date manually set, updated the test for search using date flags to create test posts with different createat dates to be able to better test the functionality
* MM-11817 Add support for after/before/on search flags with Elasticsearch
* add support to search posts to perform the search in context of the client's timezone when filtering by createat date using on: after: before: flags
* updated tests to match the new signature
* MM-11065: Allow to search and get archived channels from the API
* Fixing more tests
* Add some unit tests
* Add includeDeleted parameter to session permissions check function
* More test fixing
* Adding archive channels list in channels search
* Add restriction for archived channel edition
* Reverting permissions checks modification
* Changed the query parameter to include_deleted
* Enable search archive channels as true by default
* Adding tests for verify search on deleted channels
* Allowing to override archive channels during the imports
* Fixed test
* Search in archive channels from the API must be explicitly requested
* Removing includeDeleted parameter from GetChannelByName and GetChannelByNameForTeam
* Back to ViewArchivedChannels config
* Fixing tests
* Reverting GetChannelByName parameter
* Add include deleted parameter on GetChannel functions in plugins api
* Fixing tests
* Add missing diagnostics (#8911)
* Update diagnostics.go
* Update diagnostics.go
* Fix push notification styling backwards compatibility (#8913)
* MM-10803: remove premature user sanitization on deactivation (#8926)
* remove unused UpdateNonSSOUserActive
* MM-10803: stop prematurely sanitizing users on deactivate
This change was preceded by the removal of UpdateNonSSOUserActive to
ensure there are no APIs relying on the sanitized return value.
* MM-10803: test websocket events after UpdateUserActive
* MM-10264: Adds system scheme to permissions import/export. (#8924)
* MM-10264: Adds system scheme to permissions import/export.
* MM-10264: Switches to more likely unique name.
* MM-10264: Changed collision prevention string.
* MM-10264: Rolls back created schemes in all error cases.
* MM-10264: Test fix for more rollback cases.
* Remove api package
* Remove api dependency from cmd package
* Remove EnableAPIv3 setting
* Update web tests
* Add more websocket tests
* Move some ws and oauth tests to api4 package
* Move command tests into api4 package
* Test fixes
* Fix msg command test
* Add some app file tests
* Implementing structured logging
* Changes to en.json to allow refactor to run.
* Fixing global logger
* Structured logger initalization.
* Add caller.
* Do some log redirection.
* Auto refactor
* Cleaning up l4g reference and removing dependancy.
* Removing junk.
* Copyright headers.
* Fixing tests
* Revert "Changes to en.json to allow refactor to run."
This reverts commit fd8249e99bcad0231e6ea65cd77c32aae9a54026.
* Fixing some auto refactor strangeness and typo.
* Making keys more human readable.
* Add Scheme model and stub store.
* Port ChannelStore to be Scheme aware.
* Make almost all the API/APP layer work with ChannelSchemes.
Only thing still hacky is UpdateChannelMemberRoles().
* Add basic SchemeStore implementation.
* Migrate UpdateChannelMemberRoles properly and fix tests.
* Update store tests and mocks so they work.
* Include creating default roles in Scheme create store function.
* Implement role deletion and start scheme deletion.
* Only use non-deleted roles for authorization.
* Add GetByScheme method to Team store.
* Add GetChannelsByScheme.
* Update store mocks.
* Implement scheme deletion in the store.
* Rename is valid function.
* Add offset and limit to queries to fetch teams and channels by scheme.
* Fix queries.
* Implement scheme awareness in Team store and add a migration.
* Tidy up ChannelStore mapping functions and add exhaustive unit tests.
* Add all missing i18n.
* Proper tests for TeamStore internal functions and fix them.
* Make additional TeamMember fields nullable.
* Make new ChannelMember fields nullable.
* Create new nullable columns without defaults.
* Make new fields in large tables nullalble.
* Fix empty list of TeamMembers.
* Deduplicate SQL queries.
* Fix spelling.
* Fix review comment.
* More review fixes.
* More review fixes.
* failing to find i18n shouldn't segfault
The server was trying to handle the fact that it couldn't find the i18n
directory, by emitting a translated log message...
* fix utils.FindDir
The attempts to find the directory in the parent or grandparent directory
don't work if the current working directory was inside `enterprise`, with
`enterprise` itself being a symlink as per the usual developer setup.
Recurse to the root of the filesystem, cleaning the path along the way
to work around this limitation (and allow tests to be run from an
arbitrarily deep nesting level.)
Fix corresponding usages to employ filepath.Join.
* failing to find html templates shouldn't segfault
* fail fast if the test user cannot be created
* rework utils.FindDir to retain backwards compatibility
Only a handful of critical errors are present in the codebase.
They all occur during server startup (in `app.StartServer()`).
Currently, when one of these critical error occurs, it is simpled
mentionned in the logs – then the error is discarded, and the app
attempts to continue the execution (and probably fails pretty quickly in
a weird way).
Rather than continuing operations in an unknow state, these errors should
trigger a clean exit.
This commit rewrites critical startup errors to be correctly
propagated, logged, and then terminate the command execution.
Additionnaly, it makes the server return a proper error code to the
shell.
* XYZ-13: Update Permission and Role structs to new design.
* XYZ-10: Role store.
* XYZ-9/XYZ-44: Roles API endpoints and WebSocket message.
* XYZ-8: Switch server permissions checks to store backed roles.
* XYZ-58: Proper validation of roles where required.
* XYZ-11/XYZ-55: Migration to store backed roles from policy config.
* XYZ-37: Update unit tests to work with database roles.
* XYZ-56: Remove the "guest" role.
* Changes to SetDefaultRolesFromConfig.
* Short-circuit the store if nothing has changed.
* Address first round of review comments.
* Address second round of review comments.
