* MM-11697: Environment overrides do not overwrite config.json on save
#10388
The config store now keeps a copy of the config as loaded from the store
without environment overrides. Whenever persisting, we now check if the
current setting is different from the loaded setting. If it is, then use
the loaded setting instead.
As described in the comments to `removeEnvOverrides` in `common.go`,
this behavior will have to change if we ever let the user change a
setting that has been environmentally overriden.
This was interesting because the `load` function in `common.go` also
persists, so we have to tee the provided `io.ReadCloser` and construct a
config that doesn't have the environment overrides. And then we have to
find the path to the (maybe) changed variable in the config struct
using reflection.
Possible WIP: I had to expose a `GetWithoutEnvOverrides` function in the
Store interface just for the tests -- this is because the `file_test`
and `database_test`s are in the config_test package instead of the
`config` package.
* added function documentation
* fixed a small problem with tests
* MM-11697: big cleanup based on Jesse's PR comments
* MM-11697: edits per PR feedback
* MM-11697: licence header
* MM-11697: now testing that on disk config is not changed by env
overrides
* MM-11697: remove unneeded exports
* tweak utils.Merge docs
* move merge_test to utils_test package for easier testing
* utils: support MergeConfig and StructFieldFilter
* constrain updating certain fields by the restricted system admin
* MM-14442: Merge function, to enable merging of configs
Merge will return a new struct of the same type as base and
patch, with patch merged into base. Specifically, patch's values will be preferred
except when patch's value is `nil`.
Restrictions/guarantees:
- base or patch will not be modified
- base and patch can be pointers or values
- base and patch must be the same type
- base and patch must have no unexported types (at the moment)
- if maps or slices are different, the entire map or slice will be
replaced (at the moment)
* License header
* major rewrite addressing PR comments from Jesse
* MM-14442: merge function for config files
- simplified merge for slices and maps
- fixed many problems with nested pointers/maps/slices
- all references are cloned
- 54 new tests, simplified tests for specific problems
* MM-14442: fixing formatting, comments
* Adds elasticsearch to the user and channel autocompletion functions
* Implement channel store GetChannelsByIds test
* Style changes and govet fixes
* Add gofmt fixes
* Extract default channel search limit to a const
* Add StringSliceDiff function to the utils package
* Honor USER_SEARCH_MAX_LIMIT on the user autocomplete api handler
* Change the elasticsearch development image
* api4: break out license and config from system
* app: move some config functions from admin.go to config.go
* add ExperimentalSettings.RestrictSystemAdmin
* forbid various actions to restricted system admin
* update default.json
* fix function names in errors
* TestGetLicenseFileFromDisk: avoid using fileutils.FindConfigFile
* config: abstract config-related file access, extend memory store
* simplify config validate to avoid file knowledge
* fix relative file tests
* cluster: fix ConfigChanged event
The old and new configurations were swapped when notifying the enterprise code of configuration changes, creating needless instability in propagating config updates across a cluster.
* config/database: ignore duplicates
* test cleanup
* remove unnecessary Save() in test
* MM-14145: The config store Set will now Save automatically
When UpdateConfig (and configStore.Set) is called in admin.go and
config.go, commonStore.Set now takes a store-specific persist function.
It uses that persist function to save the configuration automatically.
Removed: Now callers do not have to call configStore.Save or
app.PersistConfig, and those functions have been removed.
Possible downside: this means a "failed to persist config" error can now
be thrown during a app.UpdateConfig or commonStore.Set call. But
considering application code never really sets a config without saving
it (except in the test cases, which were testing that -- see below), it
seems fine to group these responsibilities.
Also removed: tests for 'set without save'. Since that can not happen
anymore, the tests are not needed.
* Removed Save completely, cleaned up formatting, joined save test with
set tests.
* fixed shadowed variable error
* [MM-13828] Running tests from a new temp folder with all test resources
Possible fix for #10132
All packages which have a TestMain and use testlib.MainHelper will have a new current working directory which will have all the test
resources copied.
Note: default.json is copied as config.json as well to make sure tests don't have any impact due to changes in config by devs
* [MM-13828] Added TestMain to remaining packages to use testlib.MainHelper
This makes sure tests from all packages run with same test resources, setup in a new temp folder for each package
* Updated Jenkins file to not not config/default.json
This makes sure CI has same config files as a dev's machine
* [MM-13828] Changes requested from code review
Added accessor methods to testlib.MainHelper for accessing members
Fixed some broken tests due to change in cwd while tests run
Some other code refactoring and improvements
* [MM-13828] Added new factory method with options for creating test main helper and some code refactoring
testlib.NewMainHelperWithOptions supports options to turn on/off test dependencies and environment setup
Some other code refactoring
* Exporting members of testlib.MainHelper to make enterprise tests work
* Fixed gofmt error
* [MM-13828] removed unwanted dependency on plugins directory while setting up test resources
* [MM-13828] Fixed some tests failing due to them being running from temp folder
* [MM-13828] Some code changes suggested in PR review
* Fixed gofmt error
* vendor github.com/jmoiron/sqlx
* MM-11262: introduce a database store
* revert unnecessary fmt.Errorf
* simplify unit test helper methods
* remote TODO re: retry
* relocate initializeConfigurationsTable for clarity
* factor out a commonStore
* acquire database config lock on close for safety
* add missing header
* fix lock comment
* config file store
Introduce an interface and concrete implementation for accessing the config.
This mostly maps 1:1 with the exiting usage in `App`, except for internalizing the watcher. A future change will likely eliminate `App.PersistConfig()` and make this implicit on `Set` or `Patch`
* experimental file test changes
* emoji: move file driver checks from api4 to app
It is no longer possible to app.UpdateConfig and provide an invalid configuration, making it hard to test this case. This check doesn't really belong in the api anyway, since it's a configuration validity check and not a permissions check. Either way, the check now occurs at the App level.
* api4: generate valid public link salts for test
* TestStartServerRateLimiterCriticalError: use mock store to test invalid config
* remove config_test.go
* remove needsSave, and have Load() save to the backing store as necessary
* restore README.md
* move ldap UserFilter check to model isValid checks
* remove databaseStore until ready
* remove unimplemented Patch
* simplify unlockOnce implementation
* revert forgetting to set s.Ldap
* config/file.go: rename ReadOnlyConfigurationError to ErrReadOnlyConfiguration
* config: export FileStore
* add TestFileStoreSave
* improved config/utils test coverage
* restore config/README.md copy
* tweaks
* file store: acquire a write lock on Save/Close to safely close watcher
* fix unmarshal_test.go
* refactor utils/config* to config/
* pull validateLdapFilter into app
* clean up Config/GetConfig/GetSanitizedConfig usage
Eliminate app.GetConfig() in favour of just using app.Config() directly,
but expose app.GetSanitizedConfig() for when the old behaviour was
required.
* web: isolate config setup
* TestInvitePeopleProvider: make config explicit
* regenerateClientConfig: avoid racey map access
* integrate watch flag into app.ConfigFile option
* make app.Option return an error
* release.mk: only cp static files from config/
* release.mk: fix cp static files from config/
* api4: TestPlugin cleanup
* s/c/cfg/ for clarity
* fix merge conflict
* testlib: allow customization of testlib driver name
* Adds configuration for separate reply-to email header.
* Changes config setting name.
* Using a separate variable and value in test.
* Updates for config pointer changes in another PR.
* Adds new key to test config. Adds default value.
* CSRF Token Implementation + Tests
Remove debug statements
Implement requested changes
* Fix non-cookie authentication methods stripping auth data from requests
* Fail when CSRF cookie is not returned as part of login
* MM-13207 Add customizable timeout for link metadata and improve caching of errors
* Rename LinkMetadataTimeout to LinkMetadataTimeoutMilliseconds
* Add diagnostics for LinkMetadataTimeoutMilliseconds
* MM 10658 Change config fields to pointers (#8898)
* Change fields of config structs to pointers and set defaults
MM-10658 https://github.com/mattermost/mattermost-server/issues/8841
* Fix tests that go broken during switching config structs to pointers
MM-10658 https://github.com/mattermost/mattermost-server/issues/8841
* Apply changes of current master while switching config structs to pointers
MM-10658 https://github.com/mattermost/mattermost-server/issues/8841
* Fix new config pointer uses
* Fix app tests
* Fix mail test
* remove debugging statement
* fix TestUpdateConfig
* assign config consistently
* initialize AmazonS3Region in TestS3TestConnection
* initialize fields for TestEmailTest
* fix TestCheckMandatoryS3Fields
* MM-10417 Add local image proxy and enable by default
* Remove unused function
* Add dependencies for willnorris/imageproxy
* Fixed compilation errors
* Lock to the master version of willnorris/imageproxy
* Fix atmos/camo proxy when no SiteURL is specified
* Re-add default values for deprecated settings
* Fix unit tests added by merge
* Pass imageproxy to App struct
* Remove unneeded locking when creating the image proxy
* Remove empty test file
* Initial models, API, app, and persistence of groups and group syncing.
* Consistent letter casing in ldif.
* Moves group-specific migrations into func.
* Adds API endpoint to retrieve LDAP groups (and associated MM groups) one tree level at a time.
* Adds mattermost group id to SCIMGroup (if available).
* Splits user and group creation so that memberOf works. Returns users from ldap interface.
* Updates method name.
* Returns users IDs instead of User.
* Removes non-essential group data.
* MM-11807: Add GroupFilter to LDAP config. (#9513)
* MM-11807: Add GroupFilter to LDAP config.
* Add diagnostic.
* Adds new config option for using 'memberOf' overlay.
* Adds API endpoint to link a group.
* Removes debug statements.
* Adds unlink group API endpoint.
* Fix to LDAP API. Adds API method to client4 and app.
* Adds some missing app methods. Renames API unexported func.
* Fixes link/unlink API path to accept valid DNs.
* Allow any character for DN portion of path.
* Switches from DN to objectGUID or entryUUID as the remote identifier linking LDAP groups to MM groups.
* Formatting.
* Formatting.
* Setting group name field to an ID for phase 1.
* Adds an LDAP config field to Setting up configuration for local LDAP.
* Changes to LDAP and GroupStore interfaces.
* Draft of nesting groups in API response.
* Removes unnecessary tree models.
* Updates group membershipt create store method to also restore.
* Adds new config to test config.
* Accept AD format length.
* Switches to SetUniqueTogether method.
* Updates revert.
* Tweaks to syncing queries .
* Updates query for pending team and channel memberships.
* Removes old GroupSyncableScanner usage. Some formatting and renaming.
* Fixes bug setting syncable type in selecting paged.
* Adds tests for syncables populator.
* Only add users to teams and channels that are not deleted.
* Renames method.
* Updates test LDAP setup.
* Removes memberof config stuff.
* Renames.
* Updates test data.
* Fix for gofmt.
* Adds missing license.
* Adds missing teardowns.
* Test fix.
* Adds a cycle to the groups test data.
* Changes API to return flat list.
* Removes some unused interface and app methods.
* Returns empty braces if results are empty.
* Adds more LDAP test data.
* Fix for test data error.
* Adds error.
* Moves test groups.
* Adds OU for load test data.
* Moves load test ou creation to load data.
* Adds a new bool flag to SCIMGroups.
* Removes SCIMGroup completely.
* Removes FULL JOIN because it is not supported in MySQL.
* Adds tests for sync queries; renames constant.
* Bad merge fix.
* Vet fix.
* Returning OK on delete ldap group link
* Removes foreign key constraints.
* Adding total to the ldap getAllGroups api endpoint
* Adds get group members page.
* Removes pagination from groups syncables list API.
* Adding syncable check now that foreign key constraint is removes.
* Joins teams and channels to group syncables.
* Adds group member count.
* Adding GetAllChannels and SearchAllChannels for system admins only
* Fix.
* Test fix from pagination removal.
* Orders groupmembers by createat.
* Fixing search of all channels
* Test fix after removing pagination.
* JSON syntax error fix.
* Changing tests (for now) pending investigation.
* Adding GetAllChannels and SearchAllChannels tests for the store
* Adding GetAllChannels and SearchAllChannels API tests
* Omit empty JSON values of group syncables.
* Fixing GetAllChannels and SearchAllChannels tests
* Fixing GetAllChannels and SearchAllChannels store tests
* Fixing GetAllChannels api tests
* Adds 'LDAP groups' feature flag. (#9861)
* Migrate new client functions to idiomatic error handling
* Test fixes.
* Simplification of groups api (#9860)
* Simplification of groups api
* Fixing RequireSyncableType
* Test fix.
* Update api4/group.go
Co-Authored-By: mkraft <martinkraft@gmail.com>
* Update api4/group.go
Co-Authored-By: mkraft <martinkraft@gmail.com>
* Update api4/group.go
Co-Authored-By: mkraft <martinkraft@gmail.com>
* Update api4/group.go
Co-Authored-By: mkraft <martinkraft@gmail.com>
* Update api4/group.go
Co-Authored-By: mkraft <martinkraft@gmail.com>
* Update api4/group.go
Co-Authored-By: mkraft <martinkraft@gmail.com>
* Update api4/group.go
Co-Authored-By: mkraft <martinkraft@gmail.com>
* Update api4/group.go
Co-Authored-By: mkraft <martinkraft@gmail.com>
* Update api4/group.go
Co-Authored-By: mkraft <martinkraft@gmail.com>
* Update api4/group.go
Co-Authored-By: mkraft <martinkraft@gmail.com>
* Fix copy/paste error.
* Fix copy/paste error.
* Adds missing return, changes to correct HTTP status code.
* Adds missing return, changes status codes.
* Check for license.
* Renames variable for new signature.
* Adds client method to get a group.
* Adds client method and tests for PatchGroup.
* Adds more API tests.
* Adds groups API tests.
* Adds client method and tests for getting group syncables.
* Adds tests for patching group teams and channels.
* Update to translations.
* Removes test.
* Fix incorrect conditional.
* Removes unnecessary nil check.
* Removes unnecessary return.
* Updates comment, removes unused variable.
* Uses consistent JSON unmarshal pattern.
* Uses consistent JSON unmarshal pattern.
* Moves const block.
* Switches 'already linked' from error to success response.
* Removes commented-out code.
* Switched to status ok.
* Add parens for readability.
* Fix copy/paste error.
* Unexport some structs.
* Removes repeated validity check.
* Return without attempting commit if there's a rollback.
* Fix incorrect HTTP status code.
* Update store/sqlstore/group_supplier.go
Co-Authored-By: mkraft <martinkraft@gmail.com>
* Adds utility methods for going from groupsyncable to groupteam and groupchannel.
* Fixing george suggestions (#9911)
* Test fix.
* Adds QA data to VC with visualization.
* Fixes typo in graph image.
* Update display name when re-linking in case it has changed in LDAP.
* Adds ability to configure group display name and unique identifier. (#9923)
* Adds ability to configure group display name and unique identifier.
* Adds some configs to confi-ldap make command.
* Fix for move of session.
* Exposes method for use by SAML package.
* Switches GroupSyncableType from int to string.
* Update Jenkins build files.
* Removes unused variable assignment.
* Removes old unnecessary early return.
* Removes unnecessary variable.
* Moves param parsing before license and permissions checks.
* Removes old code.
* Compares agains underlying error rather than error id.
* Switches tests to assertions.
* Adds more assertions.
* Adds missing return.
* Adds space after comma for added legibility.
* Moves a view model to the api package.
* Unexports method.
* Uses id validator function.
* Fix docker-compose flag.
* Typo fix.
* Moves index creation to supplier.
* Removes bad merge.
* Renames parameter.
* Re-adds space.
* Removes unnecessary transaction.
* Escapes the Groups table name with backticks because it is a reserved keyword.
* Fix roles cache bug
* Removing unnecesiary deserializing function
* Switches table name rather than custom SQL everywhere for Postgres without backticks.
* Removes redundant check for sql.ErrNoRows.
* Removes redundant check for sql.ErrNoRows.
* Removes data integrity check and redundant nil conditional.
* Removes redundant check for sql.ErrNoRows.
* Removes unnecessary query.
* Removes ID length validation from persistence tier.
* Makes some supplier methods idempotent.
* Removes some empty switch defaults.
* Renames Group Type field to Source.
* Fix for mistaken field name change.
* Uses IsValidId function.
* Removes comment.
* Changes json key name.
* Removes test because no longer validating user.
* Moves model state validation to app layer.
* Don't create Groups.CanLeave column until phase 2.
* Removes state validation until properties are used in phase 2.
* Removes duplicated check.
* Removes state validation until properties are used in phase 2.
* Removes some tests until phase 2.
* Comment-out a bunch of test related to CanLeave.
* Extra unmarshal validation check. Removes more code for CanLeave.
* Removes tests for CanLeave.
* Explict error msg.
* Rewrite queries.
* Changes index name. Adds index.
* Removes assertion.
* Adds experimental feature flag.
Config Checks at StartUp Part1
Config Checks; Tests for TLS Server
HSTS header implementation + tests
make gofmt happy with new go version...
make gofmt happy with new go version #2...
fix logic bug
fix typo
Fix unnecessary code block
* disable EnablePublicChannelsMaterialization by default
Creating triggers on Amazon RDS seems to reqiure extra privileges. More
investigation is required.
* update model/config.go too
* materialize PublicChannels table
Introduce triggers for each supported database that automatically maintain a subset of the Channels table corresponding to only public channels. This improves corresponding queries that no longer need to filter out 99% DM channels.
This initial commit modifies the channel store directly for easier code reviewing, but the next wraps an experimental version around it to enable a kill switch in case there are unforeseen performance regressions.
This addresses [MM-11886](https://mattermost.atlassian.net/browse/MM-11886) and [MM-11945](https://mattermost.atlassian.net/browse/MM-11945).
* extract the experimental public channels materialization
Wrap the original channel store with an experimental version that
leverages the materialized public channels, but can be disabled to
fallback to the original implementation.
This addresses MM-11947.
* s/ExperimentalPublicChannelsMaterialization/EnablePublicChannelsMaterialization/
* simplify error handling
* move experimental config listener until after store is initialized
* MM-11065: Allow to search and get archived channels from the API
* Fixing more tests
* Add some unit tests
* Add includeDeleted parameter to session permissions check function
* More test fixing
* Adding archive channels list in channels search
* Add restriction for archived channel edition
* Reverting permissions checks modification
* Changed the query parameter to include_deleted
* Enable search archive channels as true by default
* Adding tests for verify search on deleted channels
* Allowing to override archive channels during the imports
* Fixed test
* Search in archive channels from the API must be explicitly requested
* Removing includeDeleted parameter from GetChannelByName and GetChannelByNameForTeam
* Back to ViewArchivedChannels config
* Fixing tests
* Reverting GetChannelByName parameter
* Add include deleted parameter on GetChannel functions in plugins api
* Fixing tests
* Add ExperimentalSidebarPreference in the config
* Override Sidebar preference based on GroupUnread settings
* Rename to ExperimentalChannelOrganization
* Add Sidebar config in diagnostics
* Fix diagnostics for experimental_channel_organization
* Add ucLive support
crazy testing
lovely logs
more cookie work arounds
Added Access-Control-Expose-Headers to user login
Add complete_saml_body template and revert loginWithSaml endpoint
Set Access-Control-Allow-Credentials to true in user login
Login via email instead of username
Clean up code
Add comment to give some context
Move faml logic into saml function
Communicate via chrome sendMessage api
Remove unused code
Add config to support multiple extensions
Clean up embedded complete_saml template
Fix indentation for templates
Added license header to extension.go
Add EnableExperimentalExtensions flag
Extension validated for email auth
Clean up api auth code
Remove complete_saml_body.html
* Add extension support in saml
* Clean up code
* Clean up extension validation
* Adding setting to disable email invitations.
* Adding a setting and rate limiting for email invite sending.
* Modifying email rate limit to 20/user/hour
* Adding EnableEmailInvitations to client side config and command.
