* init commit
* clean up the code
* make mocks
* fix translations
* mocks and lint fixes
* add tests
* little fixes
* Update i18n/en.json
Co-authored-by: Scott Bishel <scott.bishel@mattermost.com>
* Update i18n/en.json
Co-authored-by: Scott Bishel <scott.bishel@mattermost.com>
* Update i18n/en.json
Co-authored-by: Scott Bishel <scott.bishel@mattermost.com>
* Update i18n/en.json
Co-authored-by: Scott Bishel <scott.bishel@mattermost.com>
* Update i18n/en.json
Co-authored-by: Scott Bishel <scott.bishel@mattermost.com>
* Address Comments
* fix i18n
* update api endpoint
* add enable file and file level for conditional show of banner
* Address Comments
* Make it more clear about returns
* Create zip file utility function
* update en.json
* address comments
* write tests
* check for data in test
* remove warning string
* Correct expected and actual
* set database through environment variables
* reset environment variable at end of test
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
Co-authored-by: Scott Bishel <scott.bishel@mattermost.com>
* docker: prometheus and grafana for development
Add prometheus and grafana docker configuration when developing with access to the enterprise repository. This simplifies the setup for developers adding new metrics or even examining events from their development server. The services are disabled by default when no enterprise source is detected.
Grafana is provisioned automatically with the Prometheus datasource, as well as the canonical dashboards used with Mattermost. Furthermore, no authentication is required to access Grafana to simplify access from a development environment. The default home dashboard is customized to automatically show links to the provisioned dashboards. Dashboards can be saved, and login remains available via the default `admin/admin` credentials (which initiates a password reset), but no dashboard or system configuration is (currently) persisted if the container is destroyed.
Linux requires slightly special handling (well, really it's Docker for MacOS/Windows), in that `host.docker.internal` is the canonical way to resolve the host on MacOS/Windows, but on Linux it's usually sufficient to just use `localhost`. Until https://github.com/docker/for-linux/issues/264 is resolved, this PR includes code to customize the Prometheus configuration to point at the required address for the running platform.
* metrics: track active jobs by type
* metrics: active jobs chart
* metrics: server start and job annotations
* Update build/docker-compose.common.yml
Co-authored-by: Claudio Costa <cstcld91@gmail.com>
* prometheus: resolve docker host via 172.17.0.1 instead
* skip enabling prometheus and grafana by default
* handle JOB_STATUS_ERROR and JOB_STATUS_CANCELED end states
* handle nil srv.metrics
* lookup job to determine type for metrics
* mocked unit tests for jobs
* goimports lint fixes
* missing license
* add instance to server start tag
* filter annotations by selected instance
Co-authored-by: Claudio Costa <cstcld91@gmail.com>
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* format using `goimports -local github.com/mattermost/mattermost-server/v5 -w`
* added goimports lint check to .golangci.yml
* format using `goimports -local github.com/mattermost/mattermost-server/v5 -w` for a corner case
* make app-layers, *-mocks and store-layers for ci check
Co-authored-by: Mahmudul Haque <mahmudulhaque@protonmail.com>
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* MM-31370: Fix racy TestClusterShutdownRace
This race wasn't specific to this test, but somehow got triggered by it.
This was a critical race because the PluginsLock is widely used
throughout the codebase to gate access to the PluginsEnvironment.
However, a config listener can often run in its own goroutine leading to
a wide variety of races.
https://mattermost.atlassian.net/browse/MM-31370
```release-note
NONE
```
* incorporate review comments
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* MM-31063: Change constants to use CamelCase
* store package
* change allcaps to camel case (#16615)
* New tools.mod
Co-authored-by: Ibrahim Serdar Acikgoz <serdaracikgoz86@gmail.com>
* Added log to print golang version before running Server
* Minor fix
* Review Fixes
* Review Fixes - 2
* Update app/server.go
Co-authored-by: Claudio Costa <cstcld91@gmail.com>
Co-authored-by: Claudio Costa <cstcld91@gmail.com>
* MM-31356: Add a minimum required version check for Postgres
To keep conformance with our failing fast and obvious philosophy,
we add a check to prevent Mattermost server from starting
if the postgres version is below 10.0.
This gives customers a chance to upgrade their database before upgrading
their Mattermost version, than to run into weird compatibility issues
after they have finished the upgrade.
https://mattermost.atlassian.net/browse/MM-31356
```release-note
NONE
```
* fix lint errors
* Use a function to pretty-print version string
* rectify comment
* Remove usages of AppError on filesstore service
* Fixing a golint error
* Fixing shadowed variable
* Adding err.Error() to the NewAppError calls
* Fixing tests
* Adding missed translations
* Fix error handling and updating the translation that affects it
* Fixing two typos
* Removing supplier concept from the sql store
* Removing other metions to supplier
* Fixing gofmt
* Fixing gofmt
* Renaming NewSqlStore to New
* Fixing tests
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* MM-30988 - Fix racy test ServerSystemdNotification
The translateFunc is a global variable which was unguarded.
So we convert that into an atomic variable to prevent against races.
https://mattermost.atlassian.net/browse/MM-30988
```release-note
NONE
```
* fix tests
* move call upwards
* use race
* after translations init
* guard with a sync.Once
* missed import
* revert race
* update timeout
A cluster changed listener always runs in a separate goroutine.
This causes a race condition when a new member gets added in the cluster.
We fix it by accessing the value atomically.
I tried to write a test for this, but it wasn't triggering at all.
The best way is to just start the server with cluster enabled in race mode,
and verify that the race doesn't happen.
https://mattermost.atlassian.net/browse/MM-30319
```release-notes
NONE
```
MM-27744 disable Zap for unit tests.
Zap has no concept of shutdown or close. Zap is only shutdown when the app exits. Not a problem for console logging, but when creating a new Zap logger that outputs to files on every unit test, that leaves no easy way to clean up until process exit. Depending on what else is running this can exhaust all file handles and cause unit tests to fail.
Zap is now disabled unit tests and uses Logr instead, regardless of config settings. `make test-server` peak file handle usage dropped from ~5K to less than 100.
* Moving diagnostics into a service
* Fixing golint checks
* Fixing tests
* Renaming from diagnostics to telemetry
* Adding missing files
* Initializing telemetry earlier in the server startup
* Fixing tests
* Adding a log for the telemetryID initialization error
* Addressing PR review comments
* Fixing merge problem
* Removing some extra Diagnostics mentions
* Making tests pass
* Adding Upgrade to Enterprise version feature
* Addressing PR review comments, and adding some minor improvements
* Add tests file
* Addressing PR comments
* fix linter checks
* Storing and exposing the upgraded from TE info
* Fix showing errors on mac
* A more appropiate status code for not-supported upgrade
* Fixing tests
* Handling permissions errors
* More server logging around upgrade failures
* Apply text changes suggested from code review
Co-authored-by: Eric Sadur <57730300+esadur@users.noreply.github.com>
* Address PR review comments
* Only allow to restart the system after an upgrade
* Verify file signature before upgrade
* Adding limit to the downloaded file
* Simplifying the upgrade binary process with backup in memory
* Fixing backup/restore mechanism for the binary file
* Improve file permissions handling
* Askin the permissions for the right place (the parent directory)
* Fixing tests
* Addressing PR review comments
* Fix license headers
* Fixing retry layer
* Making it work on windows builds
* Adding license header
* Fixing 2 tests
* Fixing tests that need UpgradeFromTE System key mock
* Extracting i18n translation
* Apply suggestions from code review
Co-authored-by: Eric Sadur <57730300+esadur@users.noreply.github.com>
* Improving how the errors are written
* Fixing another error text
* Removing unneeded translation
* Fixing upgrade status strings
* Update i18n/en.json
Co-authored-by: Eric Sadur <57730300+esadur@users.noreply.github.com>
* Fixing tests
Co-authored-by: Eric Sadur <57730300+esadur@users.noreply.github.com>
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* To allow for ease of testing telemetry changes, we should make it so that the rudder key and dataplane URL can be customized through the config or environment.
* Instead of using a real config element that would be exposed to end users, we'll just use 'secret' environment variables to inject Rudder config data.
Co-authored-by: Alex Dovenmuehle <alex.dovenmuehle@mattermost.com>
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* MM-26575: Move app initialization inside Websocket router out of handler
The websocket router struct is shared amongst multiple handlers. Therefore,
there is a race condition while setting the app field and reading from it.
We move the initialization of the app field when the router struct is initialized.
And we also remove the initializations of some app fields which caused race
conditions by being written from multiple goroutines. They are already being
written once inside the AppInitializedOnce.Do method and it's redundant
to set them again to the same value.
* Add missing fields in server connector
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* MM-27116: Drain push notification channel before closing
Without this, pending push notifications will not be sent when shutting down
or restarting the server.
* Fix race
* Re-arrange server shutdown
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
Adds the advanced logging config for audit. Existing support for auditing to a single file remains for E0 and E10 licenses instances, and a new config item ExperimentalAuditSettings.AdvancedLoggingConfig is added that behaves like LogSettings.AdvancedLoggingConfig.
Supported destinations:
- file
- syslog (with out without TLS)
- raw TCP socket (with out without TLS)
ExperimentalAuditSettings.AdvancedLoggingConfig can contain a filespec to a config file, a database DSN, or JSON.
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
Co-authored-by: Claudio Costa <cstcld91@gmail.com>
Adds Advanced Logging to server. Advanced Logging is an optional logging capability that allows customers to send log records to any number of destinations.
Supported destinations:
- file
- syslog (with out without TLS)
- raw TCP socket (with out without TLS)
Allows developers to specify discrete log levels as well as the standard trace, debug, info, ... panic. Existing code and logging API usage is unchanged.
Log records are emitted asynchronously to reduce latency to the caller. Supports hot-reloading of logger config, including adding removing targets.
Advanced Logging is configured within config.json via "LogSettings.AdvancedLoggingConfig" which can contain a filespec to another config file, a database DSN, or JSON.
* Extracting email functions into a service
* Fixing two shadowing errors
* Address PR review comments
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* MM-24972: make GetHubForUserId lock-less and zero-alloc
The hubs aren't meant to be modified after the server starts up.
But the key problem was the SetHubs method which would zero out
the underlying hubs slice.
This wasn't ideally necessary because an hub would only be stopped
once during server shutdown. However, this was required by a test
which would shutdown the hub twice. And since the hub shutdown isn't
idempotent, zeroing the slice would just skip over shutting down the hub again.
To improve the overall situation, we apply several optimizations.
- We use the new hash/maphash package which exposes Go runtime's
internal hash algorithms to be used as a package. This is much faster
than hash/fnv.
- We move around the initialization of the hub to happen before
the metrics server starts. This allows us to initialize the hub
before any of the hub elements are being accessed.
- To make the test run successfully, we do not call th.TearDown.
This is fine for a test, because anyways the test process would eventually stop
and relinquish the resources to the OS.
This allows us to completely remove any mutexes and thereby
we can remove all the methods and any edge-case checks related to
index being out of bounds. As a result, the fast path becomes
very straightforward and zero-alloc.
name old time/op new time/op delta
GetHubForUserId-8 116ns ± 1% 38ns ± 7% -67.22% (p=0.000 n=10+10)
name old alloc/op new alloc/op delta
GetHubForUserId-8 36.0B ± 0% 0.0B -100.00% (p=0.000 n=10+10)
name old allocs/op new allocs/op delta
GetHubForUserId-8 2.00 ± 0% 0.00 -100.00% (p=0.000 n=10+10)
Manually tested with some load testing and running Hub tests in -race mode.
* remove mutex
* incorporate review comments
