* Use a custom user-agent when previewing links
Many websites block requests made with Go's default user-agent. We had
previous special-cased Twitter links to use a nonstandard user-agent.
This makes that behavior apply everywhere and also customizes the
user-agent to belong specifically to Mattermost.
* Correctly use custom transport for link previews
This allows us to use the custom user-agent defined in
services/httpservice/client.go.
* Stop leaking server version in custom user-agent
Since the custom user-agent is now used when previewing links, exposing
the server version could provide a vector for a malicious actor to
gather information about private deployments. To avoid this, we switch
to a generic string.
* Remove extraneous Transport creation
MakeClient already creates a transport for us, so this is unnecessary.
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* Add extract documents content command
* Adding the extraction command and making the pure go pdf library as secondary option
* Improving the memory usage and docextractor interface
* Enable content extraction by default in all the instances
* Tiny improvement on archive indexing
* Adding App interface generation and the opentracing layer
* Fixing linter errors
* Addressing PR review comments
* Addressing PR review comments
* MM-34002: Improve AddUserToChannel
When we would add a user to a channel, we would
check whether the user is removed from that team or not.
During LDAP sync, this check is not required because the
team member would have just been created. Hence, we
pass a boolean flag to bypass the check.
And with that done, we can freely query the replica.
https://mattermost.atlassian.net/browse/MM-34002
```release-note
NONE
```
* Refactor code
* Rename a struct field
* fix double negative
During LDAP sync, we would call AddTeamMember which had a read-after-write issue
where we would create a team member but then immediately after that
query the team member.
The same pattern was found in:
AddTeamMember
AddTeamMembers
AddTeamMemberByToken
To fix this, we just return the inserted team member from AddUserToTeam and use that
instead of query GetTeamMember again.
```release-note
NONE
```
https://mattermost.atlassian.net/browse/MM-33913
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
Remote Cluster Service
- provides ability for multiple Mattermost cluster instances to create a trusted connection with each other and exchange messages
- trusted connections are managed via slash commands (for now)
- facilitates features requiring inter-cluster communication, such as Shared Channels
Shared Channels Service
- provides ability to shared channels between one or more Mattermost cluster instances (using trusted connection)
- sharing/unsharing of channels is managed via slash commands (for now)
* MM-34487: Add dead queue
Just a circular buffer to store dead messages for now.
Not controlling this via config flag because this does
not have any effect except taking some more memory
per connection
```release-note
NONE
```
https://mattermost.atlassian.net/browse/MM-34487
* Wrap with config
* fix test
* MM-34086: Fix flaky test StartServerTLSOverwriteCipher
This suffered from the same race condition as
https://github.com/mattermost/mattermost-server/pull/17215.
We apply the same approach of using a memstore instead
of a file based config store.
It is likely that the test was also failing because of
the same race. Although I don't obviously see how, because
the race happens with mlog.Info. The test will fail only
if the config wasn't able to set properly. So although
not strictly related, it's somehow related.
If it fails again, we can look into it again. But this
atleast fixes the race.
While here, we also apply some cleanup of the code.
https://mattermost.atlassian.net/browse/MM-34086
```release-note
NONE
```
* Remove race test
* MM-34389: Reliable websockets: First commit
This is the first commit which makes some basic changes
to get it ready for the actual implementation.
Changes include:
- A config field to conditionally enable it.
- Refactoring the WriteMessage along with setting the deadline
to a separate method.
The basic idea is that the client sends the connection_id
and sequence_number either during the handshake (via query params),
or during challenge_auth (via added parameters in the map).
If the conn_id is empty, then we create a new one and set it.
Otherwise, we get the queues from a connection manager (TBD)
and attach them to WebConn.
```release-note
NONE
```
https://mattermost.atlassian.net/browse/MM-34389
* Incorporate review comments
* Trigger CI
* removing telemetry
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* Add feature flag for apps
* Update default to false
* Add plugin version Feature Flag
* Fix typo
* Only force shutdown, and leave the enable status dependant on the user (defaulting to enable)
* Remove unneeded tracking of status
* Handle plugin init on startup for locally installed plugin
App contains server.
Server contains WebsocketRouter.
There is no need for WebsocketRouter to contain
server too. As is evident because the code never used it.
```release-note
NONE
```
* MM-34171: Fix racy test TestSentry
The NewServer call sets the global mlog Info variable.
But before that, if we call UpdateConfig with the functional options,
then the store.Set method will call mlog.Info before
it could be set.
To fix that, we prepare the updated config
and pass that directly to NewServer to avoid
having to call UpdateConfig.
https://mattermost.atlassian.net/browse/MM-34171
```release-note
NONE
```
* disable watcher
* Trying yet again
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* MM-33893: Disable TCP_NO_DELAY for websocket connections
In very large installations, websocket messages cause too much
traffic congestion by sending too small packets and thereby cause
a drop in throughput.
To counter this, we disable the TCP_NO_DELAY flag for websocket
connections. This has shown to give noticeable improvements in
load tests.
We wrap this in a feature flag for now to let it soak in Community
first.
```release-note
NONE
```
https://mattermost.atlassian.net/browse/MM-33893
* fix gorilla specific conn
* MM-34000: Use non-epoll mode for TLS connections
A *crypto/tls.Conn does not expose the underlying TCP connection
or even a File method to get the underlying file descriptor
like the way a *net/TCPConn does. Therefore the netpoll code would
fail to get the file descriptor.
Relevant issue here: https://github.com/mailru/easygo/issues/3
It is indeed possible to use reflect black magic to get the unexported
member, but I have found unexpected errors during writing to the websocket
by getting the file descriptor this way. I do not want to spend time investigating
this especially since this is already released.
Once this is out, we can decide on the right way to fix this, most probably
by proposing to expose the File method or some other way.
https://mattermost.atlassian.net/browse/MM-34000
```release-note
Fix an issue where websockets wouldn't work with TLS connections.
In that case, we just fall back to the way it works for Windows machines,
which is to use a separate goroutine for reader connection.
```
* Ignore logging errors on non-epoll
On non-epoll systems, we needed to return an error
to break from the loop. But in that case, there is no
need to log the error
* Extracting html templates into a library
* Moving tests to the right place
* Fixing tests
* Addressing PR review comments
* Addressing PR review comments
* Replacing attomic with RWMutex
* Returning errors as channel for Templates watcher
* Address PR review comments
* Other small fixes
* Simplifying NewWithWatcher
* Addressing PR review comments
* Making error handling on rendering templates more robust
* Fixing tests
* Changing how we return errors
* Fixing shadow variables
* Addressing PR review comments
* Logging errors from the outside of sendNotificationEmail
* Fixing lock in shutdown
* Fixing the resource copy for commands tests temporary directories
* Removing unused import
* A couple of tiny fixes
* MM-31094: Adds tooling to develop and test using a MySQL instance with replication lag. Adds some lazy lookups to fallback to master if results are not found.
* MM-31094: Removes mysql-read-replica from default docker services.
* MM-31094: Switches (store..SessionStore).Get and (store.TeamStore).GetMember to using context.Context.
* MM-31094: Updates (store.UsersStore).Get to use context.
* MM-31094: Updates (store.PostStore).Get to use context.
* MM-31094: Removes feature flag and config setting.
* MM-31094: Rolls back some master reads.
* MM-31094: Rolls a non-cache read.
* MM-31094: Removes feature flag from the store.
* MM-31094: Removes unused constant and struct field.
* MM-31094: Removes some old feature flag references.
* MM-31094: Fixes some tests.
* MM-31094: App layers fix.
* MM-31094: Fixes mocks.
* MM-31094: Don't reparse flag.
* MM-31094: No reparse.
* MM-31094: Removed unused FeatureFlags field.
* MM-31094: Removes unnecessary feature flags variable declarations.
* MM-31094: Fixes copy-paste error.
* MM-31094: Fixes logical error.
* MM-30194: Removes test method from store.
* Revert "MM-30194: Removes test method from store."
This reverts commit d5a6e8529bd5f4d993824c828e239d009b05e567.
* MM-31094: Conforming to make's strange syntax.
* MM-31094: Configures helper for read replica with option.
* MM-31094: Adds some missing ctx's.
* MM-31094: WIP
* MM-31094: Updates test names.
* MM-31094: WIP
* MM-31094: Removes unnecessary master reads.
* MM-31094: ID case changes out of scope.
* MM-31094: Removes unused context.
* MM-31094: Switches to a helper. Removes some var naming changes. Fixes a merge error.
* MM-31094: Removes SQLITE db driver ref.
* MM-31094: Layer generate fix.
* MM-31094: Removes unnecessary changes.
* MM-31094: Moves test method.
* MM-31094: Re-add previous fix.
* MM-31094: Removes make command for dev.
* MM-31094: Fix for login.
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* app/server: add pprof endpoint
* reflect review comments
* make metrics link conditional
* app/server: add metrics server setup to licence listeners
* refactor a bit
* trigger CI
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* Add field to config model
Config option for disabling link previews for given domains.
* Refactor functions and corresponding tests
* Expand logic for link preview
Newly added isLinkAllowedForPreview function determines whether a link
should display a preview. It gets corresponding config values consisting
of comma separated domain values, normalizes them and checks for
matches.
* Create tests for link preview restriction
* Fix formatting issue
* Add test cases where images are expected
* Apply suggestions from code review
Co-authored-by: Ibrahim Serdar Acikgoz <serdaracikgoz86@gmail.com>
* Apply remaining code suggestions
* Add RestrictLinkPreviews value to telemetries
Co-authored-by: Ibrahim Serdar Acikgoz <serdaracikgoz86@gmail.com>
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
Co-authored-by: Harrison Healey <harrisonmhealey@gmail.com>
