* MM-39612: Make acquiring and removing connections atomic
The reconnect phase of a websocket was split into two parts:
one where we check if a connection with a given connectionID
exists or not. And second, where we remove that connection
and insert the new connection again in the index.
This would lead to a race where it would be possible
for 2 concurrent requests for the same connectionID to go through
which would lead to separate goroutines working on the same dead queue.
We simplify this by removing the connection from the index
in the check connection stage itself. And then just add that
during register phase.
And to distinguish between a fresh and an old connection, we add
a new field called reuseCount.
While here, we also cleanup some old comments and add more
in some places.
https://mattermost.atlassian.net/browse/MM-39612
```release-note
NONE
```
* remove unused method
```release-note
NONE
```
* race test
```release-note
NONE
```
- Make it a Server method.
- Pass Server instead of App.
- Remove global app instance from NewServer, rather
create local instances whenever needed.
- Remove App from Websocket router, and create dynamically
on every request.
- Remove HubStart and HubStop from App methods.
- Explicitly using s.Log instead of the global logger
to indicate dependency on Server. We could have passed the logger
explicitly but it doesn't look ideal.
```release-note
NONE
```
* add request context
* move initialialization to server
* use app interface instead of global app functions
* remove app context from webconn
* cleanup
* remove duplicated services
* move context to separate package
* remove finalize init method and move content to NewServer function
* restart workers and schedulers after adding license for tests
* reflect review comments
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* MM-34389: Reliable websockets: First commit
This is the first commit which makes some basic changes
to get it ready for the actual implementation.
Changes include:
- A config field to conditionally enable it.
- Refactoring the WriteMessage along with setting the deadline
to a separate method.
The basic idea is that the client sends the connection_id
and sequence_number either during the handshake (via query params),
or during challenge_auth (via added parameters in the map).
If the conn_id is empty, then we create a new one and set it.
Otherwise, we get the queues from a connection manager (TBD)
and attach them to WebConn.
```release-note
NONE
```
https://mattermost.atlassian.net/browse/MM-34389
* Incorporate review comments
* Trigger CI
* removing telemetry
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
App contains server.
Server contains WebsocketRouter.
There is no need for WebsocketRouter to contain
server too. As is evident because the code never used it.
```release-note
NONE
```
* MM-26575: Move app initialization inside Websocket router out of handler
The websocket router struct is shared amongst multiple handlers. Therefore,
there is a race condition while setting the app field and reading from it.
We move the initialization of the app field when the router struct is initialized.
And we also remove the initializations of some app fields which caused race
conditions by being written from multiple goroutines. They are already being
written once inside the AppInitializedOnce.Do method and it's redundant
to set them again to the same value.
* Add missing fields in server connector
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* Removing some other fake apps
* More FakeApp removed
* Removing entirely FakeApp
* Fixing some tests
* Fixing get Cluster id from get plugin status
* Fixing failing tests
* Fixing tests
* Fixing test initialization for web
* Fixing InitServer for server tests
* Fixing InitServer for server tests
* Reverting go.sum and go.mod
* Removing unneded HTMLTemplates function in App layer
* Moving back some functions to its old place to easy the review
* Moving back some functions to its old place to easy the review
* Using the last struct2interface version
* Generating store layers
* Fixing merge problems
* Addressing PR comments
* Small fix
* Fixing app tests build
* Fixing tests
* fixing tests
* Fix tests
* Fixing tests
* Fixing tests
* Fixing tests
* Moving license to server struct
* Adding some fixes to the test compilation
* Fixing cluster and some jobs initialization
* Fixing some license tests compilation problems
* Fixing recursive cache invalidation
* Regenerating app layers
* Fix test compilation
Co-authored-by: mattermod <mattermod@users.noreply.github.com>
We would call SetStatusOnline in a goroutine before actually calling HubRegister.
This could cause the message not to be sent after all, because there's no guarantee
that Register would actually happen before it.
To fix it, we just change the order of things.
Co-authored-by: mattermod <mattermod@users.noreply.github.com>
* MM-23800: remove goroutineID and stack printing
Each hub has a goroutineID which is calculated with a known hack.
The FAQ clearly explains why goroutines don't have an id:
https://golang.org/doc/faq#no_goroutine_id.
We only added that because sometimes the hub would be deadlocked and
having the goroutineID would be useful when getting the stack trace.
This is also problematic in stress tests because the hubs would
frequently get overloaded and the logs would unnecessarily have stack traces.
But that was in the past, and we have done extensive testing with
load tests and fuzz testing to smooth any rough edges remaining.
Including adding additional metrics for hub buffer size.
Monitoring the metrics is a better way to approach this problem.
Therefore, we remove these kludges from the code.
* Also remove deadlock checking code
There is no need for that anymore since
we are getting rid of the stack printing anyways.
Let's do a wholesale refactor and clean up the codebase.
* MM-23805: Refactor web_hub
This is a beginning of the refactoring of the websocket code.
To start off with, we unexport some methods and constants which did not
need to be exported. There are more remaining but some are out of scope for this PR.
The main chunk of refactor is to unexport the webconn send channel
which was the main cause of panics. Since we were directly sending
to the connection from various parts of the codebase, it would be possible
that the send channel would be closed and we could still send a message.
This would crash the server.
To fix this, we refactor the code to centralize all sending from the main
hub goroutine. This means we can leverage the connections map to check
if the connection exists or not, and only then send the message.
We also move the cluster calls to cluster.go.
* bring back cluster code inside hub
* Incorporate review comments
* Address review comments
* rename index
* MM-23807: Refactor web_conn
- Unexport some struct fields and constants which are not necessary
to be accessed from outside the package. This will help us moving
the entire websocket handling code to a separate package later.
- Change some empty string checks to check for empty string rather
than doing a len check which is more idiomatic. Both of them compile
to the same code. So it doesn't make a difference performance-wise.
- Remove redundant ToJson calls to get the length.
- Incorporate review comments
- Unexport some more methods
* Fix field name
* Run make app-layers
* Add note on hub check
* Consistent license message for all the go files
* Fixing the last set of unconsistencies with the license headers
* Addressing PR review comments
* Fixing busy.go and busy_test.go license header
* Moving goroutine pool
* Auto refactor
* Moving plugins.
* Auto refactor
* Moving fields to server
* Auto refactor
* Removing siteurl duplication.
* Moving reset of app fields
* Auto refactor
* Formatting
* Moving niling of Server to after last use
* Fixing unit tests.
* Implementing structured logging
* Changes to en.json to allow refactor to run.
* Fixing global logger
* Structured logger initalization.
* Add caller.
* Do some log redirection.
* Auto refactor
* Cleaning up l4g reference and removing dependancy.
* Removing junk.
* Copyright headers.
* Fixing tests
* Revert "Changes to en.json to allow refactor to run."
This reverts commit fd8249e99bcad0231e6ea65cd77c32aae9a54026.
* Fixing some auto refactor strangeness and typo.
* Making keys more human readable.
* Add sequence number to websocket connections and events
* Copy pointer instead of pass by value and use int64 over uint64
* Add more logging to missed events
* Add app package and move logic over from api package
* Change app package functions to return errors
* Move non-api tests into app package
* Fix merge
