* MM-32950: Reliable WebSockets: Basic single server
This PR adds reliable websocket support for a single server.
Below is a brief overview of the three states of a connection:
Normal:
- All messages are routed via web hub.
- Each web conn has a send queue to which it gets pushed.
- A message gets pulled from the queue, and before it
gets written to the wire, it is added to the dead queue.
Disconnect:
- Hub Unregister gets called, where the connection is just
marked as inactive. And new messages keep getting pushed
to the send queue.
If it gets full, the channel is closed and the conn gets removed
from conn index.
Reconnect:
- We query the hub for the connection ID, and get back the
queues.
- We construct a WebConn reusing the old queues, or a fresh one
depending on whether the connection ID was found or not.
- Now there is a tricky bit here which needs to be carefully processed.
On register, we would always send the hello message in the send queue.
But we cannot do that now because the send queue might already have messages.
Therefore, we don't send the hello message from web hub, if we reuse a connection.
Instead, we move that logic to the web conn write pump. We check if
the sequence number is in dead queue, and if it is, then we drain
the dead queue, and start consuming from the active queue.
No hello message is sent here.
But if the message does not exist in the dead queue, and the sequence number
is actually something that should have existed, then we set
a new connction id and clear the dead queue, and send a hello message.
The client, on receiving a new connection id will automatically
set its sequence number to 0, and make the sync API calls to manage
any lost data.
https://mattermost.atlassian.net/browse/MM-32590
```release-note
NONE
```
* gofmt
* Add EnableReliableWebSockets to the client config
* Refactoring isInDeadQueue
* Passing index to drainDeadQueue
* refactoring webconn
* fix pointer
* review comments
* simplify hasMsgLoss
* safety comment
* fix test
* Trigger CI
* Trigger CI
Co-authored-by: Devin Binnie <devin.binnie@mattermost.com>
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* MM-34487: Add dead queue
Just a circular buffer to store dead messages for now.
Not controlling this via config flag because this does
not have any effect except taking some more memory
per connection
```release-note
NONE
```
https://mattermost.atlassian.net/browse/MM-34487
* Wrap with config
* fix test
* MM-23800: remove goroutineID and stack printing
Each hub has a goroutineID which is calculated with a known hack.
The FAQ clearly explains why goroutines don't have an id:
https://golang.org/doc/faq#no_goroutine_id.
We only added that because sometimes the hub would be deadlocked and
having the goroutineID would be useful when getting the stack trace.
This is also problematic in stress tests because the hubs would
frequently get overloaded and the logs would unnecessarily have stack traces.
But that was in the past, and we have done extensive testing with
load tests and fuzz testing to smooth any rough edges remaining.
Including adding additional metrics for hub buffer size.
Monitoring the metrics is a better way to approach this problem.
Therefore, we remove these kludges from the code.
* Also remove deadlock checking code
There is no need for that anymore since
we are getting rid of the stack printing anyways.
Let's do a wholesale refactor and clean up the codebase.
* MM-23805: Refactor web_hub
This is a beginning of the refactoring of the websocket code.
To start off with, we unexport some methods and constants which did not
need to be exported. There are more remaining but some are out of scope for this PR.
The main chunk of refactor is to unexport the webconn send channel
which was the main cause of panics. Since we were directly sending
to the connection from various parts of the codebase, it would be possible
that the send channel would be closed and we could still send a message.
This would crash the server.
To fix this, we refactor the code to centralize all sending from the main
hub goroutine. This means we can leverage the connections map to check
if the connection exists or not, and only then send the message.
We also move the cluster calls to cluster.go.
* bring back cluster code inside hub
* Incorporate review comments
* Address review comments
* rename index
* MM-23807: Refactor web_conn
- Unexport some struct fields and constants which are not necessary
to be accessed from outside the package. This will help us moving
the entire websocket handling code to a separate package later.
- Change some empty string checks to check for empty string rather
than doing a len check which is more idiomatic. Both of them compile
to the same code. So it doesn't make a difference performance-wise.
- Remove redundant ToJson calls to get the length.
- Incorporate review comments
- Unexport some more methods
* Fix field name
* Run make app-layers
* Add note on hub check
* Make WebSocketEvent type immutable
* Update code to use updated immutable WebSocketEvent type
* Export WebSocketEvent fields and mark them as deprecated
* Consistent license message for all the go files
* Fixing the last set of unconsistencies with the license headers
* Addressing PR review comments
* Fixing busy.go and busy_test.go license header
* Rerouted the unit test logs through t.Log
* resolving merge confilvts
* Update testing.go
* Update helper_test.go
* Added godocs for NewTestingLogger
* Added go docs for NewTestingLogger
* Resolving conflicts
* api4: fix TestGetUsersNotInTeam assertions
This test was relying on data from a previous test run. With the data cleared before each test, the assertions much match reality.
* *testlib: always InitSystemAdmin
Some tests implicitly relied on the basic user having system
administrator privileges because it was the first user created as such.
Eliminate `InitSystemAdmin` and explicitly create the system admin user
instead to avoid this ambiguity going forward.
* *testlib: drop all tables before each test
* api4: split up TestChannelDelete to avoid duplicate InitBasic
* api4: teardown in TestResetPassword, for when this test comes back
* invalidate cache on DropAllTables
This is necessary since the test store persists across tests.
* disable parallel tests
While tests within a package must be explicitly parallelized using `t.Parallel()`, tests across packages are run in parallel by default. This causes problems given that the tests all currently share the same database instance.
Unfortunately, this also means that running the tests is much slower, but we can return to this later.
* Add user.DeepCopy() function
* Add omit admins/non-admins to WS broadcast and use for updating users
* Updates per feedback and adding unit test for ShouldSendEvent
