* Enable receiving binary websocket messages
* Improve error message
* Prefer anonymous declaration
* Simplify
* Improve test
* Use MessagePack to clone WebSocketRequest struct
* Use short form
* Fix test
* MM-39612: Make acquiring and removing connections atomic
The reconnect phase of a websocket was split into two parts:
one where we check if a connection with a given connectionID
exists or not. And second, where we remove that connection
and insert the new connection again in the index.
This would lead to a race where it would be possible
for 2 concurrent requests for the same connectionID to go through
which would lead to separate goroutines working on the same dead queue.
We simplify this by removing the connection from the index
in the check connection stage itself. And then just add that
during register phase.
And to distinguish between a fresh and an old connection, we add
a new field called reuseCount.
While here, we also cleanup some old comments and add more
in some places.
https://mattermost.atlassian.net/browse/MM-39612
```release-note
NONE
```
* remove unused method
```release-note
NONE
```
* race test
```release-note
NONE
```
* Return an error seperately from Response
* Remove BuildErrorResponse
* Drop Response.Error from model/client4.go
* Migrate require.Nil checks
* Migrate require.NotNil checks
* More manual fixes
* Move error check out of CheckOKStatus and CheckCreatedStatus
* Move error check out of CheckForbiddenStatus
* Move error check out of CheckUnauthorizedStatus
* Move error check out of CheckNotFoundStatus
* Move error check out of CheckBadRequestStatus
* Move error check out of CheckNotImplementedStatus and CheckRequestEntityTooLargeStatus
* Move error check out of CheckInternalErrorStatus
* Move error check out of CheckServiceUnavailableStatus
* Remove error check from checkHTTPStatus
* Remove remaining references to Response.Error
* Check previously unchecked errors
* Manually fix compile and linter errors
* Return error in CreateWebSocket methods
* Return error instead of *AppError in DoApi methods
* Manually fix bad replacments
* Conistently return Response and error
* Use err instead of seperate bool return value to indicate success
* Reduce ussage of model.AppError in web/oauth_test.go
* Remove client4.Must
* Check error in buf.ReadFrom
* Fix failing tests
* MM-24547: Fix writer leak when connection closes
When the connection is closed, the exit path does not
shut down the writer goroutine. In which case, it will keep spinning forever.
Since we already have the CAS mechanism now, we can move the closing
functionality into the main Close method and just call that in the defer block.
This makes closing the websocket client idempotent from both perspective -
- Explicitly closing.
- Closing due to connection tear down.
There are still 2 races left:
- Using the exported Conn to directly write messages. We cannot do anything about
that as long as clients directly using that.
- Setting the wsc.pingTimeoutTimer field in a separate goroutine when calling
.Connect(). This will need to be seen later.
* Fix ineffectual assignment
* Duplicate the closing of writer
The problem with refactoring the writer closing to a common
function was that we needed to wait for the reader to exit
before closing the EventChannel and ResponseChannel.
But then there is another problem that the API can be used in such
a way that the client is liable to call Close without even calling
Listen. In that case, we cannot wait for Listen to quit.
So from Close, we can only close the connection. And therefore
we need to duplicate the writer closing in the read loop's
defer block.
* Cleanup some comments
* MM-24397: Reusing the read buffer while reading messages from websockets
The core problem was that conn.ReadMessage allocated a buffer every time it was read.
This created heavy amount of allocations every single time we read a message from the websocket.
To avoid this, we bypass the ReadMessage which was more of a helper method,
and actually call the NextReader which returns a reader object.
We can then reuse a single byte.Buffer instance to read the object unmarshal
into a WebSocketEvent object. This gets rid of the allocation in the read path completly
and allows GC more time to do other tasks.
* Incorporate review comments
* Move reset buffer to top of loop
* Cleanup further
* Fix test
* Final fix
* MM-23503: Fix race in websocket_client writer
We create a separate writer goroutine where all the writes happen.
* Fixing the case of double close
* Incorporate review comments
* Use CAS
* Fix incorrect comment
* Check if client is closed in pingHandler too
Co-authored-by: mattermod <mattermod@users.noreply.github.com>
* Make WebSocketEvent type immutable
* Update code to use updated immutable WebSocketEvent type
* Export WebSocketEvent fields and mark them as deprecated
* Consistent license message for all the go files
* Fixing the last set of unconsistencies with the license headers
* Addressing PR review comments
* Fixing busy.go and busy_test.go license header
* Remove api package
* Remove api dependency from cmd package
* Remove EnableAPIv3 setting
* Update web tests
* Add more websocket tests
* Move some ws and oauth tests to api4 package
* Move command tests into api4 package
* Test fixes
* Fix msg command test
* Add some app file tests
* Add app package and move logic over from api package
* Change app package functions to return errors
* Move non-api tests into app package
* Fix merge
* Switch websocket over to post-connect authentication
* Add ability to specify token in websocket js driver, add unit tests
* Temporarily disable client websocket tests until issues are resolved
* Minor refactoring and fix status test
* Add isAuthenticated method to WebConn and minor status updates
* improve performance on sendNotifications
* Fix SQL queries
* Remove get direct profiles, not needed anymore
* Add raw data to error details if AppError fails to decode
* men
* Fix decode (#4052)
* Fixing json decode
* Adding unit test
* Initial work for client scaling (#4051)
* Begin adding paging to profiles API
* Added more paging functionality
* Finish hooking up admin console user lists
* Add API for searching users and add searching to all user lists
* Add lazy loading of profiles
* Revert config.json
* Fix unit tests and some style issues
* Add GetProfilesFromList to Go driver and fix web unit test
* Update etag for GetProfiles
* Updating ui for filters and pagination (#4044)
* Updating UI for pagination
* Adjusting margins for filter row
* Adjusting margin for specific modals
* Adding relative padding to system console
* Adjusting responsive view
* Update client user tests
* Minor fixes for direct messages modal (#4056)
* Remove some unneeded initial load calls (#4057)
* UX updates to user lists, added smart counts and bug fixes (#4059)
* Improved getExplicitMentions and unit tests (#4064)
* Refactor getting posts to lazy load profiles correctly (#4062)
* Comment out SetActiveChannel test (#4066)
* Profiler cpu, block, and memory profiler. (#4081)
* Fix TestSetActiveChannel unit test (#4071)
* Fixing build failure caused by dependancies updating (#4076)
* Adding profiler
* Fix admin_team_member_dropdown eslint errors
* Bumping session cache size (#4077)
* Bumping session cache size
* Bumping status cache
* Refactor how the client handles channel members to be large team friendly (#4106)
* Refactor how the client handles channel members to be large team friendly
* Change Id to ChannelId in ChannelStats model
* Updated getChannelMember and getProfilesByIds routes to match proposal
* Performance improvements (#4100)
* Performance improvements
* Fixing re-connect issue
* Fixing error message
* Some other minor perf tweaks
* Some other minor perf tweaks
* Fixing config file
* Fixing buffer size
* Fixing web socket send message
* adding some error logging
* fix getMe to be user required
* Fix websocket event for new user
* Fixing shutting down
* Reverting web socket changes
* Fixing logging lvl
* Adding caching to GetMember
* Adding some logging
* Fixing caching
* Fixing caching invalidate
* Fixing direct message caching
* Fixing caching
* Fixing caching
* Remove GetDirectProfiles from initial load
* Adding logging and fixing websocket client
* Adding back caching from bad merge.
* Explicitly close go driver requests (#4162)
* Refactored how the client handles team members to be more large team friendly (#4159)
* Refactor getProfilesForDirectMessageList API into getAllProfiles API
* Refactored how the client handles team members to be more large team friendly
* Fix js error when receiving a notification
* Fix JS error caused by current user being overwritten with sanitized version (#4165)
* Adding error message to status failure (#4167)
* Fix a few bugs caused by client scaling refactoring (#4170)
* When there is no read replica, don't open a second set of connections to the master database (#4173)
* Adding connection tacking to stats (#4174)
* Reduce DB writes for statuses and other status related changes (#4175)
* Fix bug preventing opening of DM channels from more modal (#4181)
* Fixing socket timing error (#4183)
* Fixing ping/pong handler
* Fixing socket timing error
* Commenting out status broadcasting
* Removing user status changes
* Removing user status changes
* Removing user status changes
* Removing user status changes
* Adding DoPreComputeJson()
* Performance improvements (#4194)
* * Fix System Console Analytics queries
* Add db.SetConnMaxLifetime to 15 minutes
* Add "net/http/pprof" for profiling
* Add FreeOSMemory() to manually release memory on reload config
* Add flag to enable http profiler
* Fix memory leak (#4197)
* Fix memory leak
* removed unneeded nil assignment
* Fixing go routine leak (#4208)
* Merge fixes
* Merge fix
* Refactored statuses to be queried by the client rather than broadcast by the server (#4212)
* Refactored server code to reduce status broadcasts and to allow getting statuses by IDs
* Refactor client code to periodically fetch statuses
* Add store unit test for getting statuses by ids
* Fix status unit test
* Add getStatusesByIds REST API and move the client over to use that instead of the WebSocket
* Adding multiple threads to websocket hub (#4230)
* Adding multiple threads to websocket hub
* Fixing unit tests
* Fixing so websocket connections from the same user end up in the same… (#4240)
* Fixing so websocket connections from the same user end up in the same list
* Removing old comment
* Refactor user autocomplete to query the server (#4239)
* Add API for autocompleting users
* Converted at mention autocomplete to query server
* Converted user search autocomplete to query server
* Switch autocomplete API naming to use term instead of username
* Split autocomplete API into two, one for channels and for teams
* Fix copy/paste error
* Some final client scaling fixes (#4246)
* Add lazy loading of profiles to integration pages
* Add lazy loading of profiles to emoji page
* Fix JS error when receiving post in select team menu and also clean up channel store
* Refactored statuses into a more real-time system
* Updated package.json with correct commit and fixed minor bug
* Minor updates to statuses based on feedback
* When setting status online, update only LastActivityAt if status already exists
