* [MM-59089] Add a compliance export constant (#27919)
* add a useful constant
* i18n
* another constant
* another i18n
* [MM-60422] Add GetChannelsWithActivityDuring (#28301)
* modify GetUsersInChannelDuring to accept a slice of channelIds
* add GetChannelsWithActivityDuring
* add compliance export progress message; remove unused custom status
* linting
* tests running too fast
* add batch size config settings
* add store tests
* linting
* empty commit
* i18n changes
* fix i18n ordering
* MM-60570 - Server-side changes consolidating the export CLI with server/ent code (#28640)
* add an i18n field; add the CLI's export directory
* int64 -> int
* Add UntilUpdateAt for MessageExport and AnalyticsPostCount
to merge
* remove now-unused i18n strings
* add TranslationsPreInitFromBuffer to allow CLI to use i18n
* use GetBuilder to simplify; rename TranslationsPreInitFromFileBytes
* [MM-59089] Improve compliance export timings (#1733 - Enterprise repo)
* MM-60422 - Performance and logic fixes for Compliance Exports (#1757 - Enterprise repo)
* MM-60570 - Enterprise-side changes consolidating the export CLI with server/ent code (#1769 - Enterprise repo)
* merge conflicts; missed file from ent branch
* MM-61038 - Add an option to sqlstore.New (#28702)
remove useless comment
add test
add an option to sqlstore.New
* MM-60976: Remove RunExport command from Mattermost binary (#28805)
* remove RunExport command from mattermost binary
* remove the code it was calling
* fix i18n
* remove test (was only testing license, not functionality)
* empty commit
* fix flaky GetChannelsWithActivityDuring test
* MM-60063: Dedicated Export Filestore fix, redo of #1772 (enterprise) (#28803)
* redo filestore fix#1772 (enterprise repo) on top of MM-59966 feature
* add new e2e tests for export filestore
* golint
* ok, note to self: shadowing bad, actually (when there's a defer)
* empty commit
* MM-61137 - Message export: Support 7.8.11 era dbs (#28824)
* support 7.8.11 era dbs by wrapping the store using only what we need
* fix flaky GetChannelsWithActivityDuring test
* add a comment
* only need to define the MEFileInfoStore (the one that'll be overridden)
* blank commit
* MM-60974 - Message Export: Add performance metrics (#28836)
* support 7.8.11 era dbs by wrapping the store using only what we need
* fix flaky GetChannelsWithActivityDuring test
* add a comment
* only need to define the MEFileInfoStore (the one that'll be overridden)
* performance metrics
* cleanup unneeded named returns
* blank commit
* MM-60975 - Message export: Add startTime and endTime to export folder name (#28840)
* support 7.8.11 era dbs by wrapping the store using only what we need
* fix flaky GetChannelsWithActivityDuring test
* add a comment
* only need to define the MEFileInfoStore (the one that'll be overridden)
* performance metrics
* output startTime and endTime in export folder
* empty commit
* merge conflict
* MM-60978 - Message export: Improve xml fields; fix delete semantics (#28873)
* support 7.8.11 era dbs by wrapping the store using only what we need
* fix flaky GetChannelsWithActivityDuring test
* add a comment
* only need to define the MEFileInfoStore (the one that'll be overridden)
* performance metrics
* output startTime and endTime in export folder
* empty commit
* add xml fields, omit when empty, tests
* fix delete semantics; test (and test for update semantics)
* clarify comments
* simplify edited post detection, now there's no edge case.
* add some spacing to help fast running tests
* merge conflicts/updates needed for new deleted post semantics
* linting; fixing tests from upstream merge
* use SafeDereference
* linting
* stronger typing; better wrapped errors; better formatting
* blank commit
* goimports formatting
* fix merge mistake
* minor fixes due to changes in master
* MM-61755 - Simplifying and Support reporting to the db from the CLI (#29281)
* finally clean up JobData struct and stringMap; prep for CLI using db
* and now simplify using StringMapToJobDataWithZeroValues
* remove unused fn
* create JobDataExported; clean up errors
* MM-60176 - Message Export: Global relay cleanup (#29168)
* move global relay logic into global_relay_export
* blank commit
* blank commit
* improve errors
* MM-60693 - Refactor CSV to use same codepath as Actiance (#29191)
* move global relay logic into global_relay_export
* blank commit
* refactor (and simplify) ExportParams into shared
* blank commit
* remove unused fn
* csv now uses pre-calculated joins/leaves like actiance
* improve errors
* remove nil post check; remove ignoredPosts metric
* remove unneeded copy
* MM-61696 - Refactor GlobalRelay to use same codepath as Actiance (#29225)
* move global relay logic into global_relay_export
* blank commit
* refactor (and simplify) ExportParams into shared
* blank commit
* remove unused fn
* csv now uses pre-calculated joins/leaves like actiance
* remove newly unneeded function and its test. goodbye.
* refactor GetPostAttachments for csv + global relay to share
* refactor global_relay_export and fix tests (no changes to output)
* improve errors
* remove nil post check; remove ignoredPosts metric
* remove unneeded copy
* remove unneeded nil check
* PR comments
* MM-61715 - Generalize e2e to all export types 🤖 (#29369)
* move global relay logic into global_relay_export
* blank commit
* refactor (and simplify) ExportParams into shared
* blank commit
* remove unused fn
* csv now uses pre-calculated joins/leaves like actiance
* remove newly unneeded function and its test. goodbye.
* refactor GetPostAttachments for csv + global relay to share
* refactor global_relay_export and fix tests (no changes to output)
* improve errors
* remove nil post check; remove ignoredPosts metric
* remove unneeded copy
* remove unneeded nil check
* PR comments
* refactor isDeletedMsg for all export types
* fix start and endtime, nasty csv createAt bug; bring closer to Actiance
* align unit tests with new logic (e.g. starttime / endtime)
* refactor a TimestampConvert fn for code + tests
* bug: pass templates to global relay (hurray for e2e tests, otherwise...)
* add global relay zip to allowed list (only for tests)
* test helpers
* new templates for e2e tests
* e2e tests... phew.
* linting
* merge conflicts
* unexport PostToRow; add test helper marker
* cleanup, shortening, thanks to PR comments
* MM-61972 - Generalize export data path - Actiance (#29399)
* extract and generalize the export data generation functions
* finish moving test (bc of previous extraction)
* lift a function from common -> shared (to break an import cycle)
* actiance now takes general export data, processes it into actiance data
* bring tests in line with correct sorting rules (upadateAt, messageId)
* fixups, PR comments
* turn strings.Repeat into a more descriptive const
amended: one letter fix; bad rebase
* MM-62009 - e2e clock heisenbug (#29434)
* consolidate assertions; output debuggable diffs (keeping for future)
* refactor test output generator to generators file
* waitUntilZeroPosts + pass through until to job = fix all clock issues
* simplify messages to model.NewId(); remove unneeded waitUntilZeroPosts
* model.NewId() -> storetest.NewTestID()
* MM-61980 - Generalize export data path - CSV (#29482)
* simple refactoring
* increase sleep times for (very) rare test failures
* add extra information to the generic export for CSV
* adj Actiance to handle new generic export (no difference in its output)
* no longer need mergePosts (yay), move getJoinLeavePosts for everyone
* adjust tests for new csv semantics (detailed in summary)
* and need to add the new exported data to the export_data_tests
* rearrange csv writing to happen after data export (more logical)
* linting
* remove debug statements
* figured out what was wrong with global relay e2e test 3; solid now
* PR comments
* MM-61718 - Generalize export data path - Global Relay (#29508)
* move global relay over to using the generalized export data
* performance pass -- not much can be done
* Update server/enterprise/message_export/global_relay_export/global_relay_export.go
Co-authored-by: Claudio Costa <cstcld91@gmail.com>
---------
Co-authored-by: Claudio Costa <cstcld91@gmail.com>
* MM-62058 - Align CSV with Actiance (#29551)
* refactoring actiance files and var names for clarity
* bug found in exported attachments (we used to miss some start/ends)
* changes needed for actiance due to new generic exports
* bringing CSV up to actiance standards
* fixing global relay b/c of new semantics (adding a note on an edge case)
* aligning e2e tests, adding comments to clarify what is expected/tested
* necessary changes; 1 more test for added functionality (ignoreDeleted)
* comment style
* MM-62059 - Align Global Relay with Actiance/CSV; many fixes (#29665)
* core logic changes to general export_data and the specific export paths
* unit tests and e2e tests, covering all new edge cases and all logic
* linting
* better var naming, const value, and cleaning up functions calls
* MM-62436 - Temporarily skip cypress tests that require download link (#29772)
---------
Co-authored-by: Claudio Costa <cstcld91@gmail.com>
The default role permissions weren't being saved
to a separate variable which would have not restored
them properly. This was a problem in various other
tests, potentially causing side-effects in other
flaky tests as well.
We fix that, as well as clean up the for loop
a bit and mention how many messages were recieved
in the failure message so that it becomes clear
whether no messages were recieved, or atleast some
were received.
https://mattermost.atlassian.net/browse/MM-62079
```release-note
NONE
```
* MM-62378: Initialize status and session cache before loading license
ps.LoadLicense could end up calling InvalidateAllCaches. Therefore,
we need to intialize those caches before calling LoadLicense
to prevent a panic from happening.
While here, we also remove some unused code.
https://mattermost.atlassian.net/browse/MM-62378
```release-note
NONE
```
* fix lint errors
```release-note
NONE
```
* fix test
```release-note
NONE
```
* revert the api4 portion of 57372bd06b
* test: Add system admin channel access tests
* skip a.HasPermissionToTeam for channels without a team
* fix TestPostGetInfo
* Add DeleteSidebarCategory method and channel category API tests
* Add test to verify channels move to default categories when custom category is deleted
* rename channel category tests and remove duplicate context
* TestNotifications fix
---------
Co-authored-by: Fume <contact@fumedev.com>
Co-authored-by: Ben Schumacher <ben.schumacher@mattermost.com>
Co-authored-by: Mattermost Build <build@mattermost.com>
This test failed because TestWebHubCloseConnOnDBFail failed
and therefore the ChannelMembers table failed to be
renamed back.
We can see that in the logs:
```
teams","request_id":"sdxjxsd1wibd9mb7kc3ixj6g1w","ip_addr":"127.0.0.1","user_id":"afw9qqjh7pnddqakraxnjf1oph","method":"POST","user_id":"afw9qqjh7pnddqakraxnjf1oph","team_id":"bocubwmn4jyrmc1p9uxrc4pwfc","error":"JoinDefaultChannels: Unable to save direct channel., channel_members_save: Error 1146 (42S02): Table 'dbrt3jzcosj3nb8g6mgwmqyss9rh.ChannelMembers' doesn't exist"}
{"timestamp":"2024-11-10 13:52:34.260 Z","level":"error","msg":"Error while invalidating channel member cache","caller":"platform/web_hub.go:524","user_id":"afw9qqjh7pnddqakraxnjf1oph","error":"failed to find ChannelMembers, TeamScheme and ChannelScheme data: Error 1146 (42S02): Table 'dbrt3jzcosj3nb8g6mgwmqyss9rh.ChannelMembers' doesn't exist"}
{"timestamp":"2024-11-10 13:52:34.260 Z","level":"error","msg":"Error while invalidating channel member cache","caller":"platform/web_hub.go:524","user_id":"afw9qqjh7pnddqakraxnjf1oph","error":"failed to find ChannelMembers, TeamScheme and ChannelScheme data: Error 1146 (42S02): Table 'dbrt3jzcosj3nb8g6mgwmqyss9rh.ChannelMembers' doesn't exist"}
```
There is nothing wrong with this test. And while
we are here, we slightly improve the for-select loop
to exit faster.
https://mattermost.atlassian.net/browse/MM-61699
```release-note
NONE
```
Finally I figured out why the log message for
/api/v4/websocket does not appear. It is because
the log gets generated only when the request returns,
and for websockets, the request doesn't return
until the client closes. And because we were closing
the client in a defer clause, the flushing of the logger
would happen before closing the client, therefore
leading to a race condition of the log not appearing
from time to time.
https://mattermost.atlassian.net/browse/MM-61698
```release-note
NONE
```
https://github.com/mattermost/mattermost/pull/29214 fixed the
race condition in web_hub initialization.
The failure in the test is strange because there's no log
for the websocket request. It's as if th.CreateWebSocketClientWithClient()
never got called at all.
I am re-arranging the lines so that even if it fails,
the table rename happens anyways. But overall, I don't see
any reason for the failure.
https://mattermost.atlassian.net/browse/MM-61698
```release-note
NONE
```
While investigating https://mattermost.atlassian.net/browse/MM-61529,
I noticed that we weren't adding the internal error which made
it impossible to understand what was really happening.
```release-note
NONE
```
I was planning to kick off a campaign for this. But aider can do this just fine.
Starting off with a single file for now. But will slowly send more PRs with larger
commit diffs.
```release-note
NONE
```
Co-authored-by: Mattermost Build <build@mattermost.com>
Drop the legacy `X` suffix from `GetMasterX` and `GetReplicaX`. The
presence of the suffix suggests there's a `non-X` version: but in fact
we migrated these away a long time ago, so remove the cognitive
overhead.
As an aside, this additionally helps avoid trip up LLMs that interpret
this as "something to fix".
We were incorrectly not broadcasting status cache updates
inspite of that being an LRU cache.
We were also not doing it for profilesInChannel cache.
Now we fix it by properly checking the invalidationEvent
which is something local to the cache itself rather
than the cache provider.
https://mattermost.atlassian.net/browse/MM-62077
```release-note
NONE
```
This introduces a new entry in the `Main Menu -> About` modal with the hostname of the currently connected websocket. This will be used to aid debugging issues in clustered environments by showing which node in the cluster is servicing requests for a particular websocket.
This information is only visible in self-managed instances. It will not be visible on cloud instances.
* [MM-61477] Fixed exif rotates images width and height is not correctly calculated
* [MM-61477] Apply rotation ony to png or jepg
* [MM-61477] Added test cases to verify correct rotation
* fix: fixed wrong format
* uff
* fix: exif rotation is only possible on jpeg images
---------
Co-authored-by: Mattermost Build <build@mattermost.com>
