* Add config settings for additional security features on mobile
* Add system console settings for mobile security
* Update svg and link
* Fix strings
* Add test for the discovery feature
* Fix tests
* Add permission migrations
* Add relevant e2e tests
* Fix key alignment
* fix tests
* Fix lint
* Mock new migration
* Fix playwright prettier
* Add new section to delegated permissions
* Update snapshots
* Fix flakyness in playwright test
---------
Co-authored-by: Elias Nahum <nahumhbl@gmail.com>
Co-authored-by: Mattermost Build <build@mattermost.com>
* feat: Add EnableCrossTeamSearch configuration option to ServiceSettings
* feat: Add EnableCrossTeamSearch configuration option to ServiceSettings
* feat: Enable cross-team search by default
* include old FF in client config
* test: Add unit tests for custom profile attributes select options
* feat: Add custom profile attributes model with validation and constants
* refactor: Trim spaces from name and color in custom profile attribute select option constructor
* gofmt
* refactor: Fix typo in custom profile attributes select option function name
* feat: Add IsValid method to validate CustomProfileAttributesSelectOptions
* refactor: Replace map[string]bool with map[string]struct{} for key existence check
* refactor: Rename NewCustomProfileAttributeSelectOption to NewCustomProfileAttributesSelectOption
* feat: Add validation to prevent empty custom profile attribute options
* refactor: Add validation and creation methods for custom profile attributes
* feat: Add index number to validation error messages in custom profile attributes
* fix tests
* add default visibility
* feat: Add comprehensive test cases for custom profile attributes field validation
* fix: Update custom profile attributes map keys to use capitalized names
* feat: Add support for lowercase and title case keys in custom profile attributes map
* test: Add comprehensive test for NewCustomProfileAttributesSelectOptionFromMap
* feat: Add validation for custom profile attributes fields
* refactor: Update CustomProfileAttributesSelectOption constructor to prioritize ID parameter
* test: Add test cases for preserving IDs in custom profile attributes
* feat: Enhance ID validation and trimming in custom profile attributes
* don't do validation in constructor
* test: Add test case for preserving option IDs when patching select field
* improve test
* i18n
* refactor: Modify CustomProfileAttributesSelectOption to use lowercase JSON keys
* fix casing in custom profilte attributes test
* refactor: Use consistent "ValidateCPAField" in error messages for custom profile attributes
* use custom types rather than string
* lint
* fix api test
* refactor: Make color field optional in custom profile attributes
* style
* generic options
* removed unused i18n
* test: Add tests for NewCPAFieldFromPropertyField and CPAFieldToPropertyField
* test: Add test case for property field with empty attributes
* refactor: Cleanup whitespace and remove empty Attrs in custom profile attributes test
* test: Add test case for CPA field with empty attributes
* refactor: Improve custom profile attributes field handling and validation
* refactor: Move validateCustomProfileAttributesField to Validate method on CPAField struct
* use CPAField
* code style
* add validation and tests
* tests
* i18n
* err->appErr
* fix TestDeleteCPAField test
* i18n
* Add SAML and LDAP attr
* rename CustomProfileAttributes in method to CPA
* rename CPASortOrder method
* rearrange consts
* use Len test method
* sanitize and validate
* manage error the same way property field and value do
* fix: Update test error ID for custom profile attributes validation
* test: Update error ID expectations in custom profile attributes tests
* refactor: Convert CPAAttrs.SortOrder from string to int
* json uses float64
* feat: Add length validation for custom profile attribute option name and color
---------
Co-authored-by: Mattermost Build <build@mattermost.com>
* Update Go version to v1.23.7
* Bump golangci-lint to a version supporting Go 1.23
* Fix golangci-lint warnings
Several rules from gosimple, revive and staticcheck linters were
failing:
- Redefinition of built-in identifiers (max, min, new, recover...)
- Use of printf-like functions with simple strings
- Check for nil slices, when len already takes it into account
* Trigger Build
* Trigger Build
---------
Co-authored-by: Mattermost Build <build@mattermost.com>
* Bump Go version to 1.23.6
* Update CodeQL Github action as well
* Use server's Go version for CodeQL action
Co-authored-by: Antonis Stamatiou <stamatiou.antonis@gmail.com>
* Empty commit to trigger CI
* Bump golangci-lint to a version supporting Go 1.23
* Fix golangci-lint warnings
Several rules from gosimple, revive and staticcheck linters were
failing:
- Redefinition of built-in identifiers (max, min, new, recover...)
- Use of printf-like functions with simple strings
- Check for nil slices, when len already takes it into account
---------
Co-authored-by: Antonis Stamatiou <stamatiou.antonis@gmail.com>
Co-authored-by: Mattermost Build <build@mattermost.com>
* MM-61947 Run DND expiry job more often and round expiry time to match interval
* Move comment to make it godoc-compatible
* Change truncateDNDEndTime to work with seconds
---------
Co-authored-by: Mattermost Build <build@mattermost.com>
* Adde MySQL and Postgres migrations
* Replaced select * with column names
* removed all * from channel SQL store
* cleanup
* Fixed a duplicate column
* cleanup
* Added migrations and store support
* WIP
* used channelname slice in a missed place
* Handled patch
* Added app level tests
* Added API layer tests
* Added API layer tests
* WIP
* converted to query builder
* cleanupo
* added not null and default constraints
* Fixed test
* fixed file name
* review fixes
* review fixes
* updated migration file
* fixed text
* Review fixes
When we added iteration by channelID, this was a known tradeoff during that.
However, it has been observed that the regular connection removal function
creates considerable blocking of the processing loop, leading to high
CPU usage and API latencies.
To fix this, we add a reverse mapping of channelIDs to connections
and their positions in the slice. This helps us to remove the connection
from the slice without iteration.
Unfortunately, this still needs to iterate through all channelIDs
during invalidation of the channel member cache. However, the user
cache invalidation is not a regular activity. So it should be an acceptable
tradeoff to make.
https://mattermost.atlassian.net/browse/MM-62960
```release-note
A new config knob ServiceSettings.EnableWebHubChannelIteration which allows a user to control the performance of websocket broadcasting. By default, this setting is turned off. If it is turned on, it improves the websocket broadcasting performance at the expense of poor performance when users join/leave a channel. It is not recommended to turn it on unless you have atleast 200,000 concurrent users actively using MM.
```
Co-authored-by: Mattermost Build <build@mattermost.com>
We add a new config setting to allow the admin to set a fixed
list of userIDs to track for all client side webapp metrics.
This gives the admin to get a deeper look at how the application
is behaving for a single user.
A new section in the system console is also added for the user
to edit this setting from the UI.
https://mattermost.atlassian.net/browse/MM-61888
```release-note
A new config setting MetricsSettings.ClientSideUserIds is added
where you can set the user ids you want to track for client side webapp
metrics.
```
* fix lint errors
```release-note
NONE
```
* fixing tests
```release-note
NONE
```
* refactor: Replace pagination with cursor-based pagination for custom profile attributes
* remove pagination loop on property value retrieval for CPA
* add migrations to optimize pagination on property fields and values
* adapt test to remove pagination check
* update migrations list
* postgres: drop index concurrently
* concurrent index manipulation must be done outside of a Tx
* fix: Correct SQL index drop syntax from "OM" to "ON" in migration files
* test: Add CountForGroup test cases for property field store
* refactor: Add CountForGroup method to PropertyFieldStore interface and implementations
* Fix style and i18n
* feat: Add optional deleted property field filtering to CountForGroup method
* refactor: Update CountForGroup to support optional deleted property fields
* test: Add comprehensive tests for CountForGroup with includeDeleted parameter
* adapt test + gen layers
* rename property service method and set the includeDelete to false
* refactor: Remove redundant constant and use CustomProfileAttributesFieldLimit directly
* fix tests
---------
Co-authored-by: Mattermost Build <build@mattermost.com>
* Adds websocket messages to Custom Profile Attributes
The app layer now fires a websocket event as part of the operations
over Custom Profile Attribute fields and values. It updates as well
the Patch method for CPA values so all the changes are commited as
part of the same transaction.
To be able to do this last operation, the change adds methods to
upsert CPA values in both the store and the property service.
* Fix i18n strings
---------
Co-authored-by: Mattermost Build <build@mattermost.com>
Co-authored-by: Miguel de la Cruz <miguel@ctrlz.es>
* refactor: Move property value sanitization to model layer
* feat: Add value sanitization for custom profile attributes
* refactor: Update custom profile attributes to use json.RawMessage
* refactor: Update patchCustomProfileAttribute to handle json.RawMessage directly
* refactor: Refactor custom profile attributes handler with improved validation
* refactor: Rename `patchCustomProfileAttribute` to `patchCPAValues`
* refactor: Replace ReturnJSON with json.NewEncoder and add error logging
* feat: Add encoding/json import to property_value.go
* refactor: Update property value tests to use json.RawMessage
* fix: Convert string value to json.RawMessage in property value test
* fix: Convert string literals to json.RawMessage in property value tests
* fix: Add missing encoding/json import in custom_profile_attributes.go
* fix: Preserve JSON RawMessage type in listCPAValues function
* fix: Update custom profile attributes test to use json.RawMessage
* feat: Add json import to custom_profile_attributes_test.go
* refactor: Update ListCPAValues and PatchCPAValues to use json.RawMessage
* refactor: Rename `actualValue` to `updatedValue` in custom profile attributes test
* refactor: Improve user permission and audit logging for custom profile attributes patch
* refactor: Optimize CPA field lookup by using ListCPAFields() and map
* fix: Correct user ID reference in custom profile attributes patch endpoint
* refactor: Change patchCPAValues to use map[string]json.RawMessage for results
* refactor: format and fix tests
* test: Add comprehensive unit tests for sanitizePropertyValue function
* test: Add test case for invalid property value type
* feat: Use `model.NewId()` to generate valid IDs in custom profile attributes tests
* refactor: Replace hardcoded IDs with dynamic variables in custom profile attributes test
* refactor: restore variable name
* refactor: drop undesired changes
* chore: refresh app layers
* feat: Update API definition to support string or string array values for custom profile attributes
* test: Add test cases for multiselect custom profile attribute values
* test: Add tests for multiselect custom profile attribute values
* test: Isolate array value test in separate t.Run
* test: Add test case for multiselect array values in custom profile attributes
* refactor: Move array value test from TestCreateCPAField to TestPatchCPAValue
* test: Update custom profile attributes test assertions
* test: add test case for handling array values in GetCPAValue
* test: Add array value tests for property value store
* refactor(store): no need to convert to json the rawmessage
* chore: lint
* i18n
* use model to interface with sqlx
* fix: Allow empty strings for text, date, and select profile attributes
* refactor: Filter out empty strings in multiselect and multiuser fields
* refactor: Update multiuser field sanitization to validate and error on invalid IDs
* refactor: Simplify sanitizePropertyValue function with reduced code duplication
* fix: Allow empty user ID in custom profile attribute sanitization
* refactor: Convert comment-based subtests to nested t.Run in TestSanitizePropertyValue
* refactor: Convert comment-based subtests to nested t.Run tests in TestSanitizePropertyValue
---------
Co-authored-by: Mattermost Build <build@mattermost.com>
* Audit calls to searchPosts API
* Update server/public/model/post_search_results.go
Co-authored-by: Ben Schumacher <ben.schumacher@mattermost.com>
* Update server/public/model/post_search_results.go
* Update server/public/model/post_search_results.go
---------
Co-authored-by: Ben Schumacher <ben.schumacher@mattermost.com>
Co-authored-by: Mattermost Build <build@mattermost.com>
We do a cluster request to get the active and dead queues
from other nodes in the cluster to sync any missing
information.
We check the dead queue in the other nodes to see
if there's been any message loss or not. Accordingly,
we send just the active queue or both active and dead queues.
There's still an edge case that is left out where
a client could have potentially connected and reconnected
to multiple nodes leaving multiple active queues
in multiple nodes. We don't handle this scenario
because then potentially we need to create
a slice of sendQueueSize * number_of_nodes. And then
this can happen again, leading to an infinite increase
in sendQueueSize.
We leave this edge-case to Redis, acknowledging
a limitation in our architecture.
In this PR, when there's no message loss, we just
take the active queue from the last node it connected
to.
And if there's message loss where the client's
seqNum is within the last node's dead queue, we also
handle that.
But if there's severe message loss where the client's
seqNum falls within the dead queue of another node, then
we just send the data from that node to reconstruct the
data as much as possible. It could be possible to set
a new connection ID in this case, but this involves
more data transfer always from all nodes and recomputing
the state in the requestor node.
https://mattermost.atlassian.net/browse/MM-61904
```release-note
NONE
```
Co-authored-by: Mattermost Build <build@mattermost.com>
* MM-61865: Add mobile client content load network metrics
* added new common label
* renaming from MobileClientContentLoad to MobileClientNetworkRequests
* content_load_group => network_request_group
* refactor more NetworkRequest-* changes
* replace contentLoadGroup to networkRequestGroup
* new metrics elapsedTime
* Refactor urlCount to totalRequests
* add averageSpeed metric
* replace contentLoadGroup with networkRequestGroup
* use h.Labels vs commonLabels for network_request_group
* add agent
* add effective latency metrics
* add total parallel requests & total sequential
* mocks generated by mockery
* did a bit of cleanup and sorting
* formatting
* updated the AcceptedNetworkRequestGroups
* cleanup and sorting
* Adds the main Property System Architecture components
This change adds the necessary migrations for the Property Groups,
Fields and Values tables to be created, the store layer and a Property
Service that can be used from the app layer.
* Adds Custom Profile Attributes endpoints and app layer
* implement get and patch cpa values
* run i18n-extract
* Update property field type to use user instead of person
* Update PropertyFields to allow for unique nondeleted fields and remove redundant indexes
* Update PropertyValues to allow for unique nondeleted fields and remove redundant indexes
* Use StringMap instead of the map[string]any on property fields
* Add i18n strings
* Revert "Use StringMap instead of the map[string]any on property fields"
This reverts commit e2735ab0f8589d2524d636419ca0cb144575c4d6.
* Cast JSON binary data to string and add todo note for StringMap use
* Add mocks to the retrylayer tests
* Cast JSON binary data to string in property value store
* Check for binary parameter instead of casting to string for JSON data
* Fix bad merge
* Check property field type is one of the allowed ones
* Avoid reusing err variable to be explicit about the returned value
* Merge Property System Migrations into one file
* Adds NOT NULL to timestamps at the DB level
* Update stores to use tableSelectQuery instead of a slice var
* Update PropertyField model translations to be more explicit and avoid repetition
* Update PropertyValue model translations to be more explicit and avoid repetition
* Use ExecBuilder instead of ToSql&Exec
* Update property field errors to add context
* Ensure PerPage is greater than zero
* Update store errors to give more context
* Use ExecBuilder in the property stores where possible
* Add an on conflict suffix to the group register to avoid race conditions
* Remove user profile API documentation changes
* Update patchCPAValues endpoint and docs to return the updated information
* Merge two similar error conditions
* Use a route function for ListCPAValues
* Remove badly used translation string
* Remove unused get in register group method
* Adds input sanitization and validation to the CPA API endpoints
* Takes login outside of one test case to make it clear it affects multiple t.Runs
* Fix wrap error and return code when property field has been deleted
* Fix receiver name
* Adds comment to move the CPA group ID to the db cache
* Set the PerPage of CPA fields to the fields limit
* Update server/channels/app/custom_profile_attributes_test.go
Co-authored-by: Alejandro García Montoro <alejandro.garciamontoro@gmail.com>
* Standardize group ID access
* Avoid polluting the state between tests
* Use specific errors for the retrieval of CPA group
---------
Co-authored-by: Scott Bishel <scott.bishel@mattermost.com>
Co-authored-by: Mattermost Build <build@mattermost.com>
Co-authored-by: Alejandro García Montoro <alejandro.garciamontoro@gmail.com>
* Updated patch/update post API to allow file modification (#29447)
* WIP
* WIP
* Atatched new files ton post
* WIP: deleting removed files
* Deleted removed files and invalidated file metadata cache
* removed file ignore logif from update post API
* Added TestFindExclusives
* Added tests for DeleteForPostByIds
* Added app layer tests
* Added tests
* Added API level tests
* test enhancements
* Fixed a test
* Edit history include file metadata (#29505)
* Send file metadata in edit history metadata
* Added app tests
* Added store tests
* Added tests for populateEditHistoryFileMetadata{
* Added cache to avoid repetitigve DB calls for edits with only message changes
* Added API tests
* i18m fix
* removed commented code
* Improved test helper
* Show attachments in edit history RHS (#29519)
* Send file metadata in edit history metadata
* Added app tests
* Added store tests
* Added tests for populateEditHistoryFileMetadata{
* Added cache to avoid repetitigve DB calls for edits with only message changes
* Added API tests
* i18m fix
* WIUP: displa files in edit
* removed commented code
* Displayed file in edit history
* Handled file icon
* Fixed closing history component on clicking on file
* Simplified selector
* Simplified selector
* Improved test helper
* Disabled action menu on edit history file
* Added tests
* Improved selector
* Updated snapshot
* review Fixes
* restructured componnets
* Updated test
* Updated test
* Restore post api (#29643)
* Restore post version API WIP
* Undelete files WIP
* Added store tests
* Created post restore API
* Updated updatepost safeUpdate signature
* review fixex and improvements
* Fixed an app test
* Added API laer tests
* Added API tests and OpenAPI specs
* Fixed a typo
* Allow editing files when editing posts (#29709)
* WIP - basic view files when editing post
* Cleanup
* bg color
* Added text editor tests for files
* WIP
* WIP
* removed debug log
* Allowed admin to add and remove files on someone else's post
* Handled drafts and scheduled posts
* linter fixes
* Updated snapshot
* server test fix
* CI
* Added doc
* Restore post api integration (#29719)
* WIP - basic view files when editing post
* Cleanup
* bg color
* Added text editor tests for files
* WIP
* WIP
* removed debug log
* Allowed admin to add and remove files on someone else's post
* Handled drafts and scheduled posts
* linter fixes
* Updated snapshot
* server test fix
* Used new API to restore post
* handled edut limit and undo
* lint fix
* added comments
* Fixed edit post item tests
* Fixed buttons
* Aded snapshots
* fix test
* Updated snapshot
* Minor fixes
* fixed snapshot
* Edit file dnd area (#29763)
* dnd wip
* DND continued
* Supported multiple unbind dragster funcs
* lint fixes
* Got center channel file drop working when editing a post
* file dnd working with center channel and rhs
* file dnd working with center channel and rhs
* removed unneeded stopPropogation calls
* cleanup
* DND overlay fix
* Lint fix
* Advanced text editor test updates for file upload overlay
* fixed use upload hook tests
* Updated some more snapshots
* minor cleanup
* Updated i18n
* removed need of array for dragster unbind events
* lint fixes
* edit history cursor
* Fixed bugu causing faliure to delete empty posts (#29778)
* Files in restore confirmation (#29781)
* Added files to restore post confirmation dialog
* Fixed post restore toast colors
* Fixed restore bug
* Fixed restore confirmation toast tests
* a11y improvement and modal width fix
* Edit attachment misc fixes (#29808)
* Removed single image actions in restore post confirmation dialog
* Fixed file drop overlay size and position
* Made edit indiator accessible
* Lint fix
* Added bunch of more tests
* ANother test migrated from enzyme to react testing library
* More test enhancements
* More test enhancements
* More test enhancements
* lint fixes
* Fixed a test
* Added missing snapshots
* Test fixes
* Adds the main Property System Architecture components
This change adds the necessary migrations for the Property Groups,
Fields and Values tables to be created, the store layer and a Property
Service that can be used from the app layer.
* Update property field type to use user instead of person
* Update PropertyFields to allow for unique nondeleted fields and remove redundant indexes
* Update PropertyValues to allow for unique nondeleted fields and remove redundant indexes
* Use StringMap instead of the map[string]any on property fields
* Add i18n strings
* Revert "Use StringMap instead of the map[string]any on property fields"
This reverts commit e2735ab0f8589d2524d636419ca0cb144575c4d6.
* Cast JSON binary data to string and add todo note for StringMap use
* Add mocks to the retrylayer tests
* Cast JSON binary data to string in property value store
* Check for binary parameter instead of casting to string for JSON data
* Check property field type is one of the allowed ones
* Avoid reusing err variable to be explicit about the returned value
* Merge Property System Migrations into one file
* Adds NOT NULL to timestamps at the DB level
* Update stores to use tableSelectQuery instead of a slice var
* Update PropertyField model translations to be more explicit and avoid repetition
* Update PropertyValue model translations to be more explicit and avoid repetition
* Use ExecBuilder instead of ToSql&Exec
* Update property field errors to add context
* Ensure PerPage is greater than zero
* Update store errors to give more context
* Use ExecBuilder in the property stores where possible
* Add an on conflict suffix to the group register to avoid race conditions
* Remove badly used translation string
* Remove unused get in register group method
---------
Co-authored-by: Mattermost Build <build@mattermost.com>
* Allow using plugin settings defined in manifest as fallback when using custom sections and plugin is disabled
* Update server/public/model/manifest.go
Co-authored-by: Christopher Poile <cpoile@gmail.com>
* Add custom setting case
---------
Co-authored-by: Christopher Poile <cpoile@gmail.com>
Co-authored-by: Mattermost Build <build@mattermost.com>
* [MM-59089] Add a compliance export constant (#27919)
* add a useful constant
* i18n
* another constant
* another i18n
* [MM-60422] Add GetChannelsWithActivityDuring (#28301)
* modify GetUsersInChannelDuring to accept a slice of channelIds
* add GetChannelsWithActivityDuring
* add compliance export progress message; remove unused custom status
* linting
* tests running too fast
* add batch size config settings
* add store tests
* linting
* empty commit
* i18n changes
* fix i18n ordering
* MM-60570 - Server-side changes consolidating the export CLI with server/ent code (#28640)
* add an i18n field; add the CLI's export directory
* int64 -> int
* Add UntilUpdateAt for MessageExport and AnalyticsPostCount
to merge
* remove now-unused i18n strings
* add TranslationsPreInitFromBuffer to allow CLI to use i18n
* use GetBuilder to simplify; rename TranslationsPreInitFromFileBytes
* [MM-59089] Improve compliance export timings (#1733 - Enterprise repo)
* MM-60422 - Performance and logic fixes for Compliance Exports (#1757 - Enterprise repo)
* MM-60570 - Enterprise-side changes consolidating the export CLI with server/ent code (#1769 - Enterprise repo)
* merge conflicts; missed file from ent branch
* MM-61038 - Add an option to sqlstore.New (#28702)
remove useless comment
add test
add an option to sqlstore.New
* MM-60976: Remove RunExport command from Mattermost binary (#28805)
* remove RunExport command from mattermost binary
* remove the code it was calling
* fix i18n
* remove test (was only testing license, not functionality)
* empty commit
* fix flaky GetChannelsWithActivityDuring test
* MM-60063: Dedicated Export Filestore fix, redo of #1772 (enterprise) (#28803)
* redo filestore fix#1772 (enterprise repo) on top of MM-59966 feature
* add new e2e tests for export filestore
* golint
* ok, note to self: shadowing bad, actually (when there's a defer)
* empty commit
* MM-61137 - Message export: Support 7.8.11 era dbs (#28824)
* support 7.8.11 era dbs by wrapping the store using only what we need
* fix flaky GetChannelsWithActivityDuring test
* add a comment
* only need to define the MEFileInfoStore (the one that'll be overridden)
* blank commit
* MM-60974 - Message Export: Add performance metrics (#28836)
* support 7.8.11 era dbs by wrapping the store using only what we need
* fix flaky GetChannelsWithActivityDuring test
* add a comment
* only need to define the MEFileInfoStore (the one that'll be overridden)
* performance metrics
* cleanup unneeded named returns
* blank commit
* MM-60975 - Message export: Add startTime and endTime to export folder name (#28840)
* support 7.8.11 era dbs by wrapping the store using only what we need
* fix flaky GetChannelsWithActivityDuring test
* add a comment
* only need to define the MEFileInfoStore (the one that'll be overridden)
* performance metrics
* output startTime and endTime in export folder
* empty commit
* merge conflict
* MM-60978 - Message export: Improve xml fields; fix delete semantics (#28873)
* support 7.8.11 era dbs by wrapping the store using only what we need
* fix flaky GetChannelsWithActivityDuring test
* add a comment
* only need to define the MEFileInfoStore (the one that'll be overridden)
* performance metrics
* output startTime and endTime in export folder
* empty commit
* add xml fields, omit when empty, tests
* fix delete semantics; test (and test for update semantics)
* clarify comments
* simplify edited post detection, now there's no edge case.
* add some spacing to help fast running tests
* merge conflicts/updates needed for new deleted post semantics
* linting; fixing tests from upstream merge
* use SafeDereference
* linting
* stronger typing; better wrapped errors; better formatting
* blank commit
* goimports formatting
* fix merge mistake
* minor fixes due to changes in master
* MM-61755 - Simplifying and Support reporting to the db from the CLI (#29281)
* finally clean up JobData struct and stringMap; prep for CLI using db
* and now simplify using StringMapToJobDataWithZeroValues
* remove unused fn
* create JobDataExported; clean up errors
* MM-60176 - Message Export: Global relay cleanup (#29168)
* move global relay logic into global_relay_export
* blank commit
* blank commit
* improve errors
* MM-60693 - Refactor CSV to use same codepath as Actiance (#29191)
* move global relay logic into global_relay_export
* blank commit
* refactor (and simplify) ExportParams into shared
* blank commit
* remove unused fn
* csv now uses pre-calculated joins/leaves like actiance
* improve errors
* remove nil post check; remove ignoredPosts metric
* remove unneeded copy
* MM-61696 - Refactor GlobalRelay to use same codepath as Actiance (#29225)
* move global relay logic into global_relay_export
* blank commit
* refactor (and simplify) ExportParams into shared
* blank commit
* remove unused fn
* csv now uses pre-calculated joins/leaves like actiance
* remove newly unneeded function and its test. goodbye.
* refactor GetPostAttachments for csv + global relay to share
* refactor global_relay_export and fix tests (no changes to output)
* improve errors
* remove nil post check; remove ignoredPosts metric
* remove unneeded copy
* remove unneeded nil check
* PR comments
* MM-61715 - Generalize e2e to all export types 🤖 (#29369)
* move global relay logic into global_relay_export
* blank commit
* refactor (and simplify) ExportParams into shared
* blank commit
* remove unused fn
* csv now uses pre-calculated joins/leaves like actiance
* remove newly unneeded function and its test. goodbye.
* refactor GetPostAttachments for csv + global relay to share
* refactor global_relay_export and fix tests (no changes to output)
* improve errors
* remove nil post check; remove ignoredPosts metric
* remove unneeded copy
* remove unneeded nil check
* PR comments
* refactor isDeletedMsg for all export types
* fix start and endtime, nasty csv createAt bug; bring closer to Actiance
* align unit tests with new logic (e.g. starttime / endtime)
* refactor a TimestampConvert fn for code + tests
* bug: pass templates to global relay (hurray for e2e tests, otherwise...)
* add global relay zip to allowed list (only for tests)
* test helpers
* new templates for e2e tests
* e2e tests... phew.
* linting
* merge conflicts
* unexport PostToRow; add test helper marker
* cleanup, shortening, thanks to PR comments
* MM-61972 - Generalize export data path - Actiance (#29399)
* extract and generalize the export data generation functions
* finish moving test (bc of previous extraction)
* lift a function from common -> shared (to break an import cycle)
* actiance now takes general export data, processes it into actiance data
* bring tests in line with correct sorting rules (upadateAt, messageId)
* fixups, PR comments
* turn strings.Repeat into a more descriptive const
amended: one letter fix; bad rebase
* MM-62009 - e2e clock heisenbug (#29434)
* consolidate assertions; output debuggable diffs (keeping for future)
* refactor test output generator to generators file
* waitUntilZeroPosts + pass through until to job = fix all clock issues
* simplify messages to model.NewId(); remove unneeded waitUntilZeroPosts
* model.NewId() -> storetest.NewTestID()
* MM-61980 - Generalize export data path - CSV (#29482)
* simple refactoring
* increase sleep times for (very) rare test failures
* add extra information to the generic export for CSV
* adj Actiance to handle new generic export (no difference in its output)
* no longer need mergePosts (yay), move getJoinLeavePosts for everyone
* adjust tests for new csv semantics (detailed in summary)
* and need to add the new exported data to the export_data_tests
* rearrange csv writing to happen after data export (more logical)
* linting
* remove debug statements
* figured out what was wrong with global relay e2e test 3; solid now
* PR comments
* MM-61718 - Generalize export data path - Global Relay (#29508)
* move global relay over to using the generalized export data
* performance pass -- not much can be done
* Update server/enterprise/message_export/global_relay_export/global_relay_export.go
Co-authored-by: Claudio Costa <cstcld91@gmail.com>
---------
Co-authored-by: Claudio Costa <cstcld91@gmail.com>
* MM-62058 - Align CSV with Actiance (#29551)
* refactoring actiance files and var names for clarity
* bug found in exported attachments (we used to miss some start/ends)
* changes needed for actiance due to new generic exports
* bringing CSV up to actiance standards
* fixing global relay b/c of new semantics (adding a note on an edge case)
* aligning e2e tests, adding comments to clarify what is expected/tested
* necessary changes; 1 more test for added functionality (ignoreDeleted)
* comment style
* MM-62059 - Align Global Relay with Actiance/CSV; many fixes (#29665)
* core logic changes to general export_data and the specific export paths
* unit tests and e2e tests, covering all new edge cases and all logic
* linting
* better var naming, const value, and cleaning up functions calls
* MM-62436 - Temporarily skip cypress tests that require download link (#29772)
---------
Co-authored-by: Claudio Costa <cstcld91@gmail.com>
We cannot simply use Elasticsearch any more because
we support Opensearch as well.
Therefore changing Elasticsearch to just say Search.
Note: There are still some cases where Elasticsearch
is used. That is intentional and necessary.
```release-note
NONE
```
