MM-59966 - Compliance Export overhaul - feature branch (#29789)

* [MM-59089] Add a compliance export constant (#27919)

* add a useful constant

* i18n

* another constant

* another i18n

* [MM-60422] Add GetChannelsWithActivityDuring (#28301)

* modify GetUsersInChannelDuring to accept a slice of channelIds

* add GetChannelsWithActivityDuring

* add compliance export progress message; remove unused custom status

* linting

* tests running too fast

* add batch size config settings

* add store tests

* linting

* empty commit

* i18n changes

* fix i18n ordering

* MM-60570 - Server-side changes consolidating the export CLI with server/ent code (#28640)

* add an i18n field; add the CLI's export directory

* int64 -> int

* Add UntilUpdateAt for MessageExport and AnalyticsPostCount

to merge

* remove now-unused i18n strings

* add TranslationsPreInitFromBuffer to allow CLI to use i18n

* use GetBuilder to simplify; rename TranslationsPreInitFromFileBytes

* [MM-59089] Improve compliance export timings (#1733 - Enterprise repo)

* MM-60422 - Performance and logic fixes for Compliance Exports (#1757 - Enterprise repo)

* MM-60570 - Enterprise-side changes consolidating the export CLI with server/ent code (#1769 - Enterprise repo)

* merge conflicts; missed file from ent branch

* MM-61038 - Add an option to sqlstore.New (#28702)

remove useless comment

add test

add an option to sqlstore.New

* MM-60976: Remove RunExport command from Mattermost binary (#28805)

* remove RunExport command from mattermost binary

* remove the code it was calling

* fix i18n

* remove test (was only testing license, not functionality)

* empty commit

* fix flaky GetChannelsWithActivityDuring test

* MM-60063: Dedicated Export Filestore fix, redo of #1772 (enterprise) (#28803)

* redo filestore fix #1772 (enterprise repo) on top of MM-59966 feature

* add new e2e tests for export filestore

* golint

* ok, note to self: shadowing bad, actually (when there's a defer)

* empty commit

* MM-61137 - Message export: Support 7.8.11 era dbs (#28824)

* support 7.8.11 era dbs by wrapping the store using only what we need

* fix flaky GetChannelsWithActivityDuring test

* add a comment

* only need to define the MEFileInfoStore (the one that'll be overridden)

* blank commit

* MM-60974 - Message Export: Add performance metrics (#28836)

* support 7.8.11 era dbs by wrapping the store using only what we need

* fix flaky GetChannelsWithActivityDuring test

* add a comment

* only need to define the MEFileInfoStore (the one that'll be overridden)

* performance metrics

* cleanup unneeded named returns

* blank commit

* MM-60975 - Message export: Add startTime and endTime to export folder name (#28840)

* support 7.8.11 era dbs by wrapping the store using only what we need

* fix flaky GetChannelsWithActivityDuring test

* add a comment

* only need to define the MEFileInfoStore (the one that'll be overridden)

* performance metrics

* output startTime and endTime in export folder

* empty commit

* merge conflict

* MM-60978 - Message export: Improve xml fields; fix delete semantics (#28873)

* support 7.8.11 era dbs by wrapping the store using only what we need

* fix flaky GetChannelsWithActivityDuring test

* add a comment

* only need to define the MEFileInfoStore (the one that'll be overridden)

* performance metrics

* output startTime and endTime in export folder

* empty commit

* add xml fields, omit when empty, tests

* fix delete semantics; test (and test for update semantics)

* clarify comments

* simplify edited post detection, now there's no edge case.

* add some spacing to help fast running tests

* merge conflicts/updates needed for new deleted post semantics

* linting; fixing tests from upstream merge

* use SafeDereference

* linting

* stronger typing; better wrapped errors; better formatting

* blank commit

* goimports formatting

* fix merge mistake

* minor fixes due to changes in master

* MM-61755 - Simplifying and Support reporting to the db from the CLI (#29281)

* finally clean up JobData struct and stringMap; prep for CLI using db

* and now simplify using StringMapToJobDataWithZeroValues

* remove unused fn

* create JobDataExported; clean up errors

* MM-60176 - Message Export: Global relay cleanup (#29168)

* move global relay logic into global_relay_export

* blank commit

* blank commit

* improve errors

* MM-60693 - Refactor CSV to use same codepath as Actiance (#29191)

* move global relay logic into global_relay_export

* blank commit

* refactor (and simplify) ExportParams into shared

* blank commit

* remove unused fn

* csv now uses pre-calculated joins/leaves like actiance

* improve errors

* remove nil post check; remove ignoredPosts metric

* remove unneeded copy

* MM-61696 - Refactor GlobalRelay to use same codepath as Actiance (#29225)

* move global relay logic into global_relay_export

* blank commit

* refactor (and simplify) ExportParams into shared

* blank commit

* remove unused fn

* csv now uses pre-calculated joins/leaves like actiance

* remove newly unneeded function and its test. goodbye.

* refactor GetPostAttachments for csv + global relay to share

* refactor global_relay_export and fix tests (no changes to output)

* improve errors

* remove nil post check; remove ignoredPosts metric

* remove unneeded copy

* remove unneeded nil check

* PR comments

* MM-61715 - Generalize e2e to all export types 🤖  (#29369)

* move global relay logic into global_relay_export

* blank commit

* refactor (and simplify) ExportParams into shared

* blank commit

* remove unused fn

* csv now uses pre-calculated joins/leaves like actiance

* remove newly unneeded function and its test. goodbye.

* refactor GetPostAttachments for csv + global relay to share

* refactor global_relay_export and fix tests (no changes to output)

* improve errors

* remove nil post check; remove ignoredPosts metric

* remove unneeded copy

* remove unneeded nil check

* PR comments

* refactor isDeletedMsg for all export types

* fix start and endtime, nasty csv createAt bug; bring closer to Actiance

* align unit tests with new logic (e.g. starttime / endtime)

* refactor a TimestampConvert fn for code + tests

* bug: pass templates to global relay (hurray for e2e tests, otherwise...)

* add global relay zip to allowed list (only for tests)

* test helpers

* new templates for e2e tests

* e2e tests... phew.

* linting

* merge conflicts

* unexport PostToRow; add test helper marker

* cleanup, shortening, thanks to PR comments

* MM-61972 - Generalize export data path - Actiance (#29399)

* extract and generalize the export data generation functions

* finish moving test (bc of previous extraction)

* lift a function from common -> shared (to break an import cycle)

* actiance now takes general export data, processes it into actiance data

* bring tests in line with correct sorting rules (upadateAt, messageId)

* fixups, PR comments

* turn strings.Repeat into a more descriptive const

amended: one letter fix; bad rebase

* MM-62009 - e2e clock heisenbug (#29434)

* consolidate assertions; output debuggable diffs (keeping for future)

* refactor test output generator to generators file

* waitUntilZeroPosts + pass through until to job = fix all clock issues

* simplify messages to model.NewId(); remove unneeded waitUntilZeroPosts

* model.NewId() -> storetest.NewTestID()

* MM-61980 - Generalize export data path - CSV (#29482)

* simple refactoring

* increase sleep times for (very) rare test failures

* add extra information to the generic export for CSV

* adj Actiance to handle new generic export (no difference in its output)

* no longer need mergePosts (yay), move getJoinLeavePosts for everyone

* adjust tests for new csv semantics (detailed in summary)

* and need to add the new exported data to the export_data_tests

* rearrange csv writing to happen after data export (more logical)

* linting

* remove debug statements

* figured out what was wrong with global relay e2e test 3; solid now

* PR comments

* MM-61718 - Generalize export data path - Global Relay (#29508)

* move global relay over to using the generalized export data

* performance pass -- not much can be done

* Update server/enterprise/message_export/global_relay_export/global_relay_export.go

Co-authored-by: Claudio Costa <cstcld91@gmail.com>

---------

Co-authored-by: Claudio Costa <cstcld91@gmail.com>

* MM-62058 - Align CSV with Actiance (#29551)

* refactoring actiance files and var names for clarity

* bug found in exported attachments (we used to miss some start/ends)

* changes needed for actiance due to new generic exports

* bringing CSV up to actiance standards

* fixing global relay b/c of new semantics (adding a note on an edge case)

* aligning e2e tests, adding comments to clarify what is expected/tested

* necessary changes; 1 more test for added functionality (ignoreDeleted)

* comment style

* MM-62059 - Align Global Relay with Actiance/CSV; many fixes (#29665)

* core logic changes to general export_data and the specific export paths

* unit tests and e2e tests, covering all new edge cases and all logic

* linting

* better var naming, const value, and cleaning up functions calls

* MM-62436 - Temporarily skip cypress tests that require download link (#29772)

---------

Co-authored-by: Claudio Costa <cstcld91@gmail.com>

server/cmd/mattermost/commands/export.go

@@ -5,7 +5,6 @@ package commands
 
 import (
 	"context"
-	"fmt"
 	"os"
 	"path/filepath"
 	"time"
@@ -33,30 +32,6 @@ var ScheduleExportCmd = &cobra.Command{
 	RunE:    scheduleExportCmdF,
 }
 
-var CsvExportCmd = &cobra.Command{
-	Use:     "csv",
-	Short:   "Export data from Mattermost in CSV format",
-	Long:    "Export data from Mattermost in CSV format",
-	Example: "export csv --exportFrom=12345",
-	RunE:    buildExportCmdF("csv"),
-}
-
-var ActianceExportCmd = &cobra.Command{
-	Use:     "actiance",
-	Short:   "Export data from Mattermost in Actiance format",
-	Long:    "Export data from Mattermost in Actiance format",
-	Example: "export actiance --exportFrom=12345",
-	RunE:    buildExportCmdF("actiance"),
-}
-
-var GlobalRelayZipExportCmd = &cobra.Command{
-	Use:     "global-relay-zip",
-	Short:   "Export data from Mattermost into a zip file containing emails to send to Global Relay for debug and testing purposes only.",
-	Long:    "Export data from Mattermost into a zip file containing emails to send to Global Relay for debug and testing purposes only. This does not archive any information in Global Relay.",
-	Example: "export global-relay-zip --exportFrom=12345",
-	RunE:    buildExportCmdF("globalrelay-zip"),
-}
-
 var BulkExportCmd = &cobra.Command{
 	Use:     "bulk [file]",
 	Short:   "Export bulk data.",
@@ -71,15 +46,6 @@ func init() {
 	ScheduleExportCmd.Flags().Int64("exportFrom", -1, "The timestamp of the earliest post to export, expressed in seconds since the unix epoch.")
 	ScheduleExportCmd.Flags().Int("timeoutSeconds", -1, "The maximum number of seconds to wait for the job to complete before timing out.")
 
-	CsvExportCmd.Flags().Int64("exportFrom", -1, "The timestamp of the earliest post to export, expressed in seconds since the unix epoch.")
-	CsvExportCmd.Flags().Int("limit", -1, "The number of posts to export. The default of -1 means no limit.")
-
-	ActianceExportCmd.Flags().Int64("exportFrom", -1, "The timestamp of the earliest post to export, expressed in seconds since the unix epoch.")
-	ActianceExportCmd.Flags().Int("limit", -1, "The number of posts to export. The default of -1 means no limit.")
-
-	GlobalRelayZipExportCmd.Flags().Int64("exportFrom", -1, "The timestamp of the earliest post to export, expressed in seconds since the unix epoch.")
-	GlobalRelayZipExportCmd.Flags().Int("limit", -1, "The number of posts to export. The default of -1 means no limit.")
-
 	BulkExportCmd.Flags().Bool("all-teams", true, "Export all teams from the server.")
 	BulkExportCmd.Flags().Bool("with-archived-channels", false, "Also exports archived channels.")
 	BulkExportCmd.Flags().Bool("with-profile-pictures", false, "Also exports profile pictures.")
@@ -87,9 +53,6 @@ func init() {
 	BulkExportCmd.Flags().Bool("archive", false, "Outputs a single archive file.")
 
 	ExportCmd.AddCommand(ScheduleExportCmd)
-	ExportCmd.AddCommand(CsvExportCmd)
-	ExportCmd.AddCommand(ActianceExportCmd)
-	ExportCmd.AddCommand(GlobalRelayZipExportCmd)
 	ExportCmd.AddCommand(BulkExportCmd)
 
 	RootCmd.AddCommand(ExportCmd)
@@ -158,57 +121,6 @@ func scheduleExportCmdF(command *cobra.Command, args []string) error {
 	return nil
 }
 
-func buildExportCmdF(format string) func(command *cobra.Command, args []string) error {
-	return func(command *cobra.Command, args []string) error {
-		a, err := InitDBCommandContextCobra(command, app.SkipPostInitialization())
-		license := a.Srv().License()
-		if err != nil {
-			return err
-		}
-		defer a.Srv().Shutdown()
-
-		rctx := request.EmptyContext(a.Log())
-
-		startTime, err := command.Flags().GetInt64("exportFrom")
-		if err != nil {
-			return errors.New("exportFrom flag error")
-		}
-		if startTime < 0 {
-			return errors.New("exportFrom must be a positive integer")
-		}
-
-		limit, err := command.Flags().GetInt("limit")
-		if err != nil {
-			return errors.New("limit flag error")
-		}
-
-		if a.MessageExport() == nil || license == nil || !*license.Features.MessageExport {
-			return errors.New("message export feature not available")
-		}
-
-		warningsCount, appErr := a.MessageExport().RunExport(rctx, format, startTime, limit)
-		if appErr != nil {
-			return appErr
-		}
-		if warningsCount == 0 {
-			CommandPrettyPrintln("SUCCESS: Your data was exported.")
-		} else {
-			if format == model.ComplianceExportTypeGlobalrelay || format == model.ComplianceExportTypeGlobalrelayZip {
-				CommandPrettyPrintln(fmt.Sprintf("WARNING: %d warnings encountered, see logs for details.", warningsCount))
-			} else {
-				CommandPrettyPrintln(fmt.Sprintf("WARNING: %d warnings encountered, see warning.txt for details.", warningsCount))
-			}
-		}
-
-		auditRec := a.MakeAuditRecord(rctx, "buildExport", audit.Success)
-		auditRec.AddMeta("format", format)
-		auditRec.AddMeta("start", startTime)
-		a.LogAuditRec(rctx, auditRec, nil)
-
-		return nil
-	}
-}
-
 func bulkExportCmdF(command *cobra.Command, args []string) error {
 	a, err := InitDBCommandContextCobra(command, app.SkipPostInitialization())
 	if err != nil {

server/cmd/mattermost/commands/export_test.go

@@ -1,74 +0,0 @@
-// Copyright (c) 2015-present Mattermost, Inc. All Rights Reserved.
-// See LICENSE.txt for license information.
-
-package commands
-
-import (
-	"testing"
-
-	"github.com/stretchr/testify/require"
-
-	"github.com/mattermost/mattermost/server/public/model"
-)
-
-// There are no tests that actually run the Message Export job, because it can take a long time to complete depending
-// on the size of the database that the config is pointing to. As such, these tests just ensure that the CLI command
-// fails fast if invalid flags are supplied
-
-func TestMessageExportNotEnabled(t *testing.T) {
-	th := Setup(t)
-	defer th.TearDown()
-
-	config := th.Config()
-	config.MessageExportSettings.EnableExport = model.NewPointer(false)
-	th.SetConfig(config)
-
-	// should fail fast because the feature isn't enabled
-	require.Error(t, th.RunCommand(t, "export", "schedule"))
-}
-
-func TestMessageExportInvalidFormat(t *testing.T) {
-	th := Setup(t)
-	defer th.TearDown()
-
-	config := th.Config()
-	config.MessageExportSettings.EnableExport = model.NewPointer(true)
-	th.SetConfig(config)
-
-	// should fail fast because format isn't supported
-	require.Error(t, th.RunCommand(t, "--format", "not_actiance", "export", "schedule"))
-}
-
-func TestMessageExportNegativeExportFrom(t *testing.T) {
-	th := Setup(t)
-	defer th.TearDown()
-
-	config := th.Config()
-	config.MessageExportSettings.EnableExport = model.NewPointer(true)
-	th.SetConfig(config)
-
-	// should fail fast because export from must be a valid timestamp
-	require.Error(t, th.RunCommand(t, "--format", "actiance", "--exportFrom", "-1", "export", "schedule"))
-}
-
-func TestMessageExportNegativeTimeoutSeconds(t *testing.T) {
-	th := Setup(t)
-	defer th.TearDown()
-
-	config := th.Config()
-	config.MessageExportSettings.EnableExport = model.NewPointer(true)
-	th.SetConfig(config)
-
-	// should fail fast because timeout seconds must be a positive int
-	require.Error(t, th.RunCommand(t, "--format", "actiance", "--exportFrom", "0", "--timeoutSeconds", "-1", "export", "schedule"))
-}
-
-func TestMessageExportFailsWithoutLicense(t *testing.T) {
-	th := Setup(t)
-	defer th.TearDown()
-
-	actual, _ := th.RunCommandWithOutput(t, "export", "csv", "export.csv", "--exportFrom", "0")
-
-	// should fail fast because timeout seconds must be a positive int
-	require.Contains(t, actual, "message export feature not available")
-}