MM-59966 - Compliance Export overhaul - feature branch (#29789)

* [MM-59089] Add a compliance export constant (#27919)

* add a useful constant

* i18n

* another constant

* another i18n

* [MM-60422] Add GetChannelsWithActivityDuring (#28301)

* modify GetUsersInChannelDuring to accept a slice of channelIds

* add GetChannelsWithActivityDuring

* add compliance export progress message; remove unused custom status

* linting

* tests running too fast

* add batch size config settings

* add store tests

* linting

* empty commit

* i18n changes

* fix i18n ordering

* MM-60570 - Server-side changes consolidating the export CLI with server/ent code (#28640)

* add an i18n field; add the CLI's export directory

* int64 -> int

* Add UntilUpdateAt for MessageExport and AnalyticsPostCount

to merge

* remove now-unused i18n strings

* add TranslationsPreInitFromBuffer to allow CLI to use i18n

* use GetBuilder to simplify; rename TranslationsPreInitFromFileBytes

* [MM-59089] Improve compliance export timings (#1733 - Enterprise repo)

* MM-60422 - Performance and logic fixes for Compliance Exports (#1757 - Enterprise repo)

* MM-60570 - Enterprise-side changes consolidating the export CLI with server/ent code (#1769 - Enterprise repo)

* merge conflicts; missed file from ent branch

* MM-61038 - Add an option to sqlstore.New (#28702)

remove useless comment

add test

add an option to sqlstore.New

* MM-60976: Remove RunExport command from Mattermost binary (#28805)

* remove RunExport command from mattermost binary

* remove the code it was calling

* fix i18n

* remove test (was only testing license, not functionality)

* empty commit

* fix flaky GetChannelsWithActivityDuring test

* MM-60063: Dedicated Export Filestore fix, redo of #1772 (enterprise) (#28803)

* redo filestore fix #1772 (enterprise repo) on top of MM-59966 feature

* add new e2e tests for export filestore

* golint

* ok, note to self: shadowing bad, actually (when there's a defer)

* empty commit

* MM-61137 - Message export: Support 7.8.11 era dbs (#28824)

* support 7.8.11 era dbs by wrapping the store using only what we need

* fix flaky GetChannelsWithActivityDuring test

* add a comment

* only need to define the MEFileInfoStore (the one that'll be overridden)

* blank commit

* MM-60974 - Message Export: Add performance metrics (#28836)

* support 7.8.11 era dbs by wrapping the store using only what we need

* fix flaky GetChannelsWithActivityDuring test

* add a comment

* only need to define the MEFileInfoStore (the one that'll be overridden)

* performance metrics

* cleanup unneeded named returns

* blank commit

* MM-60975 - Message export: Add startTime and endTime to export folder name (#28840)

* support 7.8.11 era dbs by wrapping the store using only what we need

* fix flaky GetChannelsWithActivityDuring test

* add a comment

* only need to define the MEFileInfoStore (the one that'll be overridden)

* performance metrics

* output startTime and endTime in export folder

* empty commit

* merge conflict

* MM-60978 - Message export: Improve xml fields; fix delete semantics (#28873)

* support 7.8.11 era dbs by wrapping the store using only what we need

* fix flaky GetChannelsWithActivityDuring test

* add a comment

* only need to define the MEFileInfoStore (the one that'll be overridden)

* performance metrics

* output startTime and endTime in export folder

* empty commit

* add xml fields, omit when empty, tests

* fix delete semantics; test (and test for update semantics)

* clarify comments

* simplify edited post detection, now there's no edge case.

* add some spacing to help fast running tests

* merge conflicts/updates needed for new deleted post semantics

* linting; fixing tests from upstream merge

* use SafeDereference

* linting

* stronger typing; better wrapped errors; better formatting

* blank commit

* goimports formatting

* fix merge mistake

* minor fixes due to changes in master

* MM-61755 - Simplifying and Support reporting to the db from the CLI (#29281)

* finally clean up JobData struct and stringMap; prep for CLI using db

* and now simplify using StringMapToJobDataWithZeroValues

* remove unused fn

* create JobDataExported; clean up errors

* MM-60176 - Message Export: Global relay cleanup (#29168)

* move global relay logic into global_relay_export

* blank commit

* blank commit

* improve errors

* MM-60693 - Refactor CSV to use same codepath as Actiance (#29191)

* move global relay logic into global_relay_export

* blank commit

* refactor (and simplify) ExportParams into shared

* blank commit

* remove unused fn

* csv now uses pre-calculated joins/leaves like actiance

* improve errors

* remove nil post check; remove ignoredPosts metric

* remove unneeded copy

* MM-61696 - Refactor GlobalRelay to use same codepath as Actiance (#29225)

* move global relay logic into global_relay_export

* blank commit

* refactor (and simplify) ExportParams into shared

* blank commit

* remove unused fn

* csv now uses pre-calculated joins/leaves like actiance

* remove newly unneeded function and its test. goodbye.

* refactor GetPostAttachments for csv + global relay to share

* refactor global_relay_export and fix tests (no changes to output)

* improve errors

* remove nil post check; remove ignoredPosts metric

* remove unneeded copy

* remove unneeded nil check

* PR comments

* MM-61715 - Generalize e2e to all export types 🤖  (#29369)

* move global relay logic into global_relay_export

* blank commit

* refactor (and simplify) ExportParams into shared

* blank commit

* remove unused fn

* csv now uses pre-calculated joins/leaves like actiance

* remove newly unneeded function and its test. goodbye.

* refactor GetPostAttachments for csv + global relay to share

* refactor global_relay_export and fix tests (no changes to output)

* improve errors

* remove nil post check; remove ignoredPosts metric

* remove unneeded copy

* remove unneeded nil check

* PR comments

* refactor isDeletedMsg for all export types

* fix start and endtime, nasty csv createAt bug; bring closer to Actiance

* align unit tests with new logic (e.g. starttime / endtime)

* refactor a TimestampConvert fn for code + tests

* bug: pass templates to global relay (hurray for e2e tests, otherwise...)

* add global relay zip to allowed list (only for tests)

* test helpers

* new templates for e2e tests

* e2e tests... phew.

* linting

* merge conflicts

* unexport PostToRow; add test helper marker

* cleanup, shortening, thanks to PR comments

* MM-61972 - Generalize export data path - Actiance (#29399)

* extract and generalize the export data generation functions

* finish moving test (bc of previous extraction)

* lift a function from common -> shared (to break an import cycle)

* actiance now takes general export data, processes it into actiance data

* bring tests in line with correct sorting rules (upadateAt, messageId)

* fixups, PR comments

* turn strings.Repeat into a more descriptive const

amended: one letter fix; bad rebase

* MM-62009 - e2e clock heisenbug (#29434)

* consolidate assertions; output debuggable diffs (keeping for future)

* refactor test output generator to generators file

* waitUntilZeroPosts + pass through until to job = fix all clock issues

* simplify messages to model.NewId(); remove unneeded waitUntilZeroPosts

* model.NewId() -> storetest.NewTestID()

* MM-61980 - Generalize export data path - CSV (#29482)

* simple refactoring

* increase sleep times for (very) rare test failures

* add extra information to the generic export for CSV

* adj Actiance to handle new generic export (no difference in its output)

* no longer need mergePosts (yay), move getJoinLeavePosts for everyone

* adjust tests for new csv semantics (detailed in summary)

* and need to add the new exported data to the export_data_tests

* rearrange csv writing to happen after data export (more logical)

* linting

* remove debug statements

* figured out what was wrong with global relay e2e test 3; solid now

* PR comments

* MM-61718 - Generalize export data path - Global Relay (#29508)

* move global relay over to using the generalized export data

* performance pass -- not much can be done

* Update server/enterprise/message_export/global_relay_export/global_relay_export.go

Co-authored-by: Claudio Costa <cstcld91@gmail.com>

---------

Co-authored-by: Claudio Costa <cstcld91@gmail.com>

* MM-62058 - Align CSV with Actiance (#29551)

* refactoring actiance files and var names for clarity

* bug found in exported attachments (we used to miss some start/ends)

* changes needed for actiance due to new generic exports

* bringing CSV up to actiance standards

* fixing global relay b/c of new semantics (adding a note on an edge case)

* aligning e2e tests, adding comments to clarify what is expected/tested

* necessary changes; 1 more test for added functionality (ignoreDeleted)

* comment style

* MM-62059 - Align Global Relay with Actiance/CSV; many fixes (#29665)

* core logic changes to general export_data and the specific export paths

* unit tests and e2e tests, covering all new edge cases and all logic

* linting

* better var naming, const value, and cleaning up functions calls

* MM-62436 - Temporarily skip cypress tests that require download link (#29772)

---------

Co-authored-by: Claudio Costa <cstcld91@gmail.com>

server/enterprise/message_export/message_export.go

@@ -6,34 +6,22 @@ package message_export
 import (
 	"encoding/json"
 	"errors"
-	"fmt"
-	"math"
-	"net/http"
-	"os"
-	"path"
-	"time"
-
 	"strconv"
+	"time"
 
 	"github.com/mattermost/mattermost/server/public/model"
 	"github.com/mattermost/mattermost/server/public/shared/mlog"
 	"github.com/mattermost/mattermost/server/public/shared/request"
 	"github.com/mattermost/mattermost/server/v8/channels/app"
-	"github.com/mattermost/mattermost/server/v8/channels/store"
-	"github.com/mattermost/mattermost/server/v8/channels/utils/fileutils"
 	"github.com/mattermost/mattermost/server/v8/einterfaces"
 	ejobs "github.com/mattermost/mattermost/server/v8/einterfaces/jobs"
-	"github.com/mattermost/mattermost/server/v8/platform/shared/filestore"
-	"github.com/mattermost/mattermost/server/v8/platform/shared/templates"
-
 	"github.com/mattermost/mattermost/server/v8/enterprise/message_export/actiance_export"
 	"github.com/mattermost/mattermost/server/v8/enterprise/message_export/csv_export"
 	"github.com/mattermost/mattermost/server/v8/enterprise/message_export/global_relay_export"
+	"github.com/mattermost/mattermost/server/v8/enterprise/message_export/shared"
 )
 
-const (
-	GlobalRelayExportFilename = "global-relay.zip"
-)
+const GlobalRelayExportFilename = "global-relay.zip"
 
 type MessageExportInterfaceImpl struct {
 	Server *app.Server
@@ -56,7 +44,7 @@ func (m *MessageExportInterfaceImpl) StartSynchronizeJob(rctx request.CTX, expor
 	// if a valid export time was specified, put it in the job data
 	jobData := make(map[string]string)
 	if exportFromTimestamp >= 0 {
-		jobData[JobDataBatchStartTimestamp] = strconv.FormatInt(exportFromTimestamp, 10)
+		jobData[shared.JobDataBatchStartTime] = strconv.FormatInt(exportFromTimestamp, 10)
 	}
 
 	// passing nil for job data will cause the worker to inherit start time from previously successful job
@@ -85,32 +73,137 @@ func (m *MessageExportInterfaceImpl) StartSynchronizeJob(rctx request.CTX, expor
 	return job, nil
 }
 
-func (m *MessageExportInterfaceImpl) RunExport(rctx request.CTX, exportType string, since int64, limit int) (warningCount int64, appErr *model.AppError) {
-	if limit < 0 {
-		limit = math.MaxInt64
-	}
-	postsToExport, _, err := m.Server.Store().Compliance().MessageExport(rctx, model.MessageExportCursor{LastPostUpdateAt: since}, limit)
+func RunBatch(rctx request.CTX, data shared.JobData, params shared.BackendParams) (shared.RunExportResults, shared.JobData, error) {
+	start := time.Now()
+	var err error
+	var res shared.RunExportResults
+	data, err = GetDataForBatch(rctx, data, params)
 	if err != nil {
-		return warningCount, model.NewAppError("RunExport", "ent.message_export.run_export.app_error", nil, "", http.StatusInternalServerError).Wrap(err)
-	}
-	rctx.Logger().Debug("Found posts to export", mlog.Int("number_of_posts", len(postsToExport)))
-
-	fileBackend := m.Server.FileBackend()
-	templatesDir, ok := fileutils.FindDir("templates")
-	if !ok {
-		return warningCount, model.NewAppError("RunExport", "ent.compliance.run_export.template_watcher.appError", nil, "", http.StatusAccepted)
+		return res, data, err
 	}
 
-	t, err2 := templates.New(templatesDir)
-	if err2 != nil {
-		return warningCount, model.NewAppError("RunExport", "ent.compliance.run_export.template_watcher.appError", nil, "", http.StatusAccepted).Wrap(err2)
+	if data.Finished {
+		return res, data, nil
 	}
 
-	exportDirectory := getOutputDirectoryPath(since, model.GetMillis())
-	return runExportByType(rctx, exportType, postsToExport, exportDirectory, m.Server.Store(), fileBackend, fileBackend, t, m.Server.Config())
+	// Now write the data to the export type.
+	res, err = RunExportByType(rctx, DataToExportParams(data), params)
+	if err != nil {
+		return res, data, err
+	}
+
+	data.ProcessingPostsMs = append(data.ProcessingPostsMs, res.ProcessingPostsMs)
+	data.ProcessingXmlMs = append(data.ProcessingXmlMs, res.ProcessingXmlMs)
+	data.TransferringFilesMs = append(data.TransferringFilesMs, res.TransferringFilesMs)
+	data.TransferringZipMs = append(data.TransferringZipMs, res.TransferringZipMs)
+	data.TotalBatchMs = append(data.TotalBatchMs, time.Since(start).Milliseconds())
+	data.WarningCount += res.NumWarnings
+	data.BatchStartTime = data.BatchEndTime
+
+	return res, data, err
 }
 
-func runExportByType(rctx request.CTX, exportType string, postsToExport []*model.MessageExport, exportDirectory string, db store.Store, exportBackend filestore.FileBackend, fileAttachmentBackend filestore.FileBackend, htmlTemplates *templates.Container, config *model.Config) (warningCount int64, appErr *model.AppError) {
+// GetDataForBatch gets the posts for this batch and updates JobData with the current state.
+func GetDataForBatch(rctx request.CTX, data shared.JobData, params shared.BackendParams) (shared.JobData, error) {
+	start := time.Now()
+	var err error
+	// Using BatchSize+1 is a trick to test whether or not we've reached the final batch.
+	data.PostsToExport, data.Cursor, err = params.Store.Compliance().MessageExport(rctx, data.Cursor, data.BatchSize+1)
+	if err != nil {
+		return data, err
+	}
+	data.MessageExportMs = append(data.MessageExportMs, time.Since(start).Milliseconds())
+
+	if len(data.PostsToExport) == data.BatchSize+1 {
+		// We still have posts after this current batch.
+		// Remove the last post, we have to leave it for the next batch.
+		lastPostIdx := len(data.PostsToExport) - 1
+		data.PostsToExport = data.PostsToExport[:lastPostIdx]
+		lastPostIdx = len(data.PostsToExport) - 1
+		data.Cursor.LastPostUpdateAt = *data.PostsToExport[lastPostIdx].PostUpdateAt
+		data.Cursor.LastPostId = *data.PostsToExport[lastPostIdx].PostId
+		data.BatchEndTime = data.Cursor.LastPostUpdateAt
+	} else {
+		// We've reached the final batch; we need to include all join/leave events that occur after the lastpost.
+		// This will let us also pick up the joins/leaves that occur after lastPostUpdateAt but before JobEndTime.
+		data.BatchEndTime = data.JobEndTime
+	}
+
+	if len(data.PostsToExport) == 0 {
+		data.Finished = true
+		return data, nil
+	}
+
+	rctx.Logger().Debug("Found posts to export", mlog.Int("num_posts", len(data.PostsToExport)))
+	data.MessagesExported += len(data.PostsToExport)
+	data.BatchNumber++
+	data.BatchPath = shared.GetBatchPath(data.ExportDir, data.BatchStartTime, data.BatchEndTime, data.BatchNumber)
+
+	return data, nil
+}
+
+type ExportParams struct {
+	ExportType             string
+	ChannelMetadata        map[string]*shared.MetadataChannel
+	ChannelMemberHistories map[string][]*model.ChannelMemberHistoryResult
+	PostsToExport          []*model.MessageExport
+	JobStartTime           int64
+	BatchPath              string
+	BatchStartTime         int64
+	BatchEndTime           int64
+}
+
+func DataToExportParams(data shared.JobData) ExportParams {
+	return ExportParams{
+		ExportType:             data.ExportType,
+		ChannelMetadata:        data.ChannelMetadata,
+		ChannelMemberHistories: data.ChannelMemberHistories,
+		PostsToExport:          data.PostsToExport,
+		JobStartTime:           data.JobStartTime,
+		BatchPath:              data.BatchPath,
+		BatchStartTime:         data.BatchStartTime,
+		BatchEndTime:           data.BatchEndTime,
+	}
+}
+
+func RunExportByType(rctx request.CTX, p ExportParams, b shared.BackendParams) (results shared.RunExportResults, err error) {
+	preparePosts(rctx, p.PostsToExport)
+
+	exportParams := shared.ExportParams{
+		ExportType:             p.ExportType,
+		ChannelMetadata:        p.ChannelMetadata,
+		Posts:                  p.PostsToExport,
+		ChannelMemberHistories: p.ChannelMemberHistories,
+		JobStartTime:           p.JobStartTime,
+		BatchPath:              p.BatchPath,
+		BatchStartTime:         p.BatchStartTime,
+		BatchEndTime:           p.BatchEndTime,
+		Config:                 b.Config,
+		Db:                     b.Store,
+		FileAttachmentBackend:  b.FileAttachmentBackend,
+		ExportBackend:          b.ExportBackend,
+		Templates:              b.HtmlTemplates,
+	}
+
+	switch p.ExportType {
+	case model.ComplianceExportTypeCsv:
+		rctx.Logger().Debug("Exporting CSV")
+		return csv_export.CsvExport(rctx, exportParams)
+
+	case model.ComplianceExportTypeActiance:
+		rctx.Logger().Debug("Exporting Actiance")
+		return actiance_export.ActianceExport(rctx, exportParams)
+
+	case model.ComplianceExportTypeGlobalrelay, model.ComplianceExportTypeGlobalrelayZip:
+		rctx.Logger().Debug("Exporting GlobalRelay")
+		return global_relay_export.GlobalRelayExport(rctx, exportParams)
+
+	default:
+		return results, errors.New("Unknown output format: " + p.ExportType)
+	}
+}
+
+func preparePosts(rctx request.CTX, postsToExport []*model.MessageExport) {
 	// go through all the posts and if the post's props contain 'from_bot' - override the IsBot field, since it's possible that the sender is not a user, but was a Bot and vise-versa
 	for _, post := range postsToExport {
 		if post.PostProps != nil {
@@ -178,57 +271,4 @@ func runExportByType(rctx request.CTX, exportType string, postsToExport []*model
 			post.PostCreateAt = new(int64)
 		}
 	}
-
-	switch exportType {
-	case model.ComplianceExportTypeCsv:
-		rctx.Logger().Debug("Exporting CSV")
-		return csv_export.CsvExport(rctx, postsToExport, db, exportBackend, fileAttachmentBackend, exportDirectory)
-
-	case model.ComplianceExportTypeActiance:
-		rctx.Logger().Debug("Exporting Actiance")
-		return actiance_export.ActianceExport(rctx, postsToExport, db, exportBackend, fileAttachmentBackend, exportDirectory)
-
-	case model.ComplianceExportTypeGlobalrelay, model.ComplianceExportTypeGlobalrelayZip:
-		rctx.Logger().Debug("Exporting GlobalRelay")
-		f, err := os.CreateTemp("", "")
-		if err != nil {
-			return warningCount, model.NewAppError("RunExport", "ent.compliance.global_relay.open_temporary_file.appError", nil, "", http.StatusAccepted).Wrap(err)
-		}
-		defer f.Close()
-		defer os.Remove(f.Name())
-
-		attachmentsRemovedPostIDs, warnings, appErr := global_relay_export.GlobalRelayExport(rctx, postsToExport, db, fileAttachmentBackend, f, htmlTemplates)
-		if appErr != nil {
-			return warningCount, appErr
-		}
-		warningCount = warnings
-		_, err = f.Seek(0, 0)
-		if err != nil {
-			return warningCount, model.NewAppError("RunExport", "ent.compliance.global_relay.rewind_temporary_file.appError", nil, "", http.StatusAccepted).Wrap(err)
-		}
-
-		if exportType == model.ComplianceExportTypeGlobalrelayZip {
-			// Try to disable the write timeout for the potentially big export file.
-			_, nErr := filestore.TryWriteFileContext(rctx.Context(), exportBackend, f, path.Join(exportDirectory, GlobalRelayExportFilename))
-			if nErr != nil {
-				return warningCount, model.NewAppError("runExportByType", "ent.compliance.global_relay.write_file.appError", nil, "", http.StatusInternalServerError).Wrap(nErr)
-			}
-		} else {
-			appErr = global_relay_export.Deliver(f, config)
-			if appErr != nil {
-				return warningCount, appErr
-			}
-		}
-
-		if len(attachmentsRemovedPostIDs) > 0 {
-			rctx.Logger().Debug("Global Relay Attachments Removed because they were too large to send to Global Relay", mlog.Array("attachment_ids", attachmentsRemovedPostIDs))
-			description := fmt.Sprintf("Attachments to post IDs %v were removed because they were too large to send to Global Relay.", attachmentsRemovedPostIDs)
-			appErr = model.NewAppError("RunExport", "ent.compliance.global_relay.attachments_removed.appError", map[string]any{"Description": description}, description, http.StatusAccepted)
-			return warningCount, appErr
-		}
-	default:
-		err := errors.New("Unknown output format " + exportType)
-		return warningCount, model.NewAppError("RunExport", "ent.compliance.bad_export_type.appError", map[string]any{"ExportType": exportType}, "", http.StatusBadRequest).Wrap(err)
-	}
-	return warningCount, nil
 }