MM-59966 - Compliance Export overhaul - feature branch (#29789)

* [MM-59089] Add a compliance export constant (#27919)

* add a useful constant

* i18n

* another constant

* another i18n

* [MM-60422] Add GetChannelsWithActivityDuring (#28301)

* modify GetUsersInChannelDuring to accept a slice of channelIds

* add GetChannelsWithActivityDuring

* add compliance export progress message; remove unused custom status

* linting

* tests running too fast

* add batch size config settings

* add store tests

* linting

* empty commit

* i18n changes

* fix i18n ordering

* MM-60570 - Server-side changes consolidating the export CLI with server/ent code (#28640)

* add an i18n field; add the CLI's export directory

* int64 -> int

* Add UntilUpdateAt for MessageExport and AnalyticsPostCount

to merge

* remove now-unused i18n strings

* add TranslationsPreInitFromBuffer to allow CLI to use i18n

* use GetBuilder to simplify; rename TranslationsPreInitFromFileBytes

* [MM-59089] Improve compliance export timings (#1733 - Enterprise repo)

* MM-60422 - Performance and logic fixes for Compliance Exports (#1757 - Enterprise repo)

* MM-60570 - Enterprise-side changes consolidating the export CLI with server/ent code (#1769 - Enterprise repo)

* merge conflicts; missed file from ent branch

* MM-61038 - Add an option to sqlstore.New (#28702)

remove useless comment

add test

add an option to sqlstore.New

* MM-60976: Remove RunExport command from Mattermost binary (#28805)

* remove RunExport command from mattermost binary

* remove the code it was calling

* fix i18n

* remove test (was only testing license, not functionality)

* empty commit

* fix flaky GetChannelsWithActivityDuring test

* MM-60063: Dedicated Export Filestore fix, redo of #1772 (enterprise) (#28803)

* redo filestore fix #1772 (enterprise repo) on top of MM-59966 feature

* add new e2e tests for export filestore

* golint

* ok, note to self: shadowing bad, actually (when there's a defer)

* empty commit

* MM-61137 - Message export: Support 7.8.11 era dbs (#28824)

* support 7.8.11 era dbs by wrapping the store using only what we need

* fix flaky GetChannelsWithActivityDuring test

* add a comment

* only need to define the MEFileInfoStore (the one that'll be overridden)

* blank commit

* MM-60974 - Message Export: Add performance metrics (#28836)

* support 7.8.11 era dbs by wrapping the store using only what we need

* fix flaky GetChannelsWithActivityDuring test

* add a comment

* only need to define the MEFileInfoStore (the one that'll be overridden)

* performance metrics

* cleanup unneeded named returns

* blank commit

* MM-60975 - Message export: Add startTime and endTime to export folder name (#28840)

* support 7.8.11 era dbs by wrapping the store using only what we need

* fix flaky GetChannelsWithActivityDuring test

* add a comment

* only need to define the MEFileInfoStore (the one that'll be overridden)

* performance metrics

* output startTime and endTime in export folder

* empty commit

* merge conflict

* MM-60978 - Message export: Improve xml fields; fix delete semantics (#28873)

* support 7.8.11 era dbs by wrapping the store using only what we need

* fix flaky GetChannelsWithActivityDuring test

* add a comment

* only need to define the MEFileInfoStore (the one that'll be overridden)

* performance metrics

* output startTime and endTime in export folder

* empty commit

* add xml fields, omit when empty, tests

* fix delete semantics; test (and test for update semantics)

* clarify comments

* simplify edited post detection, now there's no edge case.

* add some spacing to help fast running tests

* merge conflicts/updates needed for new deleted post semantics

* linting; fixing tests from upstream merge

* use SafeDereference

* linting

* stronger typing; better wrapped errors; better formatting

* blank commit

* goimports formatting

* fix merge mistake

* minor fixes due to changes in master

* MM-61755 - Simplifying and Support reporting to the db from the CLI (#29281)

* finally clean up JobData struct and stringMap; prep for CLI using db

* and now simplify using StringMapToJobDataWithZeroValues

* remove unused fn

* create JobDataExported; clean up errors

* MM-60176 - Message Export: Global relay cleanup (#29168)

* move global relay logic into global_relay_export

* blank commit

* blank commit

* improve errors

* MM-60693 - Refactor CSV to use same codepath as Actiance (#29191)

* move global relay logic into global_relay_export

* blank commit

* refactor (and simplify) ExportParams into shared

* blank commit

* remove unused fn

* csv now uses pre-calculated joins/leaves like actiance

* improve errors

* remove nil post check; remove ignoredPosts metric

* remove unneeded copy

* MM-61696 - Refactor GlobalRelay to use same codepath as Actiance (#29225)

* move global relay logic into global_relay_export

* blank commit

* refactor (and simplify) ExportParams into shared

* blank commit

* remove unused fn

* csv now uses pre-calculated joins/leaves like actiance

* remove newly unneeded function and its test. goodbye.

* refactor GetPostAttachments for csv + global relay to share

* refactor global_relay_export and fix tests (no changes to output)

* improve errors

* remove nil post check; remove ignoredPosts metric

* remove unneeded copy

* remove unneeded nil check

* PR comments

* MM-61715 - Generalize e2e to all export types 🤖  (#29369)

* move global relay logic into global_relay_export

* blank commit

* refactor (and simplify) ExportParams into shared

* blank commit

* remove unused fn

* csv now uses pre-calculated joins/leaves like actiance

* remove newly unneeded function and its test. goodbye.

* refactor GetPostAttachments for csv + global relay to share

* refactor global_relay_export and fix tests (no changes to output)

* improve errors

* remove nil post check; remove ignoredPosts metric

* remove unneeded copy

* remove unneeded nil check

* PR comments

* refactor isDeletedMsg for all export types

* fix start and endtime, nasty csv createAt bug; bring closer to Actiance

* align unit tests with new logic (e.g. starttime / endtime)

* refactor a TimestampConvert fn for code + tests

* bug: pass templates to global relay (hurray for e2e tests, otherwise...)

* add global relay zip to allowed list (only for tests)

* test helpers

* new templates for e2e tests

* e2e tests... phew.

* linting

* merge conflicts

* unexport PostToRow; add test helper marker

* cleanup, shortening, thanks to PR comments

* MM-61972 - Generalize export data path - Actiance (#29399)

* extract and generalize the export data generation functions

* finish moving test (bc of previous extraction)

* lift a function from common -> shared (to break an import cycle)

* actiance now takes general export data, processes it into actiance data

* bring tests in line with correct sorting rules (upadateAt, messageId)

* fixups, PR comments

* turn strings.Repeat into a more descriptive const

amended: one letter fix; bad rebase

* MM-62009 - e2e clock heisenbug (#29434)

* consolidate assertions; output debuggable diffs (keeping for future)

* refactor test output generator to generators file

* waitUntilZeroPosts + pass through until to job = fix all clock issues

* simplify messages to model.NewId(); remove unneeded waitUntilZeroPosts

* model.NewId() -> storetest.NewTestID()

* MM-61980 - Generalize export data path - CSV (#29482)

* simple refactoring

* increase sleep times for (very) rare test failures

* add extra information to the generic export for CSV

* adj Actiance to handle new generic export (no difference in its output)

* no longer need mergePosts (yay), move getJoinLeavePosts for everyone

* adjust tests for new csv semantics (detailed in summary)

* and need to add the new exported data to the export_data_tests

* rearrange csv writing to happen after data export (more logical)

* linting

* remove debug statements

* figured out what was wrong with global relay e2e test 3; solid now

* PR comments

* MM-61718 - Generalize export data path - Global Relay (#29508)

* move global relay over to using the generalized export data

* performance pass -- not much can be done

* Update server/enterprise/message_export/global_relay_export/global_relay_export.go

Co-authored-by: Claudio Costa <cstcld91@gmail.com>

---------

Co-authored-by: Claudio Costa <cstcld91@gmail.com>

* MM-62058 - Align CSV with Actiance (#29551)

* refactoring actiance files and var names for clarity

* bug found in exported attachments (we used to miss some start/ends)

* changes needed for actiance due to new generic exports

* bringing CSV up to actiance standards

* fixing global relay b/c of new semantics (adding a note on an edge case)

* aligning e2e tests, adding comments to clarify what is expected/tested

* necessary changes; 1 more test for added functionality (ignoreDeleted)

* comment style

* MM-62059 - Align Global Relay with Actiance/CSV; many fixes (#29665)

* core logic changes to general export_data and the specific export paths

* unit tests and e2e tests, covering all new edge cases and all logic

* linting

* better var naming, const value, and cleaning up functions calls

* MM-62436 - Temporarily skip cypress tests that require download link (#29772)

---------

Co-authored-by: Claudio Costa <cstcld91@gmail.com>

e2e-tests/cypress/tests/integration/channels/enterprise/system_console/compliance/compliance_export_ui_spec.js

@@ -44,7 +44,7 @@ describe('Compliance Export', () => {
         });
     });
 
-    it('MM-T3435 - Download Compliance Export Files - CSV Format', () => {
+    it.skip('MM-60115 - MM-T3435 - Download Compliance Export Files - CSV Format', () => {
         // # Navigate to a team and post an attachment
         cy.visit(`/${teamName}/channels/town-square`);
         gotoTeamAndPostImage();
@@ -65,7 +65,7 @@ describe('Compliance Export', () => {
         });
     });
 
-    it('MM-T3438 - Download Compliance Export Files when 0 messages exported', () => {
+    it.skip('MM-60115 - MM-T3438 - Download Compliance Export Files when 0 messages exported', () => {
         // # Navigate to a team and post an attachment
         cy.visit(`/${teamName}/channels/town-square`);
         gotoTeamAndPostImage();
@@ -94,7 +94,7 @@ describe('Compliance Export', () => {
         });
     });
 
-    it('MM-T1168 - Compliance Export - Run Now, entry appears in job table', () => {
+    it.skip('MM-60115 - MM-T1168 - Compliance Export - Run Now, entry appears in job table', () => {
         // # Navigate to a team and post an attachment
         cy.visit(`/${teamName}/channels/town-square`);
         gotoTeamAndPostImage();

e2e-tests/cypress/tests/integration/channels/enterprise/system_console/compliance/download_bot_compliance_export_file_spec.js

@@ -60,7 +60,7 @@ describe('Compliance Export', () => {
         cy.shellRm('-rf', downloadsFolder);
     });
 
-    it('MM-T1175_1 - UserType identifies that the message is posted by a bot', () => {
+    it.skip('MM-60115 - MM-T1175_1 - UserType identifies that the message is posted by a bot', () => {
         const message = `This is CSV bot message from ${botName} at ${Date.now()}`;
 
         // # Post bot message
@@ -81,7 +81,7 @@ describe('Compliance Export', () => {
         );
     });
 
-    it('MM-T1175_2 - UserType identifies that the message is posted by a bot', () => {
+    it.skip('MM-60115 - MM-T1175_2 - UserType identifies that the message is posted by a bot', () => {
         const message = `This is XML bot message from ${botName} at ${Date.now()}`;
 
         // # Post bot message

e2e-tests/cypress/tests/integration/channels/enterprise/system_console/compliance/download_compliance_export_file_spec.js

@@ -52,7 +52,7 @@ describe('Compliance Export', () => {
         cy.shellRm('-rf', downloadsFolder);
     });
 
-    it('MM-T1172 - Compliance Export - Deleted file is indicated in CSV File Export', () => {
+    it.skip('MM-60115 - MM-T1172 - Compliance Export - Deleted file is indicated in CSV File Export', () => {
         // # Go to compliance page and enable export
         cy.uiGoToCompliancePage();
         cy.uiEnableComplianceExport();
@@ -84,7 +84,7 @@ describe('Compliance Export', () => {
         );
     });
 
-    it('MM-T1173 - Compliance Export - Deleted file is indicated in Actiance XML File Export', () => {
+    it.skip('MM-60115 - MM-T1173 - Compliance Export - Deleted file is indicated in Actiance XML File Export', () => {
         // # Go to compliance page and enable export
         cy.uiGoToCompliancePage();
         cy.uiEnableComplianceExport(ExportFormatActiance);
@@ -121,7 +121,7 @@ describe('Compliance Export', () => {
         });
     });
 
-    it('MM-T1176 - Compliance export should include updated post after editing', () => {
+    it.skip('MM-60115 - MM-T1176 - Compliance export should include updated post after editing', () => {
         // # Go to compliance page and enable export
         cy.uiGoToCompliancePage();
         cy.uiEnableComplianceExport(ExportFormatActiance);
@@ -154,7 +154,7 @@ describe('Compliance Export', () => {
         );
     });
 
-    it('MM-T3305 - Verify Deactivated users are displayed properly in Compliance Exports', () => {
+    it.skip('MM-60115 - MM-T3305 - Verify Deactivated users are displayed properly in Compliance Exports', () => {
         // # Post a message by Admin
         cy.postMessageAs({
             sender: adminUser,

server/channels/app/channel_test.go

@@ -298,7 +298,7 @@ func TestJoinDefaultChannelsCreatesChannelMemberHistoryRecordTownSquare(t *testi
 	channel, err := th.App.Srv().Store().Channel().GetByName(th.BasicTeam.Id, "town-square", true)
 	require.NoError(t, err)
 	townSquareChannelID := channel.Id
-	users, nErr := th.App.Srv().Store().ChannelMemberHistory().GetUsersInChannelDuring(model.GetMillis()-100, model.GetMillis()+100, townSquareChannelID)
+	users, nErr := th.App.Srv().Store().ChannelMemberHistory().GetUsersInChannelDuring(model.GetMillis()-100, model.GetMillis()+100, []string{townSquareChannelID})
 	require.NoError(t, nErr)
 	initialNumTownSquareUsers := len(users)
 
@@ -308,7 +308,7 @@ func TestJoinDefaultChannelsCreatesChannelMemberHistoryRecordTownSquare(t *testi
 	require.Nil(t, appErr)
 
 	// there should be a ChannelMemberHistory record for the user
-	histories, nErr := th.App.Srv().Store().ChannelMemberHistory().GetUsersInChannelDuring(model.GetMillis()-100, model.GetMillis()+100, townSquareChannelID)
+	histories, nErr := th.App.Srv().Store().ChannelMemberHistory().GetUsersInChannelDuring(model.GetMillis()-100, model.GetMillis()+100, []string{townSquareChannelID})
 	require.NoError(t, nErr)
 	assert.Len(t, histories, initialNumTownSquareUsers+1)
 
@@ -330,7 +330,7 @@ func TestJoinDefaultChannelsCreatesChannelMemberHistoryRecordOffTopic(t *testing
 	channel, err := th.App.Srv().Store().Channel().GetByName(th.BasicTeam.Id, "off-topic", true)
 	require.NoError(t, err)
 	offTopicChannelId := channel.Id
-	users, nErr := th.App.Srv().Store().ChannelMemberHistory().GetUsersInChannelDuring(model.GetMillis()-100, model.GetMillis()+100, offTopicChannelId)
+	users, nErr := th.App.Srv().Store().ChannelMemberHistory().GetUsersInChannelDuring(model.GetMillis()-100, model.GetMillis()+100, []string{offTopicChannelId})
 	require.NoError(t, nErr)
 	initialNumTownSquareUsers := len(users)
 
@@ -340,7 +340,7 @@ func TestJoinDefaultChannelsCreatesChannelMemberHistoryRecordOffTopic(t *testing
 	require.Nil(t, appError)
 
 	// there should be a ChannelMemberHistory record for the user
-	histories, nErr := th.App.Srv().Store().ChannelMemberHistory().GetUsersInChannelDuring(model.GetMillis()-100, model.GetMillis()+100, offTopicChannelId)
+	histories, nErr := th.App.Srv().Store().ChannelMemberHistory().GetUsersInChannelDuring(model.GetMillis()-100, model.GetMillis()+100, []string{offTopicChannelId})
 	require.NoError(t, nErr)
 	assert.Len(t, histories, initialNumTownSquareUsers+1)
 
@@ -418,7 +418,7 @@ func TestCreateChannelPublicCreatesChannelMemberHistoryRecord(t *testing.T) {
 	publicChannel := th.createChannel(th.Context, th.BasicTeam, model.ChannelTypeOpen)
 
 	// there should be a ChannelMemberHistory record for the user
-	histories, err := th.App.Srv().Store().ChannelMemberHistory().GetUsersInChannelDuring(model.GetMillis()-100, model.GetMillis()+100, publicChannel.Id)
+	histories, err := th.App.Srv().Store().ChannelMemberHistory().GetUsersInChannelDuring(model.GetMillis()-100, model.GetMillis()+100, []string{publicChannel.Id})
 	require.NoError(t, err)
 	assert.Len(t, histories, 1)
 	assert.Equal(t, th.BasicUser.Id, histories[0].UserId)
@@ -433,7 +433,7 @@ func TestCreateChannelPrivateCreatesChannelMemberHistoryRecord(t *testing.T) {
 	privateChannel := th.createChannel(th.Context, th.BasicTeam, model.ChannelTypePrivate)
 
 	// there should be a ChannelMemberHistory record for the user
-	histories, err := th.App.Srv().Store().ChannelMemberHistory().GetUsersInChannelDuring(model.GetMillis()-100, model.GetMillis()+100, privateChannel.Id)
+	histories, err := th.App.Srv().Store().ChannelMemberHistory().GetUsersInChannelDuring(model.GetMillis()-100, model.GetMillis()+100, []string{privateChannel.Id})
 	require.NoError(t, err)
 	assert.Len(t, histories, 1)
 	assert.Equal(t, th.BasicUser.Id, histories[0].UserId)
@@ -557,7 +557,7 @@ func TestCreateGroupChannelCreatesChannelMemberHistoryRecord(t *testing.T) {
 	channel, appErr := th.App.CreateGroupChannel(th.Context, groupUserIds, th.BasicUser.Id)
 
 	require.Nil(t, appErr, "Failed to create group channel.")
-	histories, nErr := th.App.Srv().Store().ChannelMemberHistory().GetUsersInChannelDuring(model.GetMillis()-100, model.GetMillis()+100, channel.Id)
+	histories, nErr := th.App.Srv().Store().ChannelMemberHistory().GetUsersInChannelDuring(model.GetMillis()-100, model.GetMillis()+100, []string{channel.Id})
 	require.NoError(t, nErr)
 	assert.Len(t, histories, 3)
 
@@ -582,7 +582,7 @@ func TestCreateDirectChannelCreatesChannelMemberHistoryRecord(t *testing.T) {
 	channel, appErr := th.App.GetOrCreateDirectChannel(th.Context, user1.Id, user2.Id)
 	require.Nil(t, appErr, "Failed to create direct channel.")
 
-	histories, nErr := th.App.Srv().Store().ChannelMemberHistory().GetUsersInChannelDuring(model.GetMillis()-100, model.GetMillis()+100, channel.Id)
+	histories, nErr := th.App.Srv().Store().ChannelMemberHistory().GetUsersInChannelDuring(model.GetMillis()-100, model.GetMillis()+100, []string{channel.Id})
 	require.NoError(t, nErr)
 	assert.Len(t, histories, 2)
 
@@ -610,7 +610,7 @@ func TestGetDirectChannelCreatesChannelMemberHistoryRecord(t *testing.T) {
 	require.Nil(t, appErr, "Failed to create direct channel.")
 
 	// there should be a ChannelMemberHistory record for both users
-	histories, nErr := th.App.Srv().Store().ChannelMemberHistory().GetUsersInChannelDuring(model.GetMillis()-100, model.GetMillis()+100, channel.Id)
+	histories, nErr := th.App.Srv().Store().ChannelMemberHistory().GetUsersInChannelDuring(model.GetMillis()-100, model.GetMillis()+100, []string{channel.Id})
 	require.NoError(t, nErr)
 	assert.Len(t, histories, 2)
 
@@ -645,7 +645,7 @@ func TestAddUserToChannelCreatesChannelMemberHistoryRecord(t *testing.T) {
 	require.Nil(t, appErr, "Failed to add user to channel.")
 
 	// there should be a ChannelMemberHistory record for the user
-	histories, nErr := th.App.Srv().Store().ChannelMemberHistory().GetUsersInChannelDuring(model.GetMillis()-100, model.GetMillis()+100, channel.Id)
+	histories, nErr := th.App.Srv().Store().ChannelMemberHistory().GetUsersInChannelDuring(model.GetMillis()-100, model.GetMillis()+100, []string{channel.Id})
 	require.NoError(t, nErr)
 	assert.Len(t, histories, 2)
 	channelMemberHistoryUserIds := make([]string, 0)
@@ -656,6 +656,88 @@ func TestAddUserToChannelCreatesChannelMemberHistoryRecord(t *testing.T) {
 	assert.Equal(t, groupUserIds, channelMemberHistoryUserIds)
 }
 
+func TestUsersAndPostsCreateActivityInChannel(t *testing.T) {
+	th := Setup(t).InitBasic().DeleteBots()
+	defer th.TearDown()
+
+	user := th.CreateUser()
+	_, err := th.App.AddTeamMember(th.Context, th.BasicTeam.Id, user.Id)
+	require.Nil(t, err, "Failed to add user to team.")
+	user3 := th.CreateUser()
+	_, err = th.App.AddTeamMember(th.Context, th.BasicTeam.Id, user3.Id)
+	require.Nil(t, err, "Failed to add user to team.")
+	user4 := th.CreateUser()
+	_, err = th.App.AddTeamMember(th.Context, th.BasicTeam.Id, user4.Id)
+	require.Nil(t, err, "Failed to add user to team.")
+
+	channel1 := th.createChannel(th.Context, th.BasicTeam, model.ChannelTypeOpen)
+	channel2 := th.createChannel(th.Context, th.BasicTeam, model.ChannelTypeOpen)
+	channel3 := th.createChannel(th.Context, th.BasicTeam, model.ChannelTypeOpen)
+	channel4 := th.createChannel(th.Context, th.BasicTeam, model.ChannelTypeOpen)
+	channel5 := th.createChannel(th.Context, th.BasicTeam, model.ChannelTypeOpen)
+	channel6 := th.createChannel(th.Context, th.BasicTeam, model.ChannelTypeOpen)
+
+	// user3 is already in channel3
+	_, err = th.App.AddUserToChannel(th.Context, user3, channel3, false)
+	require.Nil(t, err, "Failed to add user to channel.")
+	// user4 is already in channel4 (for the second part of the test)
+	_, err = th.App.AddUserToChannel(th.Context, user4, channel4, false)
+	require.Nil(t, err, "Failed to add user to channel.")
+
+	// make sure we don't catch earlier posts
+	time.Sleep(10 * time.Millisecond)
+	testStart := model.GetMillis()
+
+	// Test: previous activity (user3 and 4's adds) aren't showing up:
+	channelIds, nErr := th.App.Srv().Store().ChannelMemberHistory().GetChannelsWithActivityDuring(testStart, testStart+10000)
+	require.NoError(t, nErr)
+	assert.Len(t, channelIds, 0)
+
+	// Posts, adds, and leaves should create activity
+	post := &model.Post{
+		ChannelId: channel1.Id,
+		Message:   "root post",
+		UserId:    th.BasicUser.Id,
+	}
+	_, err = th.App.CreatePost(th.Context, post, channel1, model.CreatePostFlags{})
+	require.Nil(t, err, "Failed to create post.")
+
+	_, err = th.App.AddUserToChannel(th.Context, user, channel2, false)
+	require.Nil(t, err, "Failed to add user to channel.")
+
+	err = th.App.RemoveUserFromChannel(th.Context, user3.Id, user3.Id, channel3)
+	require.Nil(t, err, "Failed to add user to channel.")
+
+	// Test: there should be a ChannelMemberHistory record for the users and the post
+	channelIds, nErr = th.App.Srv().Store().ChannelMemberHistory().GetChannelsWithActivityDuring(testStart, model.GetMillis())
+	require.NoError(t, nErr)
+	assert.Len(t, channelIds, 3)
+	assert.ElementsMatch(t, []string{channel1.Id, channel2.Id, channel3.Id}, channelIds)
+
+	testEnd := model.GetMillis()
+	// In case the tests are running very fast:
+	time.Sleep(10 * time.Millisecond)
+
+	// Now, we do not find activity for new posts, leaves, or adds after the test is over
+	post2 := &model.Post{
+		ChannelId: channel5.Id,
+		Message:   "root post",
+		UserId:    th.BasicUser.Id,
+	}
+	err = th.App.RemoveUserFromChannel(th.Context, user4.Id, user4.Id, channel4)
+	require.Nil(t, err, "Failed to create post.")
+	_, err = th.App.CreatePost(th.Context, post2, channel5, model.CreatePostFlags{})
+	require.Nil(t, err, "Failed to create post.")
+	_, err = th.App.AddUserToChannel(th.Context, user, channel6, false)
+	require.Nil(t, err, "Failed to add user to channel.")
+
+	// Test: we get the same three channels as before, not channels 4, 5, 6 which have activity after testEnd
+	channelIds, nErr = th.App.Srv().Store().ChannelMemberHistory().GetChannelsWithActivityDuring(testStart, testEnd)
+	require.NoError(t, nErr)
+	assert.Len(t, channelIds, 3)
+	assert.ElementsMatch(t, []string{channel1.Id, channel2.Id, channel3.Id}, channelIds)
+}
+
 func TestLeaveDefaultChannel(t *testing.T) {
 	th := Setup(t).InitBasic()
 	defer th.TearDown()
@@ -811,7 +893,7 @@ func TestAddChannelMemberNoUserRequestor(t *testing.T) {
 	require.Nil(t, appErr, "Failed to add user to channel.")
 
 	// there should be a ChannelMemberHistory record for the user
-	histories, nErr := th.App.Srv().Store().ChannelMemberHistory().GetUsersInChannelDuring(model.GetMillis()-100, model.GetMillis()+100, channel.Id)
+	histories, nErr := th.App.Srv().Store().ChannelMemberHistory().GetUsersInChannelDuring(model.GetMillis()-100, model.GetMillis()+100, []string{channel.Id})
 	require.NoError(t, nErr)
 	assert.Len(t, histories, 2)
 	channelMemberHistoryUserIds := make([]string, 0)

server/channels/app/file.go

@@ -103,7 +103,7 @@ func (a *App) TestFileStoreConnectionWithConfig(cfg *model.FileSettings) *model.
 		backend, err = filestore.NewFileBackend(filestore.NewFileBackendSettingsFromConfig(cfg, complianceEnabled, insecure != nil && *insecure))
 	}
 	if err != nil {
-		return model.NewAppError("FileBackend", "api.file.no_driver.app_error", nil, "", http.StatusInternalServerError).Wrap(err)
+		return model.NewAppError("FileAttachmentBackend", "api.file.no_driver.app_error", nil, "", http.StatusInternalServerError).Wrap(err)
 	}
 	nErr := backend.TestConnection()
 	if nErr != nil {

server/channels/store/opentracinglayer/opentracinglayer.go

@@ -2933,7 +2933,25 @@ func (s *OpenTracingLayerChannelMemberHistoryStore) GetChannelsLeftSince(userID
 	return result, err
 }
 
-func (s *OpenTracingLayerChannelMemberHistoryStore) GetUsersInChannelDuring(startTime int64, endTime int64, channelID string) ([]*model.ChannelMemberHistoryResult, error) {
+func (s *OpenTracingLayerChannelMemberHistoryStore) GetChannelsWithActivityDuring(startTime int64, endTime int64) ([]string, error) {
+	origCtx := s.Root.Store.Context()
+	span, newCtx := tracing.StartSpanWithParentByContext(s.Root.Store.Context(), "ChannelMemberHistoryStore.GetChannelsWithActivityDuring")
+	s.Root.Store.SetContext(newCtx)
+	defer func() {
+		s.Root.Store.SetContext(origCtx)
+	}()
+
+	defer span.Finish()
+	result, err := s.ChannelMemberHistoryStore.GetChannelsWithActivityDuring(startTime, endTime)
+	if err != nil {
+		span.LogFields(spanlog.Error(err))
+		ext.Error.Set(span, true)
+	}
+
+	return result, err
+}
+
+func (s *OpenTracingLayerChannelMemberHistoryStore) GetUsersInChannelDuring(startTime int64, endTime int64, channelID []string) ([]*model.ChannelMemberHistoryResult, error) {
 	origCtx := s.Root.Store.Context()
 	span, newCtx := tracing.StartSpanWithParentByContext(s.Root.Store.Context(), "ChannelMemberHistoryStore.GetUsersInChannelDuring")
 	s.Root.Store.SetContext(newCtx)

server/channels/store/retrylayer/retrylayer.go

@@ -3270,7 +3270,28 @@ func (s *RetryLayerChannelMemberHistoryStore) GetChannelsLeftSince(userID string
 
 }
 
-func (s *RetryLayerChannelMemberHistoryStore) GetUsersInChannelDuring(startTime int64, endTime int64, channelID string) ([]*model.ChannelMemberHistoryResult, error) {
+func (s *RetryLayerChannelMemberHistoryStore) GetChannelsWithActivityDuring(startTime int64, endTime int64) ([]string, error) {
+
+	tries := 0
+	for {
+		result, err := s.ChannelMemberHistoryStore.GetChannelsWithActivityDuring(startTime, endTime)
+		if err == nil {
+			return result, nil
+		}
+		if !isRepeatableError(err) {
+			return result, err
+		}
+		tries++
+		if tries >= 3 {
+			err = errors.Wrap(err, "giving up after 3 consecutive repeatable transaction failures")
+			return result, err
+		}
+		timepkg.Sleep(100 * timepkg.Millisecond)
+	}
+
+}
+
+func (s *RetryLayerChannelMemberHistoryStore) GetUsersInChannelDuring(startTime int64, endTime int64, channelID []string) ([]*model.ChannelMemberHistoryResult, error) {
 
 	tries := 0
 	for {

server/channels/store/sqlstore/channel_member_history_store.go

@@ -65,27 +65,78 @@ func (s SqlChannelMemberHistoryStore) LogLeaveEvent(userId string, channelId str
 	return nil
 }
 
-func (s SqlChannelMemberHistoryStore) GetUsersInChannelDuring(startTime int64, endTime int64, channelId string) ([]*model.ChannelMemberHistoryResult, error) {
+func (s SqlChannelMemberHistoryStore) GetChannelsWithActivityDuring(startTime int64, endTime int64) ([]string, error) {
+	// ChannelMemberHistory has been in production for long enough that we are assuming the export period
+	// starts after the ChannelMemberHistory table was first introduced
+	subqueryPosts := s.getSubQueryBuilder().
+		Select("p.ChannelId").
+		Distinct().
+		From("Posts AS p").
+		Where(
+			sq.And{
+				sq.GtOrEq{"p.UpdateAt": startTime},
+				sq.LtOrEq{"p.UpdateAt": endTime},
+				sq.NotLike{"p.Type": "system_%"},
+			})
+
+	subqueryCMH := s.getSubQueryBuilder().
+		Select("cmh.ChannelId").
+		Distinct().
+		From("ChannelMemberHistory AS cmh").
+		Where(
+			sq.Or{
+				sq.And{
+					sq.GtOrEq{"cmh.JoinTime": startTime},
+					sq.LtOrEq{"cmh.JoinTime": endTime},
+				},
+				sq.And{
+					sq.GtOrEq{"cmh.LeaveTime": startTime},
+					sq.LtOrEq{"cmh.LeaveTime": endTime},
+				},
+			})
+
+	unionExpr, args, err := sq.Expr("(? UNION ?) AS cm", subqueryPosts, subqueryCMH).ToSql()
+	if err != nil {
+		return nil, errors.Wrap(err, "GetChannelsWithActivityDuring unionExpr to sql")
+	}
+
+	// no bound args in this expression
+	query, _, err := s.getQueryBuilder().
+		Select("*").
+		From(unionExpr).ToSql()
+	if err != nil {
+		return nil, errors.Wrap(err, "GetChannelsWithActivityDuring query to sql")
+	}
+
+	channelIds := make([]string, 0)
+	if err := s.GetReplica().Select(&channelIds, query, args...); err != nil {
+		return nil, err
+	}
+
+	return channelIds, nil
+}
+
+func (s SqlChannelMemberHistoryStore) GetUsersInChannelDuring(startTime int64, endTime int64, channelIds []string) ([]*model.ChannelMemberHistoryResult, error) {
 	useChannelMemberHistory, err := s.hasDataAtOrBefore(startTime)
 	if err != nil {
-		return nil, errors.Wrapf(err, "hasDataAtOrBefore startTime=%d endTime=%d channelId=%s", startTime, endTime, channelId)
+		return nil, errors.Wrapf(err, "hasDataAtOrBefore startTime=%d endTime=%d channelId=%v", startTime, endTime, channelIds)
 	}
 
 	if useChannelMemberHistory {
 		// the export period starts after the ChannelMemberHistory table was first introduced, so we can use the
 		// data from it for our export
-		channelMemberHistories, err2 := s.getFromChannelMemberHistoryTable(startTime, endTime, channelId)
+		channelMemberHistories, err2 := s.getFromChannelMemberHistoryTable(startTime, endTime, channelIds)
 		if err2 != nil {
-			return nil, errors.Wrapf(err2, "getFromChannelMemberHistoryTable startTime=%d endTime=%d channelId=%s", startTime, endTime, channelId)
+			return nil, errors.Wrapf(err2, "getFromChannelMemberHistoryTable startTime=%d endTime=%d channelId=%v", startTime, endTime, channelIds)
 		}
 		return channelMemberHistories, nil
 	}
 	// the export period starts before the ChannelMemberHistory table was introduced, so we need to fake the
 	// data by assuming that anybody who has ever joined the channel in question was present during the export period.
 	// this may not always be true, but it's better than saying that somebody wasn't there when they were
-	channelMemberHistories, err := s.getFromChannelMembersTable(startTime, endTime, channelId)
+	channelMemberHistories, err := s.getFromChannelMembersTable(startTime, endTime, channelIds)
 	if err != nil {
-		return nil, errors.Wrapf(err, "getFromChannelMembersTable startTime=%d endTime=%d channelId=%s", startTime, endTime, channelId)
+		return nil, errors.Wrapf(err, "getFromChannelMembersTable startTime=%d endTime=%d channelId=%v", startTime, endTime, channelIds)
 	}
 	return channelMemberHistories, nil
 }
@@ -108,14 +159,14 @@ func (s SqlChannelMemberHistoryStore) hasDataAtOrBefore(time int64) (bool, error
 	return false, nil
 }
 
-func (s SqlChannelMemberHistoryStore) getFromChannelMemberHistoryTable(startTime int64, endTime int64, channelId string) ([]*model.ChannelMemberHistoryResult, error) {
+func (s SqlChannelMemberHistoryStore) getFromChannelMemberHistoryTable(startTime int64, endTime int64, channelIds []string) ([]*model.ChannelMemberHistoryResult, error) {
 	query, args, err := s.getQueryBuilder().
 		Select(`cmh.*, u.Email AS "Email", u.Username, Bots.UserId IS NOT NULL AS IsBot, u.DeleteAt AS UserDeleteAt`).
 		From("ChannelMemberHistory cmh").
 		Join("Users u ON cmh.UserId = u.Id").
 		LeftJoin("Bots ON Bots.UserId = u.Id").
 		Where(sq.And{
-			sq.Eq{"cmh.ChannelId": channelId},
+			sq.Eq{"cmh.ChannelId": channelIds},
 			sq.LtOrEq{"cmh.JoinTime": endTime},
 			sq.Or{
 				sq.Eq{"cmh.LeaveTime": nil},
@@ -134,14 +185,14 @@ func (s SqlChannelMemberHistoryStore) getFromChannelMemberHistoryTable(startTime
 	return histories, nil
 }
 
-func (s SqlChannelMemberHistoryStore) getFromChannelMembersTable(startTime int64, endTime int64, channelId string) ([]*model.ChannelMemberHistoryResult, error) {
+func (s SqlChannelMemberHistoryStore) getFromChannelMembersTable(startTime int64, endTime int64, channelIds []string) ([]*model.ChannelMemberHistoryResult, error) {
 	query, args, err := s.getQueryBuilder().
 		Select(`ch.ChannelId, ch.UserId, u.Email AS "Email", u.Username, Bots.UserId IS NOT NULL AS IsBot, u.DeleteAt AS UserDeleteAt`).
 		Distinct().
 		From("ChannelMembers ch").
 		Join("Users u ON ch.UserId = u.id").
 		LeftJoin("Bots ON Bots.UserId = u.id").
-		Where(sq.Eq{"ch.ChannelId": channelId}).ToSql()
+		Where(sq.Eq{"ch.ChannelId": channelIds}).ToSql()
 	if err != nil {
 		return nil, errors.Wrap(err, "channel_member_history_to_sql")
 	}

server/channels/store/sqlstore/compliance_store.go

@@ -286,24 +286,25 @@ func (s SqlComplianceStore) MessageExport(c request.CTX, cursor model.MessageExp
 		return nil, cursor, errors.Wrap(caseErr, "unable to construct case statement")
 	}
 
-	query, args, err := s.getQueryBuilder().Select(`Posts.Id AS PostId,
-			Posts.CreateAt AS PostCreateAt,
-			Posts.UpdateAt AS PostUpdateAt,
-			Posts.DeleteAt AS PostDeleteAt,
-			Posts.Message AS PostMessage,
-			Posts.Type AS PostType,
-			Posts.Props AS PostProps,
-			Posts.OriginalId AS PostOriginalId,
-			Posts.RootId AS PostRootId,
-			Posts.FileIds AS PostFileIds,
-			Teams.Id AS TeamId,
-			Teams.Name AS TeamName,
+	builder := s.getQueryBuilder().Select(`Posts.Id AS PostId,
+			Posts.CreateAt    AS PostCreateAt,
+			Posts.UpdateAt    AS PostUpdateAt,
+			Posts.DeleteAt    AS PostDeleteAt,
+            Posts.EditAt      AS PostEditAt,
+			Posts.Message     AS PostMessage,
+			Posts.Type        AS PostType,
+			Posts.Props       AS PostProps,
+			Posts.OriginalId  AS PostOriginalId,
+			Posts.RootId      AS PostRootId,
+			Posts.FileIds     AS PostFileIds,
+			Teams.Id          AS TeamId,
+			Teams.Name        AS TeamName,
 			Teams.DisplayName AS TeamDisplayName,
-			Channels.Id AS ChannelId,
-			Channels.Name AS ChannelName,
-			Channels.Type AS ChannelType,
-			Users.Id AS UserId,
-			Users.Email AS UserEmail,
+			Channels.Id       AS ChannelId,
+			Channels.Name     AS ChannelName,
+			Channels.Type     AS ChannelType,
+			Users.Id          AS UserId,
+			Users.Email       AS UserEmail,
 			Users.Username,
 			Bots.UserId IS NOT NULL AS IsBot`).
 		Column(caseStmt+" AS ChannelDisplayName", caseArgs...).
@@ -323,8 +324,13 @@ func (s SqlComplianceStore) MessageExport(c request.CTX, cursor model.MessageExp
 			sq.NotLike{"Posts.Type": "system_%"},
 		}).
 		OrderBy("PostUpdateAt, PostId").
-		Limit(uint64(limit)).
-		ToSql()
+		Limit(uint64(limit))
+
+	if cursor.UntilUpdateAt > 0 {
+		builder = builder.Where(sq.LtOrEq{"Posts.UpdateAt": cursor.UntilUpdateAt})
+	}
+
+	query, args, err := builder.ToSql()
 	if err != nil {
 		return nil, cursor, errors.Wrap(err, "unable to construct query to export messages")
 	}

server/channels/store/sqlstore/post_store.go

@@ -2404,15 +2404,14 @@ func (s *SqlPostStore) AnalyticsPostCount(options *model.PostCountOptions) (int6
 		})
 	}
 
-	queryString, args, err := query.ToSql()
-	if err != nil {
-		return 0, errors.Wrap(err, "post_tosql")
+	if options.UntilUpdateAt > 0 {
+		query = query.Where(sq.LtOrEq{"p.UpdateAt": options.UntilUpdateAt})
 	}
 
 	var v int64
-	err = s.GetReplica().Get(&v, queryString, args...)
+	err := s.GetReplica().GetBuilder(&v, query)
 	if err != nil {
-		return 0, errors.Wrap(err, "failed to count Posts")
+		return 0, fmt.Errorf("post_tosql failed or failed to count Posts: %w", err)
 	}
 
 	return v, nil

server/channels/store/sqlstore/store.go

@@ -35,6 +35,8 @@ import (
 
 type migrationDirection string
 
+type Option func(s *SqlStore) error
+
 const (
 	IndexTypeFullText                 = "full_text"
 	IndexTypeFullTextFunc             = "full_text_func"
@@ -139,12 +141,20 @@ type SqlStore struct {
 
 	isBinaryParam             bool
 	pgDefaultTextSearchConfig string
+	skipMigrations            bool
 
 	quitMonitor chan struct{}
 	wgMonitor   *sync.WaitGroup
 }
 
-func New(settings model.SqlSettings, logger mlog.LoggerIFace, metrics einterfaces.MetricsInterface) (*SqlStore, error) {
+func SkipMigrations() Option {
+	return func(s *SqlStore) error {
+		s.skipMigrations = true
+		return nil
+	}
+}
+
+func New(settings model.SqlSettings, logger mlog.LoggerIFace, metrics einterfaces.MetricsInterface, options ...Option) (*SqlStore, error) {
 	store := &SqlStore{
 		rrCounter:   0,
 		srCounter:   0,
@@ -155,6 +165,12 @@ func New(settings model.SqlSettings, logger mlog.LoggerIFace, metrics einterface
 		wgMonitor:   &sync.WaitGroup{},
 	}
 
+	for _, option := range options {
+		if err := option(store); err != nil {
+			return nil, fmt.Errorf("failed to apply option: %w", err)
+		}
+	}
+
 	err := store.initConnection()
 	if err != nil {
 		return nil, errors.Wrap(err, "error setting up connections")
@@ -178,9 +194,11 @@ func New(settings model.SqlSettings, logger mlog.LoggerIFace, metrics einterface
 		return nil, errors.Wrap(err, "error while checking DB collation")
 	}
 
-	err = store.migrate(migrationsDirectionUp, false)
-	if err != nil {
-		return nil, errors.Wrap(err, "failed to apply database migrations")
+	if !store.skipMigrations {
+		err = store.migrate(migrationsDirectionUp, false)
+		if err != nil {
+			return nil, errors.Wrap(err, "failed to apply database migrations")
+		}
 	}
 
 	store.isBinaryParam, err = store.computeBinaryParam()

server/channels/store/sqlstore/store_test.go

@@ -4,6 +4,7 @@
 package sqlstore
 
 import (
+	"database/sql"
 	"fmt"
 	"os"
 	"path/filepath"
@@ -977,3 +978,40 @@ func TestGetAppliedMigrations(t *testing.T) {
 		})
 	}
 }
+
+func TestSkipMigrationsOption(t *testing.T) {
+	testDrivers := []string{
+		model.DatabaseDriverPostgres,
+		model.DatabaseDriverMysql,
+	}
+
+	logger := mlog.CreateConsoleTestLogger(t)
+
+	for _, d := range testDrivers {
+		driver := d
+		t.Run("Should not apply migrations when skipMigrations=true for "+driver, func(t *testing.T) {
+			t.Parallel()
+			settings, err := makeSqlSettings(driver)
+			if err != nil {
+				t.Skip(err)
+			}
+
+			store, err := New(*settings, logger, nil, SkipMigrations())
+			require.NoError(t, err)
+
+			_, err = store.GetDBSchemaVersion()
+			assert.Error(t, err)
+
+			// Unfortunately there's no error for table doesn't exist, but we know it won't be ErrNoRows:
+			assert.True(t, !errors.Is(err, sql.ErrNoRows))
+
+			// And we know what each db will return:
+			if driver == model.DatabaseDriverPostgres {
+				assert.Contains(t, err.Error(), "pq: relation \"db_migrations\" does not exist")
+			} else {
+				// Mysql includes the random db name, so test the end:
+				assert.Contains(t, err.Error(), ".db_migrations' doesn't exist")
+			}
+		})
+	}
+}

server/channels/store/store.go

@@ -318,7 +318,8 @@ type ChannelStore interface {
 type ChannelMemberHistoryStore interface {
 	LogJoinEvent(userID string, channelID string, joinTime int64) error
 	LogLeaveEvent(userID string, channelID string, leaveTime int64) error
-	GetUsersInChannelDuring(startTime int64, endTime int64, channelID string) ([]*model.ChannelMemberHistoryResult, error)
+	GetUsersInChannelDuring(startTime int64, endTime int64, channelID []string) ([]*model.ChannelMemberHistoryResult, error)
+	GetChannelsWithActivityDuring(startTime int64, endTime int64) ([]string, error)
 	PermanentDeleteBatchForRetentionPolicies(now, globalPolicyEndTime, limit int64, cursor model.RetentionPolicyCursor) (int64, model.RetentionPolicyCursor, error)
 	DeleteOrphanedRows(limit int) (deleted int64, err error)
 	PermanentDeleteBatch(endTime int64, limit int64) (int64, error)

server/channels/store/storetest/channel_member_history_store.go

@@ -4,9 +4,9 @@
 package storetest
 
 import (
-	"testing"
-
 	"math"
+	"testing"
+	"time"
 
 	"github.com/stretchr/testify/assert"
 	"github.com/stretchr/testify/require"
@@ -21,6 +21,7 @@ func TestChannelMemberHistoryStore(t *testing.T, rctx request.CTX, ss store.Stor
 	t.Run("TestLogLeaveEvent", func(t *testing.T) { testLogLeaveEvent(t, rctx, ss) })
 	t.Run("TestGetUsersInChannelAtChannelMemberHistory", func(t *testing.T) { testGetUsersInChannelAtChannelMemberHistory(t, rctx, ss) })
 	t.Run("TestGetUsersInChannelAtChannelMembers", func(t *testing.T) { testGetUsersInChannelAtChannelMembers(t, rctx, ss) })
+	t.Run("TestGetChannelsWithActivityDuring", func(t *testing.T) { testGetChannelsWithActivityDuring(t, rctx, ss) })
 	t.Run("TestPermanentDeleteBatch", func(t *testing.T) { testPermanentDeleteBatch(t, rctx, ss) })
 	t.Run("TestPermanentDeleteBatchForRetentionPolicies", func(t *testing.T) { testPermanentDeleteBatchForRetentionPolicies(t, rctx, ss) })
 	t.Run("TestGetChannelsLeftSince", func(t *testing.T) { testGetChannelsLeftSince(t, rctx, ss) })
@@ -81,6 +82,160 @@ func testLogLeaveEvent(t *testing.T, rctx request.CTX, ss store.Store) {
 	assert.NoError(t, err)
 }
 
+func testGetChannelsWithActivityDuring(t *testing.T, rctx request.CTX, ss store.Store) {
+	// Need to wait to make sure channels and posts have nothing in them for this test.
+	time.Sleep(101 * time.Millisecond)
+
+	// create three test channels
+	ch1 := &model.Channel{
+		TeamId:      model.NewId(),
+		DisplayName: "Display " + model.NewId(),
+		Name:        model.NewId(),
+		Type:        model.ChannelTypeOpen,
+	}
+	channel1, err := ss.Channel().Save(rctx, ch1, -1)
+	require.NoError(t, err)
+
+	// channel2 will have no activity until case 6 (shouldn't show up until then)
+	ch2 := &model.Channel{
+		TeamId:      model.NewId(),
+		DisplayName: "Display " + model.NewId(),
+		Name:        model.NewId(),
+		Type:        model.ChannelTypeOpen,
+	}
+	channel2, err := ss.Channel().Save(rctx, ch2, -1)
+	require.NoError(t, err)
+
+	// and two test users
+	user1 := model.User{
+		Email:    MakeEmail(),
+		Nickname: model.NewId(),
+		Username: model.NewUsername(),
+	}
+	userPtr, err := ss.User().Save(rctx, &user1)
+	require.NoError(t, err)
+	user1 = *userPtr
+
+	user2 := model.User{
+		Email:    MakeEmail(),
+		Nickname: model.NewId(),
+		Username: model.NewUsername(),
+	}
+	userPtr, err = ss.User().Save(rctx, &user2)
+	require.NoError(t, err)
+	user2 = *userPtr
+
+	now := model.GetMillis()
+	originalNow := now
+
+	// user2 joins channel2 before test
+	err = ss.ChannelMemberHistory().LogJoinEvent(user2.Id, channel2.Id, now-2000)
+	require.NoError(t, err)
+
+	// case 7: assert no activity for time period before channel activity
+	channelIds, err := ss.ChannelMemberHistory().GetChannelsWithActivityDuring(originalNow-100, originalNow)
+	require.NoError(t, err)
+	assert.Empty(t, channelIds)
+
+	// case 1: user1 was in channel before period, doesn't show activity
+	err = ss.ChannelMemberHistory().LogJoinEvent(user1.Id, channel1.Id, now-1100)
+	require.NoError(t, err)
+
+	channelIds, err = ss.ChannelMemberHistory().GetChannelsWithActivityDuring(now, now+1000)
+	require.NoError(t, err)
+	assert.Empty(t, channelIds)
+
+	// case 2: user1 leaves, shows activity
+	err = ss.ChannelMemberHistory().LogLeaveEvent(user1.Id, channel1.Id, now+1)
+	require.NoError(t, err)
+
+	channelIds, err = ss.ChannelMemberHistory().GetChannelsWithActivityDuring(now, now+1000)
+	require.NoError(t, err)
+	assert.Equal(t, channelIds, []string{channel1.Id})
+
+	// case 3: user1 joins, shows activity
+	err = ss.ChannelMemberHistory().LogJoinEvent(user1.Id, channel1.Id, now+2)
+	require.NoError(t, err)
+
+	channelIds, err = ss.ChannelMemberHistory().GetChannelsWithActivityDuring(now+2, now+1000)
+	require.NoError(t, err)
+	assert.Equal(t, channelIds, []string{channel1.Id})
+
+	// case 4: new post shows activity
+	channelIds, err = ss.ChannelMemberHistory().GetChannelsWithActivityDuring(now+3, now+1000)
+	require.NoError(t, err)
+	assert.Empty(t, channelIds)
+
+	post := &model.Post{
+		ChannelId: channel1.Id,
+		Message:   "root post",
+		UserId:    user1.Id,
+		CreateAt:  now + 4,
+		UpdateAt:  now + 4,
+	}
+	post, err = ss.Post().Save(rctx, post)
+	require.NoError(t, err)
+
+	channelIds, err = ss.ChannelMemberHistory().GetChannelsWithActivityDuring(now+3, now+1000)
+	require.NoError(t, err)
+	assert.Equal(t, channelIds, []string{channel1.Id})
+
+	// case 5: update shows activity
+	// need to wait because update uses getMillis
+	time.Sleep(10 * time.Millisecond)
+	now = model.GetMillis()
+
+	channelIds, err = ss.ChannelMemberHistory().GetChannelsWithActivityDuring(now-1, now+1000)
+	require.NoError(t, err)
+	assert.Empty(t, channelIds)
+
+	newPost := post.Clone()
+	newPost.Message = "edited message"
+	_, err = ss.Post().Update(rctx, newPost, post)
+	require.NoError(t, err)
+
+	channelIds, err = ss.ChannelMemberHistory().GetChannelsWithActivityDuring(now-1, now+1000)
+	require.NoError(t, err)
+	assert.Equal(t, channelIds, []string{channel1.Id})
+
+	// case 6: get both activity from posts and from join/leave;
+	//  - also, sql deduplicates two channel1 results (from post and channel history tables)
+	time.Sleep(1 * time.Millisecond)
+	now = model.GetMillis()
+
+	channelIds, err = ss.ChannelMemberHistory().GetChannelsWithActivityDuring(now, now+1000)
+	require.NoError(t, err)
+	assert.Empty(t, channelIds)
+
+	post2 := &model.Post{
+		ChannelId: channel1.Id,
+		Message:   "root post",
+		UserId:    user1.Id,
+		CreateAt:  now + 11,
+		UpdateAt:  now + 11,
+	}
+	_, err = ss.Post().Save(rctx, post2)
+	require.NoError(t, err)
+	err = ss.ChannelMemberHistory().LogLeaveEvent(user1.Id, channel1.Id, now+12)
+	require.NoError(t, err)
+	err = ss.ChannelMemberHistory().LogLeaveEvent(user2.Id, channel2.Id, now+13)
+	require.NoError(t, err)
+
+	channelIds, err = ss.ChannelMemberHistory().GetChannelsWithActivityDuring(now+10, now+1000)
+	require.NoError(t, err)
+	assert.ElementsMatch(t, channelIds, []string{channel1.Id, channel2.Id})
+
+	// case 7: still no activity for period before tests
+	channelIds, err = ss.ChannelMemberHistory().GetChannelsWithActivityDuring(originalNow-100, originalNow)
+	require.NoError(t, err)
+	assert.Empty(t, channelIds)
+
+	// case 8: no activity for period after tests
+	channelIds, err = ss.ChannelMemberHistory().GetChannelsWithActivityDuring(now+100, now+1000)
+	require.NoError(t, err)
+	assert.Empty(t, channelIds)
+}
+
 func testGetUsersInChannelAtChannelMemberHistory(t *testing.T, rctx request.CTX, ss store.Store) {
 	// create a test channel
 	ch := &model.Channel{
@@ -119,12 +274,12 @@ func testGetUsersInChannelAtChannelMemberHistory(t *testing.T, rctx request.CTX,
 	require.NoError(t, err)
 
 	// case 1: user joins and leaves the channel before the export period begins
-	channelMembers, err := ss.ChannelMemberHistory().GetUsersInChannelDuring(joinTime-500, joinTime-100, channel.Id)
+	channelMembers, err := ss.ChannelMemberHistory().GetUsersInChannelDuring(joinTime-500, joinTime-100, []string{channel.Id})
 	require.NoError(t, err)
 	assert.Empty(t, channelMembers)
 
 	// case 2: user joins the channel after the export period begins, but has not yet left the channel when the export period ends
-	channelMembers, err = ss.ChannelMemberHistory().GetUsersInChannelDuring(joinTime-100, joinTime+500, channel.Id)
+	channelMembers, err = ss.ChannelMemberHistory().GetUsersInChannelDuring(joinTime-100, joinTime+500, []string{channel.Id})
 	require.NoError(t, err)
 	assert.Len(t, channelMembers, 1)
 	assert.Equal(t, channel.Id, channelMembers[0].ChannelId)
@@ -135,7 +290,7 @@ func testGetUsersInChannelAtChannelMemberHistory(t *testing.T, rctx request.CTX,
 	assert.Nil(t, channelMembers[0].LeaveTime)
 
 	// case 3: user joins the channel before the export period begins, but has not yet left the channel when the export period ends
-	channelMembers, err = ss.ChannelMemberHistory().GetUsersInChannelDuring(joinTime+100, joinTime+500, channel.Id)
+	channelMembers, err = ss.ChannelMemberHistory().GetUsersInChannelDuring(joinTime+100, joinTime+500, []string{channel.Id})
 	require.NoError(t, err)
 	assert.Len(t, channelMembers, 1)
 	assert.Equal(t, channel.Id, channelMembers[0].ChannelId)
@@ -150,7 +305,7 @@ func testGetUsersInChannelAtChannelMemberHistory(t *testing.T, rctx request.CTX,
 	require.NoError(t, err)
 
 	// case 4: user joins the channel before the export period begins, but has not yet left the channel when the export period ends
-	channelMembers, err = ss.ChannelMemberHistory().GetUsersInChannelDuring(joinTime+100, leaveTime-100, channel.Id)
+	channelMembers, err = ss.ChannelMemberHistory().GetUsersInChannelDuring(joinTime+100, leaveTime-100, []string{channel.Id})
 	require.NoError(t, err)
 	assert.Len(t, channelMembers, 1)
 	assert.Equal(t, channel.Id, channelMembers[0].ChannelId)
@@ -161,7 +316,7 @@ func testGetUsersInChannelAtChannelMemberHistory(t *testing.T, rctx request.CTX,
 	assert.Equal(t, leaveTime, *channelMembers[0].LeaveTime)
 
 	// case 5: user joins the channel after the export period begins, and leaves the channel before the export period ends
-	channelMembers, err = ss.ChannelMemberHistory().GetUsersInChannelDuring(joinTime-100, leaveTime+100, channel.Id)
+	channelMembers, err = ss.ChannelMemberHistory().GetUsersInChannelDuring(joinTime-100, leaveTime+100, []string{channel.Id})
 	require.NoError(t, err)
 	assert.Len(t, channelMembers, 1)
 	assert.Equal(t, channel.Id, channelMembers[0].ChannelId)
@@ -172,7 +327,7 @@ func testGetUsersInChannelAtChannelMemberHistory(t *testing.T, rctx request.CTX,
 	assert.Equal(t, leaveTime, *channelMembers[0].LeaveTime)
 
 	// case 6: user has joined and left the channel long before the export period begins
-	channelMembers, err = ss.ChannelMemberHistory().GetUsersInChannelDuring(leaveTime+100, leaveTime+200, channel.Id)
+	channelMembers, err = ss.ChannelMemberHistory().GetUsersInChannelDuring(leaveTime+100, leaveTime+200, []string{channel.Id})
 	require.NoError(t, err)
 	assert.Empty(t, channelMembers)
 }
@@ -223,7 +378,7 @@ func testGetUsersInChannelAtChannelMembers(t *testing.T, rctx request.CTX, ss st
 	// the past, even though the time that they were actually in the channel doesn't necessarily overlap with the export period
 
 	// case 1: user joins and leaves the channel before the export period begins
-	channelMembers, err := ss.ChannelMemberHistory().GetUsersInChannelDuring(joinTime-500, joinTime-100, channel.Id)
+	channelMembers, err := ss.ChannelMemberHistory().GetUsersInChannelDuring(joinTime-500, joinTime-100, []string{channel.Id})
 	require.NoError(t, err)
 	assert.Len(t, channelMembers, 1)
 	assert.Equal(t, channel.Id, channelMembers[0].ChannelId)
@@ -234,7 +389,7 @@ func testGetUsersInChannelAtChannelMembers(t *testing.T, rctx request.CTX, ss st
 	assert.Equal(t, joinTime-100, *channelMembers[0].LeaveTime)
 
 	// case 2: user joins the channel after the export period begins, but has not yet left the channel when the export period ends
-	channelMembers, err = ss.ChannelMemberHistory().GetUsersInChannelDuring(joinTime-100, joinTime+500, channel.Id)
+	channelMembers, err = ss.ChannelMemberHistory().GetUsersInChannelDuring(joinTime-100, joinTime+500, []string{channel.Id})
 	require.NoError(t, err)
 	assert.Len(t, channelMembers, 1)
 	assert.Equal(t, channel.Id, channelMembers[0].ChannelId)
@@ -245,7 +400,7 @@ func testGetUsersInChannelAtChannelMembers(t *testing.T, rctx request.CTX, ss st
 	assert.Equal(t, joinTime+500, *channelMembers[0].LeaveTime)
 
 	// case 3: user joins the channel before the export period begins, but has not yet left the channel when the export period ends
-	channelMembers, err = ss.ChannelMemberHistory().GetUsersInChannelDuring(joinTime+100, joinTime+500, channel.Id)
+	channelMembers, err = ss.ChannelMemberHistory().GetUsersInChannelDuring(joinTime+100, joinTime+500, []string{channel.Id})
 	require.NoError(t, err)
 	assert.Len(t, channelMembers, 1)
 	assert.Equal(t, channel.Id, channelMembers[0].ChannelId)
@@ -256,7 +411,7 @@ func testGetUsersInChannelAtChannelMembers(t *testing.T, rctx request.CTX, ss st
 	assert.Equal(t, joinTime+500, *channelMembers[0].LeaveTime)
 
 	// case 4: user joins the channel before the export period begins, but has not yet left the channel when the export period ends
-	channelMembers, err = ss.ChannelMemberHistory().GetUsersInChannelDuring(joinTime+100, leaveTime-100, channel.Id)
+	channelMembers, err = ss.ChannelMemberHistory().GetUsersInChannelDuring(joinTime+100, leaveTime-100, []string{channel.Id})
 	require.NoError(t, err)
 	assert.Len(t, channelMembers, 1)
 	assert.Equal(t, channel.Id, channelMembers[0].ChannelId)
@@ -267,7 +422,7 @@ func testGetUsersInChannelAtChannelMembers(t *testing.T, rctx request.CTX, ss st
 	assert.Equal(t, leaveTime-100, *channelMembers[0].LeaveTime)
 
 	// case 5: user joins the channel after the export period begins, and leaves the channel before the export period ends
-	channelMembers, err = ss.ChannelMemberHistory().GetUsersInChannelDuring(joinTime-100, leaveTime+100, channel.Id)
+	channelMembers, err = ss.ChannelMemberHistory().GetUsersInChannelDuring(joinTime-100, leaveTime+100, []string{channel.Id})
 	require.NoError(t, err)
 	assert.Len(t, channelMembers, 1)
 	assert.Equal(t, channel.Id, channelMembers[0].ChannelId)
@@ -278,7 +433,7 @@ func testGetUsersInChannelAtChannelMembers(t *testing.T, rctx request.CTX, ss st
 	assert.Equal(t, leaveTime+100, *channelMembers[0].LeaveTime)
 
 	// case 6: user has joined and left the channel long before the export period begins
-	channelMembers, err = ss.ChannelMemberHistory().GetUsersInChannelDuring(leaveTime+100, leaveTime+200, channel.Id)
+	channelMembers, err = ss.ChannelMemberHistory().GetUsersInChannelDuring(leaveTime+100, leaveTime+200, []string{channel.Id})
 	require.NoError(t, err)
 	assert.Len(t, channelMembers, 1)
 	assert.Equal(t, channel.Id, channelMembers[0].ChannelId)
@@ -332,7 +487,7 @@ func testPermanentDeleteBatch(t *testing.T, rctx request.CTX, ss store.Store) {
 	require.NoError(t, err)
 
 	// in between the join time and the leave time, both users were members of the channel
-	channelMembers, err := ss.ChannelMemberHistory().GetUsersInChannelDuring(joinTime+10, leaveTime-10, channel.Id)
+	channelMembers, err := ss.ChannelMemberHistory().GetUsersInChannelDuring(joinTime+10, leaveTime-10, []string{channel.Id})
 	require.NoError(t, err)
 	assert.Len(t, channelMembers, 2)
 
@@ -343,7 +498,7 @@ func testPermanentDeleteBatch(t *testing.T, rctx request.CTX, ss store.Store) {
 	assert.NotEqual(t, int64(0), rowsDeleted)
 
 	// after the delete, there should be one less member in the channel
-	channelMembers, err = ss.ChannelMemberHistory().GetUsersInChannelDuring(joinTime+10, leaveTime-10, channel.Id)
+	channelMembers, err = ss.ChannelMemberHistory().GetUsersInChannelDuring(joinTime+10, leaveTime-10, []string{channel.Id})
 	require.NoError(t, err)
 	assert.Len(t, channelMembers, 1)
 	assert.Equal(t, user2.Id, channelMembers[0].UserId)
@@ -387,7 +542,7 @@ func testPermanentDeleteBatchForRetentionPolicies(t *testing.T, rctx request.CTX
 	_, _, err = ss.ChannelMemberHistory().PermanentDeleteBatchForRetentionPolicies(
 		nowMillis, 0, limit, model.RetentionPolicyCursor{})
 	require.NoError(t, err)
-	result, err := ss.ChannelMemberHistory().GetUsersInChannelDuring(joinTime, leaveTime, channel.Id)
+	result, err := ss.ChannelMemberHistory().GetUsersInChannelDuring(joinTime, leaveTime, []string{channel.Id})
 	require.NoError(t, err)
 	require.Empty(t, result, "history should have been deleted by channel policy")
 	rows, err := ss.RetentionPolicy().GetIdsForDeletionByTableName("ChannelMemberHistory", 1000)

server/channels/store/storetest/compliance_store.go

@@ -58,6 +58,7 @@ func TestComplianceStore(t *testing.T, rctx request.CTX, ss store.Store) {
 	t.Run("MessageEditAfterExportMessage", func(t *testing.T) { testEditAfterExportMessage(t, rctx, ss) })
 	t.Run("MessageDeleteExportMessage", func(t *testing.T) { testDeleteExportMessage(t, rctx, ss) })
 	t.Run("MessageDeleteAfterExportMessage", func(t *testing.T) { testDeleteAfterExportMessage(t, rctx, ss) })
+	t.Run("MessageExport_UntilUpdateAt", func(t *testing.T) { testMessageExportUntilUpdateAt(t, rctx, ss) })
 }
 
 func testComplianceStore(t *testing.T, rctx request.CTX, ss store.Store) {
@@ -1174,3 +1175,105 @@ func testDeleteAfterExportMessage(t *testing.T, rctx request.CTX, ss store.Store
 	assert.Equal(t, user1.Email, *v.UserEmail)
 	assert.Equal(t, user1.Username, *v.Username)
 }
+
+func testMessageExportUntilUpdateAt(t *testing.T, rctx request.CTX, ss store.Store) {
+	defer cleanupStoreState(t, rctx, ss)
+
+	// get the starting number of message export entries
+	startTime := model.GetMillis()
+	messages, _, err := ss.Compliance().MessageExport(rctx, model.MessageExportCursor{LastPostUpdateAt: startTime - 10}, 10)
+	require.NoError(t, err)
+	assert.Equal(t, 0, len(messages))
+
+	// need a team
+	team := &model.Team{
+		DisplayName: "DisplayName",
+		Name:        model.NewId(),
+		Email:       MakeEmail(),
+		Type:        model.TeamOpen,
+	}
+	team, err = ss.Team().Save(team)
+	require.NoError(t, err)
+
+	// and two users that are a part of that team
+	user1 := &model.User{
+		Email:    MakeEmail(),
+		Username: model.NewUsername(),
+	}
+	user1, err = ss.User().Save(rctx, user1)
+	require.NoError(t, err)
+	_, nErr := ss.Team().SaveMember(rctx, &model.TeamMember{
+		TeamId: team.Id,
+		UserId: user1.Id,
+	}, -1)
+	require.NoError(t, nErr)
+
+	user2 := &model.User{
+		Email:    MakeEmail(),
+		Username: model.NewUsername(),
+	}
+	user2, err = ss.User().Save(rctx, user2)
+	require.NoError(t, err)
+	_, nErr = ss.Team().SaveMember(rctx, &model.TeamMember{
+		TeamId: team.Id,
+		UserId: user2.Id,
+	}, -1)
+	require.NoError(t, nErr)
+
+	// need a public channel
+	channel := &model.Channel{
+		TeamId:      team.Id,
+		Name:        model.NewId(),
+		DisplayName: "Public Channel",
+		Type:        model.ChannelTypeOpen,
+	}
+	channel, nErr = ss.Channel().Save(rctx, channel, -1)
+	require.NoError(t, nErr)
+
+	var posts []*model.Post
+	// user1 posts ten times in the public channel
+	for i := 0; i < 10; i++ {
+		post := &model.Post{
+			ChannelId: channel.Id,
+			UserId:    user1.Id,
+			CreateAt:  startTime + int64(i),
+			UpdateAt:  startTime + int64(i),
+			Message:   model.NewId(),
+		}
+		post, err = ss.Post().Save(rctx, post)
+		require.NoError(t, err)
+		posts = append(posts, post)
+	}
+
+	// fetch 5 starting from the third post, using LastPostUpdateAt and UntilUpdateAt.
+	// UntilUpdateAt is inclusive
+	messageExportMap := map[string]model.MessageExport{}
+	messages, _, err = ss.Compliance().MessageExport(rctx, model.MessageExportCursor{LastPostUpdateAt: posts[2].UpdateAt, UntilUpdateAt: posts[2].UpdateAt + 4}, 10000)
+	require.NoError(t, err)
+	assert.Equal(t, 5, len(messages))
+
+	for _, v := range messages {
+		messageExportMap[*v.PostId] = *v
+	}
+
+	for i := 2; i < 7; i++ {
+		assert.Equal(t, posts[i].Id, *messageExportMap[posts[i].Id].PostId)
+		assert.Equal(t, posts[i].CreateAt, *messageExportMap[posts[i].Id].PostCreateAt)
+		assert.Equal(t, posts[i].Message, *messageExportMap[posts[i].Id].PostMessage)
+		assert.Equal(t, channel.Id, *messageExportMap[posts[i].Id].ChannelId)
+		assert.Equal(t, channel.DisplayName, *messageExportMap[posts[i].Id].ChannelDisplayName)
+		assert.Equal(t, user1.Id, *messageExportMap[posts[i].Id].UserId)
+		assert.Equal(t, user1.Email, *messageExportMap[posts[i].Id].UserEmail)
+		assert.Equal(t, user1.Username, *messageExportMap[posts[i].Id].Username)
+	}
+
+	// Also test AnalyticsPostCount because they are used in tandem for MessageExports
+	count, err := ss.Post().AnalyticsPostCount(&model.PostCountOptions{
+		TeamId:             channel.TeamId,
+		ExcludeSystemPosts: true,
+		SinceUpdateAt:      posts[2].UpdateAt,
+		UntilUpdateAt:      posts[2].UpdateAt + 4,
+	})
+	require.NoError(t, err)
+	require.Equal(t, 5, int(count))
+}

server/channels/store/storetest/mocks/ChannelMemberHistoryStore.go

@@ -72,8 +72,38 @@ func (_m *ChannelMemberHistoryStore) GetChannelsLeftSince(userID string, since i
 	return r0, r1
 }
 
+// GetChannelsWithActivityDuring provides a mock function with given fields: startTime, endTime
+func (_m *ChannelMemberHistoryStore) GetChannelsWithActivityDuring(startTime int64, endTime int64) ([]string, error) {
+	ret := _m.Called(startTime, endTime)
+
+	if len(ret) == 0 {
+		panic("no return value specified for GetChannelsWithActivityDuring")
+	}
+
+	var r0 []string
+	var r1 error
+	if rf, ok := ret.Get(0).(func(int64, int64) ([]string, error)); ok {
+		return rf(startTime, endTime)
+	}
+	if rf, ok := ret.Get(0).(func(int64, int64) []string); ok {
+		r0 = rf(startTime, endTime)
+	} else {
+		if ret.Get(0) != nil {
+			r0 = ret.Get(0).([]string)
+		}
+	}
+
+	if rf, ok := ret.Get(1).(func(int64, int64) error); ok {
+		r1 = rf(startTime, endTime)
+	} else {
+		r1 = ret.Error(1)
+	}
+
+	return r0, r1
+}
+
 // GetUsersInChannelDuring provides a mock function with given fields: startTime, endTime, channelID
-func (_m *ChannelMemberHistoryStore) GetUsersInChannelDuring(startTime int64, endTime int64, channelID string) ([]*model.ChannelMemberHistoryResult, error) {
+func (_m *ChannelMemberHistoryStore) GetUsersInChannelDuring(startTime int64, endTime int64, channelID []string) ([]*model.ChannelMemberHistoryResult, error) {
 	ret := _m.Called(startTime, endTime, channelID)
 
 	if len(ret) == 0 {
@@ -82,10 +112,10 @@ func (_m *ChannelMemberHistoryStore) GetUsersInChannelDuring(startTime int64, en
 
 	var r0 []*model.ChannelMemberHistoryResult
 	var r1 error
-	if rf, ok := ret.Get(0).(func(int64, int64, string) ([]*model.ChannelMemberHistoryResult, error)); ok {
+	if rf, ok := ret.Get(0).(func(int64, int64, []string) ([]*model.ChannelMemberHistoryResult, error)); ok {
 		return rf(startTime, endTime, channelID)
 	}
-	if rf, ok := ret.Get(0).(func(int64, int64, string) []*model.ChannelMemberHistoryResult); ok {
+	if rf, ok := ret.Get(0).(func(int64, int64, []string) []*model.ChannelMemberHistoryResult); ok {
 		r0 = rf(startTime, endTime, channelID)
 	} else {
 		if ret.Get(0) != nil {
@@ -93,7 +123,7 @@ func (_m *ChannelMemberHistoryStore) GetUsersInChannelDuring(startTime int64, en
 		}
 	}
 
-	if rf, ok := ret.Get(1).(func(int64, int64, string) error); ok {
+	if rf, ok := ret.Get(1).(func(int64, int64, []string) error); ok {
 		r1 = rf(startTime, endTime, channelID)
 	} else {
 		r1 = ret.Error(1)

server/channels/store/timerlayer/timerlayer.go

@@ -2696,7 +2696,23 @@ func (s *TimerLayerChannelMemberHistoryStore) GetChannelsLeftSince(userID string
 	return result, err
 }
 
-func (s *TimerLayerChannelMemberHistoryStore) GetUsersInChannelDuring(startTime int64, endTime int64, channelID string) ([]*model.ChannelMemberHistoryResult, error) {
+func (s *TimerLayerChannelMemberHistoryStore) GetChannelsWithActivityDuring(startTime int64, endTime int64) ([]string, error) {
+	start := time.Now()
+
+	result, err := s.ChannelMemberHistoryStore.GetChannelsWithActivityDuring(startTime, endTime)
+
+	elapsed := float64(time.Since(start)) / float64(time.Second)
+	if s.Root.Metrics != nil {
+		success := "false"
+		if err == nil {
+			success = "true"
+		}
+		s.Root.Metrics.ObserveStoreMethodDuration("ChannelMemberHistoryStore.GetChannelsWithActivityDuring", success, elapsed)
+	}
+	return result, err
+}
+
+func (s *TimerLayerChannelMemberHistoryStore) GetUsersInChannelDuring(startTime int64, endTime int64, channelID []string) ([]*model.ChannelMemberHistoryResult, error) {
 	start := time.Now()
 
 	result, err := s.ChannelMemberHistoryStore.GetUsersInChannelDuring(startTime, endTime, channelID)

server/cmd/mattermost/commands/export.go

@@ -5,7 +5,6 @@ package commands
 
 import (
 	"context"
-	"fmt"
 	"os"
 	"path/filepath"
 	"time"
@@ -33,30 +32,6 @@ var ScheduleExportCmd = &cobra.Command{
 	RunE:    scheduleExportCmdF,
 }
 
-var CsvExportCmd = &cobra.Command{
-	Use:     "csv",
-	Short:   "Export data from Mattermost in CSV format",
-	Long:    "Export data from Mattermost in CSV format",
-	Example: "export csv --exportFrom=12345",
-	RunE:    buildExportCmdF("csv"),
-}
-
-var ActianceExportCmd = &cobra.Command{
-	Use:     "actiance",
-	Short:   "Export data from Mattermost in Actiance format",
-	Long:    "Export data from Mattermost in Actiance format",
-	Example: "export actiance --exportFrom=12345",
-	RunE:    buildExportCmdF("actiance"),
-}
-
-var GlobalRelayZipExportCmd = &cobra.Command{
-	Use:     "global-relay-zip",
-	Short:   "Export data from Mattermost into a zip file containing emails to send to Global Relay for debug and testing purposes only.",
-	Long:    "Export data from Mattermost into a zip file containing emails to send to Global Relay for debug and testing purposes only. This does not archive any information in Global Relay.",
-	Example: "export global-relay-zip --exportFrom=12345",
-	RunE:    buildExportCmdF("globalrelay-zip"),
-}
-
 var BulkExportCmd = &cobra.Command{
 	Use:     "bulk [file]",
 	Short:   "Export bulk data.",
@@ -71,15 +46,6 @@ func init() {
 	ScheduleExportCmd.Flags().Int64("exportFrom", -1, "The timestamp of the earliest post to export, expressed in seconds since the unix epoch.")
 	ScheduleExportCmd.Flags().Int("timeoutSeconds", -1, "The maximum number of seconds to wait for the job to complete before timing out.")
 
-	CsvExportCmd.Flags().Int64("exportFrom", -1, "The timestamp of the earliest post to export, expressed in seconds since the unix epoch.")
-	CsvExportCmd.Flags().Int("limit", -1, "The number of posts to export. The default of -1 means no limit.")
-
-	ActianceExportCmd.Flags().Int64("exportFrom", -1, "The timestamp of the earliest post to export, expressed in seconds since the unix epoch.")
-	ActianceExportCmd.Flags().Int("limit", -1, "The number of posts to export. The default of -1 means no limit.")
-
-	GlobalRelayZipExportCmd.Flags().Int64("exportFrom", -1, "The timestamp of the earliest post to export, expressed in seconds since the unix epoch.")
-	GlobalRelayZipExportCmd.Flags().Int("limit", -1, "The number of posts to export. The default of -1 means no limit.")
-
 	BulkExportCmd.Flags().Bool("all-teams", true, "Export all teams from the server.")
 	BulkExportCmd.Flags().Bool("with-archived-channels", false, "Also exports archived channels.")
 	BulkExportCmd.Flags().Bool("with-profile-pictures", false, "Also exports profile pictures.")
@@ -87,9 +53,6 @@ func init() {
 	BulkExportCmd.Flags().Bool("archive", false, "Outputs a single archive file.")
 
 	ExportCmd.AddCommand(ScheduleExportCmd)
-	ExportCmd.AddCommand(CsvExportCmd)
-	ExportCmd.AddCommand(ActianceExportCmd)
-	ExportCmd.AddCommand(GlobalRelayZipExportCmd)
 	ExportCmd.AddCommand(BulkExportCmd)
 
 	RootCmd.AddCommand(ExportCmd)
@@ -158,57 +121,6 @@ func scheduleExportCmdF(command *cobra.Command, args []string) error {
 	return nil
 }
 
-func buildExportCmdF(format string) func(command *cobra.Command, args []string) error {
-	return func(command *cobra.Command, args []string) error {
-		a, err := InitDBCommandContextCobra(command, app.SkipPostInitialization())
-		license := a.Srv().License()
-		if err != nil {
-			return err
-		}
-		defer a.Srv().Shutdown()
-
-		rctx := request.EmptyContext(a.Log())
-
-		startTime, err := command.Flags().GetInt64("exportFrom")
-		if err != nil {
-			return errors.New("exportFrom flag error")
-		}
-		if startTime < 0 {
-			return errors.New("exportFrom must be a positive integer")
-		}
-
-		limit, err := command.Flags().GetInt("limit")
-		if err != nil {
-			return errors.New("limit flag error")
-		}
-
-		if a.MessageExport() == nil || license == nil || !*license.Features.MessageExport {
-			return errors.New("message export feature not available")
-		}
-
-		warningsCount, appErr := a.MessageExport().RunExport(rctx, format, startTime, limit)
-		if appErr != nil {
-			return appErr
-		}
-		if warningsCount == 0 {
-			CommandPrettyPrintln("SUCCESS: Your data was exported.")
-		} else {
-			if format == model.ComplianceExportTypeGlobalrelay || format == model.ComplianceExportTypeGlobalrelayZip {
-				CommandPrettyPrintln(fmt.Sprintf("WARNING: %d warnings encountered, see logs for details.", warningsCount))
-			} else {
-				CommandPrettyPrintln(fmt.Sprintf("WARNING: %d warnings encountered, see warning.txt for details.", warningsCount))
-			}
-		}
-
-		auditRec := a.MakeAuditRecord(rctx, "buildExport", audit.Success)
-		auditRec.AddMeta("format", format)
-		auditRec.AddMeta("start", startTime)
-		a.LogAuditRec(rctx, auditRec, nil)
-
-		return nil
-	}
-}
-
 func bulkExportCmdF(command *cobra.Command, args []string) error {
 	a, err := InitDBCommandContextCobra(command, app.SkipPostInitialization())
 	if err != nil {

server/cmd/mattermost/commands/export_test.go

@@ -1,74 +0,0 @@
-// Copyright (c) 2015-present Mattermost, Inc. All Rights Reserved.
-// See LICENSE.txt for license information.
-
-package commands
-
-import (
-	"testing"
-
-	"github.com/stretchr/testify/require"
-
-	"github.com/mattermost/mattermost/server/public/model"
-)
-
-// There are no tests that actually run the Message Export job, because it can take a long time to complete depending
-// on the size of the database that the config is pointing to. As such, these tests just ensure that the CLI command
-// fails fast if invalid flags are supplied
-
-func TestMessageExportNotEnabled(t *testing.T) {
-	th := Setup(t)
-	defer th.TearDown()
-
-	config := th.Config()
-	config.MessageExportSettings.EnableExport = model.NewPointer(false)
-	th.SetConfig(config)
-
-	// should fail fast because the feature isn't enabled
-	require.Error(t, th.RunCommand(t, "export", "schedule"))
-}
-
-func TestMessageExportInvalidFormat(t *testing.T) {
-	th := Setup(t)
-	defer th.TearDown()
-
-	config := th.Config()
-	config.MessageExportSettings.EnableExport = model.NewPointer(true)
-	th.SetConfig(config)
-
-	// should fail fast because format isn't supported
-	require.Error(t, th.RunCommand(t, "--format", "not_actiance", "export", "schedule"))
-}
-
-func TestMessageExportNegativeExportFrom(t *testing.T) {
-	th := Setup(t)
-	defer th.TearDown()
-
-	config := th.Config()
-	config.MessageExportSettings.EnableExport = model.NewPointer(true)
-	th.SetConfig(config)
-
-	// should fail fast because export from must be a valid timestamp
-	require.Error(t, th.RunCommand(t, "--format", "actiance", "--exportFrom", "-1", "export", "schedule"))
-}
-
-func TestMessageExportNegativeTimeoutSeconds(t *testing.T) {
-	th := Setup(t)
-	defer th.TearDown()
-
-	config := th.Config()
-	config.MessageExportSettings.EnableExport = model.NewPointer(true)
-	th.SetConfig(config)
-
-	// should fail fast because timeout seconds must be a positive int
-	require.Error(t, th.RunCommand(t, "--format", "actiance", "--exportFrom", "0", "--timeoutSeconds", "-1", "export", "schedule"))
-}
-
-func TestMessageExportFailsWithoutLicense(t *testing.T) {
-	th := Setup(t)
-	defer th.TearDown()
-
-	actual, _ := th.RunCommandWithOutput(t, "export", "csv", "export.csv", "--exportFrom", "0")
-
-	// should fail fast because timeout seconds must be a positive int
-	require.Contains(t, actual, "message export feature not available")
-}

server/einterfaces/message_export.go

@@ -9,6 +9,5 @@ import (
 )
 
 type MessageExportInterface interface {
-	StartSynchronizeJob(c request.CTX, exportFromTimestamp int64) (*model.Job, *model.AppError)
-	RunExport(c request.CTX, format string, since int64, limit int) (int64, *model.AppError)
+	StartSynchronizeJob(rctx request.CTX, exportFromTimestamp int64) (*model.Job, *model.AppError)
 }

server/einterfaces/mocks/MessageExportInterface.go

@@ -15,39 +15,9 @@ type MessageExportInterface struct {
 	mock.Mock
 }
 
-// RunExport provides a mock function with given fields: c, format, since, limit
-func (_m *MessageExportInterface) RunExport(c request.CTX, format string, since int64, limit int) (int64, *model.AppError) {
-	ret := _m.Called(c, format, since, limit)
-
-	if len(ret) == 0 {
-		panic("no return value specified for RunExport")
-	}
-
-	var r0 int64
-	var r1 *model.AppError
-	if rf, ok := ret.Get(0).(func(request.CTX, string, int64, int) (int64, *model.AppError)); ok {
-		return rf(c, format, since, limit)
-	}
-	if rf, ok := ret.Get(0).(func(request.CTX, string, int64, int) int64); ok {
-		r0 = rf(c, format, since, limit)
-	} else {
-		r0 = ret.Get(0).(int64)
-	}
-
-	if rf, ok := ret.Get(1).(func(request.CTX, string, int64, int) *model.AppError); ok {
-		r1 = rf(c, format, since, limit)
-	} else {
-		if ret.Get(1) != nil {
-			r1 = ret.Get(1).(*model.AppError)
-		}
-	}
-
-	return r0, r1
-}
-
-// StartSynchronizeJob provides a mock function with given fields: c, exportFromTimestamp
-func (_m *MessageExportInterface) StartSynchronizeJob(c request.CTX, exportFromTimestamp int64) (*model.Job, *model.AppError) {
-	ret := _m.Called(c, exportFromTimestamp)
+// StartSynchronizeJob provides a mock function with given fields: rctx, exportFromTimestamp
+func (_m *MessageExportInterface) StartSynchronizeJob(rctx request.CTX, exportFromTimestamp int64) (*model.Job, *model.AppError) {
+	ret := _m.Called(rctx, exportFromTimestamp)
 
 	if len(ret) == 0 {
 		panic("no return value specified for StartSynchronizeJob")
@@ -56,10 +26,10 @@ func (_m *MessageExportInterface) StartSynchronizeJob(c request.CTX, exportFromT
 	var r0 *model.Job
 	var r1 *model.AppError
 	if rf, ok := ret.Get(0).(func(request.CTX, int64) (*model.Job, *model.AppError)); ok {
-		return rf(c, exportFromTimestamp)
+		return rf(rctx, exportFromTimestamp)
 	}
 	if rf, ok := ret.Get(0).(func(request.CTX, int64) *model.Job); ok {
-		r0 = rf(c, exportFromTimestamp)
+		r0 = rf(rctx, exportFromTimestamp)
 	} else {
 		if ret.Get(0) != nil {
 			r0 = ret.Get(0).(*model.Job)
@@ -67,7 +37,7 @@ func (_m *MessageExportInterface) StartSynchronizeJob(c request.CTX, exportFromT
 	}
 
 	if rf, ok := ret.Get(1).(func(request.CTX, int64) *model.AppError); ok {
-		r1 = rf(c, exportFromTimestamp)
+		r1 = rf(rctx, exportFromTimestamp)
 	} else {
 		if ret.Get(1) != nil {
 			r1 = ret.Get(1).(*model.AppError)

server/enterprise/internal/file/utils.go

@@ -0,0 +1,23 @@
+// Copyright (c) 2015-present Mattermost, Inc. All Rights Reserved.
+// See LICENSE.enterprise for license information.
+
+package file
+
+import (
+	"os"
+
+	"github.com/mattermost/mattermost/server/public/shared/mlog"
+)
+
+// DeleteTemp removes a file and logs the error
+// Intended to be called in a defer after the creation of a temp file to ensure cleanup
+func DeleteTemp(logger mlog.LoggerIFace, file *os.File) {
+	err := file.Close()
+	if err != nil {
+		logger.Warn("Failed to close temporary file", mlog.String("filename", file.Name()), mlog.Err(err))
+	}
+	err = os.Remove(file.Name())
+	if err != nil {
+		logger.Warn("Failed to delete temporary file", mlog.String("filename", file.Name()), mlog.Err(err))
+	}
+}

server/enterprise/message_export/actiance_export/actiance_export.go

@@ -4,29 +4,30 @@
 package actiance_export
 
 import (
+	"archive/zip"
 	"bytes"
-	"encoding/json"
 	"encoding/xml"
 	"fmt"
 	"io"
-	"net/http"
-	"path"
-	"sort"
+	"os"
+	"slices"
 	"strings"
+	"time"
 
-	"github.com/mattermost/mattermost/server/v8/enterprise/message_export/common_export"
+	"github.com/mattermost/mattermost/server/v8/enterprise/internal/file"
 
 	"github.com/mattermost/mattermost/server/public/model"
 	"github.com/mattermost/mattermost/server/public/shared/mlog"
 	"github.com/mattermost/mattermost/server/public/shared/request"
-	"github.com/mattermost/mattermost/server/v8/channels/store"
+	"github.com/mattermost/mattermost/server/v8/enterprise/message_export/shared"
 	"github.com/mattermost/mattermost/server/v8/platform/shared/filestore"
 )
 
 const (
-	XMLNS                   = "http://www.w3.org/2001/XMLSchema-instance"
-	ActianceExportFilename  = "actiance_export.xml"
-	ActianceWarningFilename = "warning.txt"
+	XMLNS                    = "http://www.w3.org/2001/XMLSchema-instance"
+	ActianceExportFilename   = "actiance_export.xml"
+	ActianceWarningFilename  = "warning.txt"
+	EnsureLastMessageInASort = "zzzzzzzzzzzzzzzzzzzzzzzzzz"
 )
 
 // The root-level element of an actiance export
@@ -39,17 +40,15 @@ type RootNode struct {
 // The Conversation element indicates an ad hoc IM conversation or a group chat room.
 // The messages from a persistent chat room are exported once a day so that a Conversation entry contains the messages posted to a chat room from 12:00:00 AM to 11:59:59 PM
 type ChannelExport struct {
-	XMLName      xml.Name     `xml:"Conversation"`
-	Perspective  string       `xml:"Perspective,attr"` // the value of this attribute doesn't seem to matter. Using the channel name makes the export more human readable
-	ChannelId    string       `xml:"-"`                // the unique id of the channel
-	RoomId       string       `xml:"RoomID"`
-	StartTime    int64        `xml:"StartTimeUTC"` // utc timestamp (seconds), start of export period or create time of channel, whichever is greater. Example: 1366611728.
-	JoinEvents   []JoinExport // start with a list of all users who were present in the channel during the export period
-	Elements     []any
-	UploadStarts []*FileUploadStartExport
-	UploadStops  []*FileUploadStopExport
-	LeaveEvents  []LeaveExport // finish with a list of all users who were present in the channel during the export period
-	EndTime      int64         `xml:"EndTimeUTC"` // utc timestamp (seconds), end of export period or delete time of channel, whichever is lesser. Example: 1366611728.
+	XMLName     xml.Name     `xml:"Conversation"`
+	Perspective string       `xml:"Perspective,attr"` // the value of this attribute doesn't seem to matter. Using the channel name makes the export more human readable
+	ChannelId   string       `xml:"-"`                // the unique id of the channel
+	RoomId      string       `xml:"RoomID"`
+	StartTime   int64        `xml:"StartTimeUTC"` // utc timestamp (seconds), start of export period or create time of channel, whichever is greater. Example: 1366611728.
+	JoinEvents  []JoinExport // start with a list of all users who were present in the channel during the export period
+	Elements    []Sortable
+	LeaveEvents []LeaveExport // finish with a list of all users who were present in the channel during the export period
+	EndTime     int64         `xml:"EndTimeUTC"` // utc timestamp (seconds), end of export period or delete time of channel, whichever is lesser. Example: 1366611728.
 }
 
 // The ParticipantEntered element indicates each user who participates in a conversation.
@@ -72,14 +71,55 @@ type LeaveExport struct {
 	CorporateEmailID string   `xml:"CorporateEmailID"`
 }
 
+type Sortable interface {
+	SortVal() (int64, string)
+}
+
+func SortableSort(a, b Sortable) int {
+	aTimestamp, aId := a.SortVal()
+	bTimestamp, bId := b.SortVal()
+	if aTimestamp == bTimestamp {
+		return strings.Compare(aId, bId)
+	}
+	return int(aTimestamp - bTimestamp)
+}
+
 // The Message element indicates the message sent by a user
 type PostExport struct {
-	XMLName      xml.Name `xml:"Message"`
-	UserEmail    string   `xml:"LoginName"`    // the email of the person that sent the post
-	UserType     string   `xml:"UserType"`     // the type of the person that sent the post
-	PostTime     int64    `xml:"DateTimeUTC"`  // utc timestamp (seconds), time at which the user sent the post. Example: 1366611728
-	Message      string   `xml:"Content"`      // the text body of the post
-	PreviewsPost string   `xml:"PreviewsPost"` // the post id of the post that is previewed by the permalink preview feature
+	XMLName   xml.Name        `xml:"Message"`
+	MessageId string          `xml:"MessageId"`   // the message id in the db
+	UserEmail string          `xml:"LoginName"`   // the email of the person that sent the post
+	UserType  shared.UserType `xml:"UserType"`    // the type of the person that sent the post: "user" or "bot"
+	CreateAt  int64           `xml:"DateTimeUTC"` // utc timestamp (unix milliseconds), the post's createAt
+
+	// Allows us to differentiate between:
+	// - "EditedOriginalMsg": the newly created message (new Id), which holds the pre-edited message contents. The
+	//    "EditedNewMsgId" field will point to the message (original Id) which has the post-edited message content.
+	// - "EditedNewMsg": the post-edited message content. This is confusing, so be careful: in the db, this EditedNewMsg
+	//    is actually the original messageId because we wanted an edited message to have the same messageId as the
+	//    pre-edited message. But for the purposes of exporting and to keep the mental model clear for end-users, we are
+	//    calling this the EditedNewMsg and EditedNewMsgId, because this will hold the NEW post-edited message contents,
+	//    and that's what's important to the end-user viewing the export.
+	//  - "UpdatedNoMsgChange": the message content hasn't changed, but the post was updated for some reason (reaction,
+	//	  replied-to, a reply was edited, a reply was deleted (as of 10.2), perhaps other reasons)
+	//  - "Deleted": the message was deleted.
+	//  - "FileDeleted": this message is recording that a file was deleted.
+	UpdatedType shared.PostUpdatedType `xml:"UpdatedType,omitempty"`
+	UpdateAt    int64                  `xml:"UpdatedDateTimeUTC,omitempty"` // if this is an updated post, this is the updated time (same as deleted time for deleted posts).
+
+	// when a message is edited, the EditedOriginalMsg points to the message Id that now has the newly edited message.
+	EditedNewMsgId string `xml:"EditedNewMsgId,omitempty"`
+
+	Message      string `xml:"Content"`                // the text body of the post
+	PreviewsPost string `xml:"PreviewsPost,omitempty"` // the post id of the post that is previewed by the permalink preview feature
+}
+
+func (p PostExport) SortVal() (int64, string) {
+	// updated messages are sorted by UpdateAt
+	if p.UpdatedType != "" {
+		return p.UpdateAt, p.MessageId
+	}
+	return p.CreateAt, p.MessageId
 }
 
 // The FileTransferStarted element indicates the beginning of a file transfer in a conversation
@@ -91,6 +131,11 @@ type FileUploadStartExport struct {
 	FilePath        string   `xml:"FileName"`     // the path to the file, as stored on the server
 }
 
+func (f FileUploadStartExport) SortVal() (int64, string) {
+	// file messages should come after the message they were with
+	return f.UploadStartTime, EnsureLastMessageInASort
+}
+
 // The FileTransferEnded element indicates the end of a file transfer in a conversation
 type FileUploadStopExport struct {
 	XMLName        xml.Name `xml:"FileTransferEnded"`
@@ -101,73 +146,105 @@ type FileUploadStopExport struct {
 	Status         string   `xml:"Status"`       // set to either "Completed" or "Failed" depending on the outcome of the upload operation
 }
 
-func ActianceExport(rctx request.CTX, posts []*model.MessageExport, db store.Store, exportBackend filestore.FileBackend, fileAttachmentBackend filestore.FileBackend, exportDirectory string) (warningCount int64, appErr *model.AppError) {
-	// sort the posts into buckets based on the channel in which they appeared
-	membersByChannel := common_export.MembersByChannel{}
-	metadata := common_export.Metadata{
-		Channels:         map[string]common_export.MetadataChannel{},
-		MessagesCount:    0,
-		AttachmentsCount: 0,
-		StartTime:        0,
-		EndTime:          0,
+func (f FileUploadStopExport) SortVal() (int64, string) {
+	// file messages should come after the message they were with
+	return f.UploadStopTime, EnsureLastMessageInASort
+}
+
+func ActianceExport(rctx request.CTX, p shared.ExportParams) (shared.RunExportResults, error) {
+	start := time.Now()
+
+	// Build the channel exports for the channels that had post or user join/leave activity this batch.
+	exportData, err := shared.GetGenericExportData(p)
+	if err != nil {
+		return exportData.Results, err
 	}
-	elementsByChannel := map[string][]any{}
-	allUploadedFiles := []*model.FileInfo{}
+	var allUploadedFiles []*model.FileInfo
 
-	for _, post := range posts {
-		if post == nil {
-			rctx.Logger().Warn("ignored a nil post reference in the list")
-			continue
+	// Convert the generic shared.ChannelExports to the Actiance-specific ChannelExports data.
+	channelExports := make([]ChannelExport, 0, len(exportData.Exports))
+	for _, channel := range exportData.Exports {
+		joinEvents := make([]JoinExport, 0, len(channel.JoinEvents))
+		leaveEvents := make([]LeaveExport, 0, len(channel.LeaveEvents))
+
+		for _, j := range channel.JoinEvents {
+			joinEvents = append(joinEvents, JoinExport{
+				UserEmail:        j.UserEmail,
+				UserType:         string(j.UserType),
+				JoinTime:         j.JoinTime,
+				CorporateEmailID: j.UserEmail,
+			})
 		}
-		elementsByChannel[*post.ChannelId] = append(elementsByChannel[*post.ChannelId], postToExportEntry(post, post.PostCreateAt, *post.PostMessage))
-
-		if post.PostDeleteAt != nil && *post.PostDeleteAt > 0 && post.PostProps != nil {
-			props := map[string]any{}
-			if json.Unmarshal([]byte(*post.PostProps), &props) == nil {
-				if _, ok := props[model.PostPropsDeleteBy]; ok {
-					elementsByChannel[*post.ChannelId] = append(elementsByChannel[*post.ChannelId], postToExportEntry(post,
-						post.PostDeleteAt, "delete "+*post.PostMessage))
-				}
-			}
+		for _, l := range channel.LeaveEvents {
+			leaveEvents = append(leaveEvents, LeaveExport{
+				UserEmail:        l.UserEmail,
+				UserType:         string(l.UserType),
+				LeaveTime:        l.LeaveTime,
+				CorporateEmailID: l.UserEmail,
+			})
 		}
 
-		startUploads, stopUploads, uploadedFiles, deleteFileMessages, err := postToAttachmentsEntries(post, db)
-		if err != nil {
-			return warningCount, err
+		elements := make([]Sortable, 0, len(channel.Posts)+len(channel.DeletedFiles)+len(channel.UploadStarts)+len(channel.UploadStops))
+		for _, p := range channel.Posts {
+			elements = append(elements, PostExport{
+				MessageId:      *p.PostId,
+				UserEmail:      *p.UserEmail,
+				UserType:       p.UserType,
+				CreateAt:       *p.PostCreateAt,
+				UpdatedType:    p.UpdatedType,
+				UpdateAt:       p.UpdateAt,
+				EditedNewMsgId: p.EditedNewMsgId,
+				Message:        p.Message,
+				PreviewsPost:   p.PreviewsPost,
+			})
 		}
-		elementsByChannel[*post.ChannelId] = append(elementsByChannel[*post.ChannelId], startUploads...)
-		elementsByChannel[*post.ChannelId] = append(elementsByChannel[*post.ChannelId], stopUploads...)
-		elementsByChannel[*post.ChannelId] = append(elementsByChannel[*post.ChannelId], deleteFileMessages...)
-
-		allUploadedFiles = append(allUploadedFiles, uploadedFiles...)
-
-		metadata.Update(post, len(uploadedFiles))
-
-		if _, ok := membersByChannel[*post.ChannelId]; !ok {
-			membersByChannel[*post.ChannelId] = common_export.ChannelMembers{}
+		for _, p := range channel.DeletedFiles {
+			elements = append(elements, PostExport{
+				MessageId:      *p.PostId,
+				UserEmail:      *p.UserEmail,
+				UserType:       p.UserType,
+				CreateAt:       *p.PostCreateAt,
+				UpdatedType:    p.UpdatedType,
+				UpdateAt:       p.UpdateAt,
+				EditedNewMsgId: p.EditedNewMsgId,
+				Message:        p.Message,
+				PreviewsPost:   p.PreviewsPost,
+			})
 		}
-		membersByChannel[*post.ChannelId][*post.UserId] = common_export.ChannelMember{
-			Email:    *post.UserEmail,
-			UserId:   *post.UserId,
-			IsBot:    post.IsBot,
-			Username: *post.Username,
+		for _, u := range channel.UploadStarts {
+			elements = append(elements, FileUploadStartExport{
+				UserEmail:       u.UserEmail,
+				UploadStartTime: u.UploadStartTime,
+				Filename:        u.FileInfo.Name,
+				FilePath:        u.FileInfo.Path,
+			})
 		}
-	}
-
-	rctx.Logger().Info("Exported data for channels", mlog.Int("number_of_channels", len(metadata.Channels)))
-
-	channelExports := []ChannelExport{}
-	for _, channel := range metadata.Channels {
-		channelExport, err := buildChannelExport(
-			channel,
-			membersByChannel[channel.ChannelId],
-			elementsByChannel[channel.ChannelId],
-			db,
-		)
-		if err != nil {
-			return warningCount, err
+		for _, u := range channel.UploadStops {
+			elements = append(elements, FileUploadStopExport{
+				UserEmail:      u.UserEmail,
+				UploadStopTime: u.UploadStopTime,
+				Filename:       u.FileInfo.Name,
+				FilePath:       u.FileInfo.Path,
+				Status:         u.Status,
+			})
 		}
-		channelExports = append(channelExports, *channelExport)
+
+		// We need to sort all the elements by (updateAt, messageId) because they were added by type above.
+		slices.SortStableFunc(elements, SortableSort)
+
+		channelExports = append(channelExports, ChannelExport{
+			Perspective: channel.DisplayName,
+			ChannelId:   channel.ChannelId,
+			RoomId: fmt.Sprintf("%v - %v - %v", shared.ChannelTypeDisplayName(channel.ChannelType),
+				channel.ChannelName, channel.ChannelId),
+			StartTime:   channel.StartTime,
+			JoinEvents:  joinEvents,
+			Elements:    elements,
+			LeaveEvents: leaveEvents,
+			EndTime:     channel.EndTime,
+		})
+
+		allUploadedFiles = append(allUploadedFiles, channel.Files...)
 	}
 
 	export := &RootNode{
@@ -175,182 +252,113 @@ func ActianceExport(rctx request.CTX, posts []*model.MessageExport, db store.Sto
 		Channels: channelExports,
 	}
 
-	return writeExport(rctx, export, allUploadedFiles, exportDirectory, exportBackend, fileAttachmentBackend)
+	results := exportData.Results
+	results.ProcessingPostsMs = time.Since(start).Milliseconds()
+
+	results.WriteExportResult, err = writeExport(rctx, export, allUploadedFiles, p.ExportBackend, p.FileAttachmentBackend, p.BatchPath)
+	results.NumChannels = len(channelExports)
+	return results, err
 }
 
-func postToExportEntry(post *model.MessageExport, createTime *int64, message string) *PostExport {
-	userType := "user"
-	if post.IsBot {
-		userType = "bot"
-	}
-	return &PostExport{
-		PostTime:     *createTime,
-		Message:      message,
-		UserType:     userType,
-		UserEmail:    *post.UserEmail,
-		PreviewsPost: post.PreviewID(),
-	}
-}
-
-func postToAttachmentsEntries(post *model.MessageExport, db store.Store) ([]any, []any, []*model.FileInfo, []any, *model.AppError) {
-	// if the post included any files, we need to add special elements to the export.
-	if len(post.PostFileIds) == 0 {
-		return nil, nil, nil, nil, nil
-	}
-
-	fileInfos, err := db.FileInfo().GetForPost(*post.PostId, true, true, false)
-	if err != nil {
-		return nil, nil, nil, nil, model.NewAppError("postToAttachmentsEntries", "ent.message_export.actiance_export.get_attachment_error", nil, "", http.StatusInternalServerError).Wrap(err)
-	}
-
-	startUploads := []any{}
-	stopUploads := []any{}
-	deleteFileMessages := []any{}
-
-	uploadedFiles := []*model.FileInfo{}
-	for _, fileInfo := range fileInfos {
-		// insert a record of the file upload into the export file
-		// path to exported file is relative to the xml file, so it's just the name of the exported file
-		startUploads = append(startUploads, &FileUploadStartExport{
-			UserEmail:       *post.UserEmail,
-			Filename:        fileInfo.Name,
-			FilePath:        fileInfo.Path,
-			UploadStartTime: *post.PostCreateAt,
-		})
-
-		stopUploads = append(stopUploads, &FileUploadStopExport{
-			UserEmail:      *post.UserEmail,
-			Filename:       fileInfo.Name,
-			FilePath:       fileInfo.Path,
-			UploadStopTime: *post.PostCreateAt,
-			Status:         "Completed",
-		})
-
-		if fileInfo.DeleteAt > 0 && post.PostDeleteAt != nil {
-			deleteFileMessages = append(deleteFileMessages, postToExportEntry(post, post.PostDeleteAt, "delete "+fileInfo.Path))
-		}
-
-		uploadedFiles = append(uploadedFiles, fileInfo)
-	}
-	return startUploads, stopUploads, uploadedFiles, deleteFileMessages, nil
-}
-
-func buildChannelExport(channel common_export.MetadataChannel, members common_export.ChannelMembers, elements []any, db store.Store) (*ChannelExport, *model.AppError) {
-	channelExport := ChannelExport{
-		ChannelId:   channel.ChannelId,
-		RoomId:      fmt.Sprintf("%v - %v - %v", common_export.ChannelTypeDisplayName(channel.ChannelType), channel.ChannelName, channel.ChannelId),
-		StartTime:   channel.StartTime,
-		EndTime:     channel.EndTime,
-		Perspective: channel.ChannelDisplayName,
-	}
-
-	channelMembersHistory, err := db.ChannelMemberHistory().GetUsersInChannelDuring(channel.StartTime, channel.EndTime, channel.ChannelId)
-	if err != nil {
-		return nil, model.NewAppError("buildChannelExport", "ent.get_users_in_channel_during", nil, "", http.StatusInternalServerError).Wrap(err)
-	}
-
-	joins, leaves := common_export.GetJoinsAndLeavesForChannel(channel.StartTime, channel.EndTime, channelMembersHistory, members)
-	type StillJoinedInfo struct {
-		Time int64
-		Type string
-	}
-	stillJoined := map[string]StillJoinedInfo{}
-	for _, join := range joins {
-		userType := "user"
-		if join.IsBot {
-			userType = "bot"
-		}
-		channelExport.JoinEvents = append(channelExport.JoinEvents, JoinExport{
-			JoinTime:         join.Datetime,
-			UserEmail:        join.Email,
-			UserType:         userType,
-			CorporateEmailID: join.Email,
-		})
-		if value, ok := stillJoined[join.Email]; !ok {
-			stillJoined[join.Email] = StillJoinedInfo{Time: join.Datetime, Type: userType}
-		} else {
-			if join.Datetime > value.Time {
-				stillJoined[join.Email] = StillJoinedInfo{Time: join.Datetime, Type: userType}
-			}
-		}
-	}
-	for _, leave := range leaves {
-		userType := "user"
-		if leave.IsBot {
-			userType = "bot"
-		}
-		channelExport.LeaveEvents = append(channelExport.LeaveEvents, LeaveExport{
-			LeaveTime:        leave.Datetime,
-			UserEmail:        leave.Email,
-			UserType:         userType,
-			CorporateEmailID: leave.Email,
-		})
-		if leave.Datetime > stillJoined[leave.Email].Time {
-			delete(stillJoined, leave.Email)
-		}
-	}
-
-	for email := range stillJoined {
-		channelExport.LeaveEvents = append(channelExport.LeaveEvents, LeaveExport{
-			LeaveTime:        channel.EndTime,
-			UserEmail:        email,
-			UserType:         stillJoined[email].Type,
-			CorporateEmailID: email,
-		})
-	}
-
-	sort.Slice(channelExport.LeaveEvents, func(i, j int) bool {
-		if channelExport.LeaveEvents[i].LeaveTime == channelExport.LeaveEvents[j].LeaveTime {
-			return channelExport.LeaveEvents[i].UserEmail < channelExport.LeaveEvents[j].UserEmail
-		}
-		return channelExport.LeaveEvents[i].LeaveTime < channelExport.LeaveEvents[j].LeaveTime
-	})
-
-	channelExport.Elements = elements
-	return &channelExport, nil
-}
-
-func writeExport(rctx request.CTX, export *RootNode, uploadedFiles []*model.FileInfo, exportDirectory string, exportBackend filestore.FileBackend, fileAttachmentBackend filestore.FileBackend) (warningCount int64, appErr *model.AppError) {
+func writeExport(rctx request.CTX, export *RootNode, uploadedFiles []*model.FileInfo, exportBackend filestore.FileBackend, fileAttachmentBackend filestore.FileBackend, batchPath string) (res shared.WriteExportResult, err error) {
+	start := time.Now()
 	// marshal the export object to xml
 	xmlData := &bytes.Buffer{}
 	xmlData.WriteString(xml.Header)
 
 	enc := xml.NewEncoder(xmlData)
 	enc.Indent("", "  ")
-	if err := enc.Encode(export); err != nil {
-		return warningCount, model.NewAppError("ActianceExport.AtianceExport", "ent.actiance.export.marshalToXml.appError", nil, "", 0).Wrap(err)
+	if err = enc.Encode(export); err != nil {
+		return res, fmt.Errorf("unable to convert export to XML: %w", err)
+	}
+	if err = enc.Flush(); err != nil {
+		return res, fmt.Errorf("unable to flush the XML encoder: %w", err)
 	}
-	enc.Flush()
 
-	// Try to disable the write timeout if the backend supports it
-	if _, err := filestore.TryWriteFileContext(rctx.Context(), exportBackend, xmlData, path.Join(exportDirectory, ActianceExportFilename)); err != nil {
-		return warningCount, model.NewAppError("ActianceExport.AtianceExport", "ent.actiance.export.write_file.appError", nil, "", 0).Wrap(err)
+	// Write this batch to a tmp zip, then copy the zip to the export directory.
+	// Using a 2M buffer because the file backend may be s3 and this optimizes speed and
+	// memory usage, see: https://github.com/mattermost/mattermost/pull/26629
+	buf := make([]byte, 1024*1024*2)
+	temp, err := os.CreateTemp("", "compliance-export-batch-*.zip")
+	if err != nil {
+		return res, fmt.Errorf("unable to create the batch temporary file: %w", err)
 	}
+	defer file.DeleteTemp(rctx.Logger(), temp)
+
+	zipFile := zip.NewWriter(temp)
+	w, err := zipFile.Create(ActianceExportFilename)
+	if err != nil {
+		return res, fmt.Errorf("unable to create the xml file in the zipFile created with the batch temporary file: %w", err)
+	}
+	if _, err = io.CopyBuffer(w, xmlData, buf); err != nil {
+		return res, fmt.Errorf("unable to write into the zipFile created with the batch temporary file: %w", err)
+	}
+	res.ProcessingXmlMs = time.Since(start).Milliseconds()
+
+	start = time.Now()
 
 	var missingFiles []string
 	for _, fileInfo := range uploadedFiles {
-		var attachmentSrc io.ReadCloser
-		attachmentSrc, nErr := fileAttachmentBackend.Reader(fileInfo.Path)
-		if nErr != nil {
-			missingFiles = append(missingFiles, "Warning:"+common_export.MissingFileMessage+" - "+fileInfo.Path)
-			rctx.Logger().Warn(common_export.MissingFileMessage, mlog.String("FileName", fileInfo.Path))
+		var attachmentReader io.ReadCloser
+		attachmentReader, err = fileAttachmentBackend.Reader(fileInfo.Path)
+		if err != nil {
+			missingFiles = append(missingFiles, "Warning:"+shared.MissingFileMessage+" - "+fileInfo.Path)
+			rctx.Logger().Warn(shared.MissingFileMessage, mlog.String("filename", fileInfo.Path))
 			continue
 		}
-		defer attachmentSrc.Close()
 
-		destPath := path.Join(exportDirectory, fileInfo.Path)
+		// There could be many uploadedFiles, so be careful about closing readers.
+		if err = func() error {
+			defer attachmentReader.Close()
+			var zipWriter io.Writer
+			zipWriter, err = zipFile.Create(fileInfo.Path)
+			if err != nil {
+				return err
+			}
 
-		_, nErr = exportBackend.WriteFile(attachmentSrc, destPath)
-		if nErr != nil {
-			return warningCount, model.NewAppError("ActianceExport.AtianceExport", "ent.actiance.export.write_file.appError", nil, "", 0).Wrap(nErr)
+			// CopyBuffer works with dirty buffers, no need to clear it.
+			if _, err = io.CopyBuffer(zipWriter, attachmentReader, buf); err != nil {
+				return err
+			}
+
+			return nil
+		}(); err != nil {
+			return res, fmt.Errorf("unable to write into the zipFile created with the batch temporary file: %w", err)
 		}
 	}
-	warningCount = int64(len(missingFiles))
-	if warningCount > 0 {
-		_, err := filestore.TryWriteFileContext(rctx.Context(), exportBackend, strings.NewReader(strings.Join(missingFiles, "\n")), path.Join(exportDirectory, ActianceWarningFilename))
+
+	res.TransferringFilesMs = time.Since(start).Milliseconds()
+	res.NumWarnings = len(missingFiles)
+	if res.NumWarnings > 0 {
+		var w io.Writer
+		w, err = zipFile.Create(ActianceWarningFilename)
 		if err != nil {
-			appErr = model.NewAppError("ActianceExport.AtianceExport", "ent.actiance.export.write_file.appError", nil, "", 0).Wrap(err)
+			return res, fmt.Errorf("unable to create the warning file in the zipFile created with the batch temporary file: %w", err)
+		}
+		r := strings.NewReader(strings.Join(missingFiles, "\n"))
+		if _, err = io.CopyBuffer(w, r, buf); err != nil {
+			return res, fmt.Errorf("unable to write into the zipFile created with the batch temporary file: %w", err)
 		}
 	}
-	return warningCount, appErr
+
+	if err = zipFile.Close(); err != nil {
+		return res, fmt.Errorf("unable to close the zipFile created with the batch temporary file: %w", err)
+	}
+
+	_, err = temp.Seek(0, io.SeekStart)
+	if err != nil {
+		return res, fmt.Errorf("unable to seek to the beginning of the the batch temporary file: %w", err)
+	}
+
+	start = time.Now()
+
+	// Try to write the file without a timeout due to the potential size of the file.
+	_, err = filestore.TryWriteFileContext(rctx.Context(), exportBackend, temp, batchPath)
+	if err != nil {
+		return res, fmt.Errorf("unable to transfer the batch zip to the file backend: %w", err)
+	}
+
+	res.TransferringZipMs = time.Since(start).Milliseconds()
+
+	return res, nil
 }

server/enterprise/message_export/actiance_export/main_test.go

@@ -0,0 +1,21 @@
+// Copyright (c) 2015-present Mattermost, Inc. All Rights Reserved.
+// See LICENSE.enterprise for license information.
+
+package actiance_export
+
+import (
+	"testing"
+
+	"github.com/mattermost/mattermost/server/v8/channels/api4"
+	"github.com/mattermost/mattermost/server/v8/channels/testlib"
+)
+
+var mainHelper *testlib.MainHelper
+
+func TestMain(m *testing.M) {
+	mainHelper = testlib.NewMainHelper()
+	defer mainHelper.Close()
+	api4.SetMainHelper(mainHelper)
+
+	mainHelper.Main(m)
+}

server/enterprise/message_export/actiance_export/test_helpers.go

@@ -0,0 +1,49 @@
+// Copyright (c) 2015-present Mattermost, Inc. All Rights Reserved.
+// See LICENSE.enterprise for license information.
+
+package actiance_export
+
+import (
+	"encoding/xml"
+	"io"
+	"testing"
+
+	"github.com/stretchr/testify/require"
+)
+
+type ChannelExportWithSpecifics struct {
+	XMLName     xml.Name                 `xml:"Conversation"`
+	Perspective string                   `xml:"Perspective,attr"`
+	ChannelId   string                   `xml:"-"`
+	RoomId      string                   `xml:"RoomID"`
+	StartTime   int64                    `xml:"StartTimeUTC"`
+	JoinEvents  []*JoinExport            `xml:"ParticipantEntered"`
+	Messages    []*PostExport            `xml:"Message"`
+	FileStarts  []*FileUploadStartExport `xml:"FileTransferStarted"`
+	FileStops   []*FileUploadStopExport  `xml:"FileTransferEnded"`
+	LeaveEvents []*LeaveExport           `xml:"ParticipantLeft"`
+	EndTime     int64                    `xml:"EndTimeUTC"`
+}
+
+func GetChannelExports(t *testing.T, r io.Reader) []*ChannelExportWithSpecifics {
+	decoder := xml.NewDecoder(r)
+	var exportedChannels []*ChannelExportWithSpecifics
+	for {
+		token, err := decoder.Token()
+		if token == nil || err != nil {
+			break
+		}
+		switch se := token.(type) {
+		case xml.StartElement:
+			if se.Name.Local == "Conversation" {
+				var a *ChannelExportWithSpecifics
+				err = decoder.DecodeElement(&a, &se)
+				require.NoError(t, err)
+				exportedChannels = append(exportedChannels, a)
+			}
+		default:
+		}
+	}
+
+	return exportedChannels
+}

server/enterprise/message_export/common_export/common_export.go

@@ -5,8 +5,13 @@ package common_export
 
 import (
 	"fmt"
+	"strconv"
+
+	"github.com/mattermost/mattermost/server/v8/enterprise/message_export/shared"
 
 	"github.com/mattermost/mattermost/server/public/model"
+	"github.com/mattermost/mattermost/server/public/shared/mlog"
+	"github.com/mattermost/mattermost/server/public/shared/request"
 )
 
 const MissingFileMessage = "File missing for post; cannot copy file to archive"
@@ -33,8 +38,6 @@ type ChannelMember struct {
 	Email    string
 	Username string
 }
-type ChannelMembers map[string]ChannelMember
-type MembersByChannel map[string]ChannelMembers
 
 type MetadataChannel struct {
 	TeamId             *string
@@ -52,7 +55,7 @@ type MetadataChannel struct {
 }
 
 type Metadata struct {
-	Channels         map[string]MetadataChannel
+	Channels         map[string]*MetadataChannel
 	MessagesCount    int
 	AttachmentsCount int
 	StartTime        int64
@@ -62,7 +65,7 @@ type Metadata struct {
 func (metadata *Metadata) Update(post *model.MessageExport, attachments int) {
 	channelMetadata, ok := metadata.Channels[*post.ChannelId]
 	if !ok {
-		channelMetadata = MetadataChannel{
+		channelMetadata = &MetadataChannel{
 			TeamId:             post.TeamId,
 			TeamName:           post.TeamName,
 			TeamDisplayName:    post.TeamDisplayName,
@@ -89,11 +92,93 @@ func (metadata *Metadata) Update(post *model.MessageExport, attachments int) {
 	metadata.Channels[*post.ChannelId] = channelMetadata
 }
 
-func GetJoinsAndLeavesForChannel(startTime int64, endTime int64, channelMembersHistory []*model.ChannelMemberHistoryResult, channelMembers ChannelMembers) ([]ChannelMemberJoin, []ChannelMemberLeave) {
-	joins := []ChannelMemberJoin{}
-	leaves := []ChannelMemberLeave{}
+type ChannelExportsParams struct {
+	Store                   shared.MessageExportStore
+	ExportPeriodStartTime   int64
+	ExportPeriodEndTime     int64
+	ChannelBatchSize        int
+	ChannelHistoryBatchSize int
+	ReportProgressMessage   func(message string)
+}
 
-	alreadyJoined := map[string]bool{}
+// CalculateChannelExports returns the channel info ( map[channelId]*MetadataChannel ) and the channel user
+// joins/leaves ( map[channelId][]*model.ChannelMemberHistoryResult ) for any channel that has had activity
+// (posts or user join/leaves) between ExportPeriodStartTime and ExportPeriodEndTime.
+func CalculateChannelExports(rctx request.CTX, opt ChannelExportsParams) (map[string]*MetadataChannel, map[string][]*model.ChannelMemberHistoryResult, error) {
+	// Which channels had user activity in the export period?
+	activeChannelIds, err := opt.Store.ChannelMemberHistory().GetChannelsWithActivityDuring(opt.ExportPeriodStartTime, opt.ExportPeriodEndTime)
+	if err != nil {
+		return nil, nil, err
+	}
+
+	if len(activeChannelIds) == 0 {
+		return nil, nil, nil
+	}
+
+	rctx.Logger().Debug("Started CalculateChannelExports", mlog.Int("export_period_start_time", opt.ExportPeriodStartTime), mlog.Int("export_period_end_time", opt.ExportPeriodEndTime), mlog.Int("num_active_channel_ids", len(activeChannelIds)))
+	message := rctx.T("ent.message_export.actiance_export.calculate_channel_exports.channel_message", model.StringMap{"NumChannels": strconv.Itoa(len(activeChannelIds))})
+	opt.ReportProgressMessage(message)
+
+	// For each channel, get its metadata.
+	channelMetadata := make(map[string]*MetadataChannel, len(activeChannelIds))
+
+	// Use batches to reduce db load and network waste.
+	for pos := 0; pos < len(activeChannelIds); pos += opt.ChannelBatchSize {
+		upTo := min(pos+opt.ChannelBatchSize, len(activeChannelIds))
+		batch := activeChannelIds[pos:upTo]
+		channels, err := opt.Store.Channel().GetMany(batch, true)
+		if err != nil {
+			return nil, nil, err
+		}
+
+		for _, channel := range channels {
+			channelMetadata[channel.Id] = &MetadataChannel{
+				TeamId:             model.NewPointer(channel.TeamId),
+				ChannelId:          channel.Id,
+				ChannelName:        channel.Name,
+				ChannelDisplayName: channel.DisplayName,
+				ChannelType:        channel.Type,
+				RoomId:             fmt.Sprintf("%v - %v", ChannelTypeDisplayName(channel.Type), channel.Id),
+				StartTime:          opt.ExportPeriodStartTime,
+				EndTime:            opt.ExportPeriodEndTime,
+			}
+		}
+	}
+
+	historiesByChannelId := make(map[string][]*model.ChannelMemberHistoryResult, len(activeChannelIds))
+
+	// Now that we have metadata, get channelMemberHistories for each channel.
+	// Use batches to reduce total db load and network waste.
+	for pos := 0; pos < len(activeChannelIds); pos += opt.ChannelHistoryBatchSize {
+		// This may take a while, so update the system console UI.
+		message := rctx.T("ent.message_export.actiance_export.calculate_channel_exports.activity_message", model.StringMap{
+			"NumChannels":  strconv.Itoa(len(activeChannelIds)),
+			"NumCompleted": strconv.Itoa(pos),
+		})
+		opt.ReportProgressMessage(message)
+
+		upTo := min(pos+opt.ChannelHistoryBatchSize, len(activeChannelIds))
+		batch := activeChannelIds[pos:upTo]
+		channelMemberHistories, err := opt.Store.ChannelMemberHistory().GetUsersInChannelDuring(opt.ExportPeriodStartTime, opt.ExportPeriodEndTime, batch)
+		if err != nil {
+			return nil, nil, err
+		}
+
+		// collect the channelMemberHistories by channelId
+		for _, entry := range channelMemberHistories {
+			historiesByChannelId[entry.ChannelId] = append(historiesByChannelId[entry.ChannelId], entry)
+		}
+	}
+
+	return channelMetadata, historiesByChannelId, nil
+}
+
+func GetJoinsAndLeavesForChannel(startTime int64, endTime int64, channelMembersHistory []*model.ChannelMemberHistoryResult,
+	postAuthors map[string]ChannelMember) ([]ChannelMemberJoin, []ChannelMemberLeave) {
+	var joins []ChannelMemberJoin
+	var leaves []ChannelMemberLeave
+
+	alreadyJoined := make(map[string]bool)
 	for _, cmh := range channelMembersHistory {
 		if cmh.UserDeleteAt > 0 && cmh.UserDeleteAt < startTime {
 			continue
@@ -129,7 +214,7 @@ func GetJoinsAndLeavesForChannel(startTime int64, endTime int64, channelMembersH
 		}
 	}
 
-	for _, member := range channelMembers {
+	for _, member := range postAuthors {
 		if alreadyJoined[member.UserId] {
 			continue
 		}

server/enterprise/message_export/common_export/common_export_test.go

@@ -13,7 +13,7 @@ import (
 
 func TestUpdateMetadata(t *testing.T) {
 	metadata := Metadata{
-		Channels:         map[string]MetadataChannel{},
+		Channels:         map[string]*MetadataChannel{},
 		MessagesCount:    0,
 		AttachmentsCount: 0,
 		StartTime:        0,
@@ -92,7 +92,7 @@ func TestGetJoinsAndLeavesForChannel(t *testing.T) {
 		name           string
 		channel        MetadataChannel
 		membersHistory []*model.ChannelMemberHistoryResult
-		usersInPosts   ChannelMembers
+		usersInPosts   map[string]ChannelMember
 		expectedJoins  int
 		expectedLeaves int
 	}{
@@ -160,9 +160,9 @@ func TestGetJoinsAndLeavesForChannel(t *testing.T) {
 			name:           "implicit-joins",
 			channel:        channel,
 			membersHistory: nil,
-			usersInPosts: ChannelMembers{
-				"test1": ChannelMember{UserId: "test1", Email: "test1", Username: "test1"},
-				"test2": ChannelMember{UserId: "test2", Email: "test2", Username: "test2"},
+			usersInPosts: map[string]ChannelMember{
+				"test1": {UserId: "test1", Email: "test1", Username: "test1"},
+				"test2": {UserId: "test2", Email: "test2", Username: "test2"},
 			},
 			expectedJoins:  2,
 			expectedLeaves: 0,
@@ -174,9 +174,9 @@ func TestGetJoinsAndLeavesForChannel(t *testing.T) {
 				{JoinTime: 130, LeaveTime: model.NewPointer(int64(150)), UserId: "test1", UserEmail: "test1", Username: "test1"},
 				{JoinTime: 130, LeaveTime: model.NewPointer(int64(150)), UserId: "test3", UserEmail: "test3", Username: "test3"},
 			},
-			usersInPosts: ChannelMembers{
-				"test1": ChannelMember{UserId: "test1", Email: "test1", Username: "test1"},
-				"test2": ChannelMember{UserId: "test2", Email: "test2", Username: "test2"},
+			usersInPosts: map[string]ChannelMember{
+				"test1": {UserId: "test1", Email: "test1", Username: "test1"},
+				"test2": {UserId: "test2", Email: "test2", Username: "test2"},
 			},
 			expectedJoins:  3,
 			expectedLeaves: 2,

server/enterprise/message_export/csv_export/csv_export.go

@@ -9,18 +9,18 @@ import (
 	"encoding/json"
 	"fmt"
 	"io"
-	"net/http"
 	"os"
 	"path"
-	"sort"
+	"slices"
 	"strconv"
+	"strings"
 
-	"github.com/mattermost/mattermost/server/v8/enterprise/message_export/common_export"
+	"github.com/mattermost/mattermost/server/v8/enterprise/internal/file"
+	"github.com/mattermost/mattermost/server/v8/enterprise/message_export/shared"
 
 	"github.com/mattermost/mattermost/server/public/model"
 	"github.com/mattermost/mattermost/server/public/shared/mlog"
 	"github.com/mattermost/mattermost/server/public/shared/request"
-	"github.com/mattermost/mattermost/server/v8/channels/store"
 	"github.com/mattermost/mattermost/server/v8/platform/shared/filestore"
 )
 
@@ -28,26 +28,87 @@ const (
 	EnterPostType            = "enter"
 	LeavePostType            = "leave"
 	PreviouslyJoinedPostType = "previously-joined"
-	CSVExportFilename        = "csv_export.zip"
 	CSVWarningFilename       = "warning.txt"
 )
 
-func CsvExport(rctx request.CTX, posts []*model.MessageExport, db store.Store, exportBackend filestore.FileBackend, fileAttachmentBackend filestore.FileBackend, exportDirectory string) (warningCount int64, appErr *model.AppError) {
-	dest, err := os.CreateTemp("", CSVExportFilename)
+type Row struct {
+	CreateAt           int64
+	UpdateAt           int64
+	UpdateType         shared.PostUpdatedType
+	TeamId             string
+	TeamName           string
+	TeamDisplayName    string
+	ChannelId          string
+	ChannelName        string
+	ChannelDisplayName string
+	ChannelType        string
+	UserId             string
+	UserEmail          string
+	Username           string
+	PostId             string
+	EditedByPostId     string
+	RepliedToPostId    string
+	PostMessage        string
+	PostType           string
+	UserType           string
+	PreviewsPostId     string
+}
+
+func CsvExport(rctx request.CTX, p shared.ExportParams) (shared.RunExportResults, error) {
+	// Build the channel exports for the channels that had post or user join/leave activity this batch.
+	exportData, err := shared.GetGenericExportData(p)
+	results := exportData.Results
 	if err != nil {
-		return warningCount, model.NewAppError("CsvExport", "ent.compliance.csv.file.creation.appError", nil, "", 0).Wrap(err)
+		return results, err
 	}
-	defer os.Remove(dest.Name())
 
-	zipFile := zip.NewWriter(dest)
+	totalRows := results.CreatedPosts + results.EditedOrigMsgPosts + results.DeletedPosts + results.EditedNewMsgPosts +
+		results.UpdatedPosts + results.UploadedFiles + results.DeletedFiles + results.Joins
+	rows := make([]Row, 0, totalRows)
+	for _, channel := range exportData.Exports {
+		rows = append(rows, getJoinLeavePosts(channel)...)
+		for _, p := range channel.Posts {
+			rows = append(rows, postToRow(p, "message", p.PostCreateAt, p.Message))
+		}
 
+		for _, u := range channel.UploadStarts {
+			rows = append(rows, attachmentToRow(shared.UploadStartToExportEntry(u)))
+		}
+		for _, d := range channel.DeletedFiles {
+			rows = append(rows, attachmentToRow(d))
+		}
+	}
+
+	// We need to sort all the elements by (CreateAt, PostId) because they were added by type and by channel above.
+	slices.SortStableFunc(rows, func(a, b Row) int {
+		if a.CreateAt == b.CreateAt {
+			return strings.Compare(a.PostId, b.PostId)
+		}
+		return int(a.CreateAt - b.CreateAt)
+	})
+
+	// We've got the data, now its write time:
+
+	// Write this batch to a tmp zip, then copy the zip to the export directory.
+	// Using a 2M buffer because the file backend may be s3 and this optimizes speed and
+	// memory usage, see: https://github.com/mattermost/mattermost/pull/26629
+	buf := make([]byte, 1024*1024*2)
+	temp, err := os.CreateTemp("", "compliance-export-batch-*.zip")
+	if err != nil {
+		return results, fmt.Errorf("unable to create temporary CSV export file: %w", err)
+	}
+	defer file.DeleteTemp(rctx.Logger(), temp)
+
+	zipFile := zip.NewWriter(temp)
 	csvFile, err := zipFile.Create("posts.csv")
 	if err != nil {
-		return warningCount, model.NewAppError("CsvExport", "ent.compliance.csv.zip.creation.appError", nil, "", 0).Wrap(err)
+		return results, fmt.Errorf("unable to create the zip export file: %w", err)
 	}
 	csvWriter := csv.NewWriter(csvFile)
 	err = csvWriter.Write([]string{
 		"Post Creation Time",
+		"Post Update Time",
+		"Post Update Type",
 		"Team Id",
 		"Team Name",
 		"Team Display Name",
@@ -68,343 +129,229 @@ func CsvExport(rctx request.CTX, posts []*model.MessageExport, db store.Store, e
 	})
 
 	if err != nil {
-		return warningCount, model.NewAppError("CsvExportPost", "ent.compliance.csv.header.export.appError", nil, "", 0).Wrap(err)
+		return results, fmt.Errorf("unable to add header to the CSV export: %w", err)
 	}
 
-	metadata := common_export.Metadata{
-		Channels:         map[string]common_export.MetadataChannel{},
-		MessagesCount:    0,
-		AttachmentsCount: 0,
-		StartTime:        0,
-		EndTime:          0,
-	}
-
-	membersByChannel := make(common_export.MembersByChannel)
-
-	for _, post := range posts {
-		attachments, err := getPostAttachments(db, post)
-		if err != nil {
-			return warningCount, err
-		}
-
-		if _, ok := membersByChannel[*post.ChannelId]; !ok {
-			membersByChannel[*post.ChannelId] = common_export.ChannelMembers{}
-		}
-
-		membersByChannel[*post.ChannelId][*post.UserId] = common_export.ChannelMember{
-			UserId:   *post.UserId,
-			Username: *post.Username,
-			IsBot:    post.IsBot,
-			Email:    *post.UserEmail,
-		}
-
-		metadata.Update(post, len(attachments))
-	}
-
-	joinLeavePosts, appErr2 := getJoinLeavePosts(metadata.Channels, membersByChannel, db)
-	if appErr2 != nil {
-		return warningCount, appErr2
-	}
-
-	postsGenerator := mergePosts(joinLeavePosts, posts)
-
-	for post := postsGenerator(); post != nil; post = postsGenerator() {
-		if err = csvWriter.Write(postToRow(post, post.PostCreateAt, *post.PostMessage)); err != nil {
-			return warningCount, model.NewAppError("CsvExportPost", "ent.compliance.csv.post.export.appError", nil, "", 0).Wrap(err)
-		}
-
-		if post.PostDeleteAt != nil && *post.PostDeleteAt > 0 && post.PostProps != nil {
-			props := map[string]any{}
-			if json.Unmarshal([]byte(*post.PostProps), &props) == nil {
-				if _, ok := props[model.PostPropsDeleteBy]; ok {
-					if err = csvWriter.Write(postToRow(post, post.PostDeleteAt, "delete "+*post.PostMessage)); err != nil {
-						return warningCount, model.NewAppError("CsvExportPost", "ent.compliance.csv.post.export.appError", nil, "", 0).Wrap(err)
-					}
-				}
-			}
-		}
-
-		var attachments []*model.FileInfo
-		attachments, appErr = getPostAttachments(db, post)
-		if appErr != nil {
-			return warningCount, appErr
-		}
-
-		for _, attachment := range attachments {
-			if err = csvWriter.Write(attachmentToRow(post, attachment)); err != nil {
-				return warningCount, model.NewAppError("CsvExport", "ent.compliance.csv.attachment.export.appError", nil, "", 0).Wrap(err)
-			}
+	for _, row := range rows {
+		if err = csvWriter.Write(rowToStringSlice(row)); err != nil {
+			return results, fmt.Errorf("unable to export a row: %w", err)
 		}
 	}
 
 	csvWriter.Flush()
 
 	var missingFiles []string
-	for _, post := range posts {
-		attachments, err := getPostAttachments(db, post)
+	for _, post := range p.Posts {
+		var attachments []*model.FileInfo
+		attachments, err = shared.GetPostAttachments(p.Db, post)
 		if err != nil {
-			return warningCount, err
+			return results, err
 		}
 
 		for _, attachment := range attachments {
-			var attachmentSrc io.ReadCloser
-			attachmentSrc, nErr := fileAttachmentBackend.Reader(attachment.Path)
+			var r io.ReadCloser
+			r, nErr := p.FileAttachmentBackend.Reader(attachment.Path)
 			if nErr != nil {
-				missingFiles = append(missingFiles, "Warning:"+common_export.MissingFileMessage+" - Post: "+*post.PostId+" - "+attachment.Path)
-				rctx.Logger().Warn(common_export.MissingFileMessage, mlog.String("PostId", *post.PostId), mlog.String("FileName", attachment.Path))
+				missingFiles = append(missingFiles, "Warning:"+shared.MissingFileMessage+" - Post: "+*post.PostId+" - "+attachment.Path)
+				rctx.Logger().Warn(shared.MissingFileMessage, mlog.String("post_id", *post.PostId), mlog.String("filename", attachment.Path))
 				continue
 			}
-			defer attachmentSrc.Close()
 
-			attachmentDst, err := zipFile.Create(path.Join("files", *post.PostId, fmt.Sprintf("%s-%s", attachment.Id, path.Base(attachment.Path))))
-			if err != nil {
-				return warningCount, model.NewAppError("CsvExport", "ent.compliance.csv.attachment.copy.appError", nil, "", 0).Wrap(err)
-			}
+			// Probably don't need to be this careful (see actiance_export.go), but may as well be consistent.
+			if err = func() error {
+				defer r.Close()
+				var attachmentDst io.Writer
+				attachmentDst, err = zipFile.Create(path.Join("files", *post.PostId, fmt.Sprintf("%s-%s", attachment.Id, path.Base(attachment.Path))))
+				if err != nil {
+					return err
+				}
 
-			_, err = io.Copy(attachmentDst, attachmentSrc)
-			if err != nil {
-				return warningCount, model.NewAppError("CsvExport", "ent.compliance.csv.attachment.copy.appError", nil, "", 0).Wrap(err)
+				_, err = io.CopyBuffer(attachmentDst, r, buf)
+				if err != nil {
+					return err
+				}
+
+				return nil
+			}(); err != nil {
+				return results, fmt.Errorf("unable to copy the attachment into the zip file: %w", err)
 			}
 		}
 	}
 
-	warningCount = int64(len(missingFiles))
-	if warningCount > 0 {
+	results.NumWarnings = len(missingFiles)
+	if results.NumWarnings > 0 {
 		metadataFile, _ := zipFile.Create(CSVWarningFilename)
 		for _, value := range missingFiles {
 			_, err = metadataFile.Write([]byte(value + "\n"))
 			if err != nil {
-				appErr = model.NewAppError("CsvExport", "ent.compliance.csv.warning.appError", nil, "", 0).Wrap(err)
-				return warningCount, appErr
+				return results, fmt.Errorf("unable to create the warning file: %w", err)
 			}
 		}
 	}
 
 	metadataFile, err := zipFile.Create("metadata.json")
 	if err != nil {
-		return warningCount, model.NewAppError("CsvExport", "ent.compliance.csv.metadata.json.zipfile.appError", nil, "", 0).Wrap(err)
+		return results, fmt.Errorf("unable to create the zip file: %w", err)
 	}
-	data, err := json.MarshalIndent(metadata, "", "  ")
+	data, err := json.MarshalIndent(exportData.Metadata, "", "  ")
 	if err != nil {
-		return warningCount, model.NewAppError("CsvExport", "ent.compliance.csv.metadata.json.marshalling.appError", nil, "", 0).Wrap(err)
+		return results, fmt.Errorf("unable to convert metadata to json: %w", err)
 	}
 	_, err = metadataFile.Write(data)
 	if err != nil {
-		return warningCount, model.NewAppError("CsvExport", "ent.compliance.csv.metadata.export.appError", nil, "", 0).Wrap(err)
+		return results, fmt.Errorf("unable to add metadata file to the zip file: %w", err)
 	}
 	err = zipFile.Close()
 	if err != nil {
-		return warningCount, model.NewAppError("CsvExport", "ent.compliance.csv.metadata.close.appError", nil, "", 0).Wrap(err)
+		return results, fmt.Errorf("unable to close the zip file: %w", err)
 	}
 
-	_, err = dest.Seek(0, 0)
+	_, err = temp.Seek(0, 0)
 	if err != nil {
-		return warningCount, model.NewAppError("CsvExport", "ent.compliance.csv.seek.appError", nil, "", 0).Wrap(err)
+		return results, fmt.Errorf("unable to seek to start of export file: %w", err)
 	}
+
 	// Try to write the file without a timeout due to the potential size of the file.
-	_, err = filestore.TryWriteFileContext(rctx.Context(), exportBackend, dest, path.Join(exportDirectory, CSVExportFilename))
+	_, err = filestore.TryWriteFileContext(rctx.Context(), p.ExportBackend, temp, p.BatchPath)
 	if err != nil {
-		return warningCount, model.NewAppError("CsvExport", "ent.compliance.csv.write_file.appError", nil, "", http.StatusInternalServerError).Wrap(err)
+		return results, fmt.Errorf("unable to write the csv file: %w", err)
 	}
-	return warningCount, appErr
+	return results, nil
 }
 
-func mergePosts(left []*model.MessageExport, right []*model.MessageExport) func() *model.MessageExport {
-	leftCursor := 0
-	rightCursor := 0
-	return func() *model.MessageExport {
-		if leftCursor >= len(left) && rightCursor >= len(right) {
-			return nil
+func getJoinLeavePosts(channel shared.ChannelExport) []Row {
+	var joinLeavePosts []Row
+
+	for _, join := range channel.JoinEvents {
+		enterMessage := fmt.Sprintf("User %s (%s) joined the channel", join.Username, join.UserEmail)
+		enterPostType := EnterPostType
+		if join.JoinTime <= channel.StartTime {
+			enterPostType = PreviouslyJoinedPostType
+			enterMessage = fmt.Sprintf("User %s (%s) was already in the channel", join.Username, join.UserEmail)
 		}
-
-		if leftCursor >= len(left) {
-			rightCursor += 1
-			return right[rightCursor-1]
-		}
-
-		if rightCursor >= len(right) {
-			leftCursor += 1
-			return left[leftCursor-1]
-		}
-
-		if *left[leftCursor].PostCreateAt <= *right[rightCursor].PostCreateAt {
-			leftCursor += 1
-			return left[leftCursor-1]
-		}
-
-		rightCursor += 1
-		return right[rightCursor-1]
-	}
-}
-
-func getJoinLeavePosts(channels map[string]common_export.MetadataChannel, membersByChannel common_export.MembersByChannel, db store.Store) ([]*model.MessageExport, *model.AppError) {
-	joinLeavePosts := []*model.MessageExport{}
-	for _, channel := range channels {
-		channelMembersHistory, err := db.ChannelMemberHistory().GetUsersInChannelDuring(channel.StartTime, channel.EndTime, channel.ChannelId)
-		if err != nil {
-			return nil, model.NewAppError("getJoinLeavePosts", "ent.get_users_in_channel_during", nil, "", http.StatusInternalServerError).Wrap(err)
-		}
-
-		joins, leaves := common_export.GetJoinsAndLeavesForChannel(channel.StartTime, channel.EndTime, channelMembersHistory, membersByChannel[channel.ChannelId])
-
-		for _, join := range joins {
-			enterMessage := fmt.Sprintf("User %s (%s) joined the channel", join.Username, join.Email)
-			enterPostType := EnterPostType
-			createAt := model.NewPointer(join.Datetime)
-			channelCopy := channel
-			if join.Datetime <= channel.StartTime {
-				enterPostType = PreviouslyJoinedPostType
-				enterMessage = fmt.Sprintf("User %s (%s) was already in the channel", join.Username, join.Email)
-				createAt = model.NewPointer(channel.StartTime)
-			}
-			joinLeavePosts = append(
-				joinLeavePosts,
-				&model.MessageExport{
-					TeamId:          channel.TeamId,
-					TeamName:        channel.TeamName,
-					TeamDisplayName: channel.TeamDisplayName,
-
-					ChannelId:          &channelCopy.ChannelId,
-					ChannelName:        &channelCopy.ChannelName,
-					ChannelDisplayName: &channelCopy.ChannelDisplayName,
-					ChannelType:        &channelCopy.ChannelType,
-
-					UserId:    model.NewPointer(join.UserId),
-					UserEmail: model.NewPointer(join.Email),
-					Username:  model.NewPointer(join.Username),
-					IsBot:     join.IsBot,
-
-					PostId:         model.NewPointer(""),
-					PostCreateAt:   createAt,
-					PostMessage:    &enterMessage,
-					PostType:       &enterPostType,
-					PostOriginalId: model.NewPointer(""),
-					PostFileIds:    []string{},
+		joinLeavePosts = append(
+			joinLeavePosts,
+			postToRow(shared.PostExport{
+				MessageExport: model.MessageExport{
+					TeamId:             &channel.TeamId,
+					TeamName:           &channel.TeamName,
+					TeamDisplayName:    &channel.TeamDisplayName,
+					ChannelId:          &channel.ChannelId,
+					ChannelName:        &channel.ChannelName,
+					ChannelDisplayName: &channel.DisplayName,
+					ChannelType:        &channel.ChannelType,
+					UserId:             &join.UserId,
+					UserEmail:          &join.UserEmail,
+					Username:           &join.Username,
+					IsBot:              join.UserType == shared.Bot,
+					PostId:             model.NewPointer(""),
+					PostCreateAt:       &join.JoinTime,
+					PostMessage:        &enterMessage,
+					PostType:           &enterPostType,
+					PostOriginalId:     model.NewPointer(""),
+					PostFileIds:        []string{},
 				},
-			)
+			}, enterPostType, &join.JoinTime, enterMessage),
+		)
+	}
+	for _, leave := range channel.LeaveEvents {
+		if leave.ClosedOut {
+			// csv does not record closed-out leaves; see export_data.go for further explanation.
+			continue
 		}
-		for _, leave := range leaves {
-			leaveMessage := fmt.Sprintf("User %s (%s) leaved the channel", leave.Username, leave.Email)
-			leavePostType := LeavePostType
-			channelCopy := channel
+		leaveMessage := fmt.Sprintf("User %s (%s) left the channel", leave.Username, leave.UserEmail)
+		leavePostType := LeavePostType
 
-			joinLeavePosts = append(
-				joinLeavePosts,
-				&model.MessageExport{
-					TeamId:          channel.TeamId,
-					TeamName:        channel.TeamName,
-					TeamDisplayName: channel.TeamDisplayName,
-
-					ChannelId:          &channelCopy.ChannelId,
-					ChannelName:        &channelCopy.ChannelName,
-					ChannelDisplayName: &channelCopy.ChannelDisplayName,
-					ChannelType:        &channelCopy.ChannelType,
-
-					UserId:    model.NewPointer(leave.UserId),
-					UserEmail: model.NewPointer(leave.Email),
-					Username:  model.NewPointer(leave.Username),
-					IsBot:     leave.IsBot,
-
-					PostId:         model.NewPointer(""),
-					PostCreateAt:   model.NewPointer(leave.Datetime),
-					PostMessage:    &leaveMessage,
-					PostType:       &leavePostType,
-					PostOriginalId: model.NewPointer(""),
-					PostFileIds:    []string{},
+		joinLeavePosts = append(
+			joinLeavePosts,
+			postToRow(shared.PostExport{
+				MessageExport: model.MessageExport{
+					TeamId:             &channel.TeamId,
+					TeamName:           &channel.TeamName,
+					TeamDisplayName:    &channel.TeamDisplayName,
+					ChannelId:          &channel.ChannelId,
+					ChannelName:        &channel.ChannelName,
+					ChannelDisplayName: &channel.DisplayName,
+					ChannelType:        &channel.ChannelType,
+					UserId:             &leave.UserId,
+					UserEmail:          &leave.UserEmail,
+					Username:           &leave.Username,
+					IsBot:              leave.UserType == shared.Bot,
+					PostId:             model.NewPointer(""),
+					PostCreateAt:       &leave.LeaveTime,
+					PostMessage:        &leaveMessage,
+					PostType:           &leavePostType,
+					PostOriginalId:     model.NewPointer(""),
+					PostFileIds:        []string{},
 				},
-			)
-		}
+			}, leavePostType, &leave.LeaveTime, leaveMessage),
+		)
 	}
 
-	sort.Slice(joinLeavePosts, func(i, j int) bool {
-		return *joinLeavePosts[i].PostCreateAt < *joinLeavePosts[j].PostCreateAt
-	})
-	return joinLeavePosts, nil
+	return joinLeavePosts
 }
 
-func getPostAttachments(db store.Store, post *model.MessageExport) ([]*model.FileInfo, *model.AppError) {
-	// if the post included any files, we need to add special elements to the export.
-	if len(post.PostFileIds) == 0 {
-		return []*model.FileInfo{}, nil
-	}
-
-	attachments, err := db.FileInfo().GetForPost(*post.PostId, true, true, false)
-	if err != nil {
-		return nil, model.NewAppError("getPostAttachments", "ent.message_export.csv_export.get_attachment_error", nil, "", http.StatusInternalServerError).Wrap(err)
-	}
-	return attachments, nil
-}
-
-func postToRow(post *model.MessageExport, createTime *int64, message string) []string {
-	teamId := ""
-	teamName := ""
-	teamDisplayName := ""
-	if post.TeamId != nil {
-		teamId = *post.TeamId
-	}
-	if post.TeamName != nil {
-		teamName = *post.TeamName
-	}
-	if post.TeamDisplayName != nil {
-		teamDisplayName = *post.TeamDisplayName
-	}
-	postType := "message"
-	if post.PostType != nil && *post.PostType != "" {
-		postType = *post.PostType
-	}
-	postRootId := ""
-	if post.PostRootId != nil {
-		postRootId = *post.PostRootId
-	}
+func postToRow(p shared.PostExport, postType string, createTime *int64, message string) Row {
 	userType := "user"
-	if post.IsBot {
+	if p.IsBot {
 		userType = "bot"
 	}
+	return Row{
+		CreateAt:           model.SafeDereference(createTime),
+		UpdateAt:           model.SafeDereference(p.PostUpdateAt),
+		UpdateType:         p.UpdatedType,
+		TeamId:             model.SafeDereference(p.TeamId),
+		TeamName:           model.SafeDereference(p.TeamName),
+		TeamDisplayName:    model.SafeDereference(p.TeamDisplayName),
+		ChannelId:          model.SafeDereference(p.ChannelId),
+		ChannelName:        model.SafeDereference(p.ChannelName),
+		ChannelDisplayName: model.SafeDereference(p.ChannelDisplayName),
+		ChannelType:        shared.ChannelTypeDisplayName(model.SafeDereference(p.ChannelType)),
+		UserId:             model.SafeDereference(p.UserId),
+		UserEmail:          model.SafeDereference(p.UserEmail),
+		Username:           model.SafeDereference(p.Username),
+		PostId:             model.SafeDereference(p.PostId),
+		EditedByPostId:     p.EditedNewMsgId,
+		RepliedToPostId:    model.SafeDereference(p.PostRootId),
+		PostMessage:        message,
+		PostType:           postType,
+		UserType:           userType,
+		PreviewsPostId:     p.PreviewID(),
+	}
+}
 
+func rowToStringSlice(r Row) []string {
 	return []string{
-		strconv.FormatInt(*createTime, 10),
-		teamId,
-		teamName,
-		teamDisplayName,
-		*post.ChannelId,
-		*post.ChannelName,
-		*post.ChannelDisplayName,
-		common_export.ChannelTypeDisplayName(*post.ChannelType),
-		*post.UserId,
-		*post.UserEmail,
-		*post.Username,
-		*post.PostId,
-		*post.PostOriginalId,
-		postRootId,
-		message,
-		postType,
-		userType,
-		post.PreviewID(),
+		strconv.FormatInt(r.CreateAt, 10),
+		strconv.FormatInt(r.UpdateAt, 10),
+		string(r.UpdateType),
+		r.TeamId,
+		r.TeamName,
+		r.TeamDisplayName,
+		r.ChannelId,
+		r.ChannelName,
+		r.ChannelDisplayName,
+		r.ChannelType,
+		r.UserId,
+		r.UserEmail,
+		r.Username,
+		r.PostId,
+		r.EditedByPostId,
+		r.RepliedToPostId,
+		r.PostMessage,
+		r.PostType,
+		r.UserType,
+		r.PreviewsPostId,
 	}
 }
 
-func attachmentToRow(post *model.MessageExport, attachment *model.FileInfo) []string {
-	row := postToRow(post, post.PostCreateAt, *post.PostMessage)
-
-	attachmentEntry := fmt.Sprintf("%s (files/%s/%s-%s)", attachment.Name, *post.PostId, attachment.Id, path.Base(attachment.Path))
-	attachmentMessage := "attachment"
-	userType := row[len(row)-2]
-
-	if attachment.DeleteAt > 0 && post.PostDeleteAt != nil {
-		deleteRow := postToRow(post, post.PostDeleteAt, *post.PostMessage)
-		row = append(
-			deleteRow[:len(deleteRow)-4],
-			attachmentEntry,
-			"deleted "+attachmentMessage,
-			userType,
-		)
-	} else {
-		row = append(
-			row[:len(row)-4],
-			attachmentEntry,
-			attachmentMessage,
-			userType,
-		)
+func attachmentToRow(post shared.PostExport) Row {
+	message := strings.TrimSpace(fmt.Sprintf("%s (files/%s/%s-%s)", post.FileInfo.Name, *post.PostId, post.FileInfo.Id, path.Base(post.FileInfo.Path)))
+	postType := "attachment"
+	if post.UpdatedType == shared.FileDeleted {
+		postType = "deleted attachment"
+		post.PostUpdateAt = model.NewPointer(post.FileInfo.DeleteAt)
 	}
-	return row
+
+	return postToRow(post, postType, post.PostCreateAt, message)
 }

server/enterprise/message_export/global_relay_export/deliver.go

@@ -6,8 +6,8 @@ package global_relay_export
 import (
 	"archive/zip"
 	"context"
+	"fmt"
 	"io"
-	"net/http"
 	"net/mail"
 	"net/smtp"
 	"os"
@@ -16,14 +16,14 @@ import (
 	"github.com/mattermost/mattermost/server/public/model"
 )
 
-func Deliver(export *os.File, config *model.Config) *model.AppError {
+func Deliver(export *os.File, config *model.Config) error {
 	info, err := export.Stat()
 	if err != nil {
-		return model.NewAppError("GlobalRelayDelivery", "ent.message_export.global_relay_export.deliver.unable_to_get_file_info.app_error", nil, "", http.StatusInternalServerError).Wrap(err)
+		return fmt.Errorf("unable to get the information of the export temporary file: %w", err)
 	}
 	zipFile, err := zip.NewReader(export, info.Size())
 	if err != nil {
-		return model.NewAppError("GlobalRelayDelivery", "ent.message_export.global_relay_export.deliver.unable_to_open_zip_file_data.app_error", nil, "", http.StatusInternalServerError).Wrap(err)
+		return fmt.Errorf("unable to open the export temporary file: %w", err)
 	}
 
 	to := *config.MessageExportSettings.GlobalRelaySettings.EmailAddress
@@ -33,7 +33,7 @@ func Deliver(export *os.File, config *model.Config) *model.AppError {
 
 	conn, err := connectToSMTPServer(ctx, config)
 	if err != nil {
-		return model.NewAppError("GlobalRelayDelivery", "ent.message_export.global_relay_export.deliver.unable_to_connect_smtp_server.app_error", nil, "", http.StatusInternalServerError).Wrap(err)
+		return fmt.Errorf("unable to connect to the smtp server: %w", err)
 	}
 	defer conn.Close()
 
@@ -55,56 +55,56 @@ func Deliver(export *os.File, config *model.Config) *model.AppError {
 			var nErr error
 			conn, nErr = connectToSMTPServer(context.Background(), config)
 			if nErr != nil {
-				return model.NewAppError("GlobalRelayDelivery", "ent.message_export.global_relay_export.deliver.unable_to_connect_smtp_server.app_error", nil, "", http.StatusInternalServerError).Wrap(nErr)
+				return fmt.Errorf("unable to connect to the smtp server: %w", nErr)
 			}
 		}
 	}
 	return nil
 }
 
-func deliverEmail(c *smtp.Client, mailFile *zip.File, from string, to string) *model.AppError {
+func deliverEmail(c *smtp.Client, mailFile *zip.File, from string, to string) error {
 	mailData, err := mailFile.Open()
 	if err != nil {
-		return model.NewAppError("GlobalRelayDelivery", "ent.message_export.global_relay_export.deliver.unable_to_open_email_file.app_error", nil, "", http.StatusInternalServerError).Wrap(err)
+		return fmt.Errorf("unable to get the an email from the temporary file: %w", err)
 	}
 	defer mailData.Close()
 
 	err = c.Mail(from)
 	if err != nil {
-		return model.NewAppError("GlobalRelayDelivery", "ent.message_export.global_relay_export.deliver.from_address.app_error", nil, "", http.StatusInternalServerError).Wrap(err)
+		return fmt.Errorf("unable to set the email From address: %w", err)
 	}
 
 	err = c.Rcpt(to)
 	if err != nil {
-		return model.NewAppError("GlobalRelayDelivery", "ent.message_export.global_relay_export.deliver.to_address.app_error", nil, "", http.StatusInternalServerError).Wrap(err)
+		return fmt.Errorf("unable to set the email To address: %w", err)
 	}
 
 	w, err := c.Data()
 	if err != nil {
-		return model.NewAppError("GlobalRelayDelivery", "ent.message_export.global_relay_export.deliver.msg_data.app_error", nil, "", http.StatusInternalServerError).Wrap(err)
+		return fmt.Errorf("unable to write the email message: %w", err)
 	}
 
 	_, err = io.Copy(w, mailData)
 	if err != nil {
-		return model.NewAppError("GlobalRelayDelivery", "ent.message_export.global_relay_export.deliver.msg.app_error", nil, "", http.StatusInternalServerError).Wrap(err)
+		return fmt.Errorf("unable to set the email message: %w", err)
 	}
 	err = w.Close()
 	if err != nil {
-		return model.NewAppError("GlobalRelayDelivery", "ent.message_export.global_relay_export.deliver.close.app_error", nil, "", http.StatusInternalServerError).Wrap(err)
+		return fmt.Errorf("unable to deliver the email to Global Relay: %w", err)
 	}
 	return nil
 }
 
-func getFrom(mailFile *zip.File) (string, *model.AppError) {
+func getFrom(mailFile *zip.File) (string, error) {
 	mailData, err := mailFile.Open()
 	if err != nil {
-		return "", model.NewAppError("GlobalRelayDelivery", "ent.message_export.global_relay_export.deliver.unable_to_open_email_file.app_error", nil, "", http.StatusInternalServerError).Wrap(err)
+		return "", fmt.Errorf("unable to get the an email from the temporary file: %w", err)
 	}
 	defer mailData.Close()
 
 	message, err := mail.ReadMessage(mailData)
 	if err != nil {
-		return "", model.NewAppError("GlobalRelayDelivery", "ent.message_export.global_relay_export.deliver.parse_mail.app_error", nil, "", http.StatusInternalServerError).Wrap(err)
+		return "", fmt.Errorf("unable to read the email information: %w", err)
 	}
 	return message.Header.Get("From"), nil
 }

server/enterprise/message_export/global_relay_export/deliver_test.go

@@ -30,8 +30,8 @@ func TestDeliver(t *testing.T) {
 		defer emptyFile.Close()
 		defer os.Remove(emptyFile.Name())
 
-		appErr := Deliver(emptyFile, config)
-		assert.NotNil(t, appErr)
+		err = Deliver(emptyFile, config)
+		assert.Error(t, err)
 	})
 
 	t.Run("Testing empty zip file", func(t *testing.T) {
@@ -46,8 +46,8 @@ func TestDeliver(t *testing.T) {
 		err = mail.DeleteMailBox(*config.MessageExportSettings.GlobalRelaySettings.EmailAddress)
 		require.NoError(t, err)
 
-		appErr := Deliver(emptyZipFile, config)
-		assert.Nil(t, appErr)
+		err = Deliver(emptyZipFile, config)
+		assert.NoError(t, err)
 
 		_, err = mail.GetMailBox(*config.MessageExportSettings.GlobalRelaySettings.EmailAddress)
 		require.Error(t, err)
@@ -87,8 +87,8 @@ func TestDeliver(t *testing.T) {
 		err = mail.DeleteMailBox(*config.MessageExportSettings.GlobalRelaySettings.EmailAddress)
 		require.NoError(t, err)
 
-		appErr := Deliver(emptyZipFile, config)
-		assert.Nil(t, appErr)
+		err = Deliver(emptyZipFile, config)
+		assert.NoError(t, err)
 
 		mailbox, err := mail.GetMailBox(*config.MessageExportSettings.GlobalRelaySettings.EmailAddress)
 		require.NoError(t, err)
@@ -130,8 +130,8 @@ func TestDeliver(t *testing.T) {
 		err = mail.DeleteMailBox(*config.MessageExportSettings.GlobalRelaySettings.EmailAddress)
 		require.NoError(t, err)
 
-		appErr := Deliver(emptyZipFile, config)
-		assert.Nil(t, appErr)
+		err = Deliver(emptyZipFile, config)
+		assert.NoError(t, err)
 
 		mailbox, err := mail.GetMailBox(*config.MessageExportSettings.GlobalRelaySettings.EmailAddress)
 		require.NoError(t, err)

server/enterprise/message_export/global_relay_export/global_relay_export.go

@@ -9,7 +9,7 @@ import (
 	"fmt"
 	"io"
 	"mime"
-	"net/http"
+	"os"
 	"sort"
 	"strings"
 	"time"
@@ -17,12 +17,11 @@ import (
 	"github.com/jaytaylor/html2text"
 	gomail "gopkg.in/mail.v2"
 
-	"github.com/mattermost/mattermost/server/v8/enterprise/message_export/common_export"
-
 	"github.com/mattermost/mattermost/server/public/model"
 	"github.com/mattermost/mattermost/server/public/shared/mlog"
 	"github.com/mattermost/mattermost/server/public/shared/request"
-	"github.com/mattermost/mattermost/server/v8/channels/store"
+	"github.com/mattermost/mattermost/server/v8/enterprise/internal/file"
+	"github.com/mattermost/mattermost/server/v8/enterprise/message_export/shared"
 	"github.com/mattermost/mattermost/server/v8/platform/shared/filestore"
 	"github.com/mattermost/mattermost/server/v8/platform/shared/templates"
 )
@@ -32,221 +31,228 @@ const (
 	GlobalRelayChannelNameHeader = "X-Mattermost-ChannelName"
 	GlobalRelayChannelIDHeader   = "X-Mattermost-ChannelID"
 	GlobalRelayChannelTypeHeader = "X-Mattermost-ChannelType"
-	MaxEmailBytes                = 250 << (10 * 2)
 	MaxEmailsPerConnection       = 400
 )
 
-type AllExport map[string][]*ChannelExport
+// MaxEmailBytes is a var because it needs to be set in tests. Otherwise it shouldn't be touched.
+var MaxEmailBytes int64 = 250 * 1024 * 1024 // 250MB
 
 type ChannelExport struct {
-	ChannelId       string            // the unique id of the channel
-	ChannelName     string            // the name of the channel
-	ChannelType     model.ChannelType // the channel type
-	StartTime       int64             // utc timestamp (seconds), start of export period or create time of channel, whichever is greater. Example: 1366611728.
-	EndTime         int64             // utc timestamp (seconds), end of export period or delete time of channel, whichever is lesser. Example: 1366611728.
-	Participants    []ParticipantRow  // summary information about the conversation participants
-	Messages        []Message         // the messages that were sent during the conversation
-	ExportedOn      int64             // utc timestamp (seconds), when this export was generated
-	numUserMessages map[string]int    // key is user id, value is number of messages that they sent during this period
-	uploadedFiles   []*model.FileInfo // any files that were uploaded to the channel during the export period
-	bytes           int64
+	TeamId             string
+	TeamName           string
+	TeamDisplayName    string
+	ChannelId          string // the unique id of the channel
+	ChannelName        string // the name of the channel
+	ChannelDisplayName string
+	ChannelType        model.ChannelType // the channel type
+	StartTime          int64             // utc timestamp (seconds), start of export period or create time of channel, whichever is greater. Example: 1366611728.
+	EndTime            int64             // utc timestamp (seconds), end of export period or delete time of channel, whichever is lesser. Example: 1366611728.
+	Participants       []ParticipantRow  // summary information about the conversation participants
+	Messages           []Message         // the messages that were sent during the conversation
+	ExportedOn         int64             // utc timestamp (seconds), when this export was generated
+	numUserMessages    map[string]int    // key is user id, value is number of messages that they sent during this period
+	uploadedFiles      []*model.FileInfo // any files that were uploaded to the channel during the export period
+	bytes              int64
 }
 
 // a row in the summary table at the top of the export
 type ParticipantRow struct {
-	Username     string
-	UserType     string
-	Email        string
-	JoinTime     int64
-	LeaveTime    int64
+	shared.JoinExport
 	MessagesSent int
 }
 
 type Message struct {
+	Id             string
 	SentTime       int64
+	SenderId       string
 	SenderUsername string
-	PostType       string
 	PostUsername   string
 	SenderUserType string
 	SenderEmail    string
 	Message        string
 	PreviewsPost   string
+	UpdateAt       int64
+	UpdateType     shared.PostUpdatedType
+	EditedNewMsgId string
 }
 
-func GlobalRelayExport(rctx request.CTX, posts []*model.MessageExport, db store.Store, fileAttachmentBackend filestore.FileBackend, dest io.Writer, templates *templates.Container) ([]string, int64, *model.AppError) {
-	var warningCount int64
-	attachmentsRemovedPostIDs := []string{}
-	allExports := make(map[string][]*ChannelExport)
-
-	zipFile := zip.NewWriter(dest)
-
-	membersByChannel := common_export.MembersByChannel{}
-	metadata := common_export.Metadata{
-		Channels:         map[string]common_export.MetadataChannel{},
-		MessagesCount:    0,
-		AttachmentsCount: 0,
-		StartTime:        0,
-		EndTime:          0,
+func GlobalRelayExport(rctx request.CTX, p shared.ExportParams) (shared.RunExportResults, error) {
+	exportData, err := shared.GetGenericExportData(p)
+	results := exportData.Results
+	if err != nil {
+		return results, err
 	}
 
-	for _, post := range posts {
-		if _, ok := membersByChannel[*post.ChannelId]; !ok {
-			membersByChannel[*post.ChannelId] = common_export.ChannelMembers{}
-		}
+	var attachmentsRemovedPostIDs []string
+	allExports := make(map[string][]*ChannelExport, len(exportData.Exports))
 
-		membersByChannel[*post.ChannelId][*post.UserId] = common_export.ChannelMember{
-			UserId:   *post.UserId,
-			Username: *post.Username,
-			IsBot:    post.IsBot,
-			Email:    *post.UserEmail,
-		}
+	// save a pointer to the joins for each channel, to be used later in getParticipants
+	joinsByChannel := make(map[string][]shared.JoinExport, len(exportData.Exports))
 
-		attachments := []*model.FileInfo{}
-		if len(post.PostFileIds) > 0 {
-			var err error
-			attachments, err = db.FileInfo().GetForPost(*post.PostId, true, true, false)
-			if err != nil {
-				return nil, warningCount, model.NewAppError("GlobalRelayExport", "ent.message_export.global_relay_export.get_attachment_error", nil, "", http.StatusInternalServerError).Wrap(err)
+	for _, channel := range exportData.Exports {
+		for _, post := range channel.Posts {
+			attachmentsRemoved := addToExports(rctx, allExports, channel, post)
+			attachmentsRemovedPostIDs = append(attachmentsRemovedPostIDs, attachmentsRemoved...)
+		}
+		if len(channel.Posts) == 0 {
+			// channel has no posts, but it was exported anyway, so it must have joins and leaves.
+			if _, present := allExports[channel.ChannelId]; !present {
+				// we found a new channel
+				allExports[channel.ChannelId] = []*ChannelExport{genericChannelToChannelExport(channel)}
 			}
 		}
-
-		attachmentsRemoved := addToExports(rctx, attachments, allExports, post)
-		attachmentsRemovedPostIDs = append(attachmentsRemovedPostIDs, attachmentsRemoved...)
-
-		metadata.Update(post, len(attachments))
+		joinsByChannel[channel.ChannelId] = channel.JoinEvents
 	}
 
+	tmpFile, err := os.CreateTemp("", "")
+	if err != nil {
+		return results, fmt.Errorf("unable to open the temporary export file: %w", err)
+	}
+	defer file.DeleteTemp(rctx.Logger(), tmpFile)
+	zipFile := zip.NewWriter(tmpFile)
+
+	// export each channelExport (sometimes multiple channelExport per real channel)
 	for _, channelExportList := range allExports {
 		for batchId, channelExport := range channelExportList {
-			participants, appErr := getParticipants(db, channelExport, membersByChannel[channelExport.ChannelId])
-			if appErr != nil {
-				return nil, warningCount, appErr
-			}
-			channelExport.Participants = participants
-			channelExport.ExportedOn = time.Now().Unix() * 1000
+			// we need to make the participant list for each channelExport "batch" (multiple "batches"per real channel)
+			// because each batch will have its own number of messages for that participant.
+			channelExport.Participants = getParticipants(channelExport, joinsByChannel[channelExport.ChannelId])
+			channelExport.ExportedOn = p.JobStartTime
 
-			channelExportFile, err := zipFile.Create(fmt.Sprintf("%s - (%s) - %d.eml", channelExport.ChannelName, channelExport.ChannelId, batchId))
+			var channelExportFile io.Writer
+			channelExportFile, err = zipFile.Create(fmt.Sprintf("%s - (%s) - %d.eml", channelExport.ChannelName, channelExport.ChannelId, batchId))
 			if err != nil {
-				return nil, warningCount, model.NewAppError("GlobalRelayExport", "ent.message_export.global_relay.create_file_in_zip.app_error", nil, "", http.StatusInternalServerError).Wrap(err)
+				return results, fmt.Errorf("unable to create the eml file: %w", err)
 			}
 
-			if appErr, warningCount = generateEmail(rctx, fileAttachmentBackend, channelExport, templates, channelExportFile); appErr != nil {
-				return nil, warningCount, appErr
+			if results.NumWarnings, err = generateEmail(rctx, p.FileAttachmentBackend, channelExport, p.Templates, channelExportFile); err != nil {
+				return results, err
 			}
 		}
 	}
 
-	err := zipFile.Close()
+	err = zipFile.Close()
 	if err != nil {
-		return nil, warningCount, model.NewAppError("GlobalRelayExport", "ent.message_export.global_relay.close_zip_file.app_error", nil, "", http.StatusInternalServerError).Wrap(err)
+		return results, fmt.Errorf("unable to close the zip file using tmpFile.Name: %v, err: %w", tmpFile.Name(), err)
 	}
 
-	return attachmentsRemovedPostIDs, warningCount, nil
+	_, err = tmpFile.Seek(0, 0)
+	if err != nil {
+		return results, fmt.Errorf("unable to re-read the Global Relay temporary export file using tmpFile.Name: %v, err: %w", tmpFile.Name(), err)
+	}
+
+	if p.ExportType == model.ComplianceExportTypeGlobalrelayZip {
+		// Try to disable the write timeout for the potentially big export file.
+		_, err = filestore.TryWriteFileContext(rctx.Context(), p.ExportBackend, tmpFile, p.BatchPath)
+		if err != nil {
+			return results, fmt.Errorf("unable to write the global relay file, using tmpFile.Name: %v, batchPath: %v, err: %w", tmpFile.Name(), p.BatchPath, err)
+		}
+	} else {
+		err = Deliver(tmpFile, p.Config)
+		if err != nil {
+			return results, fmt.Errorf("unable to deliver tmpFile.Name: %v, err: %w", tmpFile.Name(), err)
+		}
+	}
+
+	if len(attachmentsRemovedPostIDs) > 0 {
+		rctx.Logger().Warn("Global Relay Attachments Removed because they were too large to send to Global Relay",
+			mlog.Int("number_of_attachments_removed", len(attachmentsRemovedPostIDs)))
+		rctx.Logger().Warn("List of posts which had attachments removed",
+			mlog.Array("post_ids", attachmentsRemovedPostIDs))
+	}
+
+	return results, nil
 }
 
-func addToExports(rctx request.CTX, attachments []*model.FileInfo, exports map[string][]*ChannelExport, post *model.MessageExport) []string {
+// addToExports adds the post to the allExports collection. allExports keeps a map of channelId->[]*ChannelExport.
+// If a channelId has an existing []*ChannelExport, it adds post to the last ChannelExport in that list.
+// If the last ChannelExport is too big, it starts a new ChannelExport and appends it to the list (a new "batch").
+func addToExports(rctx request.CTX, allExports map[string][]*ChannelExport, genericChannel shared.ChannelExport,
+	post shared.PostExport) []string {
 	var channelExport *ChannelExport
-	attachmentsRemovedPostIDs := []string{}
-	if channelExports, present := exports[*post.ChannelId]; !present {
+	var attachmentsRemovedPostIDs []string
+	if channelExports, present := allExports[*post.ChannelId]; !present {
 		// we found a new channel
-		channelExport = &ChannelExport{
-			ChannelId:       *post.ChannelId,
-			ChannelName:     *post.ChannelDisplayName,
-			ChannelType:     *post.ChannelType,
-			StartTime:       *post.PostCreateAt,
-			Messages:        make([]Message, 0),
-			Participants:    make([]ParticipantRow, 0),
-			numUserMessages: make(map[string]int),
-			uploadedFiles:   make([]*model.FileInfo, 0),
-			bytes:           0,
-		}
-		exports[*post.ChannelId] = []*ChannelExport{channelExport}
+		channelExport = genericChannelToChannelExport(genericChannel)
+		allExports[*post.ChannelId] = []*ChannelExport{channelExport}
 	} else {
 		// we already know about this channel
 		channelExport = channelExports[len(channelExports)-1]
 	}
 
-	// Create a new ChannelExport if it would be too many bytes to add the post
-	fileBytes := fileInfoListBytes(attachments)
 	msgBytes := int64(len(*post.PostMessage))
-	postBytes := fileBytes + msgBytes
-	postTooLargeForChannelBatch := channelExport.bytes+postBytes > MaxEmailBytes
-	postAloneTooLargeToSend := postBytes > MaxEmailBytes // Attachments must be removed from export, they're too big to send.
 
-	if postAloneTooLargeToSend {
+	// Create a new ChannelExport if it would be too many bytes to add the post.
+	// NOTE: we are only exporting attachment starts.
+	attachmentStarts := make([]*model.FileInfo, 0, len(post.AttachmentCreates))
+	for _, start := range post.AttachmentCreates {
+		attachmentStarts = append(attachmentStarts, start.FileInfo)
+	}
+	fileBytes := fileInfoListBytes(attachmentStarts)
+	postBytes := fileBytes + msgBytes
+	// NOTE: This is only a rough estimate -- we're not including the txt or html portion of the email...
+	attachmentsAloneTooLargeToSend := fileBytes > MaxEmailBytes // Attachments must be removed from export, they're too big to send.
+	if attachmentsAloneTooLargeToSend {
+		postBytes -= fileBytes
+	}
+	postTooLargeForChannelBatch := channelExport.bytes+postBytes > MaxEmailBytes
+
+	if attachmentsAloneTooLargeToSend {
 		attachmentsRemovedPostIDs = append(attachmentsRemovedPostIDs, *post.PostId)
 	}
 
-	if postTooLargeForChannelBatch && !postAloneTooLargeToSend {
-		channelExport = &ChannelExport{
-			ChannelId:       *post.ChannelId,
-			ChannelName:     *post.ChannelDisplayName,
-			ChannelType:     *post.ChannelType,
-			StartTime:       *post.PostCreateAt,
-			Messages:        make([]Message, 0),
-			Participants:    make([]ParticipantRow, 0),
-			numUserMessages: make(map[string]int),
-			uploadedFiles:   make([]*model.FileInfo, 0),
-			bytes:           0,
-		}
-		exports[*post.ChannelId] = append(exports[*post.ChannelId], channelExport)
+	// new "batch"
+	if postTooLargeForChannelBatch {
+		channelExport = genericChannelToChannelExport(genericChannel)
+		allExports[*post.ChannelId] = append(allExports[*post.ChannelId], channelExport)
 	}
 
 	addPostToChannelExport(rctx, channelExport, post)
 
 	// if this post includes files, add them to the collection
-	for _, fileInfo := range attachments {
-		addAttachmentToChannelExport(channelExport, post, fileInfo, postAloneTooLargeToSend)
-	}
+	addAttachmentsToChannelExport(channelExport, post, post.AttachmentCreates, post.AttachmentDeletes, attachmentsAloneTooLargeToSend)
 	channelExport.bytes += postBytes
 	return attachmentsRemovedPostIDs
 }
 
-func getParticipants(db store.Store, channelExport *ChannelExport, members common_export.ChannelMembers) ([]ParticipantRow, *model.AppError) {
-	participantsMap := map[string]ParticipantRow{}
-	channelMembersHistory, err := db.ChannelMemberHistory().GetUsersInChannelDuring(channelExport.StartTime, channelExport.EndTime, channelExport.ChannelId)
-	if err != nil {
-		return nil, model.NewAppError("getParticipants", "ent.get_users_in_channel_during", nil, "", http.StatusInternalServerError).Wrap(err)
+func genericChannelToChannelExport(genericChannel shared.ChannelExport) *ChannelExport {
+	return &ChannelExport{
+		TeamId:             genericChannel.TeamId,
+		TeamName:           genericChannel.TeamName,
+		TeamDisplayName:    genericChannel.TeamDisplayName,
+		ChannelId:          genericChannel.ChannelId,
+		ChannelName:        genericChannel.ChannelName,
+		ChannelDisplayName: genericChannel.DisplayName,
+		ChannelType:        genericChannel.ChannelType,
+		StartTime:          genericChannel.StartTime,
+		EndTime:            genericChannel.EndTime,
+		// we can't preallocate sizes here because we don't know how many will be in this "batch"
+		Participants:    make([]ParticipantRow, 0),
+		Messages:        make([]Message, 0),
+		ExportedOn:      0,
+		numUserMessages: make(map[string]int),
+		uploadedFiles:   make([]*model.FileInfo, 0),
+		bytes:           0,
 	}
+}
 
-	joins, leaves := common_export.GetJoinsAndLeavesForChannel(channelExport.StartTime, channelExport.EndTime, channelMembersHistory, members)
-
-	for _, join := range joins {
-		userType := "user"
-		if join.IsBot {
-			userType = "bot"
-		}
-
-		if _, ok := participantsMap[join.UserId]; !ok {
-			participantsMap[join.UserId] = ParticipantRow{
-				Username:     join.Username,
-				UserType:     userType,
-				Email:        join.Email,
-				JoinTime:     join.Datetime,
-				LeaveTime:    channelExport.EndTime,
-				MessagesSent: channelExport.numUserMessages[join.UserId],
-			}
-		}
-	}
-	for _, leave := range leaves {
-		if participantRow, ok := participantsMap[leave.UserId]; ok {
-			participantRow.LeaveTime = leave.Datetime //nolint:govet
-		}
-	}
-
-	participants := []ParticipantRow{}
-	for _, participant := range participantsMap {
-		participants = append(participants, participant)
+func getParticipants(channelExport *ChannelExport, joinEvents []shared.JoinExport) []ParticipantRow {
+	participants := make([]ParticipantRow, 0, len(joinEvents))
+	for _, j := range joinEvents {
+		participants = append(participants, ParticipantRow{
+			JoinExport:   j,
+			MessagesSent: channelExport.numUserMessages[j.UserId],
+		})
 	}
 
 	sort.Slice(participants, func(i, j int) bool {
 		return participants[i].Username < participants[j].Username
 	})
-	return participants, nil
+	return participants
 }
 
-func generateEmail(rctx request.CTX, fileAttachmentBackend filestore.FileBackend, channelExport *ChannelExport, templates *templates.Container, w io.Writer) (*model.AppError, int64) {
-	var warningCount int64
+func generateEmail(rctx request.CTX, fileAttachmentBackend filestore.FileBackend, channelExport *ChannelExport, templates *templates.Container, w io.Writer) (int, error) {
+	var warningCount int
 	participantEmailAddresses := getParticipantEmails(channelExport)
 
 	// GlobalRelay expects the email to come from the person that initiated the conversation.
@@ -258,10 +264,10 @@ func generateEmail(rctx request.CTX, fileAttachmentBackend filestore.FileBackend
 
 	htmlBody, err := channelExportToHTML(rctx, channelExport, templates)
 	if err != nil {
-		return model.NewAppError("GlobalRelayExport", "ent.message_export.global_relay.generate_email.app_error", nil, "", http.StatusInternalServerError).Wrap(err), warningCount
+		return warningCount, fmt.Errorf("unable to generate eml file data: %w", err)
 	}
 
-	subject := fmt.Sprintf("Mattermost Compliance Export: %s", channelExport.ChannelName)
+	subject := fmt.Sprintf("Mattermost Compliance Export: %s", channelExport.ChannelDisplayName)
 	htmlMessage := "\r\n<html><body>" + htmlBody + "</body></html>"
 
 	txtBody, err := html2text.FromString(htmlBody)
@@ -278,9 +284,9 @@ func generateEmail(rctx request.CTX, fileAttachmentBackend filestore.FileBackend
 		"Auto-Submitted":             {"auto-generated"},
 		"Precedence":                 {"bulk"},
 		GlobalRelayMsgTypeHeader:     {"Mattermost"},
-		GlobalRelayChannelNameHeader: {encodeRFC2047Word(channelExport.ChannelName)},
+		GlobalRelayChannelNameHeader: {encodeRFC2047Word(channelExport.ChannelDisplayName)},
 		GlobalRelayChannelIDHeader:   {encodeRFC2047Word(channelExport.ChannelId)},
-		GlobalRelayChannelTypeHeader: {encodeRFC2047Word(common_export.ChannelTypeDisplayName(channelExport.ChannelType))},
+		GlobalRelayChannelTypeHeader: {encodeRFC2047Word(shared.ChannelTypeDisplayName(channelExport.ChannelType))},
 	}
 
 	m := gomail.NewMessage(gomail.SetCharset("UTF-8"))
@@ -293,9 +299,10 @@ func generateEmail(rctx request.CTX, fileAttachmentBackend filestore.FileBackend
 		path := fileInfo.Path
 
 		m.Attach(fileInfo.Name, gomail.SetCopyFunc(func(writer io.Writer) error {
-			reader, appErr := fileAttachmentBackend.Reader(path)
-			if appErr != nil {
-				rctx.Logger().Warn("File not found for export", mlog.String("Filename", path))
+			var reader filestore.ReadCloseSeeker
+			reader, err = fileAttachmentBackend.Reader(path)
+			if err != nil {
+				rctx.Logger().Warn("File not found for export", mlog.String("filename", path))
 				warningCount += 1
 				return nil
 			}
@@ -303,7 +310,7 @@ func generateEmail(rctx request.CTX, fileAttachmentBackend filestore.FileBackend
 
 			_, err = io.Copy(writer, reader)
 			if err != nil {
-				return model.NewAppError("GlobalRelayExport", "ent.message_export.global_relay.attach_file.app_error", nil, "", http.StatusInternalServerError).Wrap(err)
+				return fmt.Errorf("unable to add attachment to the Global Relay export: %w", err)
 			}
 			return nil
 		}))
@@ -311,15 +318,15 @@ func generateEmail(rctx request.CTX, fileAttachmentBackend filestore.FileBackend
 
 	_, err = m.WriteTo(w)
 	if err != nil {
-		return model.NewAppError("GlobalRelayExport", "ent.message_export.global_relay.generate_email.app_error", nil, "", http.StatusInternalServerError).Wrap(err), warningCount
+		return warningCount, fmt.Errorf("unable to generate eml file data: %w", err)
 	}
-	return nil, warningCount
+	return warningCount, nil
 }
 
 func getParticipantEmails(channelExport *ChannelExport) []string {
-	participantEmails := make([]string, len(channelExport.Participants))
-	for i, participant := range channelExport.Participants {
-		participantEmails[i] = participant.Email
+	participantEmails := make([]string, 0, len(channelExport.Participants))
+	for _, participant := range channelExport.Participants {
+		participantEmails = append(participantEmails, participant.UserEmail)
 	}
 	return participantEmails
 }
@@ -332,19 +339,15 @@ func fileInfoListBytes(fileInfoList []*model.FileInfo) int64 {
 	return totalBytes
 }
 
-func addPostToChannelExport(rctx request.CTX, channelExport *ChannelExport, post *model.MessageExport) {
-	userType := "user"
-	if post.IsBot {
-		userType = "bot"
-	}
-
+func addPostToChannelExport(rctx request.CTX, channelExport *ChannelExport, post shared.PostExport) {
 	strPostProps := post.PostProps
-	bytPostProps := []byte(*strPostProps)
+	bytePostProps := []byte(*strPostProps)
 
 	// Added to show the username if overridden by a webhook or API integration
-	postUserName := ""
+	originalUsername := model.SafeDereference(post.Username)
+	postUserName := originalUsername
 	var postPropsLocal map[string]any
-	err := json.Unmarshal(bytPostProps, &postPropsLocal)
+	err := json.Unmarshal(bytePostProps, &postPropsLocal)
 	if err != nil {
 		rctx.Logger().Warn("Failed to unmarshal post Props into JSON. Ignoring username override.", mlog.Err(err))
 	} else {
@@ -352,67 +355,59 @@ func addPostToChannelExport(rctx request.CTX, channelExport *ChannelExport, post
 			postUserName = overrideUsername.(string)
 		}
 
-		if postUserName == "" {
+		if postUserName == originalUsername {
 			if overrideUsername, ok := postPropsLocal["webhook_display_name"]; ok {
 				postUserName = overrideUsername.(string)
 			}
 		}
 	}
 
-	element := Message{
-		SentTime:       *post.PostCreateAt,
-		Message:        *post.PostMessage,
-		SenderUserType: userType,
-		PostType:       *post.PostType,
-		PostUsername:   postUserName,
-		SenderUsername: *post.Username,
-		SenderEmail:    *post.UserEmail,
-		PreviewsPost:   post.PreviewID(),
-	}
+	element := postToMessage(post)
+	element.PostUsername = postUserName
 	channelExport.Messages = append(channelExport.Messages, element)
-	channelExport.EndTime = *post.PostCreateAt
 	channelExport.numUserMessages[*post.UserId] += 1
 }
 
-func addAttachmentToChannelExport(channelExport *ChannelExport, post *model.MessageExport, fileInfo *model.FileInfo, removeAttachments bool) {
-	var uploadElement Message
-	userType := "user"
-	if post.IsBot {
-		userType = "bot"
+func postToMessage(post shared.PostExport) Message {
+	return Message{
+		Id:             model.SafeDereference(post.PostId),
+		SentTime:       model.SafeDereference(post.PostCreateAt),
+		SenderId:       model.SafeDereference(post.UserId),
+		SenderUsername: model.SafeDereference(post.Username),
+		PostUsername:   model.SafeDereference(post.Username),
+		SenderUserType: string(post.UserType),
+		SenderEmail:    model.SafeDereference(post.UserEmail),
+		Message:        post.Message,
+		PreviewsPost:   post.PreviewID(),
+		UpdateAt:       post.UpdateAt,
+		UpdateType:     post.UpdatedType,
+		EditedNewMsgId: post.EditedNewMsgId,
 	}
-	if removeAttachments {
-		// add "post" message indicating that attachments were not sent
-		uploadElement = Message{
-			SentTime:       fileInfo.CreateAt,
-			Message:        fmt.Sprintf("Uploaded file '%s' (id '%s') was removed because it was too large to send.", fileInfo.Name, fileInfo.Id),
-			SenderUsername: *post.Username,
-			SenderUserType: userType,
-			SenderEmail:    *post.UserEmail,
+}
+
+func addAttachmentsToChannelExport(channelExport *ChannelExport, post shared.PostExport,
+	attachmentStarts []*shared.FileUploadStartExport, attachmentDeletes []shared.PostExport, removeAttachments bool) {
+	for _, start := range attachmentStarts {
+		var message string
+
+		if removeAttachments {
+			message = fmt.Sprintf("Uploaded file %q (id '%s') was removed because it was too large to send.",
+				start.FileInfo.Name, start.FileInfo.Id)
+		} else {
+			channelExport.uploadedFiles = append(channelExport.uploadedFiles, start.FileInfo)
+			message = fmt.Sprintf("Uploaded file %s", start.FileInfo.Name)
 		}
 
-		if fileInfo.DeleteAt != 0 {
-			uploadElement.SentTime = fileInfo.DeleteAt
-			uploadElement.Message = fmt.Sprintf("Deleted file '%s' (id '%s') was removed because it was too large to send.", fileInfo.Name, fileInfo.Id)
-		}
-	} else {
-		channelExport.uploadedFiles = append(channelExport.uploadedFiles, fileInfo)
-
-		// add an implicit "post" to the export that includes the filename so GlobalRelay knows who uploaded each file
-		uploadElement = Message{
-			SentTime:       fileInfo.CreateAt,
-			Message:        fmt.Sprintf("Uploaded file %s", fileInfo.Name),
-			SenderUsername: *post.Username,
-			SenderUserType: userType,
-			SenderEmail:    *post.UserEmail,
-		}
-
-		if fileInfo.DeleteAt != 0 {
-			uploadElement.SentTime = fileInfo.DeleteAt
-			uploadElement.Message = fmt.Sprintf("Deleted file %s", fileInfo.Name)
-		}
+		uploadElement := postToMessage(post)
+		uploadElement.Message = message
+		channelExport.Messages = append(channelExport.Messages, uploadElement)
 	}
 
-	channelExport.Messages = append(channelExport.Messages, uploadElement)
+	for _, deleted := range attachmentDeletes {
+		uploadElement := postToMessage(deleted)
+		uploadElement.Message = fmt.Sprintf("Deleted file %s", deleted.FileInfo.Name)
+		channelExport.Messages = append(channelExport.Messages, uploadElement)
+	}
 }
 
 func encodeRFC2047Word(s string) string {

server/enterprise/message_export/global_relay_export/test_helpers.go

@@ -0,0 +1,29 @@
+// Copyright (c) 2015-present Mattermost, Inc. All Rights Reserved.
+// See LICENSE.enterprise for license information.
+
+package global_relay_export
+
+import (
+	"net/mail"
+	"strings"
+	"testing"
+
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+)
+
+func AssertHeaderContains(t *testing.T, msg string, expected map[string]string) {
+	t.Helper()
+	m, err := mail.ReadMessage(strings.NewReader(msg))
+	require.NoError(t, err)
+
+	for k, v := range expected {
+		assert.Equal(t, v, m.Header.Get(k))
+	}
+}
+
+func CleanTestOutput(msg string) string {
+	msg = strings.Replace(msg, "=\r\n", "", -1)
+	msg = strings.Replace(msg, "\r\n", "\n", -1)
+	return msg
+}

server/enterprise/message_export/global_relay_export/to_html.go

@@ -17,9 +17,15 @@ import (
 	"github.com/mattermost/mattermost/server/v8/platform/shared/templates"
 )
 
+func TimestampConvert(timestampMS int64) string {
+	return time.Unix(timestampMS/1000, 0).UTC().Format(time.RFC3339)
+
+	// for testing:  (keep in case we need to be specific -- helps when you have joins and leaves within millis of each other)
+	//return fmt.Sprintf("%d", timestampMS)
+}
+
 func channelExportToHTML(rctx request.CTX, channelExport *ChannelExport, t *templates.Container) (string, error) {
 	durationMilliseconds := channelExport.EndTime - channelExport.StartTime
-	// TODO CHECK IF WE NEED THE MILISECONS HERE OR WE CAN ROUND IT DIRECTLY HERE
 	duration := time.Duration(durationMilliseconds) * time.Millisecond
 
 	var participantRowsBuffer bytes.Buffer
@@ -35,7 +41,9 @@ func channelExportToHTML(rctx request.CTX, channelExport *ChannelExport, t *temp
 	var messagesBuffer bytes.Buffer
 	sort.Slice(channelExport.Messages, func(i, j int) bool {
 		if channelExport.Messages[i].SentTime == channelExport.Messages[j].SentTime {
-			return !strings.HasPrefix(channelExport.Messages[i].Message, "Uploaded file") && !strings.HasPrefix(channelExport.Messages[i].Message, "Deleted file")
+			return !strings.HasPrefix(channelExport.Messages[i].Message, "Uploaded file") &&
+				!strings.HasPrefix(channelExport.Messages[i].Message, "Deleted file") &&
+				channelExport.Messages[i].UpdateType == ""
 		}
 		return channelExport.Messages[i].SentTime < channelExport.Messages[j].SentTime
 	})
@@ -50,13 +58,18 @@ func channelExportToHTML(rctx request.CTX, channelExport *ChannelExport, t *temp
 
 	data := templates.Data{
 		Props: map[string]any{
-			"ChannelName":     channelExport.ChannelName,
-			"Started":         time.Unix(channelExport.StartTime/1000, 0).UTC().Format(time.RFC3339),
-			"Ended":           time.Unix(channelExport.EndTime/1000, 0).UTC().Format(time.RFC3339),
-			"Duration":        durafmt.Parse(duration.Round(time.Minute)).String(),
-			"ParticipantRows": template.HTML(participantRowsBuffer.String()),
-			"Messages":        template.HTML(messagesBuffer.String()),
-			"ExportDate":      time.Unix(channelExport.ExportedOn/1000, 0).UTC().Format(time.RFC3339),
+			"TeamId":             channelExport.TeamId,
+			"TeamName":           channelExport.TeamName,
+			"TeamDisplayName":    channelExport.TeamDisplayName,
+			"ChannelId":          channelExport.ChannelId,
+			"ChannelName":        channelExport.ChannelName,
+			"ChannelDisplayName": channelExport.ChannelDisplayName,
+			"Started":            TimestampConvert(channelExport.StartTime),
+			"Ended":              TimestampConvert(channelExport.EndTime),
+			"Duration":           durafmt.Parse(duration.Round(time.Minute)).String(),
+			"ParticipantRows":    template.HTML(participantRowsBuffer.String()),
+			"Messages":           template.HTML(messagesBuffer.String()),
+			"ExportDate":         TimestampConvert(channelExport.ExportedOn),
 		},
 	}
 
@@ -65,16 +78,16 @@ func channelExportToHTML(rctx request.CTX, channelExport *ChannelExport, t *temp
 
 func participantToHTML(participant *ParticipantRow, t *templates.Container) (string, error) {
 	durationMilliseconds := participant.LeaveTime - participant.JoinTime
-	// TODO CHECK IF WE NEED THE MILISECONS HERE OR WE CAN ROUND IT DIRECTLY HERE
 	duration := time.Duration(durationMilliseconds) * time.Millisecond
 
 	data := templates.Data{
 		Props: map[string]any{
+			"UserId":      participant.UserId,
 			"Username":    participant.Username,
 			"UserType":    participant.UserType,
-			"Email":       participant.Email,
-			"Joined":      time.Unix(participant.JoinTime/1000, 0).UTC().Format(time.RFC3339),
-			"Left":        time.Unix(participant.LeaveTime/1000, 0).UTC().Format(time.RFC3339),
+			"Email":       participant.UserEmail,
+			"Joined":      TimestampConvert(participant.JoinTime),
+			"Left":        TimestampConvert(participant.LeaveTime),
 			"Duration":    durafmt.Parse(duration.Round(time.Minute)).String(),
 			"NumMessages": participant.MessagesSent,
 		},
@@ -90,14 +103,18 @@ func messageToHTML(message *Message, t *templates.Container) (string, error) {
 	}
 	data := templates.Data{
 		Props: map[string]any{
-			"SentTime":     time.Unix(message.SentTime/1000, 0).UTC().Format(time.RFC3339),
-			"Username":     message.SenderUsername,
-			"PostUsername": postUsername,
-			"UserType":     message.SenderUserType,
-			"PostType":     message.PostType,
-			"Email":        message.SenderEmail,
-			"Message":      message.Message,
-			"PreviewsPost": message.PreviewsPost,
+			"PostId":         message.Id,
+			"SentTime":       TimestampConvert(message.SentTime),
+			"UserId":         message.SenderId,
+			"Username":       message.SenderUsername,
+			"PostUsername":   postUsername,
+			"UserType":       message.SenderUserType,
+			"Email":          message.SenderEmail,
+			"Message":        message.Message,
+			"PreviewsPost":   message.PreviewsPost,
+			"UpdateTime":     TimestampConvert(message.UpdateAt),
+			"UpdateType":     message.UpdateType,
+			"EditedNewMsgId": message.EditedNewMsgId,
 		},
 	}
 

server/enterprise/message_export/message_export.go

@@ -6,34 +6,22 @@ package message_export
 import (
 	"encoding/json"
 	"errors"
-	"fmt"
-	"math"
-	"net/http"
-	"os"
-	"path"
-	"time"
-
 	"strconv"
+	"time"
 
 	"github.com/mattermost/mattermost/server/public/model"
 	"github.com/mattermost/mattermost/server/public/shared/mlog"
 	"github.com/mattermost/mattermost/server/public/shared/request"
 	"github.com/mattermost/mattermost/server/v8/channels/app"
-	"github.com/mattermost/mattermost/server/v8/channels/store"
-	"github.com/mattermost/mattermost/server/v8/channels/utils/fileutils"
 	"github.com/mattermost/mattermost/server/v8/einterfaces"
 	ejobs "github.com/mattermost/mattermost/server/v8/einterfaces/jobs"
-	"github.com/mattermost/mattermost/server/v8/platform/shared/filestore"
-	"github.com/mattermost/mattermost/server/v8/platform/shared/templates"
-
 	"github.com/mattermost/mattermost/server/v8/enterprise/message_export/actiance_export"
 	"github.com/mattermost/mattermost/server/v8/enterprise/message_export/csv_export"
 	"github.com/mattermost/mattermost/server/v8/enterprise/message_export/global_relay_export"
+	"github.com/mattermost/mattermost/server/v8/enterprise/message_export/shared"
 )
 
-const (
-	GlobalRelayExportFilename = "global-relay.zip"
-)
+const GlobalRelayExportFilename = "global-relay.zip"
 
 type MessageExportInterfaceImpl struct {
 	Server *app.Server
@@ -56,7 +44,7 @@ func (m *MessageExportInterfaceImpl) StartSynchronizeJob(rctx request.CTX, expor
 	// if a valid export time was specified, put it in the job data
 	jobData := make(map[string]string)
 	if exportFromTimestamp >= 0 {
-		jobData[JobDataBatchStartTimestamp] = strconv.FormatInt(exportFromTimestamp, 10)
+		jobData[shared.JobDataBatchStartTime] = strconv.FormatInt(exportFromTimestamp, 10)
 	}
 
 	// passing nil for job data will cause the worker to inherit start time from previously successful job
@@ -85,32 +73,137 @@ func (m *MessageExportInterfaceImpl) StartSynchronizeJob(rctx request.CTX, expor
 	return job, nil
 }
 
-func (m *MessageExportInterfaceImpl) RunExport(rctx request.CTX, exportType string, since int64, limit int) (warningCount int64, appErr *model.AppError) {
-	if limit < 0 {
-		limit = math.MaxInt64
-	}
-	postsToExport, _, err := m.Server.Store().Compliance().MessageExport(rctx, model.MessageExportCursor{LastPostUpdateAt: since}, limit)
+func RunBatch(rctx request.CTX, data shared.JobData, params shared.BackendParams) (shared.RunExportResults, shared.JobData, error) {
+	start := time.Now()
+	var err error
+	var res shared.RunExportResults
+	data, err = GetDataForBatch(rctx, data, params)
 	if err != nil {
-		return warningCount, model.NewAppError("RunExport", "ent.message_export.run_export.app_error", nil, "", http.StatusInternalServerError).Wrap(err)
-	}
-	rctx.Logger().Debug("Found posts to export", mlog.Int("number_of_posts", len(postsToExport)))
-
-	fileBackend := m.Server.FileBackend()
-	templatesDir, ok := fileutils.FindDir("templates")
-	if !ok {
-		return warningCount, model.NewAppError("RunExport", "ent.compliance.run_export.template_watcher.appError", nil, "", http.StatusAccepted)
+		return res, data, err
 	}
 
-	t, err2 := templates.New(templatesDir)
-	if err2 != nil {
-		return warningCount, model.NewAppError("RunExport", "ent.compliance.run_export.template_watcher.appError", nil, "", http.StatusAccepted).Wrap(err2)
+	if data.Finished {
+		return res, data, nil
 	}
 
-	exportDirectory := getOutputDirectoryPath(since, model.GetMillis())
-	return runExportByType(rctx, exportType, postsToExport, exportDirectory, m.Server.Store(), fileBackend, fileBackend, t, m.Server.Config())
+	// Now write the data to the export type.
+	res, err = RunExportByType(rctx, DataToExportParams(data), params)
+	if err != nil {
+		return res, data, err
+	}
+
+	data.ProcessingPostsMs = append(data.ProcessingPostsMs, res.ProcessingPostsMs)
+	data.ProcessingXmlMs = append(data.ProcessingXmlMs, res.ProcessingXmlMs)
+	data.TransferringFilesMs = append(data.TransferringFilesMs, res.TransferringFilesMs)
+	data.TransferringZipMs = append(data.TransferringZipMs, res.TransferringZipMs)
+	data.TotalBatchMs = append(data.TotalBatchMs, time.Since(start).Milliseconds())
+	data.WarningCount += res.NumWarnings
+	data.BatchStartTime = data.BatchEndTime
+
+	return res, data, err
 }
 
-func runExportByType(rctx request.CTX, exportType string, postsToExport []*model.MessageExport, exportDirectory string, db store.Store, exportBackend filestore.FileBackend, fileAttachmentBackend filestore.FileBackend, htmlTemplates *templates.Container, config *model.Config) (warningCount int64, appErr *model.AppError) {
+// GetDataForBatch gets the posts for this batch and updates JobData with the current state.
+func GetDataForBatch(rctx request.CTX, data shared.JobData, params shared.BackendParams) (shared.JobData, error) {
+	start := time.Now()
+	var err error
+	// Using BatchSize+1 is a trick to test whether or not we've reached the final batch.
+	data.PostsToExport, data.Cursor, err = params.Store.Compliance().MessageExport(rctx, data.Cursor, data.BatchSize+1)
+	if err != nil {
+		return data, err
+	}
+	data.MessageExportMs = append(data.MessageExportMs, time.Since(start).Milliseconds())
+
+	if len(data.PostsToExport) == data.BatchSize+1 {
+		// We still have posts after this current batch.
+		// Remove the last post, we have to leave it for the next batch.
+		lastPostIdx := len(data.PostsToExport) - 1
+		data.PostsToExport = data.PostsToExport[:lastPostIdx]
+		lastPostIdx = len(data.PostsToExport) - 1
+		data.Cursor.LastPostUpdateAt = *data.PostsToExport[lastPostIdx].PostUpdateAt
+		data.Cursor.LastPostId = *data.PostsToExport[lastPostIdx].PostId
+		data.BatchEndTime = data.Cursor.LastPostUpdateAt
+	} else {
+		// We've reached the final batch; we need to include all join/leave events that occur after the lastpost.
+		// This will let us also pick up the joins/leaves that occur after lastPostUpdateAt but before JobEndTime.
+		data.BatchEndTime = data.JobEndTime
+	}
+
+	if len(data.PostsToExport) == 0 {
+		data.Finished = true
+		return data, nil
+	}
+
+	rctx.Logger().Debug("Found posts to export", mlog.Int("num_posts", len(data.PostsToExport)))
+	data.MessagesExported += len(data.PostsToExport)
+	data.BatchNumber++
+	data.BatchPath = shared.GetBatchPath(data.ExportDir, data.BatchStartTime, data.BatchEndTime, data.BatchNumber)
+
+	return data, nil
+}
+
+type ExportParams struct {
+	ExportType             string
+	ChannelMetadata        map[string]*shared.MetadataChannel
+	ChannelMemberHistories map[string][]*model.ChannelMemberHistoryResult
+	PostsToExport          []*model.MessageExport
+	JobStartTime           int64
+	BatchPath              string
+	BatchStartTime         int64
+	BatchEndTime           int64
+}
+
+func DataToExportParams(data shared.JobData) ExportParams {
+	return ExportParams{
+		ExportType:             data.ExportType,
+		ChannelMetadata:        data.ChannelMetadata,
+		ChannelMemberHistories: data.ChannelMemberHistories,
+		PostsToExport:          data.PostsToExport,
+		JobStartTime:           data.JobStartTime,
+		BatchPath:              data.BatchPath,
+		BatchStartTime:         data.BatchStartTime,
+		BatchEndTime:           data.BatchEndTime,
+	}
+}
+
+func RunExportByType(rctx request.CTX, p ExportParams, b shared.BackendParams) (results shared.RunExportResults, err error) {
+	preparePosts(rctx, p.PostsToExport)
+
+	exportParams := shared.ExportParams{
+		ExportType:             p.ExportType,
+		ChannelMetadata:        p.ChannelMetadata,
+		Posts:                  p.PostsToExport,
+		ChannelMemberHistories: p.ChannelMemberHistories,
+		JobStartTime:           p.JobStartTime,
+		BatchPath:              p.BatchPath,
+		BatchStartTime:         p.BatchStartTime,
+		BatchEndTime:           p.BatchEndTime,
+		Config:                 b.Config,
+		Db:                     b.Store,
+		FileAttachmentBackend:  b.FileAttachmentBackend,
+		ExportBackend:          b.ExportBackend,
+		Templates:              b.HtmlTemplates,
+	}
+
+	switch p.ExportType {
+	case model.ComplianceExportTypeCsv:
+		rctx.Logger().Debug("Exporting CSV")
+		return csv_export.CsvExport(rctx, exportParams)
+
+	case model.ComplianceExportTypeActiance:
+		rctx.Logger().Debug("Exporting Actiance")
+		return actiance_export.ActianceExport(rctx, exportParams)
+
+	case model.ComplianceExportTypeGlobalrelay, model.ComplianceExportTypeGlobalrelayZip:
+		rctx.Logger().Debug("Exporting GlobalRelay")
+		return global_relay_export.GlobalRelayExport(rctx, exportParams)
+
+	default:
+		return results, errors.New("Unknown output format: " + p.ExportType)
+	}
+}
+
+func preparePosts(rctx request.CTX, postsToExport []*model.MessageExport) {
 	// go through all the posts and if the post's props contain 'from_bot' - override the IsBot field, since it's possible that the sender is not a user, but was a Bot and vise-versa
 	for _, post := range postsToExport {
 		if post.PostProps != nil {
@@ -178,57 +271,4 @@ func runExportByType(rctx request.CTX, exportType string, postsToExport []*model
 			post.PostCreateAt = new(int64)
 		}
 	}
-
-	switch exportType {
-	case model.ComplianceExportTypeCsv:
-		rctx.Logger().Debug("Exporting CSV")
-		return csv_export.CsvExport(rctx, postsToExport, db, exportBackend, fileAttachmentBackend, exportDirectory)
-
-	case model.ComplianceExportTypeActiance:
-		rctx.Logger().Debug("Exporting Actiance")
-		return actiance_export.ActianceExport(rctx, postsToExport, db, exportBackend, fileAttachmentBackend, exportDirectory)
-
-	case model.ComplianceExportTypeGlobalrelay, model.ComplianceExportTypeGlobalrelayZip:
-		rctx.Logger().Debug("Exporting GlobalRelay")
-		f, err := os.CreateTemp("", "")
-		if err != nil {
-			return warningCount, model.NewAppError("RunExport", "ent.compliance.global_relay.open_temporary_file.appError", nil, "", http.StatusAccepted).Wrap(err)
-		}
-		defer f.Close()
-		defer os.Remove(f.Name())
-
-		attachmentsRemovedPostIDs, warnings, appErr := global_relay_export.GlobalRelayExport(rctx, postsToExport, db, fileAttachmentBackend, f, htmlTemplates)
-		if appErr != nil {
-			return warningCount, appErr
-		}
-		warningCount = warnings
-		_, err = f.Seek(0, 0)
-		if err != nil {
-			return warningCount, model.NewAppError("RunExport", "ent.compliance.global_relay.rewind_temporary_file.appError", nil, "", http.StatusAccepted).Wrap(err)
-		}
-
-		if exportType == model.ComplianceExportTypeGlobalrelayZip {
-			// Try to disable the write timeout for the potentially big export file.
-			_, nErr := filestore.TryWriteFileContext(rctx.Context(), exportBackend, f, path.Join(exportDirectory, GlobalRelayExportFilename))
-			if nErr != nil {
-				return warningCount, model.NewAppError("runExportByType", "ent.compliance.global_relay.write_file.appError", nil, "", http.StatusInternalServerError).Wrap(nErr)
-			}
-		} else {
-			appErr = global_relay_export.Deliver(f, config)
-			if appErr != nil {
-				return warningCount, appErr
-			}
-		}
-
-		if len(attachmentsRemovedPostIDs) > 0 {
-			rctx.Logger().Debug("Global Relay Attachments Removed because they were too large to send to Global Relay", mlog.Array("attachment_ids", attachmentsRemovedPostIDs))
-			description := fmt.Sprintf("Attachments to post IDs %v were removed because they were too large to send to Global Relay.", attachmentsRemovedPostIDs)
-			appErr = model.NewAppError("RunExport", "ent.compliance.global_relay.attachments_removed.appError", map[string]any{"Description": description}, description, http.StatusAccepted)
-			return warningCount, appErr
-		}
-	default:
-		err := errors.New("Unknown output format " + exportType)
-		return warningCount, model.NewAppError("RunExport", "ent.compliance.bad_export_type.appError", map[string]any{"ExportType": exportType}, "", http.StatusBadRequest).Wrap(err)
-	}
-	return warningCount, nil
 }

server/enterprise/message_export/scheduler.go

@@ -39,7 +39,7 @@ func (s *MessageExportScheduler) NextScheduleTime(cfg *model.Config, now time.Ti
 	if err != nil {
 		s.jobServer.Logger().Error(
 			"Cannot determine next schedule time for message export. DailyRunTime config value is invalid.",
-			mlog.String("DailyRunTime", *cfg.MessageExportSettings.DailyRunTime),
+			mlog.String("daily_run_time", *cfg.MessageExportSettings.DailyRunTime),
 		)
 		return nil
 	}

server/enterprise/message_export/shared/export_data.go

@@ -0,0 +1,543 @@
+// Copyright (c) 2015-present Mattermost, Inc. All Rights Reserved.
+// See LICENSE.enterprise for license information.
+
+package shared
+
+import (
+	"sort"
+
+	"github.com/mattermost/mattermost/server/public/model"
+)
+
+type UserType string
+
+const (
+	User UserType = "user"
+	Bot  UserType = "bot"
+)
+
+type PostExport struct {
+	model.MessageExport          // the MessageExport that this PostExport is providing more information for
+	UserType            UserType // the type of the person that sent the post: "user" or "bot"
+
+	// Allows us to differentiate between:
+	// - "EditedOriginalMsg": the newly created message (new Id), which holds the pre-edited message contents. The
+	//    "EditedNewMsgId" field will point to the message (original Id) which has the post-edited message content.
+	// - "EditedNewMsg": the post-edited message content. This is confusing, so be careful: in the db, this EditedNewMsg
+	//    is actually the original messageId because we wanted an edited message to have the same messageId as the
+	//    pre-edited message. But for the purposes of exporting and to keep the mental model clear for end-users, we are
+	//    calling this the EditedNewMsg and EditedNewMsgId, because this will hold the NEW post-edited message contents,
+	//    and that's what's important to the end-user viewing the export.
+	//  - "UpdatedNoMsgChange": the message content hasn't changed, but the post was updated for some reason (reaction,
+	//	  replied-to, a reply was edited, a reply was deleted (as of 10.2), perhaps other reasons)
+	//  - "Deleted": the message was deleted.
+	//  - "FileDeleted": this message is recording that a file was deleted.
+	UpdatedType PostUpdatedType
+	UpdateAt    int64 // if this is an updated post, this is the updated time (same as deleted time for deleted posts).
+
+	// when a message is edited, the EditedOriginalMsg points to the message Id that now has the newly edited message.
+	EditedNewMsgId    string
+	Message           string                   // the text body of the post
+	PreviewsPost      string                   // the post id of the post that is previewed by the permalink preview feature
+	AttachmentCreates []*FileUploadStartExport // the post's attachments that were uploaded this export period
+	AttachmentDeletes []PostExport             // the post's attachments that were deleted
+	FileInfo          *model.FileInfo          // if this was a file PostExport, FileInfo will contain that info. Otherwise, nil.
+}
+
+type FileUploadStartExport struct {
+	model.MessageExport        // the post that this upload was attached to
+	UserEmail           string // the email of the person that sent the file
+	UploadStartTime     int64  // utc timestamp (seconds), time at which the user started the upload. Example: 1366611728
+	FileInfo            *model.FileInfo
+}
+
+type FileUploadStopExport struct {
+	model.MessageExport        // the post that this upload was attached to
+	UserEmail           string // the email of the person that sent the file
+	UploadStopTime      int64  // utc timestamp (seconds), time at which the user finished the upload. Example: 1366611728
+	Status              string // set to either "Completed" or "Failed" depending on the outcome of the upload operation
+	FileInfo            *model.FileInfo
+}
+
+type ChannelExport struct {
+	ChannelId    string
+	ChannelType  model.ChannelType
+	ChannelName  string
+	DisplayName  string
+	StartTime    int64 // utc timestamp (milliseconds), start of export period or create time of channel, whichever is greater.
+	EndTime      int64 // utc timestamp (milliseconds), end of export period or delete time of channel, whichever is lesser.
+	Posts        []PostExport
+	Files        []*model.FileInfo
+	DeletedFiles []PostExport
+	UploadStarts []*FileUploadStartExport
+	UploadStops  []*FileUploadStopExport
+	JoinEvents   []JoinExport  // start with a list of all users who were present in the channel during the export period
+	LeaveEvents  []LeaveExport // finish with a list of all users who were present in the channel during the export period
+
+	// Used by csv, ignored by others
+	TeamId          string
+	TeamName        string
+	TeamDisplayName string
+}
+
+type JoinExport struct {
+	UserId    string
+	Username  string
+	UserEmail string   // the email of the person that joined the channel
+	UserType  UserType // the type of the user that joined the channel
+	JoinTime  int64    // utc timestamp (seconds), time at which the user joined. Example: 1366611728
+
+	// when the user left (or batch endTime if they didn't leave). Only used by GlobalRelay
+	LeaveTime int64
+}
+
+type LeaveExport struct {
+	UserId    string
+	Username  string
+	UserEmail string   // the email of the person that left the channel
+	UserType  UserType // the type of the user that left the channel
+	LeaveTime int64    // utc timestamp (seconds), time at which the user left. Example: 1366611728
+
+	// ClosedOut indicates this is a "leave" event created by closing out the channel at the end of an export period.
+	// Actiance requires all users to be closed out at the end of an export period (each join has a matching leave).
+	ClosedOut bool
+}
+
+type GenericExportData struct {
+	Exports  []ChannelExport
+	Metadata Metadata
+	Results  RunExportResults
+}
+
+// GetGenericExportData assembles all the data in an exportType-agnostic way. Each exportType will process this data into
+// the specific format they need to export.
+func GetGenericExportData(p ExportParams) (GenericExportData, error) {
+	// postAuthorsByChannel is a map so that we don't store duplicate authors
+	postAuthorsByChannel := make(map[string]map[string]ChannelMember)
+	metadata := Metadata{
+		Channels:         p.ChannelMetadata,
+		MessagesCount:    0,
+		AttachmentsCount: 0,
+		StartTime:        p.BatchStartTime,
+		EndTime:          p.BatchEndTime,
+	}
+
+	var results RunExportResults
+	channelsInThisBatch := make(map[string]bool)
+	postsByChannel := make(map[string][]PostExport)
+	filesByChannel := make(map[string][]*model.FileInfo)
+	uploadStartsByChannel := make(map[string][]*FileUploadStartExport)
+	uploadStopsByChannel := make(map[string][]*FileUploadStopExport)
+	deletedFilesByChannel := make(map[string][]PostExport)
+
+	processPostAttachments := func(post *model.MessageExport, postExport PostExport, originalPostThatWillBeDeletedLater bool) error {
+		// originalPostThatWillBeDeletedLater means we are recording this message's original file starts and stops,
+		// before it was deleted (we'll record that next call to this function)
+		//
+		// NOTE: there is an edge case here: the original post is not deleted but the attachment has been deleted.
+		//       See the note in the postToAttachmentsEntries func.
+
+		channelId := *post.ChannelId
+		uploadedFiles, startUploads, stopUploads, deleteFileMessages, err :=
+			postToAttachmentsEntries(post, p.Db, originalPostThatWillBeDeletedLater)
+		if err != nil {
+			return err
+		}
+		uploadStartsByChannel[channelId] = append(uploadStartsByChannel[channelId], startUploads...)
+		uploadStopsByChannel[channelId] = append(uploadStopsByChannel[channelId], stopUploads...)
+		deletedFilesByChannel[channelId] = append(deletedFilesByChannel[channelId], deleteFileMessages...)
+		filesByChannel[channelId] = append(filesByChannel[channelId], uploadedFiles...)
+
+		postExport.AttachmentCreates = startUploads
+		postExport.AttachmentDeletes = deleteFileMessages
+		postsByChannel[channelId] = append(postsByChannel[channelId], postExport)
+
+		results.UploadedFiles += len(startUploads)
+		results.DeletedFiles += len(deleteFileMessages)
+
+		// only count uploaded files (not deleted files)
+		if err := metadata.UpdateCounts(channelId, 1, len(startUploads)); err != nil {
+			return err
+		}
+
+		return nil
+	}
+
+	for _, post := range p.Posts {
+		channelId := *post.ChannelId
+		channelsInThisBatch[channelId] = true
+
+		// Was the post deleted (not an edited post), and originally posted during the current job window?
+		// If so, we need to record it. It may actually belong in an earlier batch, but there's no way to know that
+		// before now because of the way we export posts (by updateAt).
+		if IsDeletedMsg(post) && !isEditedOriginalMsg(post) && *post.PostCreateAt >= p.JobStartTime {
+			results.CreatedPosts++
+			postExport := createdPostToExportEntry(post)
+			if err := processPostAttachments(post, postExport, true); err != nil {
+				return GenericExportData{}, err
+			}
+		}
+		var postExport PostExport
+		postExport, results = getPostExport(post, results)
+
+		if err := processPostAttachments(post, postExport, false); err != nil {
+			return GenericExportData{}, err
+		}
+
+		if _, ok := postAuthorsByChannel[channelId]; !ok {
+			postAuthorsByChannel[channelId] = make(map[string]ChannelMember)
+		}
+		postAuthorsByChannel[channelId][*post.UserId] = ChannelMember{
+			UserId:   *post.UserId,
+			Email:    *post.UserEmail,
+			Username: *post.Username,
+			IsBot:    post.IsBot,
+		}
+	}
+
+	// If the channel is not in channelsInThisBatch (i.e. if it didn't have a post), we need to check if it had
+	// user activity between this batch's startTime-endTime. If so, add it to the channelsInThisBatch.
+	for id := range p.ChannelMetadata {
+		if !channelsInThisBatch[id] {
+			if ChannelHasActivity(p.ChannelMemberHistories[id], p.BatchStartTime, p.BatchEndTime) {
+				channelsInThisBatch[id] = true
+			}
+		}
+	}
+
+	// Build the channel exports for the channels that had post or user join/leave activity this batch.
+	channelExports := make([]ChannelExport, 0, len(channelsInThisBatch))
+	for id := range channelsInThisBatch {
+		c := metadata.Channels[id]
+
+		joinEvents, leaveEvents := getJoinsAndLeaves(p.BatchStartTime, p.BatchEndTime,
+			p.ChannelMemberHistories[id], postAuthorsByChannel[id])
+
+		// We don't have teamName and teamDisplayName from the channelMetaData, but we have it from MessageExport.
+		// However, if we don't have posts for this channel (only joins and leaves), then we don't have it at all.
+		var teamName, teamDisplayName string
+		if posts, ok := postsByChannel[id]; ok {
+			if len(posts) > 0 {
+				teamName = model.SafeDereference(posts[0].TeamName)
+				teamDisplayName = model.SafeDereference(posts[0].TeamDisplayName)
+			}
+		}
+
+		channelExports = append(channelExports, ChannelExport{
+			ChannelId:       c.ChannelId,
+			ChannelType:     c.ChannelType,
+			ChannelName:     c.ChannelName,
+			DisplayName:     c.ChannelDisplayName,
+			StartTime:       p.BatchStartTime,
+			EndTime:         p.BatchEndTime,
+			Posts:           postsByChannel[id],
+			Files:           filesByChannel[id],
+			DeletedFiles:    deletedFilesByChannel[id],
+			UploadStarts:    uploadStartsByChannel[id],
+			UploadStops:     uploadStopsByChannel[id],
+			JoinEvents:      joinEvents,
+			LeaveEvents:     leaveEvents,
+			TeamId:          model.SafeDereference(c.TeamId),
+			TeamName:        teamName,
+			TeamDisplayName: teamDisplayName,
+		})
+
+		results.Joins += len(joinEvents)
+		results.Leaves += len(leaveEvents)
+	}
+
+	return GenericExportData{channelExports, metadata, results}, nil
+}
+
+// postToAttachmentsEntries returns every fileInfo as uploadedFiles. It also adds each file into the lists:
+//
+//		startUploads, stopUploads, and deleteFileMessages (for ActianceExport).
+//	 If onlyDeleted = true, only export the deleted entry (if it exists).
+func postToAttachmentsEntries(post *model.MessageExport, db MessageExportStore, ignoreDeleted bool) (
+	uploadedFiles []*model.FileInfo, startUploads []*FileUploadStartExport, stopUploads []*FileUploadStopExport, deleteFileMessages []PostExport, err error) {
+	// if the post included any files, we need to add special elements to the export.
+	//
+	// NOTE: there is an edge case here: the original post is not deleted but the attachment has been deleted.
+	//  1. If  the attachment is deleted sometime later in the future but the post has not been updated (updateAt == createAt)
+	//     then we won't ever get here, there's nothing we can do.
+	//  2. If the attachment is deleted within this export period, or the post is updated this export period and we
+	//     now see that the attachment is deleted, then we have to output 2 files here:
+	//      a) the original file attachment
+	//      b) the deleted file attachment
+	//     These have to be added the start/stop and deletedFileMessages
+
+	if len(post.PostFileIds) == 0 {
+		return
+	}
+
+	uploadedFiles, err = db.FileInfo().GetForPost(*post.PostId, true, true, false)
+	if err != nil {
+		return
+	}
+
+	for _, fileInfo := range uploadedFiles {
+		if fileInfo.DeleteAt > 0 && !ignoreDeleted {
+			deleteFileMessages = append(deleteFileMessages, deleteFileToExportEntry(post, fileInfo))
+
+			// this was a deleted file, so do not record its start and stop. If the original message was sent in this
+			// batch, the file transfer will have been exported earlier up when the original message was exported.
+			//
+			// However, because of the edge case above, we still need to record start and stop if the post is not deleted.
+			if IsDeletedMsg(post) {
+				continue
+			} // not deleted, so need to add start and stop below.
+		}
+
+		// insert a record of the file upload into the export file
+		// path to exported file is relative to the fileAttachmentFilestore root,
+		// which could be different from the exportFilestore root
+		startUploads = append(startUploads, &FileUploadStartExport{
+			MessageExport:   *post,
+			UserEmail:       *post.UserEmail,
+			UploadStartTime: *post.PostCreateAt,
+			FileInfo:        fileInfo,
+		})
+
+		stopUploads = append(stopUploads, &FileUploadStopExport{
+			MessageExport:  *post,
+			UserEmail:      *post.UserEmail,
+			UploadStopTime: *post.PostCreateAt,
+			Status:         "Completed",
+			FileInfo:       fileInfo,
+		})
+	}
+	return
+}
+
+func getPostExport(post *model.MessageExport, results RunExportResults) (PostExport, RunExportResults) {
+	// We have three "kinds" of posts:
+	// (using "1" and "2" for simplicity)
+	// - created:                         Id = new,  CreateAt = 1,    UpdateAt = 1, DeleteAt = 0
+	// - deleted:                         Id = orig, CreateAt = orig, UpdateAt = 2, DeleteAt = 2, props: deleteBy
+	// - edited: old post gets "created": Id = new,  CreateAt = 1,    UpdateAt = 2, DeleteAt = 2, originalId: orig
+	//           existing post modified:  Id = orig, CreateAt = 1,    UpdateAt = 2, DeleteAt = 0
+	//
+	// We also have other ways for a post to be updated:
+	//  - a root post in a thread is replied to, when a reply is edited, or (as of 10.2) when a reply is deleted
+
+	if isEditedOriginalMsg(post) {
+		// Post has been edited. This is the original message.
+		results.EditedOrigMsgPosts++
+		return editedOriginalMsgToExportEntry(post), results
+	} else if IsDeletedMsg(post) {
+		// Post is deleted
+		results.DeletedPosts++
+		return deletedPostToExportEntry(post, "delete "+*post.PostMessage), results
+	} else if *post.PostUpdateAt > *post.PostCreateAt {
+		// Post has been updated. But what kind?
+		if model.SafeDereference(post.PostEditAt) > 0 {
+			// This is an edited post.
+			results.EditedNewMsgPosts++
+			return editedNewMsgToExportEntry(post), results
+		}
+		// This is just an updated post (e.g. reaction)
+		results.UpdatedPosts++
+		return updatedPostToExportEntry(post), results
+	}
+	// Post is newly created:
+	// *post.PostCreateAt == *post.PostUpdateAt && (post.PostDeleteAt == nil || *post.PostDeleteAt == 0)
+	// but also fallback to this in case there is missing data, which is better than not exporting anything.
+	results.CreatedPosts++
+	return createdPostToExportEntry(post), results
+}
+
+func getJoinsAndLeaves(startTime int64, endTime int64, channelMembersHistory []*model.ChannelMemberHistoryResult,
+	postAuthors map[string]ChannelMember) ([]JoinExport, []LeaveExport) {
+	var leaveEvents []LeaveExport
+
+	joins, leaves := GetJoinsAndLeavesForChannel(startTime, endTime, channelMembersHistory, postAuthors)
+	joinsById := make(map[string]JoinExport, len(joins))
+	type StillMemberInfo struct {
+		time     int64
+		userType UserType
+		userId   string
+		username string
+	}
+	stillMember := map[string]StillMemberInfo{}
+	for _, join := range joins {
+		userType := User
+		if join.IsBot {
+			userType = Bot
+		}
+		joinsById[join.UserId] = JoinExport{
+			UserId:    join.UserId,
+			Username:  join.Username,
+			UserEmail: join.Email,
+			JoinTime:  join.Datetime,
+			UserType:  userType,
+			LeaveTime: endTime,
+		}
+		if value, ok := stillMember[join.Email]; !ok {
+			stillMember[join.Email] = StillMemberInfo{time: join.Datetime, userType: userType, userId: join.UserId, username: join.Username}
+		} else if join.Datetime > value.time {
+			stillMember[join.Email] = StillMemberInfo{time: join.Datetime, userType: userType, userId: join.UserId, username: join.Username}
+		}
+	}
+	for _, leave := range leaves {
+		userType := User
+		if leave.IsBot {
+			userType = Bot
+		}
+		leaveEvents = append(leaveEvents, LeaveExport{
+			UserId:    leave.UserId,
+			Username:  leave.Username,
+			UserEmail: leave.Email,
+			LeaveTime: leave.Datetime,
+			UserType:  userType,
+		})
+		if leave.Datetime > stillMember[leave.Email].time {
+			delete(stillMember, leave.Email)
+		}
+
+		// record their leave in their initial join
+		if join, ok := joinsById[leave.UserId]; ok {
+			join.LeaveTime = leave.Datetime
+			joinsById[leave.UserId] = join
+		}
+	}
+
+	// Closing-out the channel for Actiance (each join must have a matching leave).
+	for email := range stillMember {
+		leaveEvents = append(leaveEvents, LeaveExport{
+			UserId:    stillMember[email].userId,
+			Username:  stillMember[email].username,
+			LeaveTime: endTime,
+			UserEmail: email,
+			UserType:  stillMember[email].userType,
+			ClosedOut: true,
+		})
+	}
+
+	joinEvents := make([]JoinExport, 0, len(joinsById))
+	for _, v := range joinsById {
+		joinEvents = append(joinEvents, v)
+	}
+
+	sort.Slice(joinEvents, func(i, j int) bool {
+		if joinEvents[i].JoinTime == joinEvents[j].JoinTime {
+			return joinEvents[i].UserEmail < joinEvents[j].UserEmail
+		}
+		return joinEvents[i].JoinTime < joinEvents[j].JoinTime
+	})
+
+	sort.Slice(leaveEvents, func(i, j int) bool {
+		if leaveEvents[i].LeaveTime == leaveEvents[j].LeaveTime {
+			return leaveEvents[i].UserEmail < leaveEvents[j].UserEmail
+		}
+		return leaveEvents[i].LeaveTime < leaveEvents[j].LeaveTime
+	})
+
+	return joinEvents, leaveEvents
+}
+
+func isEditedOriginalMsg(post *model.MessageExport) bool {
+	return model.SafeDereference(post.PostDeleteAt) > 0 && model.SafeDereference(post.PostOriginalId) != ""
+}
+
+func createdPostToExportEntry(post *model.MessageExport) PostExport {
+	userType := User
+	if post.IsBot {
+		userType = Bot
+	}
+	return PostExport{
+		MessageExport: *post,
+		Message:       *post.PostMessage,
+		UserType:      userType,
+		PreviewsPost:  post.PreviewID(),
+	}
+}
+
+func deletedPostToExportEntry(post *model.MessageExport, newMsg string) PostExport {
+	userType := User
+	if post.IsBot {
+		userType = Bot
+	}
+	return PostExport{
+		MessageExport: *post,
+		UpdateAt:      *post.PostDeleteAt,
+		UpdatedType:   Deleted,
+		Message:       newMsg,
+		UserType:      userType,
+		PreviewsPost:  post.PreviewID(),
+	}
+}
+
+func editedOriginalMsgToExportEntry(post *model.MessageExport) PostExport {
+	userType := User
+	if post.IsBot {
+		userType = Bot
+	}
+	return PostExport{
+		MessageExport:  *post,
+		UpdateAt:       *post.PostUpdateAt,
+		UpdatedType:    EditedOriginalMsg,
+		Message:        *post.PostMessage,
+		UserType:       userType,
+		PreviewsPost:   post.PreviewID(),
+		EditedNewMsgId: *post.PostOriginalId,
+	}
+}
+
+func editedNewMsgToExportEntry(post *model.MessageExport) PostExport {
+	userType := User
+	if post.IsBot {
+		userType = Bot
+	}
+	return PostExport{
+		MessageExport: *post,
+		UpdateAt:      *post.PostUpdateAt,
+		UpdatedType:   EditedNewMsg,
+		Message:       *post.PostMessage,
+		UserType:      userType,
+		PreviewsPost:  post.PreviewID(),
+	}
+}
+
+func updatedPostToExportEntry(post *model.MessageExport) PostExport {
+	userType := User
+	if post.IsBot {
+		userType = Bot
+	}
+	return PostExport{
+		MessageExport: *post,
+		UpdateAt:      *post.PostUpdateAt,
+		UpdatedType:   UpdatedNoMsgChange,
+		Message:       *post.PostMessage,
+		UserType:      userType,
+		PreviewsPost:  post.PreviewID(),
+	}
+}
+
+func deleteFileToExportEntry(post *model.MessageExport, fileInfo *model.FileInfo) PostExport {
+	userType := User
+	if post.IsBot {
+		userType = Bot
+	}
+	return PostExport{
+		MessageExport: *post,
+		UpdateAt:      fileInfo.DeleteAt,
+		UpdatedType:   FileDeleted,
+		Message:       "delete " + fileInfo.Path,
+		UserType:      userType,
+		PreviewsPost:  post.PreviewID(),
+		FileInfo:      fileInfo,
+	}
+}
+
+func UploadStartToExportEntry(u *FileUploadStartExport) PostExport {
+	userType := User
+	if u.IsBot {
+		userType = Bot
+	}
+	return PostExport{
+		MessageExport: u.MessageExport,
+		UpdateAt:      u.FileInfo.UpdateAt,
+		UserType:      userType,
+		FileInfo:      u.FileInfo,
+	}
+}

server/enterprise/message_export/shared/export_data_test.go

@@ -0,0 +1,523 @@
+// Copyright (c) 2015-present Mattermost, Inc. All Rights Reserved.
+// See LICENSE.enterprise for license information.
+
+package shared
+
+import (
+	"encoding/json"
+	"fmt"
+	"testing"
+	"time"
+
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/mock"
+	"github.com/stretchr/testify/require"
+
+	"github.com/mattermost/mattermost/server/public/model"
+	"github.com/mattermost/mattermost/server/v8/channels/api4"
+	"github.com/mattermost/mattermost/server/v8/channels/jobs"
+	"github.com/mattermost/mattermost/server/v8/channels/store/storetest"
+)
+
+func postToMessageExport(t *testing.T, p *model.Post, u *model.User, c *model.Channel, team *model.Team) model.MessageExport {
+	t.Helper()
+	props, err := json.Marshal(p.GetProps())
+	assert.NoError(t, err)
+
+	return model.MessageExport{
+		TeamId:             &team.Id,
+		TeamName:           &team.Name,
+		TeamDisplayName:    &team.DisplayName,
+		ChannelId:          &c.Id,
+		ChannelName:        &c.Name,
+		ChannelDisplayName: &c.DisplayName,
+		ChannelType:        &c.Type,
+		UserId:             &u.Id,
+		UserEmail:          model.NewPointer(u.Email),
+		Username:           &u.Username,
+		IsBot:              false,
+		PostId:             model.NewPointer(p.Id),
+		PostCreateAt:       model.NewPointer(p.CreateAt),
+		PostUpdateAt:       &p.UpdateAt,
+		PostDeleteAt:       &p.DeleteAt,
+		PostEditAt:         &p.EditAt,
+		PostMessage:        &p.Message,
+		PostType:           &p.Type,
+		PostRootId:         &p.RootId,
+		PostProps:          model.NewPointer(string(props)),
+		PostOriginalId:     &p.OriginalId,
+		PostFileIds:        p.FileIds,
+	}
+}
+
+func Test_getPostExport(t *testing.T) {
+	if testing.Short() {
+		t.Skip("skipping test in short mode.")
+	}
+
+	jobs.DefaultWatcherPollingInterval = 100
+	th := api4.SetupEnterprise(t).InitBasic()
+	th.App.Srv().SetLicense(model.NewTestLicense("message_export"))
+	defer th.TearDown()
+
+	// the post exports from the db will be random (because they all have the same updateAt), so do it a few times
+	for i := 0; i < 10; i++ {
+		time.Sleep(time.Millisecond)
+		start := model.GetMillis()
+
+		count, err := th.App.Srv().Store().Post().AnalyticsPostCount(&model.PostCountOptions{ExcludeSystemPosts: true, SincePostID: "", SinceUpdateAt: start})
+		require.NoError(t, err)
+		require.Equal(t, 0, int(count))
+
+		var posts []*model.Post
+
+		// 0 - post edited with 3 simultaneous posts in-between - forward
+		// original post with edited message
+		originalPost, err := th.App.Srv().Store().Post().Save(th.Context, &model.Post{
+			ChannelId: th.BasicChannel.Id,
+			UserId:    th.BasicUser.Id,
+			Message:   "message 0",
+		})
+		require.NoError(t, err)
+		require.NotEqual(t, 0, originalPost.UpdateAt, "originalPost's updateAt was zero, test 1")
+		posts = append(posts, originalPost)
+
+		// If we don't sleep, the two messages might not have different CreateAt and UpdateAts
+		time.Sleep(time.Millisecond)
+
+		// 1 - edited post
+		post, err := th.App.Srv().Store().Post().Update(th.Context, &model.Post{
+			Id:        originalPost.Id,
+			CreateAt:  originalPost.CreateAt,
+			EditAt:    model.GetMillis(),
+			ChannelId: th.BasicChannel.Id,
+			UserId:    th.BasicUser.Id,
+			Message:   "edited message 0",
+		}, originalPost)
+		require.NoError(t, err)
+		require.NotEqual(t, 0, originalPost.UpdateAt, "originalPost's updateAt was zero, test 2")
+		require.NotEqual(t, 0, post.UpdateAt, "edited post's updateAt was zero, test 2")
+		posts = append(posts, post)
+
+		simultaneous := post.UpdateAt
+
+		// Add 8 other posts at the same updateAt
+		for j := 1; j <= 8; j++ {
+			// 2 - post 1 at same updateAt
+			post, err = th.App.Srv().Store().Post().Save(th.Context, &model.Post{
+				ChannelId: th.BasicChannel.Id,
+				UserId:    th.BasicUser.Id,
+				Message:   fmt.Sprintf("message %d", j),
+				CreateAt:  simultaneous,
+			})
+			require.NoError(t, err)
+			require.NotEqual(t, 0, post.UpdateAt)
+			posts = append(posts, post)
+		}
+
+		// Use the config fallback for simplicity
+		th.App.UpdateConfig(func(cfg *model.Config) {
+			*cfg.MessageExportSettings.EnableExport = true
+			*cfg.MessageExportSettings.ExportFromTimestamp = start
+			*cfg.MessageExportSettings.BatchSize = 10
+		})
+
+		// the messages can be in any order because all have equal `updateAt`s
+		expectedExports := []PostExport{
+			{
+				MessageExport:  postToMessageExport(t, posts[0], th.BasicUser, th.BasicChannel, th.BasicTeam),
+				UserType:       "user",
+				Message:        posts[0].Message,
+				UpdateAt:       posts[1].UpdateAt, // the edit update at
+				UpdatedType:    EditedOriginalMsg,
+				EditedNewMsgId: posts[1].Id,
+			},
+			{
+				MessageExport: postToMessageExport(t, posts[1], th.BasicUser, th.BasicChannel, th.BasicTeam),
+				UserType:      "user",
+				Message:       posts[1].Message,
+				UpdateAt:      posts[1].UpdateAt,
+				UpdatedType:   EditedNewMsg,
+			},
+		}
+
+		for j := 2; j < 10; j++ {
+			expectedExports = append(expectedExports, PostExport{
+				MessageExport: postToMessageExport(t, posts[j], th.BasicUser, th.BasicChannel, th.BasicTeam),
+				UserType:      "user",
+				Message:       posts[j].Message,
+			})
+		}
+
+		actualMessageExports, _, err := th.App.Srv().Store().Compliance().MessageExport(th.Context, model.MessageExportCursor{
+			LastPostUpdateAt: start,
+			UntilUpdateAt:    model.GetMillis(),
+		}, 10)
+		require.NoError(t, err)
+		require.Len(t, actualMessageExports, 10)
+		for _, export := range actualMessageExports {
+			require.NotEqual(t, 0, *export.PostUpdateAt)
+		}
+		results := RunExportResults{}
+		var actualExports []PostExport
+
+		for _, m := range actualMessageExports {
+			var postExport PostExport
+			postExport, results = getPostExport(m, results)
+			actualExports = append(actualExports, postExport)
+		}
+
+		require.ElementsMatch(t, expectedExports, actualExports, fmt.Sprintf("batch %d", i))
+	}
+}
+
+func TestPostToAttachmentsEntries(t *testing.T) {
+	chanTypeDirect := model.ChannelTypeDirect
+	tt := []struct {
+		name                       string
+		post                       model.MessageExport
+		attachments                []*model.FileInfo
+		expectedStarts             []*FileUploadStartExport
+		expectedStops              []*FileUploadStopExport
+		expectedFileInfos          []*model.FileInfo
+		expectedDeleteFileMessages []PostExport
+		ignoreDeleted              bool
+		expectError                bool
+	}{
+		{
+			name: "no-attachments",
+			post: model.MessageExport{
+				ChannelId:          model.NewPointer("Test"),
+				ChannelDisplayName: model.NewPointer("Test"),
+				PostCreateAt:       model.NewPointer(int64(1)),
+				PostMessage:        model.NewPointer("Some message"),
+				UserEmail:          model.NewPointer("test@test.com"),
+				UserId:             model.NewPointer("test"),
+				Username:           model.NewPointer("test"),
+				ChannelType:        &chanTypeDirect,
+			},
+			attachments:                nil,
+			expectedStarts:             nil,
+			expectedStops:              nil,
+			expectedFileInfos:          nil,
+			expectedDeleteFileMessages: nil,
+			expectError:                false,
+		},
+		{
+			name: "one-attachment",
+			post: model.MessageExport{
+				PostId:             model.NewPointer("test"),
+				ChannelId:          model.NewPointer("Test"),
+				ChannelDisplayName: model.NewPointer("Test"),
+				PostCreateAt:       model.NewPointer(int64(1)),
+				PostMessage:        model.NewPointer("Some message"),
+				UserEmail:          model.NewPointer("test@test.com"),
+				UserId:             model.NewPointer("test"),
+				Username:           model.NewPointer("test"),
+				ChannelType:        &chanTypeDirect,
+				PostFileIds:        []string{"12345"},
+			},
+			attachments: []*model.FileInfo{
+				{Name: "test", Id: "12345", Path: "filename.txt"},
+			},
+			expectedStarts: []*FileUploadStartExport{
+				{UserEmail: "test@test.com", UploadStartTime: 1,
+					FileInfo: &model.FileInfo{Id: "12345", Name: "test", Path: "filename.txt"}},
+			},
+			expectedStops: []*FileUploadStopExport{
+				{UserEmail: "test@test.com", UploadStopTime: 1,
+					FileInfo: &model.FileInfo{Id: "12345", Name: "test", Path: "filename.txt"}, Status: "Completed"},
+			},
+			expectedFileInfos: []*model.FileInfo{
+				{Name: "test", Id: "12345", Path: "filename.txt"},
+			},
+			expectedDeleteFileMessages: nil,
+			expectError:                false,
+		},
+		{
+			name: "two-attachment",
+			post: model.MessageExport{
+				PostId:             model.NewPointer("test"),
+				ChannelId:          model.NewPointer("Test"),
+				ChannelDisplayName: model.NewPointer("Test"),
+				PostCreateAt:       model.NewPointer(int64(1)),
+				PostMessage:        model.NewPointer("Some message"),
+				UserEmail:          model.NewPointer("test@test.com"),
+				UserId:             model.NewPointer("test"),
+				Username:           model.NewPointer("test"),
+				ChannelType:        &chanTypeDirect,
+				PostFileIds:        []string{"12345", "54321"},
+			},
+			attachments: []*model.FileInfo{
+				{Name: "test", Id: "12345", Path: "filename.txt"},
+				{Name: "test2", Id: "54321", Path: "filename2.txt"},
+			},
+			expectedStarts: []*FileUploadStartExport{
+				{UserEmail: "test@test.com", UploadStartTime: 1,
+					FileInfo: &model.FileInfo{Id: "12345", Name: "test", Path: "filename.txt"}},
+				{UserEmail: "test@test.com", UploadStartTime: 1,
+					FileInfo: &model.FileInfo{Id: "54321", Name: "test2", Path: "filename2.txt"}},
+			},
+			expectedStops: []*FileUploadStopExport{
+				{UserEmail: "test@test.com", UploadStopTime: 1,
+					FileInfo: &model.FileInfo{Id: "12345", Name: "test", Path: "filename.txt"}, Status: "Completed"},
+				{UserEmail: "test@test.com", UploadStopTime: 1,
+					FileInfo: &model.FileInfo{Id: "54321", Name: "test2", Path: "filename2.txt"}, Status: "Completed"},
+			},
+			expectedFileInfos: []*model.FileInfo{
+				{Name: "test", Id: "12345", Path: "filename.txt"},
+				{Name: "test2", Id: "54321", Path: "filename2.txt"},
+			},
+			expectedDeleteFileMessages: nil,
+			expectError:                false,
+		},
+		{
+			name: "one-attachment-deleted",
+			post: model.MessageExport{
+				PostId:             model.NewPointer("test"),
+				ChannelId:          model.NewPointer("Test"),
+				ChannelDisplayName: model.NewPointer("Test"),
+				PostCreateAt:       model.NewPointer(int64(1)),
+				PostDeleteAt:       model.NewPointer(int64(2)),
+				PostMessage:        model.NewPointer("Some message"),
+				UserEmail:          model.NewPointer("test@test.com"),
+				UserId:             model.NewPointer("test"),
+				Username:           model.NewPointer("test"),
+				ChannelType:        &chanTypeDirect,
+				PostFileIds:        []string{"12345", "54321"},
+			},
+			attachments: []*model.FileInfo{
+				{Name: "test", Id: "12345", Path: "filename.txt", DeleteAt: 2},
+			},
+			expectedStarts: []*FileUploadStartExport{
+				{UserEmail: "test@test.com", UploadStartTime: 1,
+					FileInfo: &model.FileInfo{Id: "12345", Name: "test", Path: "filename.txt", DeleteAt: 2}},
+			},
+			expectedStops: []*FileUploadStopExport{
+				{UserEmail: "test@test.com", UploadStopTime: 1,
+					FileInfo: &model.FileInfo{Id: "12345", Name: "test", Path: "filename.txt", DeleteAt: 2}, Status: "Completed"},
+			},
+			expectedFileInfos: []*model.FileInfo{
+				{Name: "test", Id: "12345", Path: "filename.txt", DeleteAt: 2},
+			},
+			expectedDeleteFileMessages: []PostExport{
+				{
+					UserType:    "user",
+					UpdatedType: FileDeleted,
+					UpdateAt:    2,
+					Message:     "delete " + "filename.txt",
+					FileInfo:    &model.FileInfo{Id: "12345", Name: "test", Path: "filename.txt", DeleteAt: 2},
+				},
+			},
+			expectError: false,
+		},
+		{
+			name: "one-attachment-deleted, ignore it (only record the creation)",
+			post: model.MessageExport{
+				PostId:             model.NewPointer("test"),
+				ChannelId:          model.NewPointer("Test"),
+				ChannelDisplayName: model.NewPointer("Test"),
+				PostCreateAt:       model.NewPointer(int64(1)),
+				PostDeleteAt:       model.NewPointer(int64(2)),
+				PostMessage:        model.NewPointer("Some message"),
+				UserEmail:          model.NewPointer("test@test.com"),
+				UserId:             model.NewPointer("test"),
+				Username:           model.NewPointer("test"),
+				ChannelType:        &chanTypeDirect,
+				PostFileIds:        []string{"12345", "54321"},
+			},
+			attachments: []*model.FileInfo{
+				{Name: "test", Id: "12345", Path: "filename.txt", DeleteAt: 2},
+			},
+			expectedStarts: []*FileUploadStartExport{
+				{UserEmail: "test@test.com", UploadStartTime: 1,
+					FileInfo: &model.FileInfo{Name: "test", Id: "12345", Path: "filename.txt", DeleteAt: 2}},
+			},
+			expectedStops: []*FileUploadStopExport{
+				{UserEmail: "test@test.com", UploadStopTime: 1,
+					FileInfo: &model.FileInfo{Name: "test", Id: "12345", Path: "filename.txt", DeleteAt: 2}, Status: "Completed"},
+			},
+			expectedFileInfos: []*model.FileInfo{
+				{Name: "test", Id: "12345", Path: "filename.txt", DeleteAt: 2},
+			},
+			expectedDeleteFileMessages: nil,
+			ignoreDeleted:              true,
+			expectError:                false,
+		},
+	}
+
+	for _, tc := range tt {
+		t.Run(tc.name, func(t *testing.T) {
+			mockStore := &storetest.Store{}
+			defer mockStore.AssertExpectations(t)
+
+			if len(tc.attachments) > 0 {
+				call := mockStore.FileInfoStore.On("GetForPost", *tc.post.PostId, true, true, false)
+				call.Run(func(args mock.Arguments) {
+					call.Return(tc.attachments, nil)
+				})
+			}
+			files, uploadStarts, uploadStops, deleteFileMessages, err := postToAttachmentsEntries(&tc.post, NewMessageExportStore(mockStore), tc.ignoreDeleted)
+			if tc.expectError {
+				assert.Error(t, err)
+			} else {
+				assert.NoError(t, err)
+			}
+
+			// Need to add the post to the expected starts (to save from copy pasting in the test cases above)
+			for _, s := range tc.expectedStarts {
+				s.MessageExport = tc.post
+			}
+
+			// Need to add the post to the expected stops
+			for _, s := range tc.expectedStops {
+				s.MessageExport = tc.post
+			}
+
+			// Need to add the post to the expected deleted
+			var expectedDeleted []PostExport
+			for _, s := range tc.expectedDeleteFileMessages {
+				s.MessageExport = tc.post
+				expectedDeleted = append(expectedDeleted, s)
+			}
+
+			assert.Equal(t, tc.expectedStarts, uploadStarts)
+			assert.Equal(t, tc.expectedStops, uploadStops)
+			assert.Equal(t, tc.expectedFileInfos, files)
+			assert.Equal(t, expectedDeleted, deleteFileMessages)
+		})
+	}
+}
+
+func TestGetJoinLeavePosts(t *testing.T) {
+	mockStore := &storetest.Store{}
+	defer mockStore.AssertExpectations(t)
+
+	// This would have been retrieved during CalculateChannelExports
+	channelMemberHistories := map[string][]*model.ChannelMemberHistoryResult{
+		"good-request-1": {
+			{JoinTime: 1, UserId: "test1", UserEmail: "test1", Username: "test1"},
+			{JoinTime: 2, LeaveTime: model.NewPointer(int64(3)), UserId: "test2", UserEmail: "test2", Username: "test2"},
+			{JoinTime: 3, UserId: "test3", UserEmail: "test3", Username: "test3"},
+		},
+		"good-request-2": {
+			{JoinTime: 4, UserId: "test4", UserEmail: "test4", Username: "test4"},
+			{JoinTime: 5, LeaveTime: model.NewPointer(int64(6)), UserId: "test5", UserEmail: "test5", Username: "test5"},
+			{JoinTime: 6, UserId: "test6", UserEmail: "test6", Username: "test6"},
+		},
+	}
+
+	var joins []JoinExport
+	var leaves []LeaveExport
+	for _, id := range []string{"good-request-1", "good-request-2"} {
+		newJoins, newLeaves := getJoinsAndLeaves(
+			1,
+			7,
+			channelMemberHistories[id],
+			nil,
+		)
+		joins = append(joins, newJoins...)
+		leaves = append(leaves, newLeaves...)
+	}
+
+	assert.Len(t, joins, 6)
+	assert.Equal(t, JoinExport{
+		UserId:    "test1",
+		Username:  "test1",
+		UserEmail: "test1",
+		UserType:  User,
+		JoinTime:  1,
+		LeaveTime: 7,
+	}, joins[0])
+	assert.Equal(t, JoinExport{
+		UserId:    "test2",
+		Username:  "test2",
+		UserEmail: "test2",
+		UserType:  User,
+		JoinTime:  2,
+		LeaveTime: 3,
+	}, joins[1])
+	assert.Equal(t, JoinExport{
+		UserId:    "test3",
+		Username:  "test3",
+		UserEmail: "test3",
+		UserType:  User,
+		JoinTime:  3,
+		LeaveTime: 7,
+	}, joins[2])
+	assert.Equal(t, JoinExport{
+		UserId:    "test4",
+		Username:  "test4",
+		UserEmail: "test4",
+		UserType:  User,
+		JoinTime:  4,
+		LeaveTime: 7,
+	}, joins[3])
+	assert.Equal(t, JoinExport{
+		UserId:    "test5",
+		Username:  "test5",
+		UserEmail: "test5",
+		UserType:  User,
+		JoinTime:  5,
+		LeaveTime: 6,
+	}, joins[4])
+	assert.Equal(t, JoinExport{
+		UserId:    "test6",
+		Username:  "test6",
+		UserEmail: "test6",
+		UserType:  User,
+		JoinTime:  6,
+		LeaveTime: 7,
+	}, joins[5])
+
+	// remember that getJoinsAndLeaves sorts _for each channel_
+	assert.Len(t, leaves, 6)
+	// 1st channel:
+	assert.Equal(t, LeaveExport{
+		UserId:    "test2",
+		Username:  "test2",
+		UserEmail: "test2",
+		UserType:  User,
+		LeaveTime: 3,
+	}, leaves[0])
+	assert.Equal(t, LeaveExport{
+		UserId:    "test1",
+		Username:  "test1",
+		UserEmail: "test1",
+		UserType:  User,
+		LeaveTime: 7,
+		ClosedOut: true,
+	}, leaves[1])
+	assert.Equal(t, LeaveExport{
+		UserId:    "test3",
+		Username:  "test3",
+		UserEmail: "test3",
+		UserType:  User,
+		LeaveTime: 7,
+		ClosedOut: true,
+	}, leaves[2])
+	// 2nd channel:
+	assert.Equal(t, LeaveExport{
+		UserId:    "test5",
+		Username:  "test5",
+		UserEmail: "test5",
+		UserType:  User,
+		LeaveTime: 6,
+	}, leaves[3])
+	assert.Equal(t, LeaveExport{
+		UserId:    "test4",
+		Username:  "test4",
+		UserEmail: "test4",
+		UserType:  User,
+		LeaveTime: 7,
+		ClosedOut: true,
+	}, leaves[4])
+	assert.Equal(t, LeaveExport{
+		UserId:    "test6",
+		Username:  "test6",
+		UserEmail: "test6",
+		UserType:  User,
+		LeaveTime: 7,
+		ClosedOut: true,
+	}, leaves[5])
+}

server/enterprise/message_export/shared/main_test.go

@@ -0,0 +1,21 @@
+// Copyright (c) 2015-present Mattermost, Inc. All Rights Reserved.
+// See LICENSE.enterprise for license information.
+
+package shared
+
+import (
+	"testing"
+
+	"github.com/mattermost/mattermost/server/v8/channels/api4"
+	"github.com/mattermost/mattermost/server/v8/channels/testlib"
+)
+
+var mainHelper *testlib.MainHelper
+
+func TestMain(m *testing.M) {
+	mainHelper = testlib.NewMainHelper()
+	defer mainHelper.Close()
+	api4.SetMainHelper(mainHelper)
+
+	mainHelper.Main(m)
+}

server/enterprise/message_export/shared/shared.go

@@ -0,0 +1,609 @@
+// Copyright (c) 2015-present Mattermost, Inc. All Rights Reserved.
+// See LICENSE.enterprise for license information.
+
+package shared
+
+import (
+	"encoding/json"
+	"fmt"
+	"path"
+	"strconv"
+	"time"
+
+	"github.com/pkg/errors"
+
+	"github.com/mattermost/mattermost/server/v8/platform/shared/filestore"
+	"github.com/mattermost/mattermost/server/v8/platform/shared/templates"
+
+	"github.com/mattermost/mattermost/server/public/model"
+	"github.com/mattermost/mattermost/server/public/shared/mlog"
+	"github.com/mattermost/mattermost/server/public/shared/request"
+)
+
+const (
+	MissingFileMessage = "File missing for post; cannot copy file to archive"
+
+	EstimatedPostCount = 10_000_000
+
+	// JobDataBatchStartTime is the posts.updateat value from the previous batch. Posts are selected using
+	// keyset pagination sorted by (posts.updateat, posts.id).
+	JobDataBatchStartTime = "batch_start_time"
+
+	// JobDataJobStartTime is the start of the job (doesn't change across batches)
+	JobDataJobStartTime = "job_start_time"
+
+	// JobDataBatchStartId is the posts.id value from the previous batch.
+	JobDataBatchStartId = "batch_start_id"
+
+	// JobDataJobEndTime is the point up to which this job is exporting. It is the time the job was started,
+	// i.e., we export everything from the end of previous batch to the moment this batch started.
+	JobDataJobEndTime = "job_end_time"
+
+	JobDataJobStartId              = "job_start_id"
+	JobDataExportType              = "export_type"
+	JobDataBatchSize               = "batch_size"
+	JobDataChannelBatchSize        = "channel_batch_size"
+	JobDataChannelHistoryBatchSize = "channel_history_batch_size"
+	JobDataMessagesExported        = "messages_exported"
+	JobDataWarningCount            = "warning_count"
+	JobDataIsDownloadable          = "is_downloadable"
+	JobDataExportDir               = "export_dir"
+	JobDataBatchNumber             = "job_batch_number"
+	JobDataTotalPostsExpected      = "total_posts_expected"
+)
+
+type PostUpdatedType string
+
+const (
+	EditedOriginalMsg  PostUpdatedType = "EditedOriginalMsg"
+	EditedNewMsg       PostUpdatedType = "EditedNewMsg"
+	UpdatedNoMsgChange PostUpdatedType = "UpdatedNoMsgChange"
+	Deleted            PostUpdatedType = "Deleted"
+	FileDeleted        PostUpdatedType = "FileDeleted"
+)
+
+// JobData keeps the current state of the job.
+// When used by a worker, all fields in JobDataExported are exported to the job's job.Data prop bag.
+type JobData struct {
+	JobDataExported
+
+	ExportPeriodStartTime int64
+
+	// This section is the current state of the export
+	ChannelMetadata        map[string]*MetadataChannel
+	ChannelMemberHistories map[string][]*model.ChannelMemberHistoryResult
+	Cursor                 model.MessageExportCursor
+	PostsToExport          []*model.MessageExport
+	BatchEndTime           int64
+	BatchPath              string
+	MessageExportMs        []int64
+	ProcessingPostsMs      []int64
+	ProcessingXmlMs        []int64
+	TransferringFilesMs    []int64
+	TransferringZipMs      []int64
+	TotalBatchMs           []int64
+	Finished               bool
+}
+
+type JobDataExported struct {
+	ExportType              string
+	ExportDir               string
+	BatchStartTime          int64
+	BatchStartId            string
+	JobStartTime            int64
+	JobEndTime              int64
+	JobStartId              string
+	BatchSize               int
+	ChannelBatchSize        int
+	ChannelHistoryBatchSize int
+	BatchNumber             int
+	TotalPostsExpected      int
+	MessagesExported        int
+	WarningCount            int
+	IsDownloadable          bool
+}
+
+func JobDataToStringMap(jd JobData) map[string]string {
+	ret := make(map[string]string)
+	ret[JobDataExportType] = jd.ExportType
+	ret[JobDataExportDir] = jd.ExportDir
+	ret[JobDataBatchStartTime] = strconv.FormatInt(jd.BatchStartTime, 10)
+	ret[JobDataBatchStartId] = jd.BatchStartId
+	ret[JobDataJobStartTime] = strconv.FormatInt(jd.JobStartTime, 10)
+	ret[JobDataJobEndTime] = strconv.FormatInt(jd.JobEndTime, 10)
+	ret[JobDataJobStartId] = jd.JobStartId
+	ret[JobDataBatchSize] = strconv.Itoa(jd.BatchSize)
+	ret[JobDataChannelBatchSize] = strconv.Itoa(jd.ChannelBatchSize)
+	ret[JobDataChannelHistoryBatchSize] = strconv.Itoa(jd.ChannelHistoryBatchSize)
+	ret[JobDataBatchNumber] = strconv.Itoa(jd.BatchNumber)
+	ret[JobDataTotalPostsExpected] = strconv.Itoa(jd.TotalPostsExpected)
+	ret[JobDataMessagesExported] = strconv.Itoa(jd.MessagesExported)
+	ret[JobDataWarningCount] = strconv.Itoa(jd.WarningCount)
+	ret[JobDataIsDownloadable] = strconv.FormatBool(jd.IsDownloadable)
+	return ret
+}
+
+func StringMapToJobDataWithZeroValues(sm map[string]string) (JobData, error) {
+	var jd JobData
+	var err error
+
+	jd.ExportType = sm[JobDataExportType]
+	jd.ExportDir = sm[JobDataExportDir]
+
+	batchStartTime, ok := sm[JobDataBatchStartTime]
+	if !ok {
+		batchStartTime = "0"
+	}
+	if jd.BatchStartTime, err = strconv.ParseInt(batchStartTime, 10, 64); err != nil {
+		return jd, errors.Wrap(err, "error converting JobDataBatchStartTime")
+	}
+
+	jd.BatchStartId = sm[JobDataBatchStartId]
+
+	jobStartTime, ok := sm[JobDataJobStartTime]
+	if !ok {
+		jobStartTime = "0"
+	}
+	if jd.JobStartTime, err = strconv.ParseInt(jobStartTime, 10, 64); err != nil {
+		return jd, errors.Wrap(err, "error converting JobDataJobStartTime")
+	}
+
+	jobEndTime, ok := sm[JobDataJobEndTime]
+	if !ok {
+		jobEndTime = "0"
+	}
+	if jd.JobEndTime, err = strconv.ParseInt(jobEndTime, 10, 64); err != nil {
+		return jd, errors.Wrap(err, "error converting JobDataJobEndTime")
+	}
+
+	jd.JobStartId = sm[JobDataJobStartId]
+
+	jobBatchSize, ok := sm[JobDataBatchSize]
+	if !ok {
+		jobBatchSize = "0"
+	}
+	if jd.BatchSize, err = strconv.Atoi(jobBatchSize); err != nil {
+		return jd, errors.Wrap(err, "error converting JobDataBatchSize")
+	}
+
+	channelBatchSize, ok := sm[JobDataChannelBatchSize]
+	if !ok {
+		channelBatchSize = "0"
+	}
+	if jd.ChannelBatchSize, err = strconv.Atoi(channelBatchSize); err != nil {
+		return jd, errors.Wrap(err, "error converting JobDataChannelBatchSize")
+	}
+
+	channelHistoryBatchSize, ok := sm[JobDataChannelHistoryBatchSize]
+	if !ok {
+		channelHistoryBatchSize = "0"
+	}
+	if jd.ChannelHistoryBatchSize, err = strconv.Atoi(channelHistoryBatchSize); err != nil {
+		return jd, errors.Wrap(err, "error converting JobDataChannelHistoryBatchSize")
+	}
+
+	batchNumber, ok := sm[JobDataBatchNumber]
+	if !ok {
+		batchNumber = "0"
+	}
+	if jd.BatchNumber, err = strconv.Atoi(batchNumber); err != nil {
+		return jd, errors.Wrap(err, "error converting JobDataBatchNumber")
+	}
+
+	totalPostsExpected, ok := sm[JobDataTotalPostsExpected]
+	if !ok {
+		totalPostsExpected = "0"
+	}
+	if jd.TotalPostsExpected, err = strconv.Atoi(totalPostsExpected); err != nil {
+		return jd, errors.Wrap(err, "error converting JobDataTotalPostsExpected")
+	}
+
+	messagesExported, ok := sm[JobDataMessagesExported]
+	if !ok {
+		messagesExported = "0"
+	}
+	if jd.MessagesExported, err = strconv.Atoi(messagesExported); err != nil {
+		return jd, errors.Wrap(err, "error converting JobDataMessagesExported")
+	}
+
+	warningCount, ok := sm[JobDataWarningCount]
+	if !ok {
+		warningCount = "0"
+	}
+	if jd.WarningCount, err = strconv.Atoi(warningCount); err != nil {
+		return jd, errors.Wrap(err, "error converting JobDataWarningCount")
+	}
+
+	isDownloadable, ok := sm[JobDataIsDownloadable]
+	if !ok {
+		isDownloadable = "0"
+	}
+	if jd.IsDownloadable, err = strconv.ParseBool(isDownloadable); err != nil {
+		return jd, errors.Wrap(err, "error converting JobDataIsDownloadable")
+	}
+
+	return jd, nil
+}
+
+type BackendParams struct {
+	Config                *model.Config
+	Store                 MessageExportStore
+	FileAttachmentBackend filestore.FileBackend
+	ExportBackend         filestore.FileBackend
+	HtmlTemplates         *templates.Container
+}
+
+type ExportParams struct {
+	ExportType             string
+	ChannelMetadata        map[string]*MetadataChannel
+	Posts                  []*model.MessageExport
+	ChannelMemberHistories map[string][]*model.ChannelMemberHistoryResult
+	JobStartTime           int64
+	BatchPath              string
+	BatchStartTime         int64
+	BatchEndTime           int64
+	Config                 *model.Config
+	Db                     MessageExportStore
+	FileAttachmentBackend  filestore.FileBackend
+	ExportBackend          filestore.FileBackend
+	Templates              *templates.Container
+}
+
+type WriteExportResult struct {
+	TransferringFilesMs int64
+	ProcessingXmlMs     int64
+	TransferringZipMs   int64
+	NumWarnings         int
+}
+
+type RunExportResults struct {
+	CreatedPosts       int
+	EditedOrigMsgPosts int
+	EditedNewMsgPosts  int
+	UpdatedPosts       int
+	DeletedPosts       int
+	UploadedFiles      int
+	DeletedFiles       int
+	NumChannels        int
+	Joins              int
+	Leaves             int
+	ProcessingPostsMs  int64
+	WriteExportResult
+}
+
+type ChannelMemberJoin struct {
+	UserId   string
+	IsBot    bool
+	Email    string
+	Username string
+	Datetime int64
+}
+
+type ChannelMemberLeave struct {
+	UserId   string
+	IsBot    bool
+	Email    string
+	Username string
+	Datetime int64
+}
+
+type ChannelMember struct {
+	UserId   string
+	IsBot    bool
+	Email    string
+	Username string
+}
+
+type MetadataChannel struct {
+	TeamId             *string
+	TeamName           *string
+	TeamDisplayName    *string
+	ChannelId          string
+	ChannelName        string
+	ChannelDisplayName string
+	ChannelType        model.ChannelType
+	RoomId             string
+	StartTime          int64
+	EndTime            int64
+	MessagesCount      int
+	AttachmentsCount   int
+}
+
+type Metadata struct {
+	Channels         map[string]*MetadataChannel
+	MessagesCount    int
+	AttachmentsCount int
+	StartTime        int64
+	EndTime          int64
+}
+
+func (metadata *Metadata) UpdateCounts(channelId string, numMessages int, numAttachments int) error {
+	_, ok := metadata.Channels[channelId]
+	if !ok {
+		return fmt.Errorf("could not find channelId for post in metadata.Channels")
+	}
+
+	metadata.Channels[channelId].AttachmentsCount += numAttachments
+	metadata.AttachmentsCount += numAttachments
+	metadata.Channels[channelId].MessagesCount += numMessages
+	metadata.MessagesCount += numMessages
+
+	return nil
+}
+
+// GetInitialExportPeriodData calculates and caches the channel memberships, channel metadata, and the TotalPostsExpected.
+func GetInitialExportPeriodData(rctx request.CTX, store MessageExportStore, data JobData, reportProgress func(string)) (JobData, error) {
+	// Counting all posts may fail or timeout when the posts table is large. If this happens, log a warning, but carry
+	// on with the job anyway. The only issue is that the progress % reporting will be inaccurate.
+	count, err := store.Post().AnalyticsPostCount(&model.PostCountOptions{ExcludeSystemPosts: true, SincePostID: data.JobStartId, SinceUpdateAt: data.ExportPeriodStartTime, UntilUpdateAt: data.JobEndTime})
+	if err != nil {
+		rctx.Logger().Warn("Worker: Failed to fetch total post count for job. An estimated value will be used for progress reporting.", mlog.Err(err))
+		data.TotalPostsExpected = EstimatedPostCount
+	} else {
+		data.TotalPostsExpected = int(count)
+	}
+
+	rctx.Logger().Info("Expecting to export total posts", mlog.Int("total_posts", data.TotalPostsExpected))
+
+	// Every time we claim the job, we need to gather the membership data that every batch will use.
+	// If we're here, then either this is the start of the job, or the job was stopped (e.g., the worker stopped)
+	// and we've claimed it again. Either way, we need to recalculate channel and member history data.
+	data.ChannelMetadata, data.ChannelMemberHistories, err = CalculateChannelExports(rctx,
+		ChannelExportsParams{
+			Store:                   store,
+			ExportPeriodStartTime:   data.ExportPeriodStartTime,
+			ExportPeriodEndTime:     data.JobEndTime,
+			ChannelBatchSize:        data.ChannelBatchSize,
+			ChannelHistoryBatchSize: data.ChannelHistoryBatchSize,
+			ReportProgressMessage:   reportProgress,
+		})
+	if err != nil {
+		return data, err
+	}
+
+	data.Cursor = model.MessageExportCursor{
+		LastPostUpdateAt: data.BatchStartTime,
+		LastPostId:       data.BatchStartId,
+		UntilUpdateAt:    data.JobEndTime,
+	}
+
+	return data, nil
+}
+
+type ChannelExportsParams struct {
+	Store                   MessageExportStore
+	ExportPeriodStartTime   int64
+	ExportPeriodEndTime     int64
+	ChannelBatchSize        int
+	ChannelHistoryBatchSize int
+	ReportProgressMessage   func(message string)
+}
+
+// CalculateChannelExports returns the channel info ( map[channelId]*MetadataChannel ) and the channel user
+// joins/leaves ( map[channelId][]*model.ChannelMemberHistoryResult ) for any channel that has had activity
+// (posts or user join/leaves) between ExportPeriodStartTime and ExportPeriodEndTime.
+func CalculateChannelExports(rctx request.CTX, opt ChannelExportsParams) (map[string]*MetadataChannel, map[string][]*model.ChannelMemberHistoryResult, error) {
+	// Which channels had user activity in the export period?
+	activeChannelIds, err := opt.Store.ChannelMemberHistory().GetChannelsWithActivityDuring(opt.ExportPeriodStartTime, opt.ExportPeriodEndTime)
+	if err != nil {
+		return nil, nil, err
+	}
+
+	if len(activeChannelIds) == 0 {
+		return nil, nil, nil
+	}
+
+	rctx.Logger().Debug("Started CalculateChannelExports", mlog.Int("export_period_start_time", opt.ExportPeriodStartTime), mlog.Int("export_period_end_time", opt.ExportPeriodEndTime), mlog.Int("num_active_channel_ids", len(activeChannelIds)))
+	message := rctx.T("ent.message_export.actiance_export.calculate_channel_exports.channel_message", model.StringMap{"NumChannels": strconv.Itoa(len(activeChannelIds))})
+	opt.ReportProgressMessage(message)
+
+	// For each channel, get its metadata.
+	channelMetadata := make(map[string]*MetadataChannel, len(activeChannelIds))
+
+	// Use batches to reduce db load and network waste.
+	for pos := 0; pos < len(activeChannelIds); pos += opt.ChannelBatchSize {
+		upTo := min(pos+opt.ChannelBatchSize, len(activeChannelIds))
+		batch := activeChannelIds[pos:upTo]
+		channels, err := opt.Store.Channel().GetMany(batch, true)
+		if err != nil {
+			return nil, nil, err
+		}
+
+		for _, channel := range channels {
+			channelMetadata[channel.Id] = &MetadataChannel{
+				TeamId:             model.NewPointer(channel.TeamId),
+				ChannelId:          channel.Id,
+				ChannelName:        channel.Name,
+				ChannelDisplayName: channel.DisplayName,
+				ChannelType:        channel.Type,
+				RoomId:             fmt.Sprintf("%v - %v", ChannelTypeDisplayName(channel.Type), channel.Id),
+				StartTime:          opt.ExportPeriodStartTime,
+				EndTime:            opt.ExportPeriodEndTime,
+			}
+		}
+	}
+
+	historiesByChannelId := make(map[string][]*model.ChannelMemberHistoryResult, len(activeChannelIds))
+
+	var batchTimes []int64
+
+	// Now that we have metadata, get channelMemberHistories for each channel.
+	// Use batches to reduce total db load and network waste.
+	for pos := 0; pos < len(activeChannelIds); pos += opt.ChannelHistoryBatchSize {
+		// This may take a while, so update the system console UI.
+		message := rctx.T("ent.message_export.actiance_export.calculate_channel_exports.activity_message", model.StringMap{
+			"NumChannels":  strconv.Itoa(len(activeChannelIds)),
+			"NumCompleted": strconv.Itoa(pos),
+		})
+		opt.ReportProgressMessage(message)
+
+		start := time.Now()
+
+		upTo := min(pos+opt.ChannelHistoryBatchSize, len(activeChannelIds))
+		batch := activeChannelIds[pos:upTo]
+		channelMemberHistories, err := opt.Store.ChannelMemberHistory().GetUsersInChannelDuring(opt.ExportPeriodStartTime, opt.ExportPeriodEndTime, batch)
+		if err != nil {
+			return nil, nil, err
+		}
+
+		batchTimes = append(batchTimes, time.Since(start).Milliseconds())
+
+		// collect the channelMemberHistories by channelId
+		for _, entry := range channelMemberHistories {
+			historiesByChannelId[entry.ChannelId] = append(historiesByChannelId[entry.ChannelId], entry)
+		}
+	}
+
+	rctx.Logger().Info("GetUsersInChannelDuring batch times", mlog.Array("batch_times", batchTimes))
+
+	return channelMetadata, historiesByChannelId, nil
+}
+
+// ChannelHasActivity returns true if the channel (represented by the []*model.ChannelMemberHistoryResult slice)
+// had user activity between startTime and endTime
+func ChannelHasActivity(cmhs []*model.ChannelMemberHistoryResult, startTime int64, endTime int64) bool {
+	for _, cmh := range cmhs {
+		if (cmh.JoinTime >= startTime && cmh.JoinTime <= endTime) ||
+			(cmh.LeaveTime != nil && *cmh.LeaveTime >= startTime && *cmh.LeaveTime <= endTime) {
+			return true
+		}
+	}
+	return false
+}
+
+func GetJoinsAndLeavesForChannel(startTime int64, endTime int64, channelMembersHistory []*model.ChannelMemberHistoryResult,
+	postAuthors map[string]ChannelMember) ([]ChannelMemberJoin, []ChannelMemberLeave) {
+	var joins []ChannelMemberJoin
+	var leaves []ChannelMemberLeave
+
+	alreadyJoined := make(map[string]bool)
+	for _, cmh := range channelMembersHistory {
+		if cmh.UserDeleteAt > 0 && cmh.UserDeleteAt < startTime {
+			continue
+		}
+
+		if cmh.JoinTime > endTime {
+			continue
+		}
+
+		if cmh.LeaveTime != nil && *cmh.LeaveTime < startTime {
+			continue
+		}
+
+		if cmh.JoinTime <= endTime {
+			joins = append(joins, ChannelMemberJoin{
+				UserId:   cmh.UserId,
+				IsBot:    cmh.IsBot,
+				Email:    cmh.UserEmail,
+				Username: cmh.Username,
+				Datetime: cmh.JoinTime,
+			})
+			alreadyJoined[cmh.UserId] = true
+		}
+
+		if cmh.LeaveTime != nil && *cmh.LeaveTime <= endTime {
+			leaves = append(leaves, ChannelMemberLeave{
+				UserId:   cmh.UserId,
+				IsBot:    cmh.IsBot,
+				Email:    cmh.UserEmail,
+				Username: cmh.Username,
+				Datetime: *cmh.LeaveTime,
+			})
+		}
+	}
+
+	for _, member := range postAuthors {
+		if alreadyJoined[member.UserId] {
+			continue
+		}
+
+		joins = append(joins, ChannelMemberJoin{
+			UserId:   member.UserId,
+			IsBot:    member.IsBot,
+			Email:    member.Email,
+			Username: member.Username,
+			Datetime: startTime,
+		})
+	}
+	return joins, leaves
+}
+
+// GetPostAttachments if the post included any files, we need to add special elements to the export.
+func GetPostAttachments(db MessageExportStore, post *model.MessageExport) ([]*model.FileInfo, error) {
+	if len(post.PostFileIds) == 0 {
+		return []*model.FileInfo{}, nil
+	}
+
+	attachments, err := db.FileInfo().GetForPost(*post.PostId, true, true, false)
+	if err != nil {
+		return nil, fmt.Errorf("failed to get file info for a post: %w", err)
+	}
+	return attachments, nil
+}
+
+func ChannelTypeDisplayName(channelType model.ChannelType) string {
+	return map[model.ChannelType]string{
+		model.ChannelTypeOpen:    "public",
+		model.ChannelTypePrivate: "private",
+		model.ChannelTypeDirect:  "direct",
+		model.ChannelTypeGroup:   "group",
+	}[channelType]
+}
+
+func GetBatchPath(exportDir string, prevPostUpdateAt int64, lastPostUpdateAt int64, batchNumber int) string {
+	if exportDir == "" {
+		exportDir = path.Join(model.ComplianceExportPath, time.Now().Format(model.ComplianceExportDirectoryFormat))
+	}
+	return path.Join(exportDir,
+		fmt.Sprintf("batch%03d-%d-%d.zip", batchNumber, prevPostUpdateAt, lastPostUpdateAt))
+}
+
+// GetExportBackend returns the file backend where the export will be created.
+func GetExportBackend(rctx request.CTX, config *model.Config) (filestore.FileBackend, error) {
+	insecure := config.ServiceSettings.EnableInsecureOutgoingConnections
+	skipVerify := insecure != nil && *insecure
+
+	if config.FileSettings.DedicatedExportStore != nil && *config.FileSettings.DedicatedExportStore {
+		rctx.Logger().Debug("Worker: using dedicated export filestore", mlog.String("driver_name", *config.FileSettings.ExportDriverName))
+		backend, errFileBack := filestore.NewExportFileBackend(filestore.NewExportFileBackendSettingsFromConfig(&config.FileSettings, true, skipVerify))
+		if errFileBack != nil {
+			return nil, errFileBack
+		}
+
+		return backend, nil
+	}
+
+	backend, err := filestore.NewFileBackend(filestore.NewFileBackendSettingsFromConfig(&config.FileSettings, true, skipVerify))
+	if err != nil {
+		return nil, err
+	}
+	return backend, nil
+}
+
+// GetFileAttachmentBackend returns the file backend where file attachments are
+// located for messages that will be exported. This may be the same backend
+// where the export will be created.
+func GetFileAttachmentBackend(rctx request.CTX, config *model.Config) (filestore.FileBackend, error) {
+	insecure := config.ServiceSettings.EnableInsecureOutgoingConnections
+
+	backend, err := filestore.NewFileBackend(filestore.NewFileBackendSettingsFromConfig(&config.FileSettings, true, insecure != nil && *insecure))
+	if err != nil {
+		return nil, err
+	}
+	return backend, nil
+}
+
+func IsDeletedMsg(post *model.MessageExport) bool {
+	if model.SafeDereference(post.PostDeleteAt) > 0 && post.PostProps != nil {
+		props := map[string]any{}
+		err := json.Unmarshal([]byte(*post.PostProps), &props)
+		if err != nil {
+			return false
+		}
+
+		if _, ok := props[model.PostPropsDeleteBy]; ok {
+			return true
+		}
+	}
+	return false
+}

server/enterprise/message_export/shared/shared_test.go

@@ -0,0 +1,286 @@
+// Copyright (c) 2015-present Mattermost, Inc. All Rights Reserved.
+// See LICENSE.enterprise for license information.
+
+package shared
+
+import (
+	"testing"
+	"time"
+
+	"github.com/stretchr/testify/assert"
+
+	"github.com/mattermost/mattermost/server/public/model"
+)
+
+func TestGetJoinsAndLeavesForChannel(t *testing.T) {
+	channel := MetadataChannel{
+		StartTime:          100,
+		EndTime:            200,
+		ChannelId:          "good-request-1",
+		TeamId:             model.NewPointer("test"),
+		TeamName:           model.NewPointer("test"),
+		TeamDisplayName:    model.NewPointer("test"),
+		ChannelName:        "test",
+		ChannelDisplayName: "test",
+		ChannelType:        "O",
+	}
+
+	tt := []struct {
+		name           string
+		channel        MetadataChannel
+		membersHistory []*model.ChannelMemberHistoryResult
+		usersInPosts   map[string]ChannelMember
+		expectedJoins  int
+		expectedLeaves int
+	}{
+		{
+			name:           "no-joins-no-leaves",
+			channel:        channel,
+			membersHistory: nil,
+			usersInPosts:   nil,
+			expectedJoins:  0,
+			expectedLeaves: 0,
+		},
+		{
+			name:    "joins-and-leaves-outside-the-range",
+			channel: channel,
+			membersHistory: []*model.ChannelMemberHistoryResult{
+				{JoinTime: 1, LeaveTime: model.NewPointer(int64(10)), UserId: "test", UserEmail: "test", Username: "test"},
+				{JoinTime: 250, LeaveTime: model.NewPointer(int64(260)), UserId: "test", UserEmail: "test", Username: "test"},
+				{JoinTime: 300, UserId: "test", UserEmail: "test", Username: "test"},
+			},
+			usersInPosts:   nil,
+			expectedJoins:  0,
+			expectedLeaves: 0,
+		},
+		{
+			name:    "join-and-leave-during-the-range",
+			channel: channel,
+			membersHistory: []*model.ChannelMemberHistoryResult{
+				{JoinTime: 100, LeaveTime: model.NewPointer(int64(150)), UserId: "test", UserEmail: "test", Username: "test"},
+			},
+			usersInPosts:   nil,
+			expectedJoins:  1,
+			expectedLeaves: 1,
+		},
+		{
+			name:    "join-during-and-leave-after-the-range",
+			channel: channel,
+			membersHistory: []*model.ChannelMemberHistoryResult{
+				{JoinTime: 150, LeaveTime: model.NewPointer(int64(300)), UserId: "test", UserEmail: "test", Username: "test"},
+			},
+			usersInPosts:   nil,
+			expectedJoins:  1,
+			expectedLeaves: 0,
+		},
+		{
+			name:    "join-before-and-leave-during-the-range",
+			channel: channel,
+			membersHistory: []*model.ChannelMemberHistoryResult{
+				{JoinTime: 99, LeaveTime: model.NewPointer(int64(150)), UserId: "test", UserEmail: "test", Username: "test"},
+			},
+			usersInPosts:   nil,
+			expectedJoins:  1,
+			expectedLeaves: 1,
+		},
+		{
+			name:    "join-before-and-leave-after-the-range",
+			channel: channel,
+			membersHistory: []*model.ChannelMemberHistoryResult{
+				{JoinTime: 99, LeaveTime: model.NewPointer(int64(350)), UserId: "test", UserEmail: "test", Username: "test"},
+			},
+			usersInPosts:   nil,
+			expectedJoins:  1,
+			expectedLeaves: 0,
+		},
+		{
+			name:           "implicit-joins",
+			channel:        channel,
+			membersHistory: nil,
+			usersInPosts: map[string]ChannelMember{
+				"test1": {UserId: "test1", Email: "test1", Username: "test1"},
+				"test2": {UserId: "test2", Email: "test2", Username: "test2"},
+			},
+			expectedJoins:  2,
+			expectedLeaves: 0,
+		},
+		{
+			name:    "implicit-joins-with-explicit-joins",
+			channel: channel,
+			membersHistory: []*model.ChannelMemberHistoryResult{
+				{JoinTime: 130, LeaveTime: model.NewPointer(int64(150)), UserId: "test1", UserEmail: "test1", Username: "test1"},
+				{JoinTime: 130, LeaveTime: model.NewPointer(int64(150)), UserId: "test3", UserEmail: "test3", Username: "test3"},
+			},
+			usersInPosts: map[string]ChannelMember{
+				"test1": {UserId: "test1", Email: "test1", Username: "test1"},
+				"test2": {UserId: "test2", Email: "test2", Username: "test2"},
+			},
+			expectedJoins:  3,
+			expectedLeaves: 2,
+		},
+		{
+			name:    "join-leave-and-join-again",
+			channel: channel,
+			membersHistory: []*model.ChannelMemberHistoryResult{
+				{JoinTime: 130, LeaveTime: model.NewPointer(int64(150)), UserId: "test1", UserEmail: "test1", Username: "test1"},
+				{JoinTime: 160, LeaveTime: model.NewPointer(int64(180)), UserId: "test1", UserEmail: "test1", Username: "test1"},
+			},
+			usersInPosts:   nil,
+			expectedJoins:  2,
+			expectedLeaves: 2,
+		},
+		{
+			name:    "deactivated-members-dont-show",
+			channel: channel,
+			membersHistory: []*model.ChannelMemberHistoryResult{
+				{JoinTime: 130, LeaveTime: model.NewPointer(int64(150)), UserId: "test1", UserEmail: "test1", Username: "test1", UserDeleteAt: 50},
+				{JoinTime: 160, LeaveTime: model.NewPointer(int64(180)), UserId: "test1", UserEmail: "test1", Username: "test1", UserDeleteAt: 50},
+			},
+			usersInPosts:   nil,
+			expectedJoins:  0,
+			expectedLeaves: 0,
+		},
+		{
+			name:    "deactivated-members-show-if-deleted-after-latest-export",
+			channel: channel,
+			membersHistory: []*model.ChannelMemberHistoryResult{
+				{JoinTime: 130, LeaveTime: model.NewPointer(int64(150)), UserId: "test1", UserEmail: "test1", Username: "test1", UserDeleteAt: 150},
+				{JoinTime: 160, LeaveTime: model.NewPointer(int64(180)), UserId: "test1", UserEmail: "test1", Username: "test1", UserDeleteAt: 150},
+			},
+			usersInPosts:   nil,
+			expectedJoins:  2,
+			expectedLeaves: 2,
+		},
+		{
+			name:    "deactivated-members-show-and-dont-show",
+			channel: channel,
+			membersHistory: []*model.ChannelMemberHistoryResult{
+				{JoinTime: 130, LeaveTime: model.NewPointer(int64(150)), UserId: "test1", UserEmail: "test1", Username: "test1", UserDeleteAt: 50},
+				{JoinTime: 160, LeaveTime: model.NewPointer(int64(180)), UserId: "test1", UserEmail: "test1", Username: "test1", UserDeleteAt: 150},
+			},
+			usersInPosts:   nil,
+			expectedJoins:  1,
+			expectedLeaves: 1,
+		},
+	}
+
+	for _, tc := range tt {
+		t.Run(tc.name, func(t *testing.T) {
+			joins, leaves := GetJoinsAndLeavesForChannel(tc.channel.StartTime, tc.channel.EndTime, tc.membersHistory, tc.usersInPosts)
+			assert.Len(t, joins, tc.expectedJoins)
+			assert.Len(t, leaves, tc.expectedLeaves)
+		})
+	}
+}
+
+func Test_GetBatchPath(t *testing.T) {
+	tests := []struct {
+		name             string
+		exportDir        string
+		prevPostUpdateAt int64
+		lastPostUpdateAt int64
+		batchNumber      int
+		want             string
+	}{
+		{
+			name:             "all args given",
+			exportDir:        "/export/test_dir",
+			prevPostUpdateAt: 123,
+			lastPostUpdateAt: 456,
+			batchNumber:      21,
+			want:             "/export/test_dir/batch021-123-456.zip",
+		},
+		{
+			name:             "exportDir blank",
+			exportDir:        "",
+			prevPostUpdateAt: 12345,
+			lastPostUpdateAt: 456789,
+			batchNumber:      921,
+			want:             model.ComplianceExportPath + "/" + time.Now().Format(model.ComplianceExportDirectoryFormat) + "/batch921-12345-456789.zip",
+		},
+	}
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			assert.Equalf(t, tt.want, GetBatchPath(tt.exportDir, tt.prevPostUpdateAt, tt.lastPostUpdateAt, tt.batchNumber), "GetBatchPath(%v, %v, %v, %v)", tt.exportDir, tt.prevPostUpdateAt, tt.lastPostUpdateAt, tt.batchNumber)
+		})
+	}
+}
+
+func TestJobDataToStringMap_and_StringMapToJobData(t *testing.T) {
+	jd := JobData{
+		JobDataExported: JobDataExported{
+			ExportType:              "cli_message_export",
+			ExportDir:               "/here/there/34234-123",
+			BatchStartTime:          45,
+			BatchStartId:            "34arsitenaorsten",
+			JobStartTime:            99,
+			JobEndTime:              1234,
+			JobStartId:              "99abcdef34",
+			BatchSize:               2000,
+			ChannelBatchSize:        30000,
+			ChannelHistoryBatchSize: 30,
+			BatchNumber:             4,
+			TotalPostsExpected:      999999,
+			MessagesExported:        343499,
+			WarningCount:            39,
+		},
+		ExportPeriodStartTime: 123456,                  // not exported
+		BatchEndTime:          999999999,               // not exported
+		BatchPath:             "/another/path/123-123", // not exported
+	}
+
+	strMap := JobDataToStringMap(jd)
+
+	expected := make(map[string]string)
+	expected[JobDataExportType] = "cli_message_export"
+	expected[JobDataExportDir] = "/here/there/34234-123"
+	expected[JobDataBatchStartTime] = "45"
+	expected[JobDataBatchStartId] = "34arsitenaorsten"
+	expected[JobDataJobStartTime] = "99"
+	expected[JobDataJobEndTime] = "1234"
+	expected[JobDataJobStartId] = "99abcdef34"
+	expected[JobDataBatchSize] = "2000"
+	expected[JobDataChannelBatchSize] = "30000"
+	expected[JobDataChannelHistoryBatchSize] = "30"
+	expected[JobDataBatchNumber] = "4"
+	expected[JobDataTotalPostsExpected] = "999999"
+	expected[JobDataMessagesExported] = "343499"
+	expected[JobDataWarningCount] = "39"
+	expected[JobDataIsDownloadable] = "false"
+
+	for k, v := range expected {
+		val, ok := strMap[k]
+		assert.True(t, ok)
+		assert.Equal(t, v, val)
+	}
+
+	// not exported:
+	for _, k := range []string{"export_period_start_time", "batch_end_time", "batch_path"} {
+		_, ok := strMap[k]
+		assert.False(t, ok)
+	}
+
+	// zero the fields that weren't exported:
+	jd.ExportPeriodStartTime = 0
+	jd.BatchEndTime = 0
+	jd.BatchPath = ""
+
+	// now convert back
+	jd2, err := StringMapToJobDataWithZeroValues(strMap)
+	assert.NoError(t, err)
+	assert.Equal(t, jd, jd2)
+
+	// and test bad conversion (just a couple):
+	badStrMap := map[string]string{JobDataJobStartTime: "56aaa"}
+	_, err = StringMapToJobDataWithZeroValues(badStrMap)
+	assert.Error(t, err)
+	badStrMap = map[string]string{JobDataJobEndTime: "blah blah"}
+	_, err = StringMapToJobDataWithZeroValues(badStrMap)
+	assert.Error(t, err)
+
+	// test that zero values are used when not present
+	emptyStrMap := make(map[string]string)
+	emptyJd, err := StringMapToJobDataWithZeroValues(emptyStrMap)
+	assert.NoError(t, err)
+	assert.Equal(t, JobData{}, emptyJd)
+}

server/enterprise/message_export/shared/store.go

@@ -0,0 +1,36 @@
+// Copyright (c) 2015-present Mattermost, Inc. All Rights Reserved.
+// See LICENSE.enterprise for license information.
+
+package shared
+
+import (
+	"github.com/mattermost/mattermost/server/public/model"
+	"github.com/mattermost/mattermost/server/v8/channels/store"
+)
+
+type MessageExportStore interface {
+	Post() store.PostStore
+	ChannelMemberHistory() store.ChannelMemberHistoryStore
+	Channel() store.ChannelStore
+	Compliance() store.ComplianceStore
+	FileInfo() MEFileInfoStore
+}
+
+type MEFileInfoStore interface {
+	GetForPost(postID string, readFromMaster, includeDeleted, allowFromCache bool) ([]*model.FileInfo, error)
+}
+
+type messageExportStore struct {
+	store.Store
+}
+
+// NewMessageExportStore returns a wrapped store.Store. Why? Because the CLI tool needs to "override" the GetForPost
+// method in FileStore. Non-CLI code only needs to use the store.Store's methods. This can be removed when we deprecate
+// the CLI tool. Tracked at MM-61139.
+func NewMessageExportStore(s store.Store) messageExportStore {
+	return messageExportStore{s}
+}
+
+func (ss messageExportStore) FileInfo() MEFileInfoStore {
+	return ss.Store.FileInfo()
+}

server/enterprise/message_export/testdata/actianceE2E1Batch1ch2.tmpl

@@ -0,0 +1,101 @@
+  <Conversation Perspective="the Channel Two">
+    <RoomID>private - channel_two_name - %s</RoomID>
+    <StartTimeUTC>%d</StartTimeUTC>
+    <ParticipantEntered>
+      <LoginName>user1@email</LoginName>
+      <UserType>user</UserType>
+      <DateTimeUTC>%d</DateTimeUTC>
+      <CorporateEmailID>user1@email</CorporateEmailID>
+    </ParticipantEntered>
+    <ParticipantEntered>
+      <LoginName>user2@email</LoginName>
+      <UserType>user</UserType>
+      <DateTimeUTC>%d</DateTimeUTC>
+      <CorporateEmailID>user2@email</CorporateEmailID>
+    </ParticipantEntered>
+    <ParticipantEntered>
+      <LoginName>user3@email</LoginName>
+      <UserType>user</UserType>
+      <DateTimeUTC>%d</DateTimeUTC>
+      <CorporateEmailID>user3@email</CorporateEmailID>
+    </ParticipantEntered>
+    <Message>
+      <MessageId>%s</MessageId>
+      <LoginName>user1@email</LoginName>
+      <UserType>user</UserType>
+      <DateTimeUTC>%d</DateTimeUTC>
+      <Content>message 0</Content>
+    </Message>
+    <FileTransferStarted>
+      <LoginName>user1@email</LoginName>
+      <DateTimeUTC>%d</DateTimeUTC>
+      <UserFileName></UserFileName>
+      <FileName>path/to/attachments/file_0.txt</FileName>
+    </FileTransferStarted>
+    <FileTransferEnded>
+      <LoginName>user1@email</LoginName>
+      <DateTimeUTC>%d</DateTimeUTC>
+      <UserFileName></UserFileName>
+      <FileName>path/to/attachments/file_0.txt</FileName>
+      <Status>Completed</Status>
+    </FileTransferEnded>
+    <Message>
+      <MessageId>%s</MessageId>
+      <LoginName>user1@email</LoginName>
+      <UserType>user</UserType>
+      <DateTimeUTC>%d</DateTimeUTC>
+      <Content>message 1</Content>
+    </Message>
+    <FileTransferStarted>
+      <LoginName>user1@email</LoginName>
+      <DateTimeUTC>%d</DateTimeUTC>
+      <UserFileName></UserFileName>
+      <FileName>path/to/attachments/file_1.txt</FileName>
+    </FileTransferStarted>
+    <FileTransferEnded>
+      <LoginName>user1@email</LoginName>
+      <DateTimeUTC>%d</DateTimeUTC>
+      <UserFileName></UserFileName>
+      <FileName>path/to/attachments/file_1.txt</FileName>
+      <Status>Completed</Status>
+    </FileTransferEnded>
+    <Message>
+      <MessageId>%s</MessageId>
+      <LoginName>user1@email</LoginName>
+      <UserType>user</UserType>
+      <DateTimeUTC>%d</DateTimeUTC>
+      <Content>message 2</Content>
+    </Message>
+    <FileTransferStarted>
+      <LoginName>user1@email</LoginName>
+      <DateTimeUTC>%d</DateTimeUTC>
+      <UserFileName></UserFileName>
+      <FileName>path/to/attachments/file_2.txt</FileName>
+    </FileTransferStarted>
+    <FileTransferEnded>
+      <LoginName>user1@email</LoginName>
+      <DateTimeUTC>%d</DateTimeUTC>
+      <UserFileName></UserFileName>
+      <FileName>path/to/attachments/file_2.txt</FileName>
+      <Status>Completed</Status>
+    </FileTransferEnded>
+    <ParticipantLeft>
+      <LoginName>user2@email</LoginName>
+      <UserType>user</UserType>
+      <DateTimeUTC>%d</DateTimeUTC>
+      <CorporateEmailID>user2@email</CorporateEmailID>
+    </ParticipantLeft>
+    <ParticipantLeft>
+      <LoginName>user3@email</LoginName>
+      <UserType>user</UserType>
+      <DateTimeUTC>%d</DateTimeUTC>
+      <CorporateEmailID>user3@email</CorporateEmailID>
+    </ParticipantLeft>
+    <ParticipantLeft>
+      <LoginName>user1@email</LoginName>
+      <UserType>user</UserType>
+      <DateTimeUTC>%d</DateTimeUTC>
+      <CorporateEmailID>user1@email</CorporateEmailID>
+    </ParticipantLeft>
+    <EndTimeUTC>%d</EndTimeUTC>
+  </Conversation>

server/enterprise/message_export/testdata/actianceE2E1Batch1ch3.tmpl

@@ -0,0 +1,17 @@
+  <Conversation Perspective="the Channel Three">
+    <RoomID>public - channel_three_name - %s</RoomID>
+    <StartTimeUTC>%d</StartTimeUTC>
+    <ParticipantEntered>
+      <LoginName>user7@email</LoginName>
+      <UserType>user</UserType>
+      <DateTimeUTC>%d</DateTimeUTC>
+      <CorporateEmailID>user7@email</CorporateEmailID>
+    </ParticipantEntered>
+    <ParticipantLeft>
+      <LoginName>user7@email</LoginName>
+      <UserType>user</UserType>
+      <DateTimeUTC>%d</DateTimeUTC>
+      <CorporateEmailID>user7@email</CorporateEmailID>
+    </ParticipantLeft>
+    <EndTimeUTC>%d</EndTimeUTC>
+  </Conversation>

server/enterprise/message_export/testdata/actianceE2E1Batch2.tmpl

@@ -0,0 +1,92 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<FileDump xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
+  <Conversation Perspective="the Channel Two">
+    <RoomID>private - channel_two_name - %s</RoomID>
+    <StartTimeUTC>%d</StartTimeUTC>
+    <ParticipantEntered>
+      <LoginName>user1@email</LoginName>
+      <UserType>user</UserType>
+      <DateTimeUTC>%d</DateTimeUTC>
+      <CorporateEmailID>user1@email</CorporateEmailID>
+    </ParticipantEntered>
+    <ParticipantEntered>
+      <LoginName>user4@email</LoginName>
+      <UserType>user</UserType>
+      <DateTimeUTC>%d</DateTimeUTC>
+      <CorporateEmailID>user4@email</CorporateEmailID>
+    </ParticipantEntered>
+    <Message>
+      <MessageId>%s</MessageId>
+      <LoginName>user1@email</LoginName>
+      <UserType>user</UserType>
+      <DateTimeUTC>%d</DateTimeUTC>
+      <Content>message 3</Content>
+    </Message>
+    <FileTransferStarted>
+      <LoginName>user1@email</LoginName>
+      <DateTimeUTC>%d</DateTimeUTC>
+      <UserFileName></UserFileName>
+      <FileName>path/to/attachments/file_3.txt</FileName>
+    </FileTransferStarted>
+    <FileTransferEnded>
+      <LoginName>user1@email</LoginName>
+      <DateTimeUTC>%d</DateTimeUTC>
+      <UserFileName></UserFileName>
+      <FileName>path/to/attachments/file_3.txt</FileName>
+      <Status>Completed</Status>
+    </FileTransferEnded>
+    <Message>
+      <MessageId>%s</MessageId>
+      <LoginName>user1@email</LoginName>
+      <UserType>user</UserType>
+      <DateTimeUTC>%d</DateTimeUTC>
+      <Content>message 4</Content>
+    </Message>
+    <FileTransferStarted>
+      <LoginName>user1@email</LoginName>
+      <DateTimeUTC>%d</DateTimeUTC>
+      <UserFileName></UserFileName>
+      <FileName>path/to/attachments/file_4.txt</FileName>
+    </FileTransferStarted>
+    <FileTransferEnded>
+      <LoginName>user1@email</LoginName>
+      <DateTimeUTC>%d</DateTimeUTC>
+      <UserFileName></UserFileName>
+      <FileName>path/to/attachments/file_4.txt</FileName>
+      <Status>Completed</Status>
+    </FileTransferEnded>
+    <Message>
+      <MessageId>%s</MessageId>
+      <LoginName>user1@email</LoginName>
+      <UserType>user</UserType>
+      <DateTimeUTC>%d</DateTimeUTC>
+      <Content>message 5</Content>
+    </Message>
+    <FileTransferStarted>
+      <LoginName>user1@email</LoginName>
+      <DateTimeUTC>%d</DateTimeUTC>
+      <UserFileName></UserFileName>
+      <FileName>path/to/attachments/file_5.txt</FileName>
+    </FileTransferStarted>
+    <FileTransferEnded>
+      <LoginName>user1@email</LoginName>
+      <DateTimeUTC>%d</DateTimeUTC>
+      <UserFileName></UserFileName>
+      <FileName>path/to/attachments/file_5.txt</FileName>
+      <Status>Completed</Status>
+    </FileTransferEnded>
+    <ParticipantLeft>
+      <LoginName>user4@email</LoginName>
+      <UserType>user</UserType>
+      <DateTimeUTC>%d</DateTimeUTC>
+      <CorporateEmailID>user4@email</CorporateEmailID>
+    </ParticipantLeft>
+    <ParticipantLeft>
+      <LoginName>user1@email</LoginName>
+      <UserType>user</UserType>
+      <DateTimeUTC>%d</DateTimeUTC>
+      <CorporateEmailID>user1@email</CorporateEmailID>
+    </ParticipantLeft>
+    <EndTimeUTC>%d</EndTimeUTC>
+  </Conversation>
+</FileDump>

server/enterprise/message_export/testdata/actianceE2E1Batch3ch2.tmpl

@@ -0,0 +1,101 @@
+  <Conversation Perspective="the Channel Two">
+    <RoomID>private - channel_two_name - %s</RoomID>
+    <StartTimeUTC>%d</StartTimeUTC>
+    <ParticipantEntered>
+      <LoginName>user1@email</LoginName>
+      <UserType>user</UserType>
+      <DateTimeUTC>%d</DateTimeUTC>
+      <CorporateEmailID>user1@email</CorporateEmailID>
+    </ParticipantEntered>
+    <ParticipantEntered>
+      <LoginName>user5@email</LoginName>
+      <UserType>user</UserType>
+      <DateTimeUTC>%d</DateTimeUTC>
+      <CorporateEmailID>user5@email</CorporateEmailID>
+    </ParticipantEntered>
+    <ParticipantEntered>
+      <LoginName>user6@email</LoginName>
+      <UserType>user</UserType>
+      <DateTimeUTC>%d</DateTimeUTC>
+      <CorporateEmailID>user6@email</CorporateEmailID>
+    </ParticipantEntered>
+    <Message>
+      <MessageId>%s</MessageId>
+      <LoginName>user1@email</LoginName>
+      <UserType>user</UserType>
+      <DateTimeUTC>%d</DateTimeUTC>
+      <Content>message 6</Content>
+    </Message>
+    <FileTransferStarted>
+      <LoginName>user1@email</LoginName>
+      <DateTimeUTC>%d</DateTimeUTC>
+      <UserFileName></UserFileName>
+      <FileName>path/to/attachments/file_6.txt</FileName>
+    </FileTransferStarted>
+    <FileTransferEnded>
+      <LoginName>user1@email</LoginName>
+      <DateTimeUTC>%d</DateTimeUTC>
+      <UserFileName></UserFileName>
+      <FileName>path/to/attachments/file_6.txt</FileName>
+      <Status>Completed</Status>
+    </FileTransferEnded>
+    <Message>
+      <MessageId>%s</MessageId>
+      <LoginName>user1@email</LoginName>
+      <UserType>user</UserType>
+      <DateTimeUTC>%d</DateTimeUTC>
+      <Content>message 7</Content>
+    </Message>
+    <FileTransferStarted>
+      <LoginName>user1@email</LoginName>
+      <DateTimeUTC>%d</DateTimeUTC>
+      <UserFileName></UserFileName>
+      <FileName>path/to/attachments/file_7.txt</FileName>
+    </FileTransferStarted>
+    <FileTransferEnded>
+      <LoginName>user1@email</LoginName>
+      <DateTimeUTC>%d</DateTimeUTC>
+      <UserFileName></UserFileName>
+      <FileName>path/to/attachments/file_7.txt</FileName>
+      <Status>Completed</Status>
+    </FileTransferEnded>
+    <Message>
+      <MessageId>%s</MessageId>
+      <LoginName>user1@email</LoginName>
+      <UserType>user</UserType>
+      <DateTimeUTC>%d</DateTimeUTC>
+      <Content>message 8</Content>
+    </Message>
+    <FileTransferStarted>
+      <LoginName>user1@email</LoginName>
+      <DateTimeUTC>%d</DateTimeUTC>
+      <UserFileName></UserFileName>
+      <FileName>path/to/attachments/file_8.txt</FileName>
+    </FileTransferStarted>
+    <FileTransferEnded>
+      <LoginName>user1@email</LoginName>
+      <DateTimeUTC>%d</DateTimeUTC>
+      <UserFileName></UserFileName>
+      <FileName>path/to/attachments/file_8.txt</FileName>
+      <Status>Completed</Status>
+    </FileTransferEnded>
+    <ParticipantLeft>
+      <LoginName>user5@email</LoginName>
+      <UserType>user</UserType>
+      <DateTimeUTC>%d</DateTimeUTC>
+      <CorporateEmailID>user5@email</CorporateEmailID>
+    </ParticipantLeft>
+    <ParticipantLeft>
+      <LoginName>user6@email</LoginName>
+      <UserType>user</UserType>
+      <DateTimeUTC>%d</DateTimeUTC>
+      <CorporateEmailID>user6@email</CorporateEmailID>
+    </ParticipantLeft>
+    <ParticipantLeft>
+      <LoginName>user1@email</LoginName>
+      <UserType>user</UserType>
+      <DateTimeUTC>%d</DateTimeUTC>
+      <CorporateEmailID>user1@email</CorporateEmailID>
+    </ParticipantLeft>
+    <EndTimeUTC>%d</EndTimeUTC>
+  </Conversation>

server/enterprise/message_export/testdata/actianceE2E1Batch3ch4.tmpl

@@ -0,0 +1,17 @@
+  <Conversation Perspective="the Channel Four">
+    <RoomID>public - channel_four_name - %s</RoomID>
+    <StartTimeUTC>%d</StartTimeUTC>
+    <ParticipantEntered>
+      <LoginName>user8@email</LoginName>
+      <UserType>user</UserType>
+      <DateTimeUTC>%d</DateTimeUTC>
+      <CorporateEmailID>user8@email</CorporateEmailID>
+    </ParticipantEntered>
+    <ParticipantLeft>
+      <LoginName>user8@email</LoginName>
+      <UserType>user</UserType>
+      <DateTimeUTC>%d</DateTimeUTC>
+      <CorporateEmailID>user8@email</CorporateEmailID>
+    </ParticipantLeft>
+    <EndTimeUTC>%d</EndTimeUTC>
+  </Conversation>

server/enterprise/message_export/testdata/actianceE2E2.tmpl

@@ -0,0 +1,46 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<FileDump xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
+  <Conversation Perspective="the Channel Two">
+    <RoomID>private - channel_two_name - %s</RoomID>
+    <StartTimeUTC>%d</StartTimeUTC>
+    <ParticipantEntered>
+      <LoginName>user1@email</LoginName>
+      <UserType>user</UserType>
+      <DateTimeUTC>%d</DateTimeUTC>
+      <CorporateEmailID>user1@email</CorporateEmailID>
+    </ParticipantEntered>
+    <ParticipantEntered>
+      <LoginName>user2@email</LoginName>
+      <UserType>user</UserType>
+      <DateTimeUTC>%d</DateTimeUTC>
+      <CorporateEmailID>user2@email</CorporateEmailID>
+    </ParticipantEntered>
+    <Message>
+      <MessageId>%s</MessageId>
+      <LoginName>user1@email</LoginName>
+      <UserType>user</UserType>
+      <DateTimeUTC>%d</DateTimeUTC>
+      <Content>message 1</Content>
+    </Message>
+    <Message>
+      <MessageId>%s</MessageId>
+      <LoginName>user2@email</LoginName>
+      <UserType>user</UserType>
+      <DateTimeUTC>%d</DateTimeUTC>
+      <Content>message 2</Content>
+    </Message>
+    <ParticipantLeft>
+      <LoginName>user1@email</LoginName>
+      <UserType>user</UserType>
+      <DateTimeUTC>%d</DateTimeUTC>
+      <CorporateEmailID>user1@email</CorporateEmailID>
+    </ParticipantLeft>
+    <ParticipantLeft>
+      <LoginName>user2@email</LoginName>
+      <UserType>user</UserType>
+      <DateTimeUTC>%d</DateTimeUTC>
+      <CorporateEmailID>user2@email</CorporateEmailID>
+    </ParticipantLeft>
+    <EndTimeUTC>%d</EndTimeUTC>
+  </Conversation>
+</FileDump>

server/enterprise/message_export/testdata/actianceXMLHeader.tmpl

@@ -0,0 +1,3 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<FileDump xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
+%s%s</FileDump>

server/enterprise/message_export/testdata/csvE2E1Batch1.tmpl

@@ -0,0 +1,14 @@
+Post Creation Time,Post Update Time,Post Update Type,Team Id,Team Name,Team Display Name,Channel Id,Channel Name,Channel Display Name,Channel Type,User Id,User Email,Username,Post Id,Edited By Post Id,Replied to Post Id,Post Message,Post Type,User Type,Previews Post Id
+%[16]d,0,,%[1]s,%[2]s,%[3]s,%[4]s,channel_two_name,the Channel Two,private,%[6]s,user1@email,user1,,,,User user1 (user1@email) was already in the channel,previously-joined,user,
+%[17]d,0,,%[1]s,%[2]s,%[3]s,%[4]s,channel_two_name,the Channel Two,private,%[7]s,user2@email,user2,,,,User user2 (user2@email) joined the channel,enter,user,
+%[18]d,0,,%[1]s,%[2]s,%[3]s,%[4]s,channel_two_name,the Channel Two,private,%[7]s,user2@email,user2,,,,User user2 (user2@email) left the channel,leave,user,
+%[19]d,%[19]d,,%[1]s,%[2]s,%[3]s,%[4]s,channel_two_name,the Channel Two,private,%[6]s,user1@email,user1,%[10]s,,,message 0,message,user,
+%[19]d,%[19]d,,%[1]s,%[2]s,%[3]s,%[4]s,channel_two_name,the Channel Two,private,%[6]s,user1@email,user1,%[10]s,,,(files/%[10]s/%[13]s-file_0.txt),attachment,user,
+%[20]d,0,,%[1]s,%[2]s,%[3]s,%[4]s,channel_two_name,the Channel Two,private,%[8]s,user3@email,user3,,,,User user3 (user3@email) joined the channel,enter,user,
+%[21]d,%[21]d,,%[1]s,%[2]s,%[3]s,%[4]s,channel_two_name,the Channel Two,private,%[6]s,user1@email,user1,%[11]s,,,message 1,message,user,
+%[21]d,%[21]d,,%[1]s,%[2]s,%[3]s,%[4]s,channel_two_name,the Channel Two,private,%[6]s,user1@email,user1,%[11]s,,,(files/%[11]s/%[14]s-file_1.txt),attachment,user,
+%[22]d,0,,%[1]s,%[2]s,%[3]s,%[4]s,channel_two_name,the Channel Two,private,%[8]s,user3@email,user3,,,,User user3 (user3@email) left the channel,leave,user,
+%[23]d,0,,%[1]s,,,%[5]s,channel_three_name,the Channel Three,public,%[9]s,user7@email,user7,,,,User user7 (user7@email) was already in the channel,previously-joined,user,
+%[24]d,0,,%[1]s,,,%[5]s,channel_three_name,the Channel Three,public,%[9]s,user7@email,user7,,,,User user7 (user7@email) left the channel,leave,user,
+%[25]d,%[25]d,,%[1]s,%[2]s,%[3]s,%[4]s,channel_two_name,the Channel Two,private,%[6]s,user1@email,user1,%[12]s,,,message 2,message,user,
+%[25]d,%[25]d,,%[1]s,%[2]s,%[3]s,%[4]s,channel_two_name,the Channel Two,private,%[6]s,user1@email,user1,%[12]s,,,(files/%[12]s/%[15]s-file_2.txt),attachment,user,

server/enterprise/message_export/testdata/csvE2E1Batch2.tmpl

@@ -0,0 +1,10 @@
+Post Creation Time,Post Update Time,Post Update Type,Team Id,Team Name,Team Display Name,Channel Id,Channel Name,Channel Display Name,Channel Type,User Id,User Email,Username,Post Id,Edited By Post Id,Replied to Post Id,Post Message,Post Type,User Type,Previews Post Id
+%[13]d,0,,%[1]s,%[2]s,%[3]s,%[4]s,channel_two_name,the Channel Two,private,%[5]s,user1@email,user1,,,,User user1 (user1@email) was already in the channel,previously-joined,user,
+%[14]d,0,,%[1]s,%[2]s,%[3]s,%[4]s,channel_two_name,the Channel Two,private,%[6]s,user4@email,user4,,,,User user4 (user4@email) joined the channel,enter,user,
+%[15]d,0,,%[1]s,%[2]s,%[3]s,%[4]s,channel_two_name,the Channel Two,private,%[6]s,user4@email,user4,,,,User user4 (user4@email) left the channel,leave,user,
+%[16]d,%[16]d,,%[1]s,%[2]s,%[3]s,%[4]s,channel_two_name,the Channel Two,private,%[5]s,user1@email,user1,%[7]s,,,message 3,message,user,
+%[16]d,%[16]d,,%[1]s,%[2]s,%[3]s,%[4]s,channel_two_name,the Channel Two,private,%[5]s,user1@email,user1,%[7]s,,,(files/%[7]s/%[10]s-file_3.txt),attachment,user,
+%[17]d,%[17]d,,%[1]s,%[2]s,%[3]s,%[4]s,channel_two_name,the Channel Two,private,%[5]s,user1@email,user1,%[8]s,,,message 4,message,user,
+%[17]d,%[17]d,,%[1]s,%[2]s,%[3]s,%[4]s,channel_two_name,the Channel Two,private,%[5]s,user1@email,user1,%[8]s,,,(files/%[8]s/%[11]s-file_4.txt),attachment,user,
+%[18]d,%[18]d,,%[1]s,%[2]s,%[3]s,%[4]s,channel_two_name,the Channel Two,private,%[5]s,user1@email,user1,%[9]s,,,message 5,message,user,
+%[18]d,%[18]d,,%[1]s,%[2]s,%[3]s,%[4]s,channel_two_name,the Channel Two,private,%[5]s,user1@email,user1,%[9]s,,,(files/%[9]s/%[12]s-file_5.txt),attachment,user,

server/enterprise/message_export/testdata/csvE2E1Batch3.tmpl

@@ -0,0 +1,13 @@
+Post Creation Time,Post Update Time,Post Update Type,Team Id,Team Name,Team Display Name,Channel Id,Channel Name,Channel Display Name,Channel Type,User Id,User Email,Username,Post Id,Edited By Post Id,Replied to Post Id,Post Message,Post Type,User Type,Previews Post Id
+%[16]d,0,,%[1]s,%[2]s,%[3]s,%[4]s,channel_two_name,the Channel Two,private,%[6]s,user1@email,user1,,,,User user1 (user1@email) was already in the channel,previously-joined,user,
+%[17]d,0,,%[1]s,%[2]s,%[3]s,%[4]s,channel_two_name,the Channel Two,private,%[7]s,user5@email,user5,,,,User user5 (user5@email) joined the channel,enter,user,
+%[18]d,0,,%[1]s,%[2]s,%[3]s,%[4]s,channel_two_name,the Channel Two,private,%[7]s,user5@email,user5,,,,User user5 (user5@email) left the channel,leave,user,
+%[19]d,%[19]d,,%[1]s,%[2]s,%[3]s,%[4]s,channel_two_name,the Channel Two,private,%[6]s,user1@email,user1,%[10]s,,,message 6,message,user,
+%[19]d,%[19]d,,%[1]s,%[2]s,%[3]s,%[4]s,channel_two_name,the Channel Two,private,%[6]s,user1@email,user1,%[10]s,,,(files/%[10]s/%[13]s-file_6.txt),attachment,user,
+%[20]d,%[20]d,,%[1]s,%[2]s,%[3]s,%[4]s,channel_two_name,the Channel Two,private,%[6]s,user1@email,user1,%[11]s,,,message 7,message,user,
+%[20]d,%[20]d,,%[1]s,%[2]s,%[3]s,%[4]s,channel_two_name,the Channel Two,private,%[6]s,user1@email,user1,%[11]s,,,(files/%[11]s/%[14]s-file_7.txt),attachment,user,
+%[21]d,%[21]d,,%[1]s,%[2]s,%[3]s,%[4]s,channel_two_name,the Channel Two,private,%[6]s,user1@email,user1,%[12]s,,,message 8,message,user,
+%[21]d,%[21]d,,%[1]s,%[2]s,%[3]s,%[4]s,channel_two_name,the Channel Two,private,%[6]s,user1@email,user1,%[12]s,,,(files/%[12]s/%[15]s-file_8.txt),attachment,user,
+%[22]d,0,,%[1]s,%[2]s,%[3]s,%[4]s,channel_two_name,the Channel Two,private,%[8]s,user6@email,user6,,,,User user6 (user6@email) joined the channel,enter,user,
+%[23]d,0,,%[1]s,%[2]s,%[3]s,%[4]s,channel_two_name,the Channel Two,private,%[8]s,user6@email,user6,,,,User user6 (user6@email) left the channel,leave,user,
+%[24]d,0,,%[1]s,,,%[5]s,channel_four_name,the Channel Four,public,%[9]s,user8@email,user8,,,,User user8 (user8@email) joined the channel,enter,user,

server/enterprise/message_export/testdata/csvE2E2Batch1.tmpl

@@ -0,0 +1,5 @@
+Post Creation Time,Post Update Time,Post Update Type,Team Id,Team Name,Team Display Name,Channel Id,Channel Name,Channel Display Name,Channel Type,User Id,User Email,Username,Post Id,Edited By Post Id,Replied to Post Id,Post Message,Post Type,User Type,Previews Post Id
+%[9]d,0,,%[1]s,%[2]s,%[3]s,%[4]s,channel_two_name,the Channel Two,private,%[5]s,user1@email,user1,,,,User user1 (user1@email) was already in the channel,previously-joined,user,
+%[10]d,0,,%[1]s,%[2]s,%[3]s,%[4]s,channel_two_name,the Channel Two,private,%[6]s,user2@email,user2,,,,User user2 (user2@email) was already in the channel,previously-joined,user,
+%[11]d,%[11]d,,%[1]s,%[2]s,%[3]s,%[4]s,channel_two_name,the Channel Two,private,%[5]s,user1@email,user1,%[7]s,,,message 1,message,user,
+%[12]d,%[12]d,,%[1]s,%[2]s,%[3]s,%[4]s,channel_two_name,the Channel Two,private,%[6]s,user2@email,user2,%[8]s,,,message 2,message,user,

server/enterprise/message_export/testdata/csvE2E3Batch1Perm1.tmpl

@@ -0,0 +1,13 @@
+Post Creation Time,Post Update Time,Post Update Type,Team Id,Team Name,Team Display Name,Channel Id,Channel Name,Channel Display Name,Channel Type,User Id,User Email,Username,Post Id,Edited By Post Id,Replied to Post Id,Post Message,Post Type,User Type,Previews Post Id
+%[14]d,0,,%[1]s,%[2]s,%[3]s,%[4]s,channel_two_name,the Channel Two,private,%[5]s,user1@email,user1,,,,User user1 (user1@email) was already in the channel,previously-joined,user,
+%[15]d,%[15]d,,%[1]s,%[2]s,%[3]s,%[4]s,channel_two_name,the Channel Two,private,%[5]s,user1@email,user1,%[6]s,,,message 0,message,user,
+%[16]d,%[17]d,,%[1]s,%[2]s,%[3]s,%[4]s,channel_two_name,the Channel Two,private,%[5]s,user1@email,user1,%[7]s,,,message 1,message,user,
+%[16]d,%[17]d,Deleted,%[1]s,%[2]s,%[3]s,%[4]s,channel_two_name,the Channel Two,private,%[5]s,user1@email,user1,%[7]s,,,delete message 1,message,user,
+%[18]d,%[25]d,UpdatedNoMsgChange,%[1]s,%[2]s,%[3]s,%[4]s,channel_two_name,the Channel Two,private,%[5]s,user1@email,user1,%[8]s,,,message 2,message,user,
+%[19]d,%[20]d,,%[1]s,%[2]s,%[3]s,%[4]s,channel_two_name,the Channel Two,private,%[5]s,user1@email,user1,%[9]s,,,message 3,message,user,
+%[19]d,%[20]d,Deleted,%[1]s,%[2]s,%[3]s,%[4]s,channel_two_name,the Channel Two,private,%[5]s,user1@email,user1,%[9]s,,,delete message 3,message,user,
+%[19]d,%[20]d,,%[1]s,%[2]s,%[3]s,%[4]s,channel_two_name,the Channel Two,private,%[5]s,user1@email,user1,%[9]s,,,(files/%[9]s/%[13]s-file_3.txt),attachment,user,
+%[19]d,%[20]d,FileDeleted,%[1]s,%[2]s,%[3]s,%[4]s,channel_two_name,the Channel Two,private,%[5]s,user1@email,user1,%[9]s,,,(files/%[9]s/%[13]s-file_3.txt),deleted attachment,user,
+%[21]d,%[22]d,EditedOriginalMsg,%[1]s,%[2]s,%[3]s,%[4]s,channel_two_name,the Channel Two,private,%[5]s,user1@email,user1,%[10]s,%[11]s,,message 4,message,user,
+%[21]d,%[22]d,EditedNewMsg,%[1]s,%[2]s,%[3]s,%[4]s,channel_two_name,the Channel Two,private,%[5]s,user1@email,user1,%[11]s,,,edited message 4,message,user,
+%[23]d,%[26]d,EditedOriginalMsg,%[1]s,%[2]s,%[3]s,%[4]s,channel_two_name,the Channel Two,private,%[5]s,user1@email,user1,%[12]s,%[24]s,,message 6,message,user,

server/enterprise/message_export/testdata/csvE2E3Batch1Perm2.tmpl

@@ -0,0 +1,13 @@
+Post Creation Time,Post Update Time,Post Update Type,Team Id,Team Name,Team Display Name,Channel Id,Channel Name,Channel Display Name,Channel Type,User Id,User Email,Username,Post Id,Edited By Post Id,Replied to Post Id,Post Message,Post Type,User Type,Previews Post Id
+%[14]d,0,,%[1]s,%[2]s,%[3]s,%[4]s,channel_two_name,the Channel Two,private,%[5]s,user1@email,user1,,,,User user1 (user1@email) was already in the channel,previously-joined,user,
+%[15]d,%[15]d,,%[1]s,%[2]s,%[3]s,%[4]s,channel_two_name,the Channel Two,private,%[5]s,user1@email,user1,%[6]s,,,message 0,message,user,
+%[16]d,%[17]d,,%[1]s,%[2]s,%[3]s,%[4]s,channel_two_name,the Channel Two,private,%[5]s,user1@email,user1,%[7]s,,,message 1,message,user,
+%[16]d,%[17]d,Deleted,%[1]s,%[2]s,%[3]s,%[4]s,channel_two_name,the Channel Two,private,%[5]s,user1@email,user1,%[7]s,,,delete message 1,message,user,
+%[18]d,%[25]d,UpdatedNoMsgChange,%[1]s,%[2]s,%[3]s,%[4]s,channel_two_name,the Channel Two,private,%[5]s,user1@email,user1,%[8]s,,,message 2,message,user,
+%[19]d,%[20]d,,%[1]s,%[2]s,%[3]s,%[4]s,channel_two_name,the Channel Two,private,%[5]s,user1@email,user1,%[9]s,,,message 3,message,user,
+%[19]d,%[20]d,Deleted,%[1]s,%[2]s,%[3]s,%[4]s,channel_two_name,the Channel Two,private,%[5]s,user1@email,user1,%[9]s,,,delete message 3,message,user,
+%[19]d,%[20]d,,%[1]s,%[2]s,%[3]s,%[4]s,channel_two_name,the Channel Two,private,%[5]s,user1@email,user1,%[9]s,,,(files/%[9]s/%[13]s-file_3.txt),attachment,user,
+%[19]d,%[20]d,FileDeleted,%[1]s,%[2]s,%[3]s,%[4]s,channel_two_name,the Channel Two,private,%[5]s,user1@email,user1,%[9]s,,,(files/%[9]s/%[13]s-file_3.txt),deleted attachment,user,
+%[21]d,%[22]d,EditedNewMsg,%[1]s,%[2]s,%[3]s,%[4]s,channel_two_name,the Channel Two,private,%[5]s,user1@email,user1,%[11]s,,,edited message 4,message,user,
+%[21]d,%[22]d,EditedOriginalMsg,%[1]s,%[2]s,%[3]s,%[4]s,channel_two_name,the Channel Two,private,%[5]s,user1@email,user1,%[10]s,%[11]s,,message 4,message,user,
+%[23]d,%[26]d,EditedOriginalMsg,%[1]s,%[2]s,%[3]s,%[4]s,channel_two_name,the Channel Two,private,%[5]s,user1@email,user1,%[12]s,%[24]s,,message 6,message,user,

server/enterprise/message_export/testdata/csvE2E3Batch1Perm3.tmpl

@@ -0,0 +1,13 @@
+Post Creation Time,Post Update Time,Post Update Type,Team Id,Team Name,Team Display Name,Channel Id,Channel Name,Channel Display Name,Channel Type,User Id,User Email,Username,Post Id,Edited By Post Id,Replied to Post Id,Post Message,Post Type,User Type,Previews Post Id
+%[14]d,0,,%[1]s,%[2]s,%[3]s,%[4]s,channel_two_name,the Channel Two,private,%[5]s,user1@email,user1,,,,User user1 (user1@email) was already in the channel,previously-joined,user,
+%[15]d,%[15]d,,%[1]s,%[2]s,%[3]s,%[4]s,channel_two_name,the Channel Two,private,%[5]s,user1@email,user1,%[6]s,,,message 0,message,user,
+%[16]d,%[17]d,,%[1]s,%[2]s,%[3]s,%[4]s,channel_two_name,the Channel Two,private,%[5]s,user1@email,user1,%[7]s,,,message 1,message,user,
+%[16]d,%[17]d,Deleted,%[1]s,%[2]s,%[3]s,%[4]s,channel_two_name,the Channel Two,private,%[5]s,user1@email,user1,%[7]s,,,delete message 1,message,user,
+%[18]d,%[24]d,UpdatedNoMsgChange,%[1]s,%[2]s,%[3]s,%[4]s,channel_two_name,the Channel Two,private,%[5]s,user1@email,user1,%[8]s,,,message 2,message,user,
+%[19]d,%[20]d,,%[1]s,%[2]s,%[3]s,%[4]s,channel_two_name,the Channel Two,private,%[5]s,user1@email,user1,%[9]s,,,message 3,message,user,
+%[19]d,%[20]d,Deleted,%[1]s,%[2]s,%[3]s,%[4]s,channel_two_name,the Channel Two,private,%[5]s,user1@email,user1,%[9]s,,,delete message 3,message,user,
+%[19]d,%[20]d,,%[1]s,%[2]s,%[3]s,%[4]s,channel_two_name,the Channel Two,private,%[5]s,user1@email,user1,%[9]s,,,(files/%[9]s/%[13]s-file_3.txt),attachment,user,
+%[19]d,%[20]d,FileDeleted,%[1]s,%[2]s,%[3]s,%[4]s,channel_two_name,the Channel Two,private,%[5]s,user1@email,user1,%[9]s,,,(files/%[9]s/%[13]s-file_3.txt),deleted attachment,user,
+%[21]d,%[22]d,EditedOriginalMsg,%[1]s,%[2]s,%[3]s,%[4]s,channel_two_name,the Channel Two,private,%[5]s,user1@email,user1,%[10]s,%[11]s,,message 4,message,user,
+%[21]d,%[22]d,EditedNewMsg,%[1]s,%[2]s,%[3]s,%[4]s,channel_two_name,the Channel Two,private,%[5]s,user1@email,user1,%[11]s,,,edited message 4,message,user,
+%[25]d,%[26]d,EditedNewMsg,%[1]s,%[2]s,%[3]s,%[4]s,channel_two_name,the Channel Two,private,%[5]s,user1@email,user1,%[12]s,,,edited message 6,message,user,

server/enterprise/message_export/testdata/csvE2E3Batch1Perm4.tmpl

@@ -0,0 +1,13 @@
+Post Creation Time,Post Update Time,Post Update Type,Team Id,Team Name,Team Display Name,Channel Id,Channel Name,Channel Display Name,Channel Type,User Id,User Email,Username,Post Id,Edited By Post Id,Replied to Post Id,Post Message,Post Type,User Type,Previews Post Id
+%[14]d,0,,%[1]s,%[2]s,%[3]s,%[4]s,channel_two_name,the Channel Two,private,%[5]s,user1@email,user1,,,,User user1 (user1@email) was already in the channel,previously-joined,user,
+%[15]d,%[15]d,,%[1]s,%[2]s,%[3]s,%[4]s,channel_two_name,the Channel Two,private,%[5]s,user1@email,user1,%[6]s,,,message 0,message,user,
+%[16]d,%[17]d,,%[1]s,%[2]s,%[3]s,%[4]s,channel_two_name,the Channel Two,private,%[5]s,user1@email,user1,%[7]s,,,message 1,message,user,
+%[16]d,%[17]d,Deleted,%[1]s,%[2]s,%[3]s,%[4]s,channel_two_name,the Channel Two,private,%[5]s,user1@email,user1,%[7]s,,,delete message 1,message,user,
+%[18]d,%[24]d,UpdatedNoMsgChange,%[1]s,%[2]s,%[3]s,%[4]s,channel_two_name,the Channel Two,private,%[5]s,user1@email,user1,%[8]s,,,message 2,message,user,
+%[19]d,%[20]d,,%[1]s,%[2]s,%[3]s,%[4]s,channel_two_name,the Channel Two,private,%[5]s,user1@email,user1,%[9]s,,,message 3,message,user,
+%[19]d,%[20]d,Deleted,%[1]s,%[2]s,%[3]s,%[4]s,channel_two_name,the Channel Two,private,%[5]s,user1@email,user1,%[9]s,,,delete message 3,message,user,
+%[19]d,%[20]d,,%[1]s,%[2]s,%[3]s,%[4]s,channel_two_name,the Channel Two,private,%[5]s,user1@email,user1,%[9]s,,,(files/%[9]s/%[13]s-file_3.txt),attachment,user,
+%[19]d,%[20]d,FileDeleted,%[1]s,%[2]s,%[3]s,%[4]s,channel_two_name,the Channel Two,private,%[5]s,user1@email,user1,%[9]s,,,(files/%[9]s/%[13]s-file_3.txt),deleted attachment,user,
+%[21]d,%[22]d,EditedNewMsg,%[1]s,%[2]s,%[3]s,%[4]s,channel_two_name,the Channel Two,private,%[5]s,user1@email,user1,%[11]s,,,edited message 4,message,user,
+%[21]d,%[22]d,EditedOriginalMsg,%[1]s,%[2]s,%[3]s,%[4]s,channel_two_name,the Channel Two,private,%[5]s,user1@email,user1,%[10]s,%[11]s,,message 4,message,user,
+%[25]d,%[26]d,EditedNewMsg,%[1]s,%[2]s,%[3]s,%[4]s,channel_two_name,the Channel Two,private,%[5]s,user1@email,user1,%[12]s,,,edited message 6,message,user,

server/enterprise/message_export/testdata/csvE2E3Batch2Perm1.tmpl

@@ -0,0 +1,3 @@
+Post Creation Time,Post Update Time,Post Update Type,Team Id,Team Name,Team Display Name,Channel Id,Channel Name,Channel Display Name,Channel Type,User Id,User Email,Username,Post Id,Edited By Post Id,Replied to Post Id,Post Message,Post Type,User Type,Previews Post Id
+%[9]d,0,,%[1]s,%[2]s,%[3]s,%[4]s,channel_two_name,the Channel Two,private,%[5]s,user1@email,user1,,,,User user1 (user1@email) was already in the channel,previously-joined,user,
+%[8]d,%[10]d,EditedOriginalMsg,%[1]s,%[2]s,%[3]s,%[4]s,channel_two_name,the Channel Two,private,%[5]s,user1@email,user1,%[6]s,%[7]s,,message 6,message,user,

server/enterprise/message_export/testdata/csvE2E3Batch2Perm2.tmpl

@@ -0,0 +1,3 @@
+Post Creation Time,Post Update Time,Post Update Type,Team Id,Team Name,Team Display Name,Channel Id,Channel Name,Channel Display Name,Channel Type,User Id,User Email,Username,Post Id,Edited By Post Id,Replied to Post Id,Post Message,Post Type,User Type,Previews Post Id
+%[8]d,0,,%[1]s,%[2]s,%[3]s,%[4]s,channel_two_name,the Channel Two,private,%[5]s,user1@email,user1,,,,User user1 (user1@email) was already in the channel,previously-joined,user,
+%[7]d,%[9]d,EditedNewMsg,%[1]s,%[2]s,%[3]s,%[4]s,channel_two_name,the Channel Two,private,%[5]s,user1@email,user1,%[6]s,,,edited message 6,message,user,

server/enterprise/message_export/testdata/csvE2E4Batch1.tmpl

@@ -0,0 +1,6 @@
+%[11]d,0,,%[1]s,%[2]s,%[3]s,%[4]s,channel_two_name,the Channel Two,private,%[5]s,user1@email,user1,,,,User user1 (user1@email) was already in the channel,previously-joined,user,
+%[12]d,%[14]d,EditedOriginalMsg,%[1]s,%[2]s,%[3]s,%[4]s,channel_two_name,the Channel Two,private,%[5]s,user1@email,user1,%[6]s,%[7]s,,message 0,message,user,
+%[12]d,%[14]d,EditedNewMsg,%[1]s,%[2]s,%[3]s,%[4]s,channel_two_name,the Channel Two,private,%[5]s,user1@email,user1,%[7]s,,,edited message 0,message,user,
+%[13]d,%[14]d,,%[1]s,%[2]s,%[3]s,%[4]s,channel_two_name,the Channel Two,private,%[5]s,user1@email,user1,%[8]s,,,message 2,message,user,
+%[13]d,%[14]d,,%[1]s,%[2]s,%[3]s,%[4]s,channel_two_name,the Channel Two,private,%[5]s,user1@email,user1,%[9]s,,,message 3,message,user,
+%[13]d,%[14]d,,%[1]s,%[2]s,%[3]s,%[4]s,channel_two_name,the Channel Two,private,%[5]s,user1@email,user1,%[10]s,,,message 4,message,user,

server/enterprise/message_export/testdata/csvE2E5Job1.tmpl

@@ -0,0 +1,2 @@
+1732245676086,czoko3u8oidrtcgt58qi3afmrc,faketeambfc64t,dn_qtkwchcc7jng5mo87qw5gzxojo,zao11dtduinninqwtbjn3c6jzw,fakechannelsucatskrmt,dn_pyo5j7yqui865qt69ikoi6sgcr,public,741xw3i71frkmks64of8myh4gc,tfxgyrwfbbymzqz3c74sysbqao@localhost,fakeuser6b4riuzx5b,pf4y8d9s77domftewwfzk4fn5o,,,message 0,message,user,
+1732245676088,czoko3u8oidrtcgt58qi3afmrc,faketeambfc64t,dn_qtkwchcc7jng5mo87qw5gzxojo,zao11dtduinninqwtbjn3c6jzw,fakechannelsucatskrmt,dn_pyo5j7yqui865qt69ikoi6sgcr,public,741xw3i71frkmks64of8myh4gc,tfxgyrwfbbymzqz3c74sysbqao@localhost,fakeuser6b4riuzx5b,pf4y8d9s77domftewwfzk4fn5o,,,delete message 0,message,user,

server/enterprise/message_export/testdata/grE2E1Batch1.tmpl

@@ -0,0 +1,137 @@
+<h1>Mattermost Compliance Export</h1>
+
+<h2>Conversation Summary</h2>
+<div class=3D"summary-list">
+    <ul>
+        <li><span class=3D"bold">TeamId:</span>%[13]s</li>
+        <li><span class=3D"bold">TeamName:</span>%[14]s</li>
+        <li><span class=3D"bold">TeamDisplayName:</span>%[15]s</li>
+        <li><span class=3D"bold">ChannelId:</span>%[22]s</li>
+        <li><span class=3D"bold">ChannelName:</span>channel_two_name</li>
+        <li><span class=3D"bold">ChannelDisplayName:</span>the Channel Two</li>
+        <li><span class=3D"bold">Started:</span>%[1]s</li>
+        <li><span class=3D"bold">Ended:</span>%[2]s</li>
+        <li><span class=3D"bold">Duration:</span>0 seconds</li>
+    </ul>
+</div>
+<table class=3D"participants">
+    <tr>
+        <th class=3D"userid">UserId<br></th>
+        <th class=3D"username">Username<br></th>
+        <th class=3D"usertype">UserType<br></th>
+        <th class=3D"email">Email</th>
+        <th class=3D"joined">Joined</th>
+        <th class=3D"left">Left</th>
+        <th class=3D"duration">Duration</th>
+        <th class=3D"messages">Messages</th>
+    </tr>
+   =20
+<tr>
+    <td class=3D"userid">%[16]s</td>
+    <td class=3D"username">@user1</td>
+    <td class=3D"usertype">user</td>
+    <td class=3D"email">user1@email</td>
+    <td class=3D"joined">%[3]s</td>
+    <td class=3D"left">%[4]s</td>
+    <td class=3D"duration">0 seconds</td>
+    <td class=3D"messages">3</td>
+</tr>
+
+<tr>
+    <td class=3D"userid">%[17]s</td>
+    <td class=3D"username">@user2</td>
+    <td class=3D"usertype">user</td>
+    <td class=3D"email">user2@email</td>
+    <td class=3D"joined">%[5]s</td>
+    <td class=3D"left">%[6]s</td>
+    <td class=3D"duration">0 seconds</td>
+    <td class=3D"messages">0</td>
+</tr>
+
+<tr>
+    <td class=3D"userid">%[18]s</td>
+    <td class=3D"username">@user3</td>
+    <td class=3D"usertype">user</td>
+    <td class=3D"email">user3@email</td>
+    <td class=3D"joined">%[7]s</td>
+    <td class=3D"left">%[8]s</td>
+    <td class=3D"duration">0 seconds</td>
+    <td class=3D"messages">0</td>
+</tr>
+
+</table>
+
+<h2>Messages</h2>
+<div class=3D"message-list">
+    <ul>
+       =20
+<li class=3D"message">
+    <span class=3D"post_id">%[19]s</span>
+    <span class=3D"sent_time">%[9]s</span>
+    <span class=3D"username">@user1</span>
+    <span class=3D"userid">%[16]s</span>
+    <span class=3D"postusername">@user1</span>
+    <span class=3D"usertype">user</span>
+    <span class=3D"email">(user1@email)</span>
+    <span class=3D"message">message 0</span>
+</li>
+
+<li class=3D"message">
+    <span class=3D"post_id">%[19]s</span>
+    <span class=3D"sent_time">%[9]s</span>
+    <span class=3D"username">@user1</span>
+    <span class=3D"userid">%[16]s</span>
+    <span class=3D"postusername">@user1</span>
+    <span class=3D"usertype">user</span>
+    <span class=3D"email">(user1@email)</span>
+    <span class=3D"message">Uploaded file </span>
+</li>
+
+<li class=3D"message">
+    <span class=3D"post_id">%[20]s</span>
+    <span class=3D"sent_time">%[10]s</span>
+    <span class=3D"username">@user1</span>
+    <span class=3D"userid">%[16]s</span>
+    <span class=3D"postusername">@user1</span>
+    <span class=3D"usertype">user</span>
+    <span class=3D"email">(user1@email)</span>
+    <span class=3D"message">message 1</span>
+</li>
+
+<li class=3D"message">
+    <span class=3D"post_id">%[20]s</span>
+    <span class=3D"sent_time">%[10]s</span>
+    <span class=3D"username">@user1</span>
+    <span class=3D"userid">%[16]s</span>
+    <span class=3D"postusername">@user1</span>
+    <span class=3D"usertype">user</span>
+    <span class=3D"email">(user1@email)</span>
+    <span class=3D"message">Uploaded file </span>
+</li>
+
+<li class=3D"message">
+    <span class=3D"post_id">%[21]s</span>
+    <span class=3D"sent_time">%[11]s</span>
+    <span class=3D"username">@user1</span>
+    <span class=3D"userid">%[16]s</span>
+    <span class=3D"postusername">@user1</span>
+    <span class=3D"usertype">user</span>
+    <span class=3D"email">(user1@email)</span>
+    <span class=3D"message">message 2</span>
+</li>
+
+<li class=3D"message">
+    <span class=3D"post_id">%[21]s</span>
+    <span class=3D"sent_time">%[11]s</span>
+    <span class=3D"username">@user1</span>
+    <span class=3D"userid">%[16]s</span>
+    <span class=3D"postusername">@user1</span>
+    <span class=3D"usertype">user</span>
+    <span class=3D"email">(user1@email)</span>
+    <span class=3D"message">Uploaded file </span>
+</li>
+
+    </ul>
+</div>
+
+<p>Exported on %[12]s</p>

server/enterprise/message_export/testdata/grE2E1Batch1Ch3.tmpl

@@ -0,0 +1,47 @@
+<h2>Conversation Summary</h2>
+<div class=3D"summary-list">
+    <ul>
+        <li><span class=3D"bold">TeamId:</span>%[1]s</li>
+        <li><span class=3D"bold">TeamName:</span></li>
+        <li><span class=3D"bold">TeamDisplayName:</span></li>
+        <li><span class=3D"bold">ChannelId:</span>%[2]s</li>
+        <li><span class=3D"bold">ChannelName:</span>channel_three_name</li>
+        <li><span class=3D"bold">ChannelDisplayName:</span>the Channel Three</li>
+        <li><span class=3D"bold">Started:</span>%[3]s</li>
+        <li><span class=3D"bold">Ended:</span>%[4]s</li>
+        <li><span class=3D"bold">Duration:</span>0 seconds</li>
+    </ul>
+</div>
+<table class=3D"participants">
+    <tr>
+        <th class=3D"userid">UserId<br></th>
+        <th class=3D"username">Username<br></th>
+        <th class=3D"usertype">UserType<br></th>
+        <th class=3D"email">Email</th>
+        <th class=3D"joined">Joined</th>
+        <th class=3D"left">Left</th>
+        <th class=3D"duration">Duration</th>
+        <th class=3D"messages">Messages</th>
+    </tr>
+   =20
+<tr>
+    <td class=3D"userid">%[5]s</td>
+    <td class=3D"username">@user7</td>
+    <td class=3D"usertype">user</td>
+    <td class=3D"email">user7@email</td>
+    <td class=3D"joined">%[6]s</td>
+    <td class=3D"left">%[7]s</td>
+    <td class=3D"duration">0 seconds</td>
+    <td class=3D"messages">0</td>
+</tr>
+
+</table>
+
+<h2>Messages</h2>
+<div class=3D"message-list">
+    <ul>
+       =20
+    </ul>
+</div>
+
+<p>Exported on %[8]s</p>

server/enterprise/message_export/testdata/grE2E1Batch1Summary.tmpl

@@ -0,0 +1,29 @@
+--------------------
+Conversation Summary
+--------------------
+
+* TeamId: %[14]s
+* TeamName: %[15]s
+* TeamDisplayName: %[16]s
+* ChannelId: %[13]s
+* ChannelName: channel_two_name
+* ChannelDisplayName: %[1]s
+* Started: %[2]s
+* Ended: %[3]s
+* Duration: 0 seconds
+
+UserId
+Username
+UserType
+Email Joined Left Duration Messages %[17]s @user1 user user1@email %[4]s %[5]s 0 seconds 3 %[18]s @user2 user user2@email %[6]s %[7]s 0 seconds 0 %[19]s @user3 user user3@email %[8]s %[9]s 0 seconds 0
+
+--------
+Messages
+--------
+
+* %[20]s %[10]s @user1 %[17]s @user1 user (user1@email) message 0
+* %[20]s %[10]s @user1 %[17]s @user1 user (user1@email) Uploaded file
+* %[21]s %[11]s @user1 %[17]s @user1 user (user1@email) message 1
+* %[21]s %[11]s @user1 %[17]s @user1 user (user1@email) Uploaded file
+* %[22]s %[12]s @user1 %[17]s @user1 user (user1@email) message 2
+* %[22]s %[12]s @user1 %[17]s @user1 user (user1@email) Uploaded file

server/enterprise/message_export/testdata/grE2E1Batch1SummaryCh3.tmpl

@@ -0,0 +1,22 @@
+--------------------
+Conversation Summary
+--------------------
+
+* TeamId: %[1]s
+* TeamName:
+* TeamDisplayName:
+* ChannelId: %[4]s
+* ChannelName: channel_three_name
+* ChannelDisplayName: the Channel Three
+* Started: %[5]s
+* Ended: %[6]s
+* Duration: 0 seconds
+
+UserId
+Username
+UserType
+Email Joined Left Duration Messages %[7]s @user7 user user7@email %[8]s %[9]s 0 seconds 0
+
+--------
+Messages
+--------

server/enterprise/message_export/testdata/grE2E1Batch2.tmpl

@@ -0,0 +1,126 @@
+<h1>Mattermost Compliance Export</h1>
+
+<h2>Conversation Summary</h2>
+<div class=3D"summary-list">
+    <ul>
+        <li><span class=3D"bold">TeamId:</span>%[11]s</li>
+        <li><span class=3D"bold">TeamName:</span>%[12]s</li>
+        <li><span class=3D"bold">TeamDisplayName:</span>%[13]s</li>
+        <li><span class=3D"bold">ChannelId:</span>%[20]s</li>
+        <li><span class=3D"bold">ChannelName:</span>channel_two_name</li>
+        <li><span class=3D"bold">ChannelDisplayName:</span>the Channel Two</li>
+        <li><span class=3D"bold">Started:</span>%[1]s</li>
+        <li><span class=3D"bold">Ended:</span>%[2]s</li>
+        <li><span class=3D"bold">Duration:</span>0 seconds</li>
+    </ul>
+</div>
+<table class=3D"participants">
+    <tr>
+        <th class=3D"userid">UserId<br></th>
+        <th class=3D"username">Username<br></th>
+        <th class=3D"usertype">UserType<br></th>
+        <th class=3D"email">Email</th>
+        <th class=3D"joined">Joined</th>
+        <th class=3D"left">Left</th>
+        <th class=3D"duration">Duration</th>
+        <th class=3D"messages">Messages</th>
+    </tr>
+   =20
+<tr>
+    <td class=3D"userid">%[14]s</td>
+    <td class=3D"username">@user1</td>
+    <td class=3D"usertype">user</td>
+    <td class=3D"email">user1@email</td>
+    <td class=3D"joined">%[3]s</td>
+    <td class=3D"left">%[4]s</td>
+    <td class=3D"duration">0 seconds</td>
+    <td class=3D"messages">3</td>
+</tr>
+
+<tr>
+    <td class=3D"userid">%[16]s</td>
+    <td class=3D"username">@user4</td>
+    <td class=3D"usertype">user</td>
+    <td class=3D"email">user4@email</td>
+    <td class=3D"joined">%[5]s</td>
+    <td class=3D"left">%[6]s</td>
+    <td class=3D"duration">0 seconds</td>
+    <td class=3D"messages">0</td>
+</tr>
+
+</table>
+
+<h2>Messages</h2>
+<div class=3D"message-list">
+    <ul>
+       =20
+<li class=3D"message">
+    <span class=3D"post_id">%[17]s</span>
+    <span class=3D"sent_time">%[7]s</span>
+    <span class=3D"username">@user1</span>
+    <span class=3D"userid">%[14]s</span>
+    <span class=3D"postusername">@user1</span>
+    <span class=3D"usertype">user</span>
+    <span class=3D"email">(user1@email)</span>
+    <span class=3D"message">message 3</span>
+</li>
+
+<li class=3D"message">
+    <span class=3D"post_id">%[17]s</span>
+    <span class=3D"sent_time">%[7]s</span>
+    <span class=3D"username">@user1</span>
+    <span class=3D"userid">%[14]s</span>
+    <span class=3D"postusername">@user1</span>
+    <span class=3D"usertype">user</span>
+    <span class=3D"email">(user1@email)</span>
+    <span class=3D"message">Uploaded file </span>
+</li>
+
+<li class=3D"message">
+    <span class=3D"post_id">%[18]s</span>
+    <span class=3D"sent_time">%[8]s</span>
+    <span class=3D"username">@user1</span>
+    <span class=3D"userid">%[14]s</span>
+    <span class=3D"postusername">@user1</span>
+    <span class=3D"usertype">user</span>
+    <span class=3D"email">(user1@email)</span>
+    <span class=3D"message">message 4</span>
+</li>
+
+<li class=3D"message">
+    <span class=3D"post_id">%[18]s</span>
+    <span class=3D"sent_time">%[8]s</span>
+    <span class=3D"username">@user1</span>
+    <span class=3D"userid">%[14]s</span>
+    <span class=3D"postusername">@user1</span>
+    <span class=3D"usertype">user</span>
+    <span class=3D"email">(user1@email)</span>
+    <span class=3D"message">Uploaded file </span>
+</li>
+
+<li class=3D"message">
+    <span class=3D"post_id">%[19]s</span>
+    <span class=3D"sent_time">%[9]s</span>
+    <span class=3D"username">@user1</span>
+    <span class=3D"userid">%[14]s</span>
+    <span class=3D"postusername">@user1</span>
+    <span class=3D"usertype">user</span>
+    <span class=3D"email">(user1@email)</span>
+    <span class=3D"message">message 5</span>
+</li>
+
+<li class=3D"message">
+    <span class=3D"post_id">%[19]s</span>
+    <span class=3D"sent_time">%[9]s</span>
+    <span class=3D"username">@user1</span>
+    <span class=3D"userid">%[14]s</span>
+    <span class=3D"postusername">@user1</span>
+    <span class=3D"usertype">user</span>
+    <span class=3D"email">(user1@email)</span>
+    <span class=3D"message">Uploaded file </span>
+</li>
+
+    </ul>
+</div>
+
+<p>Exported on %[10]s</p>

server/enterprise/message_export/testdata/grE2E1Batch2Summary.tmpl

@@ -0,0 +1,29 @@
+--------------------
+Conversation Summary
+--------------------
+
+* TeamId: %[11]s
+* TeamName: %[12]s
+* TeamDisplayName: %[13]s
+* ChannelId: %[10]s
+* ChannelName: channel_two_name
+* ChannelDisplayName: the Channel Two
+* Started: %[1]s
+* Ended: %[2]s
+* Duration: 0 seconds
+
+UserId
+Username
+UserType
+Email Joined Left Duration Messages %[14]s @user1 user user1@email %[3]s %[4]s 0 seconds 3 %[15]s @user4 user user4@email %[5]s %[6]s 0 seconds 0
+
+--------
+Messages
+--------
+
+* %[16]s %[7]s @user1 %[14]s @user1 user (user1@email) message 3
+* %[16]s %[7]s @user1 %[14]s @user1 user (user1@email) Uploaded file
+* %[17]s %[8]s @user1 %[14]s @user1 user (user1@email) message 4
+* %[17]s %[8]s @user1 %[14]s @user1 user (user1@email) Uploaded file
+* %[18]s %[9]s @user1 %[14]s @user1 user (user1@email) message 5
+* %[18]s %[9]s @user1 %[14]s @user1 user (user1@email) Uploaded file

server/enterprise/message_export/testdata/grE2E1Batch3.tmpl

@@ -0,0 +1,137 @@
+<h1>Mattermost Compliance Export</h1>
+
+<h2>Conversation Summary</h2>
+<div class=3D"summary-list">
+    <ul>
+        <li><span class=3D"bold">TeamId:</span>%[13]s</li>
+        <li><span class=3D"bold">TeamName:</span>%[14]s</li>
+        <li><span class=3D"bold">TeamDisplayName:</span>%[15]s</li>
+        <li><span class=3D"bold">ChannelId:</span>%[22]s</li>
+        <li><span class=3D"bold">ChannelName:</span>channel_two_name</li>
+        <li><span class=3D"bold">ChannelDisplayName:</span>the Channel Two</li>
+        <li><span class=3D"bold">Started:</span>%[1]s</li>
+        <li><span class=3D"bold">Ended:</span>%[2]s</li>
+        <li><span class=3D"bold">Duration:</span>0 seconds</li>
+    </ul>
+</div>
+<table class=3D"participants">
+    <tr>
+        <th class=3D"userid">UserId<br></th>
+        <th class=3D"username">Username<br></th>
+        <th class=3D"usertype">UserType<br></th>
+        <th class=3D"email">Email</th>
+        <th class=3D"joined">Joined</th>
+        <th class=3D"left">Left</th>
+        <th class=3D"duration">Duration</th>
+        <th class=3D"messages">Messages</th>
+    </tr>
+   =20
+<tr>
+    <td class=3D"userid">%[16]s</td>
+    <td class=3D"username">@user1</td>
+    <td class=3D"usertype">user</td>
+    <td class=3D"email">user1@email</td>
+    <td class=3D"joined">%[3]s</td>
+    <td class=3D"left">%[4]s</td>
+    <td class=3D"duration">0 seconds</td>
+    <td class=3D"messages">3</td>
+</tr>
+
+<tr>
+    <td class=3D"userid">%[17]s</td>
+    <td class=3D"username">@user5</td>
+    <td class=3D"usertype">user</td>
+    <td class=3D"email">user5@email</td>
+    <td class=3D"joined">%[5]s</td>
+    <td class=3D"left">%[6]s</td>
+    <td class=3D"duration">0 seconds</td>
+    <td class=3D"messages">0</td>
+</tr>
+
+<tr>
+    <td class=3D"userid">%[18]s</td>
+    <td class=3D"username">@user6</td>
+    <td class=3D"usertype">user</td>
+    <td class=3D"email">user6@email</td>
+    <td class=3D"joined">%[7]s</td>
+    <td class=3D"left">%[8]s</td>
+    <td class=3D"duration">0 seconds</td>
+    <td class=3D"messages">0</td>
+</tr>
+
+</table>
+
+<h2>Messages</h2>
+<div class=3D"message-list">
+    <ul>
+       =20
+<li class=3D"message">
+    <span class=3D"post_id">%[19]s</span>
+    <span class=3D"sent_time">%[9]s</span>
+    <span class=3D"username">@user1</span>
+    <span class=3D"userid">%[16]s</span>
+    <span class=3D"postusername">@user1</span>
+    <span class=3D"usertype">user</span>
+    <span class=3D"email">(user1@email)</span>
+    <span class=3D"message">message 6</span>
+</li>
+
+<li class=3D"message">
+    <span class=3D"post_id">%[19]s</span>
+    <span class=3D"sent_time">%[9]s</span>
+    <span class=3D"username">@user1</span>
+    <span class=3D"userid">%[16]s</span>
+    <span class=3D"postusername">@user1</span>
+    <span class=3D"usertype">user</span>
+    <span class=3D"email">(user1@email)</span>
+    <span class=3D"message">Uploaded file </span>
+</li>
+
+<li class=3D"message">
+    <span class=3D"post_id">%[20]s</span>
+    <span class=3D"sent_time">%[10]s</span>
+    <span class=3D"username">@user1</span>
+    <span class=3D"userid">%[16]s</span>
+    <span class=3D"postusername">@user1</span>
+    <span class=3D"usertype">user</span>
+    <span class=3D"email">(user1@email)</span>
+    <span class=3D"message">message 7</span>
+</li>
+
+<li class=3D"message">
+    <span class=3D"post_id">%[20]s</span>
+    <span class=3D"sent_time">%[10]s</span>
+    <span class=3D"username">@user1</span>
+    <span class=3D"userid">%[16]s</span>
+    <span class=3D"postusername">@user1</span>
+    <span class=3D"usertype">user</span>
+    <span class=3D"email">(user1@email)</span>
+    <span class=3D"message">Uploaded file </span>
+</li>
+
+<li class=3D"message">
+    <span class=3D"post_id">%[21]s</span>
+    <span class=3D"sent_time">%[11]s</span>
+    <span class=3D"username">@user1</span>
+    <span class=3D"userid">%[16]s</span>
+    <span class=3D"postusername">@user1</span>
+    <span class=3D"usertype">user</span>
+    <span class=3D"email">(user1@email)</span>
+    <span class=3D"message">message 8</span>
+</li>
+
+<li class=3D"message">
+    <span class=3D"post_id">%[21]s</span>
+    <span class=3D"sent_time">%[11]s</span>
+    <span class=3D"username">@user1</span>
+    <span class=3D"userid">%[16]s</span>
+    <span class=3D"postusername">@user1</span>
+    <span class=3D"usertype">user</span>
+    <span class=3D"email">(user1@email)</span>
+    <span class=3D"message">Uploaded file </span>
+</li>
+
+    </ul>
+</div>
+
+<p>Exported on %[12]s</p>

server/enterprise/message_export/testdata/grE2E1Batch3Ch4.tmpl

@@ -0,0 +1,47 @@
+<h2>Conversation Summary</h2>
+<div class=3D"summary-list">
+    <ul>
+        <li><span class=3D"bold">TeamId:</span>%[1]s</li>
+        <li><span class=3D"bold">TeamName:</span></li>
+        <li><span class=3D"bold">TeamDisplayName:</span></li>
+        <li><span class=3D"bold">ChannelId:</span>%[2]s</li>
+        <li><span class=3D"bold">ChannelName:</span>channel_four_name</li>
+        <li><span class=3D"bold">ChannelDisplayName:</span>the Channel Four</li>
+        <li><span class=3D"bold">Started:</span>%[3]s</li>
+        <li><span class=3D"bold">Ended:</span>%[4]s</li>
+        <li><span class=3D"bold">Duration:</span>0 seconds</li>
+    </ul>
+</div>
+<table class=3D"participants">
+    <tr>
+        <th class=3D"userid">UserId<br></th>
+        <th class=3D"username">Username<br></th>
+        <th class=3D"usertype">UserType<br></th>
+        <th class=3D"email">Email</th>
+        <th class=3D"joined">Joined</th>
+        <th class=3D"left">Left</th>
+        <th class=3D"duration">Duration</th>
+        <th class=3D"messages">Messages</th>
+    </tr>
+   =20
+<tr>
+    <td class=3D"userid">%[5]s</td>
+    <td class=3D"username">@user8</td>
+    <td class=3D"usertype">user</td>
+    <td class=3D"email">user8@email</td>
+    <td class=3D"joined">%[6]s</td>
+    <td class=3D"left">%[4]s</td>
+    <td class=3D"duration">0 seconds</td>
+    <td class=3D"messages">0</td>
+</tr>
+
+</table>
+
+<h2>Messages</h2>
+<div class=3D"message-list">
+    <ul>
+       =20
+    </ul>
+</div>
+
+<p>Exported on %[8]s</p>

server/enterprise/message_export/testdata/grE2E1Batch3Summary.tmpl

@@ -0,0 +1,29 @@
+--------------------
+Conversation Summary
+--------------------
+
+* TeamId: %[13]s
+* TeamName: %[14]s
+* TeamDisplayName: %[15]s
+* ChannelId: %[12]s
+* ChannelName: channel_two_name
+* ChannelDisplayName: the Channel Two
+* Started: %[1]s
+* Ended: %[2]s
+* Duration: 0 seconds
+
+UserId
+Username
+UserType
+Email Joined Left Duration Messages %[16]s @user1 user user1@email %[3]s %[4]s 0 seconds 3 %[22]s @user5 user user5@email %[5]s %[6]s 0 seconds 0 %[23]s @user6 user user6@email %[7]s %[8]s 0 seconds 0
+
+--------
+Messages
+--------
+
+* %[19]s %[9]s @user1 %[16]s @user1 user (user1@email) message 6
+* %[19]s %[9]s @user1 %[16]s @user1 user (user1@email) Uploaded file
+* %[20]s %[10]s @user1 %[16]s @user1 user (user1@email) message 7
+* %[20]s %[10]s @user1 %[16]s @user1 user (user1@email) Uploaded file
+* %[21]s %[11]s @user1 %[16]s @user1 user (user1@email) message 8
+* %[21]s %[11]s @user1 %[16]s @user1 user (user1@email) Uploaded file

server/enterprise/message_export/testdata/grE2E1Batch3SummaryCh4.tmpl

@@ -0,0 +1,22 @@
+--------------------
+Conversation Summary
+--------------------
+
+* TeamId: %[1]s
+* TeamName:
+* TeamDisplayName:
+* ChannelId: %[4]s
+* ChannelName: channel_four_name
+* ChannelDisplayName: the Channel Four
+* Started: %[5]s
+* Ended: %[6]s
+* Duration: 0 seconds
+
+UserId
+Username
+UserType
+Email Joined Left Duration Messages %[7]s @user8 user user8@email %[8]s %[6]s 0 seconds 0
+
+--------
+Messages
+--------

server/enterprise/message_export/testdata/grE2E2Batch1.tmpl

@@ -0,0 +1,82 @@
+<h1>Mattermost Compliance Export</h1>
+
+<h2>Conversation Summary</h2>
+<div class=3D"summary-list">
+    <ul>
+        <li><span class=3D"bold">TeamId:</span>%[10]s</li>
+        <li><span class=3D"bold">TeamName:</span>%[11]s</li>
+        <li><span class=3D"bold">TeamDisplayName:</span>%[12]s</li>
+        <li><span class=3D"bold">ChannelId:</span>%[13]s</li>
+        <li><span class=3D"bold">ChannelName:</span>channel_two_name</li>
+        <li><span class=3D"bold">ChannelDisplayName:</span>the Channel Two</li>
+        <li><span class=3D"bold">Started:</span>%[1]s</li>
+        <li><span class=3D"bold">Ended:</span>%[2]s</li>
+        <li><span class=3D"bold">Duration:</span>0 seconds</li>
+    </ul>
+</div>
+<table class=3D"participants">
+    <tr>
+        <th class=3D"userid">UserId<br></th>
+        <th class=3D"username">Username<br></th>
+        <th class=3D"usertype">UserType<br></th>
+        <th class=3D"email">Email</th>
+        <th class=3D"joined">Joined</th>
+        <th class=3D"left">Left</th>
+        <th class=3D"duration">Duration</th>
+        <th class=3D"messages">Messages</th>
+    </tr>
+   =20
+<tr>
+    <td class=3D"userid">%[14]s</td>
+    <td class=3D"username">@user1</td>
+    <td class=3D"usertype">user</td>
+    <td class=3D"email">user1@email</td>
+    <td class=3D"joined">%[3]s</td>
+    <td class=3D"left">%[4]s</td>
+    <td class=3D"duration">0 seconds</td>
+    <td class=3D"messages">1</td>
+</tr>
+
+<tr>
+    <td class=3D"userid">%[15]s</td>
+    <td class=3D"username">@user2</td>
+    <td class=3D"usertype">user</td>
+    <td class=3D"email">user2@email</td>
+    <td class=3D"joined">%[5]s</td>
+    <td class=3D"left">%[6]s</td>
+    <td class=3D"duration">0 seconds</td>
+    <td class=3D"messages">1</td>
+</tr>
+
+</table>
+
+<h2>Messages</h2>
+<div class=3D"message-list">
+    <ul>
+       =20
+<li class=3D"message">
+    <span class=3D"post_id">%[16]s</span>
+    <span class=3D"sent_time">%[7]s</span>
+    <span class=3D"username">@user1</span>
+    <span class=3D"userid">%[14]s</span>
+    <span class=3D"postusername">@user1</span>
+    <span class=3D"usertype">user</span>
+    <span class=3D"email">(user1@email)</span>
+    <span class=3D"message">message 1</span>
+</li>
+
+<li class=3D"message">
+    <span class=3D"post_id">%[17]s</span>
+    <span class=3D"sent_time">%[8]s</span>
+    <span class=3D"username">@user2</span>
+    <span class=3D"userid">%[15]s</span>
+    <span class=3D"postusername">@user2</span>
+    <span class=3D"usertype">user</span>
+    <span class=3D"email">(user2@email)</span>
+    <span class=3D"message">message 2</span>
+</li>
+
+    </ul>
+</div>
+
+<p>Exported on %[9]s</p>

server/enterprise/message_export/testdata/grE2E2Batch1Summary.tmpl

@@ -0,0 +1,25 @@
+--------------------
+Conversation Summary
+--------------------
+
+* TeamId: %[9]s
+* TeamName: %[10]s
+* TeamDisplayName: %[11]s
+* ChannelId: %[12]s
+* ChannelName: channel_two_name
+* ChannelDisplayName: the Channel Two
+* Started: %[1]s
+* Ended: %[2]s
+* Duration: 0 seconds
+
+UserId
+Username
+UserType
+Email Joined Left Duration Messages %[13]s @user1 user user1@email %[3]s %[4]s 0 seconds 1 %[14]s @user2 user user2@email %[5]s %[6]s 0 seconds 1
+
+--------
+Messages
+--------
+
+* %[15]s %[7]s @user1 %[13]s @user1 user (user1@email) message 1
+* %[16]s %[8]s @user2 %[14]s @user2 user (user2@email) message 2

server/enterprise/message_export/testdata/grE2E3Batch1Perm1.tmpl

@@ -0,0 +1,191 @@
+<h1>Mattermost Compliance Export</h1>
+
+<h2>Conversation Summary</h2>
+<div class=3D"summary-list">
+    <ul>
+        <li><span class=3D"bold">TeamId:</span>%[13]s</li>
+        <li><span class=3D"bold">TeamName:</span>%[14]s</li>
+        <li><span class=3D"bold">TeamDisplayName:</span>%[15]s</li>
+        <li><span class=3D"bold">ChannelId:</span>%[16]s</li>
+        <li><span class=3D"bold">ChannelName:</span>channel_two_name</li>
+        <li><span class=3D"bold">ChannelDisplayName:</span>the Channel Two</li>
+        <li><span class=3D"bold">Started:</span>%[1]s</li>
+        <li><span class=3D"bold">Ended:</span>%[2]s</li>
+        <li><span class=3D"bold">Duration:</span>0 seconds</li>
+    </ul>
+</div>
+<table class=3D"participants">
+    <tr>
+        <th class=3D"userid">UserId<br></th>
+        <th class=3D"username">Username<br></th>
+        <th class=3D"usertype">UserType<br></th>
+        <th class=3D"email">Email</th>
+        <th class=3D"joined">Joined</th>
+        <th class=3D"left">Left</th>
+        <th class=3D"duration">Duration</th>
+        <th class=3D"messages">Messages</th>
+    </tr>
+   =20
+<tr>
+    <td class=3D"userid">%[17]s</td>
+    <td class=3D"username">@user1</td>
+    <td class=3D"usertype">user</td>
+    <td class=3D"email">user1@email</td>
+    <td class=3D"joined">%[3]s</td>
+    <td class=3D"left">%[4]s</td>
+    <td class=3D"duration">0 seconds</td>
+    <td class=3D"messages">9</td>
+</tr>
+
+</table>
+
+<h2>Messages</h2>
+<div class=3D"message-list">
+    <ul>
+       =20
+<li class=3D"message">
+    <span class=3D"post_id">%[18]s</span>
+    <span class=3D"sent_time">%[5]s</span>
+    <span class=3D"username">@user1</span>
+    <span class=3D"userid">%[17]s</span>
+    <span class=3D"postusername">@user1</span>
+    <span class=3D"usertype">user</span>
+    <span class=3D"email">(user1@email)</span>
+    <span class=3D"message">message 0</span>
+</li>
+
+<li class=3D"message">
+    <span class=3D"post_id">%[19]s</span>
+    <span class=3D"sent_time">%[6]s</span>
+    <span class=3D"username">@user1</span>
+    <span class=3D"userid">%[17]s</span>
+    <span class=3D"postusername">@user1</span>
+    <span class=3D"usertype">user</span>
+    <span class=3D"email">(user1@email)</span>
+    <span class=3D"message">message 1</span>
+</li>
+
+<li class=3D"message">
+    <span class=3D"post_id">%[19]s</span>
+    <span class=3D"sent_time">%[6]s</span>
+    <span class=3D"username">@user1</span>
+    <span class=3D"userid">%[17]s</span>
+    <span class=3D"postusername">@user1</span>
+    <span class=3D"usertype">user</span>
+    <span class=3D"email">(user1@email)</span>
+    <span class=3D"message">delete message 1</span>
+    <span class=3D"update_type">Deleted</span>
+    <span class=3D"update_time">%[26]s</span>
+    <span class=3D"edited_new_msg_id"></span>
+</li>
+
+<li class=3D"message">
+    <span class=3D"post_id">%[20]s</span>
+    <span class=3D"sent_time">%[7]s</span>
+    <span class=3D"username">@user1</span>
+    <span class=3D"userid">%[17]s</span>
+    <span class=3D"postusername">@user1</span>
+    <span class=3D"usertype">user</span>
+    <span class=3D"email">(user1@email)</span>
+    <span class=3D"message">message 2</span>
+    <span class=3D"update_type">UpdatedNoMsgChange</span>
+    <span class=3D"update_time">%[27]s</span>
+    <span class=3D"edited_new_msg_id"></span>
+</li>
+
+<li class=3D"message">
+    <span class=3D"post_id">%[21]s</span>
+    <span class=3D"sent_time">%[8]s</span>
+    <span class=3D"username">@user1</span>
+    <span class=3D"userid">%[17]s</span>
+    <span class=3D"postusername">@user1</span>
+    <span class=3D"usertype">user</span>
+    <span class=3D"email">(user1@email)</span>
+    <span class=3D"message">message 3</span>
+</li>
+
+<li class=3D"message">
+    <span class=3D"post_id">%[21]s</span>
+    <span class=3D"sent_time">%[8]s</span>
+    <span class=3D"username">@user1</span>
+    <span class=3D"userid">%[17]s</span>
+    <span class=3D"postusername">@user1</span>
+    <span class=3D"usertype">user</span>
+    <span class=3D"email">(user1@email)</span>
+    <span class=3D"message">Uploaded file </span>
+</li>
+
+<li class=3D"message">
+    <span class=3D"post_id">%[21]s</span>
+    <span class=3D"sent_time">%[8]s</span>
+    <span class=3D"username">@user1</span>
+    <span class=3D"userid">%[17]s</span>
+    <span class=3D"postusername">@user1</span>
+    <span class=3D"usertype">user</span>
+    <span class=3D"email">(user1@email)</span>
+    <span class=3D"message">delete message 3</span>
+    <span class=3D"update_type">Deleted</span>
+    <span class=3D"update_time">%[9]s</span>
+    <span class=3D"edited_new_msg_id"></span>
+</li>
+
+<li class=3D"message">
+    <span class=3D"post_id">%[21]s</span>
+    <span class=3D"sent_time">%[8]s</span>
+    <span class=3D"username">@user1</span>
+    <span class=3D"userid">%[17]s</span>
+    <span class=3D"postusername">@user1</span>
+    <span class=3D"usertype">user</span>
+    <span class=3D"email">(user1@email)</span>
+    <span class=3D"message">Deleted file </span>
+    <span class=3D"update_type">FileDeleted</span>
+    <span class=3D"update_time">%[9]s</span>
+    <span class=3D"edited_new_msg_id"></span>
+</li>
+
+<li class=3D"message">
+    <span class=3D"post_id">%[22]s</span>
+    <span class=3D"sent_time">%[10]s</span>
+    <span class=3D"username">@user1</span>
+    <span class=3D"userid">%[17]s</span>
+    <span class=3D"postusername">@user1</span>
+    <span class=3D"usertype">user</span>
+    <span class=3D"email">(user1@email)</span>
+    <span class=3D"message">message 4</span>
+    <span class=3D"update_type">EditedOriginalMsg</span>
+    <span class=3D"update_time">%[28]s</span>
+    <span class=3D"edited_new_msg_id">%[23]s</span>
+</li>
+
+<li class=3D"message">
+    <span class=3D"post_id">%[23]s</span>
+    <span class=3D"sent_time">%[11]s</span>
+    <span class=3D"username">@user1</span>
+    <span class=3D"userid">%[17]s</span>
+    <span class=3D"postusername">@user1</span>
+    <span class=3D"usertype">user</span>
+    <span class=3D"email">(user1@email)</span>
+    <span class=3D"message">edited message 4</span>
+    <span class=3D"update_type">EditedNewMsg</span>
+    <span class=3D"update_time">%[28]s</span>
+    <span class=3D"edited_new_msg_id"></span>
+</li>
+
+<li class=3D"message">
+    <span class=3D"post_id">%[24]s</span>
+    <span class=3D"sent_time">%[12]s</span>
+    <span class=3D"username">@user1</span>
+    <span class=3D"userid">%[17]s</span>
+    <span class=3D"postusername">@user1</span>
+    <span class=3D"usertype">user</span>
+    <span class=3D"email">(user1@email)</span>
+    <span class=3D"message">message 6</span>
+    <span class=3D"update_type">EditedOriginalMsg</span>
+    <span class=3D"update_time">%[29]s</span>
+    <span class=3D"edited_new_msg_id">%[25]s</span>
+</li>
+
+    </ul>
+</div>
+
+<p>Exported on %[1]s</p>

server/enterprise/message_export/testdata/grE2E3Batch1Perm2.tmpl

@@ -0,0 +1,191 @@
+<h1>Mattermost Compliance Export</h1>
+
+<h2>Conversation Summary</h2>
+<div class=3D"summary-list">
+    <ul>
+        <li><span class=3D"bold">TeamId:</span>%[13]s</li>
+        <li><span class=3D"bold">TeamName:</span>%[14]s</li>
+        <li><span class=3D"bold">TeamDisplayName:</span>%[15]s</li>
+        <li><span class=3D"bold">ChannelId:</span>%[16]s</li>
+        <li><span class=3D"bold">ChannelName:</span>channel_two_name</li>
+        <li><span class=3D"bold">ChannelDisplayName:</span>the Channel Two</li>
+        <li><span class=3D"bold">Started:</span>%[1]s</li>
+        <li><span class=3D"bold">Ended:</span>%[2]s</li>
+        <li><span class=3D"bold">Duration:</span>0 seconds</li>
+    </ul>
+</div>
+<table class=3D"participants">
+    <tr>
+        <th class=3D"userid">UserId<br></th>
+        <th class=3D"username">Username<br></th>
+        <th class=3D"usertype">UserType<br></th>
+        <th class=3D"email">Email</th>
+        <th class=3D"joined">Joined</th>
+        <th class=3D"left">Left</th>
+        <th class=3D"duration">Duration</th>
+        <th class=3D"messages">Messages</th>
+    </tr>
+   =20
+<tr>
+    <td class=3D"userid">%[17]s</td>
+    <td class=3D"username">@user1</td>
+    <td class=3D"usertype">user</td>
+    <td class=3D"email">user1@email</td>
+    <td class=3D"joined">%[3]s</td>
+    <td class=3D"left">%[4]s</td>
+    <td class=3D"duration">0 seconds</td>
+    <td class=3D"messages">9</td>
+</tr>
+
+</table>
+
+<h2>Messages</h2>
+<div class=3D"message-list">
+    <ul>
+       =20
+<li class=3D"message">
+    <span class=3D"post_id">%[18]s</span>
+    <span class=3D"sent_time">%[5]s</span>
+    <span class=3D"username">@user1</span>
+    <span class=3D"userid">%[17]s</span>
+    <span class=3D"postusername">@user1</span>
+    <span class=3D"usertype">user</span>
+    <span class=3D"email">(user1@email)</span>
+    <span class=3D"message">message 0</span>
+</li>
+
+<li class=3D"message">
+    <span class=3D"post_id">%[19]s</span>
+    <span class=3D"sent_time">%[6]s</span>
+    <span class=3D"username">@user1</span>
+    <span class=3D"userid">%[17]s</span>
+    <span class=3D"postusername">@user1</span>
+    <span class=3D"usertype">user</span>
+    <span class=3D"email">(user1@email)</span>
+    <span class=3D"message">message 1</span>
+</li>
+
+<li class=3D"message">
+    <span class=3D"post_id">%[19]s</span>
+    <span class=3D"sent_time">%[6]s</span>
+    <span class=3D"username">@user1</span>
+    <span class=3D"userid">%[17]s</span>
+    <span class=3D"postusername">@user1</span>
+    <span class=3D"usertype">user</span>
+    <span class=3D"email">(user1@email)</span>
+    <span class=3D"message">delete message 1</span>
+    <span class=3D"update_type">Deleted</span>
+    <span class=3D"update_time">%[26]s</span>
+    <span class=3D"edited_new_msg_id"></span>
+</li>
+
+<li class=3D"message">
+    <span class=3D"post_id">%[20]s</span>
+    <span class=3D"sent_time">%[7]s</span>
+    <span class=3D"username">@user1</span>
+    <span class=3D"userid">%[17]s</span>
+    <span class=3D"postusername">@user1</span>
+    <span class=3D"usertype">user</span>
+    <span class=3D"email">(user1@email)</span>
+    <span class=3D"message">message 2</span>
+    <span class=3D"update_type">UpdatedNoMsgChange</span>
+    <span class=3D"update_time">%[27]s</span>
+    <span class=3D"edited_new_msg_id"></span>
+</li>
+
+<li class=3D"message">
+    <span class=3D"post_id">%[21]s</span>
+    <span class=3D"sent_time">%[8]s</span>
+    <span class=3D"username">@user1</span>
+    <span class=3D"userid">%[17]s</span>
+    <span class=3D"postusername">@user1</span>
+    <span class=3D"usertype">user</span>
+    <span class=3D"email">(user1@email)</span>
+    <span class=3D"message">message 3</span>
+</li>
+
+<li class=3D"message">
+    <span class=3D"post_id">%[21]s</span>
+    <span class=3D"sent_time">%[8]s</span>
+    <span class=3D"username">@user1</span>
+    <span class=3D"userid">%[17]s</span>
+    <span class=3D"postusername">@user1</span>
+    <span class=3D"usertype">user</span>
+    <span class=3D"email">(user1@email)</span>
+    <span class=3D"message">Uploaded file </span>
+</li>
+
+<li class=3D"message">
+    <span class=3D"post_id">%[21]s</span>
+    <span class=3D"sent_time">%[8]s</span>
+    <span class=3D"username">@user1</span>
+    <span class=3D"userid">%[17]s</span>
+    <span class=3D"postusername">@user1</span>
+    <span class=3D"usertype">user</span>
+    <span class=3D"email">(user1@email)</span>
+    <span class=3D"message">delete message 3</span>
+    <span class=3D"update_type">Deleted</span>
+    <span class=3D"update_time">%[9]s</span>
+    <span class=3D"edited_new_msg_id"></span>
+</li>
+
+<li class=3D"message">
+    <span class=3D"post_id">%[21]s</span>
+    <span class=3D"sent_time">%[8]s</span>
+    <span class=3D"username">@user1</span>
+    <span class=3D"userid">%[17]s</span>
+    <span class=3D"postusername">@user1</span>
+    <span class=3D"usertype">user</span>
+    <span class=3D"email">(user1@email)</span>
+    <span class=3D"message">Deleted file </span>
+    <span class=3D"update_type">FileDeleted</span>
+    <span class=3D"update_time">%[9]s</span>
+    <span class=3D"edited_new_msg_id"></span>
+</li>
+
+<li class=3D"message">
+    <span class=3D"post_id">%[23]s</span>
+    <span class=3D"sent_time">%[11]s</span>
+    <span class=3D"username">@user1</span>
+    <span class=3D"userid">%[17]s</span>
+    <span class=3D"postusername">@user1</span>
+    <span class=3D"usertype">user</span>
+    <span class=3D"email">(user1@email)</span>
+    <span class=3D"message">edited message 4</span>
+    <span class=3D"update_type">EditedNewMsg</span>
+    <span class=3D"update_time">%[28]s</span>
+    <span class=3D"edited_new_msg_id"></span>
+</li>
+
+<li class=3D"message">
+    <span class=3D"post_id">%[22]s</span>
+    <span class=3D"sent_time">%[10]s</span>
+    <span class=3D"username">@user1</span>
+    <span class=3D"userid">%[17]s</span>
+    <span class=3D"postusername">@user1</span>
+    <span class=3D"usertype">user</span>
+    <span class=3D"email">(user1@email)</span>
+    <span class=3D"message">message 4</span>
+    <span class=3D"update_type">EditedOriginalMsg</span>
+    <span class=3D"update_time">%[28]s</span>
+    <span class=3D"edited_new_msg_id">%[23]s</span>
+</li>
+
+<li class=3D"message">
+    <span class=3D"post_id">%[24]s</span>
+    <span class=3D"sent_time">%[12]s</span>
+    <span class=3D"username">@user1</span>
+    <span class=3D"userid">%[17]s</span>
+    <span class=3D"postusername">@user1</span>
+    <span class=3D"usertype">user</span>
+    <span class=3D"email">(user1@email)</span>
+    <span class=3D"message">message 6</span>
+    <span class=3D"update_type">EditedOriginalMsg</span>
+    <span class=3D"update_time">%[29]s</span>
+    <span class=3D"edited_new_msg_id">%[25]s</span>
+</li>
+
+    </ul>
+</div>
+
+<p>Exported on %[1]s</p>

server/enterprise/message_export/testdata/grE2E3Batch1Perm3.tmpl

@@ -0,0 +1,191 @@
+<h1>Mattermost Compliance Export</h1>
+
+<h2>Conversation Summary</h2>
+<div class=3D"summary-list">
+    <ul>
+        <li><span class=3D"bold">TeamId:</span>%[13]s</li>
+        <li><span class=3D"bold">TeamName:</span>%[14]s</li>
+        <li><span class=3D"bold">TeamDisplayName:</span>%[15]s</li>
+        <li><span class=3D"bold">ChannelId:</span>%[16]s</li>
+        <li><span class=3D"bold">ChannelName:</span>channel_two_name</li>
+        <li><span class=3D"bold">ChannelDisplayName:</span>the Channel Two</li>
+        <li><span class=3D"bold">Started:</span>%[1]s</li>
+        <li><span class=3D"bold">Ended:</span>%[2]s</li>
+        <li><span class=3D"bold">Duration:</span>0 seconds</li>
+    </ul>
+</div>
+<table class=3D"participants">
+    <tr>
+        <th class=3D"userid">UserId<br></th>
+        <th class=3D"username">Username<br></th>
+        <th class=3D"usertype">UserType<br></th>
+        <th class=3D"email">Email</th>
+        <th class=3D"joined">Joined</th>
+        <th class=3D"left">Left</th>
+        <th class=3D"duration">Duration</th>
+        <th class=3D"messages">Messages</th>
+    </tr>
+   =20
+<tr>
+    <td class=3D"userid">%[17]s</td>
+    <td class=3D"username">@user1</td>
+    <td class=3D"usertype">user</td>
+    <td class=3D"email">user1@email</td>
+    <td class=3D"joined">%[3]s</td>
+    <td class=3D"left">%[4]s</td>
+    <td class=3D"duration">0 seconds</td>
+    <td class=3D"messages">9</td>
+</tr>
+
+</table>
+
+<h2>Messages</h2>
+<div class=3D"message-list">
+    <ul>
+       =20
+<li class=3D"message">
+    <span class=3D"post_id">%[18]s</span>
+    <span class=3D"sent_time">%[5]s</span>
+    <span class=3D"username">@user1</span>
+    <span class=3D"userid">%[17]s</span>
+    <span class=3D"postusername">@user1</span>
+    <span class=3D"usertype">user</span>
+    <span class=3D"email">(user1@email)</span>
+    <span class=3D"message">message 0</span>
+</li>
+
+<li class=3D"message">
+    <span class=3D"post_id">%[19]s</span>
+    <span class=3D"sent_time">%[6]s</span>
+    <span class=3D"username">@user1</span>
+    <span class=3D"userid">%[17]s</span>
+    <span class=3D"postusername">@user1</span>
+    <span class=3D"usertype">user</span>
+    <span class=3D"email">(user1@email)</span>
+    <span class=3D"message">message 1</span>
+</li>
+
+<li class=3D"message">
+    <span class=3D"post_id">%[19]s</span>
+    <span class=3D"sent_time">%[6]s</span>
+    <span class=3D"username">@user1</span>
+    <span class=3D"userid">%[17]s</span>
+    <span class=3D"postusername">@user1</span>
+    <span class=3D"usertype">user</span>
+    <span class=3D"email">(user1@email)</span>
+    <span class=3D"message">delete message 1</span>
+    <span class=3D"update_type">Deleted</span>
+    <span class=3D"update_time">%[26]s</span>
+    <span class=3D"edited_new_msg_id"></span>
+</li>
+
+<li class=3D"message">
+    <span class=3D"post_id">%[20]s</span>
+    <span class=3D"sent_time">%[7]s</span>
+    <span class=3D"username">@user1</span>
+    <span class=3D"userid">%[17]s</span>
+    <span class=3D"postusername">@user1</span>
+    <span class=3D"usertype">user</span>
+    <span class=3D"email">(user1@email)</span>
+    <span class=3D"message">message 2</span>
+    <span class=3D"update_type">UpdatedNoMsgChange</span>
+    <span class=3D"update_time">%[27]s</span>
+    <span class=3D"edited_new_msg_id"></span>
+</li>
+
+<li class=3D"message">
+    <span class=3D"post_id">%[21]s</span>
+    <span class=3D"sent_time">%[8]s</span>
+    <span class=3D"username">@user1</span>
+    <span class=3D"userid">%[17]s</span>
+    <span class=3D"postusername">@user1</span>
+    <span class=3D"usertype">user</span>
+    <span class=3D"email">(user1@email)</span>
+    <span class=3D"message">message 3</span>
+</li>
+
+<li class=3D"message">
+    <span class=3D"post_id">%[21]s</span>
+    <span class=3D"sent_time">%[8]s</span>
+    <span class=3D"username">@user1</span>
+    <span class=3D"userid">%[17]s</span>
+    <span class=3D"postusername">@user1</span>
+    <span class=3D"usertype">user</span>
+    <span class=3D"email">(user1@email)</span>
+    <span class=3D"message">Uploaded file </span>
+</li>
+
+<li class=3D"message">
+    <span class=3D"post_id">%[21]s</span>
+    <span class=3D"sent_time">%[8]s</span>
+    <span class=3D"username">@user1</span>
+    <span class=3D"userid">%[17]s</span>
+    <span class=3D"postusername">@user1</span>
+    <span class=3D"usertype">user</span>
+    <span class=3D"email">(user1@email)</span>
+    <span class=3D"message">delete message 3</span>
+    <span class=3D"update_type">Deleted</span>
+    <span class=3D"update_time">%[9]s</span>
+    <span class=3D"edited_new_msg_id"></span>
+</li>
+
+<li class=3D"message">
+    <span class=3D"post_id">%[21]s</span>
+    <span class=3D"sent_time">%[8]s</span>
+    <span class=3D"username">@user1</span>
+    <span class=3D"userid">%[17]s</span>
+    <span class=3D"postusername">@user1</span>
+    <span class=3D"usertype">user</span>
+    <span class=3D"email">(user1@email)</span>
+    <span class=3D"message">Deleted file </span>
+    <span class=3D"update_type">FileDeleted</span>
+    <span class=3D"update_time">%[9]s</span>
+    <span class=3D"edited_new_msg_id"></span>
+</li>
+
+<li class=3D"message">
+    <span class=3D"post_id">%[22]s</span>
+    <span class=3D"sent_time">%[10]s</span>
+    <span class=3D"username">@user1</span>
+    <span class=3D"userid">%[17]s</span>
+    <span class=3D"postusername">@user1</span>
+    <span class=3D"usertype">user</span>
+    <span class=3D"email">(user1@email)</span>
+    <span class=3D"message">message 4</span>
+    <span class=3D"update_type">EditedOriginalMsg</span>
+    <span class=3D"update_time">%[28]s</span>
+    <span class=3D"edited_new_msg_id">%[23]s</span>
+</li>
+
+<li class=3D"message">
+    <span class=3D"post_id">%[23]s</span>
+    <span class=3D"sent_time">%[11]s</span>
+    <span class=3D"username">@user1</span>
+    <span class=3D"userid">%[17]s</span>
+    <span class=3D"postusername">@user1</span>
+    <span class=3D"usertype">user</span>
+    <span class=3D"email">(user1@email)</span>
+    <span class=3D"message">edited message 4</span>
+    <span class=3D"update_type">EditedNewMsg</span>
+    <span class=3D"update_time">%[28]s</span>
+    <span class=3D"edited_new_msg_id"></span>
+</li>
+
+<li class=3D"message">
+    <span class=3D"post_id">%[25]s</span>
+    <span class=3D"sent_time">%[12]s</span>
+    <span class=3D"username">@user1</span>
+    <span class=3D"userid">%[17]s</span>
+    <span class=3D"postusername">@user1</span>
+    <span class=3D"usertype">user</span>
+    <span class=3D"email">(user1@email)</span>
+    <span class=3D"message">edited message 6</span>
+    <span class=3D"update_type">EditedNewMsg</span>
+    <span class=3D"update_time">%[29]s</span>
+    <span class=3D"edited_new_msg_id"></span>
+</li>
+
+    </ul>
+</div>
+
+<p>Exported on %[1]s</p>

server/enterprise/message_export/testdata/grE2E3Batch1Perm4.tmpl

@@ -0,0 +1,191 @@
+<h1>Mattermost Compliance Export</h1>
+
+<h2>Conversation Summary</h2>
+<div class=3D"summary-list">
+    <ul>
+        <li><span class=3D"bold">TeamId:</span>%[13]s</li>
+        <li><span class=3D"bold">TeamName:</span>%[14]s</li>
+        <li><span class=3D"bold">TeamDisplayName:</span>%[15]s</li>
+        <li><span class=3D"bold">ChannelId:</span>%[16]s</li>
+        <li><span class=3D"bold">ChannelName:</span>channel_two_name</li>
+        <li><span class=3D"bold">ChannelDisplayName:</span>the Channel Two</li>
+        <li><span class=3D"bold">Started:</span>%[1]s</li>
+        <li><span class=3D"bold">Ended:</span>%[2]s</li>
+        <li><span class=3D"bold">Duration:</span>0 seconds</li>
+    </ul>
+</div>
+<table class=3D"participants">
+    <tr>
+        <th class=3D"userid">UserId<br></th>
+        <th class=3D"username">Username<br></th>
+        <th class=3D"usertype">UserType<br></th>
+        <th class=3D"email">Email</th>
+        <th class=3D"joined">Joined</th>
+        <th class=3D"left">Left</th>
+        <th class=3D"duration">Duration</th>
+        <th class=3D"messages">Messages</th>
+    </tr>
+   =20
+<tr>
+    <td class=3D"userid">%[17]s</td>
+    <td class=3D"username">@user1</td>
+    <td class=3D"usertype">user</td>
+    <td class=3D"email">user1@email</td>
+    <td class=3D"joined">%[3]s</td>
+    <td class=3D"left">%[4]s</td>
+    <td class=3D"duration">0 seconds</td>
+    <td class=3D"messages">9</td>
+</tr>
+
+</table>
+
+<h2>Messages</h2>
+<div class=3D"message-list">
+    <ul>
+       =20
+<li class=3D"message">
+    <span class=3D"post_id">%[18]s</span>
+    <span class=3D"sent_time">%[5]s</span>
+    <span class=3D"username">@user1</span>
+    <span class=3D"userid">%[17]s</span>
+    <span class=3D"postusername">@user1</span>
+    <span class=3D"usertype">user</span>
+    <span class=3D"email">(user1@email)</span>
+    <span class=3D"message">message 0</span>
+</li>
+
+<li class=3D"message">
+    <span class=3D"post_id">%[19]s</span>
+    <span class=3D"sent_time">%[6]s</span>
+    <span class=3D"username">@user1</span>
+    <span class=3D"userid">%[17]s</span>
+    <span class=3D"postusername">@user1</span>
+    <span class=3D"usertype">user</span>
+    <span class=3D"email">(user1@email)</span>
+    <span class=3D"message">message 1</span>
+</li>
+
+<li class=3D"message">
+    <span class=3D"post_id">%[19]s</span>
+    <span class=3D"sent_time">%[6]s</span>
+    <span class=3D"username">@user1</span>
+    <span class=3D"userid">%[17]s</span>
+    <span class=3D"postusername">@user1</span>
+    <span class=3D"usertype">user</span>
+    <span class=3D"email">(user1@email)</span>
+    <span class=3D"message">delete message 1</span>
+    <span class=3D"update_type">Deleted</span>
+    <span class=3D"update_time">%[26]s</span>
+    <span class=3D"edited_new_msg_id"></span>
+</li>
+
+<li class=3D"message">
+    <span class=3D"post_id">%[20]s</span>
+    <span class=3D"sent_time">%[7]s</span>
+    <span class=3D"username">@user1</span>
+    <span class=3D"userid">%[17]s</span>
+    <span class=3D"postusername">@user1</span>
+    <span class=3D"usertype">user</span>
+    <span class=3D"email">(user1@email)</span>
+    <span class=3D"message">message 2</span>
+    <span class=3D"update_type">UpdatedNoMsgChange</span>
+    <span class=3D"update_time">%[27]s</span>
+    <span class=3D"edited_new_msg_id"></span>
+</li>
+
+<li class=3D"message">
+    <span class=3D"post_id">%[21]s</span>
+    <span class=3D"sent_time">%[8]s</span>
+    <span class=3D"username">@user1</span>
+    <span class=3D"userid">%[17]s</span>
+    <span class=3D"postusername">@user1</span>
+    <span class=3D"usertype">user</span>
+    <span class=3D"email">(user1@email)</span>
+    <span class=3D"message">message 3</span>
+</li>
+
+<li class=3D"message">
+    <span class=3D"post_id">%[21]s</span>
+    <span class=3D"sent_time">%[8]s</span>
+    <span class=3D"username">@user1</span>
+    <span class=3D"userid">%[17]s</span>
+    <span class=3D"postusername">@user1</span>
+    <span class=3D"usertype">user</span>
+    <span class=3D"email">(user1@email)</span>
+    <span class=3D"message">Uploaded file </span>
+</li>
+
+<li class=3D"message">
+    <span class=3D"post_id">%[21]s</span>
+    <span class=3D"sent_time">%[8]s</span>
+    <span class=3D"username">@user1</span>
+    <span class=3D"userid">%[17]s</span>
+    <span class=3D"postusername">@user1</span>
+    <span class=3D"usertype">user</span>
+    <span class=3D"email">(user1@email)</span>
+    <span class=3D"message">delete message 3</span>
+    <span class=3D"update_type">Deleted</span>
+    <span class=3D"update_time">%[9]s</span>
+    <span class=3D"edited_new_msg_id"></span>
+</li>
+
+<li class=3D"message">
+    <span class=3D"post_id">%[21]s</span>
+    <span class=3D"sent_time">%[8]s</span>
+    <span class=3D"username">@user1</span>
+    <span class=3D"userid">%[17]s</span>
+    <span class=3D"postusername">@user1</span>
+    <span class=3D"usertype">user</span>
+    <span class=3D"email">(user1@email)</span>
+    <span class=3D"message">Deleted file </span>
+    <span class=3D"update_type">FileDeleted</span>
+    <span class=3D"update_time">%[9]s</span>
+    <span class=3D"edited_new_msg_id"></span>
+</li>
+
+<li class=3D"message">
+    <span class=3D"post_id">%[23]s</span>
+    <span class=3D"sent_time">%[11]s</span>
+    <span class=3D"username">@user1</span>
+    <span class=3D"userid">%[17]s</span>
+    <span class=3D"postusername">@user1</span>
+    <span class=3D"usertype">user</span>
+    <span class=3D"email">(user1@email)</span>
+    <span class=3D"message">edited message 4</span>
+    <span class=3D"update_type">EditedNewMsg</span>
+    <span class=3D"update_time">%[28]s</span>
+    <span class=3D"edited_new_msg_id"></span>
+</li>
+
+<li class=3D"message">
+    <span class=3D"post_id">%[22]s</span>
+    <span class=3D"sent_time">%[10]s</span>
+    <span class=3D"username">@user1</span>
+    <span class=3D"userid">%[17]s</span>
+    <span class=3D"postusername">@user1</span>
+    <span class=3D"usertype">user</span>
+    <span class=3D"email">(user1@email)</span>
+    <span class=3D"message">message 4</span>
+    <span class=3D"update_type">EditedOriginalMsg</span>
+    <span class=3D"update_time">%[28]s</span>
+    <span class=3D"edited_new_msg_id">%[23]s</span>
+</li>
+
+<li class=3D"message">
+    <span class=3D"post_id">%[25]s</span>
+    <span class=3D"sent_time">%[12]s</span>
+    <span class=3D"username">@user1</span>
+    <span class=3D"userid">%[17]s</span>
+    <span class=3D"postusername">@user1</span>
+    <span class=3D"usertype">user</span>
+    <span class=3D"email">(user1@email)</span>
+    <span class=3D"message">edited message 6</span>
+    <span class=3D"update_type">EditedNewMsg</span>
+    <span class=3D"update_time">%[29]s</span>
+    <span class=3D"edited_new_msg_id"></span>
+</li>
+
+    </ul>
+</div>
+
+<p>Exported on %[1]s</p>

server/enterprise/message_export/testdata/grE2E3Batch1SummaryPerm1.tmpl

@@ -0,0 +1,34 @@
+--------------------
+Conversation Summary
+--------------------
+
+* TeamId: %[13]s
+* TeamName: %[14]s
+* TeamDisplayName: %[15]s
+* ChannelId: %[16]s
+* ChannelName: channel_two_name
+* ChannelDisplayName: the Channel Two
+* Started: %[1]s
+* Ended: %[2]s
+* Duration: 0 seconds
+
+UserId
+Username
+UserType
+Email Joined Left Duration Messages %[17]s @user1 user user1@email %[3]s %[4]s 0 seconds 9
+
+--------
+Messages
+--------
+
+* %[18]s %[5]s @user1 %[17]s @user1 user (user1@email) message 0
+* %[19]s %[6]s @user1 %[17]s @user1 user (user1@email) message 1
+* %[19]s %[6]s @user1 %[17]s @user1 user (user1@email) delete message 1 Deleted %[26]s
+* %[20]s %[7]s @user1 %[17]s @user1 user (user1@email) message 2 UpdatedNoMsgChange %[27]s
+* %[21]s %[8]s @user1 %[17]s @user1 user (user1@email) message 3
+* %[21]s %[8]s @user1 %[17]s @user1 user (user1@email) Uploaded file
+* %[21]s %[8]s @user1 %[17]s @user1 user (user1@email) delete message 3 Deleted %[9]s
+* %[21]s %[8]s @user1 %[17]s @user1 user (user1@email) Deleted file FileDeleted %[9]s
+* %[22]s %[10]s @user1 %[17]s @user1 user (user1@email) message 4 EditedOriginalMsg %[28]s %[23]s
+* %[23]s %[11]s @user1 %[17]s @user1 user (user1@email) edited message 4 EditedNewMsg %[28]s
+* %[24]s %[12]s @user1 %[17]s @user1 user (user1@email) message 6 EditedOriginalMsg %[29]s %[25]s

server/enterprise/message_export/testdata/grE2E3Batch1SummaryPerm2.tmpl

@@ -0,0 +1,34 @@
+--------------------
+Conversation Summary
+--------------------
+
+* TeamId: %[13]s
+* TeamName: %[14]s
+* TeamDisplayName: %[15]s
+* ChannelId: %[16]s
+* ChannelName: channel_two_name
+* ChannelDisplayName: the Channel Two
+* Started: %[1]s
+* Ended: %[2]s
+* Duration: 0 seconds
+
+UserId
+Username
+UserType
+Email Joined Left Duration Messages %[17]s @user1 user user1@email %[3]s %[4]s 0 seconds 9
+
+--------
+Messages
+--------
+
+* %[18]s %[5]s @user1 %[17]s @user1 user (user1@email) message 0
+* %[19]s %[6]s @user1 %[17]s @user1 user (user1@email) message 1
+* %[19]s %[6]s @user1 %[17]s @user1 user (user1@email) delete message 1 Deleted %[26]s
+* %[20]s %[7]s @user1 %[17]s @user1 user (user1@email) message 2 UpdatedNoMsgChange %[27]s
+* %[21]s %[8]s @user1 %[17]s @user1 user (user1@email) message 3
+* %[21]s %[8]s @user1 %[17]s @user1 user (user1@email) Uploaded file
+* %[21]s %[8]s @user1 %[17]s @user1 user (user1@email) delete message 3 Deleted %[9]s
+* %[21]s %[8]s @user1 %[17]s @user1 user (user1@email) Deleted file FileDeleted %[9]s
+* %[23]s %[11]s @user1 %[17]s @user1 user (user1@email) edited message 4 EditedNewMsg %[28]s
+* %[22]s %[10]s @user1 %[17]s @user1 user (user1@email) message 4 EditedOriginalMsg %[28]s %[23]s
+* %[24]s %[12]s @user1 %[17]s @user1 user (user1@email) message 6 EditedOriginalMsg %[29]s %[25]s

server/enterprise/message_export/testdata/grE2E3Batch1SummaryPerm3.tmpl

@@ -0,0 +1,34 @@
+--------------------
+Conversation Summary
+--------------------
+
+* TeamId: %[13]s
+* TeamName: %[14]s
+* TeamDisplayName: %[15]s
+* ChannelId: %[16]s
+* ChannelName: channel_two_name
+* ChannelDisplayName: the Channel Two
+* Started: %[1]s
+* Ended: %[2]s
+* Duration: 0 seconds
+
+UserId
+Username
+UserType
+Email Joined Left Duration Messages %[17]s @user1 user user1@email %[3]s %[4]s 0 seconds 9
+
+--------
+Messages
+--------
+
+* %[18]s %[5]s @user1 %[17]s @user1 user (user1@email) message 0
+* %[19]s %[6]s @user1 %[17]s @user1 user (user1@email) message 1
+* %[19]s %[6]s @user1 %[17]s @user1 user (user1@email) delete message 1 Deleted %[26]s
+* %[20]s %[7]s @user1 %[17]s @user1 user (user1@email) message 2 UpdatedNoMsgChange %[27]s
+* %[21]s %[8]s @user1 %[17]s @user1 user (user1@email) message 3
+* %[21]s %[8]s @user1 %[17]s @user1 user (user1@email) Uploaded file
+* %[21]s %[8]s @user1 %[17]s @user1 user (user1@email) delete message 3 Deleted %[9]s
+* %[21]s %[8]s @user1 %[17]s @user1 user (user1@email) Deleted file FileDeleted %[9]s
+* %[22]s %[10]s @user1 %[17]s @user1 user (user1@email) message 4 EditedOriginalMsg %[28]s %[23]s
+* %[23]s %[11]s @user1 %[17]s @user1 user (user1@email) edited message 4 EditedNewMsg %[28]s
+* %[25]s %[12]s @user1 %[17]s @user1 user (user1@email) edited message 6 EditedNewMsg %[29]s

server/enterprise/message_export/testdata/grE2E3Batch1SummaryPerm4.tmpl

@@ -0,0 +1,34 @@
+--------------------
+Conversation Summary
+--------------------
+
+* TeamId: %[13]s
+* TeamName: %[14]s
+* TeamDisplayName: %[15]s
+* ChannelId: %[16]s
+* ChannelName: channel_two_name
+* ChannelDisplayName: the Channel Two
+* Started: %[1]s
+* Ended: %[2]s
+* Duration: 0 seconds
+
+UserId
+Username
+UserType
+Email Joined Left Duration Messages %[17]s @user1 user user1@email %[3]s %[4]s 0 seconds 9
+
+--------
+Messages
+--------
+
+* %[18]s %[5]s @user1 %[17]s @user1 user (user1@email) message 0
+* %[19]s %[6]s @user1 %[17]s @user1 user (user1@email) message 1
+* %[19]s %[6]s @user1 %[17]s @user1 user (user1@email) delete message 1 Deleted %[26]s
+* %[20]s %[7]s @user1 %[17]s @user1 user (user1@email) message 2 UpdatedNoMsgChange %[27]s
+* %[21]s %[8]s @user1 %[17]s @user1 user (user1@email) message 3
+* %[21]s %[8]s @user1 %[17]s @user1 user (user1@email) Uploaded file
+* %[21]s %[8]s @user1 %[17]s @user1 user (user1@email) delete message 3 Deleted %[9]s
+* %[21]s %[8]s @user1 %[17]s @user1 user (user1@email) Deleted file FileDeleted %[9]s
+* %[23]s %[11]s @user1 %[17]s @user1 user (user1@email) edited message 4 EditedNewMsg %[28]s
+* %[22]s %[10]s @user1 %[17]s @user1 user (user1@email) message 4 EditedOriginalMsg %[28]s %[23]s
+* %[25]s %[12]s @user1 %[17]s @user1 user (user1@email) edited message 6 EditedNewMsg %[29]s

server/enterprise/message_export/testdata/grE2E3Batch2Perm1.tmpl

@@ -0,0 +1,63 @@
+<h1>Mattermost Compliance Export</h1>
+
+<h2>Conversation Summary</h2>
+<div class=3D"summary-list">
+    <ul>
+        <li><span class=3D"bold">TeamId:</span>%[6]s</li>
+        <li><span class=3D"bold">TeamName:</span>%[7]s</li>
+        <li><span class=3D"bold">TeamDisplayName:</span>%[8]s</li>
+        <li><span class=3D"bold">ChannelId:</span>%[9]s</li>
+        <li><span class=3D"bold">ChannelName:</span>channel_two_name</li>
+        <li><span class=3D"bold">ChannelDisplayName:</span>the Channel Two</li>
+        <li><span class=3D"bold">Started:</span>%[1]s</li>
+        <li><span class=3D"bold">Ended:</span>%[2]s</li>
+        <li><span class=3D"bold">Duration:</span>0 seconds</li>
+    </ul>
+</div>
+<table class=3D"participants">
+    <tr>
+        <th class=3D"userid">UserId<br></th>
+        <th class=3D"username">Username<br></th>
+        <th class=3D"usertype">UserType<br></th>
+        <th class=3D"email">Email</th>
+        <th class=3D"joined">Joined</th>
+        <th class=3D"left">Left</th>
+        <th class=3D"duration">Duration</th>
+        <th class=3D"messages">Messages</th>
+    </tr>
+   =20
+<tr>
+    <td class=3D"userid">%[10]s</td>
+    <td class=3D"username">@user1</td>
+    <td class=3D"usertype">user</td>
+    <td class=3D"email">user1@email</td>
+    <td class=3D"joined">%[3]s</td>
+    <td class=3D"left">%[4]s</td>
+    <td class=3D"duration">0 seconds</td>
+    <td class=3D"messages">1</td>
+</tr>
+
+</table>
+
+<h2>Messages</h2>
+<div class=3D"message-list">
+    <ul>
+       =20
+<li class=3D"message">
+    <span class=3D"post_id">%[14]s</span>
+    <span class=3D"sent_time">%[5]s</span>
+    <span class=3D"username">@user1</span>
+    <span class=3D"userid">%[10]s</span>
+    <span class=3D"postusername">@user1</span>
+    <span class=3D"usertype">user</span>
+    <span class=3D"email">(user1@email)</span>
+    <span class=3D"message">edited message 6</span>
+    <span class=3D"update_type">EditedNewMsg</span>
+    <span class=3D"update_time">%[12]s</span>
+    <span class=3D"edited_new_msg_id"></span>
+</li>
+
+    </ul>
+</div>
+
+<p>Exported on %[15]s</p>

server/enterprise/message_export/testdata/grE2E3Batch2Perm2.tmpl

@@ -0,0 +1,63 @@
+<h1>Mattermost Compliance Export</h1>
+
+<h2>Conversation Summary</h2>
+<div class=3D"summary-list">
+    <ul>
+        <li><span class=3D"bold">TeamId:</span>%[6]s</li>
+        <li><span class=3D"bold">TeamName:</span>%[7]s</li>
+        <li><span class=3D"bold">TeamDisplayName:</span>%[8]s</li>
+        <li><span class=3D"bold">ChannelId:</span>%[9]s</li>
+        <li><span class=3D"bold">ChannelName:</span>channel_two_name</li>
+        <li><span class=3D"bold">ChannelDisplayName:</span>the Channel Two</li>
+        <li><span class=3D"bold">Started:</span>%[1]s</li>
+        <li><span class=3D"bold">Ended:</span>%[2]s</li>
+        <li><span class=3D"bold">Duration:</span>0 seconds</li>
+    </ul>
+</div>
+<table class=3D"participants">
+    <tr>
+        <th class=3D"userid">UserId<br></th>
+        <th class=3D"username">Username<br></th>
+        <th class=3D"usertype">UserType<br></th>
+        <th class=3D"email">Email</th>
+        <th class=3D"joined">Joined</th>
+        <th class=3D"left">Left</th>
+        <th class=3D"duration">Duration</th>
+        <th class=3D"messages">Messages</th>
+    </tr>
+   =20
+<tr>
+    <td class=3D"userid">%[10]s</td>
+    <td class=3D"username">@user1</td>
+    <td class=3D"usertype">user</td>
+    <td class=3D"email">user1@email</td>
+    <td class=3D"joined">%[3]s</td>
+    <td class=3D"left">%[4]s</td>
+    <td class=3D"duration">0 seconds</td>
+    <td class=3D"messages">1</td>
+</tr>
+
+</table>
+
+<h2>Messages</h2>
+<div class=3D"message-list">
+    <ul>
+       =20
+<li class=3D"message">
+    <span class=3D"post_id">%[13]s</span>
+    <span class=3D"sent_time">%[5]s</span>
+    <span class=3D"username">@user1</span>
+    <span class=3D"userid">%[10]s</span>
+    <span class=3D"postusername">@user1</span>
+    <span class=3D"usertype">user</span>
+    <span class=3D"email">(user1@email)</span>
+    <span class=3D"message">message 6</span>
+    <span class=3D"update_type">EditedOriginalMsg</span>
+    <span class=3D"update_time">%[12]s</span>
+    <span class=3D"edited_new_msg_id">%[14]s</span>
+</li>
+
+    </ul>
+</div>
+
+<p>Exported on %[15]s</p>

server/enterprise/message_export/testdata/grE2E3Batch2SummaryPerm1.tmpl

@@ -0,0 +1,24 @@
+--------------------
+Conversation Summary
+--------------------
+
+* TeamId: %[6]s
+* TeamName: %[7]s
+* TeamDisplayName: %[8]s
+* ChannelId: %[9]s
+* ChannelName: channel_two_name
+* ChannelDisplayName: the Channel Two
+* Started: %[1]s
+* Ended: %[2]s
+* Duration: 0 seconds
+
+UserId
+Username
+UserType
+Email Joined Left Duration Messages %[10]s @user1 user user1@email %[3]s %[4]s 0 seconds 1
+
+--------
+Messages
+--------
+
+* %[14]s %[5]s @user1 %[10]s @user1 user (user1@email) edited message 6 EditedNewMsg %[12]s

server/enterprise/message_export/testdata/grE2E3Batch2SummaryPerm2.tmpl

@@ -0,0 +1,24 @@
+--------------------
+Conversation Summary
+--------------------
+
+* TeamId: %[6]s
+* TeamName: %[7]s
+* TeamDisplayName: %[8]s
+* ChannelId: %[9]s
+* ChannelName: channel_two_name
+* ChannelDisplayName: the Channel Two
+* Started: %[1]s
+* Ended: %[2]s
+* Duration: 0 seconds
+
+UserId
+Username
+UserType
+Email Joined Left Duration Messages %[10]s @user1 user user1@email %[3]s %[4]s 0 seconds 1
+
+--------
+Messages
+--------
+
+* %[13]s %[5]s @user1 %[10]s @user1 user (user1@email) message 6 EditedOriginalMsg %[12]s %[14]s

server/enterprise/message_export/testdata/grE2E4Summary.tmpl

@@ -0,0 +1,18 @@
+--------------------
+Conversation Summary
+--------------------
+
+* TeamId: %[5]s
+* TeamName: %[6]s
+* TeamDisplayName: %[7]s
+* ChannelId: %[8]s
+* ChannelName: channel_two_name
+* ChannelDisplayName: the Channel Two
+* Started: %[1]s
+* Ended: %[2]s
+* Duration: 0 seconds
+
+UserId
+Username
+UserType
+Email Joined Left Duration Messages %[9]s @user1 user user1@email %[3]s %[4]s 0 seconds 5

server/enterprise/message_export/worker.go

@@ -4,13 +4,10 @@
 package message_export
 
 import (
-	"archive/zip"
 	"context"
-	"encoding/json"
+	"errors"
 	"fmt"
-	"io"
 	"net/http"
-	"os"
 	"path"
 	"strconv"
 	"sync"
@@ -21,28 +18,14 @@ import (
 	"github.com/mattermost/mattermost/server/public/shared/request"
 	"github.com/mattermost/mattermost/server/v8/channels/jobs"
 	"github.com/mattermost/mattermost/server/v8/channels/utils/fileutils"
-	"github.com/mattermost/mattermost/server/v8/platform/shared/filestore"
+	"github.com/mattermost/mattermost/server/v8/enterprise/message_export/shared"
 	"github.com/mattermost/mattermost/server/v8/platform/shared/templates"
 )
 
-const (
-	JobDataBatchStartTimestamp = "batch_start_timestamp" // message export uses keyset pagination sorted by (posts.updateat, posts.id). batch_start_timestamp is the posts.updateat value from the previous batch.
-	JobDataBatchStartId        = "batch_start_id"        // message export uses keyset pagination sorted by (posts.updateat, posts.id). batch_start_id is the posts.id value from the previous batch.
+const TimeBetweenBatchesMs = 100
 
-	JobDataStartTimestamp   = "start_timestamp"
-	JobDataStartId          = "start_id"
-	JobDataExportType       = "export_type"
-	JOB_DATA_BatchSize      = "batch_size"
-	JobDataMessagesExported = "messages_exported"
-	JobDataWarningCount     = "warning_count"
-	JobDataIsDownloadable   = "is_downloadable"
-	JobDirectories          = "job_directories"
-	TimeBetweenBatches      = 100
-
-	estimatedPostCount = 10_000_000
-)
-
-const exportPath = "export"
+// testEndOfBatchCb is only used for testing
+var testEndOfBatchCb func(worker *MessageExportWorker)
 
 type MessageExportWorker struct {
 	name string
@@ -96,106 +79,70 @@ func (dr *MessageExportJobInterfaceImpl) MakeWorker() model.Worker {
 	}
 }
 
-func (worker *MessageExportWorker) IsEnabled(cfg *model.Config) bool {
-	return worker.license() != nil && *worker.license().Features.MessageExport && *cfg.MessageExportSettings.EnableExport
+func (w *MessageExportWorker) IsEnabled(cfg *model.Config) bool {
+	return w.license() != nil && *w.license().Features.MessageExport && *cfg.MessageExportSettings.EnableExport
 }
 
-func (worker *MessageExportWorker) Run() {
-	worker.stateMut.Lock()
+func (w *MessageExportWorker) Run() {
+	w.stateMut.Lock()
 	// We have to re-assign the stop channel again, because
 	// it might happen that the job was restarted due to a config change.
-	if worker.stopped {
-		worker.stopped = false
-		worker.stopCh = make(chan struct{})
-		worker.context, worker.cancel = context.WithCancel(context.Background())
+	if w.stopped {
+		w.stopped = false
+		w.stopCh = make(chan struct{})
+		w.context, w.cancel = context.WithCancel(context.Background())
 	} else {
-		worker.stateMut.Unlock()
+		w.stateMut.Unlock()
 		return
 	}
 	// Run is called from a separate goroutine and doesn't return.
 	// So we cannot Unlock in a defer clause.
-	worker.stateMut.Unlock()
+	w.stateMut.Unlock()
 
-	worker.logger.Debug("Worker Started")
+	w.logger.Debug("Worker Started")
 
 	defer func() {
-		worker.logger.Debug("Worker finished")
-		worker.stoppedCh <- struct{}{}
+		w.logger.Debug("Worker finished")
+		w.stoppedCh <- struct{}{}
 	}()
 
 	for {
 		select {
-		case <-worker.stopCh:
-			worker.logger.Debug("Worker: Received stop signal")
+		case <-w.stopCh:
+			w.logger.Debug("Worker: Received stop signal")
 			return
-		case job := <-worker.jobs:
-			worker.DoJob(&job)
+		case job := <-w.jobs:
+			w.DoJob(&job)
 		}
 	}
 }
 
-func (worker *MessageExportWorker) Stop() {
-	worker.stateMut.Lock()
-	defer worker.stateMut.Unlock()
+func (w *MessageExportWorker) Stop() {
+	w.stateMut.Lock()
+	defer w.stateMut.Unlock()
 
 	// Set to close, and if already closed before, then return.
-	if worker.stopped {
+	if w.stopped {
 		return
 	}
-	worker.stopped = true
+	w.stopped = true
 
-	worker.logger.Debug("Worker: Stopping")
-	worker.cancel()
-	close(worker.stopCh)
-	<-worker.stoppedCh
+	w.logger.Debug("Worker: Stopping")
+	w.cancel()
+	close(w.stopCh)
+	<-w.stoppedCh
 }
 
-func (worker *MessageExportWorker) JobChannel() chan<- model.Job {
-	return worker.jobs
+func (w *MessageExportWorker) JobChannel() chan<- model.Job {
+	return w.jobs
 }
 
-// getExportBackend returns the file backend where the export will be created.
-func (worker *MessageExportWorker) getExportBackend(rctx request.CTX) (filestore.FileBackend, *model.AppError) {
-	config := worker.jobServer.Config()
-	insecure := config.ServiceSettings.EnableInsecureOutgoingConnections
-
-	if config.FileSettings.DedicatedExportStore != nil && *config.FileSettings.DedicatedExportStore {
-		rctx.Logger().Debug("Worker: using dedicated export filestore", mlog.String("driver_name", *config.FileSettings.ExportDriverName))
-		backend, errFileBack := filestore.NewExportFileBackend(filestore.NewExportFileBackendSettingsFromConfig(&config.FileSettings, true, insecure != nil && *insecure))
-		if errFileBack != nil {
-			return nil, model.NewAppError("getFileBackend", "api.file.no_driver.app_error", nil, "", http.StatusInternalServerError).Wrap(errFileBack)
-		}
-
-		return backend, nil
-	}
-
-	backend, err := filestore.NewFileBackend(filestore.NewFileBackendSettingsFromConfig(&config.FileSettings, true, insecure != nil && *insecure))
-	if err != nil {
-		return nil, model.NewAppError("getFileBackend", "api.file.no_driver.app_error", nil, "", http.StatusInternalServerError).Wrap(err)
-	}
-	return backend, nil
-}
-
-// getFileAttachmentBackend returns the file backend where file attachments are
-// located for messages that will be exported. This may be the same backend
-// where the export will be created.
-func (worker *MessageExportWorker) getFileAttachmentBackend(rctx request.CTX) (filestore.FileBackend, *model.AppError) {
-	config := worker.jobServer.Config()
-	insecure := config.ServiceSettings.EnableInsecureOutgoingConnections
-
-	backend, err := filestore.NewFileBackend(filestore.NewFileBackendSettingsFromConfig(&config.FileSettings, true, insecure != nil && *insecure))
-	if err != nil {
-		return nil, model.NewAppError("getFileBackend", "api.file.no_driver.app_error", nil, "", http.StatusInternalServerError).Wrap(err)
-	}
-	return backend, nil
-}
-
-func (worker *MessageExportWorker) DoJob(job *model.Job) {
-	logger := worker.logger.With(jobs.JobLoggerFields(job)...)
+func (w *MessageExportWorker) DoJob(job *model.Job) {
+	logger := w.logger.With(jobs.JobLoggerFields(job)...)
 	logger.Debug("Worker: Received a new candidate job.")
-	defer worker.jobServer.HandleJobPanic(logger, job)
+	defer w.jobServer.HandleJobPanic(logger, job)
 
-	claimed, appErr := worker.jobServer.ClaimJob(job)
+	claimed, appErr := w.jobServer.ClaimJob(job)
 	if appErr != nil {
 		logger.Info("Worker: Error occurred while trying to claim job", mlog.Err(appErr))
 		return
@@ -205,299 +152,171 @@ func (worker *MessageExportWorker) DoJob(job *model.Job) {
 		return
 	}
 
-	var cancelContext request.CTX = request.EmptyContext(worker.logger)
+	var cancelContext request.CTX = request.EmptyContext(w.logger)
 	cancelCtx, cancelCancelWatcher := context.WithCancel(context.Background())
 	cancelWatcherChan := make(chan struct{}, 1)
 	cancelContext = cancelContext.WithContext(cancelCtx)
-	go worker.jobServer.CancellationWatcher(cancelContext, job.Id, cancelWatcherChan)
+	go w.jobServer.CancellationWatcher(cancelContext, job.Id, cancelWatcherChan)
 	defer cancelCancelWatcher()
 
 	// if job data is missing, we'll do our best to recover
-	worker.initJobData(logger, job)
-
-	// the initJobData call above populates the create_at timestamp of the first post that we should export
-	// incase of job resumption or new job
-	batchStartTime, err := strconv.ParseInt(job.Data[JobDataBatchStartTimestamp], 10, 64)
+	w.initJobData(logger, job, time.Now())
+	data, err := extractJobData(logger, job.Data)
 	if err != nil {
-		worker.setJobError(logger, job, model.NewAppError("Job.DoJob", model.NoTranslation, nil, "", http.StatusBadRequest).Wrap((err)))
-		return
-	}
-	batchStartId := job.Data[JobDataBatchStartId]
-
-	jobStartTime, err := strconv.ParseInt(job.Data[JobDataStartTimestamp], 10, 64)
-	if err != nil {
-		worker.setJobError(logger, job, model.NewAppError("Job.DoJob", model.NoTranslation, nil, "", http.StatusBadRequest).Wrap((err)))
-		return
-	}
-	jobStartId := job.Data[JobDataStartId]
-
-	batchSize, err := strconv.Atoi(job.Data[JOB_DATA_BatchSize])
-	if err != nil {
-		worker.setJobError(logger, job, model.NewAppError("Job.DoJob", model.NoTranslation, nil, "", http.StatusBadRequest).Wrap((err)))
+		// Error in conversion. Not much we can do about that. But it shouldn't happen, unless someone edited the db.
+		w.setJobError(logger, job, model.NewAppError("Job.DoJob", "ent.message_export.job_data_conversion.app_error", nil, "", http.StatusBadRequest).Wrap(err))
 		return
 	}
 
-	totalPostsExported, err := strconv.ParseInt(job.Data[JobDataMessagesExported], 10, 64)
+	rctx := request.EmptyContext(logger).WithContext(w.context)
+	reportProgress := func(message string) {
+		logger.Debug(message)
+		// Don't fail because we couldn't update progress.
+		w.setJobProgressMessage(0, message, rctx.Logger(), job)
+	}
+
+	jobParams := shared.BackendParams{
+		Config:        w.jobServer.Config(),
+		Store:         shared.NewMessageExportStore(w.jobServer.Store),
+		HtmlTemplates: w.htmlTemplateWatcher,
+	}
+	jobParams.FileAttachmentBackend, err = shared.GetFileAttachmentBackend(rctx, w.jobServer.Config())
 	if err != nil {
-		worker.setJobError(logger, job, model.NewAppError("Job.DoJob", model.NoTranslation, nil, "", http.StatusBadRequest).Wrap((err)))
+		w.setJobError(logger, job, model.NewAppError("GetFileAttachmentBackend", "api.file.no_driver.app_error", nil, "", http.StatusInternalServerError).Wrap(err))
+		return
+	}
+	jobParams.ExportBackend, err = shared.GetExportBackend(rctx, w.jobServer.Config())
+	if err != nil {
+		w.setJobError(logger, job, model.NewAppError("GetExportBackend", "api.file.no_driver.app_error", nil, "", http.StatusInternalServerError).Wrap(err))
 		return
 	}
 
-	var directories []string
-	err = json.Unmarshal([]byte(job.Data[JobDirectories]), &directories)
+	data, err = shared.GetInitialExportPeriodData(rctx, jobParams.Store, data, reportProgress)
 	if err != nil {
-		worker.setJobError(logger, job, model.NewAppError("Job.DoJob", model.NoTranslation, nil, "", http.StatusBadRequest).Wrap((err)))
+		w.setJobError(logger, job, model.NewAppError("DoJob", "ent.message_export.calculate_channel_exports.app_error", nil, "", http.StatusInternalServerError).Wrap(err))
 		return
 	}
+	job.Data[shared.JobDataTotalPostsExpected] = strconv.Itoa(data.TotalPostsExpected)
 
-	// Counting all posts may fail or timeout when the posts table is large. If this happens, log a warning, but carry
-	// on with the job anyway. The only issue is that the progress % reporting will be inaccurate.
-	var totalPosts int64
-	if count, err := worker.jobServer.Store.Post().AnalyticsPostCount(&model.PostCountOptions{ExcludeSystemPosts: true, SincePostID: jobStartId, SinceUpdateAt: jobStartTime}); err != nil {
-		logger.Warn("Worker: Failed to fetch total post count for job. An estimated value will be used for progress reporting.", mlog.Err(err))
-		totalPosts = estimatedPostCount
-	} else {
-		totalPosts = count
-	}
-
-	var totalWarningCount int64
-	cursor := model.MessageExportCursor{LastPostUpdateAt: batchStartTime, LastPostId: batchStartId}
 	for {
 		select {
 		case <-cancelWatcherChan:
 			logger.Debug("Worker: Job has been canceled via CancellationWatcher")
-			worker.setJobCanceled(logger, job)
+			w.setJobCanceled(logger, job)
 			return
 
-		case <-worker.stopCh:
+		case <-w.stopCh:
 			logger.Debug("Worker: Job has been canceled via Worker Stop. Setting the job back to pending")
-			worker.SetJobPending(logger, job)
+			w.SetJobPending(logger, job)
 			return
 
-		case <-time.After(TimeBetweenBatches * time.Millisecond):
-			logger.Debug("Starting batch export", mlog.Int("last_post_update_at", cursor.LastPostUpdateAt))
-			rctx := request.EmptyContext(logger).WithContext(worker.context)
-			prevPostUpdateAt := cursor.LastPostUpdateAt
+		case <-time.After(TimeBetweenBatchesMs * time.Millisecond):
+			logger.Debug("Starting batch export", mlog.Int("last_post_update_at", data.Cursor.LastPostUpdateAt))
 
-			var postsExported []*model.MessageExport
-			var nErr error
-			postsExported, cursor, nErr = worker.jobServer.Store.Compliance().MessageExport(rctx, cursor, batchSize)
-			if nErr != nil {
+			_, data, err = RunBatch(rctx, data, jobParams)
+			if err != nil {
 				// We ignore error if the job was explicitly cancelled
-				// and let it
-				if worker.context.Err() == context.Canceled {
+				if errors.Is(w.context.Err(), context.Canceled) {
 					logger.Debug("Worker: Job has been canceled via worker's context. Setting the job back to pending")
-					worker.SetJobPending(logger, job)
+					w.SetJobPending(logger, job)
 				} else {
-					worker.setJobError(logger, job, model.NewAppError("DoJob", "ent.message_export.run_export.app_error", nil, "", http.StatusInternalServerError).Wrap(nErr))
-				}
-				return
-			}
-			logger.Debug("Found posts to export", mlog.Int("number_of_posts", len(postsExported)))
-			totalPostsExported += int64(len(postsExported))
-			job.Data[JobDataMessagesExported] = strconv.FormatInt(totalPostsExported, 10)
-			job.Data[JobDataBatchStartTimestamp] = strconv.FormatInt(cursor.LastPostUpdateAt, 10)
-			job.Data[JobDataBatchStartId] = cursor.LastPostId
-
-			if len(postsExported) == 0 {
-				job.Data[JobDataWarningCount] = strconv.FormatInt(totalWarningCount, 10)
-				// we've exported everything up to the current time
-				logger.Debug("FormatExport complete")
-
-				// Create downloadable zip file of all batches.
-				if job.Data[JobDataExportType] != model.ComplianceExportTypeGlobalrelay {
-					exportBackend, err := worker.getExportBackend(rctx)
-					if err != nil {
-						worker.setJobError(logger, job, err)
-						return
-					}
-
-					zipErr := createZipFile(rctx, exportBackend, job.Id, directories)
-					if zipErr != nil {
-						logger.Error("Error creating zip file for export", mlog.Err(zipErr))
-						job.Data[JobDataIsDownloadable] = "false"
-					} else {
-						job.Data[JobDataIsDownloadable] = "true"
-					}
-				}
-				if totalWarningCount > 0 {
-					worker.setJobWarning(logger, job)
-				} else {
-					worker.setJobSuccess(logger, job)
+					w.setJobError(logger, job, model.NewAppError("DoJob", "ent.message_export.run_export.app_error", nil, "", http.StatusInternalServerError).Wrap(err))
 				}
 				return
 			}
 
-			exportBackend, err := worker.getExportBackend(rctx)
-			if err != nil {
-				worker.setJobError(logger, job, err)
+			setJobDataEndOfBatch(job, data)
+
+			if data.Finished {
+				w.finishExport(rctx, logger, job, data.WarningCount)
 				return
 			}
 
-			fileAttachmentBackend, err := worker.getFileAttachmentBackend(rctx)
-			if err != nil {
-				worker.setJobError(logger, job, err)
+			// also saves job.Data
+			if err := w.setJobProgress(logger, job, getJobProgress(data.MessagesExported, data.TotalPostsExpected)); err != nil {
+				// TODO: MM-59093 handle job errors (robust, recoverable)
 				return
 			}
 
-			batchDirectory := getOutputDirectoryPath(prevPostUpdateAt, cursor.LastPostUpdateAt)
-			warningCount, err := runExportByType(
-				rctx,
-				job.Data[JobDataExportType],
-				postsExported,
-				batchDirectory,
-				worker.jobServer.Store,
-				exportBackend,
-				fileAttachmentBackend,
-				worker.htmlTemplateWatcher,
-				worker.jobServer.Config(),
-			)
-			if err != nil {
-				worker.setJobError(logger, job, err)
-				return
-			}
-
-			totalWarningCount += warningCount
-
-			directories = append(directories, batchDirectory)
-			directoriesBytes, e := json.Marshal(directories)
-			if e != nil {
-				worker.setJobError(logger, job, model.NewAppError("Job.DoJob", model.NoTranslation, nil, "", http.StatusInternalServerError).Wrap((e)))
-				return
-			}
-			job.Data[JobDirectories] = string(directoriesBytes)
-
-			// also saves the last post create time
-			if err := worker.jobServer.SetJobProgress(job, getJobProgress(totalPostsExported, totalPosts)); err != nil {
-				worker.setJobError(logger, job, err)
-				return
+			// testEndOfBatchCb is only used by tests.
+			if testEndOfBatchCb != nil {
+				testEndOfBatchCb(w)
 			}
 		}
 	}
 }
 
-func createZipFile(rctx request.CTX, fileBackend filestore.FileBackend, jobId string, directories []string) error {
-	zipFileName := jobId + ".zip"
+func (w *MessageExportWorker) finishExport(rctx request.CTX, logger *mlog.Logger, job *model.Job, totalWarningCount int) {
+	job.Data[shared.JobDataWarningCount] = strconv.Itoa(totalWarningCount)
+	// we've exported everything up to the current time
+	logger.Debug("FormatExport complete")
 
-	dest, err := os.CreateTemp("", zipFileName)
-	if err != nil {
-		return err
+	job.Data[shared.JobDataIsDownloadable] = "false"
+
+	if totalWarningCount > 0 {
+		w.setJobWarning(logger, job)
+	} else {
+		w.setJobSuccess(logger, job)
 	}
-	defer os.Remove(dest.Name())
-
-	// Create a new zip archive.
-	w := zip.NewWriter(dest)
-
-	// create a 32 KiB buffer for copying files
-	buf := make([]byte, 32*1024)
-
-	// Add directories to the archive.
-	for _, directory := range directories {
-		err = addFiles(w, fileBackend, directory, buf)
-		if err != nil {
-			return err
-		}
-	}
-
-	// Make sure to check the error on Close.
-	err = w.Close()
-	if err != nil {
-		return fmt.Errorf("error closing zip file: %s %v", dest.Name(), err)
-	}
-
-	_, err = dest.Seek(0, 0)
-	if err != nil {
-		return fmt.Errorf("error seeking zip file: %s %v", dest.Name(), err)
-	}
-
-	zipPath := path.Join(exportPath, zipFileName)
-
-	// If the file backend allows it, we want to upload without a timeout
-	_, err = filestore.TryWriteFileContext(rctx.Context(), fileBackend, dest, zipPath)
-	return err
-}
-
-func addFiles(w *zip.Writer, fileBackend filestore.FileBackend, basePath string, buf []byte) error {
-	// Open the Directory
-	files, err := fileBackend.ListDirectoryRecursively(basePath)
-	if err != nil {
-		return err
-	}
-
-	for _, file := range files {
-		err = addFile(w, fileBackend, file, basePath, buf)
-		if err != nil {
-			return err
-		}
-	}
-
-	return nil
-}
-
-func addFile(w *zip.Writer, fileBackend filestore.FileBackend, file, basePath string, buf []byte) error {
-	// In some storage backends like Hitachi HCP, the first entry
-	// from a ListObjects API is always the dir entry itself.
-	if file == basePath {
-		return nil
-	}
-
-	size, err := fileBackend.FileSize(file)
-	if err != nil {
-		return fmt.Errorf("error reading file size for %s: %w", file, err)
-	}
-	if size == 0 {
-		// skip empty files
-		return nil
-	}
-
-	r, err := fileBackend.Reader(file)
-	if err != nil {
-		return fmt.Errorf("error opening file %s: %w", file, err)
-	}
-	defer r.Close()
-
-	// Add some files to the archive.
-	f, err := w.Create(file)
-	if err != nil {
-		return fmt.Errorf("error creating file %s in the archive: %w", file, err)
-	}
-	_, err = io.CopyBuffer(f, r, buf)
-	if err != nil {
-		return fmt.Errorf("error copying file %s into the archive: %w", file, err)
-	}
-
-	return nil
 }
 
 // initializes job data if it's missing, allows us to recover from failed or improperly configured jobs
-func (worker *MessageExportWorker) initJobData(logger mlog.LoggerIFace, job *model.Job) {
+func (w *MessageExportWorker) initJobData(logger mlog.LoggerIFace, job *model.Job, now time.Time) {
 	if job.Data == nil {
 		job.Data = make(map[string]string)
 	}
-	if _, exists := job.Data[JobDataMessagesExported]; !exists {
-		job.Data[JobDataMessagesExported] = "0"
+	if _, exists := job.Data[shared.JobDataMessagesExported]; !exists {
+		logger.Info("Worker: JobDataMessagesExported does not exist, starting at 0")
+		job.Data[shared.JobDataMessagesExported] = "0"
 	}
-	if _, exists := job.Data[JobDirectories]; !exists {
-		// json null value
-		job.Data[JobDirectories] = "null"
+	if _, exists := job.Data[shared.JobDataExportType]; !exists {
+		exportFormat := *w.jobServer.Config().MessageExportSettings.ExportFormat
+		logger.Info("Worker: Defaulting to configured export format", mlog.String("export_format", exportFormat))
+		job.Data[shared.JobDataExportType] = exportFormat
 	}
-	if _, exists := job.Data[JobDataExportType]; !exists {
-		// for now, we'll default to Actiance. When we support multiple export types, we'll have to fetch it from config instead
-		logger.Info("Worker: Defaulting to configured export format")
-		job.Data[JobDataExportType] = *worker.jobServer.Config().MessageExportSettings.ExportFormat
+	if _, exists := job.Data[shared.JobDataBatchSize]; !exists {
+		batchSize := strconv.Itoa(*w.jobServer.Config().MessageExportSettings.BatchSize)
+		logger.Info("Worker: Defaulting to configured batch size", mlog.String("batch_size", batchSize))
+		job.Data[shared.JobDataBatchSize] = batchSize
 	}
-	if _, exists := job.Data[JOB_DATA_BatchSize]; !exists {
-		logger.Info("Worker: Defaulting to configured batch size")
-		job.Data[JOB_DATA_BatchSize] = strconv.Itoa(*worker.jobServer.Config().MessageExportSettings.BatchSize)
+	if _, exists := job.Data[shared.JobDataChannelBatchSize]; !exists {
+		channelBatchSize := strconv.Itoa(*w.jobServer.Config().MessageExportSettings.ChannelBatchSize)
+		logger.Info("Worker: Defaulting to configured channel batch size", mlog.String("channel_batch_size", channelBatchSize))
+		job.Data[shared.JobDataChannelBatchSize] = channelBatchSize
 	}
-	if _, exists := job.Data[JobDataBatchStartTimestamp]; !exists {
-		previousJob, err := worker.jobServer.Store.Job().GetNewestJobByStatusesAndType([]string{model.JobStatusWarning, model.JobStatusSuccess}, model.JobTypeMessageExport)
+	if _, exists := job.Data[shared.JobDataChannelHistoryBatchSize]; !exists {
+		channelHistoryBatchSize := strconv.Itoa(*w.jobServer.Config().MessageExportSettings.ChannelHistoryBatchSize)
+		logger.Info("Worker: Defaulting to configured channel history batch size", mlog.String("channel_history_batch_size", channelHistoryBatchSize))
+		job.Data[shared.JobDataChannelHistoryBatchSize] = channelHistoryBatchSize
+	}
+	if _, exists := job.Data[shared.JobDataBatchNumber]; !exists {
+		logger.Info("Worker: JobDataBatchNumber does not exist, starting at 0")
+		job.Data[shared.JobDataBatchNumber] = "0"
+	}
+
+	// If this is a new job (JobEndTime doesn't exist), set it to now, because this is when the job has first started.
+	// The logic is that a job exports messages up to the moment the job was started. If the job was picked up after
+	// gracefully stopping, then run it until that original initial endTime.
+	// However, if the job was cancelled or errored out, that job will not be picked up again, so this will be a new job
+	// starting from the last successful batchStartTimestamp up until now. This is intentional (for now) because failed
+	// jobs do not get rescheduled properly yet, and when they are run again it means that new day's worth of messages
+	// need to be exported.
+	if _, exists := job.Data[shared.JobDataJobEndTime]; !exists {
+		millis := strconv.FormatInt(model.GetMillisForTime(now), 10)
+		logger.Info("Worker: JobDataJobEndTime not found in previous job, using now", mlog.String("job_data_job_end_time", millis))
+		job.Data[shared.JobDataJobEndTime] = millis
+	}
+
+	if _, exists := job.Data[shared.JobDataBatchStartTime]; !exists {
+		previousJob, err := w.jobServer.Store.Job().GetNewestJobByStatusesAndType([]string{model.JobStatusWarning, model.JobStatusSuccess}, model.JobTypeMessageExport)
 		if err != nil {
-			logger.Info("Worker: No previously successful job found, falling back to configured MessageExportSettings.ExportFromTimestamp")
-			job.Data[JobDataBatchStartTimestamp] = strconv.FormatInt(*worker.jobServer.Config().MessageExportSettings.ExportFromTimestamp, 10)
-			job.Data[JobDataBatchStartId] = ""
-			job.Data[JobDataStartTimestamp] = job.Data[JobDataBatchStartTimestamp]
-			job.Data[JobDataStartId] = job.Data[JobDataBatchStartId]
+			exportFromTimestamp := strconv.FormatInt(*w.jobServer.Config().MessageExportSettings.ExportFromTimestamp, 10)
+			logger.Info("Worker: No previously successful job found, falling back to configured MessageExportSettings.ExportFromTimestamp", mlog.String("export_from_timestamp", exportFromTimestamp))
+			job.Data[shared.JobDataBatchStartTime] = exportFromTimestamp
+			job.Data[shared.JobDataJobStartTime] = exportFromTimestamp
+			job.Data[shared.JobDataBatchStartId] = ""
+			job.Data[shared.JobDataJobStartId] = job.Data[shared.JobDataBatchStartId]
+			job.Data[shared.JobDataExportDir] = getJobExportDir(logger, job.Data, exportFromTimestamp, job.Data[shared.JobDataJobEndTime])
 			return
 		}
 
@@ -508,72 +327,174 @@ func (worker *MessageExportWorker) initJobData(logger mlog.LoggerIFace, job *mod
 		if previousJob.Data == nil {
 			previousJob.Data = make(map[string]string)
 		}
-		if _, prevExists := previousJob.Data[JobDataBatchStartTimestamp]; !prevExists {
-			logger.Info("Worker: Previously successful job lacks job data, falling back to configured MessageExportSettings.ExportFromTimestamp")
-			job.Data[JobDataBatchStartTimestamp] = strconv.FormatInt(*worker.jobServer.Config().MessageExportSettings.ExportFromTimestamp, 10)
+		if _, prevExists := previousJob.Data[shared.JobDataBatchStartTime]; !prevExists {
+			exportFromTimestamp := strconv.FormatInt(*w.jobServer.Config().MessageExportSettings.ExportFromTimestamp, 10)
+			logger.Info("Worker: Previously successful job lacks job data, falling back to configured MessageExportSettings.ExportFromTimestamp", mlog.String("export_from_timestamp", exportFromTimestamp))
+			job.Data[shared.JobDataBatchStartTime] = exportFromTimestamp
+			job.Data[shared.JobDataJobStartTime] = exportFromTimestamp
 		} else {
-			job.Data[JobDataBatchStartTimestamp] = previousJob.Data[JobDataBatchStartTimestamp]
+			job.Data[shared.JobDataBatchStartTime] = previousJob.Data[shared.JobDataBatchStartTime]
 		}
-		if _, prevExists := previousJob.Data[JobDataBatchStartId]; !prevExists {
+		if _, prevExists := previousJob.Data[shared.JobDataBatchStartId]; !prevExists {
 			logger.Info("Worker: Previously successful job lacks post ID, falling back to empty string")
-			job.Data[JobDataBatchStartId] = ""
+			job.Data[shared.JobDataBatchStartId] = ""
 		} else {
-			job.Data[JobDataBatchStartId] = previousJob.Data[JobDataBatchStartId]
+			job.Data[shared.JobDataBatchStartId] = previousJob.Data[shared.JobDataBatchStartId]
 		}
-		job.Data[JobDataStartTimestamp] = job.Data[JobDataBatchStartTimestamp]
-		job.Data[JobDataStartId] = job.Data[JobDataBatchStartId]
+		job.Data[shared.JobDataJobStartId] = job.Data[shared.JobDataBatchStartId]
 	} else {
-		logger.Info("Worker: FormatExport start time explicitly set", mlog.String("new_start_time", job.Data[JobDataBatchStartTimestamp]))
+		logger.Info("Worker: JobDataBatchStartTime start time was already set",
+			mlog.String(shared.JobDataBatchStartTime, job.Data[shared.JobDataBatchStartTime]))
 	}
+
+	if _, exists := job.Data[shared.JobDataJobStartTime]; !exists {
+		// Just in case, if we don't have this (JobDataBatchStartTime was already set, but this wasn't) set it:
+		job.Data[shared.JobDataJobStartTime] = job.Data[shared.JobDataBatchStartTime]
+		logger.Info("Worker: JobDataJobStartTime start time was not set, using batch startTimestamp",
+			mlog.String(shared.JobDataJobStartTime, job.Data[shared.JobDataJobStartTime]))
+	}
+
+	job.Data[shared.JobDataExportDir] = getJobExportDir(logger, job.Data, job.Data[shared.JobDataJobStartTime], job.Data[shared.JobDataJobEndTime])
 }
 
-func getJobProgress(totalExportedPosts, totalPosts int64) int64 {
-	return totalExportedPosts * 100 / totalPosts
+func extractJobData(logger *mlog.Logger, strmap map[string]string) (shared.JobData, error) {
+	data, err := shared.StringMapToJobDataWithZeroValues(strmap)
+	if err != nil {
+		return data, err
+	}
+
+	// ExportPeriodStartTime is initialized to BatchStartTime because this is where we will start exporting. But unlike
+	// BatchStartTime, it won't change as we process the batches.
+	// If this is the first time this job has run, BatchStartTime will be the start of the entire job. If this job has
+	// been resumed, then BatchStartTime will be the start of the newest batch. This is expected--the channel activity
+	// and total posts will be calculated from ExportPeriodStartTime (anything earlier has already been exported in
+	// previous batches).
+	// Note: ExportPeriodStartTime is different from JobStartTime because JobStartTime won't change
+	//	     if the job processes some batches, is stopped, and picked up again.
+	data.ExportPeriodStartTime = data.BatchStartTime
+
+	logger.Info("Worker: initial job variables set",
+		mlog.String("export_type", data.ExportType),
+		mlog.String("export_dir", data.ExportDir),
+		mlog.Int("job_start_time", data.JobStartTime),
+		mlog.Int("batch_start_time", data.BatchStartTime),
+		mlog.Int("export_period_start_time", data.ExportPeriodStartTime),
+		mlog.Int("job_end_time", data.JobEndTime),
+		mlog.String("job_start_id", data.JobStartId),
+		mlog.Int("batch_size", data.BatchSize),
+		mlog.Int("channel_batch_size", data.ChannelBatchSize),
+		mlog.Int("channel_history_batch_size", data.ChannelHistoryBatchSize),
+		mlog.Int("batch_number", data.BatchNumber),
+		mlog.Int("total_posts_exported", data.MessagesExported))
+
+	return data, err
 }
 
-func (worker *MessageExportWorker) setJobSuccess(logger mlog.LoggerIFace, job *model.Job) {
-	// setting progress causes the job data to be saved, which is necessary if we want the next job to pick up where this one left off
-	if err := worker.jobServer.SetJobProgress(job, 100); err != nil {
+func setJobDataEndOfBatch(job *model.Job, data shared.JobData) {
+	job.Data[shared.JobDataBatchStartTime] = strconv.FormatInt(data.BatchStartTime, 10)
+	job.Data[shared.JobDataBatchStartId] = data.Cursor.LastPostId
+	job.Data[shared.JobDataMessagesExported] = strconv.Itoa(data.MessagesExported)
+	job.Data[shared.JobDataBatchNumber] = strconv.Itoa(data.BatchNumber)
+}
+
+// getJobExportDir will use the existing JobDataExportDir if available. If it's not available, this is the first run
+// for the job, so we use the startTime and endTime passed in.
+func getJobExportDir(logger mlog.LoggerIFace, data model.StringMap, startTime string, endTime string) string {
+	exportDir, exists := data[shared.JobDataExportDir]
+	if !exists {
+		// If we don't have a jobDataExportDir, this is the first run for the job, so we use the batch startTime
+		exportDir = path.Join(model.ComplianceExportPath, fmt.Sprintf("%s-%s-%s", time.Now().Format(model.ComplianceExportDirectoryFormat), startTime, endTime))
+		logger.Info("Worker: JobDataExportDir does not exist, using current datetime", mlog.String("job_data_export_dir", exportDir))
+	}
+
+	return exportDir
+}
+
+func getJobProgress(totalExportedPosts, totalPostsExpected int) int {
+	return totalExportedPosts * 100 / totalPostsExpected
+}
+
+func (w *MessageExportWorker) setJobProgressMessage(progress int64, message string, logger mlog.LoggerIFace, job *model.Job) {
+	job.Status = model.JobStatusInProgress
+	job.Progress = progress
+	if job.Data == nil {
+		job.Data = make(map[string]string)
+	}
+	job.Data["progress_message"] = message
+
+	if _, err := w.jobServer.Store.Job().UpdateOptimistically(job, model.JobStatusInProgress); err != nil {
 		logger.Error("Worker: Failed to update progress for job", mlog.Err(err))
-		worker.setJobError(logger, job, err)
 	}
-	if err := worker.jobServer.SetJobSuccess(job); err != nil {
+}
+
+func (w *MessageExportWorker) setJobProgress(logger mlog.LoggerIFace, job *model.Job, progress int) error {
+	if job.Data != nil {
+		job.Data["progress_message"] = ""
+	}
+
+	if err := w.jobServer.SetJobProgress(job, int64(progress)); err != nil {
+		logger.Error("Worker: Failed to update progress for job", mlog.Err(err))
+		w.setJobError(logger, job, err)
+		return err
+	}
+
+	return nil
+}
+
+func (w *MessageExportWorker) setJobSuccess(logger mlog.LoggerIFace, job *model.Job) {
+	// setting progress causes the job data to be saved, which is necessary if we want the next job to pick up where this one left off
+	if job.Data != nil {
+		job.Data["progress_message"] = ""
+	}
+	if err := w.jobServer.SetJobProgress(job, 100); err != nil {
+		logger.Error("Worker: Failed to update progress for job", mlog.Err(err))
+		w.setJobError(logger, job, err)
+	}
+	if err := w.jobServer.SetJobSuccess(job); err != nil {
 		logger.Error("Worker: Failed to set success for job", mlog.Err(err))
-		worker.setJobError(logger, job, err)
+		w.setJobError(logger, job, err)
 	}
 }
 
-func (worker *MessageExportWorker) setJobWarning(logger mlog.LoggerIFace, job *model.Job) {
+func (w *MessageExportWorker) setJobWarning(logger mlog.LoggerIFace, job *model.Job) {
 	// setting progress causes the job data to be saved, which is necessary if we want the next job to pick up where this one left off
-	if err := worker.jobServer.SetJobProgress(job, 100); err != nil {
+	if job.Data != nil {
+		job.Data["progress_message"] = ""
+	}
+	if err := w.jobServer.SetJobProgress(job, 100); err != nil {
 		logger.Error("Worker: Failed to update progress for job", mlog.Err(err))
-		worker.setJobError(logger, job, err)
+		w.setJobError(logger, job, err)
 	}
-	if err := worker.jobServer.SetJobWarning(job); err != nil {
+	if err := w.jobServer.SetJobWarning(job); err != nil {
 		logger.Error("Worker: Failed to set warning for job", mlog.Err(err))
-		worker.setJobError(logger, job, err)
+		w.setJobError(logger, job, err)
 	}
 }
 
-func (worker *MessageExportWorker) setJobError(logger mlog.LoggerIFace, job *model.Job, appError *model.AppError) {
+func (w *MessageExportWorker) setJobError(logger mlog.LoggerIFace, job *model.Job, appError *model.AppError) {
+	if job.Data != nil {
+		job.Data["progress_message"] = ""
+	}
 	logger.Error("Worker: Job error", mlog.Err(appError))
-	if err := worker.jobServer.SetJobError(job, appError); err != nil {
-		logger.Error("Worker: Failed to set job errorv", mlog.Err(err), mlog.NamedErr("set_error", appError))
+	if err := w.jobServer.SetJobError(job, appError); err != nil {
+		logger.Error("Worker: Failed to set job error", mlog.Err(err), mlog.NamedErr("set_error", appError))
 	}
 }
 
-func (worker *MessageExportWorker) setJobCanceled(logger mlog.LoggerIFace, job *model.Job) {
-	if err := worker.jobServer.SetJobCanceled(job); err != nil {
+func (w *MessageExportWorker) setJobCanceled(logger mlog.LoggerIFace, job *model.Job) {
+	if job.Data != nil {
+		job.Data["progress_message"] = ""
+	}
+	if err := w.jobServer.SetJobCanceled(job); err != nil {
 		logger.Error("Worker: Failed to mark job as canceled", mlog.Err(err))
 	}
 }
 
-func (worker *MessageExportWorker) SetJobPending(logger mlog.LoggerIFace, job *model.Job) {
-	if err := worker.jobServer.SetJobPending(job); err != nil {
+func (w *MessageExportWorker) SetJobPending(logger mlog.LoggerIFace, job *model.Job) {
+	if job.Data != nil {
+		job.Data["progress_message"] = ""
+	}
+	if err := w.jobServer.SetJobPending(job); err != nil {
 		logger.Error("Worker: Failed to mark job as pending", mlog.Err(err))
 	}
 }
-
-func getOutputDirectoryPath(exportStartTime int64, exportEndTime int64) string {
-	return path.Join(exportPath, strconv.FormatInt(exportStartTime, 10)+"-"+strconv.FormatInt(exportEndTime, 10))
-}

server/enterprise/message_export/worker_test.go

@@ -4,14 +4,14 @@
 package message_export
 
 import (
-	"archive/zip"
-	"bytes"
 	"context"
-	"net/http"
+	"errors"
+	"fmt"
 	"os"
 	"path"
 	"strconv"
 	"testing"
+	"time"
 
 	"github.com/stretchr/testify/assert"
 	tmock "github.com/stretchr/testify/mock"
@@ -24,10 +24,10 @@ import (
 	"github.com/mattermost/mattermost/server/v8/channels/app"
 	"github.com/mattermost/mattermost/server/v8/channels/jobs"
 	"github.com/mattermost/mattermost/server/v8/channels/store/storetest"
+	st "github.com/mattermost/mattermost/server/v8/channels/store/storetest"
 	"github.com/mattermost/mattermost/server/v8/channels/utils/testutils"
 	"github.com/mattermost/mattermost/server/v8/einterfaces/mocks"
-	"github.com/mattermost/mattermost/server/v8/platform/shared/filestore"
-	fmocks "github.com/mattermost/mattermost/server/v8/platform/shared/filestore/mocks"
+	"github.com/mattermost/mattermost/server/v8/enterprise/message_export/shared"
 )
 
 func TestInitJobDataNoJobData(t *testing.T) {
@@ -36,14 +36,14 @@ func TestInitJobDataNoJobData(t *testing.T) {
 	defer mockStore.AssertExpectations(t)
 
 	job := &model.Job{
-		Id:       model.NewId(),
+		Id:       st.NewTestID(),
 		CreateAt: model.GetMillis(),
 		Status:   model.JobStatusPending,
 		Type:     model.JobTypeMessageExport,
 	}
 
 	// mock job store doesn't return a previously successful job, forcing fallback to config
-	mockStore.JobStore.On("GetNewestJobByStatusesAndType", []string{model.JobStatusWarning, model.JobStatusSuccess}, model.JobTypeMessageExport).Return(nil, model.NewAppError("", "", nil, "", http.StatusBadRequest))
+	mockStore.JobStore.On("GetNewestJobByStatusesAndType", []string{model.JobStatusWarning, model.JobStatusSuccess}, model.JobTypeMessageExport).Return(nil, errors.New("test"))
 
 	worker := &MessageExportWorker{
 		jobServer: &jobs.JobServer{
@@ -52,11 +52,13 @@ func TestInitJobDataNoJobData(t *testing.T) {
 				Cfg: &model.Config{
 					// mock config
 					MessageExportSettings: model.MessageExportSettings{
-						EnableExport:        model.NewPointer(true),
-						ExportFormat:        model.NewPointer(model.ComplianceExportTypeActiance),
-						DailyRunTime:        model.NewPointer("01:00"),
-						ExportFromTimestamp: model.NewPointer(int64(0)),
-						BatchSize:           model.NewPointer(10000),
+						EnableExport:            model.NewPointer(true),
+						ExportFormat:            model.NewPointer(model.ComplianceExportTypeActiance),
+						DailyRunTime:            model.NewPointer("01:00"),
+						ExportFromTimestamp:     model.NewPointer(int64(0)),
+						BatchSize:               model.NewPointer(10000),
+						ChannelBatchSize:        model.NewPointer(100),
+						ChannelHistoryBatchSize: model.NewPointer(100),
 					},
 				},
 			},
@@ -64,12 +66,14 @@ func TestInitJobDataNoJobData(t *testing.T) {
 		logger: logger,
 	}
 
-	// actually execute the code under test
-	worker.initJobData(logger, job)
+	now := time.Now()
+	worker.initJobData(logger, job, now)
 
-	assert.Equal(t, model.ComplianceExportTypeActiance, job.Data[JobDataExportType])
-	assert.Equal(t, strconv.Itoa(*worker.jobServer.Config().MessageExportSettings.BatchSize), job.Data[JOB_DATA_BatchSize])
-	assert.Equal(t, strconv.FormatInt(*worker.jobServer.Config().MessageExportSettings.ExportFromTimestamp, 10), job.Data[JobDataBatchStartTimestamp])
+	assert.Equal(t, model.ComplianceExportTypeActiance, job.Data[shared.JobDataExportType])
+	assert.Equal(t, strconv.Itoa(*worker.jobServer.Config().MessageExportSettings.BatchSize), job.Data[shared.JobDataBatchSize])
+	assert.Equal(t, strconv.FormatInt(*worker.jobServer.Config().MessageExportSettings.ExportFromTimestamp, 10), job.Data[shared.JobDataBatchStartTime])
+	expectedDir := path.Join(model.ComplianceExportPath, fmt.Sprintf("%s-%d-%d", now.Format(model.ComplianceExportDirectoryFormat), 0, now.UnixMilli()))
+	assert.Equal(t, expectedDir, job.Data[shared.JobDataExportDir])
 }
 
 func TestInitJobDataPreviousJobNoJobData(t *testing.T) {
@@ -78,7 +82,7 @@ func TestInitJobDataPreviousJobNoJobData(t *testing.T) {
 	defer mockStore.AssertExpectations(t)
 
 	previousJob := &model.Job{
-		Id:             model.NewId(),
+		Id:             st.NewTestID(),
 		CreateAt:       model.GetMillis(),
 		Status:         model.JobStatusSuccess,
 		Type:           model.JobTypeMessageExport,
@@ -87,7 +91,7 @@ func TestInitJobDataPreviousJobNoJobData(t *testing.T) {
 	}
 
 	job := &model.Job{
-		Id:       model.NewId(),
+		Id:       st.NewTestID(),
 		CreateAt: model.GetMillis(),
 		Status:   model.JobStatusPending,
 		Type:     model.JobTypeMessageExport,
@@ -103,11 +107,13 @@ func TestInitJobDataPreviousJobNoJobData(t *testing.T) {
 				Cfg: &model.Config{
 					// mock config
 					MessageExportSettings: model.MessageExportSettings{
-						EnableExport:        model.NewPointer(true),
-						ExportFormat:        model.NewPointer(model.ComplianceExportTypeActiance),
-						DailyRunTime:        model.NewPointer("01:00"),
-						ExportFromTimestamp: model.NewPointer(int64(0)),
-						BatchSize:           model.NewPointer(10000),
+						EnableExport:            model.NewPointer(true),
+						ExportFormat:            model.NewPointer(model.ComplianceExportTypeActiance),
+						DailyRunTime:            model.NewPointer("01:00"),
+						ExportFromTimestamp:     model.NewPointer(int64(0)),
+						BatchSize:               model.NewPointer(10000),
+						ChannelBatchSize:        model.NewPointer(100),
+						ChannelHistoryBatchSize: model.NewPointer(100),
 					},
 				},
 			},
@@ -115,12 +121,14 @@ func TestInitJobDataPreviousJobNoJobData(t *testing.T) {
 		logger: logger,
 	}
 
-	// actually execute the code under test
-	worker.initJobData(logger, job)
+	now := time.Now()
+	worker.initJobData(logger, job, now)
 
-	assert.Equal(t, model.ComplianceExportTypeActiance, job.Data[JobDataExportType])
-	assert.Equal(t, strconv.Itoa(*worker.jobServer.Config().MessageExportSettings.BatchSize), job.Data[JOB_DATA_BatchSize])
-	assert.Equal(t, strconv.FormatInt(*worker.jobServer.Config().MessageExportSettings.ExportFromTimestamp, 10), job.Data[JobDataBatchStartTimestamp])
+	assert.Equal(t, model.ComplianceExportTypeActiance, job.Data[shared.JobDataExportType])
+	assert.Equal(t, strconv.Itoa(*worker.jobServer.Config().MessageExportSettings.BatchSize), job.Data[shared.JobDataBatchSize])
+	assert.Equal(t, strconv.FormatInt(*worker.jobServer.Config().MessageExportSettings.ExportFromTimestamp, 10), job.Data[shared.JobDataBatchStartTime])
+	expectedDir := path.Join(model.ComplianceExportPath, fmt.Sprintf("%s-%d-%d", now.Format(model.ComplianceExportDirectoryFormat), 0, now.UnixMilli()))
+	assert.Equal(t, expectedDir, job.Data[shared.JobDataExportDir])
 }
 
 func TestInitJobDataPreviousJobWithJobData(t *testing.T) {
@@ -129,20 +137,21 @@ func TestInitJobDataPreviousJobWithJobData(t *testing.T) {
 	defer mockStore.AssertExpectations(t)
 
 	previousJob := &model.Job{
-		Id:             model.NewId(),
+		Id:             st.NewTestID(),
 		CreateAt:       model.GetMillis(),
 		Status:         model.JobStatusSuccess,
 		Type:           model.JobTypeMessageExport,
 		StartAt:        model.GetMillis() - 1000,
 		LastActivityAt: model.GetMillis() - 1000,
-		Data:           map[string]string{JobDataBatchStartTimestamp: "123"},
+		Data:           map[string]string{shared.JobDataBatchStartTime: "123"},
 	}
 
 	job := &model.Job{
-		Id:       model.NewId(),
+		Id:       st.NewTestID(),
 		CreateAt: model.GetMillis(),
 		Status:   model.JobStatusPending,
 		Type:     model.JobTypeMessageExport,
+		Data:     map[string]string{shared.JobDataExportDir: "this-is-the-export-dir"},
 	}
 
 	// mock job store returns a previously successful job that has the config that we're looking for, so we use it
@@ -155,11 +164,13 @@ func TestInitJobDataPreviousJobWithJobData(t *testing.T) {
 				Cfg: &model.Config{
 					// mock config
 					MessageExportSettings: model.MessageExportSettings{
-						EnableExport:        model.NewPointer(true),
-						ExportFormat:        model.NewPointer(model.ComplianceExportTypeActiance),
-						DailyRunTime:        model.NewPointer("01:00"),
-						ExportFromTimestamp: model.NewPointer(int64(0)),
-						BatchSize:           model.NewPointer(10000),
+						EnableExport:            model.NewPointer(true),
+						ExportFormat:            model.NewPointer(model.ComplianceExportTypeActiance),
+						DailyRunTime:            model.NewPointer("01:00"),
+						ExportFromTimestamp:     model.NewPointer(int64(0)),
+						BatchSize:               model.NewPointer(10000),
+						ChannelBatchSize:        model.NewPointer(100),
+						ChannelHistoryBatchSize: model.NewPointer(100),
 					},
 				},
 			},
@@ -167,12 +178,14 @@ func TestInitJobDataPreviousJobWithJobData(t *testing.T) {
 		logger: logger,
 	}
 
-	// actually execute the code under test
-	worker.initJobData(logger, job)
+	now := time.Now()
+	worker.initJobData(logger, job, now)
 
-	assert.Equal(t, model.ComplianceExportTypeActiance, job.Data[JobDataExportType])
-	assert.Equal(t, strconv.Itoa(*worker.jobServer.Config().MessageExportSettings.BatchSize), job.Data[JOB_DATA_BatchSize])
-	assert.Equal(t, previousJob.Data[JobDataBatchStartTimestamp], job.Data[JobDataBatchStartTimestamp])
+	assert.Equal(t, model.ComplianceExportTypeActiance, job.Data[shared.JobDataExportType])
+	assert.Equal(t, strconv.Itoa(*worker.jobServer.Config().MessageExportSettings.BatchSize), job.Data[shared.JobDataBatchSize])
+	assert.Equal(t, previousJob.Data[shared.JobDataBatchStartTime], job.Data[shared.JobDataBatchStartTime])
+	expectedDir := "this-is-the-export-dir"
+	assert.Equal(t, expectedDir, job.Data[shared.JobDataExportDir])
 }
 
 func TestDoJobNoPostsToExport(t *testing.T) {
@@ -185,7 +198,7 @@ func TestDoJobNoPostsToExport(t *testing.T) {
 	defer mockMetrics.AssertExpectations(t)
 
 	job := &model.Job{
-		Id:       model.NewId(),
+		Id:       st.NewTestID(),
 		CreateAt: model.GetMillis(),
 		Status:   model.JobStatusPending,
 		Type:     model.JobTypeMessageExport,
@@ -196,15 +209,19 @@ func TestDoJobNoPostsToExport(t *testing.T) {
 	mockMetrics.On("IncrementJobActive", model.JobTypeMessageExport)
 
 	// no previous job, data will be loaded from config
-	mockStore.JobStore.On("GetNewestJobByStatusesAndType", []string{model.JobStatusWarning, model.JobStatusSuccess}, model.JobTypeMessageExport).Return(nil, model.NewAppError("", "", nil, "", http.StatusBadRequest))
+	mockStore.JobStore.On("GetNewestJobByStatusesAndType", []string{model.JobStatusWarning, model.JobStatusSuccess}, model.JobTypeMessageExport).Return(nil, errors.New("test"))
+
+	// no channels with activity
+	mockStore.ChannelMemberHistoryStore.On("GetChannelsWithActivityDuring", mock.Anything, mock.Anything).
+		Return(make([]string, 0), nil)
 
 	// no posts found to export
-	mockStore.ComplianceStore.On("MessageExport", mock.Anything, mock.AnythingOfType("model.MessageExportCursor"), 10000).Return(
+	mockStore.ComplianceStore.On("MessageExport", mock.Anything, mock.AnythingOfType("model.MessageExportCursor"), 10001).Return(
 		make([]*model.MessageExport, 0), model.MessageExportCursor{}, nil,
 	)
 
 	mockStore.PostStore.On("AnalyticsPostCount", mock.Anything).Return(
-		int64(estimatedPostCount), nil,
+		int64(shared.EstimatedPostCount), nil,
 	)
 
 	// job completed successfully
@@ -229,11 +246,13 @@ func TestDoJobNoPostsToExport(t *testing.T) {
 						Directory:  model.NewPointer(tempDir),
 					},
 					MessageExportSettings: model.MessageExportSettings{
-						EnableExport:        model.NewPointer(true),
-						ExportFormat:        model.NewPointer(model.ComplianceExportTypeActiance),
-						DailyRunTime:        model.NewPointer("01:00"),
-						ExportFromTimestamp: model.NewPointer(int64(0)),
-						BatchSize:           model.NewPointer(10000),
+						EnableExport:            model.NewPointer(true),
+						ExportFormat:            model.NewPointer(model.ComplianceExportTypeActiance),
+						DailyRunTime:            model.NewPointer("01:00"),
+						ExportFromTimestamp:     model.NewPointer(int64(0)),
+						BatchSize:               model.NewPointer(10000),
+						ChannelBatchSize:        model.NewPointer(100),
+						ChannelHistoryBatchSize: model.NewPointer(100),
 					},
 				},
 			},
@@ -258,7 +277,7 @@ func TestDoJobWithDedicatedExportBackend(t *testing.T) {
 	defer mockMetrics.AssertExpectations(t)
 
 	job := &model.Job{
-		Id:       model.NewId(),
+		Id:       st.NewTestID(),
 		CreateAt: model.GetMillis(),
 		Status:   model.JobStatusPending,
 		Type:     model.JobTypeMessageExport,
@@ -269,15 +288,50 @@ func TestDoJobWithDedicatedExportBackend(t *testing.T) {
 	mockMetrics.On("IncrementJobActive", model.JobTypeMessageExport)
 
 	// no previous job, data will be loaded from config
-	mockStore.JobStore.On("GetNewestJobByStatusesAndType", []string{model.JobStatusWarning, model.JobStatusSuccess}, model.JobTypeMessageExport).Return(nil, model.NewAppError("", "", nil, "", http.StatusBadRequest))
+	mockStore.JobStore.On("GetNewestJobByStatusesAndType", []string{model.JobStatusWarning, model.JobStatusSuccess}, model.JobTypeMessageExport).Return(nil, errors.New("test"))
 
-	// no posts found to export
-	mockStore.ComplianceStore.On("MessageExport", mock.Anything, mock.AnythingOfType("model.MessageExportCursor"), 10000).Return(
+	channelId := st.NewTestID()
+	channelName := st.NewTestID()
+	channelDisplayName := st.NewTestID()
+	channelType := model.ChannelTypeOpen
+	messages := []*model.MessageExport{
+		{
+			TeamId:       model.NewPointer(st.NewTestID()),
+			ChannelId:    model.NewPointer(channelId),
+			ChannelName:  model.NewPointer(channelName),
+			UserId:       model.NewPointer(st.NewTestID()),
+			UserEmail:    model.NewPointer(st.NewTestID()),
+			Username:     model.NewPointer(st.NewTestID()),
+			PostId:       model.NewPointer(st.NewTestID()),
+			PostCreateAt: model.NewPointer[int64](123),
+			PostUpdateAt: model.NewPointer[int64](123),
+			PostDeleteAt: model.NewPointer[int64](123),
+			PostMessage:  model.NewPointer(st.NewTestID()),
+		},
+	}
+
+	// need to export at least one post to make an export directory and file
+
+	mockStore.ChannelMemberHistoryStore.On("GetChannelsWithActivityDuring", mock.Anything, mock.Anything).
+		Return([]string{*messages[0].ChannelId}, nil)
+	mockStore.ChannelStore.On("GetMany", []string{channelId}, true).
+		Return(model.ChannelList{{
+			Id:          channelId,
+			DisplayName: channelDisplayName,
+			Name:        channelName,
+			Type:        channelType,
+		}}, nil)
+
+	mockStore.ComplianceStore.On("MessageExport", mock.Anything, mock.AnythingOfType("model.MessageExportCursor"), 10001).Return(
+		messages, model.MessageExportCursor{}, nil,
+	).Once()
+	mockStore.ComplianceStore.On("MessageExport", mock.Anything, mock.AnythingOfType("model.MessageExportCursor"), 10001).Return(
 		make([]*model.MessageExport, 0), model.MessageExportCursor{}, nil,
-	)
+	).Once()
+	mockStore.ChannelMemberHistoryStore.On("GetUsersInChannelDuring", mock.Anything, mock.Anything, mock.Anything).Return(nil, nil)
 
 	mockStore.PostStore.On("AnalyticsPostCount", mock.Anything).Return(
-		int64(estimatedPostCount), nil,
+		int64(1), nil,
 	)
 
 	// job completed successfully
@@ -309,11 +363,13 @@ func TestDoJobWithDedicatedExportBackend(t *testing.T) {
 						ExportDirectory:      model.NewPointer(tempDedicatedDir),
 					},
 					MessageExportSettings: model.MessageExportSettings{
-						EnableExport:        model.NewPointer(true),
-						ExportFormat:        model.NewPointer(model.ComplianceExportTypeActiance),
-						DailyRunTime:        model.NewPointer("01:00"),
-						ExportFromTimestamp: model.NewPointer(int64(0)),
-						BatchSize:           model.NewPointer(10000),
+						EnableExport:            model.NewPointer(true),
+						ExportFormat:            model.NewPointer(model.ComplianceExportTypeActiance),
+						DailyRunTime:            model.NewPointer("01:00"),
+						ExportFromTimestamp:     model.NewPointer(int64(0)),
+						BatchSize:               model.NewPointer(10000),
+						ChannelBatchSize:        model.NewPointer(100),
+						ChannelHistoryBatchSize: model.NewPointer(100),
 					},
 				},
 			},
@@ -347,7 +403,7 @@ func TestDoJobCancel(t *testing.T) {
 	t.Cleanup(func() { mockMetrics.AssertExpectations(t) })
 
 	job := &model.Job{
-		Id:       model.NewId(),
+		Id:       st.NewTestID(),
 		CreateAt: model.GetMillis(),
 		Status:   model.JobStatusPending,
 		Type:     model.JobTypeMessageExport,
@@ -368,11 +424,13 @@ func TestDoJobCancel(t *testing.T) {
 							Directory:  model.NewPointer(tempDir),
 						},
 						MessageExportSettings: model.MessageExportSettings{
-							EnableExport:        model.NewPointer(true),
-							ExportFormat:        model.NewPointer(model.ComplianceExportTypeActiance),
-							DailyRunTime:        model.NewPointer("01:00"),
-							ExportFromTimestamp: model.NewPointer(int64(0)),
-							BatchSize:           model.NewPointer(10000),
+							EnableExport:            model.NewPointer(true),
+							ExportFormat:            model.NewPointer(model.ComplianceExportTypeActiance),
+							DailyRunTime:            model.NewPointer("01:00"),
+							ExportFromTimestamp:     model.NewPointer(int64(0)),
+							BatchSize:               model.NewPointer(10000),
+							ChannelBatchSize:        model.NewPointer(100),
+							ChannelHistoryBatchSize: model.NewPointer(100),
 						},
 					},
 				},
@@ -390,11 +448,26 @@ func TestDoJobCancel(t *testing.T) {
 	mockMetrics.On("IncrementJobActive", model.JobTypeMessageExport)
 
 	// No previous job, data will be loaded from config
-	mockStore.JobStore.On("GetNewestJobByStatusesAndType", []string{model.JobStatusWarning, model.JobStatusSuccess}, model.JobTypeMessageExport).Return(nil, model.NewAppError("", "", nil, "", http.StatusBadRequest))
+	mockStore.JobStore.On("GetNewestJobByStatusesAndType", []string{model.JobStatusWarning, model.JobStatusSuccess}, model.JobTypeMessageExport).Return(nil, errors.New("test"))
+
+	// Job updates the system console UI, once for getting channels, once for getting activity
+	mockStore.JobStore.On("UpdateOptimistically", mock.AnythingOfType("*model.Job"), model.JobStatusInProgress).Return(true, nil).Times(2)
+
+	// a few calls pass
+	mockStore.ChannelMemberHistoryStore.On("GetChannelsWithActivityDuring", mock.Anything, mock.Anything).
+		Return([]string{"channel-id"}, nil)
+	mockStore.ChannelStore.On("GetMany", []string{"channel-id"}, true).
+		Return(model.ChannelList{{
+			Id:          "channel-id",
+			DisplayName: "channel-display-name",
+			Name:        "channel-name",
+			Type:        model.ChannelTypeDirect,
+		}}, nil)
+	mockStore.ChannelMemberHistoryStore.On("GetUsersInChannelDuring", mock.Anything, mock.Anything, []string{"channel-id"}).Return([]*model.ChannelMemberHistoryResult{}, nil)
 
 	cancelled := make(chan struct{})
 	// Cancel the worker and return an error
-	mockStore.ComplianceStore.On("MessageExport", mock.Anything, mock.AnythingOfType("model.MessageExportCursor"), 10000).Run(func(args tmock.Arguments) {
+	mockStore.ComplianceStore.On("MessageExport", mock.Anything, mock.AnythingOfType("model.MessageExportCursor"), 10001).Run(func(args tmock.Arguments) {
 		worker.cancel()
 
 		rctx, ok := args.Get(0).(request.CTX)
@@ -408,7 +481,7 @@ func TestDoJobCancel(t *testing.T) {
 	)
 
 	mockStore.PostStore.On("AnalyticsPostCount", mock.Anything).Return(
-		int64(estimatedPostCount), nil,
+		int64(shared.EstimatedPostCount), nil,
 	)
 
 	// Job marked as pending
@@ -425,75 +498,3 @@ func TestDoJobCancel(t *testing.T) {
 	// Cleanup
 	worker.Stop()
 }
-
-func TestCreateZipFile(t *testing.T) {
-	rctx := request.TestContext(t)
-
-	tempDir, ioErr := os.MkdirTemp("", "")
-	require.NoError(t, ioErr)
-	defer os.RemoveAll(tempDir)
-
-	config := filestore.FileBackendSettings{
-		DriverName: model.ImageDriverLocal,
-		Directory:  tempDir,
-	}
-
-	fileBackend, err := filestore.NewFileBackend(config)
-	assert.NoError(t, err)
-	_ = fileBackend
-
-	b := []byte("test")
-	path1 := path.Join(exportPath, "19700101")
-	path2 := path.Join(exportPath, "19800101/subdir")
-
-	// We test with a mock to test the Hitachi HCP case
-	// where ListDirectory returns the dir itself as the first entry.
-	// Note: If the mocks fail, that means the logic in createZipFile has
-	// gone wrong and needs to be verified.
-	mock := &fmocks.FileBackend{}
-	defer mock.AssertExpectations(t)
-
-	mock.On("WriteFile", tmock.Anything, tmock.AnythingOfType("string")).Return(int64(4), nil)
-	mock.On("FileSize", tmock.Anything).Return(int64(4), nil)
-	mock.On("FileSize", tmock.Anything).Return(int64(4), nil)
-	mock.On("Reader", path.Join(path1, "testid")).Return(mockReadSeekCloser{bytes.NewReader([]byte("test"))}, nil)
-	mock.On("Reader", path.Join(path2, "testid")).Return(mockReadSeekCloser{bytes.NewReader([]byte("test"))}, nil)
-	mock.On("ListDirectoryRecursively", path1).Return([]string{path1, path.Join(path1, "testid")}, nil)
-	mock.On("ListDirectoryRecursively", path2).Return([]string{path2, path.Join(path2, "testid")}, nil)
-
-	for i, backend := range []filestore.FileBackend{fileBackend, mock} {
-		written, err := backend.WriteFile(bytes.NewReader(b), path1+"/"+model.NewId())
-		assert.NoError(t, err)
-		assert.Equal(t, int64(len(b)), written)
-
-		written, err = backend.WriteFile(bytes.NewReader(b), path2+"/"+model.NewId())
-		assert.NoError(t, err)
-		assert.Equal(t, int64(len(b)), written)
-
-		written, err = backend.WriteFile(bytes.NewReader(b), path2+"/"+model.NewId())
-		assert.NoError(t, err)
-		assert.Equal(t, int64(len(b)), written)
-
-		err = createZipFile(rctx, backend, "testjob", []string{path1, path2})
-		assert.NoError(t, err)
-
-		// Skip checking the zip file in mock case.
-		if i == 1 {
-			continue
-		}
-		r, err := zip.OpenReader(path.Join(tempDir, exportPath) + "/testjob.zip")
-		assert.NoError(t, err)
-		err = r.Close()
-		require.NoError(t, err)
-
-		assert.Equal(t, 3, len(r.File))
-	}
-}
-
-type mockReadSeekCloser struct {
-	*bytes.Reader
-}
-
-func (r mockReadSeekCloser) Close() error {
-	return nil
-}

server/i18n/en.json

@@ -7576,14 +7576,6 @@
     "id": "ent.account_migration.get_saml_users_failed",
     "translation": "Unable to get SAML users."
   },
-  {
-    "id": "ent.actiance.export.marshalToXml.appError",
-    "translation": "Unable to convert export to XML."
-  },
-  {
-    "id": "ent.actiance.export.write_file.appError",
-    "translation": "Unable to write the export file."
-  },
   {
     "id": "ent.api.post.send_notifications_and_forget.push_image_only",
     "translation": " attached a file."
@@ -7604,86 +7596,10 @@
     "id": "ent.cluster.timeout.error",
     "translation": "Timed out waiting for cluster response"
   },
-  {
-    "id": "ent.compliance.bad_export_type.appError",
-    "translation": "Unknown output format {{.ExportType}}"
-  },
-  {
-    "id": "ent.compliance.csv.attachment.copy.appError",
-    "translation": "Unable to copy the attachment into the zip file."
-  },
-  {
-    "id": "ent.compliance.csv.attachment.export.appError",
-    "translation": "Unable to add attachment to the CSV export."
-  },
-  {
-    "id": "ent.compliance.csv.file.creation.appError",
-    "translation": "Unable to create temporary CSV export file."
-  },
-  {
-    "id": "ent.compliance.csv.header.export.appError",
-    "translation": "Unable to add header to the CSV export."
-  },
-  {
-    "id": "ent.compliance.csv.metadata.close.appError",
-    "translation": "Unable to close the zip file"
-  },
-  {
-    "id": "ent.compliance.csv.metadata.export.appError",
-    "translation": "Unable to add metadata file to the zip file."
-  },
-  {
-    "id": "ent.compliance.csv.metadata.json.marshalling.appError",
-    "translation": "Unable to convert metadata to json."
-  },
-  {
-    "id": "ent.compliance.csv.metadata.json.zipfile.appError",
-    "translation": "Unable to create the zip file"
-  },
-  {
-    "id": "ent.compliance.csv.post.export.appError",
-    "translation": "Unable to export a post."
-  },
-  {
-    "id": "ent.compliance.csv.seek.appError",
-    "translation": "Unable to seek to start of export file."
-  },
-  {
-    "id": "ent.compliance.csv.warning.appError",
-    "translation": "Unable to create the warning file."
-  },
-  {
-    "id": "ent.compliance.csv.write_file.appError",
-    "translation": "Unable to write the csv file."
-  },
-  {
-    "id": "ent.compliance.csv.zip.creation.appError",
-    "translation": "Unable to create the zip export file."
-  },
-  {
-    "id": "ent.compliance.global_relay.attachments_removed.appError",
-    "translation": "Uploaded file was removed from Global Relay export because it was too large to send."
-  },
-  {
-    "id": "ent.compliance.global_relay.open_temporary_file.appError",
-    "translation": "Unable to open the temporary export file."
-  },
-  {
-    "id": "ent.compliance.global_relay.rewind_temporary_file.appError",
-    "translation": "Unable to re-read the Global Relay temporary export file."
-  },
-  {
-    "id": "ent.compliance.global_relay.write_file.appError",
-    "translation": "Unable to write the global relay file."
-  },
   {
     "id": "ent.compliance.licence_disable.app_error",
     "translation": "Compliance functionality disabled by current license. Please contact your system administrator about upgrading your enterprise license."
   },
-  {
-    "id": "ent.compliance.run_export.template_watcher.appError",
-    "translation": "Unable to load export templates. Please try again."
-  },
   {
     "id": "ent.compliance.run_failed.error",
     "translation": "Compliance export failed for job '{{.JobName}}' at '{{.FilePath}}'"
@@ -7944,10 +7860,6 @@
     "id": "ent.elasticsearch.test_config.reenter_password",
     "translation": "The Search Server URL or Username has changed. Please re-enter the password to test connection."
   },
-  {
-    "id": "ent.get_users_in_channel_during",
-    "translation": "Failed to get users in channel during specified time period."
-  },
   {
     "id": "ent.id_loaded.license_disable.app_error",
     "translation": "Your license does not support ID Loaded Push Notifications."
@@ -8081,72 +7993,20 @@
     "translation": "unable to migrate."
   },
   {
-    "id": "ent.message_export.actiance_export.get_attachment_error",
-    "translation": "Failed to get file info for a post."
+    "id": "ent.message_export.actiance_export.calculate_channel_exports.activity_message",
+    "translation": "Calculating channel activity: {{.NumCompleted}}/{{.NumChannels}} channels completed."
   },
   {
-    "id": "ent.message_export.csv_export.get_attachment_error",
-    "translation": "Failed to get file info for a post."
+    "id": "ent.message_export.actiance_export.calculate_channel_exports.channel_message",
+    "translation": "Exporting channel information for {{.NumChannels}} channels."
   },
   {
-    "id": "ent.message_export.global_relay.attach_file.app_error",
-    "translation": "Unable to add attachment to the Global Relay export."
+    "id": "ent.message_export.calculate_channel_exports.app_error",
+    "translation": "Failed to calculate channel export data."
   },
   {
-    "id": "ent.message_export.global_relay.close_zip_file.app_error",
-    "translation": "Unable to close the zip file."
-  },
-  {
-    "id": "ent.message_export.global_relay.create_file_in_zip.app_error",
-    "translation": "Unable to create the eml file."
-  },
-  {
-    "id": "ent.message_export.global_relay.generate_email.app_error",
-    "translation": "Unable to generate eml file data."
-  },
-  {
-    "id": "ent.message_export.global_relay_export.deliver.close.app_error",
-    "translation": "Unable to deliver the email to Global Relay."
-  },
-  {
-    "id": "ent.message_export.global_relay_export.deliver.from_address.app_error",
-    "translation": "Unable to set the email From address."
-  },
-  {
-    "id": "ent.message_export.global_relay_export.deliver.msg.app_error",
-    "translation": "Unable to set the email message."
-  },
-  {
-    "id": "ent.message_export.global_relay_export.deliver.msg_data.app_error",
-    "translation": "Unable to write the email message."
-  },
-  {
-    "id": "ent.message_export.global_relay_export.deliver.parse_mail.app_error",
-    "translation": "Unable to read the email information."
-  },
-  {
-    "id": "ent.message_export.global_relay_export.deliver.to_address.app_error",
-    "translation": "Unable to set the email To address."
-  },
-  {
-    "id": "ent.message_export.global_relay_export.deliver.unable_to_connect_smtp_server.app_error",
-    "translation": "Unable to connect to the smtp server"
-  },
-  {
-    "id": "ent.message_export.global_relay_export.deliver.unable_to_get_file_info.app_error",
-    "translation": "Unable to get the information of the export temporary file."
-  },
-  {
-    "id": "ent.message_export.global_relay_export.deliver.unable_to_open_email_file.app_error",
-    "translation": "Unable to get the an email from the temporary file."
-  },
-  {
-    "id": "ent.message_export.global_relay_export.deliver.unable_to_open_zip_file_data.app_error",
-    "translation": "Unable to open the export temporary file."
-  },
-  {
-    "id": "ent.message_export.global_relay_export.get_attachment_error",
-    "translation": "Failed to get file info for a post."
+    "id": "ent.message_export.job_data_conversion.app_error",
+    "translation": "Failed to convert a value from the job's data field."
   },
   {
     "id": "ent.message_export.run_export.app_error",

server/public/model/config.go

@@ -234,13 +234,19 @@ const (
 	PluginSettingsDefaultMarketplaceURL    = "https://api.integrations.mattermost.com"
 	PluginSettingsOldMarketplaceURL        = "https://marketplace.integrations.mattermost.com"
 
-	ComplianceExportTypeCsv            = "csv"
-	ComplianceExportTypeActiance       = "actiance"
-	ComplianceExportTypeGlobalrelay    = "globalrelay"
-	ComplianceExportTypeGlobalrelayZip = "globalrelay-zip"
-	GlobalrelayCustomerTypeA9          = "A9"
-	GlobalrelayCustomerTypeA10         = "A10"
-	GlobalrelayCustomerTypeCustom      = "CUSTOM"
+	ComplianceExportDirectoryFormat                = "compliance-export-2006-01-02-15h04m"
+	ComplianceExportPath                           = "export"
+	ComplianceExportPathCLI                        = "cli"
+	ComplianceExportTypeCsv                        = "csv"
+	ComplianceExportTypeActiance                   = "actiance"
+	ComplianceExportTypeGlobalrelay                = "globalrelay"
+	ComplianceExportTypeGlobalrelayZip             = "globalrelay-zip"
+	ComplianceExportChannelBatchSizeDefault        = 100
+	ComplianceExportChannelHistoryBatchSizeDefault = 10
+
+	GlobalrelayCustomerTypeA9     = "A9"
+	GlobalrelayCustomerTypeA10    = "A10"
+	GlobalrelayCustomerTypeCustom = "CUSTOM"
 
 	ClientSideCertCheckPrimaryAuth   = "primary"
 	ClientSideCertCheckSecondaryAuth = "secondary"
@@ -3403,12 +3409,14 @@ func (s *GlobalRelayMessageExportSettings) SetDefaults() {
 }
 
 type MessageExportSettings struct {
-	EnableExport          *bool   `access:"compliance_compliance_export"`
-	ExportFormat          *string `access:"compliance_compliance_export"`
-	DailyRunTime          *string `access:"compliance_compliance_export"`
-	ExportFromTimestamp   *int64  `access:"compliance_compliance_export"`
-	BatchSize             *int    `access:"compliance_compliance_export"`
-	DownloadExportResults *bool   `access:"compliance_compliance_export"`
+	EnableExport            *bool   `access:"compliance_compliance_export"`
+	ExportFormat            *string `access:"compliance_compliance_export"`
+	DailyRunTime            *string `access:"compliance_compliance_export"`
+	ExportFromTimestamp     *int64  `access:"compliance_compliance_export"`
+	BatchSize               *int    `access:"compliance_compliance_export"`
+	DownloadExportResults   *bool   `access:"compliance_compliance_export"`
+	ChannelBatchSize        *int    `access:"compliance_compliance_export"`
+	ChannelHistoryBatchSize *int    `access:"compliance_compliance_export"`
 
 	// formatter-specific settings - these are only expected to be non-nil if ExportFormat is set to the associated format
 	GlobalRelaySettings *GlobalRelayMessageExportSettings `access:"compliance_compliance_export"`
@@ -3439,6 +3447,14 @@ func (s *MessageExportSettings) SetDefaults() {
 		s.BatchSize = NewPointer(10000)
 	}
 
+	if s.ChannelBatchSize == nil || *s.ChannelBatchSize == 0 {
+		s.ChannelBatchSize = NewPointer(ComplianceExportChannelBatchSizeDefault)
+	}
+
+	if s.ChannelHistoryBatchSize == nil || *s.ChannelHistoryBatchSize == 0 {
+		s.ChannelHistoryBatchSize = NewPointer(ComplianceExportChannelHistoryBatchSizeDefault)
+	}
+
 	if s.GlobalRelaySettings == nil {
 		s.GlobalRelaySettings = &GlobalRelayMessageExportSettings{}
 	}
@@ -4434,7 +4450,7 @@ func (s *MessageExportSettings) isValid() *AppError {
 			return NewAppError("Config.IsValid", "model.config.is_valid.message_export.daily_runtime.app_error", nil, "", http.StatusBadRequest).Wrap(err)
 		} else if s.BatchSize == nil || *s.BatchSize < 0 {
 			return NewAppError("Config.IsValid", "model.config.is_valid.message_export.batch_size.app_error", nil, "", http.StatusBadRequest)
-		} else if s.ExportFormat == nil || (*s.ExportFormat != ComplianceExportTypeActiance && *s.ExportFormat != ComplianceExportTypeGlobalrelay && *s.ExportFormat != ComplianceExportTypeCsv) {
+		} else if s.ExportFormat == nil || (*s.ExportFormat != ComplianceExportTypeActiance && *s.ExportFormat != ComplianceExportTypeGlobalrelay && *s.ExportFormat != ComplianceExportTypeCsv && *s.ExportFormat != ComplianceExportTypeGlobalrelayZip) {
 			return NewAppError("Config.IsValid", "model.config.is_valid.message_export.export_type.app_error", nil, "", http.StatusBadRequest)
 		}
 

server/public/model/job.go

@@ -10,6 +10,7 @@ import (
 const (
 	JobTypeDataRetention                 = "data_retention"
 	JobTypeMessageExport                 = "message_export"
+	JobTypeCLIMessageExport              = "cli_message_export"
 	JobTypeElasticsearchPostIndexing     = "elasticsearch_post_indexing"
 	JobTypeElasticsearchPostAggregation  = "elasticsearch_post_aggregation"
 	JobTypeBlevePostIndexing             = "bleve_post_indexing"

server/public/model/message_export.go

@@ -24,6 +24,7 @@ type MessageExport struct {
 	PostCreateAt   *int64
 	PostUpdateAt   *int64
 	PostDeleteAt   *int64
+	PostEditAt     *int64
 	PostMessage    *string
 	PostType       *string
 	PostRootId     *string
@@ -32,9 +33,12 @@ type MessageExport struct {
 	PostFileIds    StringArray
 }
 
+// MessageExportCursor retrieves posts in the inclusive range:
+// [LastPostUpdateAt + LastPostId, UntilUpdateAt]
 type MessageExportCursor struct {
 	LastPostUpdateAt int64
 	LastPostId       string
+	UntilUpdateAt    int64
 }
 
 // PreviewID returns the value of the post's previewed_post prop, if present, or an empty string.

server/public/model/post.go

@@ -399,8 +399,11 @@ type PostCountOptions struct {
 	UsersPostsOnly     bool
 	// AllowFromCache looks up cache only when ExcludeDeleted and UsersPostsOnly are true and rest are falsy.
 	AllowFromCache bool
-	SincePostID    string
-	SinceUpdateAt  int64
+
+	// retrieves posts in the inclusive range: [SinceUpdateAt + LastPostId, UntilUpdateAt]
+	SincePostID   string
+	SinceUpdateAt int64
+	UntilUpdateAt int64
 }
 
 func (o *Post) Etag() string {

server/public/shared/i18n/i18n.go

@@ -78,6 +78,28 @@ func TranslationsPreInit(translationsDir string) error {
 	return initTranslationsWithDir(translationsDir)
 }
 
+// TranslationsPreInitFromFileBytes loads translations from a buffer -- useful if
+// we need to initialize i18n from an embedded i18n file (e.g., from a CLI tool)
+func TranslationsPreInitFromFileBytes(filename string, buf []byte) error {
+	if T != nil {
+		return nil
+	}
+
+	// Set T even if we fail to load the translations. Lots of shutdown handling code will
+	// segfault trying to handle the error, and the untranslated IDs are strictly better.
+	T = tfuncWithFallback(defaultLocale)
+	TDefault = tfuncWithFallback(defaultLocale)
+
+	locale := strings.Split(filename, ".")[0]
+	if !isSupportedLocale(locale) {
+		return fmt.Errorf("locale not supported: %s", locale)
+	}
+
+	locales[locale] = filename
+
+	return i18n.ParseTranslationFileBytes(filename, buf)
+}
+
 // InitTranslations set the defaults configured in the server and initialize
 // the T function using the server default as fallback language
 func InitTranslations(serverLocale, clientLocale string) error {

server/templates/globalrelay_compliance_export.html

@@ -62,14 +62,20 @@
 <h2>Conversation Summary</h2>
 <div class="summary-list">
     <ul>
-        <li><span class="bold">Channel:&nbsp;</span>{{.Props.ChannelName}}</li>
-        <li><span class="bold">Started:&nbsp;</span>{{.Props.Started}}</li>
-        <li><span class="bold">Ended:&nbsp;</span>{{.Props.Ended}}</li>
-        <li><span class="bold">Duration:&nbsp;</span>{{.Props.Duration}}</li>
+        <li><span class="bold">TeamId:</span>{{.Props.TeamId}}</li>
+        <li><span class="bold">TeamName:</span>{{.Props.TeamName}}</li>
+        <li><span class="bold">TeamDisplayName:</span>{{.Props.TeamDisplayName}}</li>
+        <li><span class="bold">ChannelId:</span>{{.Props.ChannelId}}</li>
+        <li><span class="bold">ChannelName:</span>{{.Props.ChannelName}}</li>
+        <li><span class="bold">ChannelDisplayName:</span>{{.Props.ChannelDisplayName}}</li>
+        <li><span class="bold">Started:</span>{{.Props.Started}}</li>
+        <li><span class="bold">Ended:</span>{{.Props.Ended}}</li>
+        <li><span class="bold">Duration:</span>{{.Props.Duration}}</li>
     </ul>
 </div>
 <table class="participants">
     <tr>
+        <th class="userid">UserId<br></th>
         <th class="username">Username<br></th>
         <th class="usertype">UserType<br></th>
         <th class="email">Email</th>

server/templates/globalrelay_compliance_export_message.html

@@ -1,11 +1,16 @@
 {{define "globalrelay_compliance_export_message"}}
 <li class="message">
+    <span class="post_id">{{.Props.PostId}}</span>
     <span class="sent_time">{{.Props.SentTime}}</span>
     <span class="username">@{{.Props.Username}}</span>
+    <span class="userid">{{.Props.UserId}}</span>
     <span class="postusername">{{.Props.PostUsername}}</span>
     <span class="usertype">{{.Props.UserType}}</span>
-    <span class="email">({{.Props.Email}}):</span>
-    <span class="message">{{.Props.Message}}</span>
-    <span class="previews_post">{{.Props.PreviewsPost}}</span>
+    <span class="email">({{.Props.Email}})</span>
+    <span class="message">{{.Props.Message}}</span>{{ if .Props.PreviewsPost }}
+    <span class="previews_post">{{.Props.PreviewsPost}}</span>{{ end }}{{ if .Props.UpdateType }}
+    <span class="update_type">{{.Props.UpdateType}}</span>
+    <span class="update_time">{{.Props.UpdateTime}}</span>
+    <span class="edited_new_msg_id">{{.Props.EditedNewMsgId}}</span>{{ end }}
 </li>
 {{end}}