MM-59966 - Compliance Export overhaul - feature branch (#29789)

* [MM-59089] Add a compliance export constant (#27919)

* add a useful constant

* i18n

* another constant

* another i18n

* [MM-60422] Add GetChannelsWithActivityDuring (#28301)

* modify GetUsersInChannelDuring to accept a slice of channelIds

* add GetChannelsWithActivityDuring

* add compliance export progress message; remove unused custom status

* linting

* tests running too fast

* add batch size config settings

* add store tests

* linting

* empty commit

* i18n changes

* fix i18n ordering

* MM-60570 - Server-side changes consolidating the export CLI with server/ent code (#28640)

* add an i18n field; add the CLI's export directory

* int64 -> int

* Add UntilUpdateAt for MessageExport and AnalyticsPostCount

to merge

* remove now-unused i18n strings

* add TranslationsPreInitFromBuffer to allow CLI to use i18n

* use GetBuilder to simplify; rename TranslationsPreInitFromFileBytes

* [MM-59089] Improve compliance export timings (#1733 - Enterprise repo)

* MM-60422 - Performance and logic fixes for Compliance Exports (#1757 - Enterprise repo)

* MM-60570 - Enterprise-side changes consolidating the export CLI with server/ent code (#1769 - Enterprise repo)

* merge conflicts; missed file from ent branch

* MM-61038 - Add an option to sqlstore.New (#28702)

remove useless comment

add test

add an option to sqlstore.New

* MM-60976: Remove RunExport command from Mattermost binary (#28805)

* remove RunExport command from mattermost binary

* remove the code it was calling

* fix i18n

* remove test (was only testing license, not functionality)

* empty commit

* fix flaky GetChannelsWithActivityDuring test

* MM-60063: Dedicated Export Filestore fix, redo of #1772 (enterprise) (#28803)

* redo filestore fix #1772 (enterprise repo) on top of MM-59966 feature

* add new e2e tests for export filestore

* golint

* ok, note to self: shadowing bad, actually (when there's a defer)

* empty commit

* MM-61137 - Message export: Support 7.8.11 era dbs (#28824)

* support 7.8.11 era dbs by wrapping the store using only what we need

* fix flaky GetChannelsWithActivityDuring test

* add a comment

* only need to define the MEFileInfoStore (the one that'll be overridden)

* blank commit

* MM-60974 - Message Export: Add performance metrics (#28836)

* support 7.8.11 era dbs by wrapping the store using only what we need

* fix flaky GetChannelsWithActivityDuring test

* add a comment

* only need to define the MEFileInfoStore (the one that'll be overridden)

* performance metrics

* cleanup unneeded named returns

* blank commit

* MM-60975 - Message export: Add startTime and endTime to export folder name (#28840)

* support 7.8.11 era dbs by wrapping the store using only what we need

* fix flaky GetChannelsWithActivityDuring test

* add a comment

* only need to define the MEFileInfoStore (the one that'll be overridden)

* performance metrics

* output startTime and endTime in export folder

* empty commit

* merge conflict

* MM-60978 - Message export: Improve xml fields; fix delete semantics (#28873)

* support 7.8.11 era dbs by wrapping the store using only what we need

* fix flaky GetChannelsWithActivityDuring test

* add a comment

* only need to define the MEFileInfoStore (the one that'll be overridden)

* performance metrics

* output startTime and endTime in export folder

* empty commit

* add xml fields, omit when empty, tests

* fix delete semantics; test (and test for update semantics)

* clarify comments

* simplify edited post detection, now there's no edge case.

* add some spacing to help fast running tests

* merge conflicts/updates needed for new deleted post semantics

* linting; fixing tests from upstream merge

* use SafeDereference

* linting

* stronger typing; better wrapped errors; better formatting

* blank commit

* goimports formatting

* fix merge mistake

* minor fixes due to changes in master

* MM-61755 - Simplifying and Support reporting to the db from the CLI (#29281)

* finally clean up JobData struct and stringMap; prep for CLI using db

* and now simplify using StringMapToJobDataWithZeroValues

* remove unused fn

* create JobDataExported; clean up errors

* MM-60176 - Message Export: Global relay cleanup (#29168)

* move global relay logic into global_relay_export

* blank commit

* blank commit

* improve errors

* MM-60693 - Refactor CSV to use same codepath as Actiance (#29191)

* move global relay logic into global_relay_export

* blank commit

* refactor (and simplify) ExportParams into shared

* blank commit

* remove unused fn

* csv now uses pre-calculated joins/leaves like actiance

* improve errors

* remove nil post check; remove ignoredPosts metric

* remove unneeded copy

* MM-61696 - Refactor GlobalRelay to use same codepath as Actiance (#29225)

* move global relay logic into global_relay_export

* blank commit

* refactor (and simplify) ExportParams into shared

* blank commit

* remove unused fn

* csv now uses pre-calculated joins/leaves like actiance

* remove newly unneeded function and its test. goodbye.

* refactor GetPostAttachments for csv + global relay to share

* refactor global_relay_export and fix tests (no changes to output)

* improve errors

* remove nil post check; remove ignoredPosts metric

* remove unneeded copy

* remove unneeded nil check

* PR comments

* MM-61715 - Generalize e2e to all export types 🤖  (#29369)

* move global relay logic into global_relay_export

* blank commit

* refactor (and simplify) ExportParams into shared

* blank commit

* remove unused fn

* csv now uses pre-calculated joins/leaves like actiance

* remove newly unneeded function and its test. goodbye.

* refactor GetPostAttachments for csv + global relay to share

* refactor global_relay_export and fix tests (no changes to output)

* improve errors

* remove nil post check; remove ignoredPosts metric

* remove unneeded copy

* remove unneeded nil check

* PR comments

* refactor isDeletedMsg for all export types

* fix start and endtime, nasty csv createAt bug; bring closer to Actiance

* align unit tests with new logic (e.g. starttime / endtime)

* refactor a TimestampConvert fn for code + tests

* bug: pass templates to global relay (hurray for e2e tests, otherwise...)

* add global relay zip to allowed list (only for tests)

* test helpers

* new templates for e2e tests

* e2e tests... phew.

* linting

* merge conflicts

* unexport PostToRow; add test helper marker

* cleanup, shortening, thanks to PR comments

* MM-61972 - Generalize export data path - Actiance (#29399)

* extract and generalize the export data generation functions

* finish moving test (bc of previous extraction)

* lift a function from common -> shared (to break an import cycle)

* actiance now takes general export data, processes it into actiance data

* bring tests in line with correct sorting rules (upadateAt, messageId)

* fixups, PR comments

* turn strings.Repeat into a more descriptive const

amended: one letter fix; bad rebase

* MM-62009 - e2e clock heisenbug (#29434)

* consolidate assertions; output debuggable diffs (keeping for future)

* refactor test output generator to generators file

* waitUntilZeroPosts + pass through until to job = fix all clock issues

* simplify messages to model.NewId(); remove unneeded waitUntilZeroPosts

* model.NewId() -> storetest.NewTestID()

* MM-61980 - Generalize export data path - CSV (#29482)

* simple refactoring

* increase sleep times for (very) rare test failures

* add extra information to the generic export for CSV

* adj Actiance to handle new generic export (no difference in its output)

* no longer need mergePosts (yay), move getJoinLeavePosts for everyone

* adjust tests for new csv semantics (detailed in summary)

* and need to add the new exported data to the export_data_tests

* rearrange csv writing to happen after data export (more logical)

* linting

* remove debug statements

* figured out what was wrong with global relay e2e test 3; solid now

* PR comments

* MM-61718 - Generalize export data path - Global Relay (#29508)

* move global relay over to using the generalized export data

* performance pass -- not much can be done

* Update server/enterprise/message_export/global_relay_export/global_relay_export.go

Co-authored-by: Claudio Costa <cstcld91@gmail.com>

---------

Co-authored-by: Claudio Costa <cstcld91@gmail.com>

* MM-62058 - Align CSV with Actiance (#29551)

* refactoring actiance files and var names for clarity

* bug found in exported attachments (we used to miss some start/ends)

* changes needed for actiance due to new generic exports

* bringing CSV up to actiance standards

* fixing global relay b/c of new semantics (adding a note on an edge case)

* aligning e2e tests, adding comments to clarify what is expected/tested

* necessary changes; 1 more test for added functionality (ignoreDeleted)

* comment style

* MM-62059 - Align Global Relay with Actiance/CSV; many fixes (#29665)

* core logic changes to general export_data and the specific export paths

* unit tests and e2e tests, covering all new edge cases and all logic

* linting

* better var naming, const value, and cleaning up functions calls

* MM-62436 - Temporarily skip cypress tests that require download link (#29772)

---------

Co-authored-by: Claudio Costa <cstcld91@gmail.com>

server/enterprise/message_export/common_export/common_export.go

@@ -5,8 +5,13 @@ package common_export
 
 import (
 	"fmt"
+	"strconv"
+
+	"github.com/mattermost/mattermost/server/v8/enterprise/message_export/shared"
 
 	"github.com/mattermost/mattermost/server/public/model"
+	"github.com/mattermost/mattermost/server/public/shared/mlog"
+	"github.com/mattermost/mattermost/server/public/shared/request"
 )
 
 const MissingFileMessage = "File missing for post; cannot copy file to archive"
@@ -33,8 +38,6 @@ type ChannelMember struct {
 	Email    string
 	Username string
 }
-type ChannelMembers map[string]ChannelMember
-type MembersByChannel map[string]ChannelMembers
 
 type MetadataChannel struct {
 	TeamId             *string
@@ -52,7 +55,7 @@ type MetadataChannel struct {
 }
 
 type Metadata struct {
-	Channels         map[string]MetadataChannel
+	Channels         map[string]*MetadataChannel
 	MessagesCount    int
 	AttachmentsCount int
 	StartTime        int64
@@ -62,7 +65,7 @@ type Metadata struct {
 func (metadata *Metadata) Update(post *model.MessageExport, attachments int) {
 	channelMetadata, ok := metadata.Channels[*post.ChannelId]
 	if !ok {
-		channelMetadata = MetadataChannel{
+		channelMetadata = &MetadataChannel{
 			TeamId:             post.TeamId,
 			TeamName:           post.TeamName,
 			TeamDisplayName:    post.TeamDisplayName,
@@ -89,11 +92,93 @@ func (metadata *Metadata) Update(post *model.MessageExport, attachments int) {
 	metadata.Channels[*post.ChannelId] = channelMetadata
 }
 
-func GetJoinsAndLeavesForChannel(startTime int64, endTime int64, channelMembersHistory []*model.ChannelMemberHistoryResult, channelMembers ChannelMembers) ([]ChannelMemberJoin, []ChannelMemberLeave) {
-	joins := []ChannelMemberJoin{}
-	leaves := []ChannelMemberLeave{}
+type ChannelExportsParams struct {
+	Store                   shared.MessageExportStore
+	ExportPeriodStartTime   int64
+	ExportPeriodEndTime     int64
+	ChannelBatchSize        int
+	ChannelHistoryBatchSize int
+	ReportProgressMessage   func(message string)
+}
 
-	alreadyJoined := map[string]bool{}
+// CalculateChannelExports returns the channel info ( map[channelId]*MetadataChannel ) and the channel user
+// joins/leaves ( map[channelId][]*model.ChannelMemberHistoryResult ) for any channel that has had activity
+// (posts or user join/leaves) between ExportPeriodStartTime and ExportPeriodEndTime.
+func CalculateChannelExports(rctx request.CTX, opt ChannelExportsParams) (map[string]*MetadataChannel, map[string][]*model.ChannelMemberHistoryResult, error) {
+	// Which channels had user activity in the export period?
+	activeChannelIds, err := opt.Store.ChannelMemberHistory().GetChannelsWithActivityDuring(opt.ExportPeriodStartTime, opt.ExportPeriodEndTime)
+	if err != nil {
+		return nil, nil, err
+	}
+
+	if len(activeChannelIds) == 0 {
+		return nil, nil, nil
+	}
+
+	rctx.Logger().Debug("Started CalculateChannelExports", mlog.Int("export_period_start_time", opt.ExportPeriodStartTime), mlog.Int("export_period_end_time", opt.ExportPeriodEndTime), mlog.Int("num_active_channel_ids", len(activeChannelIds)))
+	message := rctx.T("ent.message_export.actiance_export.calculate_channel_exports.channel_message", model.StringMap{"NumChannels": strconv.Itoa(len(activeChannelIds))})
+	opt.ReportProgressMessage(message)
+
+	// For each channel, get its metadata.
+	channelMetadata := make(map[string]*MetadataChannel, len(activeChannelIds))
+
+	// Use batches to reduce db load and network waste.
+	for pos := 0; pos < len(activeChannelIds); pos += opt.ChannelBatchSize {
+		upTo := min(pos+opt.ChannelBatchSize, len(activeChannelIds))
+		batch := activeChannelIds[pos:upTo]
+		channels, err := opt.Store.Channel().GetMany(batch, true)
+		if err != nil {
+			return nil, nil, err
+		}
+
+		for _, channel := range channels {
+			channelMetadata[channel.Id] = &MetadataChannel{
+				TeamId:             model.NewPointer(channel.TeamId),
+				ChannelId:          channel.Id,
+				ChannelName:        channel.Name,
+				ChannelDisplayName: channel.DisplayName,
+				ChannelType:        channel.Type,
+				RoomId:             fmt.Sprintf("%v - %v", ChannelTypeDisplayName(channel.Type), channel.Id),
+				StartTime:          opt.ExportPeriodStartTime,
+				EndTime:            opt.ExportPeriodEndTime,
+			}
+		}
+	}
+
+	historiesByChannelId := make(map[string][]*model.ChannelMemberHistoryResult, len(activeChannelIds))
+
+	// Now that we have metadata, get channelMemberHistories for each channel.
+	// Use batches to reduce total db load and network waste.
+	for pos := 0; pos < len(activeChannelIds); pos += opt.ChannelHistoryBatchSize {
+		// This may take a while, so update the system console UI.
+		message := rctx.T("ent.message_export.actiance_export.calculate_channel_exports.activity_message", model.StringMap{
+			"NumChannels":  strconv.Itoa(len(activeChannelIds)),
+			"NumCompleted": strconv.Itoa(pos),
+		})
+		opt.ReportProgressMessage(message)
+
+		upTo := min(pos+opt.ChannelHistoryBatchSize, len(activeChannelIds))
+		batch := activeChannelIds[pos:upTo]
+		channelMemberHistories, err := opt.Store.ChannelMemberHistory().GetUsersInChannelDuring(opt.ExportPeriodStartTime, opt.ExportPeriodEndTime, batch)
+		if err != nil {
+			return nil, nil, err
+		}
+
+		// collect the channelMemberHistories by channelId
+		for _, entry := range channelMemberHistories {
+			historiesByChannelId[entry.ChannelId] = append(historiesByChannelId[entry.ChannelId], entry)
+		}
+	}
+
+	return channelMetadata, historiesByChannelId, nil
+}
+
+func GetJoinsAndLeavesForChannel(startTime int64, endTime int64, channelMembersHistory []*model.ChannelMemberHistoryResult,
+	postAuthors map[string]ChannelMember) ([]ChannelMemberJoin, []ChannelMemberLeave) {
+	var joins []ChannelMemberJoin
+	var leaves []ChannelMemberLeave
+
+	alreadyJoined := make(map[string]bool)
 	for _, cmh := range channelMembersHistory {
 		if cmh.UserDeleteAt > 0 && cmh.UserDeleteAt < startTime {
 			continue
@@ -129,7 +214,7 @@ func GetJoinsAndLeavesForChannel(startTime int64, endTime int64, channelMembersH
 		}
 	}
 
-	for _, member := range channelMembers {
+	for _, member := range postAuthors {
 		if alreadyJoined[member.UserId] {
 			continue
 		}

server/enterprise/message_export/common_export/common_export_test.go

@@ -13,7 +13,7 @@ import (
 
 func TestUpdateMetadata(t *testing.T) {
 	metadata := Metadata{
-		Channels:         map[string]MetadataChannel{},
+		Channels:         map[string]*MetadataChannel{},
 		MessagesCount:    0,
 		AttachmentsCount: 0,
 		StartTime:        0,
@@ -92,7 +92,7 @@ func TestGetJoinsAndLeavesForChannel(t *testing.T) {
 		name           string
 		channel        MetadataChannel
 		membersHistory []*model.ChannelMemberHistoryResult
-		usersInPosts   ChannelMembers
+		usersInPosts   map[string]ChannelMember
 		expectedJoins  int
 		expectedLeaves int
 	}{
@@ -160,9 +160,9 @@ func TestGetJoinsAndLeavesForChannel(t *testing.T) {
 			name:           "implicit-joins",
 			channel:        channel,
 			membersHistory: nil,
-			usersInPosts: ChannelMembers{
-				"test1": ChannelMember{UserId: "test1", Email: "test1", Username: "test1"},
-				"test2": ChannelMember{UserId: "test2", Email: "test2", Username: "test2"},
+			usersInPosts: map[string]ChannelMember{
+				"test1": {UserId: "test1", Email: "test1", Username: "test1"},
+				"test2": {UserId: "test2", Email: "test2", Username: "test2"},
 			},
 			expectedJoins:  2,
 			expectedLeaves: 0,
@@ -174,9 +174,9 @@ func TestGetJoinsAndLeavesForChannel(t *testing.T) {
 				{JoinTime: 130, LeaveTime: model.NewPointer(int64(150)), UserId: "test1", UserEmail: "test1", Username: "test1"},
 				{JoinTime: 130, LeaveTime: model.NewPointer(int64(150)), UserId: "test3", UserEmail: "test3", Username: "test3"},
 			},
-			usersInPosts: ChannelMembers{
-				"test1": ChannelMember{UserId: "test1", Email: "test1", Username: "test1"},
-				"test2": ChannelMember{UserId: "test2", Email: "test2", Username: "test2"},
+			usersInPosts: map[string]ChannelMember{
+				"test1": {UserId: "test1", Email: "test1", Username: "test1"},
+				"test2": {UserId: "test2", Email: "test2", Username: "test2"},
 			},
 			expectedJoins:  3,
 			expectedLeaves: 2,