MM-59966 - Compliance Export overhaul - feature branch (#29789)

* [MM-59089] Add a compliance export constant (#27919) * add a useful constant * i18n * another constant * another i18n * [MM-60422] Add GetChannelsWithActivityDuring (#28301) * modify GetUsersInChannelDuring to accept a slice of channelIds * add GetChannelsWithActivityDuring * add compliance export progress message; remove unused custom status * linting * tests running too fast * add batch size config settings * add store tests * linting * empty commit * i18n changes * fix i18n ordering * MM-60570 - Server-side changes consolidating the export CLI with server/ent code (#28640) * add an i18n field; add the CLI's export directory * int64 -> int * Add UntilUpdateAt for MessageExport and AnalyticsPostCount to merge * remove now-unused i18n strings * add TranslationsPreInitFromBuffer to allow CLI to use i18n * use GetBuilder to simplify; rename TranslationsPreInitFromFileBytes * [MM-59089] Improve compliance export timings (#1733 - Enterprise repo) * MM-60422 - Performance and logic fixes for Compliance Exports (#1757 - Enterprise repo) * MM-60570 - Enterprise-side changes consolidating the export CLI with server/ent code (#1769 - Enterprise repo) * merge conflicts; missed file from ent branch * MM-61038 - Add an option to sqlstore.New (#28702) remove useless comment add test add an option to sqlstore.New * MM-60976: Remove RunExport command from Mattermost binary (#28805) * remove RunExport command from mattermost binary * remove the code it was calling * fix i18n * remove test (was only testing license, not functionality) * empty commit * fix flaky GetChannelsWithActivityDuring test * MM-60063: Dedicated Export Filestore fix, redo of #1772 (enterprise) (#28803) * redo filestore fix #1772 (enterprise repo) on top of MM-59966 feature * add new e2e tests for export filestore * golint * ok, note to self: shadowing bad, actually (when there's a defer) * empty commit * MM-61137 - Message export: Support 7.8.11 era dbs (#28824) * support 7.8.11 era dbs by wrapping the store using only what we need * fix flaky GetChannelsWithActivityDuring test * add a comment * only need to define the MEFileInfoStore (the one that'll be overridden) * blank commit * MM-60974 - Message Export: Add performance metrics (#28836) * support 7.8.11 era dbs by wrapping the store using only what we need * fix flaky GetChannelsWithActivityDuring test * add a comment * only need to define the MEFileInfoStore (the one that'll be overridden) * performance metrics * cleanup unneeded named returns * blank commit * MM-60975 - Message export: Add startTime and endTime to export folder name (#28840) * support 7.8.11 era dbs by wrapping the store using only what we need * fix flaky GetChannelsWithActivityDuring test * add a comment * only need to define the MEFileInfoStore (the one that'll be overridden) * performance metrics * output startTime and endTime in export folder * empty commit * merge conflict * MM-60978 - Message export: Improve xml fields; fix delete semantics (#28873) * support 7.8.11 era dbs by wrapping the store using only what we need * fix flaky GetChannelsWithActivityDuring test * add a comment * only need to define the MEFileInfoStore (the one that'll be overridden) * performance metrics * output startTime and endTime in export folder * empty commit * add xml fields, omit when empty, tests * fix delete semantics; test (and test for update semantics) * clarify comments * simplify edited post detection, now there's no edge case. * add some spacing to help fast running tests * merge conflicts/updates needed for new deleted post semantics * linting; fixing tests from upstream merge * use SafeDereference * linting * stronger typing; better wrapped errors; better formatting * blank commit * goimports formatting * fix merge mistake * minor fixes due to changes in master * MM-61755 - Simplifying and Support reporting to the db from the CLI (#29281) * finally clean up JobData struct and stringMap; prep for CLI using db * and now simplify using StringMapToJobDataWithZeroValues * remove unused fn * create JobDataExported; clean up errors * MM-60176 - Message Export: Global relay cleanup (#29168) * move global relay logic into global_relay_export * blank commit * blank commit * improve errors * MM-60693 - Refactor CSV to use same codepath as Actiance (#29191) * move global relay logic into global_relay_export * blank commit * refactor (and simplify) ExportParams into shared * blank commit * remove unused fn * csv now uses pre-calculated joins/leaves like actiance * improve errors * remove nil post check; remove ignoredPosts metric * remove unneeded copy * MM-61696 - Refactor GlobalRelay to use same codepath as Actiance (#29225) * move global relay logic into global_relay_export * blank commit * refactor (and simplify) ExportParams into shared * blank commit * remove unused fn * csv now uses pre-calculated joins/leaves like actiance * remove newly unneeded function and its test. goodbye. * refactor GetPostAttachments for csv + global relay to share * refactor global_relay_export and fix tests (no changes to output) * improve errors * remove nil post check; remove ignoredPosts metric * remove unneeded copy * remove unneeded nil check * PR comments * MM-61715 - Generalize e2e to all export types 🤖 (#29369) * move global relay logic into global_relay_export * blank commit * refactor (and simplify) ExportParams into shared * blank commit * remove unused fn * csv now uses pre-calculated joins/leaves like actiance * remove newly unneeded function and its test. goodbye. * refactor GetPostAttachments for csv + global relay to share * refactor global_relay_export and fix tests (no changes to output) * improve errors * remove nil post check; remove ignoredPosts metric * remove unneeded copy * remove unneeded nil check * PR comments * refactor isDeletedMsg for all export types * fix start and endtime, nasty csv createAt bug; bring closer to Actiance * align unit tests with new logic (e.g. starttime / endtime) * refactor a TimestampConvert fn for code + tests * bug: pass templates to global relay (hurray for e2e tests, otherwise...) * add global relay zip to allowed list (only for tests) * test helpers * new templates for e2e tests * e2e tests... phew. * linting * merge conflicts * unexport PostToRow; add test helper marker * cleanup, shortening, thanks to PR comments * MM-61972 - Generalize export data path - Actiance (#29399) * extract and generalize the export data generation functions * finish moving test (bc of previous extraction) * lift a function from common -> shared (to break an import cycle) * actiance now takes general export data, processes it into actiance data * bring tests in line with correct sorting rules (upadateAt, messageId) * fixups, PR comments * turn strings.Repeat into a more descriptive const amended: one letter fix; bad rebase * MM-62009 - e2e clock heisenbug (#29434) * consolidate assertions; output debuggable diffs (keeping for future) * refactor test output generator to generators file * waitUntilZeroPosts + pass through until to job = fix all clock issues * simplify messages to model.NewId(); remove unneeded waitUntilZeroPosts * model.NewId() -> storetest.NewTestID() * MM-61980 - Generalize export data path - CSV (#29482) * simple refactoring * increase sleep times for (very) rare test failures * add extra information to the generic export for CSV * adj Actiance to handle new generic export (no difference in its output) * no longer need mergePosts (yay), move getJoinLeavePosts for everyone * adjust tests for new csv semantics (detailed in summary) * and need to add the new exported data to the export_data_tests * rearrange csv writing to happen after data export (more logical) * linting * remove debug statements * figured out what was wrong with global relay e2e test 3; solid now * PR comments * MM-61718 - Generalize export data path - Global Relay (#29508) * move global relay over to using the generalized export data * performance pass -- not much can be done * Update server/enterprise/message_export/global_relay_export/global_relay_export.go Co-authored-by: Claudio Costa <cstcld91@gmail.com> --------- Co-authored-by: Claudio Costa <cstcld91@gmail.com> * MM-62058 - Align CSV with Actiance (#29551) * refactoring actiance files and var names for clarity * bug found in exported attachments (we used to miss some start/ends) * changes needed for actiance due to new generic exports * bringing CSV up to actiance standards * fixing global relay b/c of new semantics (adding a note on an edge case) * aligning e2e tests, adding comments to clarify what is expected/tested * necessary changes; 1 more test for added functionality (ignoreDeleted) * comment style * MM-62059 - Align Global Relay with Actiance/CSV; many fixes (#29665) * core logic changes to general export_data and the specific export paths * unit tests and e2e tests, covering all new edge cases and all logic * linting * better var naming, const value, and cleaning up functions calls * MM-62436 - Temporarily skip cypress tests that require download link (#29772) --------- Co-authored-by: Claudio Costa <cstcld91@gmail.com>
2025-01-10 16:56:02 -05:00
--- a/server/channels/store/sqlstore/channel_member_history_store.go
+++ b/server/channels/store/sqlstore/channel_member_history_store.go
@@ -65,27 +65,78 @@ func (s SqlChannelMemberHistoryStore) LogLeaveEvent(userId string, channelId str
 	return nil
 }

-func (s SqlChannelMemberHistoryStore) GetUsersInChannelDuring(startTime int64, endTime int64, channelId string) ([]*model.ChannelMemberHistoryResult, error) {
+func (s SqlChannelMemberHistoryStore) GetChannelsWithActivityDuring(startTime int64, endTime int64) ([]string, error) {
+	// ChannelMemberHistory has been in production for long enough that we are assuming the export period
+	// starts after the ChannelMemberHistory table was first introduced
+	subqueryPosts := s.getSubQueryBuilder().
+		Select("p.ChannelId").
+		Distinct().
+		From("Posts AS p").
+		Where(
+			sq.And{
+				sq.GtOrEq{"p.UpdateAt": startTime},
+				sq.LtOrEq{"p.UpdateAt": endTime},
+				sq.NotLike{"p.Type": "system_%"},
+			})
+
+	subqueryCMH := s.getSubQueryBuilder().
+		Select("cmh.ChannelId").
+		Distinct().
+		From("ChannelMemberHistory AS cmh").
+		Where(
+			sq.Or{
+				sq.And{
+					sq.GtOrEq{"cmh.JoinTime": startTime},
+					sq.LtOrEq{"cmh.JoinTime": endTime},
+				},
+				sq.And{
+					sq.GtOrEq{"cmh.LeaveTime": startTime},
+					sq.LtOrEq{"cmh.LeaveTime": endTime},
+				},
+			})
+
+	unionExpr, args, err := sq.Expr("(? UNION ?) AS cm", subqueryPosts, subqueryCMH).ToSql()
+	if err != nil {
+		return nil, errors.Wrap(err, "GetChannelsWithActivityDuring unionExpr to sql")
+	}
+
+	// no bound args in this expression
+	query, _, err := s.getQueryBuilder().
+		Select("*").
+		From(unionExpr).ToSql()
+	if err != nil {
+		return nil, errors.Wrap(err, "GetChannelsWithActivityDuring query to sql")
+	}
+
+	channelIds := make([]string, 0)
+	if err := s.GetReplica().Select(&channelIds, query, args...); err != nil {
+		return nil, err
+	}
+
+	return channelIds, nil
+}
+
+func (s SqlChannelMemberHistoryStore) GetUsersInChannelDuring(startTime int64, endTime int64, channelIds []string) ([]*model.ChannelMemberHistoryResult, error) {
 	useChannelMemberHistory, err := s.hasDataAtOrBefore(startTime)
 	if err != nil {
-		return nil, errors.Wrapf(err, "hasDataAtOrBefore startTime=%d endTime=%d channelId=%s", startTime, endTime, channelId)
+		return nil, errors.Wrapf(err, "hasDataAtOrBefore startTime=%d endTime=%d channelId=%v", startTime, endTime, channelIds)
 	}

 	if useChannelMemberHistory {
 		// the export period starts after the ChannelMemberHistory table was first introduced, so we can use the
 		// data from it for our export
-		channelMemberHistories, err2 := s.getFromChannelMemberHistoryTable(startTime, endTime, channelId)
+		channelMemberHistories, err2 := s.getFromChannelMemberHistoryTable(startTime, endTime, channelIds)
 		if err2 != nil {
-			return nil, errors.Wrapf(err2, "getFromChannelMemberHistoryTable startTime=%d endTime=%d channelId=%s", startTime, endTime, channelId)
+			return nil, errors.Wrapf(err2, "getFromChannelMemberHistoryTable startTime=%d endTime=%d channelId=%v", startTime, endTime, channelIds)
 		}
 		return channelMemberHistories, nil
 	}
 	// the export period starts before the ChannelMemberHistory table was introduced, so we need to fake the
 	// data by assuming that anybody who has ever joined the channel in question was present during the export period.
 	// this may not always be true, but it's better than saying that somebody wasn't there when they were
-	channelMemberHistories, err := s.getFromChannelMembersTable(startTime, endTime, channelId)
+	channelMemberHistories, err := s.getFromChannelMembersTable(startTime, endTime, channelIds)
 	if err != nil {
-		return nil, errors.Wrapf(err, "getFromChannelMembersTable startTime=%d endTime=%d channelId=%s", startTime, endTime, channelId)
+		return nil, errors.Wrapf(err, "getFromChannelMembersTable startTime=%d endTime=%d channelId=%v", startTime, endTime, channelIds)
 	}
 	return channelMemberHistories, nil
 }
@@ -108,14 +159,14 @@ func (s SqlChannelMemberHistoryStore) hasDataAtOrBefore(time int64) (bool, error
 	return false, nil
 }

-func (s SqlChannelMemberHistoryStore) getFromChannelMemberHistoryTable(startTime int64, endTime int64, channelId string) ([]*model.ChannelMemberHistoryResult, error) {
+func (s SqlChannelMemberHistoryStore) getFromChannelMemberHistoryTable(startTime int64, endTime int64, channelIds []string) ([]*model.ChannelMemberHistoryResult, error) {
 	query, args, err := s.getQueryBuilder().
 		Select(`cmh.*, u.Email AS "Email", u.Username, Bots.UserId IS NOT NULL AS IsBot, u.DeleteAt AS UserDeleteAt`).
 		From("ChannelMemberHistory cmh").
 		Join("Users u ON cmh.UserId = u.Id").
 		LeftJoin("Bots ON Bots.UserId = u.Id").
 		Where(sq.And{
-			sq.Eq{"cmh.ChannelId": channelId},
+			sq.Eq{"cmh.ChannelId": channelIds},
 			sq.LtOrEq{"cmh.JoinTime": endTime},
 			sq.Or{
 				sq.Eq{"cmh.LeaveTime": nil},
@@ -134,14 +185,14 @@ func (s SqlChannelMemberHistoryStore) getFromChannelMemberHistoryTable(startTime
 	return histories, nil
 }

-func (s SqlChannelMemberHistoryStore) getFromChannelMembersTable(startTime int64, endTime int64, channelId string) ([]*model.ChannelMemberHistoryResult, error) {
+func (s SqlChannelMemberHistoryStore) getFromChannelMembersTable(startTime int64, endTime int64, channelIds []string) ([]*model.ChannelMemberHistoryResult, error) {
 	query, args, err := s.getQueryBuilder().
 		Select(`ch.ChannelId, ch.UserId, u.Email AS "Email", u.Username, Bots.UserId IS NOT NULL AS IsBot, u.DeleteAt AS UserDeleteAt`).
 		Distinct().
 		From("ChannelMembers ch").
 		Join("Users u ON ch.UserId = u.id").
 		LeftJoin("Bots ON Bots.UserId = u.id").
-		Where(sq.Eq{"ch.ChannelId": channelId}).ToSql()
+		Where(sq.Eq{"ch.ChannelId": channelIds}).ToSql()
 	if err != nil {
 		return nil, errors.Wrap(err, "channel_member_history_to_sql")
 	}
--- a/server/channels/store/sqlstore/compliance_store.go
+++ b/server/channels/store/sqlstore/compliance_store.go
@@ -286,24 +286,25 @@ func (s SqlComplianceStore) MessageExport(c request.CTX, cursor model.MessageExp
 		return nil, cursor, errors.Wrap(caseErr, "unable to construct case statement")
 	}

-	query, args, err := s.getQueryBuilder().Select(`Posts.Id AS PostId,
-			Posts.CreateAt AS PostCreateAt,
-			Posts.UpdateAt AS PostUpdateAt,
-			Posts.DeleteAt AS PostDeleteAt,
-			Posts.Message AS PostMessage,
-			Posts.Type AS PostType,
-			Posts.Props AS PostProps,
-			Posts.OriginalId AS PostOriginalId,
-			Posts.RootId AS PostRootId,
-			Posts.FileIds AS PostFileIds,
-			Teams.Id AS TeamId,
-			Teams.Name AS TeamName,
+	builder := s.getQueryBuilder().Select(`Posts.Id AS PostId,
+			Posts.CreateAt    AS PostCreateAt,
+			Posts.UpdateAt    AS PostUpdateAt,
+			Posts.DeleteAt    AS PostDeleteAt,
+            Posts.EditAt      AS PostEditAt,
+			Posts.Message     AS PostMessage,
+			Posts.Type        AS PostType,
+			Posts.Props       AS PostProps,
+			Posts.OriginalId  AS PostOriginalId,
+			Posts.RootId      AS PostRootId,
+			Posts.FileIds     AS PostFileIds,
+			Teams.Id          AS TeamId,
+			Teams.Name        AS TeamName,
 			Teams.DisplayName AS TeamDisplayName,
-			Channels.Id AS ChannelId,
-			Channels.Name AS ChannelName,
-			Channels.Type AS ChannelType,
-			Users.Id AS UserId,
-			Users.Email AS UserEmail,
+			Channels.Id       AS ChannelId,
+			Channels.Name     AS ChannelName,
+			Channels.Type     AS ChannelType,
+			Users.Id          AS UserId,
+			Users.Email       AS UserEmail,
 			Users.Username,
 			Bots.UserId IS NOT NULL AS IsBot`).
 		Column(caseStmt+" AS ChannelDisplayName", caseArgs...).
@@ -323,8 +324,13 @@ func (s SqlComplianceStore) MessageExport(c request.CTX, cursor model.MessageExp
 			sq.NotLike{"Posts.Type": "system_%"},
 		}).
 		OrderBy("PostUpdateAt, PostId").
-		Limit(uint64(limit)).
-		ToSql()
+		Limit(uint64(limit))
+
+	if cursor.UntilUpdateAt > 0 {
+		builder = builder.Where(sq.LtOrEq{"Posts.UpdateAt": cursor.UntilUpdateAt})
+	}
+
+	query, args, err := builder.ToSql()
 	if err != nil {
 		return nil, cursor, errors.Wrap(err, "unable to construct query to export messages")
 	}
--- a/server/channels/store/sqlstore/post_store.go
+++ b/server/channels/store/sqlstore/post_store.go
@@ -2404,15 +2404,14 @@ func (s *SqlPostStore) AnalyticsPostCount(options *model.PostCountOptions) (int6
 		})
 	}

-	queryString, args, err := query.ToSql()
-	if err != nil {
-		return 0, errors.Wrap(err, "post_tosql")
+	if options.UntilUpdateAt > 0 {
+		query = query.Where(sq.LtOrEq{"p.UpdateAt": options.UntilUpdateAt})
 	}

 	var v int64
-	err = s.GetReplica().Get(&v, queryString, args...)
+	err := s.GetReplica().GetBuilder(&v, query)
 	if err != nil {
-		return 0, errors.Wrap(err, "failed to count Posts")
+		return 0, fmt.Errorf("post_tosql failed or failed to count Posts: %w", err)
 	}

 	return v, nil
--- a/server/channels/store/sqlstore/store.go
+++ b/server/channels/store/sqlstore/store.go
@@ -35,6 +35,8 @@ import (

 type migrationDirection string

+type Option func(s *SqlStore) error
+
 const (
 	IndexTypeFullText                 = "full_text"
 	IndexTypeFullTextFunc             = "full_text_func"
@@ -139,12 +141,20 @@ type SqlStore struct {

 	isBinaryParam             bool
 	pgDefaultTextSearchConfig string
+	skipMigrations            bool

 	quitMonitor chan struct{}
 	wgMonitor   *sync.WaitGroup
 }

-func New(settings model.SqlSettings, logger mlog.LoggerIFace, metrics einterfaces.MetricsInterface) (*SqlStore, error) {
+func SkipMigrations() Option {
+	return func(s *SqlStore) error {
+		s.skipMigrations = true
+		return nil
+	}
+}
+
+func New(settings model.SqlSettings, logger mlog.LoggerIFace, metrics einterfaces.MetricsInterface, options ...Option) (*SqlStore, error) {
 	store := &SqlStore{
 		rrCounter:   0,
 		srCounter:   0,
@@ -155,6 +165,12 @@ func New(settings model.SqlSettings, logger mlog.LoggerIFace, metrics einterface
 		wgMonitor:   &sync.WaitGroup{},
 	}

+	for _, option := range options {
+		if err := option(store); err != nil {
+			return nil, fmt.Errorf("failed to apply option: %w", err)
+		}
+	}
+
 	err := store.initConnection()
 	if err != nil {
 		return nil, errors.Wrap(err, "error setting up connections")
@@ -178,9 +194,11 @@ func New(settings model.SqlSettings, logger mlog.LoggerIFace, metrics einterface
 		return nil, errors.Wrap(err, "error while checking DB collation")
 	}

-	err = store.migrate(migrationsDirectionUp, false)
-	if err != nil {
-		return nil, errors.Wrap(err, "failed to apply database migrations")
+	if !store.skipMigrations {
+		err = store.migrate(migrationsDirectionUp, false)
+		if err != nil {
+			return nil, errors.Wrap(err, "failed to apply database migrations")
+		}
 	}

 	store.isBinaryParam, err = store.computeBinaryParam()
--- a/server/channels/store/sqlstore/store_test.go
+++ b/server/channels/store/sqlstore/store_test.go
@@ -4,6 +4,7 @@
 package sqlstore

 import (
+	"database/sql"
 	"fmt"
 	"os"
 	"path/filepath"
@@ -977,3 +978,40 @@ func TestGetAppliedMigrations(t *testing.T) {
 		})
 	}
 }
+
+func TestSkipMigrationsOption(t *testing.T) {
+	testDrivers := []string{
+		model.DatabaseDriverPostgres,
+		model.DatabaseDriverMysql,
+	}
+
+	logger := mlog.CreateConsoleTestLogger(t)
+
+	for _, d := range testDrivers {
+		driver := d
+		t.Run("Should not apply migrations when skipMigrations=true for "+driver, func(t *testing.T) {
+			t.Parallel()
+			settings, err := makeSqlSettings(driver)
+			if err != nil {
+				t.Skip(err)
+			}
+
+			store, err := New(*settings, logger, nil, SkipMigrations())
+			require.NoError(t, err)
+
+			_, err = store.GetDBSchemaVersion()
+			assert.Error(t, err)
+
+			// Unfortunately there's no error for table doesn't exist, but we know it won't be ErrNoRows:
+			assert.True(t, !errors.Is(err, sql.ErrNoRows))
+
+			// And we know what each db will return:
+			if driver == model.DatabaseDriverPostgres {
+				assert.Contains(t, err.Error(), "pq: relation \"db_migrations\" does not exist")
+			} else {
+				// Mysql includes the random db name, so test the end:
+				assert.Contains(t, err.Error(), ".db_migrations' doesn't exist")
+			}
+		})
+	}
+}