MM-59966 - Compliance Export overhaul - feature branch (#29789)
* [MM-59089] Add a compliance export constant (#27919) * add a useful constant * i18n * another constant * another i18n * [MM-60422] Add GetChannelsWithActivityDuring (#28301) * modify GetUsersInChannelDuring to accept a slice of channelIds * add GetChannelsWithActivityDuring * add compliance export progress message; remove unused custom status * linting * tests running too fast * add batch size config settings * add store tests * linting * empty commit * i18n changes * fix i18n ordering * MM-60570 - Server-side changes consolidating the export CLI with server/ent code (#28640) * add an i18n field; add the CLI's export directory * int64 -> int * Add UntilUpdateAt for MessageExport and AnalyticsPostCount to merge * remove now-unused i18n strings * add TranslationsPreInitFromBuffer to allow CLI to use i18n * use GetBuilder to simplify; rename TranslationsPreInitFromFileBytes * [MM-59089] Improve compliance export timings (#1733 - Enterprise repo) * MM-60422 - Performance and logic fixes for Compliance Exports (#1757 - Enterprise repo) * MM-60570 - Enterprise-side changes consolidating the export CLI with server/ent code (#1769 - Enterprise repo) * merge conflicts; missed file from ent branch * MM-61038 - Add an option to sqlstore.New (#28702) remove useless comment add test add an option to sqlstore.New * MM-60976: Remove RunExport command from Mattermost binary (#28805) * remove RunExport command from mattermost binary * remove the code it was calling * fix i18n * remove test (was only testing license, not functionality) * empty commit * fix flaky GetChannelsWithActivityDuring test * MM-60063: Dedicated Export Filestore fix, redo of #1772 (enterprise) (#28803) * redo filestore fix #1772 (enterprise repo) on top of MM-59966 feature * add new e2e tests for export filestore * golint * ok, note to self: shadowing bad, actually (when there's a defer) * empty commit * MM-61137 - Message export: Support 7.8.11 era dbs (#28824) * support 7.8.11 era dbs by wrapping the store using only what we need * fix flaky GetChannelsWithActivityDuring test * add a comment * only need to define the MEFileInfoStore (the one that'll be overridden) * blank commit * MM-60974 - Message Export: Add performance metrics (#28836) * support 7.8.11 era dbs by wrapping the store using only what we need * fix flaky GetChannelsWithActivityDuring test * add a comment * only need to define the MEFileInfoStore (the one that'll be overridden) * performance metrics * cleanup unneeded named returns * blank commit * MM-60975 - Message export: Add startTime and endTime to export folder name (#28840) * support 7.8.11 era dbs by wrapping the store using only what we need * fix flaky GetChannelsWithActivityDuring test * add a comment * only need to define the MEFileInfoStore (the one that'll be overridden) * performance metrics * output startTime and endTime in export folder * empty commit * merge conflict * MM-60978 - Message export: Improve xml fields; fix delete semantics (#28873) * support 7.8.11 era dbs by wrapping the store using only what we need * fix flaky GetChannelsWithActivityDuring test * add a comment * only need to define the MEFileInfoStore (the one that'll be overridden) * performance metrics * output startTime and endTime in export folder * empty commit * add xml fields, omit when empty, tests * fix delete semantics; test (and test for update semantics) * clarify comments * simplify edited post detection, now there's no edge case. * add some spacing to help fast running tests * merge conflicts/updates needed for new deleted post semantics * linting; fixing tests from upstream merge * use SafeDereference * linting * stronger typing; better wrapped errors; better formatting * blank commit * goimports formatting * fix merge mistake * minor fixes due to changes in master * MM-61755 - Simplifying and Support reporting to the db from the CLI (#29281) * finally clean up JobData struct and stringMap; prep for CLI using db * and now simplify using StringMapToJobDataWithZeroValues * remove unused fn * create JobDataExported; clean up errors * MM-60176 - Message Export: Global relay cleanup (#29168) * move global relay logic into global_relay_export * blank commit * blank commit * improve errors * MM-60693 - Refactor CSV to use same codepath as Actiance (#29191) * move global relay logic into global_relay_export * blank commit * refactor (and simplify) ExportParams into shared * blank commit * remove unused fn * csv now uses pre-calculated joins/leaves like actiance * improve errors * remove nil post check; remove ignoredPosts metric * remove unneeded copy * MM-61696 - Refactor GlobalRelay to use same codepath as Actiance (#29225) * move global relay logic into global_relay_export * blank commit * refactor (and simplify) ExportParams into shared * blank commit * remove unused fn * csv now uses pre-calculated joins/leaves like actiance * remove newly unneeded function and its test. goodbye. * refactor GetPostAttachments for csv + global relay to share * refactor global_relay_export and fix tests (no changes to output) * improve errors * remove nil post check; remove ignoredPosts metric * remove unneeded copy * remove unneeded nil check * PR comments * MM-61715 - Generalize e2e to all export types 🤖 (#29369) * move global relay logic into global_relay_export * blank commit * refactor (and simplify) ExportParams into shared * blank commit * remove unused fn * csv now uses pre-calculated joins/leaves like actiance * remove newly unneeded function and its test. goodbye. * refactor GetPostAttachments for csv + global relay to share * refactor global_relay_export and fix tests (no changes to output) * improve errors * remove nil post check; remove ignoredPosts metric * remove unneeded copy * remove unneeded nil check * PR comments * refactor isDeletedMsg for all export types * fix start and endtime, nasty csv createAt bug; bring closer to Actiance * align unit tests with new logic (e.g. starttime / endtime) * refactor a TimestampConvert fn for code + tests * bug: pass templates to global relay (hurray for e2e tests, otherwise...) * add global relay zip to allowed list (only for tests) * test helpers * new templates for e2e tests * e2e tests... phew. * linting * merge conflicts * unexport PostToRow; add test helper marker * cleanup, shortening, thanks to PR comments * MM-61972 - Generalize export data path - Actiance (#29399) * extract and generalize the export data generation functions * finish moving test (bc of previous extraction) * lift a function from common -> shared (to break an import cycle) * actiance now takes general export data, processes it into actiance data * bring tests in line with correct sorting rules (upadateAt, messageId) * fixups, PR comments * turn strings.Repeat into a more descriptive const amended: one letter fix; bad rebase * MM-62009 - e2e clock heisenbug (#29434) * consolidate assertions; output debuggable diffs (keeping for future) * refactor test output generator to generators file * waitUntilZeroPosts + pass through until to job = fix all clock issues * simplify messages to model.NewId(); remove unneeded waitUntilZeroPosts * model.NewId() -> storetest.NewTestID() * MM-61980 - Generalize export data path - CSV (#29482) * simple refactoring * increase sleep times for (very) rare test failures * add extra information to the generic export for CSV * adj Actiance to handle new generic export (no difference in its output) * no longer need mergePosts (yay), move getJoinLeavePosts for everyone * adjust tests for new csv semantics (detailed in summary) * and need to add the new exported data to the export_data_tests * rearrange csv writing to happen after data export (more logical) * linting * remove debug statements * figured out what was wrong with global relay e2e test 3; solid now * PR comments * MM-61718 - Generalize export data path - Global Relay (#29508) * move global relay over to using the generalized export data * performance pass -- not much can be done * Update server/enterprise/message_export/global_relay_export/global_relay_export.go Co-authored-by: Claudio Costa <cstcld91@gmail.com> --------- Co-authored-by: Claudio Costa <cstcld91@gmail.com> * MM-62058 - Align CSV with Actiance (#29551) * refactoring actiance files and var names for clarity * bug found in exported attachments (we used to miss some start/ends) * changes needed for actiance due to new generic exports * bringing CSV up to actiance standards * fixing global relay b/c of new semantics (adding a note on an edge case) * aligning e2e tests, adding comments to clarify what is expected/tested * necessary changes; 1 more test for added functionality (ignoreDeleted) * comment style * MM-62059 - Align Global Relay with Actiance/CSV; many fixes (#29665) * core logic changes to general export_data and the specific export paths * unit tests and e2e tests, covering all new edge cases and all logic * linting * better var naming, const value, and cleaning up functions calls * MM-62436 - Temporarily skip cypress tests that require download link (#29772) --------- Co-authored-by: Claudio Costa <cstcld91@gmail.com>
Этот коммит содержится в:
Christopher Poile
коммит произвёл
GitHub
GitHub
родитель
96195f1bd7
Коммит
aba4434dab
@@ -4,14 +4,14 @@
|
||||
package message_export
|
||||
|
||||
import (
|
||||
"archive/zip"
|
||||
"bytes"
|
||||
"context"
|
||||
"net/http"
|
||||
"errors"
|
||||
"fmt"
|
||||
"os"
|
||||
"path"
|
||||
"strconv"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/stretchr/testify/assert"
|
||||
tmock "github.com/stretchr/testify/mock"
|
||||
@@ -24,10 +24,10 @@ import (
|
||||
"github.com/mattermost/mattermost/server/v8/channels/app"
|
||||
"github.com/mattermost/mattermost/server/v8/channels/jobs"
|
||||
"github.com/mattermost/mattermost/server/v8/channels/store/storetest"
|
||||
st "github.com/mattermost/mattermost/server/v8/channels/store/storetest"
|
||||
"github.com/mattermost/mattermost/server/v8/channels/utils/testutils"
|
||||
"github.com/mattermost/mattermost/server/v8/einterfaces/mocks"
|
||||
"github.com/mattermost/mattermost/server/v8/platform/shared/filestore"
|
||||
fmocks "github.com/mattermost/mattermost/server/v8/platform/shared/filestore/mocks"
|
||||
"github.com/mattermost/mattermost/server/v8/enterprise/message_export/shared"
|
||||
)
|
||||
|
||||
func TestInitJobDataNoJobData(t *testing.T) {
|
||||
@@ -36,14 +36,14 @@ func TestInitJobDataNoJobData(t *testing.T) {
|
||||
defer mockStore.AssertExpectations(t)
|
||||
|
||||
job := &model.Job{
|
||||
Id: model.NewId(),
|
||||
Id: st.NewTestID(),
|
||||
CreateAt: model.GetMillis(),
|
||||
Status: model.JobStatusPending,
|
||||
Type: model.JobTypeMessageExport,
|
||||
}
|
||||
|
||||
// mock job store doesn't return a previously successful job, forcing fallback to config
|
||||
mockStore.JobStore.On("GetNewestJobByStatusesAndType", []string{model.JobStatusWarning, model.JobStatusSuccess}, model.JobTypeMessageExport).Return(nil, model.NewAppError("", "", nil, "", http.StatusBadRequest))
|
||||
mockStore.JobStore.On("GetNewestJobByStatusesAndType", []string{model.JobStatusWarning, model.JobStatusSuccess}, model.JobTypeMessageExport).Return(nil, errors.New("test"))
|
||||
|
||||
worker := &MessageExportWorker{
|
||||
jobServer: &jobs.JobServer{
|
||||
@@ -52,11 +52,13 @@ func TestInitJobDataNoJobData(t *testing.T) {
|
||||
Cfg: &model.Config{
|
||||
// mock config
|
||||
MessageExportSettings: model.MessageExportSettings{
|
||||
EnableExport: model.NewPointer(true),
|
||||
ExportFormat: model.NewPointer(model.ComplianceExportTypeActiance),
|
||||
DailyRunTime: model.NewPointer("01:00"),
|
||||
ExportFromTimestamp: model.NewPointer(int64(0)),
|
||||
BatchSize: model.NewPointer(10000),
|
||||
EnableExport: model.NewPointer(true),
|
||||
ExportFormat: model.NewPointer(model.ComplianceExportTypeActiance),
|
||||
DailyRunTime: model.NewPointer("01:00"),
|
||||
ExportFromTimestamp: model.NewPointer(int64(0)),
|
||||
BatchSize: model.NewPointer(10000),
|
||||
ChannelBatchSize: model.NewPointer(100),
|
||||
ChannelHistoryBatchSize: model.NewPointer(100),
|
||||
},
|
||||
},
|
||||
},
|
||||
@@ -64,12 +66,14 @@ func TestInitJobDataNoJobData(t *testing.T) {
|
||||
logger: logger,
|
||||
}
|
||||
|
||||
// actually execute the code under test
|
||||
worker.initJobData(logger, job)
|
||||
now := time.Now()
|
||||
worker.initJobData(logger, job, now)
|
||||
|
||||
assert.Equal(t, model.ComplianceExportTypeActiance, job.Data[JobDataExportType])
|
||||
assert.Equal(t, strconv.Itoa(*worker.jobServer.Config().MessageExportSettings.BatchSize), job.Data[JOB_DATA_BatchSize])
|
||||
assert.Equal(t, strconv.FormatInt(*worker.jobServer.Config().MessageExportSettings.ExportFromTimestamp, 10), job.Data[JobDataBatchStartTimestamp])
|
||||
assert.Equal(t, model.ComplianceExportTypeActiance, job.Data[shared.JobDataExportType])
|
||||
assert.Equal(t, strconv.Itoa(*worker.jobServer.Config().MessageExportSettings.BatchSize), job.Data[shared.JobDataBatchSize])
|
||||
assert.Equal(t, strconv.FormatInt(*worker.jobServer.Config().MessageExportSettings.ExportFromTimestamp, 10), job.Data[shared.JobDataBatchStartTime])
|
||||
expectedDir := path.Join(model.ComplianceExportPath, fmt.Sprintf("%s-%d-%d", now.Format(model.ComplianceExportDirectoryFormat), 0, now.UnixMilli()))
|
||||
assert.Equal(t, expectedDir, job.Data[shared.JobDataExportDir])
|
||||
}
|
||||
|
||||
func TestInitJobDataPreviousJobNoJobData(t *testing.T) {
|
||||
@@ -78,7 +82,7 @@ func TestInitJobDataPreviousJobNoJobData(t *testing.T) {
|
||||
defer mockStore.AssertExpectations(t)
|
||||
|
||||
previousJob := &model.Job{
|
||||
Id: model.NewId(),
|
||||
Id: st.NewTestID(),
|
||||
CreateAt: model.GetMillis(),
|
||||
Status: model.JobStatusSuccess,
|
||||
Type: model.JobTypeMessageExport,
|
||||
@@ -87,7 +91,7 @@ func TestInitJobDataPreviousJobNoJobData(t *testing.T) {
|
||||
}
|
||||
|
||||
job := &model.Job{
|
||||
Id: model.NewId(),
|
||||
Id: st.NewTestID(),
|
||||
CreateAt: model.GetMillis(),
|
||||
Status: model.JobStatusPending,
|
||||
Type: model.JobTypeMessageExport,
|
||||
@@ -103,11 +107,13 @@ func TestInitJobDataPreviousJobNoJobData(t *testing.T) {
|
||||
Cfg: &model.Config{
|
||||
// mock config
|
||||
MessageExportSettings: model.MessageExportSettings{
|
||||
EnableExport: model.NewPointer(true),
|
||||
ExportFormat: model.NewPointer(model.ComplianceExportTypeActiance),
|
||||
DailyRunTime: model.NewPointer("01:00"),
|
||||
ExportFromTimestamp: model.NewPointer(int64(0)),
|
||||
BatchSize: model.NewPointer(10000),
|
||||
EnableExport: model.NewPointer(true),
|
||||
ExportFormat: model.NewPointer(model.ComplianceExportTypeActiance),
|
||||
DailyRunTime: model.NewPointer("01:00"),
|
||||
ExportFromTimestamp: model.NewPointer(int64(0)),
|
||||
BatchSize: model.NewPointer(10000),
|
||||
ChannelBatchSize: model.NewPointer(100),
|
||||
ChannelHistoryBatchSize: model.NewPointer(100),
|
||||
},
|
||||
},
|
||||
},
|
||||
@@ -115,12 +121,14 @@ func TestInitJobDataPreviousJobNoJobData(t *testing.T) {
|
||||
logger: logger,
|
||||
}
|
||||
|
||||
// actually execute the code under test
|
||||
worker.initJobData(logger, job)
|
||||
now := time.Now()
|
||||
worker.initJobData(logger, job, now)
|
||||
|
||||
assert.Equal(t, model.ComplianceExportTypeActiance, job.Data[JobDataExportType])
|
||||
assert.Equal(t, strconv.Itoa(*worker.jobServer.Config().MessageExportSettings.BatchSize), job.Data[JOB_DATA_BatchSize])
|
||||
assert.Equal(t, strconv.FormatInt(*worker.jobServer.Config().MessageExportSettings.ExportFromTimestamp, 10), job.Data[JobDataBatchStartTimestamp])
|
||||
assert.Equal(t, model.ComplianceExportTypeActiance, job.Data[shared.JobDataExportType])
|
||||
assert.Equal(t, strconv.Itoa(*worker.jobServer.Config().MessageExportSettings.BatchSize), job.Data[shared.JobDataBatchSize])
|
||||
assert.Equal(t, strconv.FormatInt(*worker.jobServer.Config().MessageExportSettings.ExportFromTimestamp, 10), job.Data[shared.JobDataBatchStartTime])
|
||||
expectedDir := path.Join(model.ComplianceExportPath, fmt.Sprintf("%s-%d-%d", now.Format(model.ComplianceExportDirectoryFormat), 0, now.UnixMilli()))
|
||||
assert.Equal(t, expectedDir, job.Data[shared.JobDataExportDir])
|
||||
}
|
||||
|
||||
func TestInitJobDataPreviousJobWithJobData(t *testing.T) {
|
||||
@@ -129,20 +137,21 @@ func TestInitJobDataPreviousJobWithJobData(t *testing.T) {
|
||||
defer mockStore.AssertExpectations(t)
|
||||
|
||||
previousJob := &model.Job{
|
||||
Id: model.NewId(),
|
||||
Id: st.NewTestID(),
|
||||
CreateAt: model.GetMillis(),
|
||||
Status: model.JobStatusSuccess,
|
||||
Type: model.JobTypeMessageExport,
|
||||
StartAt: model.GetMillis() - 1000,
|
||||
LastActivityAt: model.GetMillis() - 1000,
|
||||
Data: map[string]string{JobDataBatchStartTimestamp: "123"},
|
||||
Data: map[string]string{shared.JobDataBatchStartTime: "123"},
|
||||
}
|
||||
|
||||
job := &model.Job{
|
||||
Id: model.NewId(),
|
||||
Id: st.NewTestID(),
|
||||
CreateAt: model.GetMillis(),
|
||||
Status: model.JobStatusPending,
|
||||
Type: model.JobTypeMessageExport,
|
||||
Data: map[string]string{shared.JobDataExportDir: "this-is-the-export-dir"},
|
||||
}
|
||||
|
||||
// mock job store returns a previously successful job that has the config that we're looking for, so we use it
|
||||
@@ -155,11 +164,13 @@ func TestInitJobDataPreviousJobWithJobData(t *testing.T) {
|
||||
Cfg: &model.Config{
|
||||
// mock config
|
||||
MessageExportSettings: model.MessageExportSettings{
|
||||
EnableExport: model.NewPointer(true),
|
||||
ExportFormat: model.NewPointer(model.ComplianceExportTypeActiance),
|
||||
DailyRunTime: model.NewPointer("01:00"),
|
||||
ExportFromTimestamp: model.NewPointer(int64(0)),
|
||||
BatchSize: model.NewPointer(10000),
|
||||
EnableExport: model.NewPointer(true),
|
||||
ExportFormat: model.NewPointer(model.ComplianceExportTypeActiance),
|
||||
DailyRunTime: model.NewPointer("01:00"),
|
||||
ExportFromTimestamp: model.NewPointer(int64(0)),
|
||||
BatchSize: model.NewPointer(10000),
|
||||
ChannelBatchSize: model.NewPointer(100),
|
||||
ChannelHistoryBatchSize: model.NewPointer(100),
|
||||
},
|
||||
},
|
||||
},
|
||||
@@ -167,12 +178,14 @@ func TestInitJobDataPreviousJobWithJobData(t *testing.T) {
|
||||
logger: logger,
|
||||
}
|
||||
|
||||
// actually execute the code under test
|
||||
worker.initJobData(logger, job)
|
||||
now := time.Now()
|
||||
worker.initJobData(logger, job, now)
|
||||
|
||||
assert.Equal(t, model.ComplianceExportTypeActiance, job.Data[JobDataExportType])
|
||||
assert.Equal(t, strconv.Itoa(*worker.jobServer.Config().MessageExportSettings.BatchSize), job.Data[JOB_DATA_BatchSize])
|
||||
assert.Equal(t, previousJob.Data[JobDataBatchStartTimestamp], job.Data[JobDataBatchStartTimestamp])
|
||||
assert.Equal(t, model.ComplianceExportTypeActiance, job.Data[shared.JobDataExportType])
|
||||
assert.Equal(t, strconv.Itoa(*worker.jobServer.Config().MessageExportSettings.BatchSize), job.Data[shared.JobDataBatchSize])
|
||||
assert.Equal(t, previousJob.Data[shared.JobDataBatchStartTime], job.Data[shared.JobDataBatchStartTime])
|
||||
expectedDir := "this-is-the-export-dir"
|
||||
assert.Equal(t, expectedDir, job.Data[shared.JobDataExportDir])
|
||||
}
|
||||
|
||||
func TestDoJobNoPostsToExport(t *testing.T) {
|
||||
@@ -185,7 +198,7 @@ func TestDoJobNoPostsToExport(t *testing.T) {
|
||||
defer mockMetrics.AssertExpectations(t)
|
||||
|
||||
job := &model.Job{
|
||||
Id: model.NewId(),
|
||||
Id: st.NewTestID(),
|
||||
CreateAt: model.GetMillis(),
|
||||
Status: model.JobStatusPending,
|
||||
Type: model.JobTypeMessageExport,
|
||||
@@ -196,15 +209,19 @@ func TestDoJobNoPostsToExport(t *testing.T) {
|
||||
mockMetrics.On("IncrementJobActive", model.JobTypeMessageExport)
|
||||
|
||||
// no previous job, data will be loaded from config
|
||||
mockStore.JobStore.On("GetNewestJobByStatusesAndType", []string{model.JobStatusWarning, model.JobStatusSuccess}, model.JobTypeMessageExport).Return(nil, model.NewAppError("", "", nil, "", http.StatusBadRequest))
|
||||
mockStore.JobStore.On("GetNewestJobByStatusesAndType", []string{model.JobStatusWarning, model.JobStatusSuccess}, model.JobTypeMessageExport).Return(nil, errors.New("test"))
|
||||
|
||||
// no channels with activity
|
||||
mockStore.ChannelMemberHistoryStore.On("GetChannelsWithActivityDuring", mock.Anything, mock.Anything).
|
||||
Return(make([]string, 0), nil)
|
||||
|
||||
// no posts found to export
|
||||
mockStore.ComplianceStore.On("MessageExport", mock.Anything, mock.AnythingOfType("model.MessageExportCursor"), 10000).Return(
|
||||
mockStore.ComplianceStore.On("MessageExport", mock.Anything, mock.AnythingOfType("model.MessageExportCursor"), 10001).Return(
|
||||
make([]*model.MessageExport, 0), model.MessageExportCursor{}, nil,
|
||||
)
|
||||
|
||||
mockStore.PostStore.On("AnalyticsPostCount", mock.Anything).Return(
|
||||
int64(estimatedPostCount), nil,
|
||||
int64(shared.EstimatedPostCount), nil,
|
||||
)
|
||||
|
||||
// job completed successfully
|
||||
@@ -229,11 +246,13 @@ func TestDoJobNoPostsToExport(t *testing.T) {
|
||||
Directory: model.NewPointer(tempDir),
|
||||
},
|
||||
MessageExportSettings: model.MessageExportSettings{
|
||||
EnableExport: model.NewPointer(true),
|
||||
ExportFormat: model.NewPointer(model.ComplianceExportTypeActiance),
|
||||
DailyRunTime: model.NewPointer("01:00"),
|
||||
ExportFromTimestamp: model.NewPointer(int64(0)),
|
||||
BatchSize: model.NewPointer(10000),
|
||||
EnableExport: model.NewPointer(true),
|
||||
ExportFormat: model.NewPointer(model.ComplianceExportTypeActiance),
|
||||
DailyRunTime: model.NewPointer("01:00"),
|
||||
ExportFromTimestamp: model.NewPointer(int64(0)),
|
||||
BatchSize: model.NewPointer(10000),
|
||||
ChannelBatchSize: model.NewPointer(100),
|
||||
ChannelHistoryBatchSize: model.NewPointer(100),
|
||||
},
|
||||
},
|
||||
},
|
||||
@@ -258,7 +277,7 @@ func TestDoJobWithDedicatedExportBackend(t *testing.T) {
|
||||
defer mockMetrics.AssertExpectations(t)
|
||||
|
||||
job := &model.Job{
|
||||
Id: model.NewId(),
|
||||
Id: st.NewTestID(),
|
||||
CreateAt: model.GetMillis(),
|
||||
Status: model.JobStatusPending,
|
||||
Type: model.JobTypeMessageExport,
|
||||
@@ -269,15 +288,50 @@ func TestDoJobWithDedicatedExportBackend(t *testing.T) {
|
||||
mockMetrics.On("IncrementJobActive", model.JobTypeMessageExport)
|
||||
|
||||
// no previous job, data will be loaded from config
|
||||
mockStore.JobStore.On("GetNewestJobByStatusesAndType", []string{model.JobStatusWarning, model.JobStatusSuccess}, model.JobTypeMessageExport).Return(nil, model.NewAppError("", "", nil, "", http.StatusBadRequest))
|
||||
mockStore.JobStore.On("GetNewestJobByStatusesAndType", []string{model.JobStatusWarning, model.JobStatusSuccess}, model.JobTypeMessageExport).Return(nil, errors.New("test"))
|
||||
|
||||
// no posts found to export
|
||||
mockStore.ComplianceStore.On("MessageExport", mock.Anything, mock.AnythingOfType("model.MessageExportCursor"), 10000).Return(
|
||||
channelId := st.NewTestID()
|
||||
channelName := st.NewTestID()
|
||||
channelDisplayName := st.NewTestID()
|
||||
channelType := model.ChannelTypeOpen
|
||||
messages := []*model.MessageExport{
|
||||
{
|
||||
TeamId: model.NewPointer(st.NewTestID()),
|
||||
ChannelId: model.NewPointer(channelId),
|
||||
ChannelName: model.NewPointer(channelName),
|
||||
UserId: model.NewPointer(st.NewTestID()),
|
||||
UserEmail: model.NewPointer(st.NewTestID()),
|
||||
Username: model.NewPointer(st.NewTestID()),
|
||||
PostId: model.NewPointer(st.NewTestID()),
|
||||
PostCreateAt: model.NewPointer[int64](123),
|
||||
PostUpdateAt: model.NewPointer[int64](123),
|
||||
PostDeleteAt: model.NewPointer[int64](123),
|
||||
PostMessage: model.NewPointer(st.NewTestID()),
|
||||
},
|
||||
}
|
||||
|
||||
// need to export at least one post to make an export directory and file
|
||||
|
||||
mockStore.ChannelMemberHistoryStore.On("GetChannelsWithActivityDuring", mock.Anything, mock.Anything).
|
||||
Return([]string{*messages[0].ChannelId}, nil)
|
||||
mockStore.ChannelStore.On("GetMany", []string{channelId}, true).
|
||||
Return(model.ChannelList{{
|
||||
Id: channelId,
|
||||
DisplayName: channelDisplayName,
|
||||
Name: channelName,
|
||||
Type: channelType,
|
||||
}}, nil)
|
||||
|
||||
mockStore.ComplianceStore.On("MessageExport", mock.Anything, mock.AnythingOfType("model.MessageExportCursor"), 10001).Return(
|
||||
messages, model.MessageExportCursor{}, nil,
|
||||
).Once()
|
||||
mockStore.ComplianceStore.On("MessageExport", mock.Anything, mock.AnythingOfType("model.MessageExportCursor"), 10001).Return(
|
||||
make([]*model.MessageExport, 0), model.MessageExportCursor{}, nil,
|
||||
)
|
||||
).Once()
|
||||
mockStore.ChannelMemberHistoryStore.On("GetUsersInChannelDuring", mock.Anything, mock.Anything, mock.Anything).Return(nil, nil)
|
||||
|
||||
mockStore.PostStore.On("AnalyticsPostCount", mock.Anything).Return(
|
||||
int64(estimatedPostCount), nil,
|
||||
int64(1), nil,
|
||||
)
|
||||
|
||||
// job completed successfully
|
||||
@@ -309,11 +363,13 @@ func TestDoJobWithDedicatedExportBackend(t *testing.T) {
|
||||
ExportDirectory: model.NewPointer(tempDedicatedDir),
|
||||
},
|
||||
MessageExportSettings: model.MessageExportSettings{
|
||||
EnableExport: model.NewPointer(true),
|
||||
ExportFormat: model.NewPointer(model.ComplianceExportTypeActiance),
|
||||
DailyRunTime: model.NewPointer("01:00"),
|
||||
ExportFromTimestamp: model.NewPointer(int64(0)),
|
||||
BatchSize: model.NewPointer(10000),
|
||||
EnableExport: model.NewPointer(true),
|
||||
ExportFormat: model.NewPointer(model.ComplianceExportTypeActiance),
|
||||
DailyRunTime: model.NewPointer("01:00"),
|
||||
ExportFromTimestamp: model.NewPointer(int64(0)),
|
||||
BatchSize: model.NewPointer(10000),
|
||||
ChannelBatchSize: model.NewPointer(100),
|
||||
ChannelHistoryBatchSize: model.NewPointer(100),
|
||||
},
|
||||
},
|
||||
},
|
||||
@@ -347,7 +403,7 @@ func TestDoJobCancel(t *testing.T) {
|
||||
t.Cleanup(func() { mockMetrics.AssertExpectations(t) })
|
||||
|
||||
job := &model.Job{
|
||||
Id: model.NewId(),
|
||||
Id: st.NewTestID(),
|
||||
CreateAt: model.GetMillis(),
|
||||
Status: model.JobStatusPending,
|
||||
Type: model.JobTypeMessageExport,
|
||||
@@ -368,11 +424,13 @@ func TestDoJobCancel(t *testing.T) {
|
||||
Directory: model.NewPointer(tempDir),
|
||||
},
|
||||
MessageExportSettings: model.MessageExportSettings{
|
||||
EnableExport: model.NewPointer(true),
|
||||
ExportFormat: model.NewPointer(model.ComplianceExportTypeActiance),
|
||||
DailyRunTime: model.NewPointer("01:00"),
|
||||
ExportFromTimestamp: model.NewPointer(int64(0)),
|
||||
BatchSize: model.NewPointer(10000),
|
||||
EnableExport: model.NewPointer(true),
|
||||
ExportFormat: model.NewPointer(model.ComplianceExportTypeActiance),
|
||||
DailyRunTime: model.NewPointer("01:00"),
|
||||
ExportFromTimestamp: model.NewPointer(int64(0)),
|
||||
BatchSize: model.NewPointer(10000),
|
||||
ChannelBatchSize: model.NewPointer(100),
|
||||
ChannelHistoryBatchSize: model.NewPointer(100),
|
||||
},
|
||||
},
|
||||
},
|
||||
@@ -390,11 +448,26 @@ func TestDoJobCancel(t *testing.T) {
|
||||
mockMetrics.On("IncrementJobActive", model.JobTypeMessageExport)
|
||||
|
||||
// No previous job, data will be loaded from config
|
||||
mockStore.JobStore.On("GetNewestJobByStatusesAndType", []string{model.JobStatusWarning, model.JobStatusSuccess}, model.JobTypeMessageExport).Return(nil, model.NewAppError("", "", nil, "", http.StatusBadRequest))
|
||||
mockStore.JobStore.On("GetNewestJobByStatusesAndType", []string{model.JobStatusWarning, model.JobStatusSuccess}, model.JobTypeMessageExport).Return(nil, errors.New("test"))
|
||||
|
||||
// Job updates the system console UI, once for getting channels, once for getting activity
|
||||
mockStore.JobStore.On("UpdateOptimistically", mock.AnythingOfType("*model.Job"), model.JobStatusInProgress).Return(true, nil).Times(2)
|
||||
|
||||
// a few calls pass
|
||||
mockStore.ChannelMemberHistoryStore.On("GetChannelsWithActivityDuring", mock.Anything, mock.Anything).
|
||||
Return([]string{"channel-id"}, nil)
|
||||
mockStore.ChannelStore.On("GetMany", []string{"channel-id"}, true).
|
||||
Return(model.ChannelList{{
|
||||
Id: "channel-id",
|
||||
DisplayName: "channel-display-name",
|
||||
Name: "channel-name",
|
||||
Type: model.ChannelTypeDirect,
|
||||
}}, nil)
|
||||
mockStore.ChannelMemberHistoryStore.On("GetUsersInChannelDuring", mock.Anything, mock.Anything, []string{"channel-id"}).Return([]*model.ChannelMemberHistoryResult{}, nil)
|
||||
|
||||
cancelled := make(chan struct{})
|
||||
// Cancel the worker and return an error
|
||||
mockStore.ComplianceStore.On("MessageExport", mock.Anything, mock.AnythingOfType("model.MessageExportCursor"), 10000).Run(func(args tmock.Arguments) {
|
||||
mockStore.ComplianceStore.On("MessageExport", mock.Anything, mock.AnythingOfType("model.MessageExportCursor"), 10001).Run(func(args tmock.Arguments) {
|
||||
worker.cancel()
|
||||
|
||||
rctx, ok := args.Get(0).(request.CTX)
|
||||
@@ -408,7 +481,7 @@ func TestDoJobCancel(t *testing.T) {
|
||||
)
|
||||
|
||||
mockStore.PostStore.On("AnalyticsPostCount", mock.Anything).Return(
|
||||
int64(estimatedPostCount), nil,
|
||||
int64(shared.EstimatedPostCount), nil,
|
||||
)
|
||||
|
||||
// Job marked as pending
|
||||
@@ -425,75 +498,3 @@ func TestDoJobCancel(t *testing.T) {
|
||||
// Cleanup
|
||||
worker.Stop()
|
||||
}
|
||||
|
||||
func TestCreateZipFile(t *testing.T) {
|
||||
rctx := request.TestContext(t)
|
||||
|
||||
tempDir, ioErr := os.MkdirTemp("", "")
|
||||
require.NoError(t, ioErr)
|
||||
defer os.RemoveAll(tempDir)
|
||||
|
||||
config := filestore.FileBackendSettings{
|
||||
DriverName: model.ImageDriverLocal,
|
||||
Directory: tempDir,
|
||||
}
|
||||
|
||||
fileBackend, err := filestore.NewFileBackend(config)
|
||||
assert.NoError(t, err)
|
||||
_ = fileBackend
|
||||
|
||||
b := []byte("test")
|
||||
path1 := path.Join(exportPath, "19700101")
|
||||
path2 := path.Join(exportPath, "19800101/subdir")
|
||||
|
||||
// We test with a mock to test the Hitachi HCP case
|
||||
// where ListDirectory returns the dir itself as the first entry.
|
||||
// Note: If the mocks fail, that means the logic in createZipFile has
|
||||
// gone wrong and needs to be verified.
|
||||
mock := &fmocks.FileBackend{}
|
||||
defer mock.AssertExpectations(t)
|
||||
|
||||
mock.On("WriteFile", tmock.Anything, tmock.AnythingOfType("string")).Return(int64(4), nil)
|
||||
mock.On("FileSize", tmock.Anything).Return(int64(4), nil)
|
||||
mock.On("FileSize", tmock.Anything).Return(int64(4), nil)
|
||||
mock.On("Reader", path.Join(path1, "testid")).Return(mockReadSeekCloser{bytes.NewReader([]byte("test"))}, nil)
|
||||
mock.On("Reader", path.Join(path2, "testid")).Return(mockReadSeekCloser{bytes.NewReader([]byte("test"))}, nil)
|
||||
mock.On("ListDirectoryRecursively", path1).Return([]string{path1, path.Join(path1, "testid")}, nil)
|
||||
mock.On("ListDirectoryRecursively", path2).Return([]string{path2, path.Join(path2, "testid")}, nil)
|
||||
|
||||
for i, backend := range []filestore.FileBackend{fileBackend, mock} {
|
||||
written, err := backend.WriteFile(bytes.NewReader(b), path1+"/"+model.NewId())
|
||||
assert.NoError(t, err)
|
||||
assert.Equal(t, int64(len(b)), written)
|
||||
|
||||
written, err = backend.WriteFile(bytes.NewReader(b), path2+"/"+model.NewId())
|
||||
assert.NoError(t, err)
|
||||
assert.Equal(t, int64(len(b)), written)
|
||||
|
||||
written, err = backend.WriteFile(bytes.NewReader(b), path2+"/"+model.NewId())
|
||||
assert.NoError(t, err)
|
||||
assert.Equal(t, int64(len(b)), written)
|
||||
|
||||
err = createZipFile(rctx, backend, "testjob", []string{path1, path2})
|
||||
assert.NoError(t, err)
|
||||
|
||||
// Skip checking the zip file in mock case.
|
||||
if i == 1 {
|
||||
continue
|
||||
}
|
||||
r, err := zip.OpenReader(path.Join(tempDir, exportPath) + "/testjob.zip")
|
||||
assert.NoError(t, err)
|
||||
err = r.Close()
|
||||
require.NoError(t, err)
|
||||
|
||||
assert.Equal(t, 3, len(r.File))
|
||||
}
|
||||
}
|
||||
|
||||
type mockReadSeekCloser struct {
|
||||
*bytes.Reader
|
||||
}
|
||||
|
||||
func (r mockReadSeekCloser) Close() error {
|
||||
return nil
|
||||
}
|
||||
|
||||
Ссылка в новой задаче
Block a user