MM-59966 - Compliance Export overhaul - feature branch (#29789)
* [MM-59089] Add a compliance export constant (#27919) * add a useful constant * i18n * another constant * another i18n * [MM-60422] Add GetChannelsWithActivityDuring (#28301) * modify GetUsersInChannelDuring to accept a slice of channelIds * add GetChannelsWithActivityDuring * add compliance export progress message; remove unused custom status * linting * tests running too fast * add batch size config settings * add store tests * linting * empty commit * i18n changes * fix i18n ordering * MM-60570 - Server-side changes consolidating the export CLI with server/ent code (#28640) * add an i18n field; add the CLI's export directory * int64 -> int * Add UntilUpdateAt for MessageExport and AnalyticsPostCount to merge * remove now-unused i18n strings * add TranslationsPreInitFromBuffer to allow CLI to use i18n * use GetBuilder to simplify; rename TranslationsPreInitFromFileBytes * [MM-59089] Improve compliance export timings (#1733 - Enterprise repo) * MM-60422 - Performance and logic fixes for Compliance Exports (#1757 - Enterprise repo) * MM-60570 - Enterprise-side changes consolidating the export CLI with server/ent code (#1769 - Enterprise repo) * merge conflicts; missed file from ent branch * MM-61038 - Add an option to sqlstore.New (#28702) remove useless comment add test add an option to sqlstore.New * MM-60976: Remove RunExport command from Mattermost binary (#28805) * remove RunExport command from mattermost binary * remove the code it was calling * fix i18n * remove test (was only testing license, not functionality) * empty commit * fix flaky GetChannelsWithActivityDuring test * MM-60063: Dedicated Export Filestore fix, redo of #1772 (enterprise) (#28803) * redo filestore fix #1772 (enterprise repo) on top of MM-59966 feature * add new e2e tests for export filestore * golint * ok, note to self: shadowing bad, actually (when there's a defer) * empty commit * MM-61137 - Message export: Support 7.8.11 era dbs (#28824) * support 7.8.11 era dbs by wrapping the store using only what we need * fix flaky GetChannelsWithActivityDuring test * add a comment * only need to define the MEFileInfoStore (the one that'll be overridden) * blank commit * MM-60974 - Message Export: Add performance metrics (#28836) * support 7.8.11 era dbs by wrapping the store using only what we need * fix flaky GetChannelsWithActivityDuring test * add a comment * only need to define the MEFileInfoStore (the one that'll be overridden) * performance metrics * cleanup unneeded named returns * blank commit * MM-60975 - Message export: Add startTime and endTime to export folder name (#28840) * support 7.8.11 era dbs by wrapping the store using only what we need * fix flaky GetChannelsWithActivityDuring test * add a comment * only need to define the MEFileInfoStore (the one that'll be overridden) * performance metrics * output startTime and endTime in export folder * empty commit * merge conflict * MM-60978 - Message export: Improve xml fields; fix delete semantics (#28873) * support 7.8.11 era dbs by wrapping the store using only what we need * fix flaky GetChannelsWithActivityDuring test * add a comment * only need to define the MEFileInfoStore (the one that'll be overridden) * performance metrics * output startTime and endTime in export folder * empty commit * add xml fields, omit when empty, tests * fix delete semantics; test (and test for update semantics) * clarify comments * simplify edited post detection, now there's no edge case. * add some spacing to help fast running tests * merge conflicts/updates needed for new deleted post semantics * linting; fixing tests from upstream merge * use SafeDereference * linting * stronger typing; better wrapped errors; better formatting * blank commit * goimports formatting * fix merge mistake * minor fixes due to changes in master * MM-61755 - Simplifying and Support reporting to the db from the CLI (#29281) * finally clean up JobData struct and stringMap; prep for CLI using db * and now simplify using StringMapToJobDataWithZeroValues * remove unused fn * create JobDataExported; clean up errors * MM-60176 - Message Export: Global relay cleanup (#29168) * move global relay logic into global_relay_export * blank commit * blank commit * improve errors * MM-60693 - Refactor CSV to use same codepath as Actiance (#29191) * move global relay logic into global_relay_export * blank commit * refactor (and simplify) ExportParams into shared * blank commit * remove unused fn * csv now uses pre-calculated joins/leaves like actiance * improve errors * remove nil post check; remove ignoredPosts metric * remove unneeded copy * MM-61696 - Refactor GlobalRelay to use same codepath as Actiance (#29225) * move global relay logic into global_relay_export * blank commit * refactor (and simplify) ExportParams into shared * blank commit * remove unused fn * csv now uses pre-calculated joins/leaves like actiance * remove newly unneeded function and its test. goodbye. * refactor GetPostAttachments for csv + global relay to share * refactor global_relay_export and fix tests (no changes to output) * improve errors * remove nil post check; remove ignoredPosts metric * remove unneeded copy * remove unneeded nil check * PR comments * MM-61715 - Generalize e2e to all export types 🤖 (#29369) * move global relay logic into global_relay_export * blank commit * refactor (and simplify) ExportParams into shared * blank commit * remove unused fn * csv now uses pre-calculated joins/leaves like actiance * remove newly unneeded function and its test. goodbye. * refactor GetPostAttachments for csv + global relay to share * refactor global_relay_export and fix tests (no changes to output) * improve errors * remove nil post check; remove ignoredPosts metric * remove unneeded copy * remove unneeded nil check * PR comments * refactor isDeletedMsg for all export types * fix start and endtime, nasty csv createAt bug; bring closer to Actiance * align unit tests with new logic (e.g. starttime / endtime) * refactor a TimestampConvert fn for code + tests * bug: pass templates to global relay (hurray for e2e tests, otherwise...) * add global relay zip to allowed list (only for tests) * test helpers * new templates for e2e tests * e2e tests... phew. * linting * merge conflicts * unexport PostToRow; add test helper marker * cleanup, shortening, thanks to PR comments * MM-61972 - Generalize export data path - Actiance (#29399) * extract and generalize the export data generation functions * finish moving test (bc of previous extraction) * lift a function from common -> shared (to break an import cycle) * actiance now takes general export data, processes it into actiance data * bring tests in line with correct sorting rules (upadateAt, messageId) * fixups, PR comments * turn strings.Repeat into a more descriptive const amended: one letter fix; bad rebase * MM-62009 - e2e clock heisenbug (#29434) * consolidate assertions; output debuggable diffs (keeping for future) * refactor test output generator to generators file * waitUntilZeroPosts + pass through until to job = fix all clock issues * simplify messages to model.NewId(); remove unneeded waitUntilZeroPosts * model.NewId() -> storetest.NewTestID() * MM-61980 - Generalize export data path - CSV (#29482) * simple refactoring * increase sleep times for (very) rare test failures * add extra information to the generic export for CSV * adj Actiance to handle new generic export (no difference in its output) * no longer need mergePosts (yay), move getJoinLeavePosts for everyone * adjust tests for new csv semantics (detailed in summary) * and need to add the new exported data to the export_data_tests * rearrange csv writing to happen after data export (more logical) * linting * remove debug statements * figured out what was wrong with global relay e2e test 3; solid now * PR comments * MM-61718 - Generalize export data path - Global Relay (#29508) * move global relay over to using the generalized export data * performance pass -- not much can be done * Update server/enterprise/message_export/global_relay_export/global_relay_export.go Co-authored-by: Claudio Costa <cstcld91@gmail.com> --------- Co-authored-by: Claudio Costa <cstcld91@gmail.com> * MM-62058 - Align CSV with Actiance (#29551) * refactoring actiance files and var names for clarity * bug found in exported attachments (we used to miss some start/ends) * changes needed for actiance due to new generic exports * bringing CSV up to actiance standards * fixing global relay b/c of new semantics (adding a note on an edge case) * aligning e2e tests, adding comments to clarify what is expected/tested * necessary changes; 1 more test for added functionality (ignoreDeleted) * comment style * MM-62059 - Align Global Relay with Actiance/CSV; many fixes (#29665) * core logic changes to general export_data and the specific export paths * unit tests and e2e tests, covering all new edge cases and all logic * linting * better var naming, const value, and cleaning up functions calls * MM-62436 - Temporarily skip cypress tests that require download link (#29772) --------- Co-authored-by: Claudio Costa <cstcld91@gmail.com>
Этот коммит содержится в:
Christopher Poile
коммит произвёл
GitHub
GitHub
родитель
96195f1bd7
Коммит
aba4434dab
@@ -4,9 +4,9 @@
|
||||
package storetest
|
||||
|
||||
import (
|
||||
"testing"
|
||||
|
||||
"math"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
@@ -21,6 +21,7 @@ func TestChannelMemberHistoryStore(t *testing.T, rctx request.CTX, ss store.Stor
|
||||
t.Run("TestLogLeaveEvent", func(t *testing.T) { testLogLeaveEvent(t, rctx, ss) })
|
||||
t.Run("TestGetUsersInChannelAtChannelMemberHistory", func(t *testing.T) { testGetUsersInChannelAtChannelMemberHistory(t, rctx, ss) })
|
||||
t.Run("TestGetUsersInChannelAtChannelMembers", func(t *testing.T) { testGetUsersInChannelAtChannelMembers(t, rctx, ss) })
|
||||
t.Run("TestGetChannelsWithActivityDuring", func(t *testing.T) { testGetChannelsWithActivityDuring(t, rctx, ss) })
|
||||
t.Run("TestPermanentDeleteBatch", func(t *testing.T) { testPermanentDeleteBatch(t, rctx, ss) })
|
||||
t.Run("TestPermanentDeleteBatchForRetentionPolicies", func(t *testing.T) { testPermanentDeleteBatchForRetentionPolicies(t, rctx, ss) })
|
||||
t.Run("TestGetChannelsLeftSince", func(t *testing.T) { testGetChannelsLeftSince(t, rctx, ss) })
|
||||
@@ -81,6 +82,160 @@ func testLogLeaveEvent(t *testing.T, rctx request.CTX, ss store.Store) {
|
||||
assert.NoError(t, err)
|
||||
}
|
||||
|
||||
func testGetChannelsWithActivityDuring(t *testing.T, rctx request.CTX, ss store.Store) {
|
||||
// Need to wait to make sure channels and posts have nothing in them for this test.
|
||||
time.Sleep(101 * time.Millisecond)
|
||||
|
||||
// create three test channels
|
||||
ch1 := &model.Channel{
|
||||
TeamId: model.NewId(),
|
||||
DisplayName: "Display " + model.NewId(),
|
||||
Name: model.NewId(),
|
||||
Type: model.ChannelTypeOpen,
|
||||
}
|
||||
channel1, err := ss.Channel().Save(rctx, ch1, -1)
|
||||
require.NoError(t, err)
|
||||
|
||||
// channel2 will have no activity until case 6 (shouldn't show up until then)
|
||||
ch2 := &model.Channel{
|
||||
TeamId: model.NewId(),
|
||||
DisplayName: "Display " + model.NewId(),
|
||||
Name: model.NewId(),
|
||||
Type: model.ChannelTypeOpen,
|
||||
}
|
||||
channel2, err := ss.Channel().Save(rctx, ch2, -1)
|
||||
require.NoError(t, err)
|
||||
|
||||
// and two test users
|
||||
user1 := model.User{
|
||||
Email: MakeEmail(),
|
||||
Nickname: model.NewId(),
|
||||
Username: model.NewUsername(),
|
||||
}
|
||||
userPtr, err := ss.User().Save(rctx, &user1)
|
||||
require.NoError(t, err)
|
||||
user1 = *userPtr
|
||||
|
||||
user2 := model.User{
|
||||
Email: MakeEmail(),
|
||||
Nickname: model.NewId(),
|
||||
Username: model.NewUsername(),
|
||||
}
|
||||
userPtr, err = ss.User().Save(rctx, &user2)
|
||||
require.NoError(t, err)
|
||||
user2 = *userPtr
|
||||
|
||||
now := model.GetMillis()
|
||||
originalNow := now
|
||||
|
||||
// user2 joins channel2 before test
|
||||
err = ss.ChannelMemberHistory().LogJoinEvent(user2.Id, channel2.Id, now-2000)
|
||||
require.NoError(t, err)
|
||||
|
||||
// case 7: assert no activity for time period before channel activity
|
||||
channelIds, err := ss.ChannelMemberHistory().GetChannelsWithActivityDuring(originalNow-100, originalNow)
|
||||
require.NoError(t, err)
|
||||
assert.Empty(t, channelIds)
|
||||
|
||||
// case 1: user1 was in channel before period, doesn't show activity
|
||||
err = ss.ChannelMemberHistory().LogJoinEvent(user1.Id, channel1.Id, now-1100)
|
||||
require.NoError(t, err)
|
||||
|
||||
channelIds, err = ss.ChannelMemberHistory().GetChannelsWithActivityDuring(now, now+1000)
|
||||
require.NoError(t, err)
|
||||
assert.Empty(t, channelIds)
|
||||
|
||||
// case 2: user1 leaves, shows activity
|
||||
err = ss.ChannelMemberHistory().LogLeaveEvent(user1.Id, channel1.Id, now+1)
|
||||
require.NoError(t, err)
|
||||
|
||||
channelIds, err = ss.ChannelMemberHistory().GetChannelsWithActivityDuring(now, now+1000)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, channelIds, []string{channel1.Id})
|
||||
|
||||
// case 3: user1 joins, shows activity
|
||||
err = ss.ChannelMemberHistory().LogJoinEvent(user1.Id, channel1.Id, now+2)
|
||||
require.NoError(t, err)
|
||||
|
||||
channelIds, err = ss.ChannelMemberHistory().GetChannelsWithActivityDuring(now+2, now+1000)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, channelIds, []string{channel1.Id})
|
||||
|
||||
// case 4: new post shows activity
|
||||
channelIds, err = ss.ChannelMemberHistory().GetChannelsWithActivityDuring(now+3, now+1000)
|
||||
require.NoError(t, err)
|
||||
assert.Empty(t, channelIds)
|
||||
|
||||
post := &model.Post{
|
||||
ChannelId: channel1.Id,
|
||||
Message: "root post",
|
||||
UserId: user1.Id,
|
||||
CreateAt: now + 4,
|
||||
UpdateAt: now + 4,
|
||||
}
|
||||
post, err = ss.Post().Save(rctx, post)
|
||||
require.NoError(t, err)
|
||||
|
||||
channelIds, err = ss.ChannelMemberHistory().GetChannelsWithActivityDuring(now+3, now+1000)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, channelIds, []string{channel1.Id})
|
||||
|
||||
// case 5: update shows activity
|
||||
// need to wait because update uses getMillis
|
||||
time.Sleep(10 * time.Millisecond)
|
||||
now = model.GetMillis()
|
||||
|
||||
channelIds, err = ss.ChannelMemberHistory().GetChannelsWithActivityDuring(now-1, now+1000)
|
||||
require.NoError(t, err)
|
||||
assert.Empty(t, channelIds)
|
||||
|
||||
newPost := post.Clone()
|
||||
newPost.Message = "edited message"
|
||||
_, err = ss.Post().Update(rctx, newPost, post)
|
||||
require.NoError(t, err)
|
||||
|
||||
channelIds, err = ss.ChannelMemberHistory().GetChannelsWithActivityDuring(now-1, now+1000)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, channelIds, []string{channel1.Id})
|
||||
|
||||
// case 6: get both activity from posts and from join/leave;
|
||||
// - also, sql deduplicates two channel1 results (from post and channel history tables)
|
||||
time.Sleep(1 * time.Millisecond)
|
||||
now = model.GetMillis()
|
||||
|
||||
channelIds, err = ss.ChannelMemberHistory().GetChannelsWithActivityDuring(now, now+1000)
|
||||
require.NoError(t, err)
|
||||
assert.Empty(t, channelIds)
|
||||
|
||||
post2 := &model.Post{
|
||||
ChannelId: channel1.Id,
|
||||
Message: "root post",
|
||||
UserId: user1.Id,
|
||||
CreateAt: now + 11,
|
||||
UpdateAt: now + 11,
|
||||
}
|
||||
_, err = ss.Post().Save(rctx, post2)
|
||||
require.NoError(t, err)
|
||||
err = ss.ChannelMemberHistory().LogLeaveEvent(user1.Id, channel1.Id, now+12)
|
||||
require.NoError(t, err)
|
||||
err = ss.ChannelMemberHistory().LogLeaveEvent(user2.Id, channel2.Id, now+13)
|
||||
require.NoError(t, err)
|
||||
|
||||
channelIds, err = ss.ChannelMemberHistory().GetChannelsWithActivityDuring(now+10, now+1000)
|
||||
require.NoError(t, err)
|
||||
assert.ElementsMatch(t, channelIds, []string{channel1.Id, channel2.Id})
|
||||
|
||||
// case 7: still no activity for period before tests
|
||||
channelIds, err = ss.ChannelMemberHistory().GetChannelsWithActivityDuring(originalNow-100, originalNow)
|
||||
require.NoError(t, err)
|
||||
assert.Empty(t, channelIds)
|
||||
|
||||
// case 8: no activity for period after tests
|
||||
channelIds, err = ss.ChannelMemberHistory().GetChannelsWithActivityDuring(now+100, now+1000)
|
||||
require.NoError(t, err)
|
||||
assert.Empty(t, channelIds)
|
||||
}
|
||||
|
||||
func testGetUsersInChannelAtChannelMemberHistory(t *testing.T, rctx request.CTX, ss store.Store) {
|
||||
// create a test channel
|
||||
ch := &model.Channel{
|
||||
@@ -119,12 +274,12 @@ func testGetUsersInChannelAtChannelMemberHistory(t *testing.T, rctx request.CTX,
|
||||
require.NoError(t, err)
|
||||
|
||||
// case 1: user joins and leaves the channel before the export period begins
|
||||
channelMembers, err := ss.ChannelMemberHistory().GetUsersInChannelDuring(joinTime-500, joinTime-100, channel.Id)
|
||||
channelMembers, err := ss.ChannelMemberHistory().GetUsersInChannelDuring(joinTime-500, joinTime-100, []string{channel.Id})
|
||||
require.NoError(t, err)
|
||||
assert.Empty(t, channelMembers)
|
||||
|
||||
// case 2: user joins the channel after the export period begins, but has not yet left the channel when the export period ends
|
||||
channelMembers, err = ss.ChannelMemberHistory().GetUsersInChannelDuring(joinTime-100, joinTime+500, channel.Id)
|
||||
channelMembers, err = ss.ChannelMemberHistory().GetUsersInChannelDuring(joinTime-100, joinTime+500, []string{channel.Id})
|
||||
require.NoError(t, err)
|
||||
assert.Len(t, channelMembers, 1)
|
||||
assert.Equal(t, channel.Id, channelMembers[0].ChannelId)
|
||||
@@ -135,7 +290,7 @@ func testGetUsersInChannelAtChannelMemberHistory(t *testing.T, rctx request.CTX,
|
||||
assert.Nil(t, channelMembers[0].LeaveTime)
|
||||
|
||||
// case 3: user joins the channel before the export period begins, but has not yet left the channel when the export period ends
|
||||
channelMembers, err = ss.ChannelMemberHistory().GetUsersInChannelDuring(joinTime+100, joinTime+500, channel.Id)
|
||||
channelMembers, err = ss.ChannelMemberHistory().GetUsersInChannelDuring(joinTime+100, joinTime+500, []string{channel.Id})
|
||||
require.NoError(t, err)
|
||||
assert.Len(t, channelMembers, 1)
|
||||
assert.Equal(t, channel.Id, channelMembers[0].ChannelId)
|
||||
@@ -150,7 +305,7 @@ func testGetUsersInChannelAtChannelMemberHistory(t *testing.T, rctx request.CTX,
|
||||
require.NoError(t, err)
|
||||
|
||||
// case 4: user joins the channel before the export period begins, but has not yet left the channel when the export period ends
|
||||
channelMembers, err = ss.ChannelMemberHistory().GetUsersInChannelDuring(joinTime+100, leaveTime-100, channel.Id)
|
||||
channelMembers, err = ss.ChannelMemberHistory().GetUsersInChannelDuring(joinTime+100, leaveTime-100, []string{channel.Id})
|
||||
require.NoError(t, err)
|
||||
assert.Len(t, channelMembers, 1)
|
||||
assert.Equal(t, channel.Id, channelMembers[0].ChannelId)
|
||||
@@ -161,7 +316,7 @@ func testGetUsersInChannelAtChannelMemberHistory(t *testing.T, rctx request.CTX,
|
||||
assert.Equal(t, leaveTime, *channelMembers[0].LeaveTime)
|
||||
|
||||
// case 5: user joins the channel after the export period begins, and leaves the channel before the export period ends
|
||||
channelMembers, err = ss.ChannelMemberHistory().GetUsersInChannelDuring(joinTime-100, leaveTime+100, channel.Id)
|
||||
channelMembers, err = ss.ChannelMemberHistory().GetUsersInChannelDuring(joinTime-100, leaveTime+100, []string{channel.Id})
|
||||
require.NoError(t, err)
|
||||
assert.Len(t, channelMembers, 1)
|
||||
assert.Equal(t, channel.Id, channelMembers[0].ChannelId)
|
||||
@@ -172,7 +327,7 @@ func testGetUsersInChannelAtChannelMemberHistory(t *testing.T, rctx request.CTX,
|
||||
assert.Equal(t, leaveTime, *channelMembers[0].LeaveTime)
|
||||
|
||||
// case 6: user has joined and left the channel long before the export period begins
|
||||
channelMembers, err = ss.ChannelMemberHistory().GetUsersInChannelDuring(leaveTime+100, leaveTime+200, channel.Id)
|
||||
channelMembers, err = ss.ChannelMemberHistory().GetUsersInChannelDuring(leaveTime+100, leaveTime+200, []string{channel.Id})
|
||||
require.NoError(t, err)
|
||||
assert.Empty(t, channelMembers)
|
||||
}
|
||||
@@ -223,7 +378,7 @@ func testGetUsersInChannelAtChannelMembers(t *testing.T, rctx request.CTX, ss st
|
||||
// the past, even though the time that they were actually in the channel doesn't necessarily overlap with the export period
|
||||
|
||||
// case 1: user joins and leaves the channel before the export period begins
|
||||
channelMembers, err := ss.ChannelMemberHistory().GetUsersInChannelDuring(joinTime-500, joinTime-100, channel.Id)
|
||||
channelMembers, err := ss.ChannelMemberHistory().GetUsersInChannelDuring(joinTime-500, joinTime-100, []string{channel.Id})
|
||||
require.NoError(t, err)
|
||||
assert.Len(t, channelMembers, 1)
|
||||
assert.Equal(t, channel.Id, channelMembers[0].ChannelId)
|
||||
@@ -234,7 +389,7 @@ func testGetUsersInChannelAtChannelMembers(t *testing.T, rctx request.CTX, ss st
|
||||
assert.Equal(t, joinTime-100, *channelMembers[0].LeaveTime)
|
||||
|
||||
// case 2: user joins the channel after the export period begins, but has not yet left the channel when the export period ends
|
||||
channelMembers, err = ss.ChannelMemberHistory().GetUsersInChannelDuring(joinTime-100, joinTime+500, channel.Id)
|
||||
channelMembers, err = ss.ChannelMemberHistory().GetUsersInChannelDuring(joinTime-100, joinTime+500, []string{channel.Id})
|
||||
require.NoError(t, err)
|
||||
assert.Len(t, channelMembers, 1)
|
||||
assert.Equal(t, channel.Id, channelMembers[0].ChannelId)
|
||||
@@ -245,7 +400,7 @@ func testGetUsersInChannelAtChannelMembers(t *testing.T, rctx request.CTX, ss st
|
||||
assert.Equal(t, joinTime+500, *channelMembers[0].LeaveTime)
|
||||
|
||||
// case 3: user joins the channel before the export period begins, but has not yet left the channel when the export period ends
|
||||
channelMembers, err = ss.ChannelMemberHistory().GetUsersInChannelDuring(joinTime+100, joinTime+500, channel.Id)
|
||||
channelMembers, err = ss.ChannelMemberHistory().GetUsersInChannelDuring(joinTime+100, joinTime+500, []string{channel.Id})
|
||||
require.NoError(t, err)
|
||||
assert.Len(t, channelMembers, 1)
|
||||
assert.Equal(t, channel.Id, channelMembers[0].ChannelId)
|
||||
@@ -256,7 +411,7 @@ func testGetUsersInChannelAtChannelMembers(t *testing.T, rctx request.CTX, ss st
|
||||
assert.Equal(t, joinTime+500, *channelMembers[0].LeaveTime)
|
||||
|
||||
// case 4: user joins the channel before the export period begins, but has not yet left the channel when the export period ends
|
||||
channelMembers, err = ss.ChannelMemberHistory().GetUsersInChannelDuring(joinTime+100, leaveTime-100, channel.Id)
|
||||
channelMembers, err = ss.ChannelMemberHistory().GetUsersInChannelDuring(joinTime+100, leaveTime-100, []string{channel.Id})
|
||||
require.NoError(t, err)
|
||||
assert.Len(t, channelMembers, 1)
|
||||
assert.Equal(t, channel.Id, channelMembers[0].ChannelId)
|
||||
@@ -267,7 +422,7 @@ func testGetUsersInChannelAtChannelMembers(t *testing.T, rctx request.CTX, ss st
|
||||
assert.Equal(t, leaveTime-100, *channelMembers[0].LeaveTime)
|
||||
|
||||
// case 5: user joins the channel after the export period begins, and leaves the channel before the export period ends
|
||||
channelMembers, err = ss.ChannelMemberHistory().GetUsersInChannelDuring(joinTime-100, leaveTime+100, channel.Id)
|
||||
channelMembers, err = ss.ChannelMemberHistory().GetUsersInChannelDuring(joinTime-100, leaveTime+100, []string{channel.Id})
|
||||
require.NoError(t, err)
|
||||
assert.Len(t, channelMembers, 1)
|
||||
assert.Equal(t, channel.Id, channelMembers[0].ChannelId)
|
||||
@@ -278,7 +433,7 @@ func testGetUsersInChannelAtChannelMembers(t *testing.T, rctx request.CTX, ss st
|
||||
assert.Equal(t, leaveTime+100, *channelMembers[0].LeaveTime)
|
||||
|
||||
// case 6: user has joined and left the channel long before the export period begins
|
||||
channelMembers, err = ss.ChannelMemberHistory().GetUsersInChannelDuring(leaveTime+100, leaveTime+200, channel.Id)
|
||||
channelMembers, err = ss.ChannelMemberHistory().GetUsersInChannelDuring(leaveTime+100, leaveTime+200, []string{channel.Id})
|
||||
require.NoError(t, err)
|
||||
assert.Len(t, channelMembers, 1)
|
||||
assert.Equal(t, channel.Id, channelMembers[0].ChannelId)
|
||||
@@ -332,7 +487,7 @@ func testPermanentDeleteBatch(t *testing.T, rctx request.CTX, ss store.Store) {
|
||||
require.NoError(t, err)
|
||||
|
||||
// in between the join time and the leave time, both users were members of the channel
|
||||
channelMembers, err := ss.ChannelMemberHistory().GetUsersInChannelDuring(joinTime+10, leaveTime-10, channel.Id)
|
||||
channelMembers, err := ss.ChannelMemberHistory().GetUsersInChannelDuring(joinTime+10, leaveTime-10, []string{channel.Id})
|
||||
require.NoError(t, err)
|
||||
assert.Len(t, channelMembers, 2)
|
||||
|
||||
@@ -343,7 +498,7 @@ func testPermanentDeleteBatch(t *testing.T, rctx request.CTX, ss store.Store) {
|
||||
assert.NotEqual(t, int64(0), rowsDeleted)
|
||||
|
||||
// after the delete, there should be one less member in the channel
|
||||
channelMembers, err = ss.ChannelMemberHistory().GetUsersInChannelDuring(joinTime+10, leaveTime-10, channel.Id)
|
||||
channelMembers, err = ss.ChannelMemberHistory().GetUsersInChannelDuring(joinTime+10, leaveTime-10, []string{channel.Id})
|
||||
require.NoError(t, err)
|
||||
assert.Len(t, channelMembers, 1)
|
||||
assert.Equal(t, user2.Id, channelMembers[0].UserId)
|
||||
@@ -387,7 +542,7 @@ func testPermanentDeleteBatchForRetentionPolicies(t *testing.T, rctx request.CTX
|
||||
_, _, err = ss.ChannelMemberHistory().PermanentDeleteBatchForRetentionPolicies(
|
||||
nowMillis, 0, limit, model.RetentionPolicyCursor{})
|
||||
require.NoError(t, err)
|
||||
result, err := ss.ChannelMemberHistory().GetUsersInChannelDuring(joinTime, leaveTime, channel.Id)
|
||||
result, err := ss.ChannelMemberHistory().GetUsersInChannelDuring(joinTime, leaveTime, []string{channel.Id})
|
||||
require.NoError(t, err)
|
||||
require.Empty(t, result, "history should have been deleted by channel policy")
|
||||
rows, err := ss.RetentionPolicy().GetIdsForDeletionByTableName("ChannelMemberHistory", 1000)
|
||||
|
||||
@@ -58,6 +58,7 @@ func TestComplianceStore(t *testing.T, rctx request.CTX, ss store.Store) {
|
||||
t.Run("MessageEditAfterExportMessage", func(t *testing.T) { testEditAfterExportMessage(t, rctx, ss) })
|
||||
t.Run("MessageDeleteExportMessage", func(t *testing.T) { testDeleteExportMessage(t, rctx, ss) })
|
||||
t.Run("MessageDeleteAfterExportMessage", func(t *testing.T) { testDeleteAfterExportMessage(t, rctx, ss) })
|
||||
t.Run("MessageExport_UntilUpdateAt", func(t *testing.T) { testMessageExportUntilUpdateAt(t, rctx, ss) })
|
||||
}
|
||||
|
||||
func testComplianceStore(t *testing.T, rctx request.CTX, ss store.Store) {
|
||||
@@ -1174,3 +1175,105 @@ func testDeleteAfterExportMessage(t *testing.T, rctx request.CTX, ss store.Store
|
||||
assert.Equal(t, user1.Email, *v.UserEmail)
|
||||
assert.Equal(t, user1.Username, *v.Username)
|
||||
}
|
||||
|
||||
func testMessageExportUntilUpdateAt(t *testing.T, rctx request.CTX, ss store.Store) {
|
||||
defer cleanupStoreState(t, rctx, ss)
|
||||
|
||||
// get the starting number of message export entries
|
||||
startTime := model.GetMillis()
|
||||
messages, _, err := ss.Compliance().MessageExport(rctx, model.MessageExportCursor{LastPostUpdateAt: startTime - 10}, 10)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, 0, len(messages))
|
||||
|
||||
// need a team
|
||||
team := &model.Team{
|
||||
DisplayName: "DisplayName",
|
||||
Name: model.NewId(),
|
||||
Email: MakeEmail(),
|
||||
Type: model.TeamOpen,
|
||||
}
|
||||
team, err = ss.Team().Save(team)
|
||||
require.NoError(t, err)
|
||||
|
||||
// and two users that are a part of that team
|
||||
user1 := &model.User{
|
||||
Email: MakeEmail(),
|
||||
Username: model.NewUsername(),
|
||||
}
|
||||
user1, err = ss.User().Save(rctx, user1)
|
||||
require.NoError(t, err)
|
||||
_, nErr := ss.Team().SaveMember(rctx, &model.TeamMember{
|
||||
TeamId: team.Id,
|
||||
UserId: user1.Id,
|
||||
}, -1)
|
||||
require.NoError(t, nErr)
|
||||
|
||||
user2 := &model.User{
|
||||
Email: MakeEmail(),
|
||||
Username: model.NewUsername(),
|
||||
}
|
||||
user2, err = ss.User().Save(rctx, user2)
|
||||
require.NoError(t, err)
|
||||
_, nErr = ss.Team().SaveMember(rctx, &model.TeamMember{
|
||||
TeamId: team.Id,
|
||||
UserId: user2.Id,
|
||||
}, -1)
|
||||
require.NoError(t, nErr)
|
||||
|
||||
// need a public channel
|
||||
channel := &model.Channel{
|
||||
TeamId: team.Id,
|
||||
Name: model.NewId(),
|
||||
DisplayName: "Public Channel",
|
||||
Type: model.ChannelTypeOpen,
|
||||
}
|
||||
channel, nErr = ss.Channel().Save(rctx, channel, -1)
|
||||
require.NoError(t, nErr)
|
||||
|
||||
var posts []*model.Post
|
||||
// user1 posts ten times in the public channel
|
||||
for i := 0; i < 10; i++ {
|
||||
post := &model.Post{
|
||||
ChannelId: channel.Id,
|
||||
UserId: user1.Id,
|
||||
CreateAt: startTime + int64(i),
|
||||
UpdateAt: startTime + int64(i),
|
||||
Message: model.NewId(),
|
||||
}
|
||||
post, err = ss.Post().Save(rctx, post)
|
||||
require.NoError(t, err)
|
||||
posts = append(posts, post)
|
||||
}
|
||||
|
||||
// fetch 5 starting from the third post, using LastPostUpdateAt and UntilUpdateAt.
|
||||
// UntilUpdateAt is inclusive
|
||||
messageExportMap := map[string]model.MessageExport{}
|
||||
messages, _, err = ss.Compliance().MessageExport(rctx, model.MessageExportCursor{LastPostUpdateAt: posts[2].UpdateAt, UntilUpdateAt: posts[2].UpdateAt + 4}, 10000)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, 5, len(messages))
|
||||
|
||||
for _, v := range messages {
|
||||
messageExportMap[*v.PostId] = *v
|
||||
}
|
||||
|
||||
for i := 2; i < 7; i++ {
|
||||
assert.Equal(t, posts[i].Id, *messageExportMap[posts[i].Id].PostId)
|
||||
assert.Equal(t, posts[i].CreateAt, *messageExportMap[posts[i].Id].PostCreateAt)
|
||||
assert.Equal(t, posts[i].Message, *messageExportMap[posts[i].Id].PostMessage)
|
||||
assert.Equal(t, channel.Id, *messageExportMap[posts[i].Id].ChannelId)
|
||||
assert.Equal(t, channel.DisplayName, *messageExportMap[posts[i].Id].ChannelDisplayName)
|
||||
assert.Equal(t, user1.Id, *messageExportMap[posts[i].Id].UserId)
|
||||
assert.Equal(t, user1.Email, *messageExportMap[posts[i].Id].UserEmail)
|
||||
assert.Equal(t, user1.Username, *messageExportMap[posts[i].Id].Username)
|
||||
}
|
||||
|
||||
// Also test AnalyticsPostCount because they are used in tandem for MessageExports
|
||||
count, err := ss.Post().AnalyticsPostCount(&model.PostCountOptions{
|
||||
TeamId: channel.TeamId,
|
||||
ExcludeSystemPosts: true,
|
||||
SinceUpdateAt: posts[2].UpdateAt,
|
||||
UntilUpdateAt: posts[2].UpdateAt + 4,
|
||||
})
|
||||
require.NoError(t, err)
|
||||
require.Equal(t, 5, int(count))
|
||||
}
|
||||
|
||||
@@ -72,8 +72,38 @@ func (_m *ChannelMemberHistoryStore) GetChannelsLeftSince(userID string, since i
|
||||
return r0, r1
|
||||
}
|
||||
|
||||
// GetChannelsWithActivityDuring provides a mock function with given fields: startTime, endTime
|
||||
func (_m *ChannelMemberHistoryStore) GetChannelsWithActivityDuring(startTime int64, endTime int64) ([]string, error) {
|
||||
ret := _m.Called(startTime, endTime)
|
||||
|
||||
if len(ret) == 0 {
|
||||
panic("no return value specified for GetChannelsWithActivityDuring")
|
||||
}
|
||||
|
||||
var r0 []string
|
||||
var r1 error
|
||||
if rf, ok := ret.Get(0).(func(int64, int64) ([]string, error)); ok {
|
||||
return rf(startTime, endTime)
|
||||
}
|
||||
if rf, ok := ret.Get(0).(func(int64, int64) []string); ok {
|
||||
r0 = rf(startTime, endTime)
|
||||
} else {
|
||||
if ret.Get(0) != nil {
|
||||
r0 = ret.Get(0).([]string)
|
||||
}
|
||||
}
|
||||
|
||||
if rf, ok := ret.Get(1).(func(int64, int64) error); ok {
|
||||
r1 = rf(startTime, endTime)
|
||||
} else {
|
||||
r1 = ret.Error(1)
|
||||
}
|
||||
|
||||
return r0, r1
|
||||
}
|
||||
|
||||
// GetUsersInChannelDuring provides a mock function with given fields: startTime, endTime, channelID
|
||||
func (_m *ChannelMemberHistoryStore) GetUsersInChannelDuring(startTime int64, endTime int64, channelID string) ([]*model.ChannelMemberHistoryResult, error) {
|
||||
func (_m *ChannelMemberHistoryStore) GetUsersInChannelDuring(startTime int64, endTime int64, channelID []string) ([]*model.ChannelMemberHistoryResult, error) {
|
||||
ret := _m.Called(startTime, endTime, channelID)
|
||||
|
||||
if len(ret) == 0 {
|
||||
@@ -82,10 +112,10 @@ func (_m *ChannelMemberHistoryStore) GetUsersInChannelDuring(startTime int64, en
|
||||
|
||||
var r0 []*model.ChannelMemberHistoryResult
|
||||
var r1 error
|
||||
if rf, ok := ret.Get(0).(func(int64, int64, string) ([]*model.ChannelMemberHistoryResult, error)); ok {
|
||||
if rf, ok := ret.Get(0).(func(int64, int64, []string) ([]*model.ChannelMemberHistoryResult, error)); ok {
|
||||
return rf(startTime, endTime, channelID)
|
||||
}
|
||||
if rf, ok := ret.Get(0).(func(int64, int64, string) []*model.ChannelMemberHistoryResult); ok {
|
||||
if rf, ok := ret.Get(0).(func(int64, int64, []string) []*model.ChannelMemberHistoryResult); ok {
|
||||
r0 = rf(startTime, endTime, channelID)
|
||||
} else {
|
||||
if ret.Get(0) != nil {
|
||||
@@ -93,7 +123,7 @@ func (_m *ChannelMemberHistoryStore) GetUsersInChannelDuring(startTime int64, en
|
||||
}
|
||||
}
|
||||
|
||||
if rf, ok := ret.Get(1).(func(int64, int64, string) error); ok {
|
||||
if rf, ok := ret.Get(1).(func(int64, int64, []string) error); ok {
|
||||
r1 = rf(startTime, endTime, channelID)
|
||||
} else {
|
||||
r1 = ret.Error(1)
|
||||
|
||||
Ссылка в новой задаче
Block a user