MM-59966 - Compliance Export overhaul - feature branch (#29789)
* [MM-59089] Add a compliance export constant (#27919) * add a useful constant * i18n * another constant * another i18n * [MM-60422] Add GetChannelsWithActivityDuring (#28301) * modify GetUsersInChannelDuring to accept a slice of channelIds * add GetChannelsWithActivityDuring * add compliance export progress message; remove unused custom status * linting * tests running too fast * add batch size config settings * add store tests * linting * empty commit * i18n changes * fix i18n ordering * MM-60570 - Server-side changes consolidating the export CLI with server/ent code (#28640) * add an i18n field; add the CLI's export directory * int64 -> int * Add UntilUpdateAt for MessageExport and AnalyticsPostCount to merge * remove now-unused i18n strings * add TranslationsPreInitFromBuffer to allow CLI to use i18n * use GetBuilder to simplify; rename TranslationsPreInitFromFileBytes * [MM-59089] Improve compliance export timings (#1733 - Enterprise repo) * MM-60422 - Performance and logic fixes for Compliance Exports (#1757 - Enterprise repo) * MM-60570 - Enterprise-side changes consolidating the export CLI with server/ent code (#1769 - Enterprise repo) * merge conflicts; missed file from ent branch * MM-61038 - Add an option to sqlstore.New (#28702) remove useless comment add test add an option to sqlstore.New * MM-60976: Remove RunExport command from Mattermost binary (#28805) * remove RunExport command from mattermost binary * remove the code it was calling * fix i18n * remove test (was only testing license, not functionality) * empty commit * fix flaky GetChannelsWithActivityDuring test * MM-60063: Dedicated Export Filestore fix, redo of #1772 (enterprise) (#28803) * redo filestore fix #1772 (enterprise repo) on top of MM-59966 feature * add new e2e tests for export filestore * golint * ok, note to self: shadowing bad, actually (when there's a defer) * empty commit * MM-61137 - Message export: Support 7.8.11 era dbs (#28824) * support 7.8.11 era dbs by wrapping the store using only what we need * fix flaky GetChannelsWithActivityDuring test * add a comment * only need to define the MEFileInfoStore (the one that'll be overridden) * blank commit * MM-60974 - Message Export: Add performance metrics (#28836) * support 7.8.11 era dbs by wrapping the store using only what we need * fix flaky GetChannelsWithActivityDuring test * add a comment * only need to define the MEFileInfoStore (the one that'll be overridden) * performance metrics * cleanup unneeded named returns * blank commit * MM-60975 - Message export: Add startTime and endTime to export folder name (#28840) * support 7.8.11 era dbs by wrapping the store using only what we need * fix flaky GetChannelsWithActivityDuring test * add a comment * only need to define the MEFileInfoStore (the one that'll be overridden) * performance metrics * output startTime and endTime in export folder * empty commit * merge conflict * MM-60978 - Message export: Improve xml fields; fix delete semantics (#28873) * support 7.8.11 era dbs by wrapping the store using only what we need * fix flaky GetChannelsWithActivityDuring test * add a comment * only need to define the MEFileInfoStore (the one that'll be overridden) * performance metrics * output startTime and endTime in export folder * empty commit * add xml fields, omit when empty, tests * fix delete semantics; test (and test for update semantics) * clarify comments * simplify edited post detection, now there's no edge case. * add some spacing to help fast running tests * merge conflicts/updates needed for new deleted post semantics * linting; fixing tests from upstream merge * use SafeDereference * linting * stronger typing; better wrapped errors; better formatting * blank commit * goimports formatting * fix merge mistake * minor fixes due to changes in master * MM-61755 - Simplifying and Support reporting to the db from the CLI (#29281) * finally clean up JobData struct and stringMap; prep for CLI using db * and now simplify using StringMapToJobDataWithZeroValues * remove unused fn * create JobDataExported; clean up errors * MM-60176 - Message Export: Global relay cleanup (#29168) * move global relay logic into global_relay_export * blank commit * blank commit * improve errors * MM-60693 - Refactor CSV to use same codepath as Actiance (#29191) * move global relay logic into global_relay_export * blank commit * refactor (and simplify) ExportParams into shared * blank commit * remove unused fn * csv now uses pre-calculated joins/leaves like actiance * improve errors * remove nil post check; remove ignoredPosts metric * remove unneeded copy * MM-61696 - Refactor GlobalRelay to use same codepath as Actiance (#29225) * move global relay logic into global_relay_export * blank commit * refactor (and simplify) ExportParams into shared * blank commit * remove unused fn * csv now uses pre-calculated joins/leaves like actiance * remove newly unneeded function and its test. goodbye. * refactor GetPostAttachments for csv + global relay to share * refactor global_relay_export and fix tests (no changes to output) * improve errors * remove nil post check; remove ignoredPosts metric * remove unneeded copy * remove unneeded nil check * PR comments * MM-61715 - Generalize e2e to all export types 🤖 (#29369) * move global relay logic into global_relay_export * blank commit * refactor (and simplify) ExportParams into shared * blank commit * remove unused fn * csv now uses pre-calculated joins/leaves like actiance * remove newly unneeded function and its test. goodbye. * refactor GetPostAttachments for csv + global relay to share * refactor global_relay_export and fix tests (no changes to output) * improve errors * remove nil post check; remove ignoredPosts metric * remove unneeded copy * remove unneeded nil check * PR comments * refactor isDeletedMsg for all export types * fix start and endtime, nasty csv createAt bug; bring closer to Actiance * align unit tests with new logic (e.g. starttime / endtime) * refactor a TimestampConvert fn for code + tests * bug: pass templates to global relay (hurray for e2e tests, otherwise...) * add global relay zip to allowed list (only for tests) * test helpers * new templates for e2e tests * e2e tests... phew. * linting * merge conflicts * unexport PostToRow; add test helper marker * cleanup, shortening, thanks to PR comments * MM-61972 - Generalize export data path - Actiance (#29399) * extract and generalize the export data generation functions * finish moving test (bc of previous extraction) * lift a function from common -> shared (to break an import cycle) * actiance now takes general export data, processes it into actiance data * bring tests in line with correct sorting rules (upadateAt, messageId) * fixups, PR comments * turn strings.Repeat into a more descriptive const amended: one letter fix; bad rebase * MM-62009 - e2e clock heisenbug (#29434) * consolidate assertions; output debuggable diffs (keeping for future) * refactor test output generator to generators file * waitUntilZeroPosts + pass through until to job = fix all clock issues * simplify messages to model.NewId(); remove unneeded waitUntilZeroPosts * model.NewId() -> storetest.NewTestID() * MM-61980 - Generalize export data path - CSV (#29482) * simple refactoring * increase sleep times for (very) rare test failures * add extra information to the generic export for CSV * adj Actiance to handle new generic export (no difference in its output) * no longer need mergePosts (yay), move getJoinLeavePosts for everyone * adjust tests for new csv semantics (detailed in summary) * and need to add the new exported data to the export_data_tests * rearrange csv writing to happen after data export (more logical) * linting * remove debug statements * figured out what was wrong with global relay e2e test 3; solid now * PR comments * MM-61718 - Generalize export data path - Global Relay (#29508) * move global relay over to using the generalized export data * performance pass -- not much can be done * Update server/enterprise/message_export/global_relay_export/global_relay_export.go Co-authored-by: Claudio Costa <cstcld91@gmail.com> --------- Co-authored-by: Claudio Costa <cstcld91@gmail.com> * MM-62058 - Align CSV with Actiance (#29551) * refactoring actiance files and var names for clarity * bug found in exported attachments (we used to miss some start/ends) * changes needed for actiance due to new generic exports * bringing CSV up to actiance standards * fixing global relay b/c of new semantics (adding a note on an edge case) * aligning e2e tests, adding comments to clarify what is expected/tested * necessary changes; 1 more test for added functionality (ignoreDeleted) * comment style * MM-62059 - Align Global Relay with Actiance/CSV; many fixes (#29665) * core logic changes to general export_data and the specific export paths * unit tests and e2e tests, covering all new edge cases and all logic * linting * better var naming, const value, and cleaning up functions calls * MM-62436 - Temporarily skip cypress tests that require download link (#29772) --------- Co-authored-by: Claudio Costa <cstcld91@gmail.com>
Этот коммит содержится в:
Christopher Poile
коммит произвёл
GitHub
GitHub
родитель
96195f1bd7
Коммит
aba4434dab
543
server/enterprise/message_export/shared/export_data.go
Обычный файл
543
server/enterprise/message_export/shared/export_data.go
Обычный файл
@@ -0,0 +1,543 @@
|
||||
// Copyright (c) 2015-present Mattermost, Inc. All Rights Reserved.
|
||||
// See LICENSE.enterprise for license information.
|
||||
|
||||
package shared
|
||||
|
||||
import (
|
||||
"sort"
|
||||
|
||||
"github.com/mattermost/mattermost/server/public/model"
|
||||
)
|
||||
|
||||
type UserType string
|
||||
|
||||
const (
|
||||
User UserType = "user"
|
||||
Bot UserType = "bot"
|
||||
)
|
||||
|
||||
type PostExport struct {
|
||||
model.MessageExport // the MessageExport that this PostExport is providing more information for
|
||||
UserType UserType // the type of the person that sent the post: "user" or "bot"
|
||||
|
||||
// Allows us to differentiate between:
|
||||
// - "EditedOriginalMsg": the newly created message (new Id), which holds the pre-edited message contents. The
|
||||
// "EditedNewMsgId" field will point to the message (original Id) which has the post-edited message content.
|
||||
// - "EditedNewMsg": the post-edited message content. This is confusing, so be careful: in the db, this EditedNewMsg
|
||||
// is actually the original messageId because we wanted an edited message to have the same messageId as the
|
||||
// pre-edited message. But for the purposes of exporting and to keep the mental model clear for end-users, we are
|
||||
// calling this the EditedNewMsg and EditedNewMsgId, because this will hold the NEW post-edited message contents,
|
||||
// and that's what's important to the end-user viewing the export.
|
||||
// - "UpdatedNoMsgChange": the message content hasn't changed, but the post was updated for some reason (reaction,
|
||||
// replied-to, a reply was edited, a reply was deleted (as of 10.2), perhaps other reasons)
|
||||
// - "Deleted": the message was deleted.
|
||||
// - "FileDeleted": this message is recording that a file was deleted.
|
||||
UpdatedType PostUpdatedType
|
||||
UpdateAt int64 // if this is an updated post, this is the updated time (same as deleted time for deleted posts).
|
||||
|
||||
// when a message is edited, the EditedOriginalMsg points to the message Id that now has the newly edited message.
|
||||
EditedNewMsgId string
|
||||
Message string // the text body of the post
|
||||
PreviewsPost string // the post id of the post that is previewed by the permalink preview feature
|
||||
AttachmentCreates []*FileUploadStartExport // the post's attachments that were uploaded this export period
|
||||
AttachmentDeletes []PostExport // the post's attachments that were deleted
|
||||
FileInfo *model.FileInfo // if this was a file PostExport, FileInfo will contain that info. Otherwise, nil.
|
||||
}
|
||||
|
||||
type FileUploadStartExport struct {
|
||||
model.MessageExport // the post that this upload was attached to
|
||||
UserEmail string // the email of the person that sent the file
|
||||
UploadStartTime int64 // utc timestamp (seconds), time at which the user started the upload. Example: 1366611728
|
||||
FileInfo *model.FileInfo
|
||||
}
|
||||
|
||||
type FileUploadStopExport struct {
|
||||
model.MessageExport // the post that this upload was attached to
|
||||
UserEmail string // the email of the person that sent the file
|
||||
UploadStopTime int64 // utc timestamp (seconds), time at which the user finished the upload. Example: 1366611728
|
||||
Status string // set to either "Completed" or "Failed" depending on the outcome of the upload operation
|
||||
FileInfo *model.FileInfo
|
||||
}
|
||||
|
||||
type ChannelExport struct {
|
||||
ChannelId string
|
||||
ChannelType model.ChannelType
|
||||
ChannelName string
|
||||
DisplayName string
|
||||
StartTime int64 // utc timestamp (milliseconds), start of export period or create time of channel, whichever is greater.
|
||||
EndTime int64 // utc timestamp (milliseconds), end of export period or delete time of channel, whichever is lesser.
|
||||
Posts []PostExport
|
||||
Files []*model.FileInfo
|
||||
DeletedFiles []PostExport
|
||||
UploadStarts []*FileUploadStartExport
|
||||
UploadStops []*FileUploadStopExport
|
||||
JoinEvents []JoinExport // start with a list of all users who were present in the channel during the export period
|
||||
LeaveEvents []LeaveExport // finish with a list of all users who were present in the channel during the export period
|
||||
|
||||
// Used by csv, ignored by others
|
||||
TeamId string
|
||||
TeamName string
|
||||
TeamDisplayName string
|
||||
}
|
||||
|
||||
type JoinExport struct {
|
||||
UserId string
|
||||
Username string
|
||||
UserEmail string // the email of the person that joined the channel
|
||||
UserType UserType // the type of the user that joined the channel
|
||||
JoinTime int64 // utc timestamp (seconds), time at which the user joined. Example: 1366611728
|
||||
|
||||
// when the user left (or batch endTime if they didn't leave). Only used by GlobalRelay
|
||||
LeaveTime int64
|
||||
}
|
||||
|
||||
type LeaveExport struct {
|
||||
UserId string
|
||||
Username string
|
||||
UserEmail string // the email of the person that left the channel
|
||||
UserType UserType // the type of the user that left the channel
|
||||
LeaveTime int64 // utc timestamp (seconds), time at which the user left. Example: 1366611728
|
||||
|
||||
// ClosedOut indicates this is a "leave" event created by closing out the channel at the end of an export period.
|
||||
// Actiance requires all users to be closed out at the end of an export period (each join has a matching leave).
|
||||
ClosedOut bool
|
||||
}
|
||||
|
||||
type GenericExportData struct {
|
||||
Exports []ChannelExport
|
||||
Metadata Metadata
|
||||
Results RunExportResults
|
||||
}
|
||||
|
||||
// GetGenericExportData assembles all the data in an exportType-agnostic way. Each exportType will process this data into
|
||||
// the specific format they need to export.
|
||||
func GetGenericExportData(p ExportParams) (GenericExportData, error) {
|
||||
// postAuthorsByChannel is a map so that we don't store duplicate authors
|
||||
postAuthorsByChannel := make(map[string]map[string]ChannelMember)
|
||||
metadata := Metadata{
|
||||
Channels: p.ChannelMetadata,
|
||||
MessagesCount: 0,
|
||||
AttachmentsCount: 0,
|
||||
StartTime: p.BatchStartTime,
|
||||
EndTime: p.BatchEndTime,
|
||||
}
|
||||
|
||||
var results RunExportResults
|
||||
channelsInThisBatch := make(map[string]bool)
|
||||
postsByChannel := make(map[string][]PostExport)
|
||||
filesByChannel := make(map[string][]*model.FileInfo)
|
||||
uploadStartsByChannel := make(map[string][]*FileUploadStartExport)
|
||||
uploadStopsByChannel := make(map[string][]*FileUploadStopExport)
|
||||
deletedFilesByChannel := make(map[string][]PostExport)
|
||||
|
||||
processPostAttachments := func(post *model.MessageExport, postExport PostExport, originalPostThatWillBeDeletedLater bool) error {
|
||||
// originalPostThatWillBeDeletedLater means we are recording this message's original file starts and stops,
|
||||
// before it was deleted (we'll record that next call to this function)
|
||||
//
|
||||
// NOTE: there is an edge case here: the original post is not deleted but the attachment has been deleted.
|
||||
// See the note in the postToAttachmentsEntries func.
|
||||
|
||||
channelId := *post.ChannelId
|
||||
uploadedFiles, startUploads, stopUploads, deleteFileMessages, err :=
|
||||
postToAttachmentsEntries(post, p.Db, originalPostThatWillBeDeletedLater)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
uploadStartsByChannel[channelId] = append(uploadStartsByChannel[channelId], startUploads...)
|
||||
uploadStopsByChannel[channelId] = append(uploadStopsByChannel[channelId], stopUploads...)
|
||||
deletedFilesByChannel[channelId] = append(deletedFilesByChannel[channelId], deleteFileMessages...)
|
||||
filesByChannel[channelId] = append(filesByChannel[channelId], uploadedFiles...)
|
||||
|
||||
postExport.AttachmentCreates = startUploads
|
||||
postExport.AttachmentDeletes = deleteFileMessages
|
||||
postsByChannel[channelId] = append(postsByChannel[channelId], postExport)
|
||||
|
||||
results.UploadedFiles += len(startUploads)
|
||||
results.DeletedFiles += len(deleteFileMessages)
|
||||
|
||||
// only count uploaded files (not deleted files)
|
||||
if err := metadata.UpdateCounts(channelId, 1, len(startUploads)); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
for _, post := range p.Posts {
|
||||
channelId := *post.ChannelId
|
||||
channelsInThisBatch[channelId] = true
|
||||
|
||||
// Was the post deleted (not an edited post), and originally posted during the current job window?
|
||||
// If so, we need to record it. It may actually belong in an earlier batch, but there's no way to know that
|
||||
// before now because of the way we export posts (by updateAt).
|
||||
if IsDeletedMsg(post) && !isEditedOriginalMsg(post) && *post.PostCreateAt >= p.JobStartTime {
|
||||
results.CreatedPosts++
|
||||
postExport := createdPostToExportEntry(post)
|
||||
if err := processPostAttachments(post, postExport, true); err != nil {
|
||||
return GenericExportData{}, err
|
||||
}
|
||||
}
|
||||
var postExport PostExport
|
||||
postExport, results = getPostExport(post, results)
|
||||
|
||||
if err := processPostAttachments(post, postExport, false); err != nil {
|
||||
return GenericExportData{}, err
|
||||
}
|
||||
|
||||
if _, ok := postAuthorsByChannel[channelId]; !ok {
|
||||
postAuthorsByChannel[channelId] = make(map[string]ChannelMember)
|
||||
}
|
||||
postAuthorsByChannel[channelId][*post.UserId] = ChannelMember{
|
||||
UserId: *post.UserId,
|
||||
Email: *post.UserEmail,
|
||||
Username: *post.Username,
|
||||
IsBot: post.IsBot,
|
||||
}
|
||||
}
|
||||
|
||||
// If the channel is not in channelsInThisBatch (i.e. if it didn't have a post), we need to check if it had
|
||||
// user activity between this batch's startTime-endTime. If so, add it to the channelsInThisBatch.
|
||||
for id := range p.ChannelMetadata {
|
||||
if !channelsInThisBatch[id] {
|
||||
if ChannelHasActivity(p.ChannelMemberHistories[id], p.BatchStartTime, p.BatchEndTime) {
|
||||
channelsInThisBatch[id] = true
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Build the channel exports for the channels that had post or user join/leave activity this batch.
|
||||
channelExports := make([]ChannelExport, 0, len(channelsInThisBatch))
|
||||
for id := range channelsInThisBatch {
|
||||
c := metadata.Channels[id]
|
||||
|
||||
joinEvents, leaveEvents := getJoinsAndLeaves(p.BatchStartTime, p.BatchEndTime,
|
||||
p.ChannelMemberHistories[id], postAuthorsByChannel[id])
|
||||
|
||||
// We don't have teamName and teamDisplayName from the channelMetaData, but we have it from MessageExport.
|
||||
// However, if we don't have posts for this channel (only joins and leaves), then we don't have it at all.
|
||||
var teamName, teamDisplayName string
|
||||
if posts, ok := postsByChannel[id]; ok {
|
||||
if len(posts) > 0 {
|
||||
teamName = model.SafeDereference(posts[0].TeamName)
|
||||
teamDisplayName = model.SafeDereference(posts[0].TeamDisplayName)
|
||||
}
|
||||
}
|
||||
|
||||
channelExports = append(channelExports, ChannelExport{
|
||||
ChannelId: c.ChannelId,
|
||||
ChannelType: c.ChannelType,
|
||||
ChannelName: c.ChannelName,
|
||||
DisplayName: c.ChannelDisplayName,
|
||||
StartTime: p.BatchStartTime,
|
||||
EndTime: p.BatchEndTime,
|
||||
Posts: postsByChannel[id],
|
||||
Files: filesByChannel[id],
|
||||
DeletedFiles: deletedFilesByChannel[id],
|
||||
UploadStarts: uploadStartsByChannel[id],
|
||||
UploadStops: uploadStopsByChannel[id],
|
||||
JoinEvents: joinEvents,
|
||||
LeaveEvents: leaveEvents,
|
||||
TeamId: model.SafeDereference(c.TeamId),
|
||||
TeamName: teamName,
|
||||
TeamDisplayName: teamDisplayName,
|
||||
})
|
||||
|
||||
results.Joins += len(joinEvents)
|
||||
results.Leaves += len(leaveEvents)
|
||||
}
|
||||
|
||||
return GenericExportData{channelExports, metadata, results}, nil
|
||||
}
|
||||
|
||||
// postToAttachmentsEntries returns every fileInfo as uploadedFiles. It also adds each file into the lists:
|
||||
//
|
||||
// startUploads, stopUploads, and deleteFileMessages (for ActianceExport).
|
||||
// If onlyDeleted = true, only export the deleted entry (if it exists).
|
||||
func postToAttachmentsEntries(post *model.MessageExport, db MessageExportStore, ignoreDeleted bool) (
|
||||
uploadedFiles []*model.FileInfo, startUploads []*FileUploadStartExport, stopUploads []*FileUploadStopExport, deleteFileMessages []PostExport, err error) {
|
||||
// if the post included any files, we need to add special elements to the export.
|
||||
//
|
||||
// NOTE: there is an edge case here: the original post is not deleted but the attachment has been deleted.
|
||||
// 1. If the attachment is deleted sometime later in the future but the post has not been updated (updateAt == createAt)
|
||||
// then we won't ever get here, there's nothing we can do.
|
||||
// 2. If the attachment is deleted within this export period, or the post is updated this export period and we
|
||||
// now see that the attachment is deleted, then we have to output 2 files here:
|
||||
// a) the original file attachment
|
||||
// b) the deleted file attachment
|
||||
// These have to be added the start/stop and deletedFileMessages
|
||||
|
||||
if len(post.PostFileIds) == 0 {
|
||||
return
|
||||
}
|
||||
|
||||
uploadedFiles, err = db.FileInfo().GetForPost(*post.PostId, true, true, false)
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
|
||||
for _, fileInfo := range uploadedFiles {
|
||||
if fileInfo.DeleteAt > 0 && !ignoreDeleted {
|
||||
deleteFileMessages = append(deleteFileMessages, deleteFileToExportEntry(post, fileInfo))
|
||||
|
||||
// this was a deleted file, so do not record its start and stop. If the original message was sent in this
|
||||
// batch, the file transfer will have been exported earlier up when the original message was exported.
|
||||
//
|
||||
// However, because of the edge case above, we still need to record start and stop if the post is not deleted.
|
||||
if IsDeletedMsg(post) {
|
||||
continue
|
||||
} // not deleted, so need to add start and stop below.
|
||||
}
|
||||
|
||||
// insert a record of the file upload into the export file
|
||||
// path to exported file is relative to the fileAttachmentFilestore root,
|
||||
// which could be different from the exportFilestore root
|
||||
startUploads = append(startUploads, &FileUploadStartExport{
|
||||
MessageExport: *post,
|
||||
UserEmail: *post.UserEmail,
|
||||
UploadStartTime: *post.PostCreateAt,
|
||||
FileInfo: fileInfo,
|
||||
})
|
||||
|
||||
stopUploads = append(stopUploads, &FileUploadStopExport{
|
||||
MessageExport: *post,
|
||||
UserEmail: *post.UserEmail,
|
||||
UploadStopTime: *post.PostCreateAt,
|
||||
Status: "Completed",
|
||||
FileInfo: fileInfo,
|
||||
})
|
||||
}
|
||||
return
|
||||
}
|
||||
|
||||
func getPostExport(post *model.MessageExport, results RunExportResults) (PostExport, RunExportResults) {
|
||||
// We have three "kinds" of posts:
|
||||
// (using "1" and "2" for simplicity)
|
||||
// - created: Id = new, CreateAt = 1, UpdateAt = 1, DeleteAt = 0
|
||||
// - deleted: Id = orig, CreateAt = orig, UpdateAt = 2, DeleteAt = 2, props: deleteBy
|
||||
// - edited: old post gets "created": Id = new, CreateAt = 1, UpdateAt = 2, DeleteAt = 2, originalId: orig
|
||||
// existing post modified: Id = orig, CreateAt = 1, UpdateAt = 2, DeleteAt = 0
|
||||
//
|
||||
// We also have other ways for a post to be updated:
|
||||
// - a root post in a thread is replied to, when a reply is edited, or (as of 10.2) when a reply is deleted
|
||||
|
||||
if isEditedOriginalMsg(post) {
|
||||
// Post has been edited. This is the original message.
|
||||
results.EditedOrigMsgPosts++
|
||||
return editedOriginalMsgToExportEntry(post), results
|
||||
} else if IsDeletedMsg(post) {
|
||||
// Post is deleted
|
||||
results.DeletedPosts++
|
||||
return deletedPostToExportEntry(post, "delete "+*post.PostMessage), results
|
||||
} else if *post.PostUpdateAt > *post.PostCreateAt {
|
||||
// Post has been updated. But what kind?
|
||||
if model.SafeDereference(post.PostEditAt) > 0 {
|
||||
// This is an edited post.
|
||||
results.EditedNewMsgPosts++
|
||||
return editedNewMsgToExportEntry(post), results
|
||||
}
|
||||
// This is just an updated post (e.g. reaction)
|
||||
results.UpdatedPosts++
|
||||
return updatedPostToExportEntry(post), results
|
||||
}
|
||||
// Post is newly created:
|
||||
// *post.PostCreateAt == *post.PostUpdateAt && (post.PostDeleteAt == nil || *post.PostDeleteAt == 0)
|
||||
// but also fallback to this in case there is missing data, which is better than not exporting anything.
|
||||
results.CreatedPosts++
|
||||
return createdPostToExportEntry(post), results
|
||||
}
|
||||
|
||||
func getJoinsAndLeaves(startTime int64, endTime int64, channelMembersHistory []*model.ChannelMemberHistoryResult,
|
||||
postAuthors map[string]ChannelMember) ([]JoinExport, []LeaveExport) {
|
||||
var leaveEvents []LeaveExport
|
||||
|
||||
joins, leaves := GetJoinsAndLeavesForChannel(startTime, endTime, channelMembersHistory, postAuthors)
|
||||
joinsById := make(map[string]JoinExport, len(joins))
|
||||
type StillMemberInfo struct {
|
||||
time int64
|
||||
userType UserType
|
||||
userId string
|
||||
username string
|
||||
}
|
||||
stillMember := map[string]StillMemberInfo{}
|
||||
for _, join := range joins {
|
||||
userType := User
|
||||
if join.IsBot {
|
||||
userType = Bot
|
||||
}
|
||||
joinsById[join.UserId] = JoinExport{
|
||||
UserId: join.UserId,
|
||||
Username: join.Username,
|
||||
UserEmail: join.Email,
|
||||
JoinTime: join.Datetime,
|
||||
UserType: userType,
|
||||
LeaveTime: endTime,
|
||||
}
|
||||
if value, ok := stillMember[join.Email]; !ok {
|
||||
stillMember[join.Email] = StillMemberInfo{time: join.Datetime, userType: userType, userId: join.UserId, username: join.Username}
|
||||
} else if join.Datetime > value.time {
|
||||
stillMember[join.Email] = StillMemberInfo{time: join.Datetime, userType: userType, userId: join.UserId, username: join.Username}
|
||||
}
|
||||
}
|
||||
for _, leave := range leaves {
|
||||
userType := User
|
||||
if leave.IsBot {
|
||||
userType = Bot
|
||||
}
|
||||
leaveEvents = append(leaveEvents, LeaveExport{
|
||||
UserId: leave.UserId,
|
||||
Username: leave.Username,
|
||||
UserEmail: leave.Email,
|
||||
LeaveTime: leave.Datetime,
|
||||
UserType: userType,
|
||||
})
|
||||
if leave.Datetime > stillMember[leave.Email].time {
|
||||
delete(stillMember, leave.Email)
|
||||
}
|
||||
|
||||
// record their leave in their initial join
|
||||
if join, ok := joinsById[leave.UserId]; ok {
|
||||
join.LeaveTime = leave.Datetime
|
||||
joinsById[leave.UserId] = join
|
||||
}
|
||||
}
|
||||
|
||||
// Closing-out the channel for Actiance (each join must have a matching leave).
|
||||
for email := range stillMember {
|
||||
leaveEvents = append(leaveEvents, LeaveExport{
|
||||
UserId: stillMember[email].userId,
|
||||
Username: stillMember[email].username,
|
||||
LeaveTime: endTime,
|
||||
UserEmail: email,
|
||||
UserType: stillMember[email].userType,
|
||||
ClosedOut: true,
|
||||
})
|
||||
}
|
||||
|
||||
joinEvents := make([]JoinExport, 0, len(joinsById))
|
||||
for _, v := range joinsById {
|
||||
joinEvents = append(joinEvents, v)
|
||||
}
|
||||
|
||||
sort.Slice(joinEvents, func(i, j int) bool {
|
||||
if joinEvents[i].JoinTime == joinEvents[j].JoinTime {
|
||||
return joinEvents[i].UserEmail < joinEvents[j].UserEmail
|
||||
}
|
||||
return joinEvents[i].JoinTime < joinEvents[j].JoinTime
|
||||
})
|
||||
|
||||
sort.Slice(leaveEvents, func(i, j int) bool {
|
||||
if leaveEvents[i].LeaveTime == leaveEvents[j].LeaveTime {
|
||||
return leaveEvents[i].UserEmail < leaveEvents[j].UserEmail
|
||||
}
|
||||
return leaveEvents[i].LeaveTime < leaveEvents[j].LeaveTime
|
||||
})
|
||||
|
||||
return joinEvents, leaveEvents
|
||||
}
|
||||
|
||||
func isEditedOriginalMsg(post *model.MessageExport) bool {
|
||||
return model.SafeDereference(post.PostDeleteAt) > 0 && model.SafeDereference(post.PostOriginalId) != ""
|
||||
}
|
||||
|
||||
func createdPostToExportEntry(post *model.MessageExport) PostExport {
|
||||
userType := User
|
||||
if post.IsBot {
|
||||
userType = Bot
|
||||
}
|
||||
return PostExport{
|
||||
MessageExport: *post,
|
||||
Message: *post.PostMessage,
|
||||
UserType: userType,
|
||||
PreviewsPost: post.PreviewID(),
|
||||
}
|
||||
}
|
||||
|
||||
func deletedPostToExportEntry(post *model.MessageExport, newMsg string) PostExport {
|
||||
userType := User
|
||||
if post.IsBot {
|
||||
userType = Bot
|
||||
}
|
||||
return PostExport{
|
||||
MessageExport: *post,
|
||||
UpdateAt: *post.PostDeleteAt,
|
||||
UpdatedType: Deleted,
|
||||
Message: newMsg,
|
||||
UserType: userType,
|
||||
PreviewsPost: post.PreviewID(),
|
||||
}
|
||||
}
|
||||
|
||||
func editedOriginalMsgToExportEntry(post *model.MessageExport) PostExport {
|
||||
userType := User
|
||||
if post.IsBot {
|
||||
userType = Bot
|
||||
}
|
||||
return PostExport{
|
||||
MessageExport: *post,
|
||||
UpdateAt: *post.PostUpdateAt,
|
||||
UpdatedType: EditedOriginalMsg,
|
||||
Message: *post.PostMessage,
|
||||
UserType: userType,
|
||||
PreviewsPost: post.PreviewID(),
|
||||
EditedNewMsgId: *post.PostOriginalId,
|
||||
}
|
||||
}
|
||||
|
||||
func editedNewMsgToExportEntry(post *model.MessageExport) PostExport {
|
||||
userType := User
|
||||
if post.IsBot {
|
||||
userType = Bot
|
||||
}
|
||||
return PostExport{
|
||||
MessageExport: *post,
|
||||
UpdateAt: *post.PostUpdateAt,
|
||||
UpdatedType: EditedNewMsg,
|
||||
Message: *post.PostMessage,
|
||||
UserType: userType,
|
||||
PreviewsPost: post.PreviewID(),
|
||||
}
|
||||
}
|
||||
|
||||
func updatedPostToExportEntry(post *model.MessageExport) PostExport {
|
||||
userType := User
|
||||
if post.IsBot {
|
||||
userType = Bot
|
||||
}
|
||||
return PostExport{
|
||||
MessageExport: *post,
|
||||
UpdateAt: *post.PostUpdateAt,
|
||||
UpdatedType: UpdatedNoMsgChange,
|
||||
Message: *post.PostMessage,
|
||||
UserType: userType,
|
||||
PreviewsPost: post.PreviewID(),
|
||||
}
|
||||
}
|
||||
|
||||
func deleteFileToExportEntry(post *model.MessageExport, fileInfo *model.FileInfo) PostExport {
|
||||
userType := User
|
||||
if post.IsBot {
|
||||
userType = Bot
|
||||
}
|
||||
return PostExport{
|
||||
MessageExport: *post,
|
||||
UpdateAt: fileInfo.DeleteAt,
|
||||
UpdatedType: FileDeleted,
|
||||
Message: "delete " + fileInfo.Path,
|
||||
UserType: userType,
|
||||
PreviewsPost: post.PreviewID(),
|
||||
FileInfo: fileInfo,
|
||||
}
|
||||
}
|
||||
|
||||
func UploadStartToExportEntry(u *FileUploadStartExport) PostExport {
|
||||
userType := User
|
||||
if u.IsBot {
|
||||
userType = Bot
|
||||
}
|
||||
return PostExport{
|
||||
MessageExport: u.MessageExport,
|
||||
UpdateAt: u.FileInfo.UpdateAt,
|
||||
UserType: userType,
|
||||
FileInfo: u.FileInfo,
|
||||
}
|
||||
}
|
||||
523
server/enterprise/message_export/shared/export_data_test.go
Обычный файл
523
server/enterprise/message_export/shared/export_data_test.go
Обычный файл
@@ -0,0 +1,523 @@
|
||||
// Copyright (c) 2015-present Mattermost, Inc. All Rights Reserved.
|
||||
// See LICENSE.enterprise for license information.
|
||||
|
||||
package shared
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/mock"
|
||||
"github.com/stretchr/testify/require"
|
||||
|
||||
"github.com/mattermost/mattermost/server/public/model"
|
||||
"github.com/mattermost/mattermost/server/v8/channels/api4"
|
||||
"github.com/mattermost/mattermost/server/v8/channels/jobs"
|
||||
"github.com/mattermost/mattermost/server/v8/channels/store/storetest"
|
||||
)
|
||||
|
||||
func postToMessageExport(t *testing.T, p *model.Post, u *model.User, c *model.Channel, team *model.Team) model.MessageExport {
|
||||
t.Helper()
|
||||
props, err := json.Marshal(p.GetProps())
|
||||
assert.NoError(t, err)
|
||||
|
||||
return model.MessageExport{
|
||||
TeamId: &team.Id,
|
||||
TeamName: &team.Name,
|
||||
TeamDisplayName: &team.DisplayName,
|
||||
ChannelId: &c.Id,
|
||||
ChannelName: &c.Name,
|
||||
ChannelDisplayName: &c.DisplayName,
|
||||
ChannelType: &c.Type,
|
||||
UserId: &u.Id,
|
||||
UserEmail: model.NewPointer(u.Email),
|
||||
Username: &u.Username,
|
||||
IsBot: false,
|
||||
PostId: model.NewPointer(p.Id),
|
||||
PostCreateAt: model.NewPointer(p.CreateAt),
|
||||
PostUpdateAt: &p.UpdateAt,
|
||||
PostDeleteAt: &p.DeleteAt,
|
||||
PostEditAt: &p.EditAt,
|
||||
PostMessage: &p.Message,
|
||||
PostType: &p.Type,
|
||||
PostRootId: &p.RootId,
|
||||
PostProps: model.NewPointer(string(props)),
|
||||
PostOriginalId: &p.OriginalId,
|
||||
PostFileIds: p.FileIds,
|
||||
}
|
||||
}
|
||||
|
||||
func Test_getPostExport(t *testing.T) {
|
||||
if testing.Short() {
|
||||
t.Skip("skipping test in short mode.")
|
||||
}
|
||||
|
||||
jobs.DefaultWatcherPollingInterval = 100
|
||||
th := api4.SetupEnterprise(t).InitBasic()
|
||||
th.App.Srv().SetLicense(model.NewTestLicense("message_export"))
|
||||
defer th.TearDown()
|
||||
|
||||
// the post exports from the db will be random (because they all have the same updateAt), so do it a few times
|
||||
for i := 0; i < 10; i++ {
|
||||
time.Sleep(time.Millisecond)
|
||||
start := model.GetMillis()
|
||||
|
||||
count, err := th.App.Srv().Store().Post().AnalyticsPostCount(&model.PostCountOptions{ExcludeSystemPosts: true, SincePostID: "", SinceUpdateAt: start})
|
||||
require.NoError(t, err)
|
||||
require.Equal(t, 0, int(count))
|
||||
|
||||
var posts []*model.Post
|
||||
|
||||
// 0 - post edited with 3 simultaneous posts in-between - forward
|
||||
// original post with edited message
|
||||
originalPost, err := th.App.Srv().Store().Post().Save(th.Context, &model.Post{
|
||||
ChannelId: th.BasicChannel.Id,
|
||||
UserId: th.BasicUser.Id,
|
||||
Message: "message 0",
|
||||
})
|
||||
require.NoError(t, err)
|
||||
require.NotEqual(t, 0, originalPost.UpdateAt, "originalPost's updateAt was zero, test 1")
|
||||
posts = append(posts, originalPost)
|
||||
|
||||
// If we don't sleep, the two messages might not have different CreateAt and UpdateAts
|
||||
time.Sleep(time.Millisecond)
|
||||
|
||||
// 1 - edited post
|
||||
post, err := th.App.Srv().Store().Post().Update(th.Context, &model.Post{
|
||||
Id: originalPost.Id,
|
||||
CreateAt: originalPost.CreateAt,
|
||||
EditAt: model.GetMillis(),
|
||||
ChannelId: th.BasicChannel.Id,
|
||||
UserId: th.BasicUser.Id,
|
||||
Message: "edited message 0",
|
||||
}, originalPost)
|
||||
require.NoError(t, err)
|
||||
require.NotEqual(t, 0, originalPost.UpdateAt, "originalPost's updateAt was zero, test 2")
|
||||
require.NotEqual(t, 0, post.UpdateAt, "edited post's updateAt was zero, test 2")
|
||||
posts = append(posts, post)
|
||||
|
||||
simultaneous := post.UpdateAt
|
||||
|
||||
// Add 8 other posts at the same updateAt
|
||||
for j := 1; j <= 8; j++ {
|
||||
// 2 - post 1 at same updateAt
|
||||
post, err = th.App.Srv().Store().Post().Save(th.Context, &model.Post{
|
||||
ChannelId: th.BasicChannel.Id,
|
||||
UserId: th.BasicUser.Id,
|
||||
Message: fmt.Sprintf("message %d", j),
|
||||
CreateAt: simultaneous,
|
||||
})
|
||||
require.NoError(t, err)
|
||||
require.NotEqual(t, 0, post.UpdateAt)
|
||||
posts = append(posts, post)
|
||||
}
|
||||
|
||||
// Use the config fallback for simplicity
|
||||
th.App.UpdateConfig(func(cfg *model.Config) {
|
||||
*cfg.MessageExportSettings.EnableExport = true
|
||||
*cfg.MessageExportSettings.ExportFromTimestamp = start
|
||||
*cfg.MessageExportSettings.BatchSize = 10
|
||||
})
|
||||
|
||||
// the messages can be in any order because all have equal `updateAt`s
|
||||
expectedExports := []PostExport{
|
||||
{
|
||||
MessageExport: postToMessageExport(t, posts[0], th.BasicUser, th.BasicChannel, th.BasicTeam),
|
||||
UserType: "user",
|
||||
Message: posts[0].Message,
|
||||
UpdateAt: posts[1].UpdateAt, // the edit update at
|
||||
UpdatedType: EditedOriginalMsg,
|
||||
EditedNewMsgId: posts[1].Id,
|
||||
},
|
||||
{
|
||||
MessageExport: postToMessageExport(t, posts[1], th.BasicUser, th.BasicChannel, th.BasicTeam),
|
||||
UserType: "user",
|
||||
Message: posts[1].Message,
|
||||
UpdateAt: posts[1].UpdateAt,
|
||||
UpdatedType: EditedNewMsg,
|
||||
},
|
||||
}
|
||||
|
||||
for j := 2; j < 10; j++ {
|
||||
expectedExports = append(expectedExports, PostExport{
|
||||
MessageExport: postToMessageExport(t, posts[j], th.BasicUser, th.BasicChannel, th.BasicTeam),
|
||||
UserType: "user",
|
||||
Message: posts[j].Message,
|
||||
})
|
||||
}
|
||||
|
||||
actualMessageExports, _, err := th.App.Srv().Store().Compliance().MessageExport(th.Context, model.MessageExportCursor{
|
||||
LastPostUpdateAt: start,
|
||||
UntilUpdateAt: model.GetMillis(),
|
||||
}, 10)
|
||||
require.NoError(t, err)
|
||||
require.Len(t, actualMessageExports, 10)
|
||||
for _, export := range actualMessageExports {
|
||||
require.NotEqual(t, 0, *export.PostUpdateAt)
|
||||
}
|
||||
results := RunExportResults{}
|
||||
var actualExports []PostExport
|
||||
|
||||
for _, m := range actualMessageExports {
|
||||
var postExport PostExport
|
||||
postExport, results = getPostExport(m, results)
|
||||
actualExports = append(actualExports, postExport)
|
||||
}
|
||||
|
||||
require.ElementsMatch(t, expectedExports, actualExports, fmt.Sprintf("batch %d", i))
|
||||
}
|
||||
}
|
||||
|
||||
func TestPostToAttachmentsEntries(t *testing.T) {
|
||||
chanTypeDirect := model.ChannelTypeDirect
|
||||
tt := []struct {
|
||||
name string
|
||||
post model.MessageExport
|
||||
attachments []*model.FileInfo
|
||||
expectedStarts []*FileUploadStartExport
|
||||
expectedStops []*FileUploadStopExport
|
||||
expectedFileInfos []*model.FileInfo
|
||||
expectedDeleteFileMessages []PostExport
|
||||
ignoreDeleted bool
|
||||
expectError bool
|
||||
}{
|
||||
{
|
||||
name: "no-attachments",
|
||||
post: model.MessageExport{
|
||||
ChannelId: model.NewPointer("Test"),
|
||||
ChannelDisplayName: model.NewPointer("Test"),
|
||||
PostCreateAt: model.NewPointer(int64(1)),
|
||||
PostMessage: model.NewPointer("Some message"),
|
||||
UserEmail: model.NewPointer("test@test.com"),
|
||||
UserId: model.NewPointer("test"),
|
||||
Username: model.NewPointer("test"),
|
||||
ChannelType: &chanTypeDirect,
|
||||
},
|
||||
attachments: nil,
|
||||
expectedStarts: nil,
|
||||
expectedStops: nil,
|
||||
expectedFileInfos: nil,
|
||||
expectedDeleteFileMessages: nil,
|
||||
expectError: false,
|
||||
},
|
||||
{
|
||||
name: "one-attachment",
|
||||
post: model.MessageExport{
|
||||
PostId: model.NewPointer("test"),
|
||||
ChannelId: model.NewPointer("Test"),
|
||||
ChannelDisplayName: model.NewPointer("Test"),
|
||||
PostCreateAt: model.NewPointer(int64(1)),
|
||||
PostMessage: model.NewPointer("Some message"),
|
||||
UserEmail: model.NewPointer("test@test.com"),
|
||||
UserId: model.NewPointer("test"),
|
||||
Username: model.NewPointer("test"),
|
||||
ChannelType: &chanTypeDirect,
|
||||
PostFileIds: []string{"12345"},
|
||||
},
|
||||
attachments: []*model.FileInfo{
|
||||
{Name: "test", Id: "12345", Path: "filename.txt"},
|
||||
},
|
||||
expectedStarts: []*FileUploadStartExport{
|
||||
{UserEmail: "test@test.com", UploadStartTime: 1,
|
||||
FileInfo: &model.FileInfo{Id: "12345", Name: "test", Path: "filename.txt"}},
|
||||
},
|
||||
expectedStops: []*FileUploadStopExport{
|
||||
{UserEmail: "test@test.com", UploadStopTime: 1,
|
||||
FileInfo: &model.FileInfo{Id: "12345", Name: "test", Path: "filename.txt"}, Status: "Completed"},
|
||||
},
|
||||
expectedFileInfos: []*model.FileInfo{
|
||||
{Name: "test", Id: "12345", Path: "filename.txt"},
|
||||
},
|
||||
expectedDeleteFileMessages: nil,
|
||||
expectError: false,
|
||||
},
|
||||
{
|
||||
name: "two-attachment",
|
||||
post: model.MessageExport{
|
||||
PostId: model.NewPointer("test"),
|
||||
ChannelId: model.NewPointer("Test"),
|
||||
ChannelDisplayName: model.NewPointer("Test"),
|
||||
PostCreateAt: model.NewPointer(int64(1)),
|
||||
PostMessage: model.NewPointer("Some message"),
|
||||
UserEmail: model.NewPointer("test@test.com"),
|
||||
UserId: model.NewPointer("test"),
|
||||
Username: model.NewPointer("test"),
|
||||
ChannelType: &chanTypeDirect,
|
||||
PostFileIds: []string{"12345", "54321"},
|
||||
},
|
||||
attachments: []*model.FileInfo{
|
||||
{Name: "test", Id: "12345", Path: "filename.txt"},
|
||||
{Name: "test2", Id: "54321", Path: "filename2.txt"},
|
||||
},
|
||||
expectedStarts: []*FileUploadStartExport{
|
||||
{UserEmail: "test@test.com", UploadStartTime: 1,
|
||||
FileInfo: &model.FileInfo{Id: "12345", Name: "test", Path: "filename.txt"}},
|
||||
{UserEmail: "test@test.com", UploadStartTime: 1,
|
||||
FileInfo: &model.FileInfo{Id: "54321", Name: "test2", Path: "filename2.txt"}},
|
||||
},
|
||||
expectedStops: []*FileUploadStopExport{
|
||||
{UserEmail: "test@test.com", UploadStopTime: 1,
|
||||
FileInfo: &model.FileInfo{Id: "12345", Name: "test", Path: "filename.txt"}, Status: "Completed"},
|
||||
{UserEmail: "test@test.com", UploadStopTime: 1,
|
||||
FileInfo: &model.FileInfo{Id: "54321", Name: "test2", Path: "filename2.txt"}, Status: "Completed"},
|
||||
},
|
||||
expectedFileInfos: []*model.FileInfo{
|
||||
{Name: "test", Id: "12345", Path: "filename.txt"},
|
||||
{Name: "test2", Id: "54321", Path: "filename2.txt"},
|
||||
},
|
||||
expectedDeleteFileMessages: nil,
|
||||
expectError: false,
|
||||
},
|
||||
{
|
||||
name: "one-attachment-deleted",
|
||||
post: model.MessageExport{
|
||||
PostId: model.NewPointer("test"),
|
||||
ChannelId: model.NewPointer("Test"),
|
||||
ChannelDisplayName: model.NewPointer("Test"),
|
||||
PostCreateAt: model.NewPointer(int64(1)),
|
||||
PostDeleteAt: model.NewPointer(int64(2)),
|
||||
PostMessage: model.NewPointer("Some message"),
|
||||
UserEmail: model.NewPointer("test@test.com"),
|
||||
UserId: model.NewPointer("test"),
|
||||
Username: model.NewPointer("test"),
|
||||
ChannelType: &chanTypeDirect,
|
||||
PostFileIds: []string{"12345", "54321"},
|
||||
},
|
||||
attachments: []*model.FileInfo{
|
||||
{Name: "test", Id: "12345", Path: "filename.txt", DeleteAt: 2},
|
||||
},
|
||||
expectedStarts: []*FileUploadStartExport{
|
||||
{UserEmail: "test@test.com", UploadStartTime: 1,
|
||||
FileInfo: &model.FileInfo{Id: "12345", Name: "test", Path: "filename.txt", DeleteAt: 2}},
|
||||
},
|
||||
expectedStops: []*FileUploadStopExport{
|
||||
{UserEmail: "test@test.com", UploadStopTime: 1,
|
||||
FileInfo: &model.FileInfo{Id: "12345", Name: "test", Path: "filename.txt", DeleteAt: 2}, Status: "Completed"},
|
||||
},
|
||||
expectedFileInfos: []*model.FileInfo{
|
||||
{Name: "test", Id: "12345", Path: "filename.txt", DeleteAt: 2},
|
||||
},
|
||||
expectedDeleteFileMessages: []PostExport{
|
||||
{
|
||||
UserType: "user",
|
||||
UpdatedType: FileDeleted,
|
||||
UpdateAt: 2,
|
||||
Message: "delete " + "filename.txt",
|
||||
FileInfo: &model.FileInfo{Id: "12345", Name: "test", Path: "filename.txt", DeleteAt: 2},
|
||||
},
|
||||
},
|
||||
expectError: false,
|
||||
},
|
||||
{
|
||||
name: "one-attachment-deleted, ignore it (only record the creation)",
|
||||
post: model.MessageExport{
|
||||
PostId: model.NewPointer("test"),
|
||||
ChannelId: model.NewPointer("Test"),
|
||||
ChannelDisplayName: model.NewPointer("Test"),
|
||||
PostCreateAt: model.NewPointer(int64(1)),
|
||||
PostDeleteAt: model.NewPointer(int64(2)),
|
||||
PostMessage: model.NewPointer("Some message"),
|
||||
UserEmail: model.NewPointer("test@test.com"),
|
||||
UserId: model.NewPointer("test"),
|
||||
Username: model.NewPointer("test"),
|
||||
ChannelType: &chanTypeDirect,
|
||||
PostFileIds: []string{"12345", "54321"},
|
||||
},
|
||||
attachments: []*model.FileInfo{
|
||||
{Name: "test", Id: "12345", Path: "filename.txt", DeleteAt: 2},
|
||||
},
|
||||
expectedStarts: []*FileUploadStartExport{
|
||||
{UserEmail: "test@test.com", UploadStartTime: 1,
|
||||
FileInfo: &model.FileInfo{Name: "test", Id: "12345", Path: "filename.txt", DeleteAt: 2}},
|
||||
},
|
||||
expectedStops: []*FileUploadStopExport{
|
||||
{UserEmail: "test@test.com", UploadStopTime: 1,
|
||||
FileInfo: &model.FileInfo{Name: "test", Id: "12345", Path: "filename.txt", DeleteAt: 2}, Status: "Completed"},
|
||||
},
|
||||
expectedFileInfos: []*model.FileInfo{
|
||||
{Name: "test", Id: "12345", Path: "filename.txt", DeleteAt: 2},
|
||||
},
|
||||
expectedDeleteFileMessages: nil,
|
||||
ignoreDeleted: true,
|
||||
expectError: false,
|
||||
},
|
||||
}
|
||||
|
||||
for _, tc := range tt {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
mockStore := &storetest.Store{}
|
||||
defer mockStore.AssertExpectations(t)
|
||||
|
||||
if len(tc.attachments) > 0 {
|
||||
call := mockStore.FileInfoStore.On("GetForPost", *tc.post.PostId, true, true, false)
|
||||
call.Run(func(args mock.Arguments) {
|
||||
call.Return(tc.attachments, nil)
|
||||
})
|
||||
}
|
||||
files, uploadStarts, uploadStops, deleteFileMessages, err := postToAttachmentsEntries(&tc.post, NewMessageExportStore(mockStore), tc.ignoreDeleted)
|
||||
if tc.expectError {
|
||||
assert.Error(t, err)
|
||||
} else {
|
||||
assert.NoError(t, err)
|
||||
}
|
||||
|
||||
// Need to add the post to the expected starts (to save from copy pasting in the test cases above)
|
||||
for _, s := range tc.expectedStarts {
|
||||
s.MessageExport = tc.post
|
||||
}
|
||||
|
||||
// Need to add the post to the expected stops
|
||||
for _, s := range tc.expectedStops {
|
||||
s.MessageExport = tc.post
|
||||
}
|
||||
|
||||
// Need to add the post to the expected deleted
|
||||
var expectedDeleted []PostExport
|
||||
for _, s := range tc.expectedDeleteFileMessages {
|
||||
s.MessageExport = tc.post
|
||||
expectedDeleted = append(expectedDeleted, s)
|
||||
}
|
||||
|
||||
assert.Equal(t, tc.expectedStarts, uploadStarts)
|
||||
assert.Equal(t, tc.expectedStops, uploadStops)
|
||||
assert.Equal(t, tc.expectedFileInfos, files)
|
||||
assert.Equal(t, expectedDeleted, deleteFileMessages)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestGetJoinLeavePosts(t *testing.T) {
|
||||
mockStore := &storetest.Store{}
|
||||
defer mockStore.AssertExpectations(t)
|
||||
|
||||
// This would have been retrieved during CalculateChannelExports
|
||||
channelMemberHistories := map[string][]*model.ChannelMemberHistoryResult{
|
||||
"good-request-1": {
|
||||
{JoinTime: 1, UserId: "test1", UserEmail: "test1", Username: "test1"},
|
||||
{JoinTime: 2, LeaveTime: model.NewPointer(int64(3)), UserId: "test2", UserEmail: "test2", Username: "test2"},
|
||||
{JoinTime: 3, UserId: "test3", UserEmail: "test3", Username: "test3"},
|
||||
},
|
||||
"good-request-2": {
|
||||
{JoinTime: 4, UserId: "test4", UserEmail: "test4", Username: "test4"},
|
||||
{JoinTime: 5, LeaveTime: model.NewPointer(int64(6)), UserId: "test5", UserEmail: "test5", Username: "test5"},
|
||||
{JoinTime: 6, UserId: "test6", UserEmail: "test6", Username: "test6"},
|
||||
},
|
||||
}
|
||||
|
||||
var joins []JoinExport
|
||||
var leaves []LeaveExport
|
||||
for _, id := range []string{"good-request-1", "good-request-2"} {
|
||||
newJoins, newLeaves := getJoinsAndLeaves(
|
||||
1,
|
||||
7,
|
||||
channelMemberHistories[id],
|
||||
nil,
|
||||
)
|
||||
joins = append(joins, newJoins...)
|
||||
leaves = append(leaves, newLeaves...)
|
||||
}
|
||||
|
||||
assert.Len(t, joins, 6)
|
||||
assert.Equal(t, JoinExport{
|
||||
UserId: "test1",
|
||||
Username: "test1",
|
||||
UserEmail: "test1",
|
||||
UserType: User,
|
||||
JoinTime: 1,
|
||||
LeaveTime: 7,
|
||||
}, joins[0])
|
||||
assert.Equal(t, JoinExport{
|
||||
UserId: "test2",
|
||||
Username: "test2",
|
||||
UserEmail: "test2",
|
||||
UserType: User,
|
||||
JoinTime: 2,
|
||||
LeaveTime: 3,
|
||||
}, joins[1])
|
||||
assert.Equal(t, JoinExport{
|
||||
UserId: "test3",
|
||||
Username: "test3",
|
||||
UserEmail: "test3",
|
||||
UserType: User,
|
||||
JoinTime: 3,
|
||||
LeaveTime: 7,
|
||||
}, joins[2])
|
||||
assert.Equal(t, JoinExport{
|
||||
UserId: "test4",
|
||||
Username: "test4",
|
||||
UserEmail: "test4",
|
||||
UserType: User,
|
||||
JoinTime: 4,
|
||||
LeaveTime: 7,
|
||||
}, joins[3])
|
||||
assert.Equal(t, JoinExport{
|
||||
UserId: "test5",
|
||||
Username: "test5",
|
||||
UserEmail: "test5",
|
||||
UserType: User,
|
||||
JoinTime: 5,
|
||||
LeaveTime: 6,
|
||||
}, joins[4])
|
||||
assert.Equal(t, JoinExport{
|
||||
UserId: "test6",
|
||||
Username: "test6",
|
||||
UserEmail: "test6",
|
||||
UserType: User,
|
||||
JoinTime: 6,
|
||||
LeaveTime: 7,
|
||||
}, joins[5])
|
||||
|
||||
// remember that getJoinsAndLeaves sorts _for each channel_
|
||||
assert.Len(t, leaves, 6)
|
||||
// 1st channel:
|
||||
assert.Equal(t, LeaveExport{
|
||||
UserId: "test2",
|
||||
Username: "test2",
|
||||
UserEmail: "test2",
|
||||
UserType: User,
|
||||
LeaveTime: 3,
|
||||
}, leaves[0])
|
||||
assert.Equal(t, LeaveExport{
|
||||
UserId: "test1",
|
||||
Username: "test1",
|
||||
UserEmail: "test1",
|
||||
UserType: User,
|
||||
LeaveTime: 7,
|
||||
ClosedOut: true,
|
||||
}, leaves[1])
|
||||
assert.Equal(t, LeaveExport{
|
||||
UserId: "test3",
|
||||
Username: "test3",
|
||||
UserEmail: "test3",
|
||||
UserType: User,
|
||||
LeaveTime: 7,
|
||||
ClosedOut: true,
|
||||
}, leaves[2])
|
||||
// 2nd channel:
|
||||
assert.Equal(t, LeaveExport{
|
||||
UserId: "test5",
|
||||
Username: "test5",
|
||||
UserEmail: "test5",
|
||||
UserType: User,
|
||||
LeaveTime: 6,
|
||||
}, leaves[3])
|
||||
assert.Equal(t, LeaveExport{
|
||||
UserId: "test4",
|
||||
Username: "test4",
|
||||
UserEmail: "test4",
|
||||
UserType: User,
|
||||
LeaveTime: 7,
|
||||
ClosedOut: true,
|
||||
}, leaves[4])
|
||||
assert.Equal(t, LeaveExport{
|
||||
UserId: "test6",
|
||||
Username: "test6",
|
||||
UserEmail: "test6",
|
||||
UserType: User,
|
||||
LeaveTime: 7,
|
||||
ClosedOut: true,
|
||||
}, leaves[5])
|
||||
}
|
||||
21
server/enterprise/message_export/shared/main_test.go
Обычный файл
21
server/enterprise/message_export/shared/main_test.go
Обычный файл
@@ -0,0 +1,21 @@
|
||||
// Copyright (c) 2015-present Mattermost, Inc. All Rights Reserved.
|
||||
// See LICENSE.enterprise for license information.
|
||||
|
||||
package shared
|
||||
|
||||
import (
|
||||
"testing"
|
||||
|
||||
"github.com/mattermost/mattermost/server/v8/channels/api4"
|
||||
"github.com/mattermost/mattermost/server/v8/channels/testlib"
|
||||
)
|
||||
|
||||
var mainHelper *testlib.MainHelper
|
||||
|
||||
func TestMain(m *testing.M) {
|
||||
mainHelper = testlib.NewMainHelper()
|
||||
defer mainHelper.Close()
|
||||
api4.SetMainHelper(mainHelper)
|
||||
|
||||
mainHelper.Main(m)
|
||||
}
|
||||
609
server/enterprise/message_export/shared/shared.go
Обычный файл
609
server/enterprise/message_export/shared/shared.go
Обычный файл
@@ -0,0 +1,609 @@
|
||||
// Copyright (c) 2015-present Mattermost, Inc. All Rights Reserved.
|
||||
// See LICENSE.enterprise for license information.
|
||||
|
||||
package shared
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"path"
|
||||
"strconv"
|
||||
"time"
|
||||
|
||||
"github.com/pkg/errors"
|
||||
|
||||
"github.com/mattermost/mattermost/server/v8/platform/shared/filestore"
|
||||
"github.com/mattermost/mattermost/server/v8/platform/shared/templates"
|
||||
|
||||
"github.com/mattermost/mattermost/server/public/model"
|
||||
"github.com/mattermost/mattermost/server/public/shared/mlog"
|
||||
"github.com/mattermost/mattermost/server/public/shared/request"
|
||||
)
|
||||
|
||||
const (
|
||||
MissingFileMessage = "File missing for post; cannot copy file to archive"
|
||||
|
||||
EstimatedPostCount = 10_000_000
|
||||
|
||||
// JobDataBatchStartTime is the posts.updateat value from the previous batch. Posts are selected using
|
||||
// keyset pagination sorted by (posts.updateat, posts.id).
|
||||
JobDataBatchStartTime = "batch_start_time"
|
||||
|
||||
// JobDataJobStartTime is the start of the job (doesn't change across batches)
|
||||
JobDataJobStartTime = "job_start_time"
|
||||
|
||||
// JobDataBatchStartId is the posts.id value from the previous batch.
|
||||
JobDataBatchStartId = "batch_start_id"
|
||||
|
||||
// JobDataJobEndTime is the point up to which this job is exporting. It is the time the job was started,
|
||||
// i.e., we export everything from the end of previous batch to the moment this batch started.
|
||||
JobDataJobEndTime = "job_end_time"
|
||||
|
||||
JobDataJobStartId = "job_start_id"
|
||||
JobDataExportType = "export_type"
|
||||
JobDataBatchSize = "batch_size"
|
||||
JobDataChannelBatchSize = "channel_batch_size"
|
||||
JobDataChannelHistoryBatchSize = "channel_history_batch_size"
|
||||
JobDataMessagesExported = "messages_exported"
|
||||
JobDataWarningCount = "warning_count"
|
||||
JobDataIsDownloadable = "is_downloadable"
|
||||
JobDataExportDir = "export_dir"
|
||||
JobDataBatchNumber = "job_batch_number"
|
||||
JobDataTotalPostsExpected = "total_posts_expected"
|
||||
)
|
||||
|
||||
type PostUpdatedType string
|
||||
|
||||
const (
|
||||
EditedOriginalMsg PostUpdatedType = "EditedOriginalMsg"
|
||||
EditedNewMsg PostUpdatedType = "EditedNewMsg"
|
||||
UpdatedNoMsgChange PostUpdatedType = "UpdatedNoMsgChange"
|
||||
Deleted PostUpdatedType = "Deleted"
|
||||
FileDeleted PostUpdatedType = "FileDeleted"
|
||||
)
|
||||
|
||||
// JobData keeps the current state of the job.
|
||||
// When used by a worker, all fields in JobDataExported are exported to the job's job.Data prop bag.
|
||||
type JobData struct {
|
||||
JobDataExported
|
||||
|
||||
ExportPeriodStartTime int64
|
||||
|
||||
// This section is the current state of the export
|
||||
ChannelMetadata map[string]*MetadataChannel
|
||||
ChannelMemberHistories map[string][]*model.ChannelMemberHistoryResult
|
||||
Cursor model.MessageExportCursor
|
||||
PostsToExport []*model.MessageExport
|
||||
BatchEndTime int64
|
||||
BatchPath string
|
||||
MessageExportMs []int64
|
||||
ProcessingPostsMs []int64
|
||||
ProcessingXmlMs []int64
|
||||
TransferringFilesMs []int64
|
||||
TransferringZipMs []int64
|
||||
TotalBatchMs []int64
|
||||
Finished bool
|
||||
}
|
||||
|
||||
type JobDataExported struct {
|
||||
ExportType string
|
||||
ExportDir string
|
||||
BatchStartTime int64
|
||||
BatchStartId string
|
||||
JobStartTime int64
|
||||
JobEndTime int64
|
||||
JobStartId string
|
||||
BatchSize int
|
||||
ChannelBatchSize int
|
||||
ChannelHistoryBatchSize int
|
||||
BatchNumber int
|
||||
TotalPostsExpected int
|
||||
MessagesExported int
|
||||
WarningCount int
|
||||
IsDownloadable bool
|
||||
}
|
||||
|
||||
func JobDataToStringMap(jd JobData) map[string]string {
|
||||
ret := make(map[string]string)
|
||||
ret[JobDataExportType] = jd.ExportType
|
||||
ret[JobDataExportDir] = jd.ExportDir
|
||||
ret[JobDataBatchStartTime] = strconv.FormatInt(jd.BatchStartTime, 10)
|
||||
ret[JobDataBatchStartId] = jd.BatchStartId
|
||||
ret[JobDataJobStartTime] = strconv.FormatInt(jd.JobStartTime, 10)
|
||||
ret[JobDataJobEndTime] = strconv.FormatInt(jd.JobEndTime, 10)
|
||||
ret[JobDataJobStartId] = jd.JobStartId
|
||||
ret[JobDataBatchSize] = strconv.Itoa(jd.BatchSize)
|
||||
ret[JobDataChannelBatchSize] = strconv.Itoa(jd.ChannelBatchSize)
|
||||
ret[JobDataChannelHistoryBatchSize] = strconv.Itoa(jd.ChannelHistoryBatchSize)
|
||||
ret[JobDataBatchNumber] = strconv.Itoa(jd.BatchNumber)
|
||||
ret[JobDataTotalPostsExpected] = strconv.Itoa(jd.TotalPostsExpected)
|
||||
ret[JobDataMessagesExported] = strconv.Itoa(jd.MessagesExported)
|
||||
ret[JobDataWarningCount] = strconv.Itoa(jd.WarningCount)
|
||||
ret[JobDataIsDownloadable] = strconv.FormatBool(jd.IsDownloadable)
|
||||
return ret
|
||||
}
|
||||
|
||||
func StringMapToJobDataWithZeroValues(sm map[string]string) (JobData, error) {
|
||||
var jd JobData
|
||||
var err error
|
||||
|
||||
jd.ExportType = sm[JobDataExportType]
|
||||
jd.ExportDir = sm[JobDataExportDir]
|
||||
|
||||
batchStartTime, ok := sm[JobDataBatchStartTime]
|
||||
if !ok {
|
||||
batchStartTime = "0"
|
||||
}
|
||||
if jd.BatchStartTime, err = strconv.ParseInt(batchStartTime, 10, 64); err != nil {
|
||||
return jd, errors.Wrap(err, "error converting JobDataBatchStartTime")
|
||||
}
|
||||
|
||||
jd.BatchStartId = sm[JobDataBatchStartId]
|
||||
|
||||
jobStartTime, ok := sm[JobDataJobStartTime]
|
||||
if !ok {
|
||||
jobStartTime = "0"
|
||||
}
|
||||
if jd.JobStartTime, err = strconv.ParseInt(jobStartTime, 10, 64); err != nil {
|
||||
return jd, errors.Wrap(err, "error converting JobDataJobStartTime")
|
||||
}
|
||||
|
||||
jobEndTime, ok := sm[JobDataJobEndTime]
|
||||
if !ok {
|
||||
jobEndTime = "0"
|
||||
}
|
||||
if jd.JobEndTime, err = strconv.ParseInt(jobEndTime, 10, 64); err != nil {
|
||||
return jd, errors.Wrap(err, "error converting JobDataJobEndTime")
|
||||
}
|
||||
|
||||
jd.JobStartId = sm[JobDataJobStartId]
|
||||
|
||||
jobBatchSize, ok := sm[JobDataBatchSize]
|
||||
if !ok {
|
||||
jobBatchSize = "0"
|
||||
}
|
||||
if jd.BatchSize, err = strconv.Atoi(jobBatchSize); err != nil {
|
||||
return jd, errors.Wrap(err, "error converting JobDataBatchSize")
|
||||
}
|
||||
|
||||
channelBatchSize, ok := sm[JobDataChannelBatchSize]
|
||||
if !ok {
|
||||
channelBatchSize = "0"
|
||||
}
|
||||
if jd.ChannelBatchSize, err = strconv.Atoi(channelBatchSize); err != nil {
|
||||
return jd, errors.Wrap(err, "error converting JobDataChannelBatchSize")
|
||||
}
|
||||
|
||||
channelHistoryBatchSize, ok := sm[JobDataChannelHistoryBatchSize]
|
||||
if !ok {
|
||||
channelHistoryBatchSize = "0"
|
||||
}
|
||||
if jd.ChannelHistoryBatchSize, err = strconv.Atoi(channelHistoryBatchSize); err != nil {
|
||||
return jd, errors.Wrap(err, "error converting JobDataChannelHistoryBatchSize")
|
||||
}
|
||||
|
||||
batchNumber, ok := sm[JobDataBatchNumber]
|
||||
if !ok {
|
||||
batchNumber = "0"
|
||||
}
|
||||
if jd.BatchNumber, err = strconv.Atoi(batchNumber); err != nil {
|
||||
return jd, errors.Wrap(err, "error converting JobDataBatchNumber")
|
||||
}
|
||||
|
||||
totalPostsExpected, ok := sm[JobDataTotalPostsExpected]
|
||||
if !ok {
|
||||
totalPostsExpected = "0"
|
||||
}
|
||||
if jd.TotalPostsExpected, err = strconv.Atoi(totalPostsExpected); err != nil {
|
||||
return jd, errors.Wrap(err, "error converting JobDataTotalPostsExpected")
|
||||
}
|
||||
|
||||
messagesExported, ok := sm[JobDataMessagesExported]
|
||||
if !ok {
|
||||
messagesExported = "0"
|
||||
}
|
||||
if jd.MessagesExported, err = strconv.Atoi(messagesExported); err != nil {
|
||||
return jd, errors.Wrap(err, "error converting JobDataMessagesExported")
|
||||
}
|
||||
|
||||
warningCount, ok := sm[JobDataWarningCount]
|
||||
if !ok {
|
||||
warningCount = "0"
|
||||
}
|
||||
if jd.WarningCount, err = strconv.Atoi(warningCount); err != nil {
|
||||
return jd, errors.Wrap(err, "error converting JobDataWarningCount")
|
||||
}
|
||||
|
||||
isDownloadable, ok := sm[JobDataIsDownloadable]
|
||||
if !ok {
|
||||
isDownloadable = "0"
|
||||
}
|
||||
if jd.IsDownloadable, err = strconv.ParseBool(isDownloadable); err != nil {
|
||||
return jd, errors.Wrap(err, "error converting JobDataIsDownloadable")
|
||||
}
|
||||
|
||||
return jd, nil
|
||||
}
|
||||
|
||||
type BackendParams struct {
|
||||
Config *model.Config
|
||||
Store MessageExportStore
|
||||
FileAttachmentBackend filestore.FileBackend
|
||||
ExportBackend filestore.FileBackend
|
||||
HtmlTemplates *templates.Container
|
||||
}
|
||||
|
||||
type ExportParams struct {
|
||||
ExportType string
|
||||
ChannelMetadata map[string]*MetadataChannel
|
||||
Posts []*model.MessageExport
|
||||
ChannelMemberHistories map[string][]*model.ChannelMemberHistoryResult
|
||||
JobStartTime int64
|
||||
BatchPath string
|
||||
BatchStartTime int64
|
||||
BatchEndTime int64
|
||||
Config *model.Config
|
||||
Db MessageExportStore
|
||||
FileAttachmentBackend filestore.FileBackend
|
||||
ExportBackend filestore.FileBackend
|
||||
Templates *templates.Container
|
||||
}
|
||||
|
||||
type WriteExportResult struct {
|
||||
TransferringFilesMs int64
|
||||
ProcessingXmlMs int64
|
||||
TransferringZipMs int64
|
||||
NumWarnings int
|
||||
}
|
||||
|
||||
type RunExportResults struct {
|
||||
CreatedPosts int
|
||||
EditedOrigMsgPosts int
|
||||
EditedNewMsgPosts int
|
||||
UpdatedPosts int
|
||||
DeletedPosts int
|
||||
UploadedFiles int
|
||||
DeletedFiles int
|
||||
NumChannels int
|
||||
Joins int
|
||||
Leaves int
|
||||
ProcessingPostsMs int64
|
||||
WriteExportResult
|
||||
}
|
||||
|
||||
type ChannelMemberJoin struct {
|
||||
UserId string
|
||||
IsBot bool
|
||||
Email string
|
||||
Username string
|
||||
Datetime int64
|
||||
}
|
||||
|
||||
type ChannelMemberLeave struct {
|
||||
UserId string
|
||||
IsBot bool
|
||||
Email string
|
||||
Username string
|
||||
Datetime int64
|
||||
}
|
||||
|
||||
type ChannelMember struct {
|
||||
UserId string
|
||||
IsBot bool
|
||||
Email string
|
||||
Username string
|
||||
}
|
||||
|
||||
type MetadataChannel struct {
|
||||
TeamId *string
|
||||
TeamName *string
|
||||
TeamDisplayName *string
|
||||
ChannelId string
|
||||
ChannelName string
|
||||
ChannelDisplayName string
|
||||
ChannelType model.ChannelType
|
||||
RoomId string
|
||||
StartTime int64
|
||||
EndTime int64
|
||||
MessagesCount int
|
||||
AttachmentsCount int
|
||||
}
|
||||
|
||||
type Metadata struct {
|
||||
Channels map[string]*MetadataChannel
|
||||
MessagesCount int
|
||||
AttachmentsCount int
|
||||
StartTime int64
|
||||
EndTime int64
|
||||
}
|
||||
|
||||
func (metadata *Metadata) UpdateCounts(channelId string, numMessages int, numAttachments int) error {
|
||||
_, ok := metadata.Channels[channelId]
|
||||
if !ok {
|
||||
return fmt.Errorf("could not find channelId for post in metadata.Channels")
|
||||
}
|
||||
|
||||
metadata.Channels[channelId].AttachmentsCount += numAttachments
|
||||
metadata.AttachmentsCount += numAttachments
|
||||
metadata.Channels[channelId].MessagesCount += numMessages
|
||||
metadata.MessagesCount += numMessages
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// GetInitialExportPeriodData calculates and caches the channel memberships, channel metadata, and the TotalPostsExpected.
|
||||
func GetInitialExportPeriodData(rctx request.CTX, store MessageExportStore, data JobData, reportProgress func(string)) (JobData, error) {
|
||||
// Counting all posts may fail or timeout when the posts table is large. If this happens, log a warning, but carry
|
||||
// on with the job anyway. The only issue is that the progress % reporting will be inaccurate.
|
||||
count, err := store.Post().AnalyticsPostCount(&model.PostCountOptions{ExcludeSystemPosts: true, SincePostID: data.JobStartId, SinceUpdateAt: data.ExportPeriodStartTime, UntilUpdateAt: data.JobEndTime})
|
||||
if err != nil {
|
||||
rctx.Logger().Warn("Worker: Failed to fetch total post count for job. An estimated value will be used for progress reporting.", mlog.Err(err))
|
||||
data.TotalPostsExpected = EstimatedPostCount
|
||||
} else {
|
||||
data.TotalPostsExpected = int(count)
|
||||
}
|
||||
|
||||
rctx.Logger().Info("Expecting to export total posts", mlog.Int("total_posts", data.TotalPostsExpected))
|
||||
|
||||
// Every time we claim the job, we need to gather the membership data that every batch will use.
|
||||
// If we're here, then either this is the start of the job, or the job was stopped (e.g., the worker stopped)
|
||||
// and we've claimed it again. Either way, we need to recalculate channel and member history data.
|
||||
data.ChannelMetadata, data.ChannelMemberHistories, err = CalculateChannelExports(rctx,
|
||||
ChannelExportsParams{
|
||||
Store: store,
|
||||
ExportPeriodStartTime: data.ExportPeriodStartTime,
|
||||
ExportPeriodEndTime: data.JobEndTime,
|
||||
ChannelBatchSize: data.ChannelBatchSize,
|
||||
ChannelHistoryBatchSize: data.ChannelHistoryBatchSize,
|
||||
ReportProgressMessage: reportProgress,
|
||||
})
|
||||
if err != nil {
|
||||
return data, err
|
||||
}
|
||||
|
||||
data.Cursor = model.MessageExportCursor{
|
||||
LastPostUpdateAt: data.BatchStartTime,
|
||||
LastPostId: data.BatchStartId,
|
||||
UntilUpdateAt: data.JobEndTime,
|
||||
}
|
||||
|
||||
return data, nil
|
||||
}
|
||||
|
||||
type ChannelExportsParams struct {
|
||||
Store MessageExportStore
|
||||
ExportPeriodStartTime int64
|
||||
ExportPeriodEndTime int64
|
||||
ChannelBatchSize int
|
||||
ChannelHistoryBatchSize int
|
||||
ReportProgressMessage func(message string)
|
||||
}
|
||||
|
||||
// CalculateChannelExports returns the channel info ( map[channelId]*MetadataChannel ) and the channel user
|
||||
// joins/leaves ( map[channelId][]*model.ChannelMemberHistoryResult ) for any channel that has had activity
|
||||
// (posts or user join/leaves) between ExportPeriodStartTime and ExportPeriodEndTime.
|
||||
func CalculateChannelExports(rctx request.CTX, opt ChannelExportsParams) (map[string]*MetadataChannel, map[string][]*model.ChannelMemberHistoryResult, error) {
|
||||
// Which channels had user activity in the export period?
|
||||
activeChannelIds, err := opt.Store.ChannelMemberHistory().GetChannelsWithActivityDuring(opt.ExportPeriodStartTime, opt.ExportPeriodEndTime)
|
||||
if err != nil {
|
||||
return nil, nil, err
|
||||
}
|
||||
|
||||
if len(activeChannelIds) == 0 {
|
||||
return nil, nil, nil
|
||||
}
|
||||
|
||||
rctx.Logger().Debug("Started CalculateChannelExports", mlog.Int("export_period_start_time", opt.ExportPeriodStartTime), mlog.Int("export_period_end_time", opt.ExportPeriodEndTime), mlog.Int("num_active_channel_ids", len(activeChannelIds)))
|
||||
message := rctx.T("ent.message_export.actiance_export.calculate_channel_exports.channel_message", model.StringMap{"NumChannels": strconv.Itoa(len(activeChannelIds))})
|
||||
opt.ReportProgressMessage(message)
|
||||
|
||||
// For each channel, get its metadata.
|
||||
channelMetadata := make(map[string]*MetadataChannel, len(activeChannelIds))
|
||||
|
||||
// Use batches to reduce db load and network waste.
|
||||
for pos := 0; pos < len(activeChannelIds); pos += opt.ChannelBatchSize {
|
||||
upTo := min(pos+opt.ChannelBatchSize, len(activeChannelIds))
|
||||
batch := activeChannelIds[pos:upTo]
|
||||
channels, err := opt.Store.Channel().GetMany(batch, true)
|
||||
if err != nil {
|
||||
return nil, nil, err
|
||||
}
|
||||
|
||||
for _, channel := range channels {
|
||||
channelMetadata[channel.Id] = &MetadataChannel{
|
||||
TeamId: model.NewPointer(channel.TeamId),
|
||||
ChannelId: channel.Id,
|
||||
ChannelName: channel.Name,
|
||||
ChannelDisplayName: channel.DisplayName,
|
||||
ChannelType: channel.Type,
|
||||
RoomId: fmt.Sprintf("%v - %v", ChannelTypeDisplayName(channel.Type), channel.Id),
|
||||
StartTime: opt.ExportPeriodStartTime,
|
||||
EndTime: opt.ExportPeriodEndTime,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
historiesByChannelId := make(map[string][]*model.ChannelMemberHistoryResult, len(activeChannelIds))
|
||||
|
||||
var batchTimes []int64
|
||||
|
||||
// Now that we have metadata, get channelMemberHistories for each channel.
|
||||
// Use batches to reduce total db load and network waste.
|
||||
for pos := 0; pos < len(activeChannelIds); pos += opt.ChannelHistoryBatchSize {
|
||||
// This may take a while, so update the system console UI.
|
||||
message := rctx.T("ent.message_export.actiance_export.calculate_channel_exports.activity_message", model.StringMap{
|
||||
"NumChannels": strconv.Itoa(len(activeChannelIds)),
|
||||
"NumCompleted": strconv.Itoa(pos),
|
||||
})
|
||||
opt.ReportProgressMessage(message)
|
||||
|
||||
start := time.Now()
|
||||
|
||||
upTo := min(pos+opt.ChannelHistoryBatchSize, len(activeChannelIds))
|
||||
batch := activeChannelIds[pos:upTo]
|
||||
channelMemberHistories, err := opt.Store.ChannelMemberHistory().GetUsersInChannelDuring(opt.ExportPeriodStartTime, opt.ExportPeriodEndTime, batch)
|
||||
if err != nil {
|
||||
return nil, nil, err
|
||||
}
|
||||
|
||||
batchTimes = append(batchTimes, time.Since(start).Milliseconds())
|
||||
|
||||
// collect the channelMemberHistories by channelId
|
||||
for _, entry := range channelMemberHistories {
|
||||
historiesByChannelId[entry.ChannelId] = append(historiesByChannelId[entry.ChannelId], entry)
|
||||
}
|
||||
}
|
||||
|
||||
rctx.Logger().Info("GetUsersInChannelDuring batch times", mlog.Array("batch_times", batchTimes))
|
||||
|
||||
return channelMetadata, historiesByChannelId, nil
|
||||
}
|
||||
|
||||
// ChannelHasActivity returns true if the channel (represented by the []*model.ChannelMemberHistoryResult slice)
|
||||
// had user activity between startTime and endTime
|
||||
func ChannelHasActivity(cmhs []*model.ChannelMemberHistoryResult, startTime int64, endTime int64) bool {
|
||||
for _, cmh := range cmhs {
|
||||
if (cmh.JoinTime >= startTime && cmh.JoinTime <= endTime) ||
|
||||
(cmh.LeaveTime != nil && *cmh.LeaveTime >= startTime && *cmh.LeaveTime <= endTime) {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func GetJoinsAndLeavesForChannel(startTime int64, endTime int64, channelMembersHistory []*model.ChannelMemberHistoryResult,
|
||||
postAuthors map[string]ChannelMember) ([]ChannelMemberJoin, []ChannelMemberLeave) {
|
||||
var joins []ChannelMemberJoin
|
||||
var leaves []ChannelMemberLeave
|
||||
|
||||
alreadyJoined := make(map[string]bool)
|
||||
for _, cmh := range channelMembersHistory {
|
||||
if cmh.UserDeleteAt > 0 && cmh.UserDeleteAt < startTime {
|
||||
continue
|
||||
}
|
||||
|
||||
if cmh.JoinTime > endTime {
|
||||
continue
|
||||
}
|
||||
|
||||
if cmh.LeaveTime != nil && *cmh.LeaveTime < startTime {
|
||||
continue
|
||||
}
|
||||
|
||||
if cmh.JoinTime <= endTime {
|
||||
joins = append(joins, ChannelMemberJoin{
|
||||
UserId: cmh.UserId,
|
||||
IsBot: cmh.IsBot,
|
||||
Email: cmh.UserEmail,
|
||||
Username: cmh.Username,
|
||||
Datetime: cmh.JoinTime,
|
||||
})
|
||||
alreadyJoined[cmh.UserId] = true
|
||||
}
|
||||
|
||||
if cmh.LeaveTime != nil && *cmh.LeaveTime <= endTime {
|
||||
leaves = append(leaves, ChannelMemberLeave{
|
||||
UserId: cmh.UserId,
|
||||
IsBot: cmh.IsBot,
|
||||
Email: cmh.UserEmail,
|
||||
Username: cmh.Username,
|
||||
Datetime: *cmh.LeaveTime,
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
for _, member := range postAuthors {
|
||||
if alreadyJoined[member.UserId] {
|
||||
continue
|
||||
}
|
||||
|
||||
joins = append(joins, ChannelMemberJoin{
|
||||
UserId: member.UserId,
|
||||
IsBot: member.IsBot,
|
||||
Email: member.Email,
|
||||
Username: member.Username,
|
||||
Datetime: startTime,
|
||||
})
|
||||
}
|
||||
return joins, leaves
|
||||
}
|
||||
|
||||
// GetPostAttachments if the post included any files, we need to add special elements to the export.
|
||||
func GetPostAttachments(db MessageExportStore, post *model.MessageExport) ([]*model.FileInfo, error) {
|
||||
if len(post.PostFileIds) == 0 {
|
||||
return []*model.FileInfo{}, nil
|
||||
}
|
||||
|
||||
attachments, err := db.FileInfo().GetForPost(*post.PostId, true, true, false)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("failed to get file info for a post: %w", err)
|
||||
}
|
||||
return attachments, nil
|
||||
}
|
||||
|
||||
func ChannelTypeDisplayName(channelType model.ChannelType) string {
|
||||
return map[model.ChannelType]string{
|
||||
model.ChannelTypeOpen: "public",
|
||||
model.ChannelTypePrivate: "private",
|
||||
model.ChannelTypeDirect: "direct",
|
||||
model.ChannelTypeGroup: "group",
|
||||
}[channelType]
|
||||
}
|
||||
|
||||
func GetBatchPath(exportDir string, prevPostUpdateAt int64, lastPostUpdateAt int64, batchNumber int) string {
|
||||
if exportDir == "" {
|
||||
exportDir = path.Join(model.ComplianceExportPath, time.Now().Format(model.ComplianceExportDirectoryFormat))
|
||||
}
|
||||
return path.Join(exportDir,
|
||||
fmt.Sprintf("batch%03d-%d-%d.zip", batchNumber, prevPostUpdateAt, lastPostUpdateAt))
|
||||
}
|
||||
|
||||
// GetExportBackend returns the file backend where the export will be created.
|
||||
func GetExportBackend(rctx request.CTX, config *model.Config) (filestore.FileBackend, error) {
|
||||
insecure := config.ServiceSettings.EnableInsecureOutgoingConnections
|
||||
skipVerify := insecure != nil && *insecure
|
||||
|
||||
if config.FileSettings.DedicatedExportStore != nil && *config.FileSettings.DedicatedExportStore {
|
||||
rctx.Logger().Debug("Worker: using dedicated export filestore", mlog.String("driver_name", *config.FileSettings.ExportDriverName))
|
||||
backend, errFileBack := filestore.NewExportFileBackend(filestore.NewExportFileBackendSettingsFromConfig(&config.FileSettings, true, skipVerify))
|
||||
if errFileBack != nil {
|
||||
return nil, errFileBack
|
||||
}
|
||||
|
||||
return backend, nil
|
||||
}
|
||||
|
||||
backend, err := filestore.NewFileBackend(filestore.NewFileBackendSettingsFromConfig(&config.FileSettings, true, skipVerify))
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return backend, nil
|
||||
}
|
||||
|
||||
// GetFileAttachmentBackend returns the file backend where file attachments are
|
||||
// located for messages that will be exported. This may be the same backend
|
||||
// where the export will be created.
|
||||
func GetFileAttachmentBackend(rctx request.CTX, config *model.Config) (filestore.FileBackend, error) {
|
||||
insecure := config.ServiceSettings.EnableInsecureOutgoingConnections
|
||||
|
||||
backend, err := filestore.NewFileBackend(filestore.NewFileBackendSettingsFromConfig(&config.FileSettings, true, insecure != nil && *insecure))
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return backend, nil
|
||||
}
|
||||
|
||||
func IsDeletedMsg(post *model.MessageExport) bool {
|
||||
if model.SafeDereference(post.PostDeleteAt) > 0 && post.PostProps != nil {
|
||||
props := map[string]any{}
|
||||
err := json.Unmarshal([]byte(*post.PostProps), &props)
|
||||
if err != nil {
|
||||
return false
|
||||
}
|
||||
|
||||
if _, ok := props[model.PostPropsDeleteBy]; ok {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
286
server/enterprise/message_export/shared/shared_test.go
Обычный файл
286
server/enterprise/message_export/shared/shared_test.go
Обычный файл
@@ -0,0 +1,286 @@
|
||||
// Copyright (c) 2015-present Mattermost, Inc. All Rights Reserved.
|
||||
// See LICENSE.enterprise for license information.
|
||||
|
||||
package shared
|
||||
|
||||
import (
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/stretchr/testify/assert"
|
||||
|
||||
"github.com/mattermost/mattermost/server/public/model"
|
||||
)
|
||||
|
||||
func TestGetJoinsAndLeavesForChannel(t *testing.T) {
|
||||
channel := MetadataChannel{
|
||||
StartTime: 100,
|
||||
EndTime: 200,
|
||||
ChannelId: "good-request-1",
|
||||
TeamId: model.NewPointer("test"),
|
||||
TeamName: model.NewPointer("test"),
|
||||
TeamDisplayName: model.NewPointer("test"),
|
||||
ChannelName: "test",
|
||||
ChannelDisplayName: "test",
|
||||
ChannelType: "O",
|
||||
}
|
||||
|
||||
tt := []struct {
|
||||
name string
|
||||
channel MetadataChannel
|
||||
membersHistory []*model.ChannelMemberHistoryResult
|
||||
usersInPosts map[string]ChannelMember
|
||||
expectedJoins int
|
||||
expectedLeaves int
|
||||
}{
|
||||
{
|
||||
name: "no-joins-no-leaves",
|
||||
channel: channel,
|
||||
membersHistory: nil,
|
||||
usersInPosts: nil,
|
||||
expectedJoins: 0,
|
||||
expectedLeaves: 0,
|
||||
},
|
||||
{
|
||||
name: "joins-and-leaves-outside-the-range",
|
||||
channel: channel,
|
||||
membersHistory: []*model.ChannelMemberHistoryResult{
|
||||
{JoinTime: 1, LeaveTime: model.NewPointer(int64(10)), UserId: "test", UserEmail: "test", Username: "test"},
|
||||
{JoinTime: 250, LeaveTime: model.NewPointer(int64(260)), UserId: "test", UserEmail: "test", Username: "test"},
|
||||
{JoinTime: 300, UserId: "test", UserEmail: "test", Username: "test"},
|
||||
},
|
||||
usersInPosts: nil,
|
||||
expectedJoins: 0,
|
||||
expectedLeaves: 0,
|
||||
},
|
||||
{
|
||||
name: "join-and-leave-during-the-range",
|
||||
channel: channel,
|
||||
membersHistory: []*model.ChannelMemberHistoryResult{
|
||||
{JoinTime: 100, LeaveTime: model.NewPointer(int64(150)), UserId: "test", UserEmail: "test", Username: "test"},
|
||||
},
|
||||
usersInPosts: nil,
|
||||
expectedJoins: 1,
|
||||
expectedLeaves: 1,
|
||||
},
|
||||
{
|
||||
name: "join-during-and-leave-after-the-range",
|
||||
channel: channel,
|
||||
membersHistory: []*model.ChannelMemberHistoryResult{
|
||||
{JoinTime: 150, LeaveTime: model.NewPointer(int64(300)), UserId: "test", UserEmail: "test", Username: "test"},
|
||||
},
|
||||
usersInPosts: nil,
|
||||
expectedJoins: 1,
|
||||
expectedLeaves: 0,
|
||||
},
|
||||
{
|
||||
name: "join-before-and-leave-during-the-range",
|
||||
channel: channel,
|
||||
membersHistory: []*model.ChannelMemberHistoryResult{
|
||||
{JoinTime: 99, LeaveTime: model.NewPointer(int64(150)), UserId: "test", UserEmail: "test", Username: "test"},
|
||||
},
|
||||
usersInPosts: nil,
|
||||
expectedJoins: 1,
|
||||
expectedLeaves: 1,
|
||||
},
|
||||
{
|
||||
name: "join-before-and-leave-after-the-range",
|
||||
channel: channel,
|
||||
membersHistory: []*model.ChannelMemberHistoryResult{
|
||||
{JoinTime: 99, LeaveTime: model.NewPointer(int64(350)), UserId: "test", UserEmail: "test", Username: "test"},
|
||||
},
|
||||
usersInPosts: nil,
|
||||
expectedJoins: 1,
|
||||
expectedLeaves: 0,
|
||||
},
|
||||
{
|
||||
name: "implicit-joins",
|
||||
channel: channel,
|
||||
membersHistory: nil,
|
||||
usersInPosts: map[string]ChannelMember{
|
||||
"test1": {UserId: "test1", Email: "test1", Username: "test1"},
|
||||
"test2": {UserId: "test2", Email: "test2", Username: "test2"},
|
||||
},
|
||||
expectedJoins: 2,
|
||||
expectedLeaves: 0,
|
||||
},
|
||||
{
|
||||
name: "implicit-joins-with-explicit-joins",
|
||||
channel: channel,
|
||||
membersHistory: []*model.ChannelMemberHistoryResult{
|
||||
{JoinTime: 130, LeaveTime: model.NewPointer(int64(150)), UserId: "test1", UserEmail: "test1", Username: "test1"},
|
||||
{JoinTime: 130, LeaveTime: model.NewPointer(int64(150)), UserId: "test3", UserEmail: "test3", Username: "test3"},
|
||||
},
|
||||
usersInPosts: map[string]ChannelMember{
|
||||
"test1": {UserId: "test1", Email: "test1", Username: "test1"},
|
||||
"test2": {UserId: "test2", Email: "test2", Username: "test2"},
|
||||
},
|
||||
expectedJoins: 3,
|
||||
expectedLeaves: 2,
|
||||
},
|
||||
{
|
||||
name: "join-leave-and-join-again",
|
||||
channel: channel,
|
||||
membersHistory: []*model.ChannelMemberHistoryResult{
|
||||
{JoinTime: 130, LeaveTime: model.NewPointer(int64(150)), UserId: "test1", UserEmail: "test1", Username: "test1"},
|
||||
{JoinTime: 160, LeaveTime: model.NewPointer(int64(180)), UserId: "test1", UserEmail: "test1", Username: "test1"},
|
||||
},
|
||||
usersInPosts: nil,
|
||||
expectedJoins: 2,
|
||||
expectedLeaves: 2,
|
||||
},
|
||||
{
|
||||
name: "deactivated-members-dont-show",
|
||||
channel: channel,
|
||||
membersHistory: []*model.ChannelMemberHistoryResult{
|
||||
{JoinTime: 130, LeaveTime: model.NewPointer(int64(150)), UserId: "test1", UserEmail: "test1", Username: "test1", UserDeleteAt: 50},
|
||||
{JoinTime: 160, LeaveTime: model.NewPointer(int64(180)), UserId: "test1", UserEmail: "test1", Username: "test1", UserDeleteAt: 50},
|
||||
},
|
||||
usersInPosts: nil,
|
||||
expectedJoins: 0,
|
||||
expectedLeaves: 0,
|
||||
},
|
||||
{
|
||||
name: "deactivated-members-show-if-deleted-after-latest-export",
|
||||
channel: channel,
|
||||
membersHistory: []*model.ChannelMemberHistoryResult{
|
||||
{JoinTime: 130, LeaveTime: model.NewPointer(int64(150)), UserId: "test1", UserEmail: "test1", Username: "test1", UserDeleteAt: 150},
|
||||
{JoinTime: 160, LeaveTime: model.NewPointer(int64(180)), UserId: "test1", UserEmail: "test1", Username: "test1", UserDeleteAt: 150},
|
||||
},
|
||||
usersInPosts: nil,
|
||||
expectedJoins: 2,
|
||||
expectedLeaves: 2,
|
||||
},
|
||||
{
|
||||
name: "deactivated-members-show-and-dont-show",
|
||||
channel: channel,
|
||||
membersHistory: []*model.ChannelMemberHistoryResult{
|
||||
{JoinTime: 130, LeaveTime: model.NewPointer(int64(150)), UserId: "test1", UserEmail: "test1", Username: "test1", UserDeleteAt: 50},
|
||||
{JoinTime: 160, LeaveTime: model.NewPointer(int64(180)), UserId: "test1", UserEmail: "test1", Username: "test1", UserDeleteAt: 150},
|
||||
},
|
||||
usersInPosts: nil,
|
||||
expectedJoins: 1,
|
||||
expectedLeaves: 1,
|
||||
},
|
||||
}
|
||||
|
||||
for _, tc := range tt {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
joins, leaves := GetJoinsAndLeavesForChannel(tc.channel.StartTime, tc.channel.EndTime, tc.membersHistory, tc.usersInPosts)
|
||||
assert.Len(t, joins, tc.expectedJoins)
|
||||
assert.Len(t, leaves, tc.expectedLeaves)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func Test_GetBatchPath(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
exportDir string
|
||||
prevPostUpdateAt int64
|
||||
lastPostUpdateAt int64
|
||||
batchNumber int
|
||||
want string
|
||||
}{
|
||||
{
|
||||
name: "all args given",
|
||||
exportDir: "/export/test_dir",
|
||||
prevPostUpdateAt: 123,
|
||||
lastPostUpdateAt: 456,
|
||||
batchNumber: 21,
|
||||
want: "/export/test_dir/batch021-123-456.zip",
|
||||
},
|
||||
{
|
||||
name: "exportDir blank",
|
||||
exportDir: "",
|
||||
prevPostUpdateAt: 12345,
|
||||
lastPostUpdateAt: 456789,
|
||||
batchNumber: 921,
|
||||
want: model.ComplianceExportPath + "/" + time.Now().Format(model.ComplianceExportDirectoryFormat) + "/batch921-12345-456789.zip",
|
||||
},
|
||||
}
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
assert.Equalf(t, tt.want, GetBatchPath(tt.exportDir, tt.prevPostUpdateAt, tt.lastPostUpdateAt, tt.batchNumber), "GetBatchPath(%v, %v, %v, %v)", tt.exportDir, tt.prevPostUpdateAt, tt.lastPostUpdateAt, tt.batchNumber)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestJobDataToStringMap_and_StringMapToJobData(t *testing.T) {
|
||||
jd := JobData{
|
||||
JobDataExported: JobDataExported{
|
||||
ExportType: "cli_message_export",
|
||||
ExportDir: "/here/there/34234-123",
|
||||
BatchStartTime: 45,
|
||||
BatchStartId: "34arsitenaorsten",
|
||||
JobStartTime: 99,
|
||||
JobEndTime: 1234,
|
||||
JobStartId: "99abcdef34",
|
||||
BatchSize: 2000,
|
||||
ChannelBatchSize: 30000,
|
||||
ChannelHistoryBatchSize: 30,
|
||||
BatchNumber: 4,
|
||||
TotalPostsExpected: 999999,
|
||||
MessagesExported: 343499,
|
||||
WarningCount: 39,
|
||||
},
|
||||
ExportPeriodStartTime: 123456, // not exported
|
||||
BatchEndTime: 999999999, // not exported
|
||||
BatchPath: "/another/path/123-123", // not exported
|
||||
}
|
||||
|
||||
strMap := JobDataToStringMap(jd)
|
||||
|
||||
expected := make(map[string]string)
|
||||
expected[JobDataExportType] = "cli_message_export"
|
||||
expected[JobDataExportDir] = "/here/there/34234-123"
|
||||
expected[JobDataBatchStartTime] = "45"
|
||||
expected[JobDataBatchStartId] = "34arsitenaorsten"
|
||||
expected[JobDataJobStartTime] = "99"
|
||||
expected[JobDataJobEndTime] = "1234"
|
||||
expected[JobDataJobStartId] = "99abcdef34"
|
||||
expected[JobDataBatchSize] = "2000"
|
||||
expected[JobDataChannelBatchSize] = "30000"
|
||||
expected[JobDataChannelHistoryBatchSize] = "30"
|
||||
expected[JobDataBatchNumber] = "4"
|
||||
expected[JobDataTotalPostsExpected] = "999999"
|
||||
expected[JobDataMessagesExported] = "343499"
|
||||
expected[JobDataWarningCount] = "39"
|
||||
expected[JobDataIsDownloadable] = "false"
|
||||
|
||||
for k, v := range expected {
|
||||
val, ok := strMap[k]
|
||||
assert.True(t, ok)
|
||||
assert.Equal(t, v, val)
|
||||
}
|
||||
|
||||
// not exported:
|
||||
for _, k := range []string{"export_period_start_time", "batch_end_time", "batch_path"} {
|
||||
_, ok := strMap[k]
|
||||
assert.False(t, ok)
|
||||
}
|
||||
|
||||
// zero the fields that weren't exported:
|
||||
jd.ExportPeriodStartTime = 0
|
||||
jd.BatchEndTime = 0
|
||||
jd.BatchPath = ""
|
||||
|
||||
// now convert back
|
||||
jd2, err := StringMapToJobDataWithZeroValues(strMap)
|
||||
assert.NoError(t, err)
|
||||
assert.Equal(t, jd, jd2)
|
||||
|
||||
// and test bad conversion (just a couple):
|
||||
badStrMap := map[string]string{JobDataJobStartTime: "56aaa"}
|
||||
_, err = StringMapToJobDataWithZeroValues(badStrMap)
|
||||
assert.Error(t, err)
|
||||
badStrMap = map[string]string{JobDataJobEndTime: "blah blah"}
|
||||
_, err = StringMapToJobDataWithZeroValues(badStrMap)
|
||||
assert.Error(t, err)
|
||||
|
||||
// test that zero values are used when not present
|
||||
emptyStrMap := make(map[string]string)
|
||||
emptyJd, err := StringMapToJobDataWithZeroValues(emptyStrMap)
|
||||
assert.NoError(t, err)
|
||||
assert.Equal(t, JobData{}, emptyJd)
|
||||
}
|
||||
36
server/enterprise/message_export/shared/store.go
Обычный файл
36
server/enterprise/message_export/shared/store.go
Обычный файл
@@ -0,0 +1,36 @@
|
||||
// Copyright (c) 2015-present Mattermost, Inc. All Rights Reserved.
|
||||
// See LICENSE.enterprise for license information.
|
||||
|
||||
package shared
|
||||
|
||||
import (
|
||||
"github.com/mattermost/mattermost/server/public/model"
|
||||
"github.com/mattermost/mattermost/server/v8/channels/store"
|
||||
)
|
||||
|
||||
type MessageExportStore interface {
|
||||
Post() store.PostStore
|
||||
ChannelMemberHistory() store.ChannelMemberHistoryStore
|
||||
Channel() store.ChannelStore
|
||||
Compliance() store.ComplianceStore
|
||||
FileInfo() MEFileInfoStore
|
||||
}
|
||||
|
||||
type MEFileInfoStore interface {
|
||||
GetForPost(postID string, readFromMaster, includeDeleted, allowFromCache bool) ([]*model.FileInfo, error)
|
||||
}
|
||||
|
||||
type messageExportStore struct {
|
||||
store.Store
|
||||
}
|
||||
|
||||
// NewMessageExportStore returns a wrapped store.Store. Why? Because the CLI tool needs to "override" the GetForPost
|
||||
// method in FileStore. Non-CLI code only needs to use the store.Store's methods. This can be removed when we deprecate
|
||||
// the CLI tool. Tracked at MM-61139.
|
||||
func NewMessageExportStore(s store.Store) messageExportStore {
|
||||
return messageExportStore{s}
|
||||
}
|
||||
|
||||
func (ss messageExportStore) FileInfo() MEFileInfoStore {
|
||||
return ss.Store.FileInfo()
|
||||
}
|
||||
Ссылка в новой задаче
Block a user