MM-59966 - Compliance Export overhaul - feature branch (#29789)
* [MM-59089] Add a compliance export constant (#27919) * add a useful constant * i18n * another constant * another i18n * [MM-60422] Add GetChannelsWithActivityDuring (#28301) * modify GetUsersInChannelDuring to accept a slice of channelIds * add GetChannelsWithActivityDuring * add compliance export progress message; remove unused custom status * linting * tests running too fast * add batch size config settings * add store tests * linting * empty commit * i18n changes * fix i18n ordering * MM-60570 - Server-side changes consolidating the export CLI with server/ent code (#28640) * add an i18n field; add the CLI's export directory * int64 -> int * Add UntilUpdateAt for MessageExport and AnalyticsPostCount to merge * remove now-unused i18n strings * add TranslationsPreInitFromBuffer to allow CLI to use i18n * use GetBuilder to simplify; rename TranslationsPreInitFromFileBytes * [MM-59089] Improve compliance export timings (#1733 - Enterprise repo) * MM-60422 - Performance and logic fixes for Compliance Exports (#1757 - Enterprise repo) * MM-60570 - Enterprise-side changes consolidating the export CLI with server/ent code (#1769 - Enterprise repo) * merge conflicts; missed file from ent branch * MM-61038 - Add an option to sqlstore.New (#28702) remove useless comment add test add an option to sqlstore.New * MM-60976: Remove RunExport command from Mattermost binary (#28805) * remove RunExport command from mattermost binary * remove the code it was calling * fix i18n * remove test (was only testing license, not functionality) * empty commit * fix flaky GetChannelsWithActivityDuring test * MM-60063: Dedicated Export Filestore fix, redo of #1772 (enterprise) (#28803) * redo filestore fix #1772 (enterprise repo) on top of MM-59966 feature * add new e2e tests for export filestore * golint * ok, note to self: shadowing bad, actually (when there's a defer) * empty commit * MM-61137 - Message export: Support 7.8.11 era dbs (#28824) * support 7.8.11 era dbs by wrapping the store using only what we need * fix flaky GetChannelsWithActivityDuring test * add a comment * only need to define the MEFileInfoStore (the one that'll be overridden) * blank commit * MM-60974 - Message Export: Add performance metrics (#28836) * support 7.8.11 era dbs by wrapping the store using only what we need * fix flaky GetChannelsWithActivityDuring test * add a comment * only need to define the MEFileInfoStore (the one that'll be overridden) * performance metrics * cleanup unneeded named returns * blank commit * MM-60975 - Message export: Add startTime and endTime to export folder name (#28840) * support 7.8.11 era dbs by wrapping the store using only what we need * fix flaky GetChannelsWithActivityDuring test * add a comment * only need to define the MEFileInfoStore (the one that'll be overridden) * performance metrics * output startTime and endTime in export folder * empty commit * merge conflict * MM-60978 - Message export: Improve xml fields; fix delete semantics (#28873) * support 7.8.11 era dbs by wrapping the store using only what we need * fix flaky GetChannelsWithActivityDuring test * add a comment * only need to define the MEFileInfoStore (the one that'll be overridden) * performance metrics * output startTime and endTime in export folder * empty commit * add xml fields, omit when empty, tests * fix delete semantics; test (and test for update semantics) * clarify comments * simplify edited post detection, now there's no edge case. * add some spacing to help fast running tests * merge conflicts/updates needed for new deleted post semantics * linting; fixing tests from upstream merge * use SafeDereference * linting * stronger typing; better wrapped errors; better formatting * blank commit * goimports formatting * fix merge mistake * minor fixes due to changes in master * MM-61755 - Simplifying and Support reporting to the db from the CLI (#29281) * finally clean up JobData struct and stringMap; prep for CLI using db * and now simplify using StringMapToJobDataWithZeroValues * remove unused fn * create JobDataExported; clean up errors * MM-60176 - Message Export: Global relay cleanup (#29168) * move global relay logic into global_relay_export * blank commit * blank commit * improve errors * MM-60693 - Refactor CSV to use same codepath as Actiance (#29191) * move global relay logic into global_relay_export * blank commit * refactor (and simplify) ExportParams into shared * blank commit * remove unused fn * csv now uses pre-calculated joins/leaves like actiance * improve errors * remove nil post check; remove ignoredPosts metric * remove unneeded copy * MM-61696 - Refactor GlobalRelay to use same codepath as Actiance (#29225) * move global relay logic into global_relay_export * blank commit * refactor (and simplify) ExportParams into shared * blank commit * remove unused fn * csv now uses pre-calculated joins/leaves like actiance * remove newly unneeded function and its test. goodbye. * refactor GetPostAttachments for csv + global relay to share * refactor global_relay_export and fix tests (no changes to output) * improve errors * remove nil post check; remove ignoredPosts metric * remove unneeded copy * remove unneeded nil check * PR comments * MM-61715 - Generalize e2e to all export types 🤖 (#29369) * move global relay logic into global_relay_export * blank commit * refactor (and simplify) ExportParams into shared * blank commit * remove unused fn * csv now uses pre-calculated joins/leaves like actiance * remove newly unneeded function and its test. goodbye. * refactor GetPostAttachments for csv + global relay to share * refactor global_relay_export and fix tests (no changes to output) * improve errors * remove nil post check; remove ignoredPosts metric * remove unneeded copy * remove unneeded nil check * PR comments * refactor isDeletedMsg for all export types * fix start and endtime, nasty csv createAt bug; bring closer to Actiance * align unit tests with new logic (e.g. starttime / endtime) * refactor a TimestampConvert fn for code + tests * bug: pass templates to global relay (hurray for e2e tests, otherwise...) * add global relay zip to allowed list (only for tests) * test helpers * new templates for e2e tests * e2e tests... phew. * linting * merge conflicts * unexport PostToRow; add test helper marker * cleanup, shortening, thanks to PR comments * MM-61972 - Generalize export data path - Actiance (#29399) * extract and generalize the export data generation functions * finish moving test (bc of previous extraction) * lift a function from common -> shared (to break an import cycle) * actiance now takes general export data, processes it into actiance data * bring tests in line with correct sorting rules (upadateAt, messageId) * fixups, PR comments * turn strings.Repeat into a more descriptive const amended: one letter fix; bad rebase * MM-62009 - e2e clock heisenbug (#29434) * consolidate assertions; output debuggable diffs (keeping for future) * refactor test output generator to generators file * waitUntilZeroPosts + pass through until to job = fix all clock issues * simplify messages to model.NewId(); remove unneeded waitUntilZeroPosts * model.NewId() -> storetest.NewTestID() * MM-61980 - Generalize export data path - CSV (#29482) * simple refactoring * increase sleep times for (very) rare test failures * add extra information to the generic export for CSV * adj Actiance to handle new generic export (no difference in its output) * no longer need mergePosts (yay), move getJoinLeavePosts for everyone * adjust tests for new csv semantics (detailed in summary) * and need to add the new exported data to the export_data_tests * rearrange csv writing to happen after data export (more logical) * linting * remove debug statements * figured out what was wrong with global relay e2e test 3; solid now * PR comments * MM-61718 - Generalize export data path - Global Relay (#29508) * move global relay over to using the generalized export data * performance pass -- not much can be done * Update server/enterprise/message_export/global_relay_export/global_relay_export.go Co-authored-by: Claudio Costa <cstcld91@gmail.com> --------- Co-authored-by: Claudio Costa <cstcld91@gmail.com> * MM-62058 - Align CSV with Actiance (#29551) * refactoring actiance files and var names for clarity * bug found in exported attachments (we used to miss some start/ends) * changes needed for actiance due to new generic exports * bringing CSV up to actiance standards * fixing global relay b/c of new semantics (adding a note on an edge case) * aligning e2e tests, adding comments to clarify what is expected/tested * necessary changes; 1 more test for added functionality (ignoreDeleted) * comment style * MM-62059 - Align Global Relay with Actiance/CSV; many fixes (#29665) * core logic changes to general export_data and the specific export paths * unit tests and e2e tests, covering all new edge cases and all logic * linting * better var naming, const value, and cleaning up functions calls * MM-62436 - Temporarily skip cypress tests that require download link (#29772) --------- Co-authored-by: Claudio Costa <cstcld91@gmail.com>
Этот коммит содержится в:
Christopher Poile
коммит произвёл
GitHub
GitHub
родитель
96195f1bd7
Коммит
aba4434dab
@@ -4,29 +4,30 @@
|
||||
package actiance_export
|
||||
|
||||
import (
|
||||
"archive/zip"
|
||||
"bytes"
|
||||
"encoding/json"
|
||||
"encoding/xml"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"path"
|
||||
"sort"
|
||||
"os"
|
||||
"slices"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/mattermost/mattermost/server/v8/enterprise/message_export/common_export"
|
||||
"github.com/mattermost/mattermost/server/v8/enterprise/internal/file"
|
||||
|
||||
"github.com/mattermost/mattermost/server/public/model"
|
||||
"github.com/mattermost/mattermost/server/public/shared/mlog"
|
||||
"github.com/mattermost/mattermost/server/public/shared/request"
|
||||
"github.com/mattermost/mattermost/server/v8/channels/store"
|
||||
"github.com/mattermost/mattermost/server/v8/enterprise/message_export/shared"
|
||||
"github.com/mattermost/mattermost/server/v8/platform/shared/filestore"
|
||||
)
|
||||
|
||||
const (
|
||||
XMLNS = "http://www.w3.org/2001/XMLSchema-instance"
|
||||
ActianceExportFilename = "actiance_export.xml"
|
||||
ActianceWarningFilename = "warning.txt"
|
||||
XMLNS = "http://www.w3.org/2001/XMLSchema-instance"
|
||||
ActianceExportFilename = "actiance_export.xml"
|
||||
ActianceWarningFilename = "warning.txt"
|
||||
EnsureLastMessageInASort = "zzzzzzzzzzzzzzzzzzzzzzzzzz"
|
||||
)
|
||||
|
||||
// The root-level element of an actiance export
|
||||
@@ -39,17 +40,15 @@ type RootNode struct {
|
||||
// The Conversation element indicates an ad hoc IM conversation or a group chat room.
|
||||
// The messages from a persistent chat room are exported once a day so that a Conversation entry contains the messages posted to a chat room from 12:00:00 AM to 11:59:59 PM
|
||||
type ChannelExport struct {
|
||||
XMLName xml.Name `xml:"Conversation"`
|
||||
Perspective string `xml:"Perspective,attr"` // the value of this attribute doesn't seem to matter. Using the channel name makes the export more human readable
|
||||
ChannelId string `xml:"-"` // the unique id of the channel
|
||||
RoomId string `xml:"RoomID"`
|
||||
StartTime int64 `xml:"StartTimeUTC"` // utc timestamp (seconds), start of export period or create time of channel, whichever is greater. Example: 1366611728.
|
||||
JoinEvents []JoinExport // start with a list of all users who were present in the channel during the export period
|
||||
Elements []any
|
||||
UploadStarts []*FileUploadStartExport
|
||||
UploadStops []*FileUploadStopExport
|
||||
LeaveEvents []LeaveExport // finish with a list of all users who were present in the channel during the export period
|
||||
EndTime int64 `xml:"EndTimeUTC"` // utc timestamp (seconds), end of export period or delete time of channel, whichever is lesser. Example: 1366611728.
|
||||
XMLName xml.Name `xml:"Conversation"`
|
||||
Perspective string `xml:"Perspective,attr"` // the value of this attribute doesn't seem to matter. Using the channel name makes the export more human readable
|
||||
ChannelId string `xml:"-"` // the unique id of the channel
|
||||
RoomId string `xml:"RoomID"`
|
||||
StartTime int64 `xml:"StartTimeUTC"` // utc timestamp (seconds), start of export period or create time of channel, whichever is greater. Example: 1366611728.
|
||||
JoinEvents []JoinExport // start with a list of all users who were present in the channel during the export period
|
||||
Elements []Sortable
|
||||
LeaveEvents []LeaveExport // finish with a list of all users who were present in the channel during the export period
|
||||
EndTime int64 `xml:"EndTimeUTC"` // utc timestamp (seconds), end of export period or delete time of channel, whichever is lesser. Example: 1366611728.
|
||||
}
|
||||
|
||||
// The ParticipantEntered element indicates each user who participates in a conversation.
|
||||
@@ -72,14 +71,55 @@ type LeaveExport struct {
|
||||
CorporateEmailID string `xml:"CorporateEmailID"`
|
||||
}
|
||||
|
||||
type Sortable interface {
|
||||
SortVal() (int64, string)
|
||||
}
|
||||
|
||||
func SortableSort(a, b Sortable) int {
|
||||
aTimestamp, aId := a.SortVal()
|
||||
bTimestamp, bId := b.SortVal()
|
||||
if aTimestamp == bTimestamp {
|
||||
return strings.Compare(aId, bId)
|
||||
}
|
||||
return int(aTimestamp - bTimestamp)
|
||||
}
|
||||
|
||||
// The Message element indicates the message sent by a user
|
||||
type PostExport struct {
|
||||
XMLName xml.Name `xml:"Message"`
|
||||
UserEmail string `xml:"LoginName"` // the email of the person that sent the post
|
||||
UserType string `xml:"UserType"` // the type of the person that sent the post
|
||||
PostTime int64 `xml:"DateTimeUTC"` // utc timestamp (seconds), time at which the user sent the post. Example: 1366611728
|
||||
Message string `xml:"Content"` // the text body of the post
|
||||
PreviewsPost string `xml:"PreviewsPost"` // the post id of the post that is previewed by the permalink preview feature
|
||||
XMLName xml.Name `xml:"Message"`
|
||||
MessageId string `xml:"MessageId"` // the message id in the db
|
||||
UserEmail string `xml:"LoginName"` // the email of the person that sent the post
|
||||
UserType shared.UserType `xml:"UserType"` // the type of the person that sent the post: "user" or "bot"
|
||||
CreateAt int64 `xml:"DateTimeUTC"` // utc timestamp (unix milliseconds), the post's createAt
|
||||
|
||||
// Allows us to differentiate between:
|
||||
// - "EditedOriginalMsg": the newly created message (new Id), which holds the pre-edited message contents. The
|
||||
// "EditedNewMsgId" field will point to the message (original Id) which has the post-edited message content.
|
||||
// - "EditedNewMsg": the post-edited message content. This is confusing, so be careful: in the db, this EditedNewMsg
|
||||
// is actually the original messageId because we wanted an edited message to have the same messageId as the
|
||||
// pre-edited message. But for the purposes of exporting and to keep the mental model clear for end-users, we are
|
||||
// calling this the EditedNewMsg and EditedNewMsgId, because this will hold the NEW post-edited message contents,
|
||||
// and that's what's important to the end-user viewing the export.
|
||||
// - "UpdatedNoMsgChange": the message content hasn't changed, but the post was updated for some reason (reaction,
|
||||
// replied-to, a reply was edited, a reply was deleted (as of 10.2), perhaps other reasons)
|
||||
// - "Deleted": the message was deleted.
|
||||
// - "FileDeleted": this message is recording that a file was deleted.
|
||||
UpdatedType shared.PostUpdatedType `xml:"UpdatedType,omitempty"`
|
||||
UpdateAt int64 `xml:"UpdatedDateTimeUTC,omitempty"` // if this is an updated post, this is the updated time (same as deleted time for deleted posts).
|
||||
|
||||
// when a message is edited, the EditedOriginalMsg points to the message Id that now has the newly edited message.
|
||||
EditedNewMsgId string `xml:"EditedNewMsgId,omitempty"`
|
||||
|
||||
Message string `xml:"Content"` // the text body of the post
|
||||
PreviewsPost string `xml:"PreviewsPost,omitempty"` // the post id of the post that is previewed by the permalink preview feature
|
||||
}
|
||||
|
||||
func (p PostExport) SortVal() (int64, string) {
|
||||
// updated messages are sorted by UpdateAt
|
||||
if p.UpdatedType != "" {
|
||||
return p.UpdateAt, p.MessageId
|
||||
}
|
||||
return p.CreateAt, p.MessageId
|
||||
}
|
||||
|
||||
// The FileTransferStarted element indicates the beginning of a file transfer in a conversation
|
||||
@@ -91,6 +131,11 @@ type FileUploadStartExport struct {
|
||||
FilePath string `xml:"FileName"` // the path to the file, as stored on the server
|
||||
}
|
||||
|
||||
func (f FileUploadStartExport) SortVal() (int64, string) {
|
||||
// file messages should come after the message they were with
|
||||
return f.UploadStartTime, EnsureLastMessageInASort
|
||||
}
|
||||
|
||||
// The FileTransferEnded element indicates the end of a file transfer in a conversation
|
||||
type FileUploadStopExport struct {
|
||||
XMLName xml.Name `xml:"FileTransferEnded"`
|
||||
@@ -101,73 +146,105 @@ type FileUploadStopExport struct {
|
||||
Status string `xml:"Status"` // set to either "Completed" or "Failed" depending on the outcome of the upload operation
|
||||
}
|
||||
|
||||
func ActianceExport(rctx request.CTX, posts []*model.MessageExport, db store.Store, exportBackend filestore.FileBackend, fileAttachmentBackend filestore.FileBackend, exportDirectory string) (warningCount int64, appErr *model.AppError) {
|
||||
// sort the posts into buckets based on the channel in which they appeared
|
||||
membersByChannel := common_export.MembersByChannel{}
|
||||
metadata := common_export.Metadata{
|
||||
Channels: map[string]common_export.MetadataChannel{},
|
||||
MessagesCount: 0,
|
||||
AttachmentsCount: 0,
|
||||
StartTime: 0,
|
||||
EndTime: 0,
|
||||
func (f FileUploadStopExport) SortVal() (int64, string) {
|
||||
// file messages should come after the message they were with
|
||||
return f.UploadStopTime, EnsureLastMessageInASort
|
||||
}
|
||||
|
||||
func ActianceExport(rctx request.CTX, p shared.ExportParams) (shared.RunExportResults, error) {
|
||||
start := time.Now()
|
||||
|
||||
// Build the channel exports for the channels that had post or user join/leave activity this batch.
|
||||
exportData, err := shared.GetGenericExportData(p)
|
||||
if err != nil {
|
||||
return exportData.Results, err
|
||||
}
|
||||
elementsByChannel := map[string][]any{}
|
||||
allUploadedFiles := []*model.FileInfo{}
|
||||
var allUploadedFiles []*model.FileInfo
|
||||
|
||||
for _, post := range posts {
|
||||
if post == nil {
|
||||
rctx.Logger().Warn("ignored a nil post reference in the list")
|
||||
continue
|
||||
// Convert the generic shared.ChannelExports to the Actiance-specific ChannelExports data.
|
||||
channelExports := make([]ChannelExport, 0, len(exportData.Exports))
|
||||
for _, channel := range exportData.Exports {
|
||||
joinEvents := make([]JoinExport, 0, len(channel.JoinEvents))
|
||||
leaveEvents := make([]LeaveExport, 0, len(channel.LeaveEvents))
|
||||
|
||||
for _, j := range channel.JoinEvents {
|
||||
joinEvents = append(joinEvents, JoinExport{
|
||||
UserEmail: j.UserEmail,
|
||||
UserType: string(j.UserType),
|
||||
JoinTime: j.JoinTime,
|
||||
CorporateEmailID: j.UserEmail,
|
||||
})
|
||||
}
|
||||
elementsByChannel[*post.ChannelId] = append(elementsByChannel[*post.ChannelId], postToExportEntry(post, post.PostCreateAt, *post.PostMessage))
|
||||
|
||||
if post.PostDeleteAt != nil && *post.PostDeleteAt > 0 && post.PostProps != nil {
|
||||
props := map[string]any{}
|
||||
if json.Unmarshal([]byte(*post.PostProps), &props) == nil {
|
||||
if _, ok := props[model.PostPropsDeleteBy]; ok {
|
||||
elementsByChannel[*post.ChannelId] = append(elementsByChannel[*post.ChannelId], postToExportEntry(post,
|
||||
post.PostDeleteAt, "delete "+*post.PostMessage))
|
||||
}
|
||||
}
|
||||
for _, l := range channel.LeaveEvents {
|
||||
leaveEvents = append(leaveEvents, LeaveExport{
|
||||
UserEmail: l.UserEmail,
|
||||
UserType: string(l.UserType),
|
||||
LeaveTime: l.LeaveTime,
|
||||
CorporateEmailID: l.UserEmail,
|
||||
})
|
||||
}
|
||||
|
||||
startUploads, stopUploads, uploadedFiles, deleteFileMessages, err := postToAttachmentsEntries(post, db)
|
||||
if err != nil {
|
||||
return warningCount, err
|
||||
elements := make([]Sortable, 0, len(channel.Posts)+len(channel.DeletedFiles)+len(channel.UploadStarts)+len(channel.UploadStops))
|
||||
for _, p := range channel.Posts {
|
||||
elements = append(elements, PostExport{
|
||||
MessageId: *p.PostId,
|
||||
UserEmail: *p.UserEmail,
|
||||
UserType: p.UserType,
|
||||
CreateAt: *p.PostCreateAt,
|
||||
UpdatedType: p.UpdatedType,
|
||||
UpdateAt: p.UpdateAt,
|
||||
EditedNewMsgId: p.EditedNewMsgId,
|
||||
Message: p.Message,
|
||||
PreviewsPost: p.PreviewsPost,
|
||||
})
|
||||
}
|
||||
elementsByChannel[*post.ChannelId] = append(elementsByChannel[*post.ChannelId], startUploads...)
|
||||
elementsByChannel[*post.ChannelId] = append(elementsByChannel[*post.ChannelId], stopUploads...)
|
||||
elementsByChannel[*post.ChannelId] = append(elementsByChannel[*post.ChannelId], deleteFileMessages...)
|
||||
|
||||
allUploadedFiles = append(allUploadedFiles, uploadedFiles...)
|
||||
|
||||
metadata.Update(post, len(uploadedFiles))
|
||||
|
||||
if _, ok := membersByChannel[*post.ChannelId]; !ok {
|
||||
membersByChannel[*post.ChannelId] = common_export.ChannelMembers{}
|
||||
for _, p := range channel.DeletedFiles {
|
||||
elements = append(elements, PostExport{
|
||||
MessageId: *p.PostId,
|
||||
UserEmail: *p.UserEmail,
|
||||
UserType: p.UserType,
|
||||
CreateAt: *p.PostCreateAt,
|
||||
UpdatedType: p.UpdatedType,
|
||||
UpdateAt: p.UpdateAt,
|
||||
EditedNewMsgId: p.EditedNewMsgId,
|
||||
Message: p.Message,
|
||||
PreviewsPost: p.PreviewsPost,
|
||||
})
|
||||
}
|
||||
membersByChannel[*post.ChannelId][*post.UserId] = common_export.ChannelMember{
|
||||
Email: *post.UserEmail,
|
||||
UserId: *post.UserId,
|
||||
IsBot: post.IsBot,
|
||||
Username: *post.Username,
|
||||
for _, u := range channel.UploadStarts {
|
||||
elements = append(elements, FileUploadStartExport{
|
||||
UserEmail: u.UserEmail,
|
||||
UploadStartTime: u.UploadStartTime,
|
||||
Filename: u.FileInfo.Name,
|
||||
FilePath: u.FileInfo.Path,
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
rctx.Logger().Info("Exported data for channels", mlog.Int("number_of_channels", len(metadata.Channels)))
|
||||
|
||||
channelExports := []ChannelExport{}
|
||||
for _, channel := range metadata.Channels {
|
||||
channelExport, err := buildChannelExport(
|
||||
channel,
|
||||
membersByChannel[channel.ChannelId],
|
||||
elementsByChannel[channel.ChannelId],
|
||||
db,
|
||||
)
|
||||
if err != nil {
|
||||
return warningCount, err
|
||||
for _, u := range channel.UploadStops {
|
||||
elements = append(elements, FileUploadStopExport{
|
||||
UserEmail: u.UserEmail,
|
||||
UploadStopTime: u.UploadStopTime,
|
||||
Filename: u.FileInfo.Name,
|
||||
FilePath: u.FileInfo.Path,
|
||||
Status: u.Status,
|
||||
})
|
||||
}
|
||||
channelExports = append(channelExports, *channelExport)
|
||||
|
||||
// We need to sort all the elements by (updateAt, messageId) because they were added by type above.
|
||||
slices.SortStableFunc(elements, SortableSort)
|
||||
|
||||
channelExports = append(channelExports, ChannelExport{
|
||||
Perspective: channel.DisplayName,
|
||||
ChannelId: channel.ChannelId,
|
||||
RoomId: fmt.Sprintf("%v - %v - %v", shared.ChannelTypeDisplayName(channel.ChannelType),
|
||||
channel.ChannelName, channel.ChannelId),
|
||||
StartTime: channel.StartTime,
|
||||
JoinEvents: joinEvents,
|
||||
Elements: elements,
|
||||
LeaveEvents: leaveEvents,
|
||||
EndTime: channel.EndTime,
|
||||
})
|
||||
|
||||
allUploadedFiles = append(allUploadedFiles, channel.Files...)
|
||||
}
|
||||
|
||||
export := &RootNode{
|
||||
@@ -175,182 +252,113 @@ func ActianceExport(rctx request.CTX, posts []*model.MessageExport, db store.Sto
|
||||
Channels: channelExports,
|
||||
}
|
||||
|
||||
return writeExport(rctx, export, allUploadedFiles, exportDirectory, exportBackend, fileAttachmentBackend)
|
||||
results := exportData.Results
|
||||
results.ProcessingPostsMs = time.Since(start).Milliseconds()
|
||||
|
||||
results.WriteExportResult, err = writeExport(rctx, export, allUploadedFiles, p.ExportBackend, p.FileAttachmentBackend, p.BatchPath)
|
||||
results.NumChannels = len(channelExports)
|
||||
return results, err
|
||||
}
|
||||
|
||||
func postToExportEntry(post *model.MessageExport, createTime *int64, message string) *PostExport {
|
||||
userType := "user"
|
||||
if post.IsBot {
|
||||
userType = "bot"
|
||||
}
|
||||
return &PostExport{
|
||||
PostTime: *createTime,
|
||||
Message: message,
|
||||
UserType: userType,
|
||||
UserEmail: *post.UserEmail,
|
||||
PreviewsPost: post.PreviewID(),
|
||||
}
|
||||
}
|
||||
|
||||
func postToAttachmentsEntries(post *model.MessageExport, db store.Store) ([]any, []any, []*model.FileInfo, []any, *model.AppError) {
|
||||
// if the post included any files, we need to add special elements to the export.
|
||||
if len(post.PostFileIds) == 0 {
|
||||
return nil, nil, nil, nil, nil
|
||||
}
|
||||
|
||||
fileInfos, err := db.FileInfo().GetForPost(*post.PostId, true, true, false)
|
||||
if err != nil {
|
||||
return nil, nil, nil, nil, model.NewAppError("postToAttachmentsEntries", "ent.message_export.actiance_export.get_attachment_error", nil, "", http.StatusInternalServerError).Wrap(err)
|
||||
}
|
||||
|
||||
startUploads := []any{}
|
||||
stopUploads := []any{}
|
||||
deleteFileMessages := []any{}
|
||||
|
||||
uploadedFiles := []*model.FileInfo{}
|
||||
for _, fileInfo := range fileInfos {
|
||||
// insert a record of the file upload into the export file
|
||||
// path to exported file is relative to the xml file, so it's just the name of the exported file
|
||||
startUploads = append(startUploads, &FileUploadStartExport{
|
||||
UserEmail: *post.UserEmail,
|
||||
Filename: fileInfo.Name,
|
||||
FilePath: fileInfo.Path,
|
||||
UploadStartTime: *post.PostCreateAt,
|
||||
})
|
||||
|
||||
stopUploads = append(stopUploads, &FileUploadStopExport{
|
||||
UserEmail: *post.UserEmail,
|
||||
Filename: fileInfo.Name,
|
||||
FilePath: fileInfo.Path,
|
||||
UploadStopTime: *post.PostCreateAt,
|
||||
Status: "Completed",
|
||||
})
|
||||
|
||||
if fileInfo.DeleteAt > 0 && post.PostDeleteAt != nil {
|
||||
deleteFileMessages = append(deleteFileMessages, postToExportEntry(post, post.PostDeleteAt, "delete "+fileInfo.Path))
|
||||
}
|
||||
|
||||
uploadedFiles = append(uploadedFiles, fileInfo)
|
||||
}
|
||||
return startUploads, stopUploads, uploadedFiles, deleteFileMessages, nil
|
||||
}
|
||||
|
||||
func buildChannelExport(channel common_export.MetadataChannel, members common_export.ChannelMembers, elements []any, db store.Store) (*ChannelExport, *model.AppError) {
|
||||
channelExport := ChannelExport{
|
||||
ChannelId: channel.ChannelId,
|
||||
RoomId: fmt.Sprintf("%v - %v - %v", common_export.ChannelTypeDisplayName(channel.ChannelType), channel.ChannelName, channel.ChannelId),
|
||||
StartTime: channel.StartTime,
|
||||
EndTime: channel.EndTime,
|
||||
Perspective: channel.ChannelDisplayName,
|
||||
}
|
||||
|
||||
channelMembersHistory, err := db.ChannelMemberHistory().GetUsersInChannelDuring(channel.StartTime, channel.EndTime, channel.ChannelId)
|
||||
if err != nil {
|
||||
return nil, model.NewAppError("buildChannelExport", "ent.get_users_in_channel_during", nil, "", http.StatusInternalServerError).Wrap(err)
|
||||
}
|
||||
|
||||
joins, leaves := common_export.GetJoinsAndLeavesForChannel(channel.StartTime, channel.EndTime, channelMembersHistory, members)
|
||||
type StillJoinedInfo struct {
|
||||
Time int64
|
||||
Type string
|
||||
}
|
||||
stillJoined := map[string]StillJoinedInfo{}
|
||||
for _, join := range joins {
|
||||
userType := "user"
|
||||
if join.IsBot {
|
||||
userType = "bot"
|
||||
}
|
||||
channelExport.JoinEvents = append(channelExport.JoinEvents, JoinExport{
|
||||
JoinTime: join.Datetime,
|
||||
UserEmail: join.Email,
|
||||
UserType: userType,
|
||||
CorporateEmailID: join.Email,
|
||||
})
|
||||
if value, ok := stillJoined[join.Email]; !ok {
|
||||
stillJoined[join.Email] = StillJoinedInfo{Time: join.Datetime, Type: userType}
|
||||
} else {
|
||||
if join.Datetime > value.Time {
|
||||
stillJoined[join.Email] = StillJoinedInfo{Time: join.Datetime, Type: userType}
|
||||
}
|
||||
}
|
||||
}
|
||||
for _, leave := range leaves {
|
||||
userType := "user"
|
||||
if leave.IsBot {
|
||||
userType = "bot"
|
||||
}
|
||||
channelExport.LeaveEvents = append(channelExport.LeaveEvents, LeaveExport{
|
||||
LeaveTime: leave.Datetime,
|
||||
UserEmail: leave.Email,
|
||||
UserType: userType,
|
||||
CorporateEmailID: leave.Email,
|
||||
})
|
||||
if leave.Datetime > stillJoined[leave.Email].Time {
|
||||
delete(stillJoined, leave.Email)
|
||||
}
|
||||
}
|
||||
|
||||
for email := range stillJoined {
|
||||
channelExport.LeaveEvents = append(channelExport.LeaveEvents, LeaveExport{
|
||||
LeaveTime: channel.EndTime,
|
||||
UserEmail: email,
|
||||
UserType: stillJoined[email].Type,
|
||||
CorporateEmailID: email,
|
||||
})
|
||||
}
|
||||
|
||||
sort.Slice(channelExport.LeaveEvents, func(i, j int) bool {
|
||||
if channelExport.LeaveEvents[i].LeaveTime == channelExport.LeaveEvents[j].LeaveTime {
|
||||
return channelExport.LeaveEvents[i].UserEmail < channelExport.LeaveEvents[j].UserEmail
|
||||
}
|
||||
return channelExport.LeaveEvents[i].LeaveTime < channelExport.LeaveEvents[j].LeaveTime
|
||||
})
|
||||
|
||||
channelExport.Elements = elements
|
||||
return &channelExport, nil
|
||||
}
|
||||
|
||||
func writeExport(rctx request.CTX, export *RootNode, uploadedFiles []*model.FileInfo, exportDirectory string, exportBackend filestore.FileBackend, fileAttachmentBackend filestore.FileBackend) (warningCount int64, appErr *model.AppError) {
|
||||
func writeExport(rctx request.CTX, export *RootNode, uploadedFiles []*model.FileInfo, exportBackend filestore.FileBackend, fileAttachmentBackend filestore.FileBackend, batchPath string) (res shared.WriteExportResult, err error) {
|
||||
start := time.Now()
|
||||
// marshal the export object to xml
|
||||
xmlData := &bytes.Buffer{}
|
||||
xmlData.WriteString(xml.Header)
|
||||
|
||||
enc := xml.NewEncoder(xmlData)
|
||||
enc.Indent("", " ")
|
||||
if err := enc.Encode(export); err != nil {
|
||||
return warningCount, model.NewAppError("ActianceExport.AtianceExport", "ent.actiance.export.marshalToXml.appError", nil, "", 0).Wrap(err)
|
||||
if err = enc.Encode(export); err != nil {
|
||||
return res, fmt.Errorf("unable to convert export to XML: %w", err)
|
||||
}
|
||||
if err = enc.Flush(); err != nil {
|
||||
return res, fmt.Errorf("unable to flush the XML encoder: %w", err)
|
||||
}
|
||||
enc.Flush()
|
||||
|
||||
// Try to disable the write timeout if the backend supports it
|
||||
if _, err := filestore.TryWriteFileContext(rctx.Context(), exportBackend, xmlData, path.Join(exportDirectory, ActianceExportFilename)); err != nil {
|
||||
return warningCount, model.NewAppError("ActianceExport.AtianceExport", "ent.actiance.export.write_file.appError", nil, "", 0).Wrap(err)
|
||||
// Write this batch to a tmp zip, then copy the zip to the export directory.
|
||||
// Using a 2M buffer because the file backend may be s3 and this optimizes speed and
|
||||
// memory usage, see: https://github.com/mattermost/mattermost/pull/26629
|
||||
buf := make([]byte, 1024*1024*2)
|
||||
temp, err := os.CreateTemp("", "compliance-export-batch-*.zip")
|
||||
if err != nil {
|
||||
return res, fmt.Errorf("unable to create the batch temporary file: %w", err)
|
||||
}
|
||||
defer file.DeleteTemp(rctx.Logger(), temp)
|
||||
|
||||
zipFile := zip.NewWriter(temp)
|
||||
w, err := zipFile.Create(ActianceExportFilename)
|
||||
if err != nil {
|
||||
return res, fmt.Errorf("unable to create the xml file in the zipFile created with the batch temporary file: %w", err)
|
||||
}
|
||||
if _, err = io.CopyBuffer(w, xmlData, buf); err != nil {
|
||||
return res, fmt.Errorf("unable to write into the zipFile created with the batch temporary file: %w", err)
|
||||
}
|
||||
res.ProcessingXmlMs = time.Since(start).Milliseconds()
|
||||
|
||||
start = time.Now()
|
||||
|
||||
var missingFiles []string
|
||||
for _, fileInfo := range uploadedFiles {
|
||||
var attachmentSrc io.ReadCloser
|
||||
attachmentSrc, nErr := fileAttachmentBackend.Reader(fileInfo.Path)
|
||||
if nErr != nil {
|
||||
missingFiles = append(missingFiles, "Warning:"+common_export.MissingFileMessage+" - "+fileInfo.Path)
|
||||
rctx.Logger().Warn(common_export.MissingFileMessage, mlog.String("FileName", fileInfo.Path))
|
||||
var attachmentReader io.ReadCloser
|
||||
attachmentReader, err = fileAttachmentBackend.Reader(fileInfo.Path)
|
||||
if err != nil {
|
||||
missingFiles = append(missingFiles, "Warning:"+shared.MissingFileMessage+" - "+fileInfo.Path)
|
||||
rctx.Logger().Warn(shared.MissingFileMessage, mlog.String("filename", fileInfo.Path))
|
||||
continue
|
||||
}
|
||||
defer attachmentSrc.Close()
|
||||
|
||||
destPath := path.Join(exportDirectory, fileInfo.Path)
|
||||
// There could be many uploadedFiles, so be careful about closing readers.
|
||||
if err = func() error {
|
||||
defer attachmentReader.Close()
|
||||
var zipWriter io.Writer
|
||||
zipWriter, err = zipFile.Create(fileInfo.Path)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
_, nErr = exportBackend.WriteFile(attachmentSrc, destPath)
|
||||
if nErr != nil {
|
||||
return warningCount, model.NewAppError("ActianceExport.AtianceExport", "ent.actiance.export.write_file.appError", nil, "", 0).Wrap(nErr)
|
||||
// CopyBuffer works with dirty buffers, no need to clear it.
|
||||
if _, err = io.CopyBuffer(zipWriter, attachmentReader, buf); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
return nil
|
||||
}(); err != nil {
|
||||
return res, fmt.Errorf("unable to write into the zipFile created with the batch temporary file: %w", err)
|
||||
}
|
||||
}
|
||||
warningCount = int64(len(missingFiles))
|
||||
if warningCount > 0 {
|
||||
_, err := filestore.TryWriteFileContext(rctx.Context(), exportBackend, strings.NewReader(strings.Join(missingFiles, "\n")), path.Join(exportDirectory, ActianceWarningFilename))
|
||||
|
||||
res.TransferringFilesMs = time.Since(start).Milliseconds()
|
||||
res.NumWarnings = len(missingFiles)
|
||||
if res.NumWarnings > 0 {
|
||||
var w io.Writer
|
||||
w, err = zipFile.Create(ActianceWarningFilename)
|
||||
if err != nil {
|
||||
appErr = model.NewAppError("ActianceExport.AtianceExport", "ent.actiance.export.write_file.appError", nil, "", 0).Wrap(err)
|
||||
return res, fmt.Errorf("unable to create the warning file in the zipFile created with the batch temporary file: %w", err)
|
||||
}
|
||||
r := strings.NewReader(strings.Join(missingFiles, "\n"))
|
||||
if _, err = io.CopyBuffer(w, r, buf); err != nil {
|
||||
return res, fmt.Errorf("unable to write into the zipFile created with the batch temporary file: %w", err)
|
||||
}
|
||||
}
|
||||
return warningCount, appErr
|
||||
|
||||
if err = zipFile.Close(); err != nil {
|
||||
return res, fmt.Errorf("unable to close the zipFile created with the batch temporary file: %w", err)
|
||||
}
|
||||
|
||||
_, err = temp.Seek(0, io.SeekStart)
|
||||
if err != nil {
|
||||
return res, fmt.Errorf("unable to seek to the beginning of the the batch temporary file: %w", err)
|
||||
}
|
||||
|
||||
start = time.Now()
|
||||
|
||||
// Try to write the file without a timeout due to the potential size of the file.
|
||||
_, err = filestore.TryWriteFileContext(rctx.Context(), exportBackend, temp, batchPath)
|
||||
if err != nil {
|
||||
return res, fmt.Errorf("unable to transfer the batch zip to the file backend: %w", err)
|
||||
}
|
||||
|
||||
res.TransferringZipMs = time.Since(start).Milliseconds()
|
||||
|
||||
return res, nil
|
||||
}
|
||||
|
||||
Разница между файлами не показана из-за своего большого размера
Загрузить разницу
21
server/enterprise/message_export/actiance_export/main_test.go
Обычный файл
21
server/enterprise/message_export/actiance_export/main_test.go
Обычный файл
@@ -0,0 +1,21 @@
|
||||
// Copyright (c) 2015-present Mattermost, Inc. All Rights Reserved.
|
||||
// See LICENSE.enterprise for license information.
|
||||
|
||||
package actiance_export
|
||||
|
||||
import (
|
||||
"testing"
|
||||
|
||||
"github.com/mattermost/mattermost/server/v8/channels/api4"
|
||||
"github.com/mattermost/mattermost/server/v8/channels/testlib"
|
||||
)
|
||||
|
||||
var mainHelper *testlib.MainHelper
|
||||
|
||||
func TestMain(m *testing.M) {
|
||||
mainHelper = testlib.NewMainHelper()
|
||||
defer mainHelper.Close()
|
||||
api4.SetMainHelper(mainHelper)
|
||||
|
||||
mainHelper.Main(m)
|
||||
}
|
||||
49
server/enterprise/message_export/actiance_export/test_helpers.go
Обычный файл
49
server/enterprise/message_export/actiance_export/test_helpers.go
Обычный файл
@@ -0,0 +1,49 @@
|
||||
// Copyright (c) 2015-present Mattermost, Inc. All Rights Reserved.
|
||||
// See LICENSE.enterprise for license information.
|
||||
|
||||
package actiance_export
|
||||
|
||||
import (
|
||||
"encoding/xml"
|
||||
"io"
|
||||
"testing"
|
||||
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
type ChannelExportWithSpecifics struct {
|
||||
XMLName xml.Name `xml:"Conversation"`
|
||||
Perspective string `xml:"Perspective,attr"`
|
||||
ChannelId string `xml:"-"`
|
||||
RoomId string `xml:"RoomID"`
|
||||
StartTime int64 `xml:"StartTimeUTC"`
|
||||
JoinEvents []*JoinExport `xml:"ParticipantEntered"`
|
||||
Messages []*PostExport `xml:"Message"`
|
||||
FileStarts []*FileUploadStartExport `xml:"FileTransferStarted"`
|
||||
FileStops []*FileUploadStopExport `xml:"FileTransferEnded"`
|
||||
LeaveEvents []*LeaveExport `xml:"ParticipantLeft"`
|
||||
EndTime int64 `xml:"EndTimeUTC"`
|
||||
}
|
||||
|
||||
func GetChannelExports(t *testing.T, r io.Reader) []*ChannelExportWithSpecifics {
|
||||
decoder := xml.NewDecoder(r)
|
||||
var exportedChannels []*ChannelExportWithSpecifics
|
||||
for {
|
||||
token, err := decoder.Token()
|
||||
if token == nil || err != nil {
|
||||
break
|
||||
}
|
||||
switch se := token.(type) {
|
||||
case xml.StartElement:
|
||||
if se.Name.Local == "Conversation" {
|
||||
var a *ChannelExportWithSpecifics
|
||||
err = decoder.DecodeElement(&a, &se)
|
||||
require.NoError(t, err)
|
||||
exportedChannels = append(exportedChannels, a)
|
||||
}
|
||||
default:
|
||||
}
|
||||
}
|
||||
|
||||
return exportedChannels
|
||||
}
|
||||
Ссылка в новой задаче
Block a user