MM-59966 - Compliance Export overhaul - feature branch (#29789)
* [MM-59089] Add a compliance export constant (#27919) * add a useful constant * i18n * another constant * another i18n * [MM-60422] Add GetChannelsWithActivityDuring (#28301) * modify GetUsersInChannelDuring to accept a slice of channelIds * add GetChannelsWithActivityDuring * add compliance export progress message; remove unused custom status * linting * tests running too fast * add batch size config settings * add store tests * linting * empty commit * i18n changes * fix i18n ordering * MM-60570 - Server-side changes consolidating the export CLI with server/ent code (#28640) * add an i18n field; add the CLI's export directory * int64 -> int * Add UntilUpdateAt for MessageExport and AnalyticsPostCount to merge * remove now-unused i18n strings * add TranslationsPreInitFromBuffer to allow CLI to use i18n * use GetBuilder to simplify; rename TranslationsPreInitFromFileBytes * [MM-59089] Improve compliance export timings (#1733 - Enterprise repo) * MM-60422 - Performance and logic fixes for Compliance Exports (#1757 - Enterprise repo) * MM-60570 - Enterprise-side changes consolidating the export CLI with server/ent code (#1769 - Enterprise repo) * merge conflicts; missed file from ent branch * MM-61038 - Add an option to sqlstore.New (#28702) remove useless comment add test add an option to sqlstore.New * MM-60976: Remove RunExport command from Mattermost binary (#28805) * remove RunExport command from mattermost binary * remove the code it was calling * fix i18n * remove test (was only testing license, not functionality) * empty commit * fix flaky GetChannelsWithActivityDuring test * MM-60063: Dedicated Export Filestore fix, redo of #1772 (enterprise) (#28803) * redo filestore fix #1772 (enterprise repo) on top of MM-59966 feature * add new e2e tests for export filestore * golint * ok, note to self: shadowing bad, actually (when there's a defer) * empty commit * MM-61137 - Message export: Support 7.8.11 era dbs (#28824) * support 7.8.11 era dbs by wrapping the store using only what we need * fix flaky GetChannelsWithActivityDuring test * add a comment * only need to define the MEFileInfoStore (the one that'll be overridden) * blank commit * MM-60974 - Message Export: Add performance metrics (#28836) * support 7.8.11 era dbs by wrapping the store using only what we need * fix flaky GetChannelsWithActivityDuring test * add a comment * only need to define the MEFileInfoStore (the one that'll be overridden) * performance metrics * cleanup unneeded named returns * blank commit * MM-60975 - Message export: Add startTime and endTime to export folder name (#28840) * support 7.8.11 era dbs by wrapping the store using only what we need * fix flaky GetChannelsWithActivityDuring test * add a comment * only need to define the MEFileInfoStore (the one that'll be overridden) * performance metrics * output startTime and endTime in export folder * empty commit * merge conflict * MM-60978 - Message export: Improve xml fields; fix delete semantics (#28873) * support 7.8.11 era dbs by wrapping the store using only what we need * fix flaky GetChannelsWithActivityDuring test * add a comment * only need to define the MEFileInfoStore (the one that'll be overridden) * performance metrics * output startTime and endTime in export folder * empty commit * add xml fields, omit when empty, tests * fix delete semantics; test (and test for update semantics) * clarify comments * simplify edited post detection, now there's no edge case. * add some spacing to help fast running tests * merge conflicts/updates needed for new deleted post semantics * linting; fixing tests from upstream merge * use SafeDereference * linting * stronger typing; better wrapped errors; better formatting * blank commit * goimports formatting * fix merge mistake * minor fixes due to changes in master * MM-61755 - Simplifying and Support reporting to the db from the CLI (#29281) * finally clean up JobData struct and stringMap; prep for CLI using db * and now simplify using StringMapToJobDataWithZeroValues * remove unused fn * create JobDataExported; clean up errors * MM-60176 - Message Export: Global relay cleanup (#29168) * move global relay logic into global_relay_export * blank commit * blank commit * improve errors * MM-60693 - Refactor CSV to use same codepath as Actiance (#29191) * move global relay logic into global_relay_export * blank commit * refactor (and simplify) ExportParams into shared * blank commit * remove unused fn * csv now uses pre-calculated joins/leaves like actiance * improve errors * remove nil post check; remove ignoredPosts metric * remove unneeded copy * MM-61696 - Refactor GlobalRelay to use same codepath as Actiance (#29225) * move global relay logic into global_relay_export * blank commit * refactor (and simplify) ExportParams into shared * blank commit * remove unused fn * csv now uses pre-calculated joins/leaves like actiance * remove newly unneeded function and its test. goodbye. * refactor GetPostAttachments for csv + global relay to share * refactor global_relay_export and fix tests (no changes to output) * improve errors * remove nil post check; remove ignoredPosts metric * remove unneeded copy * remove unneeded nil check * PR comments * MM-61715 - Generalize e2e to all export types 🤖 (#29369) * move global relay logic into global_relay_export * blank commit * refactor (and simplify) ExportParams into shared * blank commit * remove unused fn * csv now uses pre-calculated joins/leaves like actiance * remove newly unneeded function and its test. goodbye. * refactor GetPostAttachments for csv + global relay to share * refactor global_relay_export and fix tests (no changes to output) * improve errors * remove nil post check; remove ignoredPosts metric * remove unneeded copy * remove unneeded nil check * PR comments * refactor isDeletedMsg for all export types * fix start and endtime, nasty csv createAt bug; bring closer to Actiance * align unit tests with new logic (e.g. starttime / endtime) * refactor a TimestampConvert fn for code + tests * bug: pass templates to global relay (hurray for e2e tests, otherwise...) * add global relay zip to allowed list (only for tests) * test helpers * new templates for e2e tests * e2e tests... phew. * linting * merge conflicts * unexport PostToRow; add test helper marker * cleanup, shortening, thanks to PR comments * MM-61972 - Generalize export data path - Actiance (#29399) * extract and generalize the export data generation functions * finish moving test (bc of previous extraction) * lift a function from common -> shared (to break an import cycle) * actiance now takes general export data, processes it into actiance data * bring tests in line with correct sorting rules (upadateAt, messageId) * fixups, PR comments * turn strings.Repeat into a more descriptive const amended: one letter fix; bad rebase * MM-62009 - e2e clock heisenbug (#29434) * consolidate assertions; output debuggable diffs (keeping for future) * refactor test output generator to generators file * waitUntilZeroPosts + pass through until to job = fix all clock issues * simplify messages to model.NewId(); remove unneeded waitUntilZeroPosts * model.NewId() -> storetest.NewTestID() * MM-61980 - Generalize export data path - CSV (#29482) * simple refactoring * increase sleep times for (very) rare test failures * add extra information to the generic export for CSV * adj Actiance to handle new generic export (no difference in its output) * no longer need mergePosts (yay), move getJoinLeavePosts for everyone * adjust tests for new csv semantics (detailed in summary) * and need to add the new exported data to the export_data_tests * rearrange csv writing to happen after data export (more logical) * linting * remove debug statements * figured out what was wrong with global relay e2e test 3; solid now * PR comments * MM-61718 - Generalize export data path - Global Relay (#29508) * move global relay over to using the generalized export data * performance pass -- not much can be done * Update server/enterprise/message_export/global_relay_export/global_relay_export.go Co-authored-by: Claudio Costa <cstcld91@gmail.com> --------- Co-authored-by: Claudio Costa <cstcld91@gmail.com> * MM-62058 - Align CSV with Actiance (#29551) * refactoring actiance files and var names for clarity * bug found in exported attachments (we used to miss some start/ends) * changes needed for actiance due to new generic exports * bringing CSV up to actiance standards * fixing global relay b/c of new semantics (adding a note on an edge case) * aligning e2e tests, adding comments to clarify what is expected/tested * necessary changes; 1 more test for added functionality (ignoreDeleted) * comment style * MM-62059 - Align Global Relay with Actiance/CSV; many fixes (#29665) * core logic changes to general export_data and the specific export paths * unit tests and e2e tests, covering all new edge cases and all logic * linting * better var naming, const value, and cleaning up functions calls * MM-62436 - Temporarily skip cypress tests that require download link (#29772) --------- Co-authored-by: Claudio Costa <cstcld91@gmail.com>
Этот коммит содержится в:
Christopher Poile
коммит произвёл
GitHub
GitHub
родитель
96195f1bd7
Коммит
aba4434dab
@@ -9,18 +9,18 @@ import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"os"
|
||||
"path"
|
||||
"sort"
|
||||
"slices"
|
||||
"strconv"
|
||||
"strings"
|
||||
|
||||
"github.com/mattermost/mattermost/server/v8/enterprise/message_export/common_export"
|
||||
"github.com/mattermost/mattermost/server/v8/enterprise/internal/file"
|
||||
"github.com/mattermost/mattermost/server/v8/enterprise/message_export/shared"
|
||||
|
||||
"github.com/mattermost/mattermost/server/public/model"
|
||||
"github.com/mattermost/mattermost/server/public/shared/mlog"
|
||||
"github.com/mattermost/mattermost/server/public/shared/request"
|
||||
"github.com/mattermost/mattermost/server/v8/channels/store"
|
||||
"github.com/mattermost/mattermost/server/v8/platform/shared/filestore"
|
||||
)
|
||||
|
||||
@@ -28,26 +28,87 @@ const (
|
||||
EnterPostType = "enter"
|
||||
LeavePostType = "leave"
|
||||
PreviouslyJoinedPostType = "previously-joined"
|
||||
CSVExportFilename = "csv_export.zip"
|
||||
CSVWarningFilename = "warning.txt"
|
||||
)
|
||||
|
||||
func CsvExport(rctx request.CTX, posts []*model.MessageExport, db store.Store, exportBackend filestore.FileBackend, fileAttachmentBackend filestore.FileBackend, exportDirectory string) (warningCount int64, appErr *model.AppError) {
|
||||
dest, err := os.CreateTemp("", CSVExportFilename)
|
||||
type Row struct {
|
||||
CreateAt int64
|
||||
UpdateAt int64
|
||||
UpdateType shared.PostUpdatedType
|
||||
TeamId string
|
||||
TeamName string
|
||||
TeamDisplayName string
|
||||
ChannelId string
|
||||
ChannelName string
|
||||
ChannelDisplayName string
|
||||
ChannelType string
|
||||
UserId string
|
||||
UserEmail string
|
||||
Username string
|
||||
PostId string
|
||||
EditedByPostId string
|
||||
RepliedToPostId string
|
||||
PostMessage string
|
||||
PostType string
|
||||
UserType string
|
||||
PreviewsPostId string
|
||||
}
|
||||
|
||||
func CsvExport(rctx request.CTX, p shared.ExportParams) (shared.RunExportResults, error) {
|
||||
// Build the channel exports for the channels that had post or user join/leave activity this batch.
|
||||
exportData, err := shared.GetGenericExportData(p)
|
||||
results := exportData.Results
|
||||
if err != nil {
|
||||
return warningCount, model.NewAppError("CsvExport", "ent.compliance.csv.file.creation.appError", nil, "", 0).Wrap(err)
|
||||
return results, err
|
||||
}
|
||||
defer os.Remove(dest.Name())
|
||||
|
||||
zipFile := zip.NewWriter(dest)
|
||||
totalRows := results.CreatedPosts + results.EditedOrigMsgPosts + results.DeletedPosts + results.EditedNewMsgPosts +
|
||||
results.UpdatedPosts + results.UploadedFiles + results.DeletedFiles + results.Joins
|
||||
rows := make([]Row, 0, totalRows)
|
||||
for _, channel := range exportData.Exports {
|
||||
rows = append(rows, getJoinLeavePosts(channel)...)
|
||||
for _, p := range channel.Posts {
|
||||
rows = append(rows, postToRow(p, "message", p.PostCreateAt, p.Message))
|
||||
}
|
||||
|
||||
for _, u := range channel.UploadStarts {
|
||||
rows = append(rows, attachmentToRow(shared.UploadStartToExportEntry(u)))
|
||||
}
|
||||
for _, d := range channel.DeletedFiles {
|
||||
rows = append(rows, attachmentToRow(d))
|
||||
}
|
||||
}
|
||||
|
||||
// We need to sort all the elements by (CreateAt, PostId) because they were added by type and by channel above.
|
||||
slices.SortStableFunc(rows, func(a, b Row) int {
|
||||
if a.CreateAt == b.CreateAt {
|
||||
return strings.Compare(a.PostId, b.PostId)
|
||||
}
|
||||
return int(a.CreateAt - b.CreateAt)
|
||||
})
|
||||
|
||||
// We've got the data, now its write time:
|
||||
|
||||
// Write this batch to a tmp zip, then copy the zip to the export directory.
|
||||
// Using a 2M buffer because the file backend may be s3 and this optimizes speed and
|
||||
// memory usage, see: https://github.com/mattermost/mattermost/pull/26629
|
||||
buf := make([]byte, 1024*1024*2)
|
||||
temp, err := os.CreateTemp("", "compliance-export-batch-*.zip")
|
||||
if err != nil {
|
||||
return results, fmt.Errorf("unable to create temporary CSV export file: %w", err)
|
||||
}
|
||||
defer file.DeleteTemp(rctx.Logger(), temp)
|
||||
|
||||
zipFile := zip.NewWriter(temp)
|
||||
csvFile, err := zipFile.Create("posts.csv")
|
||||
if err != nil {
|
||||
return warningCount, model.NewAppError("CsvExport", "ent.compliance.csv.zip.creation.appError", nil, "", 0).Wrap(err)
|
||||
return results, fmt.Errorf("unable to create the zip export file: %w", err)
|
||||
}
|
||||
csvWriter := csv.NewWriter(csvFile)
|
||||
err = csvWriter.Write([]string{
|
||||
"Post Creation Time",
|
||||
"Post Update Time",
|
||||
"Post Update Type",
|
||||
"Team Id",
|
||||
"Team Name",
|
||||
"Team Display Name",
|
||||
@@ -68,343 +129,229 @@ func CsvExport(rctx request.CTX, posts []*model.MessageExport, db store.Store, e
|
||||
})
|
||||
|
||||
if err != nil {
|
||||
return warningCount, model.NewAppError("CsvExportPost", "ent.compliance.csv.header.export.appError", nil, "", 0).Wrap(err)
|
||||
return results, fmt.Errorf("unable to add header to the CSV export: %w", err)
|
||||
}
|
||||
|
||||
metadata := common_export.Metadata{
|
||||
Channels: map[string]common_export.MetadataChannel{},
|
||||
MessagesCount: 0,
|
||||
AttachmentsCount: 0,
|
||||
StartTime: 0,
|
||||
EndTime: 0,
|
||||
}
|
||||
|
||||
membersByChannel := make(common_export.MembersByChannel)
|
||||
|
||||
for _, post := range posts {
|
||||
attachments, err := getPostAttachments(db, post)
|
||||
if err != nil {
|
||||
return warningCount, err
|
||||
}
|
||||
|
||||
if _, ok := membersByChannel[*post.ChannelId]; !ok {
|
||||
membersByChannel[*post.ChannelId] = common_export.ChannelMembers{}
|
||||
}
|
||||
|
||||
membersByChannel[*post.ChannelId][*post.UserId] = common_export.ChannelMember{
|
||||
UserId: *post.UserId,
|
||||
Username: *post.Username,
|
||||
IsBot: post.IsBot,
|
||||
Email: *post.UserEmail,
|
||||
}
|
||||
|
||||
metadata.Update(post, len(attachments))
|
||||
}
|
||||
|
||||
joinLeavePosts, appErr2 := getJoinLeavePosts(metadata.Channels, membersByChannel, db)
|
||||
if appErr2 != nil {
|
||||
return warningCount, appErr2
|
||||
}
|
||||
|
||||
postsGenerator := mergePosts(joinLeavePosts, posts)
|
||||
|
||||
for post := postsGenerator(); post != nil; post = postsGenerator() {
|
||||
if err = csvWriter.Write(postToRow(post, post.PostCreateAt, *post.PostMessage)); err != nil {
|
||||
return warningCount, model.NewAppError("CsvExportPost", "ent.compliance.csv.post.export.appError", nil, "", 0).Wrap(err)
|
||||
}
|
||||
|
||||
if post.PostDeleteAt != nil && *post.PostDeleteAt > 0 && post.PostProps != nil {
|
||||
props := map[string]any{}
|
||||
if json.Unmarshal([]byte(*post.PostProps), &props) == nil {
|
||||
if _, ok := props[model.PostPropsDeleteBy]; ok {
|
||||
if err = csvWriter.Write(postToRow(post, post.PostDeleteAt, "delete "+*post.PostMessage)); err != nil {
|
||||
return warningCount, model.NewAppError("CsvExportPost", "ent.compliance.csv.post.export.appError", nil, "", 0).Wrap(err)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
var attachments []*model.FileInfo
|
||||
attachments, appErr = getPostAttachments(db, post)
|
||||
if appErr != nil {
|
||||
return warningCount, appErr
|
||||
}
|
||||
|
||||
for _, attachment := range attachments {
|
||||
if err = csvWriter.Write(attachmentToRow(post, attachment)); err != nil {
|
||||
return warningCount, model.NewAppError("CsvExport", "ent.compliance.csv.attachment.export.appError", nil, "", 0).Wrap(err)
|
||||
}
|
||||
for _, row := range rows {
|
||||
if err = csvWriter.Write(rowToStringSlice(row)); err != nil {
|
||||
return results, fmt.Errorf("unable to export a row: %w", err)
|
||||
}
|
||||
}
|
||||
|
||||
csvWriter.Flush()
|
||||
|
||||
var missingFiles []string
|
||||
for _, post := range posts {
|
||||
attachments, err := getPostAttachments(db, post)
|
||||
for _, post := range p.Posts {
|
||||
var attachments []*model.FileInfo
|
||||
attachments, err = shared.GetPostAttachments(p.Db, post)
|
||||
if err != nil {
|
||||
return warningCount, err
|
||||
return results, err
|
||||
}
|
||||
|
||||
for _, attachment := range attachments {
|
||||
var attachmentSrc io.ReadCloser
|
||||
attachmentSrc, nErr := fileAttachmentBackend.Reader(attachment.Path)
|
||||
var r io.ReadCloser
|
||||
r, nErr := p.FileAttachmentBackend.Reader(attachment.Path)
|
||||
if nErr != nil {
|
||||
missingFiles = append(missingFiles, "Warning:"+common_export.MissingFileMessage+" - Post: "+*post.PostId+" - "+attachment.Path)
|
||||
rctx.Logger().Warn(common_export.MissingFileMessage, mlog.String("PostId", *post.PostId), mlog.String("FileName", attachment.Path))
|
||||
missingFiles = append(missingFiles, "Warning:"+shared.MissingFileMessage+" - Post: "+*post.PostId+" - "+attachment.Path)
|
||||
rctx.Logger().Warn(shared.MissingFileMessage, mlog.String("post_id", *post.PostId), mlog.String("filename", attachment.Path))
|
||||
continue
|
||||
}
|
||||
defer attachmentSrc.Close()
|
||||
|
||||
attachmentDst, err := zipFile.Create(path.Join("files", *post.PostId, fmt.Sprintf("%s-%s", attachment.Id, path.Base(attachment.Path))))
|
||||
if err != nil {
|
||||
return warningCount, model.NewAppError("CsvExport", "ent.compliance.csv.attachment.copy.appError", nil, "", 0).Wrap(err)
|
||||
}
|
||||
// Probably don't need to be this careful (see actiance_export.go), but may as well be consistent.
|
||||
if err = func() error {
|
||||
defer r.Close()
|
||||
var attachmentDst io.Writer
|
||||
attachmentDst, err = zipFile.Create(path.Join("files", *post.PostId, fmt.Sprintf("%s-%s", attachment.Id, path.Base(attachment.Path))))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
_, err = io.Copy(attachmentDst, attachmentSrc)
|
||||
if err != nil {
|
||||
return warningCount, model.NewAppError("CsvExport", "ent.compliance.csv.attachment.copy.appError", nil, "", 0).Wrap(err)
|
||||
_, err = io.CopyBuffer(attachmentDst, r, buf)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
return nil
|
||||
}(); err != nil {
|
||||
return results, fmt.Errorf("unable to copy the attachment into the zip file: %w", err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
warningCount = int64(len(missingFiles))
|
||||
if warningCount > 0 {
|
||||
results.NumWarnings = len(missingFiles)
|
||||
if results.NumWarnings > 0 {
|
||||
metadataFile, _ := zipFile.Create(CSVWarningFilename)
|
||||
for _, value := range missingFiles {
|
||||
_, err = metadataFile.Write([]byte(value + "\n"))
|
||||
if err != nil {
|
||||
appErr = model.NewAppError("CsvExport", "ent.compliance.csv.warning.appError", nil, "", 0).Wrap(err)
|
||||
return warningCount, appErr
|
||||
return results, fmt.Errorf("unable to create the warning file: %w", err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
metadataFile, err := zipFile.Create("metadata.json")
|
||||
if err != nil {
|
||||
return warningCount, model.NewAppError("CsvExport", "ent.compliance.csv.metadata.json.zipfile.appError", nil, "", 0).Wrap(err)
|
||||
return results, fmt.Errorf("unable to create the zip file: %w", err)
|
||||
}
|
||||
data, err := json.MarshalIndent(metadata, "", " ")
|
||||
data, err := json.MarshalIndent(exportData.Metadata, "", " ")
|
||||
if err != nil {
|
||||
return warningCount, model.NewAppError("CsvExport", "ent.compliance.csv.metadata.json.marshalling.appError", nil, "", 0).Wrap(err)
|
||||
return results, fmt.Errorf("unable to convert metadata to json: %w", err)
|
||||
}
|
||||
_, err = metadataFile.Write(data)
|
||||
if err != nil {
|
||||
return warningCount, model.NewAppError("CsvExport", "ent.compliance.csv.metadata.export.appError", nil, "", 0).Wrap(err)
|
||||
return results, fmt.Errorf("unable to add metadata file to the zip file: %w", err)
|
||||
}
|
||||
err = zipFile.Close()
|
||||
if err != nil {
|
||||
return warningCount, model.NewAppError("CsvExport", "ent.compliance.csv.metadata.close.appError", nil, "", 0).Wrap(err)
|
||||
return results, fmt.Errorf("unable to close the zip file: %w", err)
|
||||
}
|
||||
|
||||
_, err = dest.Seek(0, 0)
|
||||
_, err = temp.Seek(0, 0)
|
||||
if err != nil {
|
||||
return warningCount, model.NewAppError("CsvExport", "ent.compliance.csv.seek.appError", nil, "", 0).Wrap(err)
|
||||
return results, fmt.Errorf("unable to seek to start of export file: %w", err)
|
||||
}
|
||||
|
||||
// Try to write the file without a timeout due to the potential size of the file.
|
||||
_, err = filestore.TryWriteFileContext(rctx.Context(), exportBackend, dest, path.Join(exportDirectory, CSVExportFilename))
|
||||
_, err = filestore.TryWriteFileContext(rctx.Context(), p.ExportBackend, temp, p.BatchPath)
|
||||
if err != nil {
|
||||
return warningCount, model.NewAppError("CsvExport", "ent.compliance.csv.write_file.appError", nil, "", http.StatusInternalServerError).Wrap(err)
|
||||
return results, fmt.Errorf("unable to write the csv file: %w", err)
|
||||
}
|
||||
return warningCount, appErr
|
||||
return results, nil
|
||||
}
|
||||
|
||||
func mergePosts(left []*model.MessageExport, right []*model.MessageExport) func() *model.MessageExport {
|
||||
leftCursor := 0
|
||||
rightCursor := 0
|
||||
return func() *model.MessageExport {
|
||||
if leftCursor >= len(left) && rightCursor >= len(right) {
|
||||
return nil
|
||||
func getJoinLeavePosts(channel shared.ChannelExport) []Row {
|
||||
var joinLeavePosts []Row
|
||||
|
||||
for _, join := range channel.JoinEvents {
|
||||
enterMessage := fmt.Sprintf("User %s (%s) joined the channel", join.Username, join.UserEmail)
|
||||
enterPostType := EnterPostType
|
||||
if join.JoinTime <= channel.StartTime {
|
||||
enterPostType = PreviouslyJoinedPostType
|
||||
enterMessage = fmt.Sprintf("User %s (%s) was already in the channel", join.Username, join.UserEmail)
|
||||
}
|
||||
|
||||
if leftCursor >= len(left) {
|
||||
rightCursor += 1
|
||||
return right[rightCursor-1]
|
||||
}
|
||||
|
||||
if rightCursor >= len(right) {
|
||||
leftCursor += 1
|
||||
return left[leftCursor-1]
|
||||
}
|
||||
|
||||
if *left[leftCursor].PostCreateAt <= *right[rightCursor].PostCreateAt {
|
||||
leftCursor += 1
|
||||
return left[leftCursor-1]
|
||||
}
|
||||
|
||||
rightCursor += 1
|
||||
return right[rightCursor-1]
|
||||
}
|
||||
}
|
||||
|
||||
func getJoinLeavePosts(channels map[string]common_export.MetadataChannel, membersByChannel common_export.MembersByChannel, db store.Store) ([]*model.MessageExport, *model.AppError) {
|
||||
joinLeavePosts := []*model.MessageExport{}
|
||||
for _, channel := range channels {
|
||||
channelMembersHistory, err := db.ChannelMemberHistory().GetUsersInChannelDuring(channel.StartTime, channel.EndTime, channel.ChannelId)
|
||||
if err != nil {
|
||||
return nil, model.NewAppError("getJoinLeavePosts", "ent.get_users_in_channel_during", nil, "", http.StatusInternalServerError).Wrap(err)
|
||||
}
|
||||
|
||||
joins, leaves := common_export.GetJoinsAndLeavesForChannel(channel.StartTime, channel.EndTime, channelMembersHistory, membersByChannel[channel.ChannelId])
|
||||
|
||||
for _, join := range joins {
|
||||
enterMessage := fmt.Sprintf("User %s (%s) joined the channel", join.Username, join.Email)
|
||||
enterPostType := EnterPostType
|
||||
createAt := model.NewPointer(join.Datetime)
|
||||
channelCopy := channel
|
||||
if join.Datetime <= channel.StartTime {
|
||||
enterPostType = PreviouslyJoinedPostType
|
||||
enterMessage = fmt.Sprintf("User %s (%s) was already in the channel", join.Username, join.Email)
|
||||
createAt = model.NewPointer(channel.StartTime)
|
||||
}
|
||||
joinLeavePosts = append(
|
||||
joinLeavePosts,
|
||||
&model.MessageExport{
|
||||
TeamId: channel.TeamId,
|
||||
TeamName: channel.TeamName,
|
||||
TeamDisplayName: channel.TeamDisplayName,
|
||||
|
||||
ChannelId: &channelCopy.ChannelId,
|
||||
ChannelName: &channelCopy.ChannelName,
|
||||
ChannelDisplayName: &channelCopy.ChannelDisplayName,
|
||||
ChannelType: &channelCopy.ChannelType,
|
||||
|
||||
UserId: model.NewPointer(join.UserId),
|
||||
UserEmail: model.NewPointer(join.Email),
|
||||
Username: model.NewPointer(join.Username),
|
||||
IsBot: join.IsBot,
|
||||
|
||||
PostId: model.NewPointer(""),
|
||||
PostCreateAt: createAt,
|
||||
PostMessage: &enterMessage,
|
||||
PostType: &enterPostType,
|
||||
PostOriginalId: model.NewPointer(""),
|
||||
PostFileIds: []string{},
|
||||
joinLeavePosts = append(
|
||||
joinLeavePosts,
|
||||
postToRow(shared.PostExport{
|
||||
MessageExport: model.MessageExport{
|
||||
TeamId: &channel.TeamId,
|
||||
TeamName: &channel.TeamName,
|
||||
TeamDisplayName: &channel.TeamDisplayName,
|
||||
ChannelId: &channel.ChannelId,
|
||||
ChannelName: &channel.ChannelName,
|
||||
ChannelDisplayName: &channel.DisplayName,
|
||||
ChannelType: &channel.ChannelType,
|
||||
UserId: &join.UserId,
|
||||
UserEmail: &join.UserEmail,
|
||||
Username: &join.Username,
|
||||
IsBot: join.UserType == shared.Bot,
|
||||
PostId: model.NewPointer(""),
|
||||
PostCreateAt: &join.JoinTime,
|
||||
PostMessage: &enterMessage,
|
||||
PostType: &enterPostType,
|
||||
PostOriginalId: model.NewPointer(""),
|
||||
PostFileIds: []string{},
|
||||
},
|
||||
)
|
||||
}, enterPostType, &join.JoinTime, enterMessage),
|
||||
)
|
||||
}
|
||||
for _, leave := range channel.LeaveEvents {
|
||||
if leave.ClosedOut {
|
||||
// csv does not record closed-out leaves; see export_data.go for further explanation.
|
||||
continue
|
||||
}
|
||||
for _, leave := range leaves {
|
||||
leaveMessage := fmt.Sprintf("User %s (%s) leaved the channel", leave.Username, leave.Email)
|
||||
leavePostType := LeavePostType
|
||||
channelCopy := channel
|
||||
leaveMessage := fmt.Sprintf("User %s (%s) left the channel", leave.Username, leave.UserEmail)
|
||||
leavePostType := LeavePostType
|
||||
|
||||
joinLeavePosts = append(
|
||||
joinLeavePosts,
|
||||
&model.MessageExport{
|
||||
TeamId: channel.TeamId,
|
||||
TeamName: channel.TeamName,
|
||||
TeamDisplayName: channel.TeamDisplayName,
|
||||
|
||||
ChannelId: &channelCopy.ChannelId,
|
||||
ChannelName: &channelCopy.ChannelName,
|
||||
ChannelDisplayName: &channelCopy.ChannelDisplayName,
|
||||
ChannelType: &channelCopy.ChannelType,
|
||||
|
||||
UserId: model.NewPointer(leave.UserId),
|
||||
UserEmail: model.NewPointer(leave.Email),
|
||||
Username: model.NewPointer(leave.Username),
|
||||
IsBot: leave.IsBot,
|
||||
|
||||
PostId: model.NewPointer(""),
|
||||
PostCreateAt: model.NewPointer(leave.Datetime),
|
||||
PostMessage: &leaveMessage,
|
||||
PostType: &leavePostType,
|
||||
PostOriginalId: model.NewPointer(""),
|
||||
PostFileIds: []string{},
|
||||
joinLeavePosts = append(
|
||||
joinLeavePosts,
|
||||
postToRow(shared.PostExport{
|
||||
MessageExport: model.MessageExport{
|
||||
TeamId: &channel.TeamId,
|
||||
TeamName: &channel.TeamName,
|
||||
TeamDisplayName: &channel.TeamDisplayName,
|
||||
ChannelId: &channel.ChannelId,
|
||||
ChannelName: &channel.ChannelName,
|
||||
ChannelDisplayName: &channel.DisplayName,
|
||||
ChannelType: &channel.ChannelType,
|
||||
UserId: &leave.UserId,
|
||||
UserEmail: &leave.UserEmail,
|
||||
Username: &leave.Username,
|
||||
IsBot: leave.UserType == shared.Bot,
|
||||
PostId: model.NewPointer(""),
|
||||
PostCreateAt: &leave.LeaveTime,
|
||||
PostMessage: &leaveMessage,
|
||||
PostType: &leavePostType,
|
||||
PostOriginalId: model.NewPointer(""),
|
||||
PostFileIds: []string{},
|
||||
},
|
||||
)
|
||||
}
|
||||
}, leavePostType, &leave.LeaveTime, leaveMessage),
|
||||
)
|
||||
}
|
||||
|
||||
sort.Slice(joinLeavePosts, func(i, j int) bool {
|
||||
return *joinLeavePosts[i].PostCreateAt < *joinLeavePosts[j].PostCreateAt
|
||||
})
|
||||
return joinLeavePosts, nil
|
||||
return joinLeavePosts
|
||||
}
|
||||
|
||||
func getPostAttachments(db store.Store, post *model.MessageExport) ([]*model.FileInfo, *model.AppError) {
|
||||
// if the post included any files, we need to add special elements to the export.
|
||||
if len(post.PostFileIds) == 0 {
|
||||
return []*model.FileInfo{}, nil
|
||||
}
|
||||
|
||||
attachments, err := db.FileInfo().GetForPost(*post.PostId, true, true, false)
|
||||
if err != nil {
|
||||
return nil, model.NewAppError("getPostAttachments", "ent.message_export.csv_export.get_attachment_error", nil, "", http.StatusInternalServerError).Wrap(err)
|
||||
}
|
||||
return attachments, nil
|
||||
}
|
||||
|
||||
func postToRow(post *model.MessageExport, createTime *int64, message string) []string {
|
||||
teamId := ""
|
||||
teamName := ""
|
||||
teamDisplayName := ""
|
||||
if post.TeamId != nil {
|
||||
teamId = *post.TeamId
|
||||
}
|
||||
if post.TeamName != nil {
|
||||
teamName = *post.TeamName
|
||||
}
|
||||
if post.TeamDisplayName != nil {
|
||||
teamDisplayName = *post.TeamDisplayName
|
||||
}
|
||||
postType := "message"
|
||||
if post.PostType != nil && *post.PostType != "" {
|
||||
postType = *post.PostType
|
||||
}
|
||||
postRootId := ""
|
||||
if post.PostRootId != nil {
|
||||
postRootId = *post.PostRootId
|
||||
}
|
||||
func postToRow(p shared.PostExport, postType string, createTime *int64, message string) Row {
|
||||
userType := "user"
|
||||
if post.IsBot {
|
||||
if p.IsBot {
|
||||
userType = "bot"
|
||||
}
|
||||
return Row{
|
||||
CreateAt: model.SafeDereference(createTime),
|
||||
UpdateAt: model.SafeDereference(p.PostUpdateAt),
|
||||
UpdateType: p.UpdatedType,
|
||||
TeamId: model.SafeDereference(p.TeamId),
|
||||
TeamName: model.SafeDereference(p.TeamName),
|
||||
TeamDisplayName: model.SafeDereference(p.TeamDisplayName),
|
||||
ChannelId: model.SafeDereference(p.ChannelId),
|
||||
ChannelName: model.SafeDereference(p.ChannelName),
|
||||
ChannelDisplayName: model.SafeDereference(p.ChannelDisplayName),
|
||||
ChannelType: shared.ChannelTypeDisplayName(model.SafeDereference(p.ChannelType)),
|
||||
UserId: model.SafeDereference(p.UserId),
|
||||
UserEmail: model.SafeDereference(p.UserEmail),
|
||||
Username: model.SafeDereference(p.Username),
|
||||
PostId: model.SafeDereference(p.PostId),
|
||||
EditedByPostId: p.EditedNewMsgId,
|
||||
RepliedToPostId: model.SafeDereference(p.PostRootId),
|
||||
PostMessage: message,
|
||||
PostType: postType,
|
||||
UserType: userType,
|
||||
PreviewsPostId: p.PreviewID(),
|
||||
}
|
||||
}
|
||||
|
||||
func rowToStringSlice(r Row) []string {
|
||||
return []string{
|
||||
strconv.FormatInt(*createTime, 10),
|
||||
teamId,
|
||||
teamName,
|
||||
teamDisplayName,
|
||||
*post.ChannelId,
|
||||
*post.ChannelName,
|
||||
*post.ChannelDisplayName,
|
||||
common_export.ChannelTypeDisplayName(*post.ChannelType),
|
||||
*post.UserId,
|
||||
*post.UserEmail,
|
||||
*post.Username,
|
||||
*post.PostId,
|
||||
*post.PostOriginalId,
|
||||
postRootId,
|
||||
message,
|
||||
postType,
|
||||
userType,
|
||||
post.PreviewID(),
|
||||
strconv.FormatInt(r.CreateAt, 10),
|
||||
strconv.FormatInt(r.UpdateAt, 10),
|
||||
string(r.UpdateType),
|
||||
r.TeamId,
|
||||
r.TeamName,
|
||||
r.TeamDisplayName,
|
||||
r.ChannelId,
|
||||
r.ChannelName,
|
||||
r.ChannelDisplayName,
|
||||
r.ChannelType,
|
||||
r.UserId,
|
||||
r.UserEmail,
|
||||
r.Username,
|
||||
r.PostId,
|
||||
r.EditedByPostId,
|
||||
r.RepliedToPostId,
|
||||
r.PostMessage,
|
||||
r.PostType,
|
||||
r.UserType,
|
||||
r.PreviewsPostId,
|
||||
}
|
||||
}
|
||||
|
||||
func attachmentToRow(post *model.MessageExport, attachment *model.FileInfo) []string {
|
||||
row := postToRow(post, post.PostCreateAt, *post.PostMessage)
|
||||
|
||||
attachmentEntry := fmt.Sprintf("%s (files/%s/%s-%s)", attachment.Name, *post.PostId, attachment.Id, path.Base(attachment.Path))
|
||||
attachmentMessage := "attachment"
|
||||
userType := row[len(row)-2]
|
||||
|
||||
if attachment.DeleteAt > 0 && post.PostDeleteAt != nil {
|
||||
deleteRow := postToRow(post, post.PostDeleteAt, *post.PostMessage)
|
||||
row = append(
|
||||
deleteRow[:len(deleteRow)-4],
|
||||
attachmentEntry,
|
||||
"deleted "+attachmentMessage,
|
||||
userType,
|
||||
)
|
||||
} else {
|
||||
row = append(
|
||||
row[:len(row)-4],
|
||||
attachmentEntry,
|
||||
attachmentMessage,
|
||||
userType,
|
||||
)
|
||||
func attachmentToRow(post shared.PostExport) Row {
|
||||
message := strings.TrimSpace(fmt.Sprintf("%s (files/%s/%s-%s)", post.FileInfo.Name, *post.PostId, post.FileInfo.Id, path.Base(post.FileInfo.Path)))
|
||||
postType := "attachment"
|
||||
if post.UpdatedType == shared.FileDeleted {
|
||||
postType = "deleted attachment"
|
||||
post.PostUpdateAt = model.NewPointer(post.FileInfo.DeleteAt)
|
||||
}
|
||||
return row
|
||||
|
||||
return postToRow(post, postType, post.PostCreateAt, message)
|
||||
}
|
||||
|
||||
Разница между файлами не показана из-за своего большого размера
Загрузить разницу
Ссылка в новой задаче
Block a user