PLT-7503: Create Message Export Scheduled Task and CLI Command (#7612)

* Created message export scheduled task

* Added CLI command to immediately kick off an export job

* Added email addresses for users joining and leaving the channel to the export

* Added support for both MySQL and PostgreSQL

* Fixing gofmt error

* Added a new ChannelMemberHistory store and associated tests

* Updating the ChannelMemberHistory channel as users create/join/leave channels

* Added user email to the message export object so it can be included in the actiance export xml

* Don't fail to log a leave event if a corresponding join event wasn't logged

* Adding copyright notices

* Adding message export settings to daily diagnostics report

* Added System Console integration for message export

* Cleaned up TODOs

* Made batch size configurable

* Added export from timestamp to CLI command

* Made ChannelMemberHistory table updates best effort

* Added a context-based timeout option to the message export CLI

* Minor PR updates/improvements

* Removed unnecessary fields from MessageExport object to reduce query overhead

* Removed JSON functions from the message export query in an effort to optimize performance

* Changed the way that channel member history queries and purges work to better account for edge cases

* Fixing a test I missed with the last refactor

* Added file copy functionality to file backend, improved config validation, added default config values

* Fixed file copy tests

* More concise use of the testing libraries

* Fixed context leak error

* Changed default export path to correctly place an 'export' directory under the 'data' directory

* Can't delete records from a read replica

* Fixed copy file tests

* Start job workers when license is applied, if configured to do so

* Suggestions from the PR

* Moar unit tests

* Fixed test imports
Этот коммит содержится в:
Jonathan
2017-11-30 09:07:04 -05:00
коммит произвёл GitHub
родитель d0d9ba4a7e
Коммит 375c0632fa
41 изменённых файлов: 1446 добавлений и 59 удалений

Просмотреть файл

@@ -55,6 +55,7 @@ type Features struct {
ThemeManagement *bool `json:"theme_management"`
EmailNotificationContents *bool `json:"email_notification_contents"`
DataRetention *bool `json:"data_retention"`
MessageExport *bool `json:"message_export"`
// after we enabled more features for webrtc we'll need to control them with this
FutureFeatures *bool `json:"future_features"`
@@ -76,6 +77,7 @@ func (f *Features) ToMap() map[string]interface{} {
"elastic_search": *f.Elasticsearch,
"email_notification_contents": *f.EmailNotificationContents,
"data_retention": *f.DataRetention,
"message_export": *f.MessageExport,
"future": *f.FutureFeatures,
}
}
@@ -152,6 +154,10 @@ func (f *Features) SetDefaults() {
if f.DataRetention == nil {
f.DataRetention = NewBool(*f.FutureFeatures)
}
if f.MessageExport == nil {
f.MessageExport = NewBool(*f.FutureFeatures)
}
}
func (l *License) IsExpired() bool {