* Audit logging - new schema added, old schema removed.
* fix linter error by running goimports
* Address review comments
* Address review comments
* Example usage of new audit logging API for the updateUserAuth call
* fixed unit test on auditing updating user record
* Changed the `TestUpdateConfigDiffInAuditRecord` testcase---it failed, because this PR changes how the `meta` field is serialized into the audit log records.
* fix linter error
* use string constants for record keys
* new audit api calls for api4/bot
* `Auditable` interface implementations for model classes
* New audit calls for channel api
* New audit calls for channel_local
* renamed receivers for required style reasons
* New audit calls for api4/command
* renamed receiver
* New audit calls for api4/command_local
* renamed receiver
* fix unit test to reflect changes in the Auditable implementation of the user class
* new audit calls for compliance
* new audit calls for configs
* remove auditRec.addMeta from updateConfig and patchConfig
* new audit calls for config_local
* new audit calls
* new audit calls for ldap, license apis
* new audit calls
* new audit calls
* new audit calls
* new audit calls
* new audit calls
* new audit calls
* new audit calls
* new audit calls
* fix linter error
* fixed linter error
* fixed "user update" test
* Don't include all of config when audit logging config changes. Also fix unit test on TestUpdateConfigDiffInAuditRecord
* address review comments
* Added Auditable() method for UserPatch
* Fix duplicative method declaration from merge
* Fix styling and API changes issues introduced with merge
* Fix broken test
Co-authored-by: Daniel Schalla <daniel@schalla.me>
* Add new Job to keep updating the last_accessible_post time
* Filter out posts for funcs returning PostList model
* Separate methods to get and compute cache
* filter pinned posts
* For posts with sorted CreateAt order, support a faster form of filtering.
* Add inaccessible header for getPost and getPostsByIDs APIs
* replace manual binary search with the std. library
* in-place filter posts
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
Co-authored-by: Nathaniel Allred <neallred@protonmail.com>
* Working on refactoring jobs service
* Making more consistent with the previous existing code
* Remove no longer needed functions
* Making a base PeridicScheduler to use it in most of the schedulers implementations
* Removing accidental complexity from on of the jobs
* Removing accidental complexity from expirynotify
* Fixing compilation from previous commit
* Remove accidental complexity from the export_delete job
* Simplifying the workers by making a reusable worker
* Using simple worker for export_delete job
* Simpliying export process job
* Simpliying extract content job
* Simpliying import delete job
* Simpliying import process job
* Simpliying product noticies job
* Simpliying fix crt channel unreads job (only removing the uneeded register function)
* Simpliying migrations job (only removing the uneeded register function)
* fixup
* Simpliying plugins job (only removing the uneeded register function)
* Simpliying bleve indexing job (only removing the uneeded register function)
* Simpliying resend invitation email job (only removing the uneeded register function)
* Fixing tests
* Simplifying migration tests infrastructure
* Adding missed license to files
* Adding an empty file to imports package to ensure this package exist even without enterprise repo
* Regenerating einterfaces mocks
* Adding missed license to files
* Updating i18n/en.json file
* help fixing enterprise tests compilation
* Adding new DailyScheduler
* Fixing typo and changing the waitTime type for periodic sechduler
* Making the daily scheduler more generic
* Adding comments to clarify not used parameters in interface scheduler interface implementations
* Using merror to handle multiple errors in jobs workers
* Fixing linter errors
* Addressing PR review comments
* Reverting go.tools.mod changes
* Removing the static check for worker type in the model (moving it to the insertion of new jobs
* Moving migrations job to the jobs directory
* Fixing (and improving a bit) tests
* Apply suggestions from code review
Co-authored-by: Doug Lauder <wiggin77@warpmail.net>
* Fixing enterprise tests
* Removing unneeded InitWorkers/InitSchedulers calls
* Fix expirenotify job when error happens
* Fixing govet errors
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
Co-authored-by: Doug Lauder <wiggin77@warpmail.net>
Summary
The addition of the TotalMsgCountRoot and MsgCountRoot columns to support CRT caused several issues with previously read threads and channels being marked as unread. Previously we attempted to fix this purely in a SQL migration [MM-35345][MM-35494] fixes for incorrect mentions and unreads for threads and channels #17803 but that turned out to be too heavy and it was decided to break up some of the fixes into async jobs.
This PR implements an async job to mark channels as read if there are no user posts since the last time the user viewed the channel.
Ticket Link
https://mattermost.atlassian.net/browse/MM-37013
* Include filepaths for post attachments
* Cleanup
* Enable exporting file attachments
* Fix file import
* Enable zip export
* Support creating missing directories when unzipping
* Add test
* Add translations
* Export direct channel posts attachments
* Fix returned values order
Remove pointer to slice in return
* [MM-31597] Implement export process job (#16626)
* Implement export process job
* Add translations
* Remove unused value
* [MM-31249] Add /exports API endpoint (#16633)
* Implement API endpoints to list, download and delete export files
* Add endpoint for single resource
* Update i18n/en.json
Co-authored-by: Ibrahim Serdar Acikgoz <serdaracikgoz86@gmail.com>
* Update i18n/en.json
Co-authored-by: Ibrahim Serdar Acikgoz <serdaracikgoz86@gmail.com>
Co-authored-by: Ibrahim Serdar Acikgoz <serdaracikgoz86@gmail.com>
* Implement job to delete export files
* Fix app layers
* Fix typo
Co-authored-by: Ibrahim Serdar Acikgoz <serdaracikgoz86@gmail.com>
* Include filepaths for post attachments
* Cleanup
* Enable exporting file attachments
* Fix file import
* Enable zip export
* Support creating missing directories when unzipping
* Add test
* Add translations
* Export direct channel posts attachments
* Fix returned values order
Remove pointer to slice in return
* [MM-31597] Implement export process job (#16626)
* Implement export process job
* Add translations
* Remove unused value
* [MM-31249] Add /exports API endpoint (#16633)
* Implement API endpoints to list, download and delete export files
* Add endpoint for single resource
* Update i18n/en.json
Co-authored-by: Ibrahim Serdar Acikgoz <serdaracikgoz86@gmail.com>
* Update i18n/en.json
Co-authored-by: Ibrahim Serdar Acikgoz <serdaracikgoz86@gmail.com>
Co-authored-by: Ibrahim Serdar Acikgoz <serdaracikgoz86@gmail.com>
* Fix var name
Co-authored-by: Ibrahim Serdar Acikgoz <serdaracikgoz86@gmail.com>
* Implement unzip function
* Implement FileSize method
* Implement path rewriting for bulk import
* Small improvements
* Add ImportSettings to config
* Implement ListImports API endpoint
* Enable uploading import files
* Implement import process job
* Add missing license headers
* Address reviews
* Make path sanitization a bit smarter
* Clean path before calculating Dir
* [MM-30008] Add mmctl support for file imports (#16301)
* Add mmctl support for import files
* Improve test
* Remove unnecessary handlers
* Use th.TestForSystemAdminAndLocal
* Make nouser id a constant
* Adding files commiting to do something else
* Stashing to merge master
* Adding files, commit of working code, pre-test
* Changing up logic according to new conversations, adding template and functions for 7 day and 14 day emails
* Add subscription active check
* Add a comment around subscription status
* Fix i18n
* Update jobs/cloud/worker.go
Co-authored-by: Maria A Nunez <maria.nunez@mattermost.com>
* Add a check for cloud license and exit early
* remove log
* Update jobs/cloud/worker.go
Co-authored-by: Maria A Nunez <maria.nunez@mattermost.com>
* Remove case for 91st day
* Change var name
* Fix i18n
* Change value for dates in email
* Add email template and send logic for support email on 30th day of arrears
* Fix EETests?
* add one to the day so the count starts on first day of next cycle
* use the license for the enabled check
* Moved scheduler and worker into enterprise
* Add the user count to the support email
* remove a line
* Use the date for the 14 day email
* Fix for design review
* More font changes
* Fixes for PR
* Change feature flag name
* Add cloud_interface to einterfaces/jobs
* Add back & for running job server
* Remove unused constant
Co-authored-by: Maria A Nunez <maria.nunez@mattermost.com>
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* Add metrics to MM server for total enabled user count and include installation ID from a new env var
* diagnostics context
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
* new job type created that checks for expired mobile sessions and pushes notifications.
* only send session expired notifications if ExtendSessionLengthWithActivity is enabled.
* includes schema change: field added to Sessions table
* [MM-20979] Add first implementation of the Bleve search engine
* Fix i18n
* Migrate searchengine utils tests
* Fix linter
* Don't add allTermsQ if both termQueries and notTermQueries are empty
* Fix test that should work if user is system admin
* Modify naming according to review comments
* Abstract getIndexDir function
* Extracting bleve engine name as a constant
* Merge both Indexer interfaces into one
* Add worker stopped message
* Allow worker to be started/stopped with config change
* Use constants for index names
* Modify test order
* Fix linter
* Trying to unlock the CI
* add warning count as return value
* add warning count as return value
* fix file name
* update mock
* add setting warning to db
* replace wrongly removed string
* add dummy function to see if it will build
* remove dummy function
Co-authored-by: mattermod <mattermod@users.noreply.github.com>
* Consistent license message for all the go files
* Fixing the last set of unconsistencies with the license headers
* Addressing PR review comments
* Fixing busy.go and busy_test.go license header
Expire K/V Values
Regenerate Code
pathfix
Update Expiry on Update
Check for Exit Signal
gofmt
Rewrote Go Routine
Remove tempoarily cleanup loop
fix expiretime
TEST: Expired Watchdog as GoRoutine
Check if Srv is nil
Use Scheduler/Worker for Expired Key CleanUp
add license
fix scheduler job type; DoJob Restructuring
Remove unused imports and constants
move db migration from 5.4 to 5.5
* MM-9728: Online migration for advanced permissions phase 2
* Add unit tests for new store functions.
* Move migration specific code to own file.
* Add migration state function test.
* Style fixes.
* Add i18n strings.
* Fix mocks.
* Add TestMain to migrations package tests.
* Fix typo.
* Fix review comments.
* Fix up the "Check if migration is done" check to actually work.
* Created message export scheduled task
* Added CLI command to immediately kick off an export job
* Added email addresses for users joining and leaving the channel to the export
* Added support for both MySQL and PostgreSQL
* Fixing gofmt error
* Added a new ChannelMemberHistory store and associated tests
* Updating the ChannelMemberHistory channel as users create/join/leave channels
* Added user email to the message export object so it can be included in the actiance export xml
* Don't fail to log a leave event if a corresponding join event wasn't logged
* Adding copyright notices
* Adding message export settings to daily diagnostics report
* Added System Console integration for message export
* Cleaned up TODOs
* Made batch size configurable
* Added export from timestamp to CLI command
* Made ChannelMemberHistory table updates best effort
* Added a context-based timeout option to the message export CLI
* Minor PR updates/improvements
* Removed unnecessary fields from MessageExport object to reduce query overhead
* Removed JSON functions from the message export query in an effort to optimize performance
* Changed the way that channel member history queries and purges work to better account for edge cases
* Fixing a test I missed with the last refactor
* Added file copy functionality to file backend, improved config validation, added default config values
* Fixed file copy tests
* More concise use of the testing libraries
* Fixed context leak error
* Changed default export path to correctly place an 'export' directory under the 'data' directory
* Can't delete records from a read replica
* Fixed copy file tests
* Start job workers when license is applied, if configured to do so
* Suggestions from the PR
* Moar unit tests
* Fixed test imports
* PLT-7302: Aggregate Elasticsearch indexes over a certain age.
This is done by a scheduled daily job, in order to keep the shard count
to a sensible level in Elasticsearch.
* Use map[string]string instead of StringMap
This commit redesigns the jobserver to be based around an architecture
of "workers", which carry out jobs of a particular type, and "jobs"
which are a unit of work carried by a particular worker. It also
introduces "schedulers" which are responsible for scheduling jobs of a
particular type automatically (jobs can also be scheduled manually when
apropriate).
Workers may be run many times, either in instances of the platform
binary, or the standalone jobserver binary. In any mattermost cluster,
only one instance of platform OR jobserver must run the schedulers. At
the moment this is controlled by a config variable, but in future will
be controlled through the cluster leader election process.
* Added initial job server
* Added job server to be ran as part of platform
* Added test job to the enterprise repo
* Fixed job server not loading license
* Renamed job package to jobs
* Fixed TE not being buildable
* Added JobStatus table to database
* Changed fields used by JobStatus
* Added APIs to query job status
* Added config change listener to server
* Added option to run job server from Makefile
* Added ability to enable/disable jobs from config
* Commented out placeholder for search indexing job
* Fixed govet
* Removed debug messages and fixed job api init message
* Adding initial race detector
* Remove setting of config twice
* Fixing config file watch and config reload on license save
* Fixing config file watch and config reload on license save
* Fixing build error
* Fixing locking issue
* Fixing makefile
* Fixing race in config
* Fixing race in status unit test
* Adding EE race tests
* Fixing race in cluster info
* Removing code that's isn't needed
* Fixing some more races
* Fixing govet issue
* Adding go vet to the api package
* Adding go vet to app package
* Adding go vet to manualtesting package
* Adding go vet to the model package
* Adding go vet to the store dir
* Adding go vet to utils package
* Adding missing dirs with go files
* Fixing up makefile
* Fixing up makefile
* Removing root dir
