Граф коммитов

59 Коммитов

Автор SHA1 Сообщение Дата
Siyuan Liu
c7f7bef9ec move cache2 package to cache (#14921)
Co-authored-by: Mattermod <mattermod@users.noreply.github.com>
2020-07-18 13:31:06 +05:30
Eli Yukelzon
11812ae8e4 MM-25450 deleted posts should be sanitised (#14769) 2020-06-11 13:12:13 +03:00
Siyuan Liu
6a5dd550c8 migrate direct usage of lru to cache2 (#14508)
Automatic Merge
2020-06-02 06:01:30 -07:00
Agniva De Sarker
15ed9a8f20 MM-25006: Fix occurences of unclosed http bodies (#14521)
* MM-25006: Fix occurences of unclosed http bodies

And while here, we also use ioutil.Discard to read the remaining
body instead of reading with ioutil.ReadAll which allocates a separate
byte buffer.

* Fix mistake

* Address review comments

Co-authored-by: mattermod <mattermod@users.noreply.github.com>
2020-05-20 08:41:37 +05:30
Claudio Costa
1e53fe85ad [MM-21378] Add mutex to model.Post to guard against race conditions on Post.Props (#13884)
* Add mutex to model.Post to guard against race conditions on Post.Props

* Rename mutex

* Add GetProp() method to Post

* Fix more tests

* Fix flaky test

Benchmarks:

BenchmarkPostPropsGet_indirect
BenchmarkPostPropsGet_indirect-2     	85026746	        13.0 ns/op	       0 B/op	       0 allocs/op
BenchmarkPostPropsGet_indirect-4     	90273747	        13.0 ns/op	       0 B/op	       0 allocs/op
BenchmarkPostPropsGet_indirect-8     	88324293	        13.0 ns/op	       0 B/op	       0 allocs/op
BenchmarkPostPropsGet_indirect-16    	91427720	        13.1 ns/op	       0 B/op	       0 allocs/op
BenchmarkPostPropsGet_direct
BenchmarkPostPropsGet_direct-2       	1000000000	         0.242 ns/op	       0 B/op	       0 allocs/op
BenchmarkPostPropsGet_direct-4       	1000000000	         0.241 ns/op	       0 B/op	       0 allocs/op
BenchmarkPostPropsGet_direct-8       	1000000000	         0.240 ns/op	       0 B/op	       0 allocs/op
BenchmarkPostPropsGet_direct-16      	1000000000	         0.241 ns/op	       0 B/op	       0 allocs/op
BenchmarkPostPropsAdd_indirect
BenchmarkPostPropsAdd_indirect-2     	 5602224	       203 ns/op	     336 B/op	       2 allocs/op
BenchmarkPostPropsAdd_indirect-4     	 5959496	       206 ns/op	     336 B/op	       2 allocs/op
BenchmarkPostPropsAdd_indirect-8     	 5833999	       205 ns/op	     336 B/op	       2 allocs/op
BenchmarkPostPropsAdd_indirect-16    	 5802493	       225 ns/op	     336 B/op	       2 allocs/op
BenchmarkPostPropsAdd_direct
BenchmarkPostPropsAdd_direct-2       	100000000	        11.3 ns/op	       0 B/op	       0 allocs/op
BenchmarkPostPropsAdd_direct-4       	100000000	        11.3 ns/op	       0 B/op	       0 allocs/op
BenchmarkPostPropsAdd_direct-8       	100000000	        11.6 ns/op	       0 B/op	       0 allocs/op
BenchmarkPostPropsAdd_direct-16      	99840794	        11.4 ns/op	       0 B/op	       0 allocs/op
BenchmarkPostPropsDel_indirect
BenchmarkPostPropsDel_indirect-2     	18824002	        61.9 ns/op	      48 B/op	       1 allocs/op
BenchmarkPostPropsDel_indirect-4     	19470736	        63.8 ns/op	      48 B/op	       1 allocs/op
BenchmarkPostPropsDel_indirect-8     	17640460	        65.3 ns/op	      48 B/op	       1 allocs/op
BenchmarkPostPropsDel_indirect-16    	18692962	        65.4 ns/op	      48 B/op	       1 allocs/op
BenchmarkPostPropsDel_direct
BenchmarkPostPropsDel_direct-2       	516257440	         2.34 ns/op	       0 B/op	       0 allocs/op
BenchmarkPostPropsDel_direct-4       	514865216	         2.43 ns/op	       0 B/op	       0 allocs/op
BenchmarkPostPropsDel_direct-8       	511330477	         2.37 ns/op	       0 B/op	       0 allocs/op
BenchmarkPostPropsDel_direct-16      	499504010	         2.38 ns/op	       0 B/op	       0 allocs/op
2020-03-13 21:12:20 +01:00
JtheBAB
cd36c9f041 MM-22355 - Fix typos detected by goreportcad.com (#13842)
* Update webhook_test.go

* Update user_test.go

* Update filesstore_test.go

* Update plugin_requests.go

* Update syncables.go

* Update helper.go

* Update html_entities.go

* Update user.go

* Update team.go

* Update notification.go

* Update notification_test.go

* Update plugin_api_test.go

* Update post_metadata.go

* Update channel_test.go

* Update database.go

* Update channel.go

* Update user_store.go

* Update team_test.go

* revert andd

* Revert back to infintie

Co-authored-by: Jesús Espino <jespinog@gmail.com>
Co-authored-by: mattermod <mattermod@users.noreply.github.com>
2020-03-02 17:10:40 +01:00
Eli Yukelzon
17523fa5d9 MM-21898 - Part 1: Generate and use an interface instead of *A… (#13840)
* Generate and use an interface instead of *App
2020-02-13 13:26:58 +01:00
Jesús Espino
a76643116f Making private opengraph methods in the app layer (#13759) 2020-01-28 10:56:45 +01:00
Jesús Espino
06fb1458ca Use always strings for cache keys (#13631)
* Use always strings for cache keys

* fixing tests

* Addressing PR review comments

* Adding the base to FormatInt

* Fix typo
2020-01-22 15:59:59 +01:00
Hector
62b57143c8 [MM 19840] Create a Cache Interface abstraction to support multiple cache backends (#13384)
* Refactor to use structured logging

* Properly formatted with gofmt

* created interface Cache, but construction of cache is still coupled.

* Implementing cache factories to build caches

* Simple redis implementation without error handling. Keys and values by default are string

* refactor NewLocalCacheLayer to inject cache factory

* Removed redis impl to focus on cache abstraction

* CacheFactory injected on sqlsupplier and saved in Store struct

* remove useless private method

* replace concrete declaration of lru cache to cache abstraction

* discard spaces

* Renamed factory to provider because concrete implementations of factories may hold an state (such as a redis client for example)

* refactor to include all caches in the same package cache and subpackages

* method close cache. This method will be used for concrete implementations that need to release resources (close a connection, etc)

* closing cacheprovider and releasing resources while closing sql store

* fixed merge conflict fail

* gofmt files

* remove unused property from post_store

* naming refactor to avoid stutter. Added godocs on interface

* Store doesnt know anything about the cache and provider. Cache provider will be built after loading config and injected in localCacheLayer

* fixed broken test

* cache provider initialized before RunOldAppInitialization which initializes the localcachelayer

* move statusCache to server to initialize it with the new cache provider

* update terms_service and channel_layer to have new cacheProvider

* gofmt

* Add Connect method to the cache provider

* mock cacheprovider in user_layer_test

Co-authored-by: Ben Schumacher <ben.schumacher@mattermost.com>
Co-authored-by: mattermod <mattermod@users.noreply.github.com>
2020-01-09 09:57:28 +01:00
Jesús Espino
a63684fcb5 Consistent license message for all the go files (#13235)
* Consistent license message for all the go files

* Fixing the last set of unconsistencies with the license headers

* Addressing PR review comments

* Fixing busy.go and busy_test.go license header
2019-11-29 12:59:40 +01:00
Miguel de la Cruz
2259b7f2a8 [MM-19948] Set version on module file and internal paths (#13186)
* [MM-19948] Set version on module file and internal paths

* Fixes after merge

* Fix i18n checker error
2019-11-28 14:39:38 +01:00
Ben Schumacher
7a665aacdd Run gosimple against codebase (#12928) 2019-10-29 07:45:09 +01:00
Harrison Healey
1234e68575 MM-19574 Don't return metadata for deleted posts (#12878) 2019-10-23 14:31:22 -04:00
Miguel de la Cruz
a9ed8fab28 Return POST_EMBED_LINK for links that are not images nor og (#11723) 2019-08-22 23:00:37 +02:00
Harrison Healey
23ef7d9a62 MM-17609 Add EnableSVGs setting (#11874)
* MM-17609 Add EnableSVGs setting

* MM-17609 Return SVG images in post metadata
2019-08-14 13:40:40 -04:00
Harrison Healey
fe9e8a4493 MM-17426 Properly specify multiple Accept headers for post metadata (#11766)
* MM-17326 Properly specify multiple Accept headers for post metadata

* Add q= weight to text/html
2019-08-01 13:52:38 -04:00
krjn
4ada8aacd4 MM-16389 - complement for emoji_icon issue (#11650) 2019-07-18 14:54:01 +02:00
krjn
dd30488a09 [MM-16389] Add icon_emoji parameter to webhooks (#11586)
[MM-16389] Add icon_emoji field to posts from incoming webhooks

To be used to substitute the profile picture on posts.
2019-07-17 11:01:18 +02:00
Saturnino Abril
b832985f1d MM-11210 Add "GET /posts/unread" API to support landing on the last unread post (#11486)
* [MM-11210] Add API GET 'api/v4/channels/{channel_id:[A-Za-z0-9]+}/posts/unread' for scrolling overhaul (#9108)

* Add API GET 'api/v4/channels/{channel_id:[A-Za-z0-9]+}/posts/unread'

* add constants

* refactor GetPostSince and added more tests

* move constants to app package

* [MM-11528 &&  MM-11583] Add userId to in the "posts/unread" path and update test with time delay to fix intermittent failure (#9229)

* add userId to in the "posts/unread" path and update test with time delay to fix intermittent failure

* add limit before and after to query

* remove time delay on test and put pretermined value of Post.CreateAt

* Fix conflict

* [MM-11876] Add cursor to posts list such as next_post_id and previous_post_id (#9707)

* add cursor to posts list such as next_post_id and previous_post_id

add publish previous_post_id on WEBSOCKET_EVENT_POSTED and only get next or previous post IDs if necessary

revert change on adding previous_post_id in WEBSOCKET_EVENT_POSTED

add missing strings import

fix merge conflicts

* update per comment

* update per feedback

* corrected the logic in getting the next and previous post ID

* fix logic to determine next and post IDs, and rename function to have suffix of "Time"

* rearrange logics and add mote tests

* fix merge conflict

* fix missing message when using unread API (#10233)

* MM-15569 Fixes failing test on TestGetPostsForChannelAroundLastUnread (#11039)

* Fix missing posts when getting posts since

* revert changes to GetPostsSince

* migrate Post.GetPostAfterTime and Post.GetPostBeforeTime to sync by default

* revert change to cacheItem

* Fix post ID validation, build query on squirrel and only return post ID as necessary
2019-07-04 02:53:04 +05:30
Kévin Etienne
ea8d441345 MM 16523 Migrate "LinkMetadata.Save" to Sync by default (#11416)
* Update LinkMetadata.Save store and interface

* Update mocks with make store-mocks

* Update tests and implementation
2019-06-28 10:28:08 +01:00
Kévin Etienne
df9af66a28 MM-16524 Migrate "LinkMetadata.Get" to Sync by default (#11376)
* Update LinkMetadata.Get store and interface

* Update mocks with make store-mocks

* Update tests and implementation

* Avoid casting LinkMetadata.Data

* Fix indent

* Make error more explicit

* Fix indent (was containing spaces)

* Test value returned by LinkMetadata().Get()
2019-06-26 09:36:29 -04:00
Harrison Healey
ff0696c667 MM-16228 Remove DisablePostMetadata setting (#11138)
* MM-16228 Remove DisablePostMetadata setting

* Remove DEPRECATED_DO_NOT_USE_DisablePostMetadata
2019-06-13 17:32:36 -04:00
Guillermo Vayá
ad69002f9e Truncate data from OpenGraph metadata (#10532)
* Truncate strings from OpenGraph metada

* remove unwanted opengraph fields, limit to 1 image

* test helper functions

* Add truncating test

* fix typo

* change test into not comparing for the pointer value

* truncate only once

* only shorten fields that were already present

* set maximum of 5 images

* fix original tests

* fix test

* limit to 5 images

* fix typo

* place functions below types, simplify function commentaries

* Rewrite how to empty opengraph's structure
2019-06-11 17:39:35 +02:00
Angela Chang
8e6a158449 [MM-8421] Special PR to have parseImages return nil when format is tiff (#10928) 2019-05-27 22:20:10 +02:00
Jesús Espino
87ff64ea6d MM-15035: Read from file infos from master on post creation (#10899)
* MM-15035: Read from file infos from master on post creation

* Handling edit post cases for get files metadata
2019-05-23 14:29:41 +02:00
Harrison Healey
99ea780f20 MM-13142 Replace mlog.Any with mlog.Err (#10848) 2019-05-15 16:04:43 -04:00
Harrison Healey
e50b642e43 MM-14030 Add format and frame count to image metadata (#10757)
* MM-14030 Add initial CountFrames

* MM-14030 Add format and frame count to image metadata

* Fix tests and stop using image dimensions from OpenGraph

* Fix copyright header

* Move license to NOTICE.txt
2019-04-30 16:45:26 -04:00
Harrison Healey
a5501cf3f8 MM-15302 Don't get post embeds with link previews disabled (#10587)
* MM-15302 Don't get post embeds with link previews disabled

* Enable link previews for other metadata tests
2019-04-10 15:48:15 -04:00
Harrison Healey
f9a6e8dfd9 MM-14558 Downgrade warnings printed by post metadata to be debug messages (#10445)
* MM-14558 Downgrade warnings printed by post metadata to be debug messages

* Revert log level for some post metadata messages
2019-03-14 10:21:43 +01:00
Harrison Healey
0e31ce9110 MM-13997 Prioritize images over OpenGraph metadata when requesting metadata (#10241) 2019-02-09 10:44:38 +01:00
Harrison Healey
3a8e8739b2 MM-13996 Properly get image dimensions for OpenGraph images using secure_url (#10240)
For images in the OpenGraph metadata, we only looked for the `url` field, but we should've also been looking for the `secure_url` field for sites that defined it. We also set the `secure_url` field when proxying OpenGraph images as well, so we were not properly giving image dimensions for OpenGraph images.

#### Ticket Link
https://mattermost.atlassian.net/browse/MM-13996
2019-02-07 19:43:25 +01:00
Harrison Healey
dbf54b3599 MM-13838 Bypass the HTTP client when getting image dimensions from the image proxy (#10208)
* MM-13838 Bypass the HTTP client when getting image dimensions from the proxy

* Add additional log messages to debug failing test

* Fix unit test to work on Jenkins
2019-02-04 12:43:30 -05:00
Harrison Healey
46b05499cc MM-13932 Fix image metadata for invalid image links (#10217) 2019-02-01 18:23:04 -04:00
Daniel Schalla
a2f46c0466 [MM-13889] Limit OpenGraph data response size (#10213)
* Limit OpenGraph data response size

* Add LimitReader to Metadata Image
2019-02-01 18:55:43 +01:00
Harrison Healey
ee9395f1b6 MM-13917 Enable post metadata by default (#10210) 2019-01-31 14:57:50 -05:00
Harrison Healey
7c677b6196 MM-13207 Add customizable timeout for link metadata and improve caching of errors (#10188)
* MM-13207 Add customizable timeout for link metadata and improve caching of errors

* Rename LinkMetadataTimeout to LinkMetadataTimeoutMilliseconds

* Add diagnostics for LinkMetadataTimeoutMilliseconds
2019-01-31 09:40:23 -05:00
Hanzei
0c981aa010 Prohibit variable shadowing (#10177)
* Prohibit variable shadowing

* Fix outstanding shadowed variables
2019-01-31 12:16:16 +01:00
Harrison Healey
26684716aa MM-13664 Cache external link metadata when populating post metadata (#10128)
* MM-13664 Added LinkMetadata types

* MM-13664 Use LinkMetadata when populating post metadata

* Fix unused import

* Fix index name on SQLite

* Finish adding unit tests

* Address feedback

* Increase max length of URL column to 2048 characters
2019-01-28 16:25:08 +01:00
Harrison Healey
ba5566d1a0 MM-10417 Add local image proxy and enable by default (#9967)
* MM-10417 Add local image proxy and enable by default

* Remove unused function

* Add dependencies for willnorris/imageproxy

* Fixed compilation errors

* Lock to the master version of willnorris/imageproxy

* Fix atmos/camo proxy when no SiteURL is specified

* Re-add default values for deprecated settings

* Fix unit tests added by merge

* Pass imageproxy to App struct

* Remove unneeded locking when creating the image proxy

* Remove empty test file
2019-01-24 16:11:32 -04:00
Harrison Healey
1a3ccaf305 MM-13606 Remove consumeAndClose and clean up integration response handling (#10066)
* MM-13606 Remove consumeAndClose

* Allow overriding HTTPService's request timeout

* MM-13606 Clean up integration response handling

* Properly close httptest servers

* Address feedback

* Only call buf.Bytes when necessary

* Properly check for errors in doOutgoingWebhookRequest

* Add comment explaining ignored ioutil.ReadAll errors
2019-01-09 17:07:08 -05:00
Harrison Healey
aff3156c56 MM-13238 Fix link cache being cleared too often with image proxy (#9981) 2018-12-12 13:11:53 -05:00
Harrison Healey
1607d18e55 MM-12009 Resolve relative links in posts as the web app does (#9896) 2018-11-28 11:01:35 -05:00
Harrison Healey
4294758793 MM-13208 Rename DisablePostMetadata setting to EnablePostMetadata and turn off by default (#9894) 2018-11-27 16:13:02 -05:00
Harrison Healey
ba627c0f92 Skip getting the post again when getting file metadata for a post (#9879) 2018-11-26 10:47:01 +00:00
Harrison Healey
71dba81e91 MM-13015 Add safety valve setting to disable post metadata (#9877)
* MM-13015 Add safety valve setting to disable post metadata

* Remove setting from client config since it's no longer needed
2018-11-23 11:59:34 -05:00
Harrison Healey
d018554ef3 MM-11434 Only call PreparePostForClient once when creating a post (#9868) (#9876)
* MM-11434 Only call PreparePostForClient once when creating a post

* Have PreparePostForClient provide new metadata when a post already has it and update tests
2018-11-23 10:20:02 -05:00
Harrison Healey
eff68da254 MM-11434 Revert "Parallelize PreparePostListForClient" 2018-11-22 13:14:55 -05:00
Harrison Healey
5f65356a3f Address issues 2018-11-22 13:10:36 -05:00
Harrison Healey
5388afd37e Remove comments about deprecating FileIds and HasReactions (#9869) 2018-11-22 13:09:27 -05:00