ef85001523
* MM-25710: Use an efficient cache serialization algorithm We investigate 3 packages for selecting a suitable replacement for gob encoding. The algorithm chosen was msgpack which gives a decent boost over the standard gob encoding. Any external schema dependent algorithms like protobuf, flatbuffers, avro, capn'proto were not considered as that would entail converting the model structs into separate schema objects and then code generating the Go structs. It could be done theoretically at a later stage specifically for structs which are in the hot path. This is a general solution for now. The packages considered were: - github.com/tinylib/msgp - github.com/ugorji/go/codec - github.com/vmihailenco/msgpack/v5 msgp uses code generation to generate encoding/decoding code without the reflection overhead. Theoretically, therefore this is supposed to give the fastest performance. However, a major flaw in that package is that it works only at a file/directory level, not at a package level. Therefore, for structs which are spread across multiple files, it becomes near to impossible to chase down all transitive dependencies to generate the code. Even if that's done, it fails on some complex type like xml.Name and time.Time. (See: https://github.com/tinylib/msgp/issues/274#issuecomment-643654611) Therefore, we are left with 2 choices. Both of them use the same underlying algorithm. But msgpack/v5 wraps the encoders/decoders in a sync.Pool. To make a perfect apples-apples comparison, I wrote a sync.Pool for ugorji/go/codec too and compared performance. msgpack/v5 came out to be the fastest by a small margin. benchstat master.txt ugorji.txt name old time/op new time/op delta LRU/simple=new-8 5.62µs ± 3% 3.68µs ± 2% -34.64% (p=0.000 n=10+10) LRU/complex=new-8 38.4µs ± 2% 9.1µs ± 2% -76.38% (p=0.000 n=9+9) LRU/User=new-8 75.8µs ± 2% 23.5µs ± 2% -69.01% (p=0.000 n=10+10) LRU/Post=new-8 125µs ± 2% 21µs ± 3% -82.92% (p=0.000 n=9+10) LRU/Status=new-8 27.6µs ± 1% 5.4µs ± 4% -80.34% (p=0.000 n=10+10) name old alloc/op new alloc/op delta LRU/simple=new-8 3.20kB ± 0% 1.60kB ± 0% -49.97% (p=0.000 n=10+10) LRU/complex=new-8 15.7kB ± 0% 4.4kB ± 0% -71.89% (p=0.000 n=9+10) LRU/User=new-8 33.5kB ± 0% 9.2kB ± 0% -72.48% (p=0.000 n=10+8) LRU/Post=new-8 38.7kB ± 0% 4.8kB ± 0% -87.48% (p=0.000 n=10+10) LRU/Status=new-8 10.6kB ± 0% 1.7kB ± 0% -83.50% (p=0.000 n=10+10) name old allocs/op new allocs/op delta LRU/simple=new-8 46.0 ± 0% 20.0 ± 0% -56.52% (p=0.000 n=10+10) LRU/complex=new-8 324 ± 0% 48 ± 0% -85.19% (p=0.000 n=10+10) LRU/User=new-8 622 ± 0% 108 ± 0% -82.64% (p=0.000 n=10+10) LRU/Post=new-8 902 ± 0% 74 ± 0% -91.80% (p=0.000 n=10+10) LRU/Status=new-8 242 ± 0% 22 ± 0% -90.91% (p=0.000 n=10+10) 11:31:48-~/mattermost/mattermost-server/services/cache2$benchstat master.txt vmi.txt name old time/op new time/op delta LRU/simple=new-8 5.62µs ± 3% 3.68µs ± 3% -34.59% (p=0.000 n=10+10) LRU/complex=new-8 38.4µs ± 2% 8.7µs ± 3% -77.45% (p=0.000 n=9+10) LRU/User=new-8 75.8µs ± 2% 20.9µs ± 1% -72.45% (p=0.000 n=10+10) LRU/Post=new-8 125µs ± 2% 21µs ± 2% -83.08% (p=0.000 n=9+10) LRU/Status=new-8 27.6µs ± 1% 5.1µs ± 3% -81.66% (p=0.000 n=10+10) name old alloc/op new alloc/op delta LRU/simple=new-8 3.20kB ± 0% 1.60kB ± 0% -49.89% (p=0.000 n=10+10) LRU/complex=new-8 15.7kB ± 0% 4.6kB ± 0% -70.87% (p=0.000 n=9+8) LRU/User=new-8 33.5kB ± 0% 10.3kB ± 0% -69.40% (p=0.000 n=10+9) LRU/Post=new-8 38.7kB ± 0% 6.0kB ± 0% -84.62% (p=0.000 n=10+10) LRU/Status=new-8 10.6kB ± 0% 1.9kB ± 0% -82.41% (p=0.000 n=10+10) name old allocs/op new allocs/op delta LRU/simple=new-8 46.0 ± 0% 20.0 ± 0% -56.52% (p=0.000 n=10+10) LRU/complex=new-8 324 ± 0% 46 ± 0% -85.80% (p=0.000 n=10+10) LRU/User=new-8 622 ± 0% 106 ± 0% -82.96% (p=0.000 n=10+10) LRU/Post=new-8 902 ± 0% 89 ± 0% -90.13% (p=0.000 n=10+10) LRU/Status=new-8 242 ± 0% 23 ± 0% -90.50% (p=0.000 n=10+10) In general, we can see that the time to marshal/unmarshal pays off as the size of the struct increases. We can see that msgpack/v5 is faster for CPU but very slightly heavier on memory. Since we are interested in fastest speed, we choose msgpack/v5. As a future optimization, we can use a mix of msgpack and msgp for hot structs. To do that, we would need to shuffle around some code so that for the hot struct, all its dependencies are in the same file. Let's use this in production for some time, watch grafana graphs for the hottest caches and come back to optimizing this more once we have more data. Side note: we have to do with micro-benchmarks for the time being, because all the caches aren't migrated to cache2 interface yet. Once that's in, we can actually run some load tests and do comparisons. * Bring back missing import * Fix tests
139 строки
6.6 KiB
Modula-2
139 строки
6.6 KiB
Modula-2
module github.com/mattermost/mattermost-server/v5
|
|
|
|
go 1.14
|
|
|
|
require (
|
|
github.com/Masterminds/squirrel v1.2.0
|
|
github.com/NYTimes/gziphandler v1.1.1
|
|
github.com/RoaringBitmap/roaring v0.4.23 // indirect
|
|
github.com/armon/go-metrics v0.3.0 // indirect
|
|
github.com/avct/uasurfer v0.0.0-20191028135549-26b5daa857f1
|
|
github.com/beevik/etree v1.1.0 // indirect
|
|
github.com/blang/semver v3.5.1+incompatible
|
|
github.com/blevesearch/bleve v1.0.7
|
|
github.com/bmizerany/assert v0.0.0-20160611221934-b7ed37b82869 // indirect
|
|
github.com/codahale/hdrhistogram v0.0.0-20161010025455-3a0bb77429bd // indirect
|
|
github.com/corpix/uarand v0.1.1 // indirect
|
|
github.com/cznic/b v0.0.0-20181122101859-a26611c4d92d // indirect
|
|
github.com/cznic/mathutil v0.0.0-20181122101859-297441e03548 // indirect
|
|
github.com/cznic/strutil v0.0.0-20181122101858-275e90344537 // indirect
|
|
github.com/dgryski/dgoogauth v0.0.0-20190221195224-5a805980a5f3
|
|
github.com/disintegration/imaging v1.6.2
|
|
github.com/dyatlov/go-opengraph v0.0.0-20180429202543-816b6608b3c8
|
|
github.com/facebookgo/ensure v0.0.0-20200202191622-63f1cf65ac4c // indirect
|
|
github.com/facebookgo/stack v0.0.0-20160209184415-751773369052 // indirect
|
|
github.com/facebookgo/subset v0.0.0-20200203212716-c811ad88dec4 // indirect
|
|
github.com/fortytw2/leaktest v1.3.0 // indirect
|
|
github.com/francoispqt/gojay v1.2.13
|
|
github.com/fsnotify/fsnotify v1.4.9
|
|
github.com/getsentry/sentry-go v0.6.0
|
|
github.com/glycerine/go-unsnap-stream v0.0.0-20190901134440-81cf024a9e0a // indirect
|
|
github.com/go-asn1-ber/asn1-ber v1.4.1 // indirect
|
|
github.com/go-gorp/gorp v2.2.0+incompatible // indirect
|
|
github.com/go-sql-driver/mysql v1.5.0
|
|
github.com/golang/freetype v0.0.0-20170609003504-e2365dfdc4a0
|
|
github.com/golang/protobuf v1.4.2 // indirect
|
|
github.com/google/uuid v1.1.1 // indirect
|
|
github.com/gorilla/handlers v1.4.2
|
|
github.com/gorilla/mux v1.7.4
|
|
github.com/gorilla/schema v1.1.0
|
|
github.com/gorilla/websocket v1.4.2
|
|
github.com/gregjones/httpcache v0.0.0-20190611155906-901d90724c79 // indirect
|
|
github.com/h2non/go-is-svg v0.0.0-20160927212452-35e8c4b0612c
|
|
github.com/hako/durafmt v0.0.0-20191009132224-3f39dc1ed9f4
|
|
github.com/hashicorp/go-hclog v0.12.2
|
|
github.com/hashicorp/go-immutable-radix v1.2.0 // indirect
|
|
github.com/hashicorp/go-msgpack v1.1.5 // indirect
|
|
github.com/hashicorp/go-multierror v1.1.0 // indirect
|
|
github.com/hashicorp/go-plugin v1.2.2
|
|
github.com/hashicorp/go-sockaddr v1.0.2 // indirect
|
|
github.com/hashicorp/go-uuid v1.0.1 // indirect
|
|
github.com/hashicorp/golang-lru v0.5.4 // indirect
|
|
github.com/hashicorp/memberlist v0.2.2
|
|
github.com/hashicorp/yamux v0.0.0-20190923154419-df201c70410d // indirect
|
|
github.com/icrowley/fake v0.0.0-20180203215853-4178557ae428
|
|
github.com/jaytaylor/html2text v0.0.0-20200412013138-3577fbdbcff7
|
|
github.com/jmhodges/levigo v1.0.0 // indirect
|
|
github.com/jmoiron/sqlx v1.2.0
|
|
github.com/jonboulle/clockwork v0.1.0 // indirect
|
|
github.com/konsorten/go-windows-terminal-sequences v1.0.2 // indirect
|
|
github.com/lib/pq v1.4.0
|
|
github.com/magiconair/properties v1.8.1 // indirect
|
|
github.com/mailru/easyjson v0.7.1 // indirect
|
|
github.com/mattermost/go-i18n v1.11.0
|
|
github.com/mattermost/gorp v2.0.1-0.20200527092429-d62b7b9cadfc+incompatible
|
|
github.com/mattermost/gosaml2 v0.3.2
|
|
github.com/mattermost/ldap v0.0.0-20191128190019-9f62ba4b8d4d
|
|
github.com/mattermost/rsc v0.0.0-20160330161541-bbaefb05eaa0
|
|
github.com/mattermost/viper v1.0.4
|
|
github.com/mattn/go-colorable v0.1.6 // indirect
|
|
github.com/mattn/go-runewidth v0.0.9 // indirect
|
|
github.com/mattn/go-sqlite3 v2.0.3+incompatible
|
|
github.com/miekg/dns v1.1.29 // indirect
|
|
github.com/minio/minio-go/v6 v6.0.55
|
|
github.com/mitchellh/go-testing-interface v1.14.1 // indirect
|
|
github.com/mitchellh/mapstructure v1.2.3 // indirect
|
|
github.com/muesli/smartcrop v0.3.0 // indirect
|
|
github.com/oklog/run v1.1.0 // indirect
|
|
github.com/olekukonko/tablewriter v0.0.4 // indirect
|
|
github.com/olivere/elastic v6.2.30+incompatible // indirect
|
|
github.com/onsi/ginkgo v1.12.0 // indirect
|
|
github.com/onsi/gomega v1.9.0 // indirect
|
|
github.com/opentracing/opentracing-go v1.1.0
|
|
github.com/pborman/uuid v1.2.0
|
|
github.com/pelletier/go-toml v1.7.0 // indirect
|
|
github.com/pkg/errors v0.9.1
|
|
github.com/poy/onpar v1.0.0 // indirect
|
|
github.com/prometheus/client_golang v1.5.1
|
|
github.com/prometheus/client_model v0.2.0 // indirect
|
|
github.com/prometheus/procfs v0.0.11 // indirect
|
|
github.com/remyoudompheng/bigfft v0.0.0-20190728182440-6a916e37a237 // indirect
|
|
github.com/rs/cors v1.7.0
|
|
github.com/rudderlabs/analytics-go v3.2.1+incompatible
|
|
github.com/russellhaering/goxmldsig v0.0.0-20180430223755-7acd5e4a6ef7
|
|
github.com/rwcarlsen/goexif v0.0.0-20190401172101-9e8deecbddbd
|
|
github.com/segmentio/analytics-go v3.1.0+incompatible
|
|
github.com/segmentio/backo-go v0.0.0-20200129164019-23eae7c10bd3 // indirect
|
|
github.com/sirupsen/logrus v1.5.0
|
|
github.com/smartystreets/assertions v1.0.0 // indirect
|
|
github.com/spf13/afero v1.2.2 // indirect
|
|
github.com/spf13/cast v1.3.1 // indirect
|
|
github.com/spf13/cobra v1.0.0
|
|
github.com/spf13/jwalterweatherman v1.1.0 // indirect
|
|
github.com/spf13/pflag v1.0.5 // indirect
|
|
github.com/ssor/bom v0.0.0-20170718123548-6386211fdfcf // indirect
|
|
github.com/stretchr/objx v0.2.0 // indirect
|
|
github.com/stretchr/testify v1.5.1
|
|
github.com/tecbot/gorocksdb v0.0.0-20191217155057-f0fad39f321c // indirect
|
|
github.com/throttled/throttled v2.2.4+incompatible
|
|
github.com/tinylib/msgp v1.1.2 // indirect
|
|
github.com/tylerb/graceful v1.2.15
|
|
github.com/uber/jaeger-client-go v2.23.0+incompatible
|
|
github.com/uber/jaeger-lib v2.2.0+incompatible
|
|
github.com/vmihailenco/msgpack/v5 v5.0.0-beta.1
|
|
github.com/wiggin77/logr v1.0.4
|
|
github.com/wiggin77/merror v1.0.2
|
|
github.com/wiggin77/srslog v1.0.1
|
|
github.com/xtgo/uuid v0.0.0-20140804021211-a0b114877d4c // indirect
|
|
github.com/ziutek/mymysql v1.5.4 // indirect
|
|
go.uber.org/zap v1.15.0
|
|
golang.org/x/crypto v0.0.0-20200429183012-4b2356b1ed79
|
|
golang.org/x/image v0.0.0-20200119044424-58c23975cae1
|
|
golang.org/x/lint v0.0.0-20200302205851-738671d3881b // indirect
|
|
golang.org/x/mod v0.3.0 // indirect
|
|
golang.org/x/net v0.0.0-20200501053045-e0ff5e5a1de5
|
|
golang.org/x/sys v0.0.0-20200515095857-1151b9dac4a9 // indirect
|
|
golang.org/x/text v0.3.2
|
|
golang.org/x/tools v0.0.0-20200612220849-54c614fe050c
|
|
google.golang.org/genproto v0.0.0-20200424135956-bca184e23272 // indirect
|
|
google.golang.org/grpc v1.29.1 // indirect
|
|
gopkg.in/alexcesaro/quotedprintable.v3 v3.0.0-20150716171945-2caba252f4dc // indirect
|
|
gopkg.in/ini.v1 v1.55.0 // indirect
|
|
gopkg.in/mail.v2 v2.3.1
|
|
gopkg.in/natefinch/lumberjack.v2 v2.0.0
|
|
gopkg.in/olivere/elastic.v6 v6.2.30
|
|
gopkg.in/yaml.v2 v2.2.8
|
|
honnef.co/go/tools v0.0.1-2020.1.3 // indirect
|
|
willnorris.com/go/imageproxy v0.10.0
|
|
)
|