Merge pull request #5 from TriggerMail/dangermike/NOTICKET/benchmarks

Improved benchmarks
TriggerMail · May 24, 2021 · b420ce6 · b420ce6
2 parents 8632828 + dbce141
commit b420ce6
Show file tree

Hide file tree

Showing 14 changed files with 640 additions and 198 deletions.
diff --git a/bench/README.md b/bench/README.md
@@ -0,0 +1,50 @@
+# LazyLRU Benchmarking
+
+Because this implementation is designed for groups of keys that come in waves, a simple [testing benchmark ](https://golang.org/pkg/testing/#hdr-Benchmarks) that reads and writes random keys would not be an accurate representation of this library. For those kinds of general loads, [hashicorp/golang-lru](https://github.com/hashicorp/golang-lru) is just as good as LazyLRU when the cache is >25% full. So these benchmarks try to fill that gap.
+
+Benchmarking independently is interesting, but not as instructive. The candidates for all the tests were:
+
+* **null**: Do nothing. Don't save anything. All `get` operations are misses.
+* **mapcache.{hour|50ms}**: A map of `key => {value, expiration}`. If the map is full, items are dropped at random. The time indicates the expiration -- _50ms_ for expiring frequently relative to read/write operations, _hour_ for exprining infrequently relative to read/write operations.
+* **lazylru.{hour|50ms}**: The thing in this repo. The one we're here to test.
+* **hashicorp.lru**: This is the default implementation in the [hashicorp/golang-lru](https://pkg.go.dev/github.com/hashicorp/golang-lru?utm_source=godoc) package. This is the implementation based on [groupcache](https://github.com/golang/groupcache/blob/master/lru/lru.go). _This implementation does not support expiration._
+* **hashicorp.exp_{hour|50ms}**: This is the hashicorp.lru, but instead of storing raw values, we store `key => {value, expiration} ` like we did in the mapcache above. Expiry is checked on read and stale values are discarded.
+* **hashicorp.arc**: hashicorp's implementation of the [Adaptive Relay Cache](https://www.usenix.org/legacy/event/fast03/tech/full_papers/megiddo/megiddo.pdf). _This implementation does not support expiration._
+* **hashicorp.2Q**: hashicorp's implementation of the [multi-queue replacement algorithm](https://static.usenix.org/event/usenix01/full_papers/zhou/zhou.pdf). 
+
+These tests define sets of keys, then rotate through those sets. This is meant to simulate the waves of requests for a set of keys that would come as marketing sends run through a day. Tests have the following parameters: 
+
+* **algorithm**: What we are testing
+* **ranges**: How many ranges of keys are in the test
+* **keys/range**: How big each range is
+* **cycles/range**: How many times each range is read
+* **threads**: The number of concurrent reader/writer workers
+* **size**: Capacity of the cache under test
+* **work_time_µs**: On each operation, spin-wait to alleviate lock contention while not releasing the CPU
+* **sleep_time_µs**: On each operation, sleep to allevaite lock contention while yielding
+* **cycles**: How may read or write operations are in the test
+* **duration_ms**: How long the test took
+* **rate_kHz**: Cycles/duration
+* **hit_rate_%**: How efficient the cache was
+
+I ran 253 variations of these tests on a Google Cloud [n1-standard-8](https://cloud.google.com/compute/docs/machine-types#n1_machine_types) (8-core) VM running Go 1.16.4. I've included the [raw results](results.csv.gz) in this repo.
+
+* **Test A**: 5 ranges of 1000 keys, 1000000 cycles/range, size 10000, 1 thread, 0 work, 0 sleep.
+* **Test B**: 5 ranges of 1000 keys, 1000000 cycles/range, size 10000, 64 thread, 0 work, 0 sleep
+* **Test C**: 1 ranges of 20000 keys, 1000000 cycles/range, size 10000, 64 thread, 0 work, 0 sleep
+
+| Algorithm          | rate (kHz) | hit rate (%) | rate (kHz) | hit rate (%) | rate (kHz) | hit rate (%) |
+| ------------------ | ---------: | -----------: | ---------: | -----------: | ---------: | -----------: |
+|                    | **Test A** |   **Test A** | **Test B** |   **Test B** | **Test C** |   **Test C** |
+| null               |   18532.44 |         0.00 |    3755.14 |         0.00 |    3131.50 |         0.00 |
+| mapcache.hour      |    3177.00 |        99.90 |    1854.16 |        99.90 |     881.83 |        10.00 |
+| mapcache.50ms      |    3108.52 |        96.83 |    1654.61 |        94.20 |     865.27 |         8.20 |
+| lazylru.hour       |    4811.95 |        99.90 |    2719.89 |        99.90 |     462.65 |         9.96 |
+| lazylru.50ms       |    3977.11 |        97.50 |    1466.69 |        93.29 |     458.97 |         9.94 |
+| hashicorp.lru      |    3796.49 |        99.90 |    1457.54 |        99.90 |     696.47 |        10.00 |
+| hashicorp.exp_hour |    2627.22 |        99.90 |    1343.52 |        99.90 |     591.02 |         9.97 |
+| hashicorp.exp_50ms |    2616.85 |        99.90 |    1342.45 |        99.90 |     587.46 |        10.00 |
+| hashicorp.arc      |    3496.26 |        99.90 |    1455.67 |        99.90 |     338.93 |         9.95 |
+| hashicorp.2Q       |    3804.71 |        99.90 |    1476.98 |        99.90 |     357.44 |         9.97 |
+
+The reason that HashiCorp's implementations were used as the reference is that they are well done. For general purpose caching needs, they are hard to beat. However, the Tests A and B are a reasonable facsimilie of what we see in real life. And in that environment, it performs very well. In Test C, where the cache is undersized, LazyLRU worse than the regular LRU algorithm. The two "smart" algorithms, ARC and 2Q, shouldn't be expected to perform well in this test because of the random request pattern.
diff --git a/bench/go.mod b/bench/go.mod
@@ -0,0 +1,11 @@
+module github.com/TriggerMail/lazylru/bench
+
+go 1.16
+
+replace github.com/TriggerMail/lazylru => ../
+
+require (
+	github.com/TriggerMail/lazylru v0.0.0-00010101000000-000000000000
+	github.com/hashicorp/golang-lru v0.5.4
+	go.uber.org/zap v1.16.0
+)
diff --git a/bench/go.sum b/bench/go.sum
@@ -0,0 +1,57 @@
+github.com/BurntSushi/toml v0.3.1 h1:WXkYYl6Yr3qBf1K79EBnL4mak0OimBfB0XUf9Vl28OQ=
+github.com/BurntSushi/toml v0.3.1/go.mod h1:xHWCNGjB5oqiDr8zfno3MHue2Ht5sIBksp03qcyfWMU=
+github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
+github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
+github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
+github.com/google/renameio v0.1.0/go.mod h1:KWCgfxg9yswjAJkECMjeO8J8rahYeXnNhOm40UhjYkI=
+github.com/hashicorp/golang-lru v0.5.4 h1:YDjusn29QI/Das2iO9M0BHnIbxPeyuCHsjMW+lJfyTc=
+github.com/hashicorp/golang-lru v0.5.4/go.mod h1:iADmTwqILo4mZ8BN3D2Q6+9jd8WM5uGBxy+E8yxSoD4=
+github.com/kisielk/gotool v1.0.0/go.mod h1:XhKaO+MFFWcvkIS/tQcRk01m1F5IRFswLeQ+oQHNcck=
+github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo=
+github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ=
+github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI=
+github.com/pkg/errors v0.8.1 h1:iURUrRGxPUNPdy5/HRSm+Yj6okJ6UtLINN0Q9M4+h3I=
+github.com/pkg/errors v0.8.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
+github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
+github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
+github.com/rogpeppe/go-internal v1.3.0/go.mod h1:M8bDsm7K2OlrFYOpmOWEs/qY81heoFRclV5y23lUDJ4=
+github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
+github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
+github.com/stretchr/testify v1.4.0/go.mod h1:j7eGeouHqKxXV5pUuKE4zz7dFj8WfuZ+81PSLYec5m4=
+github.com/stretchr/testify v1.7.0 h1:nwc3DEeHmmLAfoZucVR881uASk0Mfjw8xYJ99tb5CcY=
+github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
+go.uber.org/atomic v1.6.0 h1:Ezj3JGmsOnG1MoRWQkPBsKLe9DwWD9QeXzTRzzldNVk=
+go.uber.org/atomic v1.6.0/go.mod h1:sABNBOSYdrvTF6hTgEIbc7YasKWGhgEQZyfxyTvoXHQ=
+go.uber.org/multierr v1.5.0 h1:KCa4XfM8CWFCpxXRGok+Q0SS/0XBhMDbHHGABQLvD2A=
+go.uber.org/multierr v1.5.0/go.mod h1:FeouvMocqHpRaaGuG9EjoKcStLC43Zu/fmqdUMPcKYU=
+go.uber.org/tools v0.0.0-20190618225709-2cfd321de3ee h1:0mgffUl7nfd+FpvXMVz4IDEaUSmT1ysygQC7qYo7sG4=
+go.uber.org/tools v0.0.0-20190618225709-2cfd321de3ee/go.mod h1:vJERXedbb3MVM5f9Ejo0C68/HhF8uaILCdgjnY+goOA=
+go.uber.org/zap v1.16.0 h1:uFRZXykJGK9lLY4HtgSw44DnIcAM+kRBP7x5m+NpAOM=
+go.uber.org/zap v1.16.0/go.mod h1:MA8QOfq0BHJwdXa996Y4dYkAqRKB8/1K1QMMZVaNZjQ=
+golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
+golang.org/x/crypto v0.0.0-20190510104115-cbcb75029529/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
+golang.org/x/lint v0.0.0-20190930215403-16217165b5de h1:5hukYrvBGR8/eNkX5mdUezrA6JiaEZDtJb9Ei+1LlBs=
+golang.org/x/lint v0.0.0-20190930215403-16217165b5de/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc=
+golang.org/x/mod v0.0.0-20190513183733-4bf6d317e70e/go.mod h1:mXi4GBBbnImb6dmsKGUJ2LatrhH/nqhxcFungHvyanc=
+golang.org/x/net v0.0.0-20190311183353-d8887717615a/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
+golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
+golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
+golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
+golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
+golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
+golang.org/x/tools v0.0.0-20190311212946-11955173bddd/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs=
+golang.org/x/tools v0.0.0-20190621195816-6e04913cbbac/go.mod h1:/rFqwRUd4F7ZHNgwSSTFct+R/Kf4OFW1sUzUTQQTgfc=
+golang.org/x/tools v0.0.0-20191029041327-9cc4af7d6b2c/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
+golang.org/x/tools v0.0.0-20191029190741-b9c20aec41a5 h1:hKsoRgsbwY1NafxrwTs+k64bikrLBkAgPir1TNCj3Zs=
+golang.org/x/tools v0.0.0-20191029190741-b9c20aec41a5/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
+golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
+gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
+gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
+gopkg.in/errgo.v2 v2.1.0/go.mod h1:hNsd1EY+bozCKY1Ytp96fpM3vjJbqLJn88ws8XvfDNI=
+gopkg.in/yaml.v2 v2.2.2 h1:ZCJp+EgiOT7lHqUV2J862kp8Qj64Jo6az82+3Td9dZw=
+gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
+gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c h1:dUUwHk2QECo/6vqA44rthZ8ie2QXMNeKRTHCNY2nXvo=
+gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
+honnef.co/go/tools v0.0.1-2019.2.3 h1:3JgtbtFHMiCmsznwGVTUWbgGov+pVqnlf1dEJTNAXeM=
+honnef.co/go/tools v0.0.1-2019.2.3/go.mod h1:a3bituU0lyd329TUQxRnasdCoJDkEUEAqEt0JzvZhAg=
diff --git a/bench/hashicorp_wrapper.go b/bench/hashicorp_wrapper.go
@@ -0,0 +1,113 @@
+package main
+
+import (
+	"time"
+
+	hlru "github.com/hashicorp/golang-lru"
+)
+
+type HashicorpWrapper struct {
+	cache *hlru.Cache
+}
+
+func NewHashicorpWrapper(size int) *HashicorpWrapper {
+	retval, err := hlru.New(size)
+	if err != nil {
+		panic(err)
+	}
+	return &HashicorpWrapper{retval}
+}
+
+func (c *HashicorpWrapper) Get(key string) (interface{}, bool) {
+	return c.cache.Get(key)
+}
+
+func (c *HashicorpWrapper) Set(key string, value interface{}) {
+	c.cache.Add(key, value)
+}
+
+func (c *HashicorpWrapper) Close() {
+	c.cache.Purge()
+}
+
+type HashicorpARCWrapper struct {
+	cache *hlru.ARCCache
+}
+
+func NewHashicorpARCWrapper(size int) *HashicorpARCWrapper {
+	retval, err := hlru.NewARC(size)
+	if err != nil {
+		panic(err)
+	}
+	return &HashicorpARCWrapper{retval}
+}
+
+func (c *HashicorpARCWrapper) Get(key string) (interface{}, bool) {
+	return c.cache.Get(key)
+}
+
+func (c *HashicorpARCWrapper) Set(key string, value interface{}) {
+	c.cache.Add(key, value)
+}
+
+func (c *HashicorpARCWrapper) Close() {
+	c.cache.Purge()
+}
+
+type Hashicorp2QWrapper struct {
+	cache *hlru.TwoQueueCache
+}
+
+func NewHashicorp2QWrapper(size int) *Hashicorp2QWrapper {
+	retval, err := hlru.New2Q(size)
+	if err != nil {
+		panic(err)
+	}
+	return &Hashicorp2QWrapper{retval}
+}
+
+func (c *Hashicorp2QWrapper) Get(key string) (interface{}, bool) {
+	return c.cache.Get(key)
+}
+
+func (c *Hashicorp2QWrapper) Set(key string, value interface{}) {
+	c.cache.Add(key, value)
+}
+
+func (c *Hashicorp2QWrapper) Close() {
+	c.cache.Purge()
+}
+
+type HashicorpWrapperExp struct {
+	cache *hlru.Cache
+	ttl   time.Duration
+}
+
+func NewHashicorpWrapperExp(size int, ttl time.Duration) *HashicorpWrapperExp {
+	retval, err := hlru.New(size)
+	if err != nil {
+		panic(err)
+	}
+	return &HashicorpWrapperExp{retval, ttl}
+}
+
+func (c *HashicorpWrapperExp) Get(key string) (interface{}, bool) {
+	ret, ok := c.cache.Get(key)
+	if !ok {
+		return nil, false
+	}
+	item, _ := ret.(mapCacheElement)
+	if item.expiration.Before(time.Now()) {
+		return nil, ok
+	}
+
+	return ret, ok
+}
+
+func (c *HashicorpWrapperExp) Set(key string, value interface{}) {
+	c.cache.Add(key, mapCacheElement{value: value, expiration: time.Now().Add(c.ttl)})
+}
+
+func (c *HashicorpWrapperExp) Close() {
+	c.cache.Purge()
+}