Introduction

ROR is a more cost-effective alternative to redis, uses RocksDB as the storage engine and can save about 2/3 of the cost.

ROR extends SWAP feature based on redis codebase, it is compatible with almost all redis commands (including lua and multi/exec) and redis replication (RDB + RESP).

Architecture

SWAP

ROR stores hot and cold data in redis and RocksDB respectively, and exchanges hot and cold data automatically:

• SWAP IN: If key is cold, ROR reads cold key from RocksDB to redis, and then processes the command.
• SWAP OUT: If maxmemory is reached, ROR picks least frequently used key(LFU), and then writes the key to RocksDB.

Replication

Redis can be the slave of ror by directly executing the slaveof command. ROR replication process is almost the same with redis, the only difference lies in RDB generation for cold data: RocksDB CHECKPOINT is obtained first, and then the cold data is scanned and converted into RDB format.

Getting Started with ROR

1. Running with Docker

1. Compilation Process

   git clone https://github.com/ctripcorp/Redis-On-Rocks.git
   cd Redis-On-Rocks
   # use Dockerfile in current dir to build images, image name is ror
   docker build -t ror .
   # run ror docker image, and interact with docker by bash
   docker run -it ror /bin/bash

2. Compilation from Source

1. Supported Platforms

   • Linux - CentOS
   • Linux - Ubuntu
   • macOS(Darwin)

2. Required Library Software

   • snappy-devel
   • zlib-devel
   • gflags-devel
   • libstdc++

3. Compilation Process

   • 3.1 Get the source code

   git clone https://github.com/ctripcorp/Redis-On-Rocks/

   • 3.2 Switch to the latest release version

   git tag          # Check the latest release tag (e.g., ror-1.2.4)
   git checkout TAG # Switch to the latest version (e.g., git checkout ror-1.2.4)

   • 3.3 Execute compilation

   yum install -y zlib-devel-1.2.11  \
       snappy-devel-1.1.8  \
       jemalloc-5.2.1  \
       procps-ng-3.3.17
   cd Redis-On-Rocks
   git submodule update --init --recursive
   make

Migration

From redis

RDB

1. ROR is compatible with redis rdb, so it can directly read redis rdb file. In ror disk swap node, when migrate data from ror to redis, just put rdb file in ror data dir, and start ror node. All data in rdb file, would be trear as cold data. When cold data is fetched, it would be loaded into memory as hot data.

To Redis

1. In ror disk swap node, when migrate data from ror to redis, just put rdb file in ror data dir, and start ror node. All data in rdb file, would be trear as cold data. When cold data is fetched, it would be loaded into memory as hot data.

Benchmark ROR Performance

Performance

1. Two scenarios of performance testing

   1. 100thd: 100 concurrency, no speed limit
   2. 10Kqps: 1000 concurrency, speed limit to 10000 QPS

2. Public cloud performance

   1. Cold refers to conf mammemory's size. Hot refers to data size exceeds maxmemory conf size.
   2. SIN-AWS & FRA-AWS (r6gd.4xlarge)

   Scenario\Program			xredis	ror
   Hot	Get	100thd	QPS=100344 Latency(mean,p99): 952 1804(us)	QPS=85566 Latency(mean,p99):1131 2135(us)
   		10Kqps	QPS=9962 Latency(mean,p99): 92 249(us)	QPS=9960 Latency(mean,p99):111 320(us)
   	HGETALL	100thd	QPS=81024 Latency(mean,p99):1190 2383(us)	QPS=70468 Latency(mean,p99):1366 2592(us)
   		10Kqps	QPS=9967 Latency(mean,p99): 119 404(us)	QPS=9965 Latency(mean,p99):125 362(us)
   	HGET	100thd	QPS=94653 Latency(mean,p99): 998 1937(us)	QPS=80700 Latency(mean,p99):1186 2272(us)
   		10Kqps	QPS=9965 Latency(mean,p99):95 191(us)	QPS=9967 Latency(mean,p99):97 228(us)
   Cold	Get	100thd	-	QPS=54457 Latency(mean,p99):1760 6814(us)
   		10Kqps	-	QPS=9959 Latency(mean,p99):111 356(us)
   	HGETALL	100thd	-	QPS=32565 Latency(mean,p99):2968 5732(us)
   		10Kqps	-	QPS=9965 Latency(mean,p99):163 524(us)
   	HGET	100thd	-	QPS=43512 Latency(mean,p99):2263 12751(us)
   		10Kqps	-	QPS=9966 Latency(mean,p99):118 472(us)

   3. ALI (ecs.si3r.4xlarge)

   Scenario\Program			xredis	ror
   Hot	Get	100thd	QPS=96679 Latency(mean,p99):1006 1267(us)	QPS=82631 Latency(mean,p99):1177 1651(us)
   		10Kqps	QPS=9983 Latency(mean,p99):92 204(us)	QPS=9983 Latency(mean,p99):112 250(us)
   	HGETALL	100thd	QPS=76308 Latency(mean,p99):1283 1917(us)	QPS=66645 Latency(mean,p99):1468 2146(us)
   		10Kqps	QPS=9988 Latency(mean,p99):122 279(us)	QPS=9988 Latency(mean,p99):139 294(us)
   	HGET	100thd	QPS=86230 Latency(mean,p99):1122 1610(us)	QPS=77459 Latency(mean,p99):1249 1716(us)
   		10Kqps	QPS=9995 Latency(mean,p99):114 269(us)	QPS=9988 Latency(mean,p99):126 259(us)
   Cold	Get	100thd	-	QPS=56408 Latency(mean,p99):1743 2710(us)
   		10Kqps	-	QPS=9983 Latency(mean,p99):200 706(us)
   	HGETALL	100thd	-	QPS=38082 Latency(mean,p99):2599 3791(us)
   		10Kqps	-	QPS=9991 Latency(mean,p99):148 442(us)
   	HGET	100thd	-	QPS=47821 Latency(mean,p99):2045 3202(us)
   		10Kqps	-	QPS=9996 Latency(mean,p99):177 891(us)

Configuration

Conf Options

1. maxmemory

   • Set a memory usage limit to the specified amount of bytes. When the memory limit is reached Redis will try to remove keys according to the eviction policy selected (see maxmemory-policy).If Redis can't remove keys according to the policy, or if the policy is set to 'noeviction', Redis will start to reply with errors to commands that would use more memory, like SET, LPUSH, and so on, and will continue to reply to read-only commands like GET.This option is usually useful when using Redis as an LRU or LFU cache, or to set a hard memory limit for an instance (using the 'noeviction' policy).

2. swap-mode

   • swap-mode controls how ror do swap, could be: disk or memory. disk: store cold keys on disk, hot keys in memory and do SWAP hot cold if needed. memory: disable swap feature, act as vanilla redis (not ready yet).

3. swap-max-db-size

   • maximum size of disk usage allowed. if disk usage execeeds the limit redis will reject DENYOOM commands. default is 0 (unlimited).

Detail Confs

1. Detailed Confs

Observability

1. keyspace

   # Keyspace
   db0:keys=2,evicts=20000,metas=0,expires=0,avg_ttl=0
   

   • db0: current db id
   • evicts: cold keys' num in current db, when data turns warm or deleted, evicts may -1; when data turns cold, evicts may +1
   • metas: when cold data swaps in, add code data's key to db's meta; when swap data gets deleted, delete db's meta
   • expires: stores expired keys count
   • avg_ttl: stores average ttl of key objects, for statistics
2. cpu

   # CPU
   used_cpu_sys:0.045921
   used_cpu_user:0.187862
   used_cpu_sys_children:0.001902
   used_cpu_user_children:0.000000
   used_cpu_sys_main_thread:0.043579
   used_cpu_user_main_thread:0.183033
   swap_main_thread_cpu_usage:1.60%
   swap_swap_threads_cpu_usage:0.00%
   swap_other_threads_cpu_usage:0.00%
   

   • used_cpu_sys: the total amount of time spent executing in kernel mode of current process
   • used_cpu_user: the total amount of time spent executing in user mode of current process
   • used_cpu_sys_children: the total amount of time spent executing in kernel mode of child process
   • used_cpu_user_children: the total amount of time spent executing in user mode of child process
   • used_cpu_sys_main_thread: the total amount of time spent executing in kernel mode of the calling thread
   • used_cpu_user_main_thread: the total amount of time spent executing in user mode of the calling thread
   • swap_main_thread_cpu_usage: cal main thread cpu usage according to /proc/pid
   • swap_swap_threads_cpu_usage: cal swap thread cpu usage according to /proc/pid/task/tid
   • swap_other_threads_cpu_usage: cal other threads cpu usage according to (process cpu usage-swap_main_thread_cpu_usage-swap_swap_threads_cpu_usage)