Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add Redis OBJECT command support #650

Closed
wants to merge 9 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
53 changes: 53 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -74,3 +74,56 @@ test_all
*.trs
tags

# netbeans project
nbproject
nbproject/*
*.bak
*bak

# debian packaging
debian/.debhelper/*
debian/.debhelper

# https://github.com/phusion/debian-packaging-for-the-modern-developer/blob/master/.gitignore
.DS_Store
.debhelper
*.debhelper
*.cache
*.cache/*
*.deb
*.dsc
*.build
*.buildinfo
*.changes
*.tar.gz
*.log
*.substvars
# Ignore generated files
/*.deb
/*.dsc
/*.changes
/*.build
/*.buildinfo
/*.tar.gz

# Ignore files generated during build
# https://github.com/vincentbernat/pragmatic-debian-packages/blob/master/.gitignore
debian/files
debian/*.substvars
debian/*.log
debian/.debhelper/*
debian/*.debhelper
debian/*-stamp
debian/*.after
debian/*.before
debian/nutcracker/*

/*/debian/files
/*/debian/*.substvars
/*/debian/*.log
/*/debian/.debhelper/*
/*/debian/*.debhelper
/*/debian/*-stamp
/*/debian/*.after
/*/debian/*.before
/*/debian/nutcracker/*
3 changes: 3 additions & 0 deletions .travis.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
language: c
script: bash ./travis.sh

2 changes: 1 addition & 1 deletion Makefile.am
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ MAINTAINERCLEANFILES = Makefile.in aclocal.m4 configure config.h.in config.h.in~

ACLOCAL_AMFLAGS = -I m4

SUBDIRS = contrib src
SUBDIRS = src

dist_man_MANS = man/nutcracker.8

Expand Down
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -70,7 +70,7 @@ The build steps are the same (`./configure; make; sudo make install`).
-D, --describe-stats : print stats description and exit
-v, --verbose=N : set logging level (default: 5, min: 0, max: 11)
-o, --output=S : set logging file (default: stderr)
-c, --conf-file=S : set configuration file (default: conf/nutcracker.yml)
-c, --conf-file=S : set configuration file (default: /etc/nutcracker/nutcracker.yml)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is backwards incompatible, only works on *nix-like systems, and is unrelated to new redis command support. Remove it.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

noted, thanks. I will revert it.

-s, --stats-port=N : set stats monitoring port (default: 22222)
-a, --stats-addr=S : set stats monitoring ip (default: 0.0.0.0)
-i, --stats-interval=N : set stats aggregation interval in msec (default: 30000 msec)
Expand Down Expand Up @@ -121,7 +121,7 @@ Twemproxy can be configured through a YAML file specified by the -c or --conf-fi
+ **servers**: A list of server address, port and weight (name:port:weight or ip:port:weight) for this server pool.


For example, the configuration file in [conf/nutcracker.yml](conf/nutcracker.yml), also shown below, configures 5 server pools with names - _alpha_, _beta_, _gamma_, _delta_ and omega. Clients that intend to send requests to one of the 10 servers in pool delta connect to port 22124 on 127.0.0.1. Clients that intend to send request to one of 2 servers in pool omega connect to unix path /tmp/gamma. Requests sent to pool alpha and omega have no timeout and might require timeout functionality to be implemented on the client side. On the other hand, requests sent to pool beta, gamma and delta timeout after 400 msec, 400 msec and 100 msec respectively when no response is received from the server. Of the 5 server pools, only pools alpha, gamma and delta are configured to use server ejection and hence are resilient to server failures. All the 5 server pools use ketama consistent hashing for key distribution with the key hasher for pools alpha, beta, gamma and delta set to fnv1a_64 while that for pool omega set to hsieh. Also only pool beta uses [nodes names](notes/recommendation.md#node-names-for-consistent-hashing) for consistent hashing, while pool alpha, gamma, delta and omega use 'host:port:weight' for consistent hashing. Finally, only pool alpha and beta can speak the redis protocol, while pool gamma, delta and omega speak memcached protocol.
For example, the configuration file in [/etc/nutcracker/nutcracker.yml](/etc/nutcracker/nutcracker.yml), also shown below, configures 5 server pools with names - _alpha_, _beta_, _gamma_, _delta_ and omega. Clients that intend to send requests to one of the 10 servers in pool delta connect to port 22124 on 127.0.0.1. Clients that intend to send request to one of 2 servers in pool omega connect to unix path /tmp/gamma. Requests sent to pool alpha and omega have no timeout and might require timeout functionality to be implemented on the client side. On the other hand, requests sent to pool beta, gamma and delta timeout after 400 msec, 400 msec and 100 msec respectively when no response is received from the server. Of the 5 server pools, only pools alpha, gamma and delta are configured to use server ejection and hence are resilient to server failures. All the 5 server pools use ketama consistent hashing for key distribution with the key hasher for pools alpha, beta, gamma and delta set to fnv1a_64 while that for pool omega set to hsieh. Also only pool beta uses [nodes names](notes/recommendation.md#node-names-for-consistent-hashing) for consistent hashing, while pool alpha, gamma, delta and omega use 'host:port:weight' for consistent hashing. Finally, only pool alpha and beta can speak the redis protocol, while pool gamma, delta and omega speak memcached protocol.

alpha:
listen: 127.0.0.1:22121
Expand Down
7 changes: 7 additions & 0 deletions conf/nutcracker.yml
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ alpha:
server_failure_limit: 1
servers:
- 127.0.0.1:6379:1
- 127.0.0.1:7379:1

beta:
listen: 127.0.0.1:22122
Expand All @@ -22,6 +23,7 @@ beta:
- 127.0.0.1:6381:1 server2
- 127.0.0.1:6382:1 server3
- 127.0.0.1:6383:1 server4
- 127.0.0.1:7379:1 server5

gamma:
listen: 127.0.0.1:22123
Expand All @@ -36,6 +38,8 @@ gamma:
servers:
- 127.0.0.1:11212:1
- 127.0.0.1:11213:1
- 127.0.0.1:11211:1
- 127.0.0.1:33211:1

delta:
listen: 127.0.0.1:22124
Expand All @@ -56,6 +60,7 @@ delta:
- 127.0.0.1:11221:1
- 127.0.0.1:11222:1
- 127.0.0.1:11223:1
- 127.0.0.1:33211:1

omega:
listen: /tmp/gamma
Expand All @@ -65,3 +70,5 @@ omega:
servers:
- 127.0.0.1:11214:100000
- 127.0.0.1:11215:1
- 127.0.0.1:11211:1
- 127.0.0.1:33211:1
7 changes: 0 additions & 7 deletions configure.ac
Original file line number Diff line number Diff line change
Expand Up @@ -197,15 +197,8 @@ AS_IF([test "x$disable_stats" = xyes],
[AC_DEFINE([HAVE_STATS], [1], [Define to 1 if stats is not disabled])])
AC_MSG_RESULT($disable_stats)

# Untar the yaml-0.2.5 in contrib/ before config.status is rerun
AC_CONFIG_COMMANDS_PRE([tar xvfz contrib/yaml-0.2.5.tar.gz -C contrib])

# Call yaml-0.2.5 ./configure recursively
AC_CONFIG_SUBDIRS([contrib/yaml-0.2.5])

# Define Makefiles
AC_CONFIG_FILES([Makefile
contrib/Makefile
src/Makefile
src/hashkit/Makefile
src/proto/Makefile
Expand Down
4 changes: 2 additions & 2 deletions contrib/Makefile.am
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
SUBDIRS = yaml-0.2.5
#SUBDIRS = yaml-0.2.5

EXTRA_DIST = yaml-0.2.5.tar.gz
#EXTRA_DIST = yaml-0.2.5.tar.gz
30 changes: 30 additions & 0 deletions debian/changelog
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
nutcracker (0.4.1+dfsg-1) unstable; urgency=medium
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Debian support is completely unrelated to new redis command support and should not be part of the same PR.

Additionally, I don't have any experience with debian packaging and wouldn't be able to review this, anyway - The debian package maintainers who originally wrote these files could continue to do so without these being checked in.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes. you may ignore all the debian folder


* New upstream release.
* Bump debhelper compat level to 10; remove dh-autoreconf B-D.
* Update Standards-Version to 3.9.8, no changes needed.
* Fix a few d/copyright structural errors.
* Update Vcs-Git/Vcs-Browser to point to https URLs and cgit.
* Add dependency on lsb-base for /lib/lsb/init-functions.
* Add hardening build flags (hardening=+all).
* Switch pidfile location from /var/run to /run.
* Update debian/patches/sysconfdir to only write a pidfile if --daemonize
was passed.
* Add systemd service file.
* Remove upstart job, dead and unmaintained code.

-- Faidon Liambotis <[email protected]> Sat, 17 Dec 2016 04:48:28 +0200

nutcracker (0.4.0+dfsg-1) unstable; urgency=medium

* New upstream release.
* Switch debian/watch to GitHub instead of Google Code, as this is
apparently where new tarballs are being shipped nowadays.

-- Faidon Liambotis <[email protected]> Tue, 21 Oct 2014 03:26:49 +0300

nutcracker (0.3.0+dfsg-1) unstable; urgency=medium

* Initial release. (Closes: #712107)

-- Faidon Liambotis <[email protected]> Mon, 06 Oct 2014 20:02:52 +0300
1 change: 1 addition & 0 deletions debian/compat
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
10
26 changes: 26 additions & 0 deletions debian/control
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
Source: nutcracker
Section: net
Priority: optional
Maintainer: Faidon Liambotis <[email protected]>
Build-Depends: debhelper (>= 10), libyaml-dev
Standards-Version: 3.9.8
Homepage: https://github.com/twitter/twemproxy
Vcs-Git: https://anonscm.debian.org/git/collab-maint/nutcracker.git
Vcs-Browser: https://anonscm.debian.org/cgit/collab-maint/nutcracker.git

Package: nutcracker
Architecture: any
Depends: ${shlibs:Depends}, ${misc:Depends}, adduser, lsb-base (>= 3.0-6)
Description: Fast, light-weight proxy for memcached and Redis
nutcracker, also known as twemproxy (pronounced "two-em-proxy"), is a
fast and lightweight proxy for the memcached and Redis protocols. It was
primarily built to reduce the connection count on backend caching
servers, but it has a number of features, such as:
* Maintains persistent server connections to backend servers.
* Enables pipelining of requests and responses.
* Supports multiple server pools simultaneously.
* Shard data automatically across multiple servers.
* Supports multiple hashing modes including consistent hashing and
distribution.
* High-availability by disabling nodes on failures.
* Observability through stats exposed on stats monitoring port.
143 changes: 143 additions & 0 deletions debian/copyright
Original file line number Diff line number Diff line change
@@ -0,0 +1,143 @@
Format: http://www.debian.org/doc/packaging-manuals/copyright-format/1.0/
Upstream-Name: nutcracker
Source: https://github.com/twitter/twemproxy

Files: *
Copyright: 2011, 2013 Twitter, Inc.
License: Apache-2.0

Files: debian/*
Copyright: 2014 Faidon Liambotis <[email protected]>
License: Apache-2.0

Files: src/nc_array.c src/nc_array.h src/nc_rbtree.c src/nc_rbtree.h
Copyright: 2002-2010 Igor Sysoev
2011 Twitter, Inc.
License: BSD-2-clause

Files: src/nc_queue.h
Copyright: 1991, 1993 The Regents of the University of California
2011 Twitter, Inc.
License: BSD-4-clause
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions
are met:
1. Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright
notice, this list of conditions and the following disclaimer in the
documentation and/or other materials provided with the distribution.
3. All advertising materials mentioning features or use of this software
must display the following acknowledgement:
This product includes software developed by the University of
California, Berkeley and its contributors.
4. Neither the name of the University nor the names of its contributors
may be used to endorse or promote products derived from this software
without specific prior written permission.
.
THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND
ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE
FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
SUCH DAMAGE.

Files: src/hashkit/*
Copyright: 2007-2010 TangentOrg (Brian Aker)
2011 Data Differential (http://datadifferential.com/)
2011 Twitter, Inc.
License: BSD-4-clause-TangentOrg
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are
met:
.
* Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
.
* Redistributions in binary form must reproduce the above
copyright notice, this list of conditions and the following disclaimer
in the documentation and/or other materials provided with the
distribution.
.
* Neither the name of TangentOrg nor the names of its
contributors may be used to endorse or promote products derived from
this software without specific prior written permission.
.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

Files: src/hashkit/nc_one_at_a_time.c
Copyright: 2009 Brian Aker
2011 Twitter, Inc.
License: Apache-2.0

Files: src/hashkit/nc_hsieh.c
Copyright: 2004, 2005, Paul Hsieh
2011 Twitter, Inc.
License: PaulHsieh-license
The derivative content includes raw computer source code, ideas, opinions,
and excerpts whose original source is covered under another license and
transformations of such derivatives. Note that mere excerpts by themselves
(with the exception of raw source code) are not considered derivative works
under this license. Use and redistribution is limited to the following
conditions:
* One may not create a derivative work which, in any way, violates the
Paul Hsieh exposition license described above on the original content.
* One may not apply a license to a derivative work that precludes anyone
else from using and redistributing derivative content.
* One may not attribute any derivative content to authors not involved in
the creation of the content, though an attribution to the author is not
necessary.
Comment: http://www.azillionmonkeys.com/qed/weblicense.html

License: BSD-2-clause
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions
are met:
1. Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright
notice, this list of conditions and the following disclaimer in the
documentation and/or other materials provided with the distribution.
.
THIS SOFTWARE IS PROVIDED BY AUTHOR AND CONTRIBUTORS ``AS IS'' AND
ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
ARE DISCLAIMED. IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
SUCH DAMAGE.

License: Apache-2.0
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
.
http://www.apache.org/licenses/LICENSE-2.0
.
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
.
On Debian systems, the complete text of the Apache version 2.0 license
can be found in "/usr/share/common-licenses/Apache-2.0".
2 changes: 2 additions & 0 deletions debian/default
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
# extra daemon options; see nutcracker --help
DAEMON_OPTS=""
2 changes: 2 additions & 0 deletions debian/dirs
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
etc/nutcracker
var/log/nutcracker
3 changes: 3 additions & 0 deletions debian/docs
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
README.md
notes/recommendation.md
notes/redis.md
1 change: 1 addition & 0 deletions debian/examples
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
debian/nutcracker.yml
Loading