A lot of miscellaneous work #14

PeterScott · 2011-01-23T00:53:02Z

Nice project you've got! I made a bunch of changes here:

Merge in the wip-reconnect branch, which gives a big improvement in robustness.
Rename README and TUTORIAL to .md files, for GitHub user-friendliness, and expand the README.
Add a connection timeout. We can't do a general socket timeout, but we can add a timeout to the socket's initial connection attempt, which could be useful.
Get rid of the dependency on PyYaml, replacing it with a custom parser that's more compatible with beanstalkd, and about 30 times faster. Adjust documentation to match.
Add basic docstrings to all public methods.

There was also a brief attempt to make connections thread-local, but that patch was reverted. The Right Thing in this case is probably to keep thread-local connection pools, like they do over in redis-py.

Anyway, you've made what is easily the best beanstalkd client for Python, so thanks for that.

This further centralises socket-related errors to originate either from within Connection#_interact or Connection#connect.

`Connection#connect` is now a no-op for a non-closed connection. The closed property is not yet correctly maintained in the case of socket errors.

By making Connection inherit from threading.local, we can make a single connection object and use it from several threads, safely. This is enormously useful when using libraries like eventlet, and is common in similar libraries.

This reverts commit bd11204.

We can't use a general socket timeout because of our use of buffered socket I/O with socket.makefile(), but we can add a timeout to the part where we connect to the server, and remove the timeout once the connection succeeds.

We can get rid of the bothersome PyYaml dependency by using a very simple regexp-based parser, which happens to be a lot faster than a full YAML parser. It also allows us to work around some oddities in the way beanstalkd emits YAML: if a tube name looks like a number, the new parser will still parse it as a string, which is probably what users expect to happen.

The docstrings don't tell everything, and so there's a need for some more extensive documentation, but they serve as a basic guide to what each function does, which is very helpful.

Docstrings and README.md were wrapped at 70 columns, which was not consistent with TUTORIAL.md, and uncomfortably short.

Add a title at the top, and include a note about what exactly you can stick in a job body -- arbitrary bytes -- and how to make this play nice with Unicode.

The parser has a list of keys which it will not attempt to parse as numbers, in order to correctly parse messages from beanstalkd. The version, which may look like a number in the future, should be part of this list.

When using a connection with exponential or constant backoff, you should pass connect_automatically=False to the constructor, and call connect() manually. This may fail. If you then call connect() again, there will be a small delay before it attempts to reconnect. If it fails again, the next delay will be larger, and so on, exponentially. A successful connection resets this.

This behaves the same as the behavior in the main branch. Also, to simplify the API, it connects automatically iff the reconnect strategy is exp_backoff or constant.

earl · 2011-01-28T02:37:05Z

Peter, thanks a lot for this contribution. I'll review and handle it in separate parts over the next few days.

First up, the easiest part: the very documentation improvements are merged in 07a9b29.

earl · 2011-01-28T03:47:51Z

I merged the connection timeout parameter in cc22cef. This is a good idea, even independently of the reconnection stuff in wip-reconnect.

Note that I made two minor modifications:

I renamed the parameter to connect_timeout
Instead of a custom default timeout, we're now falling back on socket.getdefaulttimeout (which is typically None, unless set by the user)

arturhoo · 2011-08-10T13:56:36Z

Any updates on this?

…estarts.

… load.

dsully · 2012-02-24T22:24:13Z

I would also love to see these fixes merged.

Thanks.

dlo · 2012-03-22T18:32:33Z

Any update here?

earl · 2012-03-22T23:13:44Z

Dan and Dan, thanks for your comments. Is there anything specific you are particularly interested in seeing merged?

PeterScott · 2012-03-23T00:36:01Z

The thread-safety stuff is the part that I personally care about the most.

dsully · 2012-03-23T02:07:28Z

Thread-safety & reconnects for me.

dsully · 2012-12-13T18:21:27Z

Ping.

hurrycane · 2013-03-16T10:50:32Z

Hey Guys! I would love thread safety too!

hit9 · 2014-07-29T08:37:40Z

Ping

ultrabug · 2014-08-12T09:07:10Z

+1 also for reconnection / thread safety

@earl : is it because the PR is too big that you hesitate in going forward towards this ? how could we help you with it mate ?

Thanks

earl and others added 24 commits May 22, 2010 03:42

Move Job body reading to a central location

ce266cf

This further centralises socket-related errors to originate either from within Connection#_interact or Connection#connect.

Start to doctest our network error handling behaviour

c5bd1a5

Add closed property and adapt connect

d07fd8d

`Connection#connect` is now a no-op for a non-closed connection. The closed property is not yet correctly maintained in the case of socket errors.

Doctest SocketError

38836c0

Close the connection on socket errors

50cbe73

Make closing an already closed socket a no-op

c2a5449

Slightly speed up the network test

79a054e

Make the example worker do something midly useful

3bdb9d7

Document how to reconnect upon errors

40f07a3

Make connections thread-local.

bd11204

By making Connection inherit from threading.local, we can make a single connection object and use it from several threads, safely. This is enormously useful when using libraries like eventlet, and is common in similar libraries.

Convert README and TUTORIAL to .md files, for GitHub readability.

0583d71

Revert "Make connections thread-local."

928573d

This reverts commit bd11204.

Merge branch 'wip-reconnect'

05f6c3d

Made unit testing use TUTORIAL.md instead of the old name TUTORIAL.

561edec

Add connection timeout.

2fdab3a

We can't use a general socket timeout because of our use of buffered socket I/O with socket.makefile(), but we can add a timeout to the part where we connect to the server, and remove the timeout once the connection succeeds.

Add basic docstrings to all public methods.

1e4e9af

The docstrings don't tell everything, and so there's a need for some more extensive documentation, but they serve as a basic guide to what each function does, which is very helpful.

Expanded README file.

6162708

Switched from 70 to 80 column fill, for consistency.

f29a446

Docstrings and README.md were wrapped at 70 columns, which was not consistent with TUTORIAL.md, and uncomfortably short.

Mention the connection_timeout parameter in the tutorial.

cfb83b9

Expand tutorial slightly.

8ff3267

Add a title at the top, and include a note about what exactly you can stick in a job body -- arbitrary bytes -- and how to make this play nice with Unicode.

Add version to the parser's list of known non-numeric keys.

38b076d

The parser has a list of keys which it will not attempt to parse as numbers, in order to correctly parse messages from beanstalkd. The version, which may look like a number in the future, should be part of this list.

Added None as the default reconnect strategy.

a85aa7d

This behaves the same as the behavior in the main branch. Also, to simplify the API, it connects automatically iff the reconnect strategy is exp_backoff or constant.

PeterScott added 3 commits September 19, 2011 12:01

Improve docstrings and version bump.

54cae63

Merge remote branch 'remotes/origin/exponential-backoff'

15375f9

Improve tube handling. Version bump.

f4a88e4

PeterScott added 8 commits September 19, 2011 13:03

Get rid of git trickery in setup.py

cf73a13

Improve docstrings.

be6a7a1

Automatically do blocking reconnects, transparently handling server r…

231aa8f

…estarts.

Version bump.

36e95cb

Make connections safe to use from multiple threads.

8b5e7fc

Version bump

eb66a52

Fix tiny typo.

abcc4d4

Fix race condition that caused insertion into wrong queue under heavy…

8d648ab

… load.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

A lot of miscellaneous work #14

A lot of miscellaneous work #14

PeterScott commented Jan 23, 2011

earl commented Jan 28, 2011

earl commented Jan 28, 2011

arturhoo commented Aug 10, 2011

dsully commented Feb 24, 2012

dlo commented Mar 22, 2012

earl commented Mar 22, 2012

PeterScott commented Mar 23, 2012

dsully commented Mar 23, 2012

dsully commented Dec 13, 2012

hurrycane commented Mar 16, 2013

hit9 commented Jul 29, 2014

ultrabug commented Aug 12, 2014

A lot of miscellaneous work #14

Are you sure you want to change the base?

A lot of miscellaneous work #14

Conversation

PeterScott commented Jan 23, 2011

earl commented Jan 28, 2011

earl commented Jan 28, 2011

arturhoo commented Aug 10, 2011

dsully commented Feb 24, 2012

dlo commented Mar 22, 2012

earl commented Mar 22, 2012

PeterScott commented Mar 23, 2012

dsully commented Mar 23, 2012

dsully commented Dec 13, 2012

hurrycane commented Mar 16, 2013

hit9 commented Jul 29, 2014

ultrabug commented Aug 12, 2014