Like a modern code version of the mythical beast with 100 serpent heads, Typhoeus runs HTTP requests in parallel while cleanly encapsulating handling logic.
A single request:
Typhoeus.get("www.example.com", followlocation: true)Parallel requests:
hydra = Typhoeus::Hydra.new
10.times.map{ hydra.queue(Typhoeus::Request.new("www.example.com", followlocation: true)) }
hydra.rungem install typhoeus
gem "typhoeus"
- Documentation (GitHub master)
- Mailing list
The primary interface for Typhoeus is comprised of three classes: Request, Response, and Hydra. Request represents an HTTP request object, response represents an HTTP response, and Hydra manages making parallel HTTP connections.
request = Typhoeus::Request.new(
"www.example.com",
method: :post,
body: "this is a request body",
params: {field1: "a field"},
headers: {Accept: "text/html"}
)We can see from this that the first argument is the url. The second is a set of options. The options are all optional. The default for :method is :get. You can run the query either on its own or through the hydra:
request.run
#=> <Typhoeus::Response ... >hydra = Typhoeus::Hydra.hydra
hydra.queue(request)
hydra.runThe response object will be set after the request is run.
response = request.response
response.code
response.total_time
response.headers_has
response.bodyTyphoeus has some convenience methods for performing single HTTP requests. The arguments are the same as those you pass into the request constructor.
Typhoeus.get("www.example.com")
Typhoeus.head("www.example.com")
Typhoeus.put("www.example.com/posts/1", body: "whoo, a body")
Typhoeus.post("www.example.com/posts", body: { title: "test post", content: "this is my test"})
Typhoeus.delete("www.example.com/posts/1")You can query the response object to figure out if you had a successful request or not. Here’s some example code that you might use to handle errors.
request.on_complete do |response|
if response.success?
# hell yeah
elsif response.timed_out?
# aw hell no
log("got a time out")
elsif response.code == 0
# Could not get an http response, something's wrong.
log(response.curl_error_message)
else
# Received a non-successful http response.
log("HTTP request failed: " + response.code.to_s)
end
end
request.execute_callbacksThis also works with serial (blocking) requests in the same fashion. Both serial and parallel requests return a Response object.
A File object can be passed as a param for a POST request to handle uploading files to the server. Typhoeus will upload the file as the original file name and use Mime::Types to set the content type.
Typhoeus.post(
"http://localhost:3000/posts",
body: {
title: "test post",
content: "this is my test",
file: File.open("thesis.txt","r")
}
)Generally, you should be running requests through hydra. Here is how that looks
hydra = Typhoeus::Hydra.hydra
first_request = Typhoeus::Request.new("www.example.com/posts/1.json")
first_request.on_complete do |response|
third_request = Typhoeus::Request.new(www.example.com/posts/3.json)
hydra.queue third_request
end
second_request = Typhoeus::Request.new("www.example.com/posts/2.json")
hydra.queue first_request
hydra.queue second_request
# this is a blocking call that returns once all requests are complete
hydra.runThe execution of that code goes something like this. The first and second requests are built and queued. When hydra is run the first and second requests run in parallel. When the first request completes, the third request is then built and queued up. The moment it is queued Hydra starts executing it. Meanwhile the second request would continue to run (or it could have completed before the first). Once the third request is done, hydra.run returns.
Hydra will also handle how many requests you can make in parallel. Things will get flakey if you try to make too many requests at the same time. The built in limit is 200. When more requests than that are queued up, hydra will save them for later and start the requests as others are finished. You can raise or lower the concurrency limit through the Hydra constructor.
Typhoeus::Hydra.new(max_concurrency: 20)Hydra memoizes requests within a single run call. You have to enable memoization. This will result in a single request being issued. However, the on_complete handlers of both will be called.
Typhoeus::Config.memoize = true
hydra = Typhoeus::Hydra.new(max_concurrency: 1)
2.times do
hydra.queue Typhoeus::Request.new("www.example.com")
end
hydra.runThis will result in a two requests.
Typhoeus::Config.memoize = false
hydra = Typhoeus::Hydra.new(max_concurrency: 1)
2.times do
hydra.queue Typhoeus::Request.new("www.example.com")
end
hydra.runTyphoeus includes built in support for caching. In the following example, if there is a cache hit, the cached object is passed to the on_complete handler of the request object.
class Cache
attr_accessor :memory
def initialize
@memory = {}
end
def get(request)
memory[request]
end
def set(request, reseponse)
memory[request] = reseponse
end
end
Typhoeus::Config.cache = Cache.new
Typhoeus.get("www.example.com") == Typhoeus.get("www.example.com")
#=> trueHydra allows you to stub out specific urls and patterns to avoid hitting remote servers while testing.
response = Response.new(code: 200, body: "{'name' : 'paul'}")
Typhoeus.stub('www.example.com').and_return(response)
Typhoeus.get("www.example.com") == response
#=> trueThe queued request will hit the stub. You can also specify a regex to match urls.
response = Response.new(code: 200, body: "{'name' : 'paul'}")
Typhoeus.stub(/example/).and_return(response)
Typhoeus.get("www.example.com") == response
#=> trueNo exceptions are raised on HTTP timeouts. You can check whether a request timed out with the following methods:
Typhoeus.get("www.example.com").timed_out?Use followlocation: true, eg:
Typhoeus.get("www.example.com", followlocation: true)Typhoeus::Request.get("www.example.com", userpwd: "user:password")Typhoeus::Request.get("www.example.com", cookiefile: "/path/to/file", cookiejar: "/path/to/file")Here, cookiefile is a file to read cookies to send from, and cookiejar is a file to write received cookies to.
So, if you just want cookies enabled, you need to pass in a same filename in both options.
Are available and documented here
SSL comes built in to libcurl so it’s in Typhoeus as well. If you pass in a url with "https" it should just work assuming that you have your cert bundle in order and the server is verifiable. You must also have libcurl built with SSL support enabled. You can check that by doing this:
curl --version
Now, even if you have libcurl built with OpenSSL you may still have a messed up cert bundle or if you’re hitting a non-verifiable SSL server then you’ll have to disable peer verification to make SSL work. Like this:
Typhoeus.get("https://www.example.com", ssl_verifypeer: false)If you are getting "SSL: certificate subject name does not match target host name" from curl (ex:- you are trying to access to b.c.host.com when the certificate subject is *.host.com). You can disable host verification. Like this:
Typhoeus.get("https://www.example.com", ssl_verifyhost: 2)Sometime it’s useful to see verbose output from curl. You may now enable it on a per request basis:
Typhoeus.get("http://example.com", verbose: true)or global:
Typhoeus::Config.verbose = trueJust remember that libcurl prints it’s debug output to the console (to STDERR), so you’ll need to run your scripts from the console to see it.
Running the specs should be as easy as:
bundle install
bundle exec rake
(The MIT License)
Copyright © 2009-2010 Paul Dix
Copyright © 2011-2012 David Balatero
Copyright © 2012-2013 Hans Hasselberg
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.