undefined method `closed?' for nil:NilClass on find object

[gerner]

Last night one of our batch processing systems pulled in 0.3.15 for the first time and failed while downloading a file from s3 with the following stack trace.

NoMethodError: undefined method `closed?' for nil:NilClass
        from /usr/lib/ruby/1.9.1/net/http.rb:2457:in `stream_check'
        from /usr/lib/ruby/1.9.1/net/http.rb:2377:in `read_body'
        from /usr/lib/ruby/1.9.1/net/http.rb:2404:in `body'
        from /home/hadoop/resources/gems/ruby/1.9.1/gems/s3-0.3.15/lib/s3/object.rb:268:in `parse_headers'
        from /home/hadoop/resources/gems/ruby/1.9.1/gems/s3-0.3.15/lib/s3/object.rb:197:in `object_headers'
        from /home/hadoop/resources/gems/ruby/1.9.1/gems/s3-0.3.15/lib/s3/object.rb:61:in `retrieve'
        from /home/hadoop/resources/gems/ruby/1.9.1/gems/s3-0.3.15/lib/s3/objects_extension.rb:12:in `find_first'

For us, this is 100% repro. Here is the code I'm using:

require 's3'
service = S3::Service.new(:access_key_id => "your_aws_access_id",
                          :secret_access_key => "your_secret_key")
bucket = service.buckets.find("bucket")
object = bucket.objects.find("some/object/key")

here's my Gemfile:

source 'http://rubygems.org'
gem 's3', '0.3.15'

GEM
  remote: http://rubygems.org/
  specs:
    proxies (0.2.1)
    s3 (0.3.15)
      proxies (~> 0.2.0)

PLATFORMS
  ruby

DEPENDENCIES
  s3 (= 0.3.15)

[gerner]

just to be super clear: if you repeat these steps with 0.3.13 you don't get the exception

are these methods deprecated or not supported? it seems like this is core functionality and would be in some core regression tests. Is there some other way one ought to fetch content from s3?

[qoobaa]

Sure, it seems to be caused by recent changes with streaming. Streaming seems to not work properly, can you lock your project on 0.3.13 for now?

[ghost]

+1 here. Was about to log the same bug.

[qoobaa]

Reverted streams support in 0.3.16.

Sorry for that, but it's difficult to test S3 without actual S3 interaction. AFAIK they don't have any free option for testing, so I'd need to pay just for running tests. It sucks.

[ghost]

http://aws.amazon.com/free/
http://docs.aws.amazon.com/gettingstarted/latest/awsgsg-intro/gsg-aws-free-tier.html

[gerner]

What we do to test this kind of interaction with AWS is pull creds from environment variables. As @marinerJB suggests there's a free tier for AWS. Using these two approaches you can get free S3 access and avoid having creds in your source code.

it might also be reasonable to have a test suite that you only run before submitting a new release (especially a patch release). In general, I'm surprised to see new functionality in a path release. Any reason you didn't bump to 0.4 minor version for streams support? This would have saved us (we're locked into minor versions, but open to patch releases)

[qoobaa]

True.

I generally try to maintain the gem, but I'm not using it since 2 years or so. It's probably the only work that I've done that is still being used, so I guess I should put some more attention to it.

About the free tier - I already have an account on AWS (it's from 2009). I'm not able to start a free tier any more. Moreover Amazon blocked some services on my account, I have no idea why. I'm not able to access CloudFront for instance, because "my subscription is being processed". For that reason I moved away from AWS recently.