aws/s3 0.1.0 (initial release)

This is the initial release of aws/s3, a ruby library for Amazon's Simple Store Service's (S3) REST API.

It implements the entire REST API aiming to be simple, comprehensive and extensible.

Aside from the basics, it supports ACLs (reading and writing), BitTorrent downloads and bucket access logging (enabling, configuring, fetching). Data can be streamed both up and down.

Installation instructions, a tutorial and API documentation can be found here:    This is the initial release, but it has comprehensive test coverage and we are using it in production at 37signals.

Thanks! marcel

Let me be the first to say thank you!

We're using S3 in one of our apps but the support was hacked in so I
look forward to checking this out with the hope it will improve our

Great stuff!


Looks really cool! I’ve checked out a few of the S3 libs that have been released recently, and I like structure you’ve come up with…

Quick question… I tried swapping out some S3 code in an app to use aws/s3, and for some reason I’m having routing difficulties in my app now. My namespaced controllers aren’t being loaded, for some reason. Is there any obvious reason why this would happen? (and, yes, I know namespaces are on the top ten don’t-do things. I may eliminate them, but for now they make sense.)


Looks really cool! I've checked out a few of the S3 libs that have been released recently, and I like structure you've come up with...

Hey Matt, glad you like it so far.

Quick question... I tried swapping out some S3 code in an app to use aws/s3, and for some reason I'm having routing difficulties in my app now. My namespaced controllers aren't being loaded, for some reason. Is there any obvious reason why this would happen? (and, yes, I know namespaces are on the top ten don't-do things. I may eliminate them, but for now they make sense.)

Nothing comes to mind just from what you are describing. A stack trace would likely help, or a quick summary of how you plugged aws/s3 into your application (via rubygems or by putting it in the vendor/plugins directory) as well as what namedspaced controllers you have that aren't being loaded.

When it comes to things not being loaded and/or routes Nicholas Seckar is the expert. I'm not too familiar with the subtleties of it these days.



I’ve tried it both ways, via RubyGems and via svn:externals…

This is truly bizarre… At this point I’m assuming that I must be doing something wrong. I’m trying to set the base connection info from environment.rb so that it’s available within my app, as well as within Backgroundrb.

As soon as I call

AWS::S3::Base.establish_connection!( :access_key_id => ‘my_id’, :secret_access_key => ‘my_key’


in environment.rb, all of my controllers that are namespaced (i.e. Account::IndexController < Account:ActionController) stop being recognized. For instance, “account/index” will now render “index”. Any errors that I’m getting are just routing recognition errors, so I don’t have much of a stack trace to show.

I’d be interested to see if anyone else can duplicate this. It’s just so weird that I can’t believe that it’s the fault of aws-s3. I’ll keep looking at it and see if I can find anything obvious.




I did notice one thing. Check out this stack trace:

NameError (uninitialized constant Account::ArtistController): /vendor/rails/activerecord/lib/…/…/activesupport/lib/active_support/dependencies.rb:260:in `load_missing_constant’

/vendor/rails/activerecord/lib/../../activesupport/lib/active_support/dependencies.rb:446:in `const_missing_not_from_s3_library'
/vendor/plugins/aws/lib/aws/s3/extensions.rb:111:in `const_missing'
/vendor/rails/activerecord/lib/../../activesupport/lib/active_support/inflector.rb:162:in `constantize'

/vendor/rails/activerecord/lib/../../activesupport/lib/active_support/core_ext/string/inflections.rb:148:in `constantize'
/vendor/rails/actionpack/lib/action_controller/routing.rb:1246:in `recognize'
/vendor/rails/railties/lib/dispatcher.rb:40:in `dispatch'

/usr/local/lib/ruby/gems/1.8/gems/mongrel-0.3.18/lib/mongrel/rails.rb:78:in `process'
/usr/local/lib/ruby/gems/1.8/gems/mongrel-0.3.18/lib/mongrel/rails.rb:76:in `process'

0.3.18/lib/mongrel.rb:587:in process_client' /usr/local/lib/ruby/gems/1.8/gems/mongrel-0.3.18/lib/mongrel.rb:586:in process_client’ /usr/local/lib/ruby/gems/1.8/gems/mongrel-0.3.18/lib/mongrel.rb:700:in `run’

/usr/local/lib/ruby/gems/1.8/gems/mongrel-0.3.18/lib/mongrel.rb:700:in `run'
/usr/local/lib/ruby/gems/1.8/gems/mongrel-0.3.18/lib/mongrel.rb:687:in `run'

/lib/mongrel/configurator.rb:271:in run' /usr/local/lib/ruby/gems/1.8/gems/mongrel-0.3.18/lib/mongrel/configurator.rb:270:in run’ /usr/local/lib/ruby/gems/1.8/gems/mongrel-0.3.18/bin/mongrel_rails:125:in `run’

/usr/local/lib/ruby/gems/1.8/gems/mongrel-0.3.18/lib/mongrel/command.rb:211:in `run'

Looks like the S3 library extends ActiveSupport (const_missing_not_from_s3_library)? I’m not a ruby guru (sigh), so it’s not immediately apparent to me how this could affect things, but I suspect that this is the culprit…


Thanks Marcel! I've been investigating S3 storage for my upcoming app. How does this library compare with s33r <;?


S3 does alias and redefine const_missing but it chains back to the original if its particular use case doesn't apply to the given constant and then just does what ActiveSupport would have done anyway, which is reflected in that strack trace as you can see eventually the ActiveSupport version gets called.

Just to sanity check, you could go into lib/aws/s3/extensions.rb and remove the definition to const_missing_from_s3_library and the two calls to alias_method below it. I threw in that const missing recently just to add a tiny convenience to the API. What it provides isn't even documented and the library doesn't depend on it so you can safely take it away. I probably will remove it. I was just trying it on to see how it felt. But either way, my const_missing shouldn't be getting in the way since it's chaining back to the original.


Hey Shane,

From looking around at the s33r code for a bit, it appears to be one of the more complete S3 libraries I've seen. It supports similar functionality to aws/s3. The differences are mostly in the way the library is decomposed and the API it exposes to you, its client. Sort of like, what's the difference between Ruby and Python? In some ways not a lot. It comes back to what people prefer. What they find natural and comfortable.

At a glance s33r seems like a fine library. It would be up to you to determine which is a better fit for you. I'd encourage you to try them both out.



For some whacky reason it has to do with your extension to String:

class String def underscore gsub(/([A-Z]+)([A-Z][a-z])/,‘\1_\2’). gsub(/([a-z\d])([A-Z])/,‘\1_\2’). downcase

end end

If I get rid of that, I’m able to get at my controllers.

Basically what I’ve been doing is to load the dev console, test that I can access the controllers (i.e. Account::AccountController), then require the s3/extensions.rb file manually, and then see what happens when I require another namespaced controller ( i.e. Account::ArtistController). With the library as it is, once I require the extensions file I can no longer get at my controllers, and it fails with load_missing_constant throwing “uninitialized constant”. If I get rid of the extension to String, it doesn’t fail anymore and I get get at all of my controllers even after requiring extensions.

So, it doesn’t appear to be your const_missing_from_s3_library function (which is very cool, by the way! I’ll be sure to use that. I don’t think it made it into the docs, did it?), but is instead the underscore function on String.

I believe that it is overriding the underscore extension to string in ActiveSupport::CoreExtensions::String::Inflections#underscore. Since Inflector is used to derive the controller file names for inclusion, I think this is what’s causing the problem.


Ah, that makes sense. I don't bundle ActiveSupport with the library, but I do define an underscore method which ActiveSupport also defines, except mine is simpler and doesn't take into account all the stuff that ActiveSupport's does because it's not intended to to do all the constant name manipulation that ActiveSupport's does. It slipped my mind that this would mess up ActiveSupport's underscore method. The fix should be easy, I'll just do this:

  def underscore     # ...   end unless public_instance_methods.include? 'underscore'

Thanks for tracking that down. That's my bad. Sorry about that. I'll be more mindful of that kind of thing when I extend built in classes. I tried to keep my extensions to a minimum.


Thanks again for tracking that down. I'm about to push a 0.1.1 point release because I think it's justifiable. I should have known better :slight_smile:



No worries, mate ;). I’m just glad I was able to track it down… A good ego-booster for a rubynewby such as myself.

I’ll update my copy when you get that checked in…




One quick observation about your implementation…

One thing that S33r deals with, and I had dealt with in my previous hacky S3 implementation, is the small block size (1k, I believe) that Net::HTTP uses when streaming requests. Unfortunately it’s hard-coded in so you can’t set it yourself. Apparently this causes requests to be extremely CPU-bound, and can lead to unreasonable CPU usage when streaming files. S33r has a fix in there to override the block size used by Net::HTTP, and allows you to set it yourself. Perhaps this would be a useful extension? I didn’t see it in there anywhere, but it from what I’ve read it’s a pretty easy way to boost the performance when sending files.



Yeah, I noticed earlier while reading around in the code that he opened up Net::HTTP and redefined that code to allow for customization of the block size. Others have also bemoaned unfortunately small default sizes in Net::HTTP.

Thanks for pointing that out. It seems like a sensible change to make.


Nice release with the S3 Marcel.

Question about the service itself which I have been contemplating. I want to utilize a service such as S3 seeing as the cost for large amounts of bandwidth far exceed even the best prices I have seen from hosting providers. My concern is that I will still have to take on the initial burden of having to first upload the file to my servers, then stream it to S3. Is there a way to directly upload to S3 without having to spend bandwidth on my servers only to relay it to theirs?

Thanks in advance. Nathaniel.


AFAIK, there is not. All files loaded onto S3 have to be authenticated, for obvious reasons, so all transmission to S3 should be done server-side. Otherwise, I suppose you could send them directly to S3 client-side with Flash or something, but you don’t want to ever be in a position where your AWS authorization stuff is available to a client machine.

If anyone else knows better, please enlighten me :wink: