This is the initial release of aws/s3, a ruby library for Amazon's Simple Store
Service's (S3) REST API.
It implements the entire REST API aiming to be simple, comprehensive and
extensible.
Aside from the basics, it supports ACLs (reading and writing), BitTorrent
downloads and bucket access logging (enabling, configuring, fetching). Data can
be streamed both up and down.
Installation instructions, a tutorial and API documentation can be found here:
http://amazon.rubyforge.org
This is the initial release, but it has comprehensive test coverage and we
are using it in production at 37signals.
Looks really cool! I’ve checked out a few of the S3 libs that have been released recently, and I like structure you’ve come up with…
Quick question… I tried swapping out some S3 code in an app to use aws/s3, and for some reason I’m having routing difficulties in my app now. My namespaced controllers aren’t being loaded, for some reason. Is there any obvious reason why this would happen? (and, yes, I know namespaces are on the top ten don’t-do things. I may eliminate them, but for now they make sense.)
Looks really cool! I've checked out a few of the S3 libs that have been
released recently, and I like structure you've come up with...
Hey Matt, glad you like it so far.
Quick question... I tried swapping out some S3 code in an app to use aws/s3,
and for some reason I'm having routing difficulties in my app now. My
namespaced controllers aren't being loaded, for some reason. Is there any
obvious reason why this would happen? (and, yes, I know namespaces are on
the top ten don't-do things. I may eliminate them, but for now they make
sense.)
Nothing comes to mind just from what you are describing. A stack trace would
likely help, or a quick summary of how you plugged aws/s3 into your
application (via rubygems or by putting it in the vendor/plugins directory)
as well as what namedspaced controllers you have that aren't being loaded.
When it comes to things not being loaded and/or routes Nicholas Seckar is the expert. I'm
not too familiar with the subtleties of it these days.
I’ve tried it both ways, via RubyGems and via svn:externals…
This is truly bizarre… At this point I’m assuming that I must be doing something wrong. I’m trying to set the base connection info from environment.rb so that it’s available within my app, as well as within Backgroundrb.
in environment.rb, all of my controllers that are namespaced (i.e. Account::IndexController < Account:ActionController) stop being recognized. For instance, “account/index” will now render “index”. Any errors that I’m getting are just routing recognition errors, so I don’t have much of a stack trace to show.
I’d be interested to see if anyone else can duplicate this. It’s just so weird that I can’t believe that it’s the fault of aws-s3. I’ll keep looking at it and see if I can find anything obvious.
Looks like the S3 library extends ActiveSupport (const_missing_not_from_s3_library)? I’m not a ruby guru (sigh), so it’s not immediately apparent to me how this could affect things, but I suspect that this is the culprit…
S3 does alias and redefine const_missing but it chains back to the original
if its particular use case doesn't apply to the given constant and then just
does what ActiveSupport would have done anyway, which is reflected in that
strack trace as you can see eventually the ActiveSupport version gets called.
Just to sanity check, you could go into lib/aws/s3/extensions.rb and remove
the definition to const_missing_from_s3_library and the two calls to
alias_method below it. I threw in that const missing recently just to add a
tiny convenience to the API. What it provides isn't even documented and the
library doesn't depend on it so you can safely take it away. I probably will
remove it. I was just trying it on to see how it felt. But either way, my
const_missing shouldn't be getting in the way since it's chaining back to the
original.
From looking around at the s33r code for a bit, it appears to be one of the
more complete S3 libraries I've seen. It supports similar functionality to
aws/s3. The differences are mostly in the way the library is decomposed and
the API it exposes to you, its client. Sort of like, what's the difference
between Ruby and Python? In some ways not a lot. It comes back to what people
prefer. What they find natural and comfortable.
At a glance s33r seems like a fine library. It would be up to you to determine which is
a better fit for you. I'd encourage you to try them both out.
For some whacky reason it has to do with your extension to String:
class String
def underscore
gsub(/([A-Z]+)([A-Z][a-z])/,‘\1_\2’).
gsub(/([a-z\d])([A-Z])/,‘\1_\2’).
downcase
end
end
If I get rid of that, I’m able to get at my controllers.
Basically what I’ve been doing is to load the dev console, test that I can access the controllers (i.e. Account::AccountController), then require the s3/extensions.rb file manually, and then see what happens when I require another namespaced controller (
i.e. Account::ArtistController). With the library as it is, once I require the extensions file I can no longer get at my controllers, and it fails with load_missing_constant throwing “uninitialized constant”. If I get rid of the extension to String, it doesn’t fail anymore and I get get at all of my controllers even after requiring extensions.
So, it doesn’t appear to be your const_missing_from_s3_library function (which is very cool, by the way! I’ll be sure to use that. I don’t think it made it into the docs, did it?), but is instead the underscore function on String.
I believe that it is overriding the underscore extension to string in ActiveSupport::CoreExtensions::String::Inflections#underscore. Since Inflector is used to derive the controller file names for inclusion, I think this is what’s causing the problem.
Ah, that makes sense. I don't bundle ActiveSupport with the library, but I do
define an underscore method which ActiveSupport also defines, except mine is
simpler and doesn't take into account all the stuff that ActiveSupport's does
because it's not intended to to do all the constant name manipulation that
ActiveSupport's does. It slipped my mind that this would mess up
ActiveSupport's underscore method. The fix should be easy, I'll just do this:
def underscore
# ...
end unless public_instance_methods.include? 'underscore'
Thanks for tracking that down. That's my bad. Sorry about that. I'll be more
mindful of that kind of thing when I extend built in classes. I tried to keep
my extensions to a minimum.
One thing that S33r deals with, and I had dealt with in my previous hacky S3 implementation, is the small block size (1k, I believe) that Net::HTTP uses when streaming requests. Unfortunately it’s hard-coded in so you can’t set it yourself. Apparently this causes requests to be extremely CPU-bound, and can lead to unreasonable CPU usage when streaming files. S33r has a fix in there to override the block size used by Net::HTTP, and allows you to set it yourself. Perhaps this would be a useful extension? I didn’t see it in there anywhere, but it from what I’ve read it’s a pretty easy way to boost the performance when sending files.
Yeah, I noticed earlier while reading around in the code that he opened up
Net::HTTP and redefined that code to allow for customization of the block
size. Others have also bemoaned unfortunately small default sizes in
Net::HTTP.
Thanks for pointing that out. It seems like a sensible change to make.
Question about the service itself which I have been contemplating. I
want to utilize a service such as S3 seeing as the cost for large
amounts of bandwidth far exceed even the best prices I have seen from
hosting providers. My concern is that I will still have to take on the
initial burden of having to first upload the file to my servers, then
stream it to S3. Is there a way to directly upload to S3 without
having to spend bandwidth on my servers only to relay it to theirs?
AFAIK, there is not. All files loaded onto S3 have to be authenticated, for obvious reasons, so all transmission to S3 should be done server-side. Otherwise, I suppose you could send them directly to S3 client-side with Flash or something, but you don’t want to ever be in a position where your AWS authorization stuff is available to a client machine.
