Well, that is a pretty sticky problem. You'll have to decent to system level I guess. On debian systems you can call 'free' with returns the current memory usage. This command can of course be called in Ruby(and thus Rails) and then parsed to see how much memory there is still available. But I am not sure that will do you any good. As I remember from attachment_fu you can 'validate' the file size but only _after_ it been uploaded. So that won't help you. Perhaps javascript could intervene clientside?
Sample output free: $ free total used free shared buffers cached Mem: 239820 216752 23068 0 35112 47252 -/+ buffers/cache: 134388 105432 Swap: 498004 249752 248252 $