Can only get RMagick to work writing with 777

Hi -

I am at a loss with chmod. The only way I can get RMagick to work properly writing to a file is if I set the dir to 777 permissions, but I feel like this is a huge security risk. Can someone give me a push in the right direction with chown/chmod, etc. for RMagick?

All I want is for everyone to be able to upload images, and then my app processes and saves those images in a directory.

Thanks for any help, Dino