I'm trying to decide if I should set up my own server at home or I should use a hosting company for my website. I'm inclined to use a hosting company at this point because I don't know have experience setting up a public facing server that is secure,
There's the rub. You *will* get in over your head. Not if, when. And while it will be a learning experience, you will pull your hair out and it will be generally very uncomfortable when something goes wrong.
plus I would need to upgrade my internet plan. There's also the issue of reliable power supply. However, what I'd gain is complete flexbility and lots of memory and cpu power.
You would also get these benefits from colocation, without having to multi-home Internet connections to your house and install backup generators 
In all seriousness, if you're interested in learning how to set up a secure, stable platform, consider buying a server and using it at home for development and testing. You can then throw it in a colo once you gain experience.
If I go with a hosting company(shared or VPS), the ongoing cost may be high, plus I would only have very little RAM and disk space. Some of these companies only offer 256MB of RAM. Is that enough for a MySQL-driven website? Would I have complete flexibility to install software such as various RoR gems, RMagick, TinyMce, memcached? I hope to have millions of users one day.
On a VPS, you'll generally have root access and you can install whatever you'd like.
As a general rule, if you're a developer, stick to what you're good at. I got into the hosting game for a while and it was just too much stress to handle. That's my $0.02.
Brad
