Some people like to only deal with one package manager (gem is just portage for Ruby really). However, Portage is probably going to be a bit behind the gem version, and there will probably be some gems that just aren't available in Portage.
I guess it really just depends on how big a fan of Portage you are. A sane strategy might be to just install Ruby and Rubygems from Portage and then do everything else from gems. Ymmv.