What version of RubyGems?
Are you getting your pugins with Git or Subversion?
Have you tried to ssh into another box on your network? Is that slow
also?
Are you on a wired or wireless connection?
Are you on a company LAN?
Have you tried installing gems or plugins from a different location
(maybe a Wi-Fi hotspot)?
I have seen odd connection issues in the past on networks with improper
DNS configurations.
You might what to install it with the verbose flag ( -v ) to see why
its taking so long. Also , with atleast gem <1.2 , passing the flags --
no-rdoc --no-ri to the gem install command would speed it up , you get
no documentation though.
Also you might want to ping the server itself to see if its just the
server.