Use a join table Contacts with user_id and contact_id fields. Then use a has_many :through association to find a user's contacts.
Also, check out this post and the comments that follow: http://blog.hasmanythrough.com/2006/4/21/self-referential-through
Hope this helps,