You're passing the :popup options to the url options hash. You want to pass it as part of the html_options hash, like this:
<%= link_to document.name, {:controller => 'member', :action => :show_attachment, :id => document.id}, :popup => ['new window', 'width=800, height=700, left = 450, top = 100, resizable, scrollbars=yes' ] %>