has-many-through relationship - deleting DB rows

I have group memberships modeled in a HMT relationship. My problem is, when a member is removed from a group, the relationship is not deleted, but the userid is merely set to 0.

http://pastie.caboo.se/52837

Which makes me realize that I haven’t declared :dependent => :destroy in any of my relationships:

http://pastie.caboo.se/52842

Is this possible with HTM? Sure, I could have the relationship destroyed if either of the two objects are destroyed. But what I also need is for the relationship to be destroyed if either of the object leave it. There must be a common solution for this?

Thanks!