I struggled with this problem the other night and managed to come up with solution that allowed me to to use the code from my Sweeper instead of duplicating it in rake. I wrote a blog entry explaining what I did along with code samples.
http://stevencummings.name/2010/09/07/expire_fragment-with-rake-and-as-a-model-only-observer
