Dealing with large chunks of data

Before you do that, try doing as much of the calculating as possible in SQL. I had a similar problem to yours recently, and brought the statistics generation time down from 45 seconds to 0.6 by letting the database do the computation.