Handling ActiveRecord::RecordNotFound errors

Can anyone explain why an ActiveRecord::RecordNotFound error is only
thrown on a regular id-based find, such as Product.find(123)? For
example, if you're using a dynamically generated attribute find method,
such as Product.find_by_name('asdfd'), and no results match, it just
returns nil, and no error is thrown.

That's expected. The reasoning being that if you look up a record by
id then you're expecting the record to be there, so if it's missing
that's an exceptional condition.
On the other hand, with a find :first (which is what find_by_xxx ends
up doing) or find :all, you don't have that expectation. For example
the user has decided to search for records matching a set of criteria
or something along those lines and so there not being any records to
return is a perfectly normal thing.
How you handle this is completely application dependant (it might be
as simple as rendering 'No results found')