self managed recurring billing

I'm working on setting up the credit card processing part of a rails app
for my company and running into some issues. I'm currently working
with the ActiveMerchant plugin and connecting to our account at
Authorize.net. Everything's been relatively easy to setup, until I got
to the recurring billing part. The problems I'm finding are...

A. ActiveMerchant doesn't appear (yet?) to support Authorize.net's *new*
Automatic Recurring Billing (ARB) API.

B. ARB doesn't appear to directly notify our application about what
charges failed or succeeded, instead sending a daily email that a
person would have to read and then manually go take appropriate actions
on user accounts, which completely surprises me. I wrote a recurring
billing interface to Paypal in Perl over two years ago, and this was a
standard part of the interface for them all the way back then.

C. ARB doesn't allow the kind of control we need for deciding when and
how much to charge people. ARB wants us to setup a schedule to charge
card X Y times at Z frequency. However, our system would be a little
more akin to allowing users to use a card already on file for future
purchases without having to re-enter the card. The amounts and
frequencies are going to change a lot. We don't want to store the card
info on our sever, but would rather store in on the gateway and just
send them a key referencing the previously used card and tell them to,
"charge that one again for this new amount, and respond with the status."

Has anybody else run into any of these issues, and if so, how did you
deal with it. We're not married to ActiveMerchant or Authorize.net, so
if there are alternatives to either/both that would allow the kind of
control we need, I'd love to hear about them.

To summarize, we need to charge irregular amounts at irregular intervals
to a specific card keyed to a specific customer *without* storing the
credit card info on our server *or* requiring the customer to re-enter
their card info every time, and find out immediately (*not* via email)
if the charge succeeded or failed. Pipe dream?

Joe,

I wrote a ruby library for interacting with the recurring billing API
for TrustCommerce (www.trustcommerce.com). You could write one for
the ARB API. It's really not that hard. Take a look at
http://trustcommerce.rubyforge.org for the code. Maybe you can use it
as a template for building an ARB one...

I wrote about my experiences with using Paypal Payflow for Lighthouse:
http://activereload.net/2007/5/17/dealing-with-subscription-payments

Those basic subscription services are fine, except there's no way to
tell when a payment fails. You just have to watch the emails and
close accounts manually, or write something to parse the returned
emails. I'd love it if they had something similar to Paypal IPN.